Image processing method and device, electronic equipment and computer readable storage medium

Abstract

Embodiments of the present application disclose an image processing method and device, electronic equipment and computer readable storage medium; the embodiments of the present application obtain the initial image feature of the image to be processed in the spatial domain, and the initial image feature is subjected to frequency domain transformation, and the frequency domain feature is obtained, then, according to the frequency domain feature, the initial image feature is subjected to denoising processing, and the denoised image feature is obtained, then, the feature relationship between the plurality of channel features in the initial image feature is identified, and the feature sequence relationship data is obtained, then, according to the feature sequence relationship data, the initial image feature is subjected to feature adjustment, and the serialized image feature of the image to be processed is obtained, finally, based on the denoised image feature and the serialized image feature, the image resolution of the image to be processed is adjusted, and the target image is obtained; the scheme can effectively capture the long-range dependence between image structures while reducing the computational complexity, and improve the super-resolution effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically to an image processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Image super-resolution (SR) is a common computer vision task. Its main purpose is to recover a high-resolution (HR) image from a low-resolution (LR) image. This requires enlarging the image size while enhancing details, reducing blur, and recovering as much information as possible lost due to insufficient resolution. Image super-resolution has applications in many fields, such as real-time video communication, multimedia imaging and video, mobile photography, video inspection, medical imaging, satellite and remote sensing, and more.

[0003] Image super-resolution methods can be categorized into traditional methods and deep learning-based methods. Traditional methods mainly consist of interpolation methods and sparse coding methods, while deep learning methods primarily include convolutional neural network (CNN) methods and Transformer (a type of deep learning model) methods. In traditional methods, interpolation methods such as bilinear interpolation and bicubic convolution perform poorly, while many sparse coding methods rely on manually crafted priors. In deep learning-based methods, although CNNs surpass traditional methods in their ability to model complex data and are widely used in super-resolution, the limited receptive field of convolutional layers prevents them from learning long-range dependencies. The Transformer, based on a self-attention mechanism, can overcome this problem, but the quadratic computational complexity of self-attention limits the range of token selection, restricting their ability to fully utilize long-range dependencies and thus affecting super-resolution performance. Summary of the Invention

[0004] This application provides an image processing method, apparatus, electronic device, and computer-readable storage medium that can effectively capture long-range dependencies between image structures and improve super-resolution performance while reducing computational complexity.

[0005] This application provides an image processing method, including:

[0006] The initial image features of the image to be processed in the spatial domain are obtained, and the initial image features are transformed in the frequency domain to obtain the frequency domain features.

[0007] Based on the frequency domain features, the initial image features are denoised to obtain denoised image features;

[0008] The feature relationships between multiple channel features in the initial image features are identified to obtain feature sequence relationship data;

[0009] Based on the feature sequence relationship data, the initial image features are adjusted to obtain the serialized image features of the image to be processed;

[0010] Based on the denoised image features and the serialized image features, the image resolution of the image to be processed is adjusted to obtain the target image.

[0011] Accordingly, embodiments of this application provide an image processing apparatus, including:

[0012] The transformation unit is used to acquire the initial image features of the image to be processed in the spatial domain, and to perform frequency domain transformation on the initial image features to obtain frequency domain features;

[0013] A denoising unit is used to denoise the initial image features based on the frequency domain features to obtain denoised image features;

[0014] The recognition unit is used to recognize the feature relationships between multiple channel features in the initial image features to obtain feature sequence relationship data;

[0015] An adjustment unit is used to adjust the initial image features according to the feature sequence relationship data to obtain the serialized image features of the image to be processed;

[0016] The super-resolution unit is used to adjust the image resolution of the image to be processed based on the denoised image features and the serialized image features to obtain the target image.

[0017] In some embodiments, the transformation unit may be used to acquire the image to be processed, extract semantic features from the image to be processed in the spatial domain to obtain multiple channel features, and perform dimensionality reduction processing on the channel features to obtain the initial image features.

[0018] In some embodiments, the transformation unit may be specifically used to obtain attention parameters corresponding to the initial image features; fuse the attention parameters with the initial image features to obtain fused image features; and perform frequency domain transformation on the fused image features to obtain the frequency domain features.

[0019] In some embodiments, the transformation unit may be specifically used to fuse the channel features with the query parameters to obtain fused query features; fuse the channel features with the key parameters to obtain fused key features; fuse the channel features with the value parameters to obtain fused value features; and use the fused query features, the fused key features, and the fused value features as the fused image features.

[0020] In some embodiments, the transformation unit may be specifically used to perform frequency domain transformation on the fused query features to obtain frequency domain query features; perform frequency domain transformation on the fused key features to obtain frequency domain key features; perform frequency domain transformation on the fused value features to obtain frequency domain value features; and use the frequency domain query features, the frequency domain key features, and the frequency domain value features as the frequency domain features.

[0021] In some embodiments, the denoising unit may be specifically used to group the frequency domain features according to a preset frequency interval threshold to obtain grouped frequency domain features, wherein the grouped frequency domain features include frequency domain interval features corresponding to multiple frequency domain intervals, and the number of features of the frequency domain interval features is the same; to perform attention weighting on the grouped frequency domain features to obtain target frequency domain features, wherein the target frequency domain features indicate the feature relationship between the frequency domain interval features; and to perform denoising processing on the initial image features according to the target frequency domain features to obtain the denoised image features.

[0022] In some embodiments, the denoising unit may be specifically used to mark the position of the pixel corresponding to the frequency domain feature to obtain pixel position index data; determine the pixel frequency domain feature corresponding to the pixel from the frequency domain feature based on the pixel position index data; and group the pixel frequency domain feature according to the preset frequency interval threshold to obtain the grouped frequency domain feature, wherein the grouped frequency domain feature contains the position information of the pixel.

[0023] In some embodiments, the denoising unit may be specifically configured to: identify frequency domain features corresponding to a preset low-frequency interval threshold in the target frequency domain features to obtain low-frequency interval features, wherein the low-frequency interval features indicate noise features in the initial image features; remove the low-frequency interval features from the target frequency domain features to obtain denoised frequency domain features; reshape the denoised frequency domain features to obtain reshaped frequency domain features, wherein the reshaped frequency domain features have the same number of features as the initial image features; and perform spatial domain transformation on the reshaped frequency domain features to obtain the denoised image features.

[0024] In some embodiments, the recognition unit may be specifically configured to: recognize the feature relationships between channel features in the initial image features according to a first feature order of the channel features, to obtain first feature sequence relationship data corresponding to the channel features; adjust the first feature order to obtain a second feature order, wherein the first feature order and the second feature order are in reverse order; and recognize the feature relationships between channel features in the initial image features according to the second feature order, to obtain second feature sequence relationship data corresponding to the channel features.

[0025] In some embodiments, the adjustment unit may be specifically used to perform channel attention weighting based on the initial image features to obtain channel weights, wherein the channel weights indicate the importance of the channel features; perform spatial attention weighting based on the initial image features to obtain spatial position weights, wherein the spatial position weights indicate the importance of the pixel positions in the image to be processed; and perform feature adjustment on the initial image features according to the channel weights, the spatial position weights, and the feature sequence relationship data to obtain the serialized image features.

[0026] In some embodiments, the adjustment unit may be specifically used to fuse the spatial location weights with the initial image features to obtain fused spatial features, and to fuse the channel weights with the initial image features to obtain fused channel features; and to fuse the fused spatial features and the fused channel features according to the feature sequence relationship data to obtain the serialized image features.

[0027] In some embodiments, the adjustment unit may be specifically used to perform feature adjustment on the fused channel features according to the first feature sequence relationship data to obtain the adjusted first channel features; to perform feature fusion of the adjusted first channel features and the fused spatial features to obtain the fused first image features; and to perform feature adjustment on the fused first image features according to the second feature sequence relationship data to obtain the serialized image features.

[0028] In some embodiments, the super-resolution unit may be used to fuse the denoised image features and the serialized image features to obtain target image features; and based on the target image features, perform dimensionality upscaling on the image to be processed to obtain the target image.

[0029] Furthermore, this application also provides an electronic device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to execute the image processing method provided in this application.

[0030] Furthermore, embodiments of this application also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the image processing method provided in embodiments of this application.

[0031] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the image processing methods provided in embodiments of this application.

[0032] This application embodiment obtains the initial image features of the image to be processed in the spatial domain, and performs frequency domain transformation on the initial image features to obtain frequency domain features; then, based on the frequency domain features, the initial image features are denoised to obtain denoised image features; next, the feature relationships between multiple channel features in the initial image features are identified to obtain feature sequence relationship data; then, according to the feature sequence relationship data, the initial image features are adjusted to obtain serialized image features of the image to be processed; finally, based on the denoised image features and the serialized image features, the image resolution of the image to be processed is adjusted to obtain the target image. This scheme can transform the initial image features of the image to be processed into frequency domain features. Based on these frequency domain features, it can capture global information of the image, improving the ability to model long-range dependencies. Furthermore, it processes the frequency domain features of different frequency components encoding different features (such as smooth regions, edges, textures, and noise) to obtain denoised image features, thus removing noise information from the image and avoiding noise interference in the super-resolution process. Additionally, the scheme identifies the feature relationships between multiple channel features in the initial image features and then adjusts these features to obtain serialized image features, ensuring that linear dependencies between channel features are preserved and enhancing the coherence of the channel feature sequence. Finally, the denoised image features and serialized image features are used as guiding conditions for super-resolution processing of the image to be processed, resulting in a high-resolution target image. This method can improve super-resolution performance while reducing computational complexity. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of an application scenario of the image processing method provided in the embodiments of this application;

[0035] Figure 2This is a schematic diagram illustrating another application scenario of the image processing method provided in the embodiments of this application;

[0036] Figure 3 This is an overall schematic diagram of the image processing method provided in the embodiments of this application;

[0037] Figure 4 This is a schematic flowchart of the image processing method provided in the embodiments of this application;

[0038] Figure 5 This is a schematic diagram illustrating the process of determining frequency domain features provided in an embodiment of this application;

[0039] Figure 6 This is a schematic diagram illustrating the grouping of frequency domain features according to an embodiment of this application;

[0040] Figure 7 This is a schematic diagram of the operation of spectral band reconstruction of frequency domain features provided in an embodiment of this application;

[0041] Figure 8 This is a schematic diagram illustrating the process of determining a target frequency domain feature according to an embodiment of this application;

[0042] Figure 9A This is a schematic diagram illustrating the determination of denoised image features according to an embodiment of this application;

[0043] Figure 9B This is another schematic diagram illustrating the determination of denoised image features provided in an embodiment of this application;

[0044] Figure 10A This is a schematic diagram of an image to be processed provided in an embodiment of this application;

[0045] Figure 10B This is a schematic diagram of another image to be processed provided in an embodiment of this application;

[0046] Figure 10C This is a schematic diagram of another image to be processed provided in an embodiment of this application;

[0047] Figure 11 This is a schematic diagram illustrating the determination of a feature sequence relationship data provided in an embodiment of this application;

[0048] Figure 12 This is a schematic diagram illustrating the determination of a serialized image feature provided in an embodiment of this application;

[0049] Figure 13 These are visual representations of images obtained using different super-resolution methods, as well as high-resolution native images.

[0050] Figure 14 This is an improved schematic diagram provided by an embodiment of this application to meet real-time requirements;

[0051] Figure 15 This is another schematic flowchart of the image processing method provided in the embodiments of this application;

[0052] Figure 16 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application;

[0053] Figure 17 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0054] The technical solutions described below, with reference to the accompanying drawings, will be clearly and completely explained. Obviously, the described embodiments are merely some, not all, of the embodiments described in this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0055] This application provides an image processing method, apparatus, electronic device, and computer-readable storage medium. The image processing apparatus can be integrated into an electronic device, which may be a server or a user terminal, etc.

[0056] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud-preset databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN) acceleration services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0057] Figure 1 A schematic diagram illustrating an application scenario of the image processing method provided in an embodiment of this application is shown. For example... Figure 1As shown, the image processing device can acquire a low-resolution image to be processed and process it to obtain a high-resolution target image. The source of the image to be processed can be varied; for example, it can be any image requiring super-resolution processing, or an image frame contained in a low-resolution video stream. Specifically, the image processing device can acquire the initial image features of the image to be processed in the spatial domain, and perform frequency domain transformation on the initial image features to obtain frequency domain features; based on the frequency domain features, perform denoising processing on the initial image features to obtain denoised image features; identify the feature relationships between multiple channel features in the initial image features to obtain feature sequence relationship data; adjust the initial image features according to the feature sequence relationship data to obtain serialized image features of the image to be processed; and based on the denoised image features and the serialized image features, adjust the image resolution of the image to be processed to obtain the target image.

[0058] Figure 2 This illustration shows another application scenario of the image processing method provided in this application. In real-time video communication (RTC) scenarios, there are high requirements for video clarity and real-time performance, which places high demands on cloud service providers. To ensure real-time performance and maintain low costs, video streams are often kept at low resolutions and bitrates, but this sacrifices user experience. For example, in cloud gaming scenarios, high-resolution native video brings high rendering and transmission pressure, but reducing resolution and bitrate leads to a decrease in game image quality. Therefore, as... Figure 2 As shown, the user terminal can establish a communication connection with the cloud device where the cloud service provider is located. The cloud device can provide high-definition video streams and low-definition video streams. The cloud device transmits the low-definition video stream to the user terminal. After receiving the low-definition video stream, the user terminal can use its computing power to perform super-resolution processing on the low-definition video stream to obtain a video stream with the same resolution as the high-definition video stream, and then render and display it on the terminal.

[0059] High-definition (HD) and low-definition (LD) video streams provided by cloud devices can be understood as video streams with different resolutions for the same video content. Both HD and LLD streams contain multiple video frames. For ease of distinction, the video frames in an HD stream are called HD video frames, and those in a LLD stream are called LLD video frames. HD video frames have a higher resolution than LLD video frames; for example, an HD video frame might have a resolution of 1080p, while a LLD video frame might have a resolution of 720p. Different video frame resolutions correspond to different numbers of bits transmitted per second (in Mbps). For example, a 1080p HD video frame transmits 4 Mbps, while a 720p LLD video frame transmits 2.5 Mbps.

[0060] It is understandable that the specific resolutions of high-definition and low-definition video streams can be other values, as long as the resolution of the high-definition video frame is greater than that of the low-definition video frame.

[0061] It is understood that in the specific embodiments of this application, images and other related data are involved. When the following embodiments of this application are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0062] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0063] This embodiment will be described from the perspective of an image processing device, which can be integrated into an electronic device, such as a server or a terminal. The terminal can include tablet computers, laptops, personal computers (PCs), wearable devices, virtual reality devices, or other smart devices that can generate image files.

[0064] An image processing method includes: acquiring initial image features of an image to be processed in the spatial domain, and performing frequency domain transformation on the initial image features to obtain frequency domain features; performing denoising processing on the initial image features based on the frequency domain features to obtain denoised image features; identifying the feature relationships between multiple channel features in the initial image features to obtain feature sequence relationship data; adjusting the initial image features according to the feature sequence relationship data to obtain serialized image features of the image to be processed; and adjusting the image resolution of the image to be processed based on the denoised image features and the serialized image features to obtain a target image.

[0065] Figure 3 A schematic diagram of an image processing method provided in an embodiment of this application is shown. Figure 3 As shown, the image to be processed is a low-resolution image requiring super-resolution processing. After acquiring the image, initial image features in the spatial domain can be extracted. The processing of these initial image features involves two branches: a spectral self-attention block and a bidirectional state-space model block. The initial image features are processed by both the spectral self-attention block and the bidirectional state-space model block, yielding the results of the two branches. These results are then fused, and based on the fused data and the initial image features, super-resolution processing is performed on the image to be processed, resulting in a high-resolution target image.

[0066] The processing method for the spectral self-attention block branch includes: performing frequency domain transformation on the initial image features to obtain frequency domain features; and performing denoising processing on the initial image features based on the frequency domain features to obtain denoised image features.

[0067] The processing method for the bidirectional state-space model block branch includes: identifying the feature relationships between multiple channel features in the initial image features to obtain feature sequence relationship data; and adjusting the initial image features according to the feature sequence relationship data to obtain the serialized image features of the image to be processed.

[0068] After obtaining the processing results of the two branches mentioned above, the image resolution of the image to be processed can be adjusted based on the denoised image features and the serialized image features to obtain the target image.

[0069] Figure 4 A schematic flowchart of the image processing method provided in an embodiment of this application is shown. Figure 4 As shown, the specific process of this image processing method is as follows:

[0070] 101. Obtain the initial image features of the image to be processed in the spatial domain, and perform frequency domain transformation on the initial image features to obtain frequency domain features.

[0071] The spatial domain refers to the representation of image data in the original pixel coordinate system. In image processing and computer vision, the spatial domain usually refers to the two-dimensional plane of the image, where each point (pixel) has a specific coordinate position and a corresponding pixel value.

[0072] The frequency domain, also known as the frequency spectrum domain, represents a signal at the frequency level. Unlike the spatial domain, the frequency domain focuses on the frequency components of a signal rather than its spatial distribution. Features in the frequency domain can effectively reflect the high-frequency structure distribution of an image.

[0073] In this embodiment, the image to be processed can be understood as an image requiring super-resolution processing, which has a low resolution. The initial image features are the features corresponding to the image to be processed in the spatial domain. The initial image features can include various types of features, such as pixel value features, texture features, edge features, shape features, spatial relationship features, etc.

[0074] Among these features, pixel value features can represent the color or brightness information of each point in the image to be processed, usually represented by the red, green, and blue (RGB) color space or grayscale values. Texture features can represent the local patterns of regions in the image to be processed, which can be smooth, coarse, regular, irregular, etc. Edge features can represent areas of drastic brightness changes in the image to be processed, and can mark the boundaries between different regions. Shape features can represent closed regions composed of pixels, which can be the outlines or internal structures of objects contained in the image to be processed. Spatial relationship features can represent information such as the relative position and orientation between different features in the image to be processed.

[0075] In this embodiment of the application, obtaining the initial image features of the image to be processed in the spatial domain may include: obtaining the image to be processed, extracting semantic features of the image to be processed in the spatial domain to obtain multiple channel features; and performing dimensionality reduction processing on the channel features to obtain the initial image features.

[0076] As mentioned earlier, the image to be processed is a low-resolution image that requires super-resolution processing. Therefore, after obtaining the image to be processed, feature extraction can be performed to obtain the initial image features. Specifically, semantic features can be extracted from the image to be processed in the spatial domain to obtain multiple channel features. Then, dimensionality reduction processing is performed on each channel feature to obtain the dimensionality-reduced channel features corresponding to each channel feature. The initial image features include multiple dimensionality-reduced channel features.

[0077] In the spatial domain, semantic feature extraction of the image to be processed can be achieved by using different convolutional kernels to extract features from the image, resulting in multiple channel features. Different convolutional kernels can extract different semantic features, which are called channel features. By using multiple convolutional kernels to extract features from the image to be processed, a corresponding number of channel features can be obtained. For example, using 10 different convolutional kernels to extract features from the image to be processed can yield 10 channel features; similarly, using 20 different convolutional kernels to extract features from the image to be processed can yield 20 channel features.

[0078] After obtaining the channel features, each channel feature can be dimensionality reduced (i.e., downsampled) to obtain multiple dimensionality-reduced channel features. These multiple dimensionality-reduced channel features can then be combined to form the initial image features. There are several ways to perform dimensionality reduction on channel features. For example, downsampling convolution can be used to reduce the feature dimension of the channel features, thereby reducing the amount of subsequent computation.

[0079] It is understandable that the process of obtaining the initial image features of the image to be processed in the spatial domain can also include: obtaining the image to be processed and performing dimensionality reduction processing on the image to be processed to obtain the dimensionality-reduced image to be processed; extracting semantic features from the dimensionality-reduced image to be processed in the spatial domain to obtain multiple channel features, and using the channel features as the initial image features. This method first performs downsampling processing on the image to be processed to reduce the image size, and then, based on the dimensionality-reduced image to be processed, uses multiple convolutional kernels to extract features from the image to be processed to obtain the corresponding number of channel features, thereby obtaining the initial image features.

[0080] In this embodiment of the application, performing frequency domain transformation on the initial image features to obtain frequency domain features may include: obtaining attention parameters corresponding to the initial image features; fusing the attention parameters with the initial image features to obtain fused image features; and performing frequency domain transformation on the fused image features to obtain frequency domain features.

[0081] Figure 5 A schematic diagram illustrating the process of determining frequency domain features provided in an embodiment of this application is shown. Figure 5 As shown, in the process of transforming the initial image features into frequency domain features, it is necessary to first fuse the initial image features with the attention parameters to obtain the fused image features; then, the fused image features are transformed into frequency domain features to obtain the frequency domain features.

[0082] In this context, attention parameters can be understood as the parameters that transform the initial image features into the query, key, and value required by the attention mechanism. Attention parameters can include query parameters, value parameters, and key parameters. Query parameters can be understood as the parameters corresponding to the query matrix in the attention mechanism, value parameters as the parameters corresponding to the value matrix, and key parameters as the parameters corresponding to the key matrix.

[0083] Based on this, the attention parameters are fused with the initial image features to obtain fused image features. This can include: fusing channel features with query parameters to obtain fused query features; fusing channel features with key parameters to obtain fused key features; and fusing channel features with value parameters to obtain fused value features. The fused query features, fused key features, and fused value features are then used as the fused image features. Subsequently, frequency domain transformations can be performed on the fused query features, fused key features, and fused value features respectively to obtain their corresponding features in the frequency domain.

[0084] Specifically, performing frequency domain transformation on the fused image features to obtain frequency domain features can include: performing frequency domain transformation on the fused query features to obtain frequency domain query features; performing frequency domain transformation on the fused key features to obtain frequency domain key features; performing frequency domain transformation on the fused value features to obtain frequency domain value features; and using the frequency domain query features, frequency domain key features, and frequency domain value features as frequency domain features.

[0085] The frequency domain transformation can be achieved through Fourier transform. By performing a Fourier transform on the fused image features, the features can be converted from the original spatial domain to the frequency domain, yielding frequency domain features. Transforming the fused image features (including fused query features, fused key features, and fused value features) to the frequency domain via Fourier transform allows multiple feature tokens to be assigned to different frequency bands. In the frequency domain, each pixel embeds a sinusoidal component of the original spatial features with a specific frequency. In other words, after transforming the fused image features from the spatial domain to the frequency domain via Fourier transform, they are decomposed into components of different frequencies, yielding frequency features. Low-frequency and high-frequency components correspond to slowly changing and rapidly changing components in the signal, respectively.

[0086] Each token in the frequency domain features represents a sinusoidal component of the image to be processed, thus providing a global representation of a specific frequency band. The sinusoidal component of a two-dimensional image refers to the periodic variation described by a sinusoidal function. In image processing, the sinusoidal component can be analyzed and processed using either the Discrete Fourier Transform (DFT) or the Fast Fourier Transform (FFT).

[0087] 102. Based on the frequency domain characteristics, the initial image features are denoised to obtain the denoised image features.

[0088] In the frequency domain, low-frequency components can encode smooth regions and weak noise, while high-frequency components correspond to rich edge and texture information. Noise in the low-frequency components can interfere with other features during super-resolution, thus affecting the accuracy of super-resolution. Therefore, it is necessary to identify the noise contained in the initial image features based on the frequency domain features to denoise the initial image features, obtaining denoised image features, and then performing super-resolution based on the denoised image features.

[0089] In this embodiment, denoising the initial image features based on frequency domain features to obtain denoised image features may include: grouping the frequency domain features according to a preset frequency interval threshold to obtain grouped frequency domain features, wherein the grouped frequency domain features include frequency interval features corresponding to multiple frequency intervals, and the number of features in each frequency interval feature is the same; performing attention weighting on the grouped frequency domain features to obtain target frequency domain features, wherein the target frequency domain features indicate the feature relationship between the frequency interval features; and denoising the initial image features based on the target frequency domain features to obtain denoised image features.

[0090] A preset frequency range threshold can be understood as the frequency values ​​corresponding to multiple boundary points set within the frequency range covered by the frequency domain feature. These boundary points can divide the frequency domain feature into multiple frequency domain range features with different frequency domain ranges, and each frequency domain range feature contains the same number of features. That is, according to the preset frequency range threshold, the frequency domain feature is evenly divided into multiple frequency domain ranges, and each frequency domain range corresponds to a frequency domain range feature containing the same number of features. After the above grouping, the resulting frequency domain range features can be classified as high-frequency range features, low-frequency range features, etc.

[0091] Figure 6 This illustration shows a schematic diagram of grouping frequency domain features according to an embodiment of this application. For example... Figure 6 As shown, the frequency range covered by the frequency domain feature is x0-x n Between x0 and x, for this frequency domain characteristic, n Three preset frequency range thresholds were set within the range, namely x1, x2, and x3, ranked from low to high frequency, with x1 > x0 and x3 > x3. <x n Based on the three preset frequency interval thresholds mentioned above, the frequency domain features can be divided into four frequency domain interval features: the first frequency domain interval feature, the second frequency domain interval feature, the third frequency domain interval feature, and the fourth frequency domain interval feature. Specifically, the frequency domain interval corresponding to the first frequency domain interval feature is [x0, x1), the frequency domain interval corresponding to the second frequency domain interval feature is [x1, x2), the frequency domain interval corresponding to the third frequency domain interval feature is [x2, x3), and the frequency domain interval corresponding to the fourth frequency domain interval feature is [x3, x4]. n ].

[0092] The four frequency domain features mentioned above are ordered from lowest to highest frequency range as follows: first frequency domain feature, second frequency domain feature, third frequency domain feature, and fourth frequency domain feature. Furthermore, the interval window values ​​for these four frequency domain intervals are the same, meaning that each of the four frequency domain intervals contains the same number of features.

[0093] It is understandable that the number and frequency values ​​of preset frequency interval thresholds can be set or adjusted according to actual conditions, as long as the frequency domain features can be evenly divided into multiple frequency domain bands of equal length. Correspondingly, the number of frequency domain intervals included in the grouped frequency domain features can be set or adjusted according to actual conditions, and the frequency domain interval features corresponding to each frequency interval can also be set or adjusted according to actual conditions. It is understood that the above examples are merely an introduction to the grouping process of frequency domain features and do not impose limitations on the specific grouping process of frequency domain features.

[0094] In this embodiment of the application, the frequency domain features are grouped according to a preset frequency interval threshold to obtain grouped frequency domain features. This may include: marking the position of the pixels corresponding to the frequency domain features to obtain pixel position index data; determining the pixel frequency domain features corresponding to the pixels from the frequency domain features based on the pixel position index data; and grouping the pixel frequency domain features according to the preset frequency interval threshold to obtain grouped frequency domain features, wherein the grouped frequency domain features contain the position information of the pixels.

[0095] Since each feature token in the frequency domain represents a sinusoidal component of the image to be processed, and the image to be processed is composed of multiple pixels with different frequencies, the process of grouping the frequency domain features can be understood as the process of recombining the spectral bands of the pixels in the image to be processed according to their frequencies. By recombining the spectral bands of the frequency domain features, the pixels in the image to be processed can be divided into different frequency domain intervals, thereby determining which pixels in the image to be processed correspond to high-frequency components and which correspond to low-frequency components.

[0096] Figure 7 This illustration shows a schematic diagram of the operation of spectral band reconstruction of frequency domain features provided in an embodiment of this application. For example... Figure 7As shown, the pixel points corresponding to the frequency domain features can be marked with their positions to obtain pixel point position index data, which indicates the correspondence between the pixel points and the frequencies. This step can be performed using a grid index, which is a technique that divides space or data into regular grid units and uses these units to quickly locate the target object. The core principles of the grid index include: (1) Spatial division: dividing the target area (two-dimensional plane or three-dimensional space) into grid units of equal size (such as squares or cubes); (2) Data mapping: associating each object (such as points, lines, and surfaces) with the grid unit in which it is located; (3) Fast query: by calculating the grid range of the target area, objects in the relevant grid units can be directly retrieved, avoiding a full scan.

[0097] After obtaining the pixel location index data, the pixel frequency domain features corresponding to the pixel can be determined from the frequency domain features based on the pixel location index data. Then, the pixel frequency domain features are grouped according to the preset frequency interval threshold to obtain the grouped frequency domain features, which contain the pixel location information.

[0098] For example, in the example above where the frequency domain features are divided into four frequency domain intervals, after determining the pixel frequency domain features corresponding to each pixel from the frequency domain features, the pixel frequency domain features can be arranged in order from high frequency to low frequency (or from low frequency to high frequency). Then, according to a preset frequency domain interval threshold, the pixel frequency domain features are grouped to obtain the grouped frequency domain features. The grouped frequency domain features include the frequency domain interval features corresponding to the four frequency domain intervals, and each frequency domain interval contains the same number of pixel frequency domain features. The above four frequency domain interval features can be regarded as a sequence of frequency domain feature bands of the same length. Concatenating the above four frequency domain interval features yields the grouped frequency domain features. The grouped frequency domain features include a first frequency domain interval feature, a second frequency domain interval feature, a third frequency domain interval feature, and a fourth frequency domain interval feature. Each frequency domain interval feature includes the pixel position information within that frequency domain interval, thereby determining which pixels in the image to be processed correspond to high-frequency components and which pixels correspond to low-frequency components.

[0099] In this embodiment, the position markers of pixels are collected by grid index, and then the frequency domain features are sorted from high frequency to low frequency (or low frequency to high frequency). The reordered frequency domain features are then classified into band sequences of the same length and inter-band splicing is performed to obtain grouped frequency domain features, so as to facilitate subsequent self-attention calculation on the grouped frequency domain features.

[0100] After grouping the frequency domain features to obtain grouped frequency domain features, attention weighting can be applied to these grouped frequency domain features to obtain the target frequency domain features. The target frequency domain features indicate the feature relationships between frequency domain intervals. Since frequency domain features are divided into frequency domain query features, frequency domain key features, and frequency domain value features, the grouped frequency domain features obtained after grouping the frequency domain features can be categorized into grouped query features, grouped key features, and grouped value features.

[0101] Based on this, attention weighting is applied to the grouped frequency domain features to obtain the target frequency domain features. This can include: calculating the dot product of the grouped query features and the grouped key features to obtain the attention weights of the grouped frequency domain features; and fusing the attention weights with the grouped value features to obtain the target frequency domain features.

[0102] Figure 8 This diagram illustrates the process of determining a target frequency domain feature according to an embodiment of this application. Figure 8 The process involves performing a dot product on the grouped query features and grouped key features, and then applying a soft maximization to the result to obtain the attention weights corresponding to the grouped frequency domain features. The soft maximization can be performed in various ways, such as using the Softmax function or other functions (e.g., the Sigmoid function) to normalize the dot product result, thus obtaining the attention weights for the grouped frequency domain features. After obtaining the attention weights for the grouped frequency domain features, these weights can be fused with the grouped key features to obtain the target frequency domain features.

[0103] The process of determining the target frequency domain features described above can be understood as: performing spectral self-attention calculation based on frequency domain features within a frequency range. By grouping the frequency domain features and performing self-attention calculation based on the grouped frequency domain features, target frequency domain features that can indicate the feature relationships between frequency domain interval features are obtained. Through spectral self-attention calculation, the high-frequency bands of the encoded texture (i.e., the frequency domain interval features corresponding to the high-frequency intervals) can be enhanced, and the low-frequency bands of the encoded noise (i.e., the frequency domain interval features corresponding to the low-frequency intervals) can be weakened in the grouped frequency domain features, thereby highlighting high-frequency image details.

[0104] After obtaining the target frequency domain features, the initial image features can be denoised based on the target frequency domain features to remove the frequency domain features corresponding to the low-frequency intervals in the target frequency domain features, so as to obtain the denoised image features. Specifically, denoising the initial image features based on the target frequency domain features to obtain the denoised image features may include: identifying the frequency domain features corresponding to a preset low-frequency interval threshold in the target frequency domain features to obtain low-frequency interval features, which indicate noise features in the initial image features; removing the low-frequency interval features from the target frequency domain features to obtain the denoised frequency domain features; reshaping the denoised frequency domain features to obtain reshaped frequency domain features, which have the same number of features as the initial image features; and performing a spatial domain transformation on the reshaped frequency domain features to obtain the denoised image features.

[0105] Figure 9A This illustration shows a schematic diagram of determining the features of a denoised image according to an embodiment of this application. For example... Figure 9A As shown, denoising can be performed on the target frequency domain features to obtain denoised frequency domain features. Specifically, this step involves identifying the frequency domain interval features corresponding to a preset low-frequency interval threshold from the target frequency domain features, obtaining the low-frequency interval features, and then removing the low-frequency interval features from the target frequency domain features to obtain the denoised frequency domain features. The preset low-frequency interval threshold can be determined by referring to a preset frequency interval threshold. For example, the preset low-frequency interval threshold can be the same as one or more preset frequency interval thresholds. Taking the above example of dividing the frequency domain features into four frequency domain interval features based on the preset frequency interval threshold, the preset low-frequency interval threshold can be set to be the same as the preset frequency interval threshold corresponding to the first frequency domain interval feature. In this case, the first frequency domain interval feature can be used as the low-frequency interval feature.

[0106] For example, the preset low-frequency interval threshold can also be selected within the range of preset frequency interval thresholds. Again, taking the example of dividing the frequency domain features into four frequency domain intervals based on the preset frequency interval thresholds, the range corresponding to the preset low-frequency interval threshold can be selected within the preset frequency interval threshold range corresponding to the first frequency domain interval feature. That is, the range corresponding to the preset low-frequency interval threshold can be selected within the interval [x0, x1). When selecting the range corresponding to the preset low-frequency interval threshold within the interval [x0, x1), the magnitude of the feature values ​​of each frequency domain interval feature in the target frequency domain feature can be referenced. For example, the frequency range corresponding to the frequency domain interval feature with the smaller feature value can be used as the basis for determining the preset low-frequency interval threshold. In this case, the frequency range of the low-frequency interval feature is smaller than the frequency range of the first frequency domain interval feature. This method allows for a more refined determination of noise features in the initial image features.

[0107] Compared to the target frequency domain features, the denoised frequency domain features are missing some (noise) features. Therefore, feature reshaping is necessary to obtain the reshaped frequency domain features. Feature reshaping, a key step in machine learning and data preprocessing, can adjust the dimension or structure of data to suit the input requirements of a specific model or algorithm. Feature reshaping does not change the essential information of the data; it optimizes the data representation by rearranging, splitting, or merging features. In this embodiment, feature reshaping of the denoised frequency domain features ensures that the reshaped frequency domain features have the same number of features as the initial image features.

[0108] Then, a spatial domain transformation can be performed on the reconstructed frequency domain features to obtain the denoised image features. The spatial domain transformation process can be understood as the inverse process of the frequency domain transformation. As mentioned above, a Fourier transform is performed on the initial image features to obtain the frequency domain features. At this point, an inverse Fourier transform can be performed on the reconstructed frequency domain features to transform them into denoised image features in the spatial domain.

[0109] Figure 9B This illustration shows another schematic diagram of determining denoised image features according to an embodiment of this application. For example... Figure 9B As shown, after obtaining the reshaped frequency domain features, a feature scattering operation can be performed on the reshaped frequency domain features to obtain scattered frequency domain features; then, a spatial domain transformation can be performed on the scattered frequency domain features to obtain the denoised image features. By performing feature scattering processing on the reshaped frequency domain features, the feature values ​​can be dispersed to specific locations or structures.

[0110] 103. Identify the feature relationships between multiple channel features in the initial image features to obtain feature sequence relationship data.

[0111] Figure 10A A schematic diagram of an image to be processed according to an embodiment of this application is shown; Figure 10B This illustration shows a schematic diagram of another image to be processed provided in an embodiment of this application; Figure 10C This illustration shows another image to be processed according to an embodiment of this application. Figures 10A-10C As shown, structural similarities exist within the images; for example, Figure 10A windows of buildings in the middle Figure 10B zebra stripes in Figure 10C While the local sharpness of game scenes varies, their structures are similar. In low-resolution images, details in blurry areas can be compensated for by using similar, sharper structures, thus achieving a better super-resolution effect. This is the dependency relationship between image content.

[0112] exist Figure 3The bidirectional state-space model block shown allows for the identification of feature relationships among multiple channel features in the initial image features, yielding feature sequence relationship data. This feature sequence relationship data can be understood as the dependencies between multiple channel features in the initial image features. The initial image features of the image to be processed can be understood as a global feature, and the individual channel features contained within it can be understood as local features; these local features collectively constitute the global feature. Therefore, identifying the feature relationships between channel features ensures that linear dependencies between channel features are preserved, enhancing the coherence of the channel feature sequence.

[0113] In this embodiment of the application, the feature sequence relationship data may include first feature sequence relationship data and second feature sequence relationship data. The first feature sequence relationship data is feature sequence relationship data obtained by identifying feature relationships according to the first feature order of the channel features, and the second feature sequence relationship data is feature sequence relationship data obtained by identifying feature relationships according to the second feature order of the channel features.

[0114] In the process of feature extraction from the image to be processed, the feature order of each channel feature can be marked according to the order of feature extraction. For example, if five channel features are extracted in chronological order during the feature extraction process of the image to be processed, namely channel feature 1, channel feature 2, channel feature 3, channel feature 4 and channel feature 5, then the feature order of the above five channel features is arranged in the following order: channel feature 1 → channel feature 2 → channel feature 3 → channel feature 4 → channel feature 5.

[0115] The first feature order can be a forward channel feature order, i.e., channel feature 1 → channel feature 2 → channel feature 3 → channel feature 4 → channel feature 5, or any channel feature order you desire, i.e., channel feature 5 → channel feature 4 → channel feature 3 → channel feature 2 → channel feature 1. When the first feature order is a forward channel feature order, the second feature order is a reverse channel feature order; conversely, when the first feature order is a reverse channel feature order, the second feature order is a forward channel feature order.

[0116] Based on this, the feature relationships between multiple channel features in the initial image features are identified to obtain feature sequence relationship data. This may include: identifying the feature relationships between channel features in the initial image features according to the first feature order of the channel features to obtain the first feature sequence relationship data corresponding to the channel features; adjusting the first feature order to obtain a second feature order, wherein the order of the first feature order and the second feature order are reversed; and identifying the feature relationships between channel features in the initial image features according to the second feature order to obtain the second feature sequence relationship data corresponding to the channel features.

[0117] Figure 11 This illustration shows a schematic diagram of determining a feature sequence relationship data provided in an embodiment of this application. For example... Figure 11 As shown, the feature relationships between channel features in the initial image features can be identified first according to the first feature order to obtain the first feature sequence relationship data. A Structured Space Model (SSM) can be used to identify the feature relationships between channel features. As a mathematical framework for describing dynamic systems, the SSM can model the sequential evolution process of a system through the association between latent states and observations. In the embodiments of this application, the channel features can be sequentially input into the SSM according to the first feature order, and the association (dependency) between each channel feature under the first feature order can be constructed through the SSM.

[0118] Compared to the Transformer based on self-attention, the computational complexity of the state-space model when processing sequential data is far less than the quadratic computational complexity of the Transformer based on self-attention. Therefore, in this embodiment, by introducing a cascaded bidirectional state-space model to capture and model local and global information, the super-resolution effect can be guaranteed as much as possible while reducing complexity.

[0119] After obtaining the first feature sequence relationship data, the order of the first features can be adjusted to obtain the second feature order. Specifically, the first feature order can be reversed to the second feature order by channel flipping, making the order of the first and second features opposite. Then, following the second feature order, the channel features can continue to be input into the state space model, and the state space model can be used to construct the correlation relationships between the various channel features under the second feature order.

[0120] In this embodiment, a bidirectional strategy is employed to capture the correlation between channel features from both the forward and reverse directions, which can alleviate the state-space model's forgetting of channel features. Especially when there are many channel features, multiple channel features constitute the long-range features of the image to be processed. By capturing the dependencies between multiple channel features through the bidirectional strategy, the state-space model's forgetting of long-range features can be alleviated, and the extraction effect of long-range information can be better enhanced.

[0121] 104. Based on the feature sequence relationship data, adjust the initial image features to obtain the serialized image features of the image to be processed.

[0122] Serialized image features are the image features obtained after the initial image features of the image to be processed are processed by a bidirectional state-space model block. Serialized image features contain the dependencies between multiple channel features of the image to be processed.

[0123] In this embodiment of the application, feature adjustment is performed on the initial image features according to the feature sequence relationship data to obtain the serialized image features of the image to be processed. This may include: performing channel attention weighting based on the initial image features to obtain channel weights, where the channel weights indicate the importance of the channel features; performing spatial attention weighting based on the initial image features to obtain spatial position weights, where the spatial position weights indicate the importance of the pixel positions in the image to be processed; and performing feature adjustment on the initial image features according to the channel weights, spatial position weights, and feature sequence relationship data to obtain the serialized image features.

[0124] Figure 12 This illustration shows a schematic diagram of determining a serialized image feature according to an embodiment of this application. For example... Figure 12 As shown, spatial attention weighting and channel attention weighting can be applied to the initial image features to obtain the corresponding weight data. Specifically, channel attention weighting of the initial image features yields the channel weights, while spatial attention weighting yields the spatial location weights.

[0125] Spatial attention and channel attention are two commonly used attention mechanisms in deep learning.

[0126] Spatial attention focuses on the importance of spatial locations, concentrating on different regions (pixels or local areas) in an image or feature map to determine which spatial locations are more important to the task. The spatial location weights obtained after spatial attention weighting are a two-dimensional weight matrix with the same width and height as the initial image feature map, used to weight features at different spatial locations. Therefore, the spatial location weights can indicate the importance of pixel locations in the image to be processed.

[0127] Channel attention focuses on the importance of channels, that is, on the semantic information (such as color, texture, category features, etc.) of different channels. It can analyze the importance of different feature channels (such as the output of different filters after convolution) and determine which channel features are more critical to the task. The channel weights obtained after channel attention weighting are a one-dimensional weight vector (the length of which is equal to the number of channels), used to weight the features of different channels. Therefore, the channel weights can indicate the importance of channel features.

[0128] Before applying channel attention weighting and spatial attention weighting to the initial image features, further two-dimensional convolution processing can be performed on the initial image features to extract more channel features representing local details, thereby further expanding the number of channels. Then, channel attention weighting and spatial attention weighting are applied to the obtained multiple channel features. The size and number of convolution kernels used for the two-dimensional convolution processing can be set according to the actual situation. For example, the size of the convolution kernel can be 5×5, 3×3, etc., and the number of convolution kernels can be 5, 10, etc.

[0129] After calculating the channel weights and spatial location weights, the initial image features can be adjusted based on the channel weights, spatial location weights, and feature sequence relationship data to obtain serialized image features. In this embodiment, adjusting the initial image features based on the channel weights, spatial location weights, and feature sequence relationship data to obtain serialized image features may include: fusing the spatial location weights with the initial image features to obtain fused spatial features, and fusing the channel weights with the initial image features to obtain fused channel features; and fusing the fused spatial features and fused channel features based on the feature sequence relationship data to obtain serialized image features.

[0130] The initial image features are fused with spatial location weights and channel weights to obtain fused spatial features and fused channel features. The fused spatial features reflect the importance of different pixel positions in the image to be processed, while the fused channel features reflect the importance of each channel feature. Then, based on the feature sequence relationship data, the fused spatial features and fused channel features are further fused to obtain serialized image features. The serialized image features are the output of the bidirectional state-space model block.

[0131] In this embodiment of the application, fusing the fused spatial features and the fused channel features according to the feature sequence relationship data to obtain serialized image features may include: adjusting the fused channel features according to the first feature sequence relationship data to obtain the adjusted first channel features; fusing the adjusted first channel features with the fused spatial features to obtain the fused first image features; and adjusting the fused first image features according to the second feature sequence relationship data to obtain the serialized image features.

[0132] The first feature sequence relationship data is calculated based on the fused channel features, and the second feature sequence relationship data is calculated based on the fused channel features after channel flipping. The method for adjusting the fused channel features based on the first feature sequence relationship data can be to fuse the first feature sequence relationship data and the fused channel features to obtain the adjusted first channel feature, which includes the correlation relationships between the various channel features.

[0133] After obtaining the adjusted first channel features, the adjusted first channel features can be fused with the fused spatial features. This feature fusion can be achieved through methods such as feature stitching, resulting in the fused first image features. Subsequently, for the second feature sequence relationship data obtained after channel flipping, the fused first image features can be further adjusted based on this data. For example, the second feature sequence relationship data can be fused with the fused first image features to obtain serialized image features.

[0134] After obtaining the fused first image features and performing channel flipping, these fused first image features can be used as the initial image features after channel flipping. Spatial attention weighting is then applied to these fused first image features to obtain new spatial position weights, and channel attention weighting is applied to obtain new channel weights. Simultaneously, the fused first image features are fused with the new spatial position weights to obtain new fused spatial features, and the fused first image features are fused with the new channel weights to obtain new fused channel features. Next, the feature relationships between the channel features in the fused first image features are identified to obtain second feature sequence relationship data. Finally, based on the second feature sequence relationship data, the new fused spatial features and the new fused channel features are fused to obtain serialized image features.

[0135] It is understandable that before using a state-space model to identify the feature relationships between channel features, a one-dimensional deep convolution kernel can be used to convolve the channel features, serializing the channel features into a one-dimensional vector, so that the state-space model can be used to identify the feature relationships later.

[0136] 105. Based on the features of the denoised image and the features of the serialized image, the image resolution of the image to be processed is adjusted to obtain the target image.

[0137] After obtaining the denoised image features from the spectral self-attention block branch output and the serialized image features from the bidirectional state space model block output, the image resolution of the image to be processed can be adjusted based on the data from the outputs of the two branches to obtain a high-resolution target image.

[0138] In this embodiment of the application, adjusting the image resolution of the image to be processed based on the denoised image features and the serialized image features to obtain the target image may include: fusing the denoised image features and the serialized image features to obtain the target image features; and performing dimensionality upscaling on the image to be processed based on the target image features to obtain the target image.

[0139] Specifically, the denoised image features and the serialized image features can be fused through channel concatenation and reduction to obtain the target image features. These target image features can then be further fused with the initial image features of the image to be processed to obtain fused target image features. Finally, upsampling convolution processing is performed on the fused target image features to increase the resolution of the image to be processed, resulting in a high-resolution target image, thus achieving super-resolution processing of the image to be processed.

[0140] It is understandable that before fusing the target image features with the initial image features, convolution processing can be performed on the target image features to ensure that their feature sizes are consistent with those of the initial image features, facilitating subsequent feature fusion. The size of the convolution kernel used in this process can also be set according to actual conditions, and this application does not impose any restrictions on it.

[0141] Furthermore, the fused target image is subjected to upsampling convolution processing. Specifically, the fused target image can be interpolated and its features enhanced by the convolution kernel to achieve dimensionality upsampling of the image to be processed, thereby obtaining the target image.

[0142] The image processing method provided in this application is quantitatively compared with other super-resolution methods in related technologies under a 2x super-resolution experiment. Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used to evaluate the technical effectiveness of each method. The experimental results are shown in Table 1. The datasets used in the experiment include the publicly available datasets Set5, Set14, and Urban100. Other super-resolution methods sampled in the experiment include SMSR, ESRT, GLADSR, and LatticeNet-CL.

[0143] The core idea of ​​SMSR (Sparse Mask Super-Resolution) is to use sparse masks to guide the network to focus on high-frequency detail regions (such as edges and textures), suppressing redundant computation in smooth regions. It is usually combined with multi-stage progressive reconstruction to gradually recover high-frequency information. This method is mainly for optimizing high-frequency details and edges in images.

[0144] The core idea of ​​ESRT (Efficient Super-Resolution Transformer) is to combine the global modeling capability of Transformer with the local feature extraction capability of Convolutional Neural Network (CNN) to design a lightweight network. It reduces the amount of computation by reusing features across stages and sharing parameters. This method is mainly aimed at balancing super-resolution performance and improving computational efficiency.

[0145] The core idea of ​​GLADSR (Global-Local Adjusting Dense Network) is to optimize resource allocation through a global-local collaborative adjustment mechanism and achieve efficient image super-resolution by combining lightweight module design. It dynamically focuses on key regions and enhances high-frequency features through global guidance and local selection, introduces nested dense connections to achieve cross-stage feature reuse, and finally reduces computational complexity through separable pyramid upsampling.

[0146] LatticeNet-CL is a deep learning model that combines LatticeNet (a lattice-structured network) with contrastive learning (CL), primarily used to improve feature representation capabilities and model robustness. Its core design utilizes multi-path feature interaction and contrastive loss optimization. In image super-resolution processing, LatticeNet fuses multi-scale features, and contrastive learning constrains the latent spatial similarity between generated images and real images.

[0147] Table 1. Comparison Results of Quantitative Indicators

[0148]

[0149] As shown in Table 1, the image processing method provided in this application performs super-resolution on the image to be processed, and the peak signal-to-noise ratio and structural similarity of the obtained target image are higher than those of other methods in related technologies.

[0150] From a visual perspective, this application also compares the target image obtained using the image processing method provided in this application's embodiments with images obtained using other methods (including) related technologies for the same low-resolution image to be processed. Furthermore, a high-resolution native image corresponding to the image to be processed is used as a reference.

[0151] Figure 13 The images shown are visual representations of images obtained using different super-resolution methods, as well as high-resolution native images. For example... Figure 13 As shown, compared with bilinear interpolation and cubic convolution, the target image obtained by the image processing method provided in this application has clearer texture details and is almost indistinguishable from the high-resolution native image in terms of visual effect.

[0152] In addition, to meet the requirements of real-time performance, the embodiments of this application can also accelerate and lightweight the model side; deploy the image processing device on multiple terminals and perform hardware acceleration on the deployment side. Figure 14 This illustration shows an improved schematic diagram that meets real-time requirements, provided by an embodiment of this application. For example... Figure 14 As shown, the image processing device can execute image processing methods. For these methods, model acceleration and lightweighting can be implemented, for example, through methods such as image patch parallel processing, reparameterization, and knowledge distillation. The image processing device can be deployed across multiple endpoints to distribute computational load, and hardware acceleration can be applied, for example, through methods such as NNAPI, TFLite, and Qualcomm QNN.

[0153] NNAPI is Android's Neural Network API, used to accelerate machine learning inference on Android devices. TensorFlow Lite is Google's inference framework for mobile and embedded devices, supporting various hardware acceleration methods. QualcommQNN is Qualcomm's neural network library optimized for its Snapdragon platform, designed to improve performance.

[0154] The following comparison will focus on the time required for super-resolution. Taking super-resolution on a Snapdragon 8gen2 device as an example, the time required for super-resolution using the image processing method of this application (including before and after acceleration) and other methods in related technologies is tested for the same low-resolution (540p) image to be processed. The resolution of the target image after super-resolution is 1080p. The experimental results are shown in Table 2.

[0155] Table 2 Comparison of Over-Scoring Time

[0156]

[0157] As shown in Table 2, after lightweighting and hardware acceleration, the model requires 6.9ms to super-resolution one frame, which meets the real-time requirements and has the potential for practical application.

[0158] As can be seen from the above, the embodiments of this application obtain the initial image features of the image to be processed in the spatial domain, and perform frequency domain transformation on the initial image features to obtain frequency domain features; then, based on the frequency domain features, the initial image features are denoised to obtain denoised image features; then, the feature relationships between multiple channel features in the initial image features are identified to obtain feature sequence relationship data; then, according to the feature sequence relationship data, the initial image features are feature-adjusted to obtain the serialized image features of the image to be processed; finally, based on the denoised image features and the serialized image features, the image resolution of the image to be processed is adjusted to obtain the target image. This scheme can transform the initial image features of the image to be processed into the spectral domain to obtain frequency domain features. Based on these frequency domain features, global information of the image to be processed can be captured, improving the ability to model long-range dependencies. Furthermore, by processing the frequency domain features of different frequency components encoding different features (such as smooth regions, edges, textures, and noise), denoised image features are obtained, thereby removing noise information from the image to be processed and avoiding noise interference in the super-resolution process. Additionally, by identifying the feature relationships between multiple channel features in the initial image features, feature adjustments are made to the initial image features to obtain serialized image features, ensuring that the linear dependencies between channel features are preserved and enhancing the coherence of the channel feature sequence. Finally, the denoised image features and serialized image features are used as guiding conditions for super-resolution processing of the image to be processed, resulting in a high-resolution target image. This method can improve the super-resolution effect while reducing computational complexity.

[0159] Based on the method described in the above embodiments, the following examples will provide further detailed explanations.

[0160] In this embodiment, the image processing device will be specifically integrated into an electronic device, with the electronic device serving as a server and the user terminal as an example.

[0161] Figure 15 Another schematic flowchart of the image processing method provided in an embodiment of this application is shown. Figure 15 As shown, an image processing method has the following specific steps:

[0162] 201. The terminal acquires the initial image features of the image to be processed in the spatial domain.

[0163] For example, the terminal can acquire the image to be processed and extract semantic features from the image in the spatial domain to obtain multiple channel features; then perform dimensionality reduction processing on the channel features to obtain the initial image features.

[0164] 202. The terminal performs frequency domain transformation on the initial image features to obtain frequency domain features.

[0165] For example, the terminal can obtain the attention parameters corresponding to the initial image features; fuse the attention parameters with the initial image features to obtain the fused image features; and perform frequency domain transformation on the fused image features to obtain frequency domain features.

[0166] For example, the terminal can fuse channel features with query parameters to obtain fused query features; fuse channel features with key parameters to obtain fused key features; fuse channel features with value parameters to obtain fused value features; and use the fused query features, fused key features, and fused value features as fused image features.

[0167] For example, the terminal can perform frequency domain transformation on the fused query features to obtain frequency domain query features; perform frequency domain transformation on the fused key features to obtain frequency domain key features; perform frequency domain transformation on the fused value features to obtain frequency domain value features; and use the frequency domain query features, frequency domain key features, and frequency domain value features as frequency domain features.

[0168] 203. The terminal groups the frequency domain features according to a preset frequency range threshold to obtain the grouped frequency domain features.

[0169] For example, after grouping, the frequency domain features include frequency domain interval features corresponding to multiple frequency domain intervals, and the number of features in the frequency domain interval features is the same.

[0170] For example, the terminal can mark the position of the pixels corresponding to the frequency domain features to obtain pixel position index data; based on the pixel position index data, determine the pixel frequency domain features corresponding to the pixels from the frequency domain features; and group the pixel frequency domain features according to a preset frequency interval threshold to obtain grouped frequency domain features, which contain the position information of the pixels.

[0171] 204. The terminal performs attention weighting on the grouped frequency domain features to obtain the target frequency domain features, which indicate the feature relationships between frequency domain interval features.

[0172] For example, grouped frequency domain features can be divided into grouped query features, grouped key features, and grouped value features.

[0173] For example, the terminal can perform a dot product calculation on the grouped query features and the grouped key features to obtain the attention weight of the grouped frequency domain features; and fuse the attention weight with the grouped value features to obtain the target frequency domain features.

[0174] 205. The terminal performs denoising processing on the initial image features based on the target frequency domain features to obtain the denoised image features.

[0175] For example, the terminal can identify the frequency domain interval features corresponding to a preset low-frequency interval threshold in the target frequency domain features, and obtain the low-frequency interval features, which indicate the noise features in the initial image features; remove the low-frequency interval features from the target frequency domain features to obtain the denoised frequency domain features; reshape the denoised frequency domain features to obtain the reshaped frequency domain features, which have the same number of features as the initial image features; and perform spatial domain transformation on the reshaped frequency domain features to obtain the denoised image features.

[0176] 206. The terminal identifies the feature relationships between multiple channel features in the initial image features and obtains feature sequence relationship data.

[0177] For example, the feature sequence relationship data includes the first feature sequence relationship data and the second feature sequence relationship data.

[0178] For example, the terminal can identify the feature relationships between channel features in the initial image features according to the first feature order of the channel features to obtain the first feature sequence relationship data corresponding to the channel features; adjust the first feature order to obtain the second feature order, the order of the first feature order and the second feature order being reversed; and identify the feature relationships between channel features in the initial image features according to the second feature order to obtain the second feature sequence relationship data corresponding to the channel features.

[0179] 207. The terminal adjusts the initial image features according to the feature sequence relationship data to obtain the serialized image features of the image to be processed.

[0180] For example, the terminal can perform channel attention weighting based on the initial image features to obtain channel weights, which indicate the importance of the channel features; perform spatial attention weighting based on the initial image features to obtain spatial position weights, which indicate the importance of the pixel positions in the image to be processed; and perform feature adjustment on the initial image features according to the channel weights, spatial position weights, and feature sequence relationship data to obtain serialized image features.

[0181] For example, the terminal can fuse spatial location weights with initial image features to obtain fused spatial features, and fuse channel weights with initial image features to obtain fused channel features; based on feature sequence relationship data, the fused spatial features and fused channel features are fused to obtain serialized image features.

[0182] For example, the terminal can adjust the fused channel features based on the first feature sequence relationship data to obtain the adjusted first channel features; fuse the adjusted first channel features with the fused spatial features to obtain the fused first image features; and adjust the fused first image features based on the second feature sequence relationship data to obtain the serialized image features.

[0183] 208. The terminal adjusts the image resolution of the image to be processed based on the features of the denoised image and the features of the serialized image to obtain the target image.

[0184] For example, the terminal can fuse the features of the denoised image and the features of the serialized image to obtain the features of the target image; based on the features of the target image, the image to be processed is upgraded to obtain the target image.

[0185] As described above, this embodiment of the application obtains the initial image features of the image to be processed in the spatial domain through a terminal; then, the terminal performs frequency domain transformation on the initial image features to obtain frequency domain features; subsequently, the terminal groups the frequency domain features according to a preset frequency interval threshold to obtain grouped frequency domain features; then, the terminal performs attention weighting on the grouped frequency domain features to obtain target frequency domain features, which indicate the feature relationships between frequency domain interval features; the terminal then performs denoising processing on the initial image features based on the target frequency domain features to obtain denoised image features; furthermore, the terminal identifies the feature relationships between multiple channel features in the initial image features to obtain feature sequence relationship data; the terminal performs feature adjustment on the initial image features according to the feature sequence relationship data to obtain serialized image features of the image to be processed; finally, the terminal adjusts the image resolution of the image to be processed based on the denoised image features and the serialized image features to obtain the target image. This scheme can transform the initial image features of the image to be processed into the spectral domain to obtain frequency domain features, and then perform spectral self-attention calculation based on the frequency domain features. This allows for the capture of global information of the image to be processed, improving the ability to model long-range dependencies. After spectral self-attention calculation, the high-frequency bands of encoded textures can be enhanced, while the low-frequency bands of encoded noise can be weakened. This allows for the processing of the frequency domain features of different frequency components encoding different features (such as smooth regions, edges, textures, and noise) to obtain denoised image features, thereby removing noise information from the image to be processed and avoiding noise interference in the super-resolution process. Furthermore, by identifying the feature relationships between multiple channel features in the initial image features, the initial image features are adjusted to obtain serialized image features, ensuring that the linear dependencies between channel features are preserved and enhancing the coherence of the channel feature sequence. Finally, the denoised image features and serialized image features are used as guiding conditions for super-resolution processing of the image to be processed, resulting in a high-resolution target image. This method can improve the super-resolution effect while reducing computational complexity.

[0186] To better implement the above methods, this application also provides an image processing apparatus that can be integrated into a network device, such as a server or terminal. The terminal may include a tablet computer, a laptop computer, and / or a personal computer.

[0187] Figure 16 A schematic diagram of the structure of the image processing apparatus provided in an embodiment of this application is shown. Figure 16 As shown, the image processing device may include a transformation unit 301, a noise reduction unit 302, a recognition unit 303, an adjustment unit 304, and a super-resolution unit 305, as follows:

[0188] (1) Transformation unit 301;

[0189] The transformation unit 301 is used to acquire the initial image features of the image to be processed in the spatial domain, and to perform frequency domain transformation on the initial image features to obtain frequency domain features.

[0190] For example, the transformation unit 301 can be used to obtain the attention parameters corresponding to the initial image features; fuse the attention parameters with the initial image features to obtain the fused image features; and perform frequency domain transformation on the fused image features to obtain frequency domain features.

[0191] For example, the transformation unit 301 can be used to obtain the attention parameters corresponding to the initial image features; fuse the attention parameters with the initial image features to obtain the fused image features; and perform frequency domain transformation on the fused image features to obtain frequency domain features.

[0192] For example, the transformation unit 301 can be used to fuse channel features with query parameters to obtain fused query features; fuse channel features with key parameters to obtain fused key features; fuse channel features with value parameters to obtain fused value features; and use the fused query features, fused key features, and fused value features as fused image features.

[0193] For example, the transformation unit 301 can be used to perform frequency domain transformation on the fused query features to obtain frequency domain query features; perform frequency domain transformation on the fused key features to obtain frequency domain key features; perform frequency domain transformation on the fused value features to obtain frequency domain value features; and use the frequency domain query features, frequency domain key features, and frequency domain value features as frequency domain features.

[0194] (2) Noise reduction unit 302;

[0195] The denoising unit 302 is used to denoise the initial image features based on the frequency domain features to obtain the denoised image features.

[0196] For example, the denoising unit 302 can be used to group frequency domain features according to a preset frequency interval threshold to obtain grouped frequency domain features. The grouped frequency domain features include frequency domain interval features corresponding to multiple frequency domain intervals, and the number of features in the frequency domain interval features is the same. Attention weighting is applied to the grouped frequency domain features to obtain target frequency domain features. The target frequency domain features indicate the feature relationship between frequency domain interval features. Based on the target frequency domain features, the initial image features are denoised to obtain denoised image features.

[0197] For example, the denoising unit 302 can be used to mark the position of the pixel corresponding to the frequency domain feature to obtain the pixel position index data; determine the pixel frequency domain feature corresponding to the pixel from the frequency domain feature based on the pixel position index data; and group the pixel frequency domain feature according to the preset frequency interval threshold to obtain the grouped frequency domain feature, which contains the position information of the pixel.

[0198] For example, the denoising unit 302 can be specifically used to identify frequency domain features corresponding to a preset low-frequency interval threshold in the target frequency domain features, thereby obtaining low-frequency interval features, which indicate noise features in the initial image features; remove low-frequency interval features from the target frequency domain features to obtain denoised frequency domain features; reshape the denoised frequency domain features to obtain reshaped frequency domain features, which have the same number of features as the initial image features; and perform spatial domain transformation on the reshaped frequency domain features to obtain denoised image features.

[0199] (3) Identification unit 303;

[0200] The recognition unit 303 is used to recognize the feature relationships between multiple channel features in the initial image features to obtain feature sequence relationship data.

[0201] For example, the recognition unit 303 can be specifically used to recognize the feature relationship between channel features in the initial image features according to the first feature order of the channel features, and obtain the first feature sequence relationship data corresponding to the channel features; adjust the first feature order to obtain the second feature order, the order of the first feature order and the second feature order being reversed; and recognize the feature relationship between channel features in the initial image features according to the second feature order, and obtain the second feature sequence relationship data corresponding to the channel features.

[0202] (4) Adjustment unit 304;

[0203] The adjustment unit 304 is used to adjust the initial image features according to the feature sequence relationship data to obtain the serialized image features of the image to be processed.

[0204] For example, the adjustment unit 304 can be used to perform channel attention weighting based on the initial image features to obtain channel weights, which indicate the importance of the channel features; perform spatial attention weighting based on the initial image features to obtain spatial position weights, which indicate the importance of the pixel positions in the image to be processed; and perform feature adjustment on the initial image features according to the channel weights, spatial position weights, and feature sequence relationship data to obtain serialized image features.

[0205] For example, the adjustment unit 304 can be used to fuse spatial location weights with initial image features to obtain fused spatial features, and to fuse channel weights with initial image features to obtain fused channel features; based on feature sequence relationship data, the fused spatial features and fused channel features are fused to obtain serialized image features.

[0206] For example, the adjustment unit 304 can be used to adjust the fused channel features according to the first feature sequence relationship data to obtain the adjusted first channel features; to fuse the adjusted first channel features with the fused spatial features to obtain the fused first image features; and to adjust the fused first image features according to the second feature sequence relationship data to obtain the serialized image features.

[0207] (5) Super-resolution unit 305;

[0208] The super-resolution unit 305 is used to adjust the image resolution of the image to be processed based on the features of the denoised image and the features of the serialized image to obtain the target image.

[0209] For example, super-resolution unit 305 can be used to fuse the features of the denoised image and the features of the serialized image to obtain the features of the target image; based on the features of the target image, the image to be processed is upgraded to obtain the target image.

[0210] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0211] As can be seen from the above, in this embodiment, the transformation unit 301 obtains the initial image features of the image to be processed in the spatial domain, and performs frequency domain transformation on the initial image features to obtain frequency domain features; then, the denoising unit 302 performs denoising processing on the initial image features according to the frequency domain features to obtain denoised image features; then, the recognition unit 303 recognizes the feature relationships between multiple channel features in the initial image features to obtain feature sequence relationship data; the adjustment unit 304 then performs feature adjustment on the initial image features according to the feature sequence relationship data to obtain the serialized image features of the image to be processed; finally, the super-resolution unit 305 adjusts the image resolution of the image to be processed based on the denoised image features and the serialized image features to obtain the target image. This scheme can transform the initial image features of the image to be processed into the spectral domain to obtain frequency domain features. Based on these frequency domain features, global information of the image to be processed can be captured, improving the ability to model long-range dependencies. Furthermore, by processing the frequency domain features of different frequency components encoding different features (such as smooth regions, edges, textures, and noise), denoised image features are obtained, thereby removing noise information from the image to be processed and avoiding noise interference in the super-resolution process. Additionally, by identifying the feature relationships between multiple channel features in the initial image features, feature adjustments are made to the initial image features to obtain serialized image features, ensuring that the linear dependencies between channel features are preserved and enhancing the coherence of the channel feature sequence. Finally, the denoised image features and serialized image features are used as guiding conditions for super-resolution processing of the image to be processed, resulting in a high-resolution target image. This method can improve the super-resolution effect while reducing computational complexity.

[0212] This application also provides an electronic device, such as... Figure 17 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0213] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 17 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0214] The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 402, and calls data stored in the memory 402, to perform various functions and process data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0215] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0216] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0217] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0218] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0219] The process involves: acquiring initial image features of the image to be processed in the spatial domain; performing frequency domain transformation on these initial image features to obtain frequency domain features; denoising the initial image features based on the frequency domain features to obtain denoised image features; identifying the feature relationships between multiple channel features in the initial image features to obtain feature sequence relationship data; adjusting the initial image features according to the feature sequence relationship data to obtain serialized image features of the image to be processed; and adjusting the image resolution of the image to be processed based on the denoised image features and the serialized image features to obtain the target image.

[0220] For example, an electronic device can acquire the initial image features of the image to be processed in the spatial domain; perform frequency domain transformation on the initial image features to obtain frequency domain features; group the frequency domain features according to a preset frequency interval threshold to obtain grouped frequency domain features; apply attention weighting to the grouped frequency domain features to obtain target frequency domain features, which indicate the feature relationships between frequency interval features; perform denoising processing on the initial image features based on the target frequency domain features to obtain denoised image features; identify the feature relationships between multiple channel features in the initial image features to obtain feature sequence relationship data; perform feature adjustment on the initial image features according to the feature sequence relationship data to obtain serialized image features of the image to be processed; and adjust the image resolution of the image to be processed based on the denoised image features and the serialized image features to obtain the target image, and so on.

[0221] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0222] As can be seen from the above, the embodiments of this application obtain the initial image features of the image to be processed in the spatial domain, and perform frequency domain transformation on the initial image features to obtain frequency domain features; then, based on the frequency domain features, the initial image features are denoised to obtain denoised image features; then, the feature relationships between multiple channel features in the initial image features are identified to obtain feature sequence relationship data; then, according to the feature sequence relationship data, the initial image features are feature-adjusted to obtain the serialized image features of the image to be processed; finally, based on the denoised image features and the serialized image features, the image resolution of the image to be processed is adjusted to obtain the target image. This scheme can transform the initial image features of the image to be processed into the spectral domain to obtain frequency domain features. Based on these frequency domain features, global information of the image to be processed can be captured, improving the ability to model long-range dependencies. Furthermore, by processing the frequency domain features of different frequency components encoding different features (such as smooth regions, edges, textures, and noise), denoised image features are obtained, thereby removing noise information from the image to be processed and avoiding noise interference in the super-resolution process. Additionally, by identifying the feature relationships between multiple channel features in the initial image features, feature adjustments are made to the initial image features to obtain serialized image features, ensuring that the linear dependencies between channel features are preserved and enhancing the coherence of the channel feature sequence. Finally, the denoised image features and serialized image features are used as guiding conditions for super-resolution processing of the image to be processed, resulting in a high-resolution target image. This method can improve the super-resolution effect while reducing computational complexity.

[0223] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0224] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the image processing methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0225] The process involves: acquiring initial image features of the image to be processed in the spatial domain; performing frequency domain transformation on these initial image features to obtain frequency domain features; denoising the initial image features based on the frequency domain features to obtain denoised image features; identifying the feature relationships between multiple channel features in the initial image features to obtain feature sequence relationship data; adjusting the initial image features according to the feature sequence relationship data to obtain serialized image features of the image to be processed; and adjusting the image resolution of the image to be processed based on the denoised image features and the serialized image features to obtain the target image.

[0226] For example, the process involves: acquiring initial image features of the image to be processed in the spatial domain; performing frequency domain transformation on the initial image features to obtain frequency domain features; grouping the frequency domain features according to a preset frequency interval threshold to obtain grouped frequency domain features; applying attention weighting to the grouped frequency domain features to obtain target frequency domain features, which indicate the feature relationships between frequency interval features; denoising the initial image features based on the target frequency domain features to obtain denoised image features; identifying the feature relationships between multiple channel features in the initial image features to obtain feature sequence relationship data; adjusting the initial image features according to the feature sequence relationship data to obtain serialized image features of the image to be processed; and adjusting the image resolution of the image to be processed based on the denoised image features and the serialized image features to obtain the target image, and so on.

[0227] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0228] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0229] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the image processing methods provided in the embodiments of this application, the beneficial effects that any of the image processing methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0230] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various alternative implementations of the data access aspect described above.

[0231] The foregoing has provided a detailed description of an image processing method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized in that, include: The initial image features of the image to be processed in the spatial domain are obtained, and the initial image features are transformed in the frequency domain to obtain the frequency domain features. Based on the frequency domain features, the initial image features are denoised to obtain denoised image features; The feature relationships between multiple channel features in the initial image features are identified to obtain feature sequence relationship data; Based on the feature sequence relationship data, the initial image features are adjusted to obtain the serialized image features of the image to be processed; Based on the denoised image features and the serialized image features, the image resolution of the image to be processed is adjusted to obtain the target image.

2. The image processing method as described in claim 1, characterized in that, The process of obtaining the initial image features of the image to be processed in the spatial domain includes: The image to be processed is acquired, and semantic features are extracted from the image to be processed in the spatial domain to obtain multiple channel features; The channel features are subjected to dimensionality reduction processing to obtain the initial image features.

3. The image processing method as described in claim 1, characterized in that, The step of performing a frequency domain transformation on the initial image features to obtain frequency domain features includes: Obtain the attention parameters corresponding to the initial image features; The attention parameters are fused with the initial image features to obtain the fused image features; The fused image features are transformed in the frequency domain to obtain the frequency domain features.

4. The image processing method as described in claim 3, characterized in that, The attention parameters include query parameters, value parameters, and key parameters. The step of fusing the attention parameters with the initial image features to obtain fused image features includes: The channel features are fused with the query parameters to obtain the fused query features; The channel features are fused with the key parameters to obtain the fused key features; The channel features are fused with the value parameters to obtain the fused value features; The fused query features, the fused key features, and the fused value features are used as the fused image features.

5. The image processing method as described in claim 4, characterized in that, The step of performing a frequency domain transformation on the fused image features to obtain the frequency domain features includes: The fused query features are transformed in the frequency domain to obtain frequency domain query features; The fused key features are transformed in the frequency domain to obtain the frequency domain key features; The fused value features are subjected to frequency domain transformation to obtain frequency domain value features; The frequency domain query feature, the frequency domain key feature, and the frequency domain value feature are used as the frequency domain feature.

6. The image processing method as described in claim 1, characterized in that, The step of denoising the initial image features based on the frequency domain features to obtain denoised image features includes: The frequency domain features are grouped according to a preset frequency interval threshold to obtain grouped frequency domain features. The grouped frequency domain features include frequency domain interval features corresponding to multiple frequency domain intervals, and the number of features in each frequency domain interval feature is the same. Attention weighting is applied to the grouped frequency domain features to obtain target frequency domain features, which indicate the feature relationships between the frequency domain interval features. Based on the target frequency domain features, the initial image features are denoised to obtain the denoised image features.

7. The image processing method as described in claim 6, characterized in that, The step of grouping the frequency domain features according to a preset frequency interval threshold to obtain grouped frequency domain features includes: The pixel points corresponding to the frequency domain features are marked at their positions to obtain pixel point position index data; Based on the pixel position index data, determine the pixel frequency domain feature corresponding to the pixel from the frequency domain feature; The pixel frequency domain features are grouped according to the preset frequency range threshold to obtain the grouped frequency domain features, which include the pixel location information.

8. The image processing method as described in claim 6, characterized in that, The step of denoising the initial image features based on the target frequency domain features to obtain the denoised image features includes: In the target frequency domain features, frequency domain interval features corresponding to a preset low-frequency interval threshold are identified to obtain low-frequency interval features, and the low-frequency interval features indicate noise features in the initial image features; Remove the low-frequency region features from the target frequency domain features to obtain the denoised frequency domain features; The denoised frequency domain features are reshaped to obtain reshaped frequency domain features, and the number of features in the reshaped frequency domain features is the same as that in the initial image features. The reshaped frequency domain features are transformed in the spatial domain to obtain the denoised image features.

9. The image processing method as described in claim 1, characterized in that, The feature sequence relationship data includes first feature sequence relationship data and second feature sequence relationship data. The step of identifying the feature relationships between multiple channel features in the initial image features to obtain feature sequence relationship data includes: According to the first feature order of the channel features, the feature relationship between the channel features in the initial image features is identified to obtain the first feature sequence relationship data corresponding to the channel features; The first feature order is adjusted to obtain a second feature order, wherein the first feature order and the second feature order are in reverse order. According to the second feature order, the feature relationships between the channel features in the initial image features are identified to obtain the second feature sequence relationship data corresponding to the channel features.

10. The image processing method as described in claim 9, characterized in that, The step of adjusting the initial image features according to the feature sequence relationship data to obtain the serialized image features of the image to be processed includes: Channel attention weighting is performed based on the initial image features to obtain channel weights, where the channel weights indicate the importance of the channel features. Spatial attention weighting is performed based on the initial image features to obtain spatial location weights, which indicate the importance of pixel locations in the image to be processed. Based on the channel weights, spatial location weights, and feature sequence relationship data, the initial image features are adjusted to obtain the serialized image features.

11. The image processing method as described in claim 10, characterized in that, The step of adjusting the initial image features based on the channel weights, spatial location weights, and feature sequence relationship data to obtain the serialized image features includes: The spatial location weights are fused with the initial image features to obtain fused spatial features, and the channel weights are fused with the initial image features to obtain fused channel features. Based on the feature sequence relationship data, the fused spatial features and the fused channel features are fused to obtain the serialized image features.

12. The image processing method as described in claim 11, characterized in that, The step of fusing the fused spatial features and the fused channel features according to the feature sequence relationship data to obtain the serialized image features includes: Based on the first feature sequence relationship data, the fused channel features are adjusted to obtain the adjusted first channel features; The adjusted first channel features are fused with the fused spatial features to obtain the fused first image features. Based on the second feature sequence relationship data, feature adjustment is performed on the fused first image features to obtain the serialized image features.

13. The image processing method as described in claim 1, characterized in that, The step of adjusting the image resolution of the image to be processed based on the denoised image features and the serialized image features to obtain the target image includes: The denoised image features and the serialized image features are fused to obtain the target image features; Based on the features of the target image, the image to be processed is subjected to dimensionality upscaling to obtain the target image.

14. An image processing apparatus, characterized in that, include: The transformation unit is used to acquire the initial image features of the image to be processed in the spatial domain, and to perform frequency domain transformation on the initial image features to obtain frequency domain features; A denoising unit is used to denoise the initial image features based on the frequency domain features to obtain denoised image features; The recognition unit is used to recognize the feature relationships between multiple channel features in the initial image features to obtain feature sequence relationship data; An adjustment unit is used to adjust the initial image features according to the feature sequence relationship data to obtain the serialized image features of the image to be processed; The super-resolution unit is used to adjust the image resolution of the image to be processed based on the denoised image features and the serialized image features to obtain the target image.

15. An electronic device, characterized in that, It includes a processor and a memory, the memory storing an application program, and the processor running the application program in the memory to perform the steps of the image processing method according to any one of claims 1 to 12.

16. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the image processing method according to any one of claims 1 to 12.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the image processing method according to any one of claims 1 to 12.

Images