Image processing method, and apparatus

Abstract

Provided in the embodiments of the present application are an image processing method, and an apparatus in the field of computer vision, which are used for converting image features into spatial domains for modulation, so as to perform adaptive modulation on different frequency components to obtain images having a better enhancement effect. The method comprises: acquiring an input image, wherein the input image may specifically comprise an image read from a storage space, or may be an image collected online by using an electronic device, etc.; and then inputting the input image into a pre-trained image processing model to obtain an output image, wherein the image processing model may be a diffusion model implemented on the basis of a transformer architecture, the image processing model comprises a plurality of modules, and said modules are used for converting input first image features into a spectrogram, modulating the spectrogram by using time steps to obtain a modulated spectrogram, and converting the modulated spectrogram into second image features, the second image features being used for obtaining the output image.

Description

Image processing method and device

[0001] The present application claims priority to the Chinese patent application No.CN202411346694.4, filed on September 25, 2024, and entitled "Image processing method and device", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of computer vision, and in particular to an image processing method and device. BACKGROUND

[0003] Image processing, such as image super-resolution, image dehazing or image deblurring, is widely used in the field of computer vision, and is very important for the function of computer vision. The effect of image processing greatly affects the user experience.

[0004] Taking image super-resolution as an example, image super-resolution aims to reconstruct a high-resolution image from an input low-resolution image. Diffusion model is a generative model that learns to recover target data from noisy data step by step, and finally generates very realistic samples. Diffusion model can be implemented through various architectures, such as transformer architecture, which is referred to as DiT (diffusion Transformer). Diffusion model usually performs multiple denoising and reuses the same denoiser. In the denoising process, time steps and spatial features are usually combined for modulation, but the actual output image still has poor effect. SUMMARY

[0005] The present application provides an image processing method and device for converting image features to spatial domain for modulation, thereby adaptively modulating different frequency components to obtain an image with better enhancement effect.

[0006] Therefore, in a first aspect, embodiments of the present application provide an image processing method, comprising: obtaining an input image, which can specifically include an image read from a storage space, or an image collected online by an electronic device, etc.; then inputting the input image into a pre-trained image processing model to obtain an output image, the image processing model can be a diffusion model implemented based on a transformer architecture, which can be used to reconstruct a higher definition image from the input image through diffusion; wherein the image processing model includes a plurality of modules, each module is configured to convert an input first image feature into a frequency spectrum, if the module is the first module, the first image feature is a feature extracted from the input image, if the module is an intermediate module, the first image feature can be a feature output by the previous module, and the frequency spectrum is modulated by a time step to obtain a modulated frequency spectrum, and the modulated frequency spectrum is converted into a second image feature, which is used to obtain the output image, for example, the output of the module as an intermediate layer can be input into the next module, and the output of the last module can be output through an output layer to obtain the output image.

[0007] In the embodiments of the present application, a diffusion model implemented based on a transformer architecture is provided, which can convert image features into frequency components in a frequency domain for adaptive modulation, so that different frequency components such as high frequency components and low frequency components can be trained to realize adaptive modulation of high frequency components and low frequency components. Generally, the time step in the diffusion model has a correlation with the frequency, so that the method provided in the embodiments of the present application can modulate the image from the dimension of frequency to obtain an output image with better enhancement effect.

[0008] In a possible implementation, each of the aforementioned modules can be configured to obtain a frequency spectrum corresponding to the first image feature, such as converting the image feature in a spatial domain into a frequency spectrum in a frequency domain through a conversion algorithm; then obtain a frequency modulation matrix according to the time step, that is, the time step is the time interval for diffusing the first image feature; then modulate the frequency spectrum according to the frequency modulation matrix to obtain a modulated frequency spectrum, the elements in the frequency modulation matrix are used to weight the frequency components in the frequency spectrum; then convert the modulated frequency spectrum into a second image feature, that is, convert the frequency spectrum in the frequency domain into the image feature in the spatial domain.

[0009] Therefore, in the embodiments of the present application, the spatial domain feature can be converted into the frequency domain for modulation within the module, so that adaptive modulation can be performed on different frequency components, and better image enhancement effect can be achieved compared with spatial domain modulation.

[0010] In a possible implementation, the aforementioned obtaining the spectrum diagram corresponding to the first image feature includes: dividing the first image feature to obtain a plurality of sub-blocks, that is, performing feature division in the spatial domain to divide the first image feature into sub-blocks of smaller granularity; and then converting the plurality of sub-blocks into the spectrum diagram. Thus, for input features of different resolutions, the features can be divided into sub-blocks of smaller granularity for processing, so that the plurality of modules in the image processing model can implement feature processing of different scales, the features in the image can be fully enhanced, the image enhancement effect can be improved, and a higher-definition output image can be obtained. Moreover, processing the image features of smaller granularity can improve the efficiency of image processing.

[0011] In a possible implementation, the aforementioned obtaining the frequency modulation matrix according to the time step can include: mapping the time step to a preset space to obtain a mapping vector, the preset space can be a space corresponding to the frequency domain, for example, if the frequency domain is a two-dimensional space (or referred to as a vector), the time step can also be mapped to a two-dimensional space; and then reshaping the mapping vector to obtain the frequency modulation matrix. Thus, in the implementation of the present application, the time step can be mapped to a space and reshaped into a frequency modulation matrix, so as to adaptively modulate the high-frequency components and the low-frequency components to modulate the image features in the frequency domain, and obtain an output image that performs better in both high frequency and low frequency.

[0012] In a possible implementation, the aforementioned obtaining the spectrum diagram corresponding to the first image feature includes: performing Fourier transform on the first image feature to obtain the spectrum diagram corresponding to the first image feature; and the aforementioned converting the modulated spectrum diagram into the second image feature includes: performing inverse Fourier transform on the modulated spectrum diagram to obtain the second image feature. Thus, in the implementation of the present application, Fourier transform and inverse Fourier transform can be used to convert between the spatial domain and the frequency domain, so as to support feature modulation in the frequency domain and improve the image enhancement effect.

[0013] In a possible implementation, in the image processing model, the plurality of modules are connected in a U-shaped structure, and the number of processing channels in each module is the same. The image processing model can process features of multiple different scales. For example, each pair of modules in the plurality of modules can form a pair of encoding modules and decoding modules, that is, the same scale features are encoded and decoded. Thus, in the implementation of the present application, the internal structure of the transformer module can refer to the internal structure of the straight-cylinder transformer module, so that the U-shaped DiT structure is implemented by using the internal structure of the straight-cylinder DiT module, so as to process image features of different scales and improve the image enhancement effect.

[0014] In a possible implementation, the foregoing image processing model can be specifically used in a visual task, such as image super-resolution, deblurring, or defogging, to adapt to various image processing tasks and have strong generalization.

[0015] In addition, in a possible scenario, the image processing model can be pre-trained, for example, a low-resolution image is used as the model input, and a high-definition image is used as the supervision to learn the image processing model, so that the image model learns the parameters that can be used to output a high-definition image, and the image enhancement effect of the image processing model is improved.

[0016] In a second aspect, an embodiment of the present application provides an image processing device, comprising:

[0017] The input module is configured to input an input image.

[0018] The processing module is configured to input the input image into the image processing model to obtain an output image.

[0019] The image processing model comprises a plurality of modules, each module is configured to convert the input first image feature into a spectrum graph, modulate the spectrum graph by using a time step to obtain a modulated spectrum graph, and convert the modulated spectrum graph into a second image feature, and the second image feature is used to obtain the output image.

[0020] The effects achieved by the second aspect or any optional implementation of the second aspect can be referred to the description of the foregoing first aspect or any optional implementation of the first aspect, which will not be described here.

[0021] In a possible implementation, each of the foregoing modules is specifically configured to: obtain a spectrum graph corresponding to the first image feature; obtain a frequency modulation matrix according to a time step, the time step being a time interval for diffusing the first image feature; modulate the spectrum graph according to the frequency modulation matrix to obtain a modulated spectrum graph, elements in the frequency modulation matrix being used to weight frequency components in the spectrum graph; and convert the modulated spectrum graph into the second image feature.

[0022] In a possible implementation, each of the foregoing modules is specifically configured to: divide the first image feature to obtain a plurality of blocks; and convert the plurality of blocks into the spectrum graph.

[0023] In a possible implementation, each of the foregoing modules is specifically configured to: map the time step to a preset space to obtain a mapping vector; and reshape the mapping vector to obtain the frequency modulation matrix.

[0024] In a possible implementation, each of the foregoing modules is specifically configured to: perform Fourier transform on the first image feature to obtain a frequency spectrum corresponding to the first image feature; and perform inverse Fourier transform on the modulated frequency spectrum to obtain the second image feature.

[0025] In a possible implementation, the plurality of modules are connected in a U shape, and the number of processing channels in each module is the same, and the plurality of modules process features of multiple scales.

[0026] In a possible implementation, the image processing model is used for super-resolution, deblurring, or defogging of an input image.

[0027] In a third aspect, an embodiment of the present application provides an image processing apparatus, including a processor and a memory, wherein the processor and the memory are interconnected through a circuit, the processor invokes program code in the memory to execute functions related to processing in the action prediction method shown in any one of the third aspect.

[0028] In a fourth aspect, an embodiment of the present application provides a digital processing chip or a chip, the chip including a processing unit and a communication interface, the processing unit obtaining program instructions through the communication interface, the program instructions being executed by the processing unit, and the processing unit being configured to execute functions related to processing in the first aspect or any one of the optional implementation manners of the first aspect.

[0029] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, including instructions, when the instructions are executed on a computer, causing the computer to execute the method in the first aspect or any one of the optional implementation manners of the first aspect.

[0030] In a sixth aspect, an embodiment of the present application provides a computer program product including computer programs / instructions, when the computer programs / instructions are executed by a processor, causing the processor to execute the method in the first aspect or any one of the optional implementation manners of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0031] FIG. 1 is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0032] FIG. 2 is a schematic diagram of an image processing method provided by an embodiment of the present application;

[0033] FIG. 3 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0034] FIG. 4 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0035] FIG. 5 is a schematic diagram of a structure of an image processing model provided by an embodiment of the present application;

[0036] FIG. 6 is a structural schematic diagram of another image processing model provided by an embodiment of the present application;

[0037] FIG. 7 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0038] FIG. 8 is a structural schematic diagram of another image processing model provided by an embodiment of the present application;

[0039] FIG. 9 is a structural schematic diagram of another image processing model provided by an embodiment of the present application;

[0040] FIG. 10 is a structural comparison schematic diagram of an image processing model provided by an embodiment of the present application;

[0041] FIG. 11 is a schematic diagram of the amount of computing resources occupied by an image processing model provided by an embodiment of the present application;

[0042] FIG. 12 is a flowchart of a model training stage provided by an embodiment of the present application;

[0043] FIG. 13 is a flowchart of an inference training stage provided by an embodiment of the present application;

[0044] FIG. 14 is a structural schematic diagram of an image processing apparatus provided by an embodiment of the present application;

[0045] FIG. 15 is a structural schematic diagram of another image processing apparatus provided by an embodiment of the present application;

[0046] FIG. 16 is a structural schematic diagram of a chip provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0048] First, for ease of understanding, some terms or concepts related to the embodiments of the present application are introduced.

[0049] (1) Neural network

[0050] The neural network can be composed of neural units, and the neural unit can refer to an operation unit with x s as input. The output of the operation unit can be:

[0051] where s = 1, 2, … n, n is a natural number greater than 1, and W sFor x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to perform a non-linear transformation on the features acquired in the neural network, converting the input signal of the neural unit into an output signal. The output signal of this activation function can be used as the input to the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together; that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be a region composed of several neural units.

[0052] (2) Convolutional Neural Network

[0053] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane can share weights, which are called the convolutional kernel. Shared weights can be understood as the way features are extracted being independent of their location. The convolutional kernel can be formalized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is that it reduces the connections between layers in the CNN, while also reducing the risk of overfitting.

[0054] (3) Probability diffusion process

[0055] Diffusion is the process by which data changes over time according to a certain pattern. Typically, in probability theory and statistics, diffusion is a solution to a stochastic differential equation. This is a continuous-time Markov process, representing a continuous sample path. Brownian motion, reflective Brownian motion, and the Ornstein-Uhlenbeck process are all examples of diffusion.

[0056] (4) Markov-related concepts

[0057] Markov decision process (MDP): a mathematical model for analyzing decision problems based on discrete-time stochastic control. It assumes that the environment has Markov property (the conditional probability distribution of the future state of the environment depends only on the current state), and the decision maker makes decisions (also known as actions) according to the current state of the environment by periodically observing the state of the environment, and gets the next state and reward after interacting with the environment. In other words, at each time t, the state s(t) observed by the decision maker will be transferred to the next state s(t+1) under the influence of the action a(t) taken, and the reward r(t) will be fed back. Where s(t) represents the state function, a(t) represents the action function, r(t) represents the reward, and t represents the time.

[0058] Markov chain (MC): a stochastic process that assumes a transition from one state to another. For a state s, the next state depends on the current state, and is independent of the previous states, i.e. s(t+1) is only related to s(t). Therefore, when state transition is performed by Markov chain, the state transition path may have a ring structure. The path formed by the plurality of state nodes in the method provided in the application can include the path formed based on Markov chain.

[0059] State transition matrix: in Markov decision process, each state is transformed into the next state with a certain probability, and the transition probability between a plurality of state nodes can be represented as a matrix. In the method provided in the application, the transition probability between states can be learned through a policy network, i.e. the state transition matrix can be output through the policy network, and the learning can be completed after the transition probability converges.

[0060] (5) Generative model

[0061] Generative model refers to a model that can randomly generate observation data, especially under the condition of giving certain hidden parameters. It specifies a joint probability distribution for observation values and labeled data sequences. In machine learning, a generative model can be used to directly model data (e.g. sampling data according to the probability density function of a certain variable), or to establish the conditional probability distribution between variables. The conditional probability distribution can be formed by the generative model according to Bayes' theorem.

[0062] (6) Diffusion model

[0063] Diffusion models are a class of generative models that learn how to recover target data from noisy data step by step, eventually generating realistic samples. The core idea of this class of models is to gradually destroy the training data (i.e., add noise) and learn how to recover the original data in reverse. Diffusion models have received widespread attention in image generation, audio generation, and other tasks due to their high-quality and stable generation.

[0064] Taking the image super-resolution task as an example, given a low-resolution image y and its corresponding high-resolution image x0, the image super-resolution method based on residual transfer diffusion model aims to model the conditional distribution q(x0|y). Generally, these methods define a forward process containing T steps, which gradually introduces random noise to x0, and this process can be realized in one step through the reparameterization trick: q(x t |x0,y0)=N(x t ; x0+η t e0,κ 2 η t I),t=1,2,…T,

[0065] where the transition sequence gradually changes from η1→0 to η T →1,κ controls the noise variance. The denoising process can be represented as:

[0066] where the clean high-resolution image x0 is directly predicted by the denoiser f θ (x t ,y0,t) at each step, and gradually refined, finally outputting the optimized high-resolution image x0.

[0067] (7) Image super-resolution

[0068] Image super-resolution, also known as image super-resolution, is a solution to enhance image imaging effects and improve image quality. Its goal is to enlarge low-resolution images while maintaining image visual quality and details, so that they can still retain as much original information as possible at higher resolution. Generally, low-resolution images are caused by insufficient sampling rate, information loss during compression or transmission, etc., and super-resolution technology tries to recover these lost details.

[0069] (8)transformer

[0070] The transformer structure is a feature extraction network that includes an encoder and a decoder. Of course, there are cases where the transformer structure does not include an encoder, but includes a decoder.

[0071] Encoder: learn features in global receptive field through self-attention mechanism, such as the features of pixels.

[0072] Decoder: learn features of required modules through self-attention and cross-attention mechanisms, such as the features of output frames.

[0073] Exemplarily, the structure of the Transformer layer in an existing scheme can include a multi-head attention network and a forward network module. In the case of natural language processing, the multi-head attention network obtains corresponding weight values by calculating the correlation between words and words, and obtains context-related word representations, which is the core part of the transformer structure; the forward network further transforms the obtained representations to obtain the final output of the Transformer layer. In addition to the two important components, a residual layer (ADD) and a linear normalization (Norm) are respectively stacked on the two components to optimize the output of the Transformer layer.

[0074] (9) Attention mechanism

[0075] The attention mechanism can quickly extract important features of sparse data. The attention mechanism provides an effective modeling method for capturing global context information through QKV. Assuming that the input is Q (query), and the context is stored in the form of key-value pair (K, V). Then the attention mechanism is actually a mapping function from query to a series of key-value pairs. The essence of the attention function can be described as a mapping from a query to a series of (key, value) pairs. Attention essentially assigns a weight coefficient to each element in the sequence, which can also be understood as soft addressing. If each element in the sequence is stored in the form of (K, V), then attention completes the addressing by calculating the similarity between Q and K. The similarity calculated by Q and K reflects the importance of the extracted V value, i.e., the weight, and then the weighted sum is obtained to obtain the final feature value.

[0076] The calculation of attention mainly includes three steps. The first step is to calculate the similarity between query and each key to obtain the weight. Common similarity functions include dot product, concatenation, and perception, etc. Then the second step is to normalize these weights using a softmax function (on the one hand, it can be normalized to obtain a probability distribution whose sum of all weight coefficients is 1. On the other hand, the characteristic of the softmax function can highlight the weight of important elements); Finally, the weighted sum of the weight and the corresponding key-value value is obtained to obtain the final feature value. The specific calculation formula can be as follows:

[0077] where d is the dimension of the matrix Q, K.

[0078] In addition, attention includes self-attention and cross-attention. Self-attention can be understood as a special attention, that is, the input of QKV is consistent. While the input of QKV in cross-attention is inconsistent. Attention is to use the similarity (such as inner product) between features as weight to integrate the query feature as the update value of the current feature. Self-attention is the attention extracted based on the attention of the feature map itself.

[0079] For convolution, the setting of the convolution kernel limits the size of the receptive field, resulting in the network often needing multiple layers of stacking to focus on the entire feature map. The advantage of self-attention is that its attention is global, and it can obtain the global spatial information of the feature map through simple query and assignment.

[0080] (10) Multi-head attention (MHA)

[0081] When given the same set of queries, keys, and values, the model is expected to learn different behaviors based on the same attention mechanism, and then combine different behaviors as knowledge, for example, to capture various ranges of dependencies within a sequence (e.g., short-range dependencies and long-range dependencies). Therefore, the attention mechanism combines different subspaces of representations of queries, keys, and values. Compared with the self-attention model, the multi-head attention model increases the number of heads, for example, Q, K, V are first passed through a linear layer, and then input to the scaled dot-product attention (i.e., Scaled Dot-Product Attention), repeated h times, that is, multi-head, each time a head is calculated, the parameters are not shared between heads, and the parameters W of the linear transformation of Q, K, and V are different each time. Then concatenate the results of the h times of scaled dot-product attention, and then perform a linear transformation (linear layer) on the obtained value to obtain the result of the multi-head attention.

[0082] (11) Diffusion transformer (DiT)

[0083] is a diffusion model combined with the transformer architecture, aiming to efficiently capture data dependencies and generate high-quality results in image and video generation tasks. It simulates a diffusion process to gradually add noise to the data, and then learns to reverse this process to construct the required data samples from the noise.

[0084] (12) time step

[0085] The diffusion model simulates the generation process of data by gradually adding Gaussian noise to the data during the training process, which can be regarded as a series of steps on the time axis. Each time step represents a specific stage in the data generation or conversion process on the time axis. Through the gradual conversion of these stages, the model can finally generate meaningful data samples from random noise.

[0086] In the field of computer vision, image processing includes various scenarios such as image super-resolution, dehazing, or deblurring. Taking image super-resolution as an example, image super-resolution aims to reconstruct high-resolution images from low-resolution inputs. Diffusion models have shown excellent performance in image generation, so applying diffusion models to image processing fields such as image super-resolution scenarios will become a more effective image processing method.

[0087] In conventional diffusion models, a denoising step is usually set, and the same denoiser is usually reused. In the diffusion process, the time step is explicitly injected into the denoiser as a condition to modulate the intermediate features, and the effectiveness of the modulation strategy will directly affect the performance of the diffusion model. The existing modulation method usually uses the time step to directly modulate the extracted image features, and the output modulation effect cannot achieve very good results, and there is a lot of room for improvement in the modulation effect. And the commonly used DiT usually adopts a straight tube structure, all sub-modules are configured the same, and the feature map always maintains the same resolution and channel dimension.

[0088] For example, in the LDM (High-Resolution Image Synthesis with Latent Diffusion Models) scheme provided by the existing scheme, the input is first compressed from the pixel space to the latent space through the VAE encoder, and the diffusion model performs noise addition and denoising in the latent space. The architecture of the denoiser is a U-Net, which is stacked by residual modules and attention modules at each layer. Further, ResShift follows the LDM architecture, improves the noise scheduling strategy of the diffusion model, and shortens the Markov chain, reducing the number of denoising steps to 15. This method does not require large-scale pre-training, and when the model is deployed in a lightweight manner, it can be easily retrained for the improved structure. However, the performance upper limit of the diffusion model image super-resolution based on the LDM architecture is usually lower than that of the pre-training prior method.

[0089] For example, in the case of DiffBIR, the pre-trained text-to-image model is frozen during training, and an additional ControlNet is added to introduce low-resolution images and only train this part. The rich prior knowledge contained in the frozen text-to-image model can guide the reconstruction of high-resolution images, and its performance upper limit is higher than that of the method of training from scratch. However, the pre-trained model has a fixed structure and a large number of parameters, which makes the resources required for pre-training expensive, and it is usually difficult to improve the network structure, which limits its application in lightweight scenarios.

[0090] For example, in a specific DiT architecture, the convolutional neural network is discarded, and a straight-cylinder architecture using pure Transformer is used, and the resolution and channels of the feature map do not change. Time modulation is performed after the attention module and the linear layer, respectively, and time step information is injected into the attention module. However, the time modulation strategy usually acts on the channel dimension, and the same modulation parameters are simply used for high and low frequency components of the image, which limits the performance upper limit of the diffusion model. Moreover, the straight-cylinder DiT architecture lacks the multi-scale feature extraction capability required in bottom-level vision, which limits its image processing effect.

[0091] Therefore, the embodiments of the present application provide an image processing method, which can convert image features into the frequency domain and modulate frequency domain features using time steps, thereby realizing adaptive modulation of each frequency component in the frequency domain and outputting an image with better performance.

[0092] The method provided by the embodiments of the present application can be applied to various image processing fields and can be deployed in functional servers or terminal electronic devices. For example, the method provided by the present application can be deployed in an electronic device, and a user can directly use the electronic device for image processing, or it can also be deployed in the cloud to provide image processing services for user terminals. Different deployment methods will be introduced below.

[0093] Deployment method one, deployed in an electronic device

[0094] The electronic device provided in the embodiments of the present application can specifically include a handheld device, a vehicle-mounted device, a data processing device, a computing device, and the like, and can include an image sensor or be connected with an image sensor. The electronic device can also include a digital camera, a cellular phone, a camera, a smart phone, a personal digital assistant (PDA) computer, a tablet computer, a laptop computer, a machine type communication (MTC) terminal, a point of sales (POS), a vehicle-mounted computer, a head-mounted device, a data processing device (such as a bracelet, a smart watch, and the like), a security device, a virtual reality (VR) device, an augmented reality (AR) device, and other electronic devices with an imaging function.

[0095] Deployment mode two: deployed in the cloud

[0096] The embodiments of the present application also provide a cloud, and one or more terminals access the platform. The method provided in the present application can be deployed in the cloud to provide image denoising or enhancement services for the terminal.

[0097] For example, FIG. 1 is an application scenario of the method provided in the present application, which can include a cloud 11 and a terminal 12. The cloud 11 and the terminal 12 can be connected through a wire or wirelessly.

[0098] The cloud 11 can specifically include a server cluster with storage and processing functions. The method provided in the embodiments of the present application can be deployed in the cloud 11, specifically, can receive multiple frames of images from the terminal 12, and after enhancing the images through the method provided in the embodiments of the present application, feed back the enhanced images to the terminal 12.

[0099] The terminal 12 can implement image processing by interacting with the cloud 11. The terminal can specifically include, but is not limited to, for example, a personal computer, a computer workstation, a smart phone, a tablet computer, a notebook computer, a smart car, and the like. The terminal 12 can transmit an image to the cloud 11, which can be an image captured by the terminal itself, an image including user input, or an image stored locally in the terminal, and the like. For example, the cloud can provide services for users through a client deployed in the terminal or a web page in the terminal, and taking the deployment of the client in the terminal as an example, the user can send an image collected by the terminal, such as an image captured by the terminal, to the cloud through the client deployed in the terminal, and the cloud 11 performs super-resolution, dehazing, or deblurring on the image through the method provided in the embodiments of the present application, and outputs a high-definition image and feeds back the terminal 12.

[0100] In a possible scenario, it can also be applied to a scenario of multiple terminals, for example, a user can use another terminal different from the terminal 12 to capture the aforementioned image, and transmit the image to the terminal 12, and the terminal 12 feeds back the enhanced image to the terminal 12 after uploading the image to the cloud 11 for image denoising or enhancement.

[0101] The method provided in the embodiments of the present application will be described below.

[0102] Referring to FIG. 2, a flowchart of an image processing method provided in the embodiments of the present application is shown,

[0103] 201, an input image is acquired.

[0104] The input image can specifically include a received image, an image read from a storage space, or a captured image, and the like. The input image can specifically be an image that needs to be enhanced. For example, in some scenarios, the captured image is not high-definition, a received low-code-rate compressed image, or a stored compressed image, and the like, which can be used as the input image of the method provided in the embodiments of the present application to perform image enhancement to obtain a higher-definition output image.

[0105] For example, if the method provided in the embodiments of the present application is deployed in a terminal, the input image can be an image captured by the terminal, an image captured by another device received by the terminal, or a low-resolution image read from a storage space, and the like.

[0106] For another example, if the method provided in the embodiments of the present application is deployed in a server, the input image can be an image sent by a client received by the server, or an image received from another server, and the like.

[0107] 202、inputting the input image into the image processing model to obtain an output image, wherein the image processing model converts image features into frequency domain features, modulates the frequency domain features using time steps, converts the modulated frequency domain features into spatial domain features, and obtains the output image based on the spatial domain features.

[0108] The image processing model includes a plurality of modules, each of which can be used to convert the input first image features into a frequency spectrum, modulate the frequency spectrum using time steps to obtain a modulated frequency spectrum, and convert the modulated frequency spectrum into second image features, which are used to obtain the output image.

[0109] Specifically, the image processing model can specifically include a model implemented based on the aforementioned DiT, and in the diffusion process, the frequency components in the frequency domain features are adaptively modulated using time steps, thereby realizing frequency domain modulation of the image features.

[0110] Optionally, the image processing model can be specifically used to perform tasks such as super-resolution, defogging, or deblurring, and can be applied in various underlying vision tasks, and has very strong generalization.

[0111] In the embodiments of the present application, in the process of enhancing the image, the diffusion mechanism is used to modulate the frequency domain components in the image features using time steps in the frequency domain, thereby realizing adaptive modulation of the frequency domain components. Generally, there is a correlation between the time steps and the frequencies in the diffusion process of image processing, and therefore adaptive modulation of the frequency components in the image features using time steps can enhance the image in the frequency dimension to realize effects such as super-resolution, defogging, or deblurring, and greatly improves the image enhancement effect.

[0112] Specifically, each module can be used to obtain a spectrum corresponding to the first image feature, that is, to convert the image feature in the spatial domain into a spectrum in the frequency domain; then a frequency modulation matrix is obtained according to a time step, which is a time interval for diffusing the first image feature, which can be a preset amount, or a value trained in the model training process, etc.; then the spectrum is modulated according to the frequency modulation matrix to obtain a modulated spectrum, and the elements in the frequency modulation matrix are used to weight the frequency components in the spectrum, that is, to adaptively modulate each component in the spectrum based on the time step based on the frequency modulation matrix; then the modulated spectrum is converted into a second image feature, that is, the modulated spectrum is converted into an image feature in the spatial domain; and the output image is obtained according to the second image feature. In the embodiment of the application, the image feature in the spatial domain is converted into the spectrum in the frequency domain, so that the frequency domain components can be adaptively modulated based on the time step. In the diffusion process, the time step has a correlation with the frequency components in the image, so that the adaptive modulation of the frequency components based on the time step in the embodiment of the application can achieve better image processing effect.

[0113] In a possible implementation, in order to improve the effect of image processing, further, in each module, the input first image feature can also be divided into a plurality of patches, and then the plurality of patches are respectively converted into corresponding spectrums. Therefore, in the embodiment of the application, the spatial domain features of the image can be divided into a plurality of patches, that is, the spatial domain features are divided into smaller granularity, so that when modulating, different resolution input features can be cut to adapt to different resolution input features to achieve higher inference efficiency.

[0114] Optionally, in the process of determining the frequency modulation matrix based on the time step, the time step can be mapped to a space of a preset dimension, and the frequency modulation matrix is obtained by reshaping in the space. For example, the time step can be mapped to a two-dimensional vector, and the time step mapped to the vector is reshaped to output the frequency modulation matrix corresponding to each frequency component. The frequency modulation matrix can include elements corresponding to each frequency component, and the elements in the frequency modulation matrix are equivalent to a weight value of a frequency component, and different weight values can be learned for high frequency components and low frequency components to adaptively enhance the high frequency components and the low frequency components, so as to realize modulation in the frequency domain and improve the image processing effect.

[0115] Optionally, the manner of realizing the conversion between the spatial domain and the frequency domain can specifically adopt the manner of Fourier transform and inverse Fourier transform. For example, Fourier transform can be performed on the first image feature to obtain a frequency spectrum corresponding to the first image feature; and inverse Fourier transform can be performed on the modulated frequency spectrum to obtain the second image feature. Therefore, in the embodiments of the present application, Fourier transform can be used to realize the conversion between the spatial domain and the frequency domain, so as to perform modulation in the frequency domain, thereby realizing adaptive modulation of the frequency components.

[0116] In addition, optionally, the DiT provided in the embodiments of the present application can adopt a U-shaped structure, and the number of channels processed by each transformer module in the U-shaped structure is the same, so that multi-scale feature processing is realized through the U-shaped structure, the accuracy of feature processing is improved, more accurate image features are obtained, and the same number of channel processing is adopted, which is equivalent to performing calculation resource reallocation within each resolution, so that the same number of channel processing can be used for processing features of each resolution. Compared with the existing U-shaped transformer architecture, more channel numbers are allocated to high-resolution features. Generally, high-frequency details captured in the high-resolution stage are crucial for image super-resolution, and therefore the image super-resolution effect of the final output can be improved.

[0117] Specifically, for the U-shaped DiT provided in the embodiments of the present application, different scale feature processing is realized through the U-shaped structure, and the same number of channels is processed in each transformer. For example, different scale spatial features can be mapped to a space with the same number of channels, so that different scale feature processing is performed in the space with the same number of channels, and the output features can be up-sampled or down-sampled to output features with a scale that is adapted to the input features of the next layer transformer. It can be understood that the transformer modules in the DiT architecture can form an encoding module-decoding module pair, which is used to encode and decode features of the same scale, so as to realize the U-shaped DiT that can process multiple scales.

[0118] Compared with the existing U-shaped model, the DiT with a U-shaped structure provided in the embodiments of the present application adopts the same size transformer module in the transformer of different scales, and compared with the existing scale-adaptive channel number, the DiT with a U-shaped structure provided in the embodiments of the present application is equivalent to allocating more calculation resources to high-resolution features, so as to improve the attention to high-resolution features and make the enhancement effect of the final output image better.

[0119] The foregoing introduces the method flow provided by the embodiments of the present application. In the following, the method flow provided by the embodiments of the present application is further introduced in combination with a specific application scenario.

[0120] The embodiments of the present application provide a diffusion model architecture trained from scratch, which can be comparable to the prior-based method, thereby achieving a balance between performance and flexibility. The Diffusion Transformer architecture is referred to the image processing task, such as image super-resolution, deblurring or defogging, and the like, to convert the inefficient spatial-temporal modulation to the frequency domain and realize adaptive modulation of different frequency components. For the DiT model structure, the embodiments of the present application provide a U-shaped DiT architecture combined with a straight tube module design to make up for the lack of multi-scale characteristics of the straight tube architecture.

[0121] Firstly, the application scenarios of the method provided by the embodiments of the present application can include various image enhancement scenarios, such as image super-resolution, defogging or deblurring scenarios.

[0122] For example, as shown in FIG. 3, low-quality video data collected by various environmental shooting devices can be collected and stored in a memory, such as smart city, home monitoring, in-vehicle monitoring, pet monitoring and the like. When playing the video data, the image in the video data can be enhanced by the image processing method provided by the present application, such as defogging or deblurring, thereby obtaining video data with higher clarity and improving the user's viewing experience.

[0123] For another example, the image processing method provided by the present application can be applied to a video live streaming scenario, as shown in FIG. 4, a server can send a video stream to a client used by a user. After the client receives the data stream sent by the server, the image in the data stream can be enhanced by the image processing method provided by the present application, thereby obtaining video data with higher clarity and improving the user's viewing experience.

[0124] For another example, in the intelligent driving scenario, the images around the vehicle can be shot by the camera arranged on the intelligent vehicle, and the images can be used to plan the driving path for the vehicle or make driving decisions. The image shot by the camera can be enhanced by the image processing method provided by the present application to obtain clearer images, especially in the foggy scenario, to realize the defogging effect, thereby improving the driving safety of the vehicle and improving the user experience.

[0125] For another example, the user can use the terminal to take a picture in the environment with fog, and the image shot may not be clear due to the influence of the fog. The image shot can be defogged by the image processing method provided by the present application, thereby improving the clarity of the image and improving the user experience.

[0126] For example, in some scenes with large differences in ambient light, the image captured by the user can be unclear due to low contrast, and the image processing method provided in the present application can be used to process the captured image to make the image clearer and improve the user experience.

[0127] In combination with the foregoing application scenarios, the DiT architecture and method provided in the embodiments of the present application will be described in detail below taking image super-resolution as an example. The super-resolution mentioned below can also be replaced by dehazing or deblurring according to actual scenarios.

[0128] Referring to FIG. 5, the architecture of a DiT provided in an embodiment of the present application is shown.

[0129] In the DiT architecture provided in the embodiments of the present application, the DiT includes N transformer modules. In each transformer module, the input feature is converted from spatial domain to frequency domain, and the frequency domain feature is modulated based on the time step in the diffusion process to achieve adaptive modulation.

[0130] Specifically, in each transformer module, the time step modulation is converted from spatial domain to frequency domain, and different modulation parameters are used for different frequency components, so that the diffusion model can better focus on the corresponding frequency components at different time steps, enhancing the frequency perception ability of the diffusion model and improving the model performance. Through analysis of the denoising process of the diffusion model, it can be found that there is a strong correlation between the time step and the frequency, and usually low frequencies are generated first and then high frequencies are generated. Therefore, by modulating the time step in the frequency domain, adaptive modulation of different frequency components in the frequency domain can be achieved, and better image super-resolution effect can be achieved.

[0131] Exemplarily, the structure of one of the transformer modules can be as shown in FIG. 6.

[0132] Each transformer module can include Norm, ADD, MHA and other network layers. The method provided in the embodiments of the present application can be deployed as an adaptive frequency modulation (AdaFM) layer, which is used to convert the time step modulation from spatial domain to frequency domain for modulation, and convert the modulated frequency domain information into spatial domain features. After upsampling and downsampling, each transformer module normalizes the channel number to the same value by calculating the resource redistribution. Inside each module, it can include a self-attention module and a feedforward neural network for performing channel mixing. The normalization layer and the AdaFM proposed in the embodiments of the present application are added before the two modules, respectively, and a residual connection is added.

[0133] The specific process performed in the AdaFM module is as follows:

[0134] Firstly, to adapt to different input resolutions and improve efficiency, the spatial feature map f spat ∈R C×H×W is divided into p×p size windows to facilitate the subsequent conversion of window features to frequency domain features.

[0135] Subsequently, a fast Fourier transform is performed within each window to convert the time modulation of the diffusion model from the spatial domain to the frequency domain. Specifically, a fast Fourier transform (FFT) can be used to convert each window to a frequency spectrum f The time step is mapped to a p 2 dimensional vector and reshaped into a frequency modulation matrix S spec ∈R p×p for adaptively reweighting different frequency components, thereby enhancing the ability of the diffusion model to emphasize specific frequencies at different time steps. This is equivalent to converting the time step information from the spatial domain to the frequency domain, adaptively modulating the weights of different frequency components, enhancing the perception ability of the diffusion model at different time steps for corresponding frequencies, improving the model performance while greatly reducing the parameter amount. And this module is plug and play, which can be applied to any diffusion model to replace its original time modulation module.

[0136] Subsequently, the modulated frequency spectrum f′ spec is converted from the frequency domain to the spatial domain through an inverse fast Fourier transform. That is, the time step is mapped to a two-dimensional modulation matrix of the same size as the window, and the two-dimensional and frequency spectrum are multiplied element by element. The modulated feature is converted from the frequency domain to the spatial domain through an inverse fast Fourier transform.

[0137] In the frequency spectrum, each pixel at a specific spatial position corresponds to a predetermined frequency component, which is defined only by the spatial dimension of the feature map and is independent of its content. Therefore, the consistency relationship between its frequency and spatial features allows the same frequency scaling matrix S spec to be applied across all windows and channels, greatly reducing the required parameter amount and enabling adaptive adjustment of each frequency domain component of the feature. This is equivalent to converting the spatial feature to a frequency domain feature, and converting it to the same number of frequency domain features in each scale module, so that the same number of frequency domain features are processed in each transformer.

[0138] The entire process of AdaFM can be represented as:

[0139] Time step reshaping: S spec = reshape(f time ,p×p),

[0140] Conversion of spatial features to frequency components: f spec = FFT(P(fspat )),

[0141] Modulation: f' spec = S spec ⊙f spec ,

[0142] Frequency component conversion to spatial feature: f out = P -1 (iFFT(f' spec )).

[0143] where P and P -1 represent the splitting and merging operations of the window, and FFT and iFFT represent the fast Fourier transform and its inverse transform. Since different frequencies correspond to different spatial positions on the feature map, the proposed frequency time step adjustment module actually uses different modulation parameters for different spatial positions, that is, it performs adaptive modulation based on the time step for different frequency components, thereby more accurately adapting the relationship between the time step and the frequency component in the actual scene, and outputting more accurate image features.

[0144] Of course, the algorithm in the foregoing process can also be replaced by other algorithms with the same or similar functions, which can be determined according to the actual application scenario, and the embodiments of the present application are not limited to the specific algorithm used.

[0145] Since the diffusion model uses the same denoiser at different time steps, it is crucial to explicitly include the time step as a condition, which directly affects the performance upper limit of the diffusion model. Through the analysis of the time evolution of the image predicted by the diffusion-based super-resolution model, the model usually focuses on different frequency components at different denoising stages. For example, as shown in FIG. 7, the model initially reconstructs the low-frequency components related to the structure of the image, and then gradually refines the high-frequency details related to the texture, that is, the low-frequency components affect the structure of the image, and the high-frequency components affect the texture of the image. Through the analysis of the denoising process of the diffusion model, it can be known that the time step has a strong correlation with the frequency, and generally low frequencies are generated first and then high frequencies. This module fully considers this characteristic and converts the time step modulation from the spatial domain to the frequency domain, using different modulation parameters for different frequency components, so that the diffusion model pays more attention to the corresponding frequency components at different time steps, enhancing the frequency perception ability of the diffusion model and improving the model performance. Therefore, in the embodiments of the present application, the time step is adaptively adjusted for different frequency components, different modulation parameters are used for high-frequency and low-frequency regions, so as to adaptively adjust the low-frequency components and high-frequency components in the image features, and improve the image enhancement effect.

[0146] Different from image generation tasks starting from pure noise and mainly focusing on semantic information, the super-resolution task emphasizes the recovery of high-frequency details of the image, requiring the diffusion model to have strong frequency perception ability. The existing adaptive layer normalization (AdaLN) can also be applied to the DiT model. AdaLN usually modulates the feature map in the channel dimension, and applies the same modulation parameter to all spatial positions. It does not modulate high-frequency components and low-frequency components. And it is challenging to generate spatially related modulation parameters from a one-dimensional time step vector. Therefore, it can be replaced with the AdaFM provided in the embodiments of the present application, or the AdaFM provided in the embodiments of the present application is added to the transformer module to adaptively distinguish the positions of high-frequency components and low-frequency components in the image.

[0147] For example, another structure of the transformer module provided in the embodiments of the present application can be as shown in FIG. 8. In one transformer module, an AdaLN module and an AdaFM module can be arranged, the channel modulation, that is, the modulation in the spatial dimension is performed by the AdaLN module, and the modulation in the frequency domain dimension is performed by the AdaFM module, so as to realize the modulation in the spatial and frequency domain dimensions of the image, the effects can be complementary, and thus the effect of image super-resolution can be further improved. Of course, the positions of the AdaLN module and the AdaFM module shown in FIG. 8 can also be interchanged or adjusted to other positions. Specifically, the positions of the AdaLN module and the AdaFM module can be exemplarily introduced in the embodiments of the present application, and are not limited.

[0148] Generally, the frequency adaptive modulation module is more efficient, and the parameter amount is greatly reduced. Compared with the existing spatial modulation module (such as AdaLN), the frequency adaptive modulation module uses different modulation parameters for different frequency components, enhances the frequency perception ability of the diffusion model, and is more efficient than the method of only modulating in the channel dimension. The module utilizes the consistency of frequency and resolution in the frequency spectrum diagram, and shares the modulation parameters for all windows, and the parameter amount is greatly reduced.

[0149] In addition, the DiT provided in the embodiments of the present application can adopt a U-shaped structure, but the number of channels processed by each transformer remains the same or consistent. The frequency domain modulation is performed on features of different scales through the U-shaped structure, and the straight tube type transformer architecture is combined to configure processing resources for the processing process of each scale.

[0150] For example, the DiT architecture provided in the embodiments of the present application can be as shown in FIG. 9, in which the U-shaped structure is used to connect between the transformer modules, and the transformer modules of adjacent scales are connected. A plurality of encoding layers are first arranged to encode the input features of different scales, and a decoding layer corresponding to each scale is also arranged to process the transformer modules of the same resolution, that is, the encoding layer and the decoding layer are connected. The specific structure of each transformer can be referred to FIG. 6 or FIG. 8. The processing channel dimension of each transformer is the same, and the plurality of transformer modules can process input features of multiple scales.

[0151] Generally, the DiT with consistent channels and depth, that is, the straight-cylinder DiT can effectively process input of different window sizes, such as DiT-XL / 2, DiT-XL / 4 or DiT-XL / 8 model. Secondly, the DiT at high resolution (such as DiT / 2) has better effect than the DiT at low resolution (such as DiT / 8) and benefits more from model enlargement. However, compared with the network for multi-scale feature extraction, the straight-cylinder DiT lacks the processing ability of features of different scales. Therefore, in the DiT architecture provided in the embodiments of the present application, the U-shaped structure is arranged on the basis of the transformer module structure of the straight-cylinder DiT to obtain the U-shaped DiT with multi-scale feature processing ability.

[0152] It can be understood that the U-shaped structure is used on the model, and as the depth increases, the resolution of the feature map decreases and the channel dimension increases. The straight-cylinder isotropic design is used for each transformer module, and the channel numbers of all modules are normalized to the same value after up-sampling and down-sampling. This strategy allocates more computing resources to more important high-resolution layers, which pay more attention to high-frequency details in the super-resolution task, thereby improving the super-resolution effect of the image.

[0153] For example, the comparison between the straight-cylinder DiT, the DiT with different channels and depths of U-shaped structure (referred to as U-shaped DiT) and the U-shaped structure DiT provided in the embodiments of the present application can be as shown in FIG. 10.

[0154] In the straight-cylinder DiT, the number of channels of the transformer module is uniformly set as c1, and each transformer module processes image features of the same scale. In the U-shaped DiT, different transformer modules can be set to different numbers of channels, such as Nc2. For example, in the 3-layer scale structure shown in FIG. 10, the corresponding numbers of channels are set as c2, 2c2, and 4c2, respectively, and each transformer module processes features of the corresponding scale. In the DiT structure provided in the embodiments of the present application, the same number of processing channels is configured for each transformer module, such as c3, that is, c3>c2. Generally, c2<c3<Nc2. Compared with the U-shaped DiT, the DiT provided in the embodiments of the present application configures more processing resources for high-resolution features, thereby improving the attention to high-resolution features. It can be understood that the multi-scale U-shaped DiT provided in the embodiments of the present application introduces the characteristic that the number of channel dimensions of the transformer module in the straight-cylinder DiT is consistent, and more computing resources are redistributed to more important high-resolution layers. Exemplarily, the comparison of the computing resources of each scale of the U-shaped DiT and the DiT (i.e., Isotropic Design) provided in the embodiments of the present application can be as shown in FIG. 11. Specifically, each scale is composed of several transformer modules operating at the same resolution. First, the overall structure conforms to the U-shaped design, and as the depth increases, the feature map resolution decreases and the number of channels increases. Second, after upsampling and downsampling, the channel dimension of the normalized features is standardized within each scale, that is, Nc2 is replaced by c3, so that the processing channel dimension of each scale in the model remains consistent, the transformer module is executed in the redistributed feature space, and then the feature scale is mapped back to the corresponding scale, such as mapping the features belonging to the c3 dimension back to the Nc2 scale, so that the features of different scales are processed in the same number of channels. The high-frequency details captured in the high-resolution stage are crucial for image super-resolution and benefit more from model enlargement, so the standardized channel dimension is set to be greater than the original high-resolution stage in the U-Net, but significantly smaller than the number of channels in the low-resolution stage. The DiT provided in the embodiments of the present application allocates computing resources to the key high-resolution layer without deploying complex scheduling strategies, greatly improves the ability of the Transformer architecture, and requires far fewer parameters than the traditional U-Net.

[0155] In the embodiments of the present application, a U-shaped structure is generally used, as the depth increases, the feature map resolution decreases, the channel dimension increases, and a multi-scale characteristic is introduced; a straight cylinder type isotropic design is used for each sub-module, the channel numbers of all modules are normalized to the same value after upsampling and downsampling, and the channel dimension after normalization is set to be greater than the high-resolution stage in the U-Net provided by the existing scheme, but significantly smaller than the channel number of the low-resolution stage. Therefore, by reallocating the computing resources of the U-shaped architecture, better performance is achieved with less parameter amount. Through analysis of DiT, it can be concluded that the performance of high-resolution DiT is better, the model amplification benefit is higher, and DiT with consistent channels and depth can effectively process inputs of different window sizes, and the straight cylinder type isotropic module design is introduced into the U-Net in the embodiments of the present application, avoiding the design of a complicated scheduling strategy, and allocating more computing resources to more important high-resolution layers to pay attention to high-frequency details, greatly reducing the parameter amount while improving the performance. It is equivalent to reallocating the computing resources of the U-shaped architecture, allocating more computing resources to more important high-resolution layers to improve the image processing effect.

[0156] In addition, the diffusion model all needs time modulation, and the correlation between frequency and time step in the denoising process exists in various tasks, so the AdaFM module provided in the embodiments of the present application can be applied to diffusion models in various scenes, and can be extended to more bottom visual tasks and even generation tasks, and has stronger generalization.

[0157] In addition, before deploying the DiT provided in the embodiments of the present application, the initial DiT can also be trained, and the trained DiT is deployed in the inference device.

[0158] For example, the flow of the training stage can be as shown in FIG. 12, which is an end-to-end training model from scratch, the diffusion model is randomly initialized, and after single-stage training on the synthetic degraded image super-resolution dataset, the inference task can be performed. In the inference stage, as shown in FIG. 13, any real scene low-resolution image can be directly input, and a higher quality high-resolution image can be generated.

[0159] To facilitate the understanding of the effect of the method provided by the embodiments of the present application, some existing schemes and the effect of the method provided by the embodiments of the present application are compared, for example, tested on open source real scene data sets RealSR, RealSet65 and synthetic data set LSDIR, as shown in Table 1, wherein the evaluation indexes can include the parameter quantity (#Params) and the indexes CLIPIAQ, MUSIQ (Multi-scale Image Quality Transformer) and MANIQA (Multi-dimension Attention Network for No-Reference Image Quality Assessment) for evaluating the image enhancement effect. Compared with the existing diffusion model super-resolution method trained from scratch, the method provided by the embodiments of the present application is far superior to other schemes in various indexes, and the parameter quantity is only about 50% of that of the other schemes. The existing schemes such as GAN based Methods (such as RealSR-JPEG, BSRGAN, RealESRGAN and SwinIR), Prior based method (such as StableSR-200, DiffBIR-50, PASD-20 and SeeSR-50) and Training-from-scratch diff based methods (such as LDM-100 and ResShin-15) can be classified as the method provided by the embodiments of the present application, that is, ours in the table is the scheme provided by the embodiments of the present application.

[0160] Table 1

[0161] Test on open source real scene data sets RealSR, RealSet65 and synthetic data set LSDIR, as shown in Table 2, compared with the existing diffusion model super-resolution method based on pre-training prior, the present patent achieves comparable performance, but the parameter quantity is less than 5% of that of the prior.

[0162] Table 2

[0163] Test the subjective effect on open source real scene data sets RealSR, RealSet65 and synthetic data set LSDIR, the method provided by the embodiments of the present application generates better details.

[0164] Table 3

[0165] In addition, in the ablation experiment, as shown in Table 3, the isotropic straight-cylinder standard DiT architecture has poor performance on super-resolution, and the performance is significantly improved after introducing the U-shaped architecture, that is, U-shape, but the parameter amount is increased by 6 times under the same FLOPs. After introducing the structure provided in the present application, the performance is obviously improved, and the parameter amount is reduced by 62%. After introducing the frequency adaptive modulation module AdaFM provided in the present application, the parameter amount is further reduced by 40%, the performance is further improved, and is superior to the TMSA modulation strategy proposed in DiffiT (diffusion vision transformers).

[0166] Obviously, with better objective indicators and subjective visual effects, the method provided in the embodiments of the present application can be applied to image super-resolution tasks, such as super-resolution of mobile phones / screen devices. The method provided in the embodiments of the present application can greatly surpass the current most advanced diffusion model super-resolution method trained from scratch, and can rival the current most advanced diffusion model super-resolution method based on pre-training prior, and the model parameter amount is less than 5% of it.

[0167] The foregoing introduces the method flow provided in the embodiments of the present application, and the following introduces the device structure for executing the method flow provided in the present application.

[0168] Referring to FIG. 14, the embodiments of the present application provide a structural schematic diagram of an image processing device, which includes:

[0169] The input module 1401 is configured to input the input image.

[0170] The processing module 1402 is configured to input the input image into the image processing model to obtain an output image.

[0171] The image processing model includes a plurality of modules, each module is configured to convert the input first image feature into a frequency spectrum, modulate the frequency spectrum using a time step to obtain a modulated frequency spectrum, and convert the modulated frequency spectrum into a second image feature, and the second image feature is used to obtain the output image.

[0172] In a possible implementation, each of the foregoing modules is specifically configured to: obtain a frequency spectrum corresponding to the first image feature; obtain a frequency modulation matrix according to a time step, the time step being a time interval for diffusion of the first image feature; modulate the frequency spectrum according to the frequency modulation matrix to obtain a modulated frequency spectrum, elements in the frequency modulation matrix being used to weight frequency components in the frequency spectrum; and convert the modulated frequency spectrum into the second image feature.

[0173] In a possible implementation, each of the foregoing modules is specifically configured to: divide the first image feature to obtain a plurality of blocks; and convert the plurality of blocks into the frequency spectrum.

[0174] In a possible implementation, each of the foregoing modules is specifically configured to: map the time step to a preset space to obtain a mapping vector; and reshape the mapping vector to obtain the frequency modulation matrix.

[0175] In a possible implementation, each of the foregoing modules is specifically configured to: perform Fourier transform on the first image feature to obtain a frequency spectrum corresponding to the first image feature; and perform inverse Fourier transform on the modulated frequency spectrum to obtain the second image feature.

[0176] In a possible implementation, the plurality of modules are connected in a U shape, the number of processing channels in each module is the same, and the plurality of modules process features of multiple scales.

[0177] In a possible implementation, the image processing model is used for super-resolution, deblurring or defogging of an input image.

[0178] As shown in FIG. 15, it is a schematic diagram of a hardware structure of an image processing apparatus 150 provided by an embodiment of the present application. The image processing apparatus 150 can be used to implement the steps of the method in the foregoing inference stage.

[0179] The image processing apparatus 150 shown in FIG. 15 can include a processor 1501, a memory 1502, a communication interface 1503 and a bus 1504. The processor 1501, the memory 1502 and the communication interface 1503 can be connected through the bus 1504.

[0180] The processor 1501 is the control center of the image processing apparatus 150, and can be a general central processing unit (CPU), or other general-purpose processors, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc., which can specifically include a GPU or an NPU, etc., and can be adaptively set according to actual application scenarios.

[0181] As an example, the processor 1501 can include one or more CPUs, and can also include other processors, such as the CPU, NPU or GPU shown in FIG. 15, etc.

[0182] The memory 1502 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0183] In a possible implementation, the memory 1502 can exist independently of the processor 1501. The memory 1502 can be connected to the processor 1501 through the bus 1504, for storing data, instructions, or program codes. When the processor 1501 invokes and executes the instructions or program codes stored in the memory 1502, the method provided by the embodiments of the present application can be implemented, for example, the action prediction method shown in the reasoning stage.

[0184] In another possible implementation, the memory 1502 can also be integrated with the processor 1501.

[0185] The communication interface 1503 is used for connecting the image processing apparatus 150 to other devices through a communication network, which can be an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface 1503 can include a receiving unit for receiving data, and a sending unit for sending data.

[0186] The bus 1504 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is shown in FIG. 15, but it does not mean that there is only one bus or only one type of bus.

[0187] It should be noted that the structure shown in FIG. 15 does not constitute a limitation on the image processing apparatus 150, and the image processing apparatus 150 can include more or fewer components than those shown in FIG. 15, or combine certain components, or different component arrangements, in addition to the components shown in FIG. 15.

[0188] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware, and of course can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits, or special circuits, etc. However, for the present application, software program implementation is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., including a plurality of instructions for causing an image processing apparatus (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0189] The computer readable storage medium in the embodiments of the present application stores a program for training a model or performing an inference task, which, when running on a computer, causes the computer to execute all or part of the steps of the methods described in the foregoing embodiments of FIG. 4 to FIG. 15.

[0190] The embodiments of the present application also provide a digital processing chip. The digital processing chip integrates a circuit for implementing the above processor or the function of the processor and one or more interfaces. When the digital processing chip integrates a memory, the digital processing chip can complete the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected with an external memory through a communication interface. The digital processing chip implements the method steps of any one or more of the above embodiments according to the program code stored in the external memory.

[0191] In the above embodiments, all or part can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part.

[0192] The embodiments of the present application further provide a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be stored by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0193] The image processing device provided by the embodiments of the present application can be a chip, which includes a processing unit and a communication unit. The processing unit can be a processor, and the communication unit can be an input / output interface, a pin, a circuit, etc. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the server executes the methods described in the embodiments shown in FIGS. 2-13. Alternatively, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM), etc.

[0194] In particular, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and the like. The general-purpose processor can be a microprocessor or any conventional processor, and the like.

[0195] Exemplarily, refer to FIG. 16, which is a structural schematic diagram of a chip provided by an embodiment of the present application. The chip can be a neural-network processing unit (NPU) 160, which is mounted on a host CPU as a coprocessor and is assigned tasks by the host CPU. The core part of the NPU is an operation circuit 1603, which extracts matrix data in a memory and performs multiplication operation under the control of a controller 1604.

[0196] In some implementations, the operation circuit 1603 internally includes a plurality of processing engines (PEs). In some implementations, the operation circuit 1603 is a two-dimensional systolic array. The operation circuit 1603 can also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1603 is a general-purpose matrix processor.

[0197] For example, assume that there are an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes corresponding data of the matrix B from a weight memory 1602 and buffers the data on each PE of the operation circuit. The operation circuit takes matrix A data from an input memory 1601 and performs matrix operation with the matrix B, and the partial result or final result of the matrix is saved in an accumulator 1608.

[0198] The unified memory 1606 is used to store input data and output data. The weight data is transferred to the weight memory 1602 through a direct memory access controller (DMAC) 1605. The input data is also transferred to the unified memory 1606 through the DMAC.

[0199] A bus interface unit (BIU) 1610 is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1609.

[0200] The bus interface unit 1610 (BIU) is used for the instruction fetch buffer 1609 to fetch instructions from the external memory, and is also used for the direct memory access controller 1605 to fetch the original data of the input matrix A or the weight matrix B from the external memory.

[0201] The DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 1606, or to transfer the weight data to the weight memory 1602, or to transfer the input data to the input memory 1601.

[0202] The vector calculation unit 1607 includes a plurality of operation processing units, which further process the output of the operation circuit as needed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / full connection layer network calculation in neural networks, such as batch normalization, pixel-level summation, upsampling of feature planes, etc.

[0203] In some implementations, the vector calculation unit 1607 can store the processed output vector to the unified memory 1606. For example, the vector calculation unit 1607 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1603, such as linear interpolation of the feature planes extracted by the convolutional layer, and for example, a vector of accumulated values to generate activation values. In some implementations, the vector calculation unit 1607 generates normalized values, pixel-level summation values, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1603, for example, for use in subsequent layers in the neural network.

[0204] The controller 1604 is connected to the instruction fetch buffer 1609, which is used to store instructions used by the controller 1604;

[0205] The unified memory 1606, the input memory 1601, the weight memory 1602, and the instruction memory 1609 are on-chip memories. The external memory is private to the NPU hardware architecture.

[0206] The operations of the layers in the recurrent neural network can be performed by the operation circuit 1603 or the vector calculation unit 1607.

[0207] The processor mentioned in any of the above can be a general central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the program of the method of FIGS. 2-13.

[0208] It should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. In addition, the connection relationship between the modules in the apparatus embodiment provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0209] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits, or special circuits. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., including a plurality of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0210] The terms "first", "second", and the like in the description and in the claims of the present application and above drawings are used for distinguishing between similar objects and not necessarily for describing a specific sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or inverse order, depending upon the circumstances. The term "and / or" in the present application is merely used to represent an association between associated objects, and it is possible that three relationships exist, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present application generally represents an "or" relationship between the associated objects. Furthermore, the terms "include" and "have" and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device that includes a list of steps or modules as an example does not have to be limited to those steps or modules, but can include other steps or modules that are not expressly listed or inherent to such process, method, product or device. The naming or numbering of steps in the present application does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering, and the named or numbered flow steps can be executed in a different order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in the present application is a logical division, and in actual application, there can be another division manner, for example, multiple modules can be combined or integrated in another system, or some features can be ignored or not executed, in addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be through some ports, and the indirect coupling or communication connection between the modules can be electrical or other similar forms, which are not limited in the present application. Furthermore, the modules or sub-modules described as separate components can or can not be physically separated, and can or can not be physical modules, or can be distributed to multiple circuit modules, and part or all of the modules can be selected according to actual needs to achieve the purpose of the present application.

Claims

1. An image processing method, characterized by, The method comprises: obtaining an input image; inputting the input image into an image processing model to obtain an output image; wherein the image processing model comprises a plurality of modules, each module is configured to convert an input first image feature into a frequency spectrum, modulate the frequency spectrum using a time step to obtain a modulated frequency spectrum, and convert the modulated frequency spectrum into a second image feature, and the second image feature is used to obtain the output image.

2. The method of claim 1, wherein, Each module is specifically configured to: obtain a frequency spectrum corresponding to the first image feature; obtain a frequency modulation matrix according to a time step, the time step being a time interval for diffusion of the first image feature; modulate the frequency spectrum according to the frequency modulation matrix to obtain the modulated frequency spectrum, and elements in the frequency modulation matrix are used to weight frequency components in the frequency spectrum; convert the modulated frequency spectrum into the second image feature.

3. The method of claim 2, wherein, The method further comprises: dividing the first image feature to obtain a plurality of blocks; and converting the plurality of blocks into the frequency spectrum.

4. The method according to claim 2 or 3, characterized in that, The method further comprises: mapping the time step to a preset space to obtain a mapping vector; reshaping the mapping vector to obtain the frequency modulation matrix.

5. The method of any one of claims 2-4, wherein: obtaining a frequency spectrum corresponding to the first image feature comprises: performing Fourier transform on the first image feature to obtain a frequency spectrum corresponding to the first image feature; and converting the modulated frequency spectrum into a second image feature comprises: performing inverse Fourier transform on the modulated frequency spectrum to obtain the second image feature.

6. The method according to any one of claims 1-5, characterized in that, The plurality of modules are connected in a U shape, the number of processing channels in each module is the same, and the plurality of modules process features of multiple scales.

7. The method according to any one of claims 1 to 6, characterized in that, The image processing model is used for super-resolution, deblurring, or defogging of the input image.

8. An image processing apparatus characterized by comprising: The method comprises: an input module configured to obtain an input image; a processing module configured to input the input image into an image processing model to obtain an output image; wherein the image processing model comprises a plurality of modules, each module is configured to convert an input first image feature into a frequency spectrum, modulate the frequency spectrum using a time step to obtain a modulated frequency spectrum, and convert the modulated frequency spectrum into a second image feature, and the second image feature is used to obtain the output image.

9. The apparatus of claim 8, wherein, Each module is specifically configured to: obtain a frequency spectrum corresponding to the first image feature; obtain a frequency modulation matrix according to a time step, the time step being a time interval for diffusion of the first image feature; modulate the frequency spectrum according to the frequency modulation matrix to obtain the modulated frequency spectrum, and elements in the frequency modulation matrix are used to weight frequency components in the frequency spectrum; convert the modulated frequency spectrum into the second image feature.

10. The apparatus of claim 9, wherein, Each module is specifically configured to: divide the first image feature to obtain a plurality of blocks; convert the plurality of patches into the spectrum graph.

11. The apparatus of claim 9 or 10, wherein, The each module is specifically configured to: map the time steps into a preset space to obtain a mapping vector; reshape the mapping vector to obtain the frequency modulation matrix.

12. The apparatus of any one of claims 9-11, wherein, The each module is specifically configured to: perform Fourier transform on the first image feature to obtain a spectrum graph corresponding to the first image feature; perform inverse Fourier transform on the modulated spectrum graph to obtain the second image feature.

13. The apparatus of any one of claims 8-12, wherein, The plurality of modules are connected in a U shape, the number of processing channels in each module is the same, and the plurality of modules process features of multiple scales.

14. The apparatus of any one of claims 8-13, wherein, The image processing model is used for super-resolution, deblurring or defogging of the input image.

15. An image processing apparatus characterized by comprising: The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, and when the code is executed, the device executes the method in any one of claims 1-7.

16. A computer storage medium, comprising, The computer storage medium stores instructions, and the instructions cause the computer to implement the method in any one of claims 1 to 7 when executed by the computer.

17. A computer program product, characterised in that, The computer program product stores instructions, and the instructions cause the computer to implement the method in any one of claims 1 to 7 when executed by the computer.

Images