Image processing method, apparatus and system

Abstract

Provided in the present application is an image processing method, comprising: acquiring an input image; performing feature extraction on the input image to obtain a first feature; inputting the first feature into a first super-resolution network to obtain a second feature; and, on the basis of the second feature, obtaining an output image by means of reconstruction, the resolution of the output image being higher than the resolution of the input image, wherein the first super-resolution network comprises N2 feature extraction module groups, each feature extraction module group comprises N1 feature extraction modules, and each feature extraction module comprises N3 cascaded multi-scale sub-modules, the N3 cascaded multi-scale sub-modules being used for successively performing feature extraction on an input feature in different scanning directions, and N1, N2 and N3 all being positive integers. Compared with parallel scanning in multiple directions, extracting features by means of a plurality of cascaded multi-scale sub-modules having different scanning directions improves the peak signal-to-noise ratio and super-resolution accuracy without increasing the amount of calculation and the number of parameters.

Description

An image processing method, device and system

[0001] The present application claims priority to the Chinese patent application No. 202411149524.7, filed on August 20, 2024, and entitled "An image processing method, device and system", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of image technology, and in particular to an image processing method, device and system. BACKGROUND

[0003] Super-resolution (SR) technology refers to processing a low resolution (LR) image to obtain a high resolution (HR) image. With the development of computer science, in order to improve the effect of SR technology, an image processing method based on artificial intelligence (AI) has become a development trend of SR technology.

[0004] However, there is a contradiction between the amount of calculation and the accuracy in the current SR technology. A simple network structure pursues the efficiency of image processing, and discards the accuracy of the reconstructed image. Although a complex network structure can greatly improve the performance of SR, it usually brings an exponential increase in calculation and power consumption.

[0005] Therefore, how to balance the demand for accuracy and calculation in SR technology is a problem to be solved. SUMMARY

[0006] The present application provides an image processing method, device and system, which can balance the demand for accuracy and calculation in SR technology, and reduce the overhead while ensuring the quality of image reconstruction.

[0007] In a first aspect, an image processing method is provided, comprising: obtaining an input image; performing feature extraction on the input image to obtain a first feature; inputting the first feature into a first super-resolution network to obtain a second feature; and reconstructing an output image based on the second feature, the resolution of the output image being higher than that of the input image; wherein the first super-resolution network comprises N2 groups of feature extraction modules, each group of feature extraction modules comprising N1 feature extraction modules, and each feature extraction module comprising N3 cascaded multi-scale sub-modules, the N3 cascaded multi-scale sub-modules being used for sequentially performing feature extraction on the input feature according to different scanning directions, and N1, N2 and N3 are all positive integers.

[0008] Compared with the scanning manner in which multiple scanning directions are connected in parallel, the feature extraction is performed through multiple cascaded multi-scale sub-modules with different scanning directions, the complexity of the system is reduced, the peak signal-to-noise ratio is effectively improved without additional calculation and parameter quantity, and the accuracy of the high-resolution image reconstructed based on the low-resolution image is improved.

[0009] With reference to the first aspect, in some implementations of the first aspect, the feature extraction module includes four cascaded multi-scale sub-modules, which are configured to sequentially perform feature extraction on the input feature according to four different scanning directions; and the four different scanning directions include horizontal direction scanning, vertical direction scanning, reverse horizontal direction scanning, and reverse vertical direction scanning.

[0010] Based on the above scheme, the multi-scale sub-modules with different scanning directions are used for feature extraction, the spatial relationship of the model is enriched, the correlation between pixels can be obtained from more dimensions, and the foundation for improving the accuracy of the reconstructed image of the first super-resolution network is laid.

[0011] With reference to the first aspect, in some implementations of the first aspect, the multi-scale sub-module is configured to: perform first processing on the input feature based on a state space model (SSM) to obtain a third feature; perform second processing on the input feature based on the SSM after down-sampling to obtain a fourth feature; perform feature fusion on the third feature and the fourth feature to obtain a fifth feature; perform third processing on the input feature based on a deep separable convolution attention (DSCA) to obtain a sixth feature; and determine the output feature of the multi-scale sub-module according to the input feature, the fifth feature, and the sixth feature.

[0012] Based on the above scheme, different size pixel-level features of the input image are obtained through the first processing and the second processing, and channel-level features of the input image are obtained through the third processing. The three different processing manners can obtain feature extraction results of different dimensions, different scale representations are aggregated through the adaptive fusion module, the spatial perception ability of the first super-resolution network is enhanced, more comprehensive context information can be obtained, and the accuracy of the high-resolution image reconstruction is improved.

[0013] With reference to the first aspect, in some implementations of the first aspect, the multi-scale sub-module is configured to: perform first processing on the input feature based on a state space model (SSM) to obtain a third feature; perform second processing on the input feature based on the SSM after down-sampling to obtain a fourth feature; perform feature fusion on the third feature and the fourth feature to obtain a fifth feature; and determine the output feature of the multi-scale sub-module according to the input feature and the fifth feature.

[0014] Based on the above scheme, different precision pixel-level features of the input image can be obtained through the first processing and the second processing, different scale representations are aggregated, the spatial perception ability of the first super-resolution network is enhanced, more abundant context information can be obtained, and the precision of high-resolution image reconstruction is improved.

[0015] With reference to the first aspect, in some implementations of the first aspect, the multi-scale sub-module is configured to: perform first processing on the input feature based on a state space model (SSM) to obtain a third feature; and determine the output feature of the multi-scale sub-module according to the input feature and the third feature.

[0016] With reference to the first aspect, in some implementations of the first aspect, the determining the output feature of the multi-scale sub-module includes: determining a seventh feature according to the input feature, the fifth feature and the sixth feature, and performing fourth processing on the seventh feature based on a feed-forward network (FFN) to obtain the output feature of the multi-scale sub-module; or determining a seventh feature according to the input feature and the fifth feature, and performing fourth processing on the seventh feature based on a feed-forward network (FFN) to obtain the output feature of the multi-scale sub-module; or determining a seventh feature according to the input feature and the third feature, and performing fourth processing on the seventh feature based on a feed-forward network (FFN) to obtain the output feature of the multi-scale sub-module.

[0017] Based on the above scheme, the fourth processing on the seventh feature based on the FFN can further capture more complex features and representations through nonlinear transformation, which helps to improve the precision of high-resolution image reconstruction.

[0018] With reference to the first aspect, in some implementations of the first aspect, each feature extraction module group in the N2 feature extraction module groups is configured to: perform convolution on the output feature of the N1th feature extraction module in each feature extraction module group to obtain an eighth feature; and perform residual processing on the input feature of the first feature extraction module in each feature extraction module group and the eighth feature to obtain the output feature of each feature extraction module group.

[0019] Based on the above scheme, the output feature is processed through convolution, which can realize the refinement of the feature, and the residual structure allows the network to reuse the features learned by the previous layers in the deep layer, and directly transmits the shallow layer features to the deep layer through the connection of skipping certain layers, so that the deep network can more easily learn effective feature representation, and the problem of gradient disappearance or gradient explosion is alleviated.

[0020] With reference to the first aspect, in some implementations of the first aspect, the first super-resolution network is configured to: perform convolution on the output features of the N2th group of feature extraction modules in the first super-resolution network to obtain ninth features; and perform residual processing on the first features and the ninth features to obtain the second features.

[0021] Based on the above scheme, the convolution processing can realize the refinement of the features, and the residual structure can enable the deep network to learn effective feature representation more easily, and alleviate the gradient vanishing or gradient explosion problem.

[0022] With reference to the first aspect, in some implementations of the first aspect, before the input image is obtained, the method further includes: dividing a to-be-processed image into blocks to obtain the input image, the input image being any one of a plurality of input image blocks obtained by dividing the to-be-processed image; and after the output image is reconstructed, the method further includes: splicing a plurality of output images corresponding to the plurality of input image blocks into a target image.

[0023] Based on the above scheme, compared with processing the entire to-be-processed image, the block processing can reduce the amount of computation on the one hand, and improve the quality of the output image on the other hand. Since the image has local correlation, the block processing can divide the to-be-processed image into a plurality of input image blocks with high internal correlation, so that the network can obtain relatively accurate results using a smaller number of computation layers. If the block processing is not performed, the number of computation layers required by the network is related to the most complex part of the to-be-processed image, so a larger number of computation layers is required.

[0024] With reference to the first aspect, in some implementations of the first aspect, the first super-resolution network is trained according to image samples, the image samples including a first image and a second image corresponding to the first image, and the resolution of the first image is lower than the resolution of the second image.

[0025] With reference to the first aspect, in some implementations of the first aspect, the first super-resolution network is trained according to the following manner: performing the first feature extraction on the first image to obtain first image input features; inputting the first image features into an initial model of the first super-resolution network to obtain first image output features; reconstructing a third image based on the first image output features; and adjusting parameters of the initial model of the first super-resolution network according to the third image and the second image to train the first super-resolution network.

[0026] The second aspect provides an image processing apparatus, which includes an obtaining unit and a processing unit.

[0027] The acquisition unit is configured to acquire an input image.

[0028] The processing unit is configured to perform the following operations: performing feature extraction on the input image to obtain first features; inputting the first features into a first super-resolution network to obtain second features; and reconstructing an output image based on the second features, the output image having a higher resolution than the input image; wherein the first super-resolution network comprises N2 groups of feature extraction modules, each group of feature extraction modules comprises N1 feature extraction modules, and each feature extraction module comprises N3 cascaded multi-scale sub-modules, the N3 cascaded multi-scale sub-modules being configured to sequentially perform feature extraction on input features according to different scanning directions, and N1, N2 and N3 are positive integers.

[0029] With reference to the second aspect, in some implementations of the second aspect, the multi-scale sub-module is configured to: perform first processing on the input features based on a state space model (SSM) to obtain third features; perform down-sampling and second processing on the input features based on the SSM to obtain fourth features; perform feature fusion on the third features and the fourth features to obtain fifth features; perform third processing on the input features based on a depth separable convolution attention (DSCA) to obtain sixth features; and determine an output feature of the multi-scale sub-module according to the input features, the fifth features and the sixth features.

[0030] With reference to the second aspect, in some implementations of the second aspect, the multi-scale sub-module is configured to: perform first processing on the input features based on a state space model (SSM) to obtain third features; perform down-sampling and second processing on the input features based on the SSM to obtain fourth features; perform feature fusion on the third features and the fourth features to obtain fifth features; and determine an output feature of the multi-scale sub-module according to the input features and the fifth features.

[0031] With reference to the second aspect, in some implementations of the second aspect, the multi-scale sub-module is configured to: perform first processing on the input features based on a state space model (SSM) to obtain third features; and determine an output feature of the multi-scale sub-module according to the input features and the third features.

[0032] In some implementations of the second aspect, in combination with the second aspect, the determining the output feature of the multi-scale sub-module comprises: determining a seventh feature according to the input feature, the fifth feature and the sixth feature, performing fourth processing based on a feed-forward network (FFN) on the seventh feature to obtain the output feature of the multi-scale sub-module; or determining a seventh feature according to the input feature and the fifth feature, performing fourth processing based on a feed-forward network (FFN) on the seventh feature to obtain the output feature of the multi-scale sub-module; or determining a seventh feature according to the input feature and the third feature, performing fourth processing based on a feed-forward network (FFN) on the seventh feature to obtain the output feature of the multi-scale sub-module.

[0033] In some implementations of the second aspect, in combination with the second aspect, each feature extraction module group in the N2 feature extraction module groups is configured to: perform convolution on the output feature of the N1th feature extraction module in each feature extraction module group to obtain an eighth feature; and perform residual processing on the input feature of the first feature extraction module in each feature extraction module group and the eighth feature to obtain the output feature of each feature extraction module group.

[0034] In some implementations of the second aspect, in combination with the second aspect, the first super-resolution network is configured to: perform convolution on the output feature of the N2th feature extraction module group in the first super-resolution network to obtain a ninth feature; and perform residual processing on the first feature and the ninth feature to obtain the second feature.

[0035] In some implementations of the second aspect, in combination with the second aspect, the processing unit is further configured to divide a to-be-processed image into blocks to obtain the input image, the input image being any one of a plurality of input image blocks obtained by dividing the to-be-processed image into blocks; and after the output image is reconstructed, the method further comprises: splicing a plurality of output images corresponding to the plurality of input image blocks into a target image.

[0036] It should be understood that the beneficial effects of the second aspect and the various implementations of the second aspect can refer to the beneficial effects of the first aspect and the various implementations of the first aspect, which will not be repeated here.

[0037] In a third aspect, a device for road prediction is provided, which comprises: an acquisition unit, a training unit.

[0038] The acquisition unit is configured to acquire an image sample pair, the image sample pair comprising a first image and a second image corresponding to the first image, the resolution of the first image being lower than the resolution of the second image.

[0039] The training unit is configured to train the initial model of the first super-resolution network to obtain the first super-resolution network according to the image sample pair.

[0040] With reference to the third aspect, in some implementations of the third aspect, the training module is further configured to perform the first feature extraction on the first image to obtain first image input features, input the first image features into the initial model of the first super-resolution network to obtain first image output features, and reconstruct a third image based on the first image output features, and adjust parameters of the initial model of the first super-resolution network based on the third image and the second image to train the first super-resolution network.

[0041] It should be understood that the beneficial effects of the third aspect and the various implementations of the third aspect are the same as those of the first aspect and the various implementations of the first aspect, which will not be repeated here.

[0042] In a fourth aspect, a computing device is provided, which includes a processor and a memory, and optionally, an input / output interface. The processor is configured to control the input / output interface to receive and send information, and the memory is configured to store a computer program. The processor is configured to call and run the computer program from the memory, so as to execute the method in the first aspect or any possible implementation of the first aspect.

[0043] Optionally, the processor can be a general-purpose processor, which can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, or the like. When implemented by software, the processor can be a general-purpose processor, which reads software codes stored in the memory to implement the processor. The memory can be integrated in the processor or exist independently outside the processor.

[0044] In a fifth aspect, a computing device cluster is provided, which includes at least one computing device, and each computing device includes a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so as to make the computing device cluster execute the method in the first aspect or any possible implementation of the first aspect.

[0045] In a sixth aspect, a chip is provided, which acquires instructions and executes the instructions to implement the method in the first aspect or any possible implementation of the first aspect.

[0046] Optionally, as an implementation, the chip includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface, and executes the method in the first aspect or any possible implementation of the first aspect.

[0047] Optionally, as an implementation form, the chip further comprises a memory, and the memory stores instructions, and the processor is configured to execute the instructions stored in the memory, and the processor is configured to execute the method in the first aspect and any implementation form of the first aspect when the instructions are executed.

[0048] In a seventh aspect, a computer program product is provided, which comprises instructions, and when the instructions are executed by a computing device, the computing device is caused to execute the method in the first aspect and any implementation form of the first aspect.

[0049] In an eighth aspect, a computer program product is provided, which comprises instructions, and when the instructions are executed by a computing device cluster, the computing device cluster is caused to execute the method in the first aspect and any implementation form of the first aspect.

[0050] In a ninth aspect, a computer-readable storage medium is provided, which comprises computer program instructions, and when the computer program instructions are executed by a computing device, the computing device executes the method in the first aspect and any implementation form of the first aspect.

[0051] As examples, the computer-readable storage includes, but is not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and hard drive.

[0052] Optionally, as an implementation form, the storage medium can be a non-volatile storage medium.

[0053] In a tenth aspect, a computer-readable storage medium is provided, which comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method in the first aspect and any implementation form of the first aspect.

[0054] As examples, the computer-readable storage includes, but is not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and hard drive.

[0055] Optionally, as an implementation manner, the storage medium can be a nonvolatile storage medium. BRIEF DESCRIPTION OF DRAWINGS

[0056] FIG. 1 is a schematic diagram of a training phase and an inference phase of a model.

[0057] FIG. 2 is a schematic diagram of an architecture 200 of an image processing system suitable for embodiments of the present application.

[0058] FIG. 3 is a functional schematic diagram of an image processing apparatus suitable for embodiments of the present application.

[0059] FIG. 4 is a schematic diagram of a method 400 of image processing suitable for embodiments of the present application.

[0060] FIG. 5 is a schematic block diagram of a method 500 of image processing suitable for embodiments of the present application.

[0061] FIG. 6 is a structural schematic block diagram of a DA-HMG suitable for embodiments of the present application.

[0062] FIG. 7 is a schematic diagram of an image unfolding manner suitable for embodiments of the present application.

[0063] FIG. 8 is a structural schematic block diagram of an HMB suitable for embodiments of the present application.

[0064] FIG. 9 is a structural schematic block diagram of an L-SSM suitable for embodiments of the present application.

[0065] FIG. 10 is a structural schematic block diagram of an R-SSM suitable for embodiments of the present application.

[0066] FIG. 11 is a structural schematic block diagram of a fusion module suitable for embodiments of the present application.

[0067] FIG. 12 is a schematic diagram of an effect of a method of image processing suitable for embodiments of the present application.

[0068] FIG. 13 is a schematic diagram of an effect of a method of image processing suitable for embodiments of the present application.

[0069] FIG. 14 is a schematic diagram of an effect of a method of image processing suitable for embodiments of the present application.

[0070] FIG. 15 is a schematic diagram of a model training method 1500 of a first super-resolution network provided by embodiments of the present application.

[0071] FIG. 16 is a schematic block diagram of an image processing apparatus 1600 provided by embodiments of the present application.

[0072] FIG. 17 is a schematic block diagram of another image processing apparatus 1700 provided by embodiments of the present application.

[0073] FIG. 18 is a schematic diagram of an architecture of a computing device 1800 according to an embodiment of the present application.

[0074] FIG. 19 is a schematic diagram of an architecture of a computing device cluster according to an embodiment of the present application.

[0075] FIG. 20 is a schematic diagram of a connection between computing devices 1800A and 1800B via a network according to an embodiment of the present application. DETAILED DESCRIPTION

[0076] The technical solutions in the present application will be described below with reference to the accompanying drawings.

[0077] Artificial intelligence (AI) is the use of digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, so that the machine has the functions of perception, reasoning and decision-making. The research in the field of artificial intelligence includes robots, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, AI basic theory, etc.

[0078] The basic principle of AI is to combine massive data with super strong operation processing capability and intelligent algorithm to establish an AI model for solving specific problems, so that the AI model can automatically induce and learn potential patterns or features from data, thereby realizing a thinking mode close to human.

[0079] AI model, i.e. AI algorithm (or AI operator), is a general term for mathematical algorithms constructed based on the principle of artificial intelligence, and is also the basis for solving specific problems using AI. According to different specific methods and / or technologies for realizing artificial intelligence, the AI model can also be called a machine learning model, a deep learning model or a reinforcement learning model.

[0080] In order to facilitate the understanding of the embodiments of the present application, first, the terms involved in the present application are simply explained.

[0081] It should be understood that the relevant concept explanation may be limited due to the specific circumstances of the embodiments of the present application, but it does not mean that the present application can only be limited to this specific situation, and there may be differences in the specific circumstances of different embodiments, which are not limited here.

[0082] 1. Peak signal-to-noise ratio (PSNR)

[0083] In the field of image processing, PSNR is a method for evaluating the quality of image reconstruction, which calculates the difference between each pixel of the original image and the processed image (or noise image) based on the mean square error. The minimum value of PSNR is 0. Generally speaking, the higher the PSNR of the reconstructed image, the smaller the gap between the reconstructed image and the true value image, i.e. the better the image quality; on the contrary, the lower the PSNR of the reconstructed image, the greater the image distortion, i.e. the worse the image quality. The specific calculation formula is as follows:

[0084] wherein MSE represents the mean square error of the original image and the processed image; MaxValue is the maximum value of the pixel color in the image (for example, for an 8-bit image, MaxValue = 2 8 -1 = 255), and the unit of PSNR is decibel (dB).

[0085] 2. Strutral similarity (SSIM)

[0086] SSIM is an index for measuring the similarity between two images. The higher the SSIM of the reconstructed image, the more similar the structure of the reconstructed image and the true value image.

[0087] 3. floating point operations (FLOPs)

[0088] FLOPs is an index for evaluating the computational complexity of a model, which represents the number of floating point operations required for a single forward propagation (or backward propagation) of the model. The calculation method of FLOPs varies depending on the type of network layer (for example, in a fully connected layer, the number of FLOPs usually depends on the weight matrix and the number of input nodes; in a convolutional layer, the number of FLOPs depends on the size of the convolution kernel, the number of input and output channels, the size of the input feature map, and the size of the output feature map), and a higher FLOPs means that the model requires more computational resources and time for a single inference or training process.

[0089] 4. Neural network

[0090] A neural network is a model, which can be composed of neural units, which can be an operation unit with X s and intercept 1 as input, and the output of the operation unit can be:

[0091] wherein s = 1, 2, … n, n is a natural number greater than 1, W s is xs The weight of the neural unit, b is the bias of the neural unit. f is an activation function of the neural unit, used to introduce a nonlinear characteristic into the neural network, converting the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a Relu function. The neural network is a network formed by connecting a plurality of the above-described single neural units, i.e., the output of one neural unit can be the input of another neural unit, and the input of each neural unit can be connected to the local receptive field of the previous layer, thereby extracting the features of the local receptive field, wherein the local receptive field can be a region composed of a plurality of neural units.

[0092] Based on the above, the neural network can be specifically understood as a network composed of neurons and having an input layer, a hidden layer and an output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are the hidden layers. The input of the neural network module can be a single frame image or an image sequence. The neural network with many layers of hidden layers is called a deep neural network (DNN). The work of each layer in the neural network can be described by the mathematical expression y = a(W x + b). From a physical perspective, the work of each layer in the neural network can be understood as completing the transformation of the input space (the set of input vectors) to the output space (i.e., the row space of the matrix to the column space) through five operations on the input space, which include: 1, dimensionality increase / decrease; 2, magnification / reduction; 3, rotation; 4, translation; 5, "bending". Among them, the operations of 1, 2 and 3 are completed by W, the operation of 4 is completed by "+b", and the operation of 5 is realized by "a()". The reason why "space" is used here is that the object to be classified is not a single thing, but a class of things, and the space refers to the set of all individuals of this class of things. W is the weight matrix of each layer of the neural network, and each value in the matrix represents the weight value of a neuron in the layer. The matrix W determines the spatial transformation of the input space to the output space described above, i.e., W of each layer of the neural network controls how to transform the space. The purpose of training the neural network is to obtain the weight matrix of all layers of the trained neural network. Therefore, the training process of the neural network is essentially learning the way to control the spatial transformation, more specifically, learning the weight matrix.

[0093] The neural network can include a feed-forward network (FFN), a convolutional neural network (CNN) or a Transformer model, which will be introduced below.

[0094] 5、Feed-forward network

[0095] Feedforward Neural Networks (FNNs) were among the earliest invented simple artificial neural networks. In a feedforward network, each neuron belongs to a different layer. Neurons in each layer receive signals from neurons in the previous layer and output signals to the next layer. There is no feedback throughout the network; signals propagate unidirectionally from the input layer to the output layer.

[0096] 6. Convolutional Neural Networks

[0097] A CNN (Convolutional Neural Network) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as performing convolution between the same trainable filter and an input image or a convolutional feature map. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature maps, each composed of rectangularly arranged neural units. Neural units within the same feature map share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of an image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, the same learned image information can be used for all locations in the image. Within the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more kernels there are, the richer the image information reflected by the convolution operation. Convolutional kernels can be initialized as matrices of random size, and during CNN training, they can learn appropriate weights. Furthermore, sharing weights directly reduces the connections between layers in the convolutional neural network, while also lowering the risk of overfitting.

[0098] 7. Transformer

[0099] Transformer is a feature extraction network based on an attention mechanism that can contain an encoder and / or a decoder.

[0100] An encoder is used to encode an input sequence into a series of consecutive hidden states, which contain semantic information about the input sequence. An encoder consists of multiple identical layers, each including an attention mechanism and a Front-Factor Node (FFN). The attention mechanism computes a representation for each position in the input sequence, while the FFN further processes these representations.

[0101] Decoder: for generating the target sequence according to the output of the encoder and the generated partial sequence. The encoder is also composed of multiple identical layers, in addition to the attention mechanism and the FFN, the decoder also includes an encoder-decoder attention mechanism for associating the output of the encoder with the generated partial sequence.

[0102] The attention mechanism is described as follows: the attention mechanism (also referred to as attention) occurs between the encoder and the decoder (also understood to occur between the input sequence and the generated sequence), which can quickly extract important features of sparse data, so that the model dynamically calculates the degree of association between each position and other positions in the input sequence when processing sequence data.

[0103] In simple terms, attention uses the degree of similarity (such as inner product) between features as a weight to integrate the queried features as the updated value of the current features. Assuming that the input is Q (query) and the context is stored in the form of key-value pairs (K (key), V (value)). Then the attention mechanism is actually a mapping function from query to a series of (key, value). The Attention function that implements its mapping can then describe a mapping relationship from a query to a series of key-value pairs. The Attention function essentially assigns a weight coefficient (which can also be understood as soft addressing) to each element in the sequence. If each element in the sequence is stored in the form of (K, V), then the Attention function completes the addressing by calculating the similarity between Q and K. The similarity calculated by Q and K reflects the importance of the extracted V value, that is, the weight, and then the weighted sum is obtained. The final feature value.

[0104] The calculation of attention mainly includes three steps, the first step is to calculate the similarity between query and each key to obtain the weight, the commonly used similarity function is dot product, concatenation, perception, etc.; the second step is to normalize these weights by using the softmax function (on the one hand, it can be normalized to obtain a probability distribution whose sum of all weight coefficients is 1. On the other hand, the characteristic of the softmax function can highlight the weight of important elements); finally, the weight and the corresponding value are weighted and summed to obtain the final feature value. The specific calculation formula is as follows:

[0105] Where d is the dimension of the QK matrix.

[0106] Attention includes self-attention and cross-attention. Self-attention can be understood as a special attention, that is, the input of QKV is consistent. The self-attention mechanism in the self-attention model occurs within the input sequence or within the output sequence, and can extract the relationship between words far apart in the same sentence, such as syntactic features (phrase structure). Self-attention is an attention extracted based on the attention of the feature map itself, and provides an effective modeling method for capturing global context information through QKV. The input of QKV in cross-attention is inconsistent.

[0107] It should be understood that the above is only described based on the most basic structure of the Transformer, and the encoder and the decoder thereof can also have other implementation manners, or can also include other structures in addition to the encoder and the decoder. For example, in the super-resolution task, a structure based on Vision Transformer (ViT) is usually used. Specifically, the ViT structure only includes an encoder, and after the input is encoded into tokens according to pixels, it is sent to a Transformer block composed of a multi-head self-attention (MSA) and an FFN, and the required tokens are adaptively aggregated through attention. For specific content of the Transformer and its derivative models, reference can be made to the related description in the prior art, and the present application does not limit this.

[0108] 8、State Space Model (SSM)

[0109] SSM is a mathematical model used to describe and analyze the behavior of linear time-invariant systems, which is widely used in various fields. In the field of deep learning, SSM is used to process sequence data, such as time series analysis, natural language processing (NLP) and video understanding, etc. The core idea of SSM is to represent the relationship between the current state of the system and the input and output through state equations and observation equations:

[0110] (1) State Equation: Describes the evolution of the system state over time. The state equation usually contains the current state and input, as well as possible system parameters, and the state equation can be expressed as: h'(t) = Ah(t) + Bx(t)

[0111] Where h(t) is the system state at time step t, h'(t) is the derivative of the state vector with respect to time t, x(t) is the input at time step t, A is the state transition matrix, and B is the input matrix.

[0112] (2) Observation Equation: describes the relationship between system output and state, which can be expressed as: y(t) = Ch(t) + Dx(t)

[0113] where y(t) is the system output at time step t, C is the observation matrix, and D is the feedforward matrix.

[0114] It should be understood that the above is only a description based on the most basic structure of SSM in the time series modeling of continuous space, and in the time series discretization and parallel processing, the state equation and observation equation can have other representations. For details, please refer to the relevant description of the prior art, which will not be repeated here.

[0115] 9、Mamba model

[0116] Mamba is a new selective structured state space model based on SSM architecture, which can achieve linear time inference speed and perform well in long sequence modeling tasks. Mamba alleviates the modeling constraints of convolutional neural networks through global receptive fields and dynamic weighting, and provides advanced modeling capabilities similar to Transformers. Mamba improves the structured state space sequence model (S4) architecture, also known as S6 or selective SSM. Mamba makes two important modifications to S4:

[0117] (1) Selective scan algorithm: Unlike other SSMs that require time and input invariance, Mamba introduces a selection mechanism that controls the way information propagates along the sequence dimension, allowing the model to distinguish and retain key information and filter out irrelevant information.

[0118] (2) Hardware-aware algorithm: allows efficient storage of intermediate results through parallel scan, kernel fusion, and recomputation. Parallel scan is a technique that treats recursive computation in SSM as a scan for parallel processing, and the computation between the existing pairs in the same hierarchy can be parallelized to improve efficiency. Kernel fusion allows the model to prevent writing intermediate results and continuously perform calculations, limiting the number of times switching from DRAM to SRAM. Recomputation refers to the intermediate states necessary for backpropagation, which are not saved but recalculated during the backward pass, thereby reducing memory requirements and significantly reducing overhead compared to reading all intermediate states from DRAM.

[0119] 10. Model training and inference

[0120] In embodiments of the present application, the models used for image processing are essentially neural networks, or part of the structure of the model is a neural network. The application of the model generally includes two stages of training and inference, the training of the model refers to the calculation of the training data using the specified initial model, and the parameters in the initial model are adjusted according to the calculation results using a certain method, so that the model gradually learns certain rules and has a specific function. The model with stable function after training can be used for inference. The inference of the model is a process of using the trained AI model to calculate the input data and obtain the predicted inference result.

[0121] Figure 1 is a schematic diagram of the training and inference stages of the model.

[0122] As shown in Figure 1, in the training stage, first, a training set for a deep learning model needs to be constructed based on the target, the training set includes multiple training data, each training data is provided with a label, the label of the training data is the correct answer of the training data on a specific problem, and the label can represent the target of training the deep learning model using the training data. For example, for training a deep learning model that can be used to implement super-resolution, the training set can include multiple original images (i.e. training data) and the true value image corresponding to each original image (as a label), and the resolution of the true value image is greater than that of the original image.

[0123] When training the deep learning model, the training data can be input to the deep learning model after parameter initialization in batches, and the deep learning model calculates (i.e. infers) the training data to obtain the inference result for the training data. The inference result obtained by inference and the label corresponding to the training data are used as data for calculating the loss according to the loss function. The loss function is a function used to calculate the gap (i.e. loss value loss) between the inference result of the model for the training data and the label of the training data in the model training stage. The loss function can be implemented using different mathematical functions, and the expressions of commonly used loss functions are: mean square error loss function, logarithmic loss function, least squares method, etc.

[0124] The loss value calculated based on the loss function can be used to update the parameters of the deep learning model, and the gradient descent method is commonly used for parameter updating. The training of the model is a repeated iterative process, and different training data is inferred and the loss value is calculated each time. The goal of multiple iterations is to continuously update the parameters of the deep learning model to find the parameter configuration that makes the loss value of the loss function the lowest or tends to be stable.

[0125] It should be understood that the loss function is a function that maps a random event or its related random variable to a non-negative real number to represent the "risk" or "loss" of the random event. In application, the loss function is usually associated with an optimization problem as a learning criterion, that is, the model is solved and evaluated by minimizing the loss function. For example, in machine learning, the loss function is used for parameteric estimation of the model, and the loss value obtained based on the loss function can be used to describe the difference between the predicted value and the actual value of the model. Common loss functions include mean square error loss function, support vector machine (SVM) hinge loss function, cross-entropy loss function, etc.

[0126] In the training phase, in order to make the training efficiency of the model and the performance of the model after training more optimal, some reasonable hyperparameters need to be set for training. The hyperparameters of the deep learning model refer to a class of parameters that cannot be obtained by learning training data or cannot be changed due to training data driving in the training process, which is a concept relative to the parameters in the model. The hyperparameters of the deep learning model are usually set by artificial experience or experiment, and the hyperparameters include: learning rate, batch size, network structure hyperparameters (such as: network layer number (also called depth), interaction mode between network layers, convolution kernel number and convolution kernel size, activation function) and the like. Among them, the learning rate as a hyperparameter is used to control the amplitude of the parameter weight update of the model in the training process, which greatly affects the speed and accuracy of the training.

[0127] As shown in FIG. 1, the trained deep learning model can be used for inference on input data. In the inference phase, the data of the actual application scenario is usually used as the input data, and the inference of the trained deep learning model can obtain an inference result. The inference phase is the actual application of the trained deep learning model, which can quickly use the AI capability to solve specific technical problems.

[0128] The above only takes the training of the most typical deep learning model as an example for introduction, and the training of other types of models has slight differences, but the principle is similar, which is to infer the training data, adjust the parameters in the model according to the inference result, and obtain the parameter combination that stabilizes the performance of the model as the goal.

[0129] 11. Feature fusion

[0130] The different features extracted by the neural network are generated into new features by some method, so that the new features are more effective for classification, identification or detection, and feature fusion generally has two ways: concat and add. Among them, concat is a series of feature fusion, that is, two features are directly connected, and the dimensions of two input features x and y are p and q, and the dimension of the output feature z is p+q; add is a parallel fusion strategy, which combines two feature vectors, and for input features x and y, a new feature z with unchanged channel number is obtained. Taking image data as an example, add increases the amount of information under the feature describing the image, but the dimension describing the image itself does not increase, only the amount of information under each dimension increases; and concat is the merging of the number of channels, that is, the features describing the image increase, but the amount of information under each feature dimension does not increase.

[0131] It should be understood that feature fusion can also be realized by other ways, and for the specific content, reference can be made to the related description in the prior art, which is not limited in the present application.

[0132] Fig. 2 is a schematic diagram of an architecture 200 of an image processing system suitable for embodiments of the present application.

[0133] As shown in Fig. 2, the data acquisition device 260 is configured to acquire training data and store the training data into the database 230, and the training device 220 trains the target model / rule 201 based on the training data maintained in the database 230.

[0134] It should be noted that the data acquisition device 260, the execution device 210 and the training device 220 can be the same or different devices. The training data can include a plurality of training image groups, each training image group including a first resolution image and a second resolution image of the same scene, the first resolution image having a higher resolution than the second resolution image. Optionally, each training image group is generated according to computer graphics technology.

[0135] How the training device 220 obtains the target model / rule 201 based on the training data will be described in more detail below, and the target model / rule 201 can be used to implement the method of image processing provided by the embodiments of the present application.

[0136] It should be noted that in actual application, the training data maintained in the database 230 can come from the data acquisition device 260 or from other devices. The training device 220 also does not necessarily train the target model / rule 201 based on the training data maintained in the database 230, but can also obtain training data from the cloud or other places for model training, and the above description should not be regarded as a limitation on the embodiments of the present application. The target model / rule 201 trained by the training device 220 can be applied to different systems or devices, for example, to the execution device 210 shown in FIG. 2.

[0137] It should be understood that the execution device 210 can be a terminal. Illustratively, the terminal can be, for example, a digital camera, a monitoring camera, a mobile phone, a personal computer (PC), a notebook computer, a server, a tablet computer, a smart television, a vehicle-mounted terminal, a mobile internet device (MID), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc., and can also be an edge device (for example, a box carrying a chip with processing capability) and the like; the execution device 210 can also be a server or a cloud, etc.

[0138] The execution device 210 is configured with an I / O interface 212 for data interaction with external devices, and a user can input data to the I / O interface 212 through the client device 240. In addition, the input data can be input by the user, can also be uploaded by the user through a shooting device, or can come from a database, which is not limited in the present application.

[0139] As a possible embodiment, the execution device 210 and the training device 220 are different processors deployed on different physical devices, such as servers or servers in a cluster. For example, the execution device 210 can be a graphic processing unit (GPU), a central processing unit (CPU), other general-purpose processor, a digital signal processing (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The training device 220 can be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the program execution of the scheme of the present application. The training device 220 can configure the trained neural network to the plurality of execution devices 210. Each execution device 210 implements the function of super-resolution reconstruction, etc. by using the trained neural network.

[0140] In another possible embodiment, the execution device 210 and the training device 220 are deployed on the same physical device, or the execution device 210 and the training device 220 are the same physical device. The computing device can configure the trained neural network to itself, and implement the function of super-resolution reconstruction, etc. by using the trained neural network.

[0141] The training device 220 is configured to train the neural network by using the training data until the loss function in the neural network converges, and the loss function value is less than a certain threshold, so that the neural network training is completed, so that the neural network reaches a certain accuracy. Or, all the training data in the database 230 are used for training, and then the neural network training is completed, so that the trained neural network has the function of super-resolution reconstruction, etc. Further, the training device 220 configures the trained neural network to the execution device 210. The execution device 210 is configured to implement the function of processing application data according to the trained neural network.

[0142] The preprocessing module 213 is configured to perform preprocessing on the input data received by the I / O interface 212. In the embodiment of the present application, the preprocessing module 213 can be configured to obtain the features of the image to be predicted. During the preprocessing of the input data performed by the execution device 210, or during the processing performed by the computing module 211 of the execution device 210, the execution device 210 can call the data, code, etc. in the data storage system 250 for the corresponding processing, and can also store the data, instructions, etc. obtained through the corresponding processing in the data storage system 250. Finally, the I / O interface 212 returns the processing result to the client device 240, thereby providing the user.

[0143] It should be understood that FIG. 2 is only a schematic diagram of a system architecture, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG. 2, the data storage system 250 is an external memory relative to the execution device 210, and in other cases, the data storage system 250 can also be placed in the execution device 210.

[0144] It should be understood that the system architecture and business scenarios described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems as the system architecture evolves and new business scenarios appear.

[0145] FIG. 3 is a functional schematic diagram of an image processing apparatus suitable for the embodiments of the present application.

[0146] As shown in FIG. 3, the image processing apparatus 300 can include some or all of the following: a blocking unit 310, a super-resolution processing unit 320, a stitching unit 330, and a size adjustment unit 340. That is, in actual applications, the various units shown in FIG. 3 are not necessarily present, for example, the image processing apparatus 300 can only include the super-resolution processing unit 320, for another example, the image processing apparatus 300 can include the super-resolution processing unit 320, the blocking unit 310, and the stitching unit 330, for another example, the image processing apparatus 300 can include the super-resolution processing unit 320, the blocking unit 310, the stitching unit 330, and the size adjustment unit 340, for another example, the image processing apparatus 300 can include the super-resolution processing unit 320 and the size adjustment unit 340, and so on. In some implementations, some or all of the blocking unit 310, the stitching unit 330, and the size adjustment unit 340 described above can be integrated in the super-resolution processing unit 320. It should also be understood that the image processing apparatus 300 can also be in the form of a chip, an integrated circuit, a software development kit (SDK), etc.

[0147] The block unit 310 is configured to divide the to-be-processed image (low definition, i.e., low resolution) into a plurality of input image blocks (low definition). The input image blocks can be input into the super-resolution processing unit 320 to obtain a plurality of output image blocks (high definition, i.e., high resolution). The splicing unit 330 is configured to splice the plurality of input image blocks to obtain a target image (high definition). The size adjustment unit 340 is configured to adjust the size of the target image to adapt to display requirements, i.e., to obtain a display image that adapts to display requirements.

[0148] The super-resolution processing unit 320 is configured to process the low-definition input image to obtain a high-definition output image. The input image can be the to-be-processed image or a divided block of the to-be-processed image. The input image is a low-definition image, and the output image is a high-definition image obtained by performing super-resolution processing (hereinafter referred to as super-resolution processing) on the input image by using the super-resolution processing unit 320.

[0149] It should be understood that the image in the embodiments of the present application can be a static image (or static picture) or a dynamic image (or dynamic picture), such as an RGB image, a black-and-white image, or a grayscale image, etc. For ease of description, the static image or the dynamic image is uniformly referred to as an image in the following embodiments of the present application.

[0150] In the embodiments of the present application, the super-resolution processing unit 320 is mainly configured to perform super-resolution processing on the input image. Common SR processing methods can include a traditional SR processing method and a neural network-based SR processing method.

[0151] The traditional SR processing method is based on pixel interpolation to improve the resolution of the image, i.e., inserting new elements between pixel point values based on the original image pixels. For example, linear interpolation, bilinear interpolation, nearest neighbor interpolation, or bicubic interpolation, etc. However, this kind of image up-sampling scheme based on interpolation supplements the missing pixel values by calculating the surrounding pixels of the to-be-supplemented points, which cannot solve the problems of connecting contour breakpoints and eliminating jaggies in the image. The above simple network structure pursues the efficiency of image processing, and sacrifices the accuracy of the reconstructed image.

[0152] In the neural network-based SR processing method, the Transformer-based SR architecture introduces a self-attention mechanism in the bottom layer of the visual task, which can significantly improve the SR performance, but also leads to a sub-exponential increase in computational complexity with the sequence length. Although the Mamba-based SR architecture has linear complexity, it mostly adopts a multi-directional parallel scanning strategy to compensate. Therefore, although the complex network structure can greatly improve the performance of SR, it usually brings exponential increase in computational and power overheads.

[0153] Therefore, the embodiments of the present application provide a method, device and system for image processing, which can balance the requirements for accuracy and computational load in the SR technology, reduce the overhead while ensuring the image reconstruction quality.

[0154] The technical solutions provided by the present application will be described in detail below with reference to the drawings.

[0155] FIG. 4 is a schematic diagram of a method 400 for image processing suitable for the embodiments of the present application.

[0156] It should be understood that FIG. 4 shows the steps or operations of the method for image processing, but these steps or operations are only to help those skilled in the art to understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to the specific values or specific scenarios shown in FIG. 4. Those skilled in the art can obviously make various equivalent modifications or changes according to the examples given below in FIG. 4, and such modifications and changes also fall within the scope of the embodiments of the present application. The embodiments of the present application can also perform other operations or variations of the operations in FIG. 4.

[0157] It should be understood that the embodiments shown below do not particularly limit the specific structure of the subject performing the method provided by the embodiments of the present application, as long as the subject can perform image processing according to the method provided by the embodiments of the present application by running a program in which the code of the method provided by the embodiments of the present application is recorded. For example, the method provided by the embodiments of the present application can be performed by a processing device, which can be the execution device in FIG. 2 or a component (such as a processor, a chip, or a chip system, etc.) of the execution device. The execution device can be a cloud device or a terminal device (such as a mobile phone) or the like. Of course, the method provided by the embodiments of the present application can also be performed by a system of a cloud device and a terminal device.

[0158] Optionally, the method can be processed by the CPU in the execution device, or can be processed by the CPU and the GPU together, or can not use the GPU, but use other processors suitable for neural network calculation, which is not limited by the present application.

[0159] As shown in FIG. 4, the method 400 can include the following steps:

[0160] S401: obtaining an input image.

[0161] It should be understood that the input image is a low-resolution image that needs to be processed by SR. The input image can be a complete image to be processed, or a part of the complete image to be processed, which is not limited by the present application.

[0162] Optionally, before the input image is acquired, the image to be processed can also be divided into blocks to obtain the input image, the input image being any one of a plurality of input blocks obtained by dividing the image to be processed.

[0163] It should be understood that, compared to processing the entire image to be processed, block processing can reduce the amount of computation on the one hand and improve the quality of the output image on the other hand. Since images have local correlation, block processing can divide the image to be processed into a plurality of input blocks with high internal correlation, so that the network can obtain relatively accurate results using a smaller number of computation layers. If block processing is not performed, the number of computation layers required by the network is related to the most complex part of the image to be processed, so a larger number of computation layers is required.

[0164] S402: Feature extraction is performed on the input image to obtain first features.

[0165] The first features are basic, low-level shallow features extracted from the input low-resolution image, such as edges, corners, textures, etc., which are used for further processing to generate high-resolution images.

[0166] Optionally, the first features can be implemented through at least one convolutional layer. The convolutional layer is a basic unit for processing image data in deep learning, and can extract local features from the image through a sliding window.

[0167] S403: The first features are input into a first super-resolution network to obtain second features.

[0168] The first super-resolution network is used to perform super-resolution processing on the first features to generate second features with higher pixels. The first super-resolution network includes N2 feature extraction module groups, each feature extraction module group including N1 feature extraction modules, and each feature extraction module including N3 cascaded multi-scale sub-modules. The N3 cascaded multi-scale sub-modules are used to sequentially extract features from the input features in different scanning directions. N1, N2 and N3 are positive integers.

[0169] In this case, compared to a scanning mode in which a plurality of scanning directions are connected in parallel, the feature extraction through the plurality of cascaded multi-scale sub-modules with different scanning directions reduces the complexity of the system. The above scheme can effectively improve the peak signal-to-noise ratio without increasing the amount of calculation and the number of parameters, thereby improving the accuracy of the high-resolution image reconstructed based on the low-resolution image.

[0170] Optionally, the feature extraction module comprises four cascaded multi-scale sub-modules, which are used for sequentially performing feature extraction on the input features in four different scanning directions.

[0171] In this case, the multi-scale sub-modules with different scanning directions are used for feature extraction, which enriches the spatial relationship of the model, so that the correlation between pixels can be obtained from more dimensions, thereby laying a foundation for improving the reconstruction image accuracy of the first super-resolution network.

[0172] For specific content of the four different scanning directions, refer to the related description in FIG. 7 below, which will not be repeated here.

[0173] It should be understood that "the feature extraction module comprises four cascaded multi-scale sub-modules" is only a preferred solution, and the number of multi-scale sub-modules in the feature extraction module is not limited, and the scanning direction corresponding to each multi-scale sub-module is not limited.

[0174] In addition, the N3 cascaded multi-scale sub-modules can have N4 different unfolding directions, N4 is less than or equal to N3, and N4 is a positive integer. That is, in the embodiment of the application, as long as each feature extraction module includes multi-scale sub-modules with different scanning directions, and the plurality of cascaded multi-scale sub-modules in each feature extraction module do not have different scanning directions.

[0175] In a possible implementation, the multi-scale sub-module is configured to: perform first processing based on a state space model (SSM) on the input features to obtain third features; perform down-sampling and second processing based on the SSM on the input features to obtain fourth features; perform feature fusion on the third features and the fourth features to obtain fifth features; perform third processing based on a deep separable convolution attention (DSCA) on the input features to obtain sixth features; and determine an output feature of the multi-scale sub-module according to the input features, the fifth features and the sixth features.

[0176] It should be noted that the specific content of the first processing can refer to the related description of the local SSM (Local SSM, L-SSM) processing implemented by the L-SSM module 820 below, the specific content of the second processing can refer to the related description of the region SSM (Region SSM, R-SSM) processing implemented by the R-SSM module 830 below, and the specific content of the third processing can refer to the related description of the deep separable convolutional attention (Deep Separable Convolutional Attention, DSCA) processing implemented by the DSCA module 810 below, which will not be repeated here.

[0177] Specifically, the "feature fusion of the third feature and the fourth feature to obtain a fifth feature" can include: determining pixel values of the fourth feature in the first pixel range according to the first pixel range and pixel values of the fourth feature in the second pixel range; and determining a fifth feature according to pixel values of the third feature in the first pixel range and pixel values of the fourth feature in the first pixel range. The first pixel range is a pixel range corresponding to the third feature, the second pixel range is a pixel range corresponding to the fourth feature, and the second pixel range is contained in the first pixel range.

[0178] Optionally, the specific content of the feature fusion can refer to the related description of the fusion module 840 below, which will not be repeated here.

[0179] Optionally, before the input features are respectively subjected to the first processing, the second processing and the third processing, the input features can also be subjected to a layer normalization (Layer Normalization, LN) processing. LN does not depend on batch statistical characteristics, can reduce overfitting phenomenon, makes the activation outputs of different layers have more stable distribution, helps the network to better learn feature representation, and improves the generalization ability of the model. In addition, LN also reduces the sensitivity of the network to the initialized weights, makes the network training more robust, and reduces the demand for hyperparameter adjustment. It should be understood that the specific process of the LN processing can refer to the related description of the prior art, which will not be repeated here.

[0180] In this case, different size pixel-level features of the input image are obtained through the first processing and the second processing, and the channel-level feature of the input image is obtained through the third processing. The above three different processing methods can obtain feature extraction results of different dimensions, and different scale representations are aggregated through an adaptive fusion module, thereby enhancing the spatial perception ability of the first super-resolution network, obtaining more comprehensive context information, and improving the accuracy of high-resolution image reconstruction.

[0181] In another possible implementation, the multi-scale sub-module is configured to: perform first processing on the input feature based on a state space model (SSM) to obtain a third feature; perform second processing on the input feature based on the SSM after down-sampling the input feature to obtain a fourth feature; perform feature fusion on the third feature and the fourth feature to obtain a fifth feature; and determine the output feature of the multi-scale sub-module according to the input feature and the fifth feature.

[0182] It should be understood that the first processing, the second processing, and the feature fusion can refer to the related description above, and thus will not be repeated here.

[0183] Optionally, the input feature can be processed by LN before being processed by the first processing and the second processing.

[0184] In this case, different-precision pixel-level features of the input image can be obtained through the first processing and the second processing, different scales of representation can be aggregated, the spatial perception capability of the first super-resolution network can be enhanced, richer context information can be obtained, and the accuracy of high-resolution image reconstruction can be improved.

[0185] In another possible implementation, the multi-scale sub-module is configured to: perform first processing on the input feature based on a state space model (SSM) to obtain a third feature; and determine the output feature of the multi-scale sub-module according to the input feature and the third feature.

[0186] It should be understood that the first processing can refer to the related description above, and thus will not be repeated here.

[0187] Optionally, the input feature can be processed by LN before being processed by the first processing.

[0188] It should be noted that, based on the above implementations, the determination of the output feature of the multi-scale sub-module can include: determining a seventh feature according to the input feature, the fifth feature, and the sixth feature, performing fourth processing on the seventh feature based on a feed-forward network (FFN) to obtain the output feature of the multi-scale sub-module; or determining a seventh feature according to the input feature and the fifth feature, performing fourth processing on the seventh feature based on a feed-forward network (FFN) to obtain the output feature of the multi-scale sub-module; or determining a seventh feature according to the input feature and the third feature, performing fourth processing on the seventh feature based on a feed-forward network (FFN) to obtain the output feature of the multi-scale sub-module.

[0189] It should be understood that the fourth processing on the seventh feature based on the FFN can further capture more complex features and representations through nonlinear transformation, which helps to improve the accuracy of high-resolution image reconstruction.

[0190] In a possible implementation, the determining the seventh feature according to the input feature, the fifth feature and the sixth feature can be implemented based on a Residual Network (ResNet) strategy.

[0191] It should be understood that the embodiments of the present application are not limited to the implementation of the residual network, and specific implementation can be referred to the related description in the prior art.

[0192] For example, the fifth feature and the sixth feature can be fused and summed with the input feature to obtain the seventh feature after residual processing.

[0193] For example, the information in the input feature can be adjusted by a learnable scaling factor S1 in the process of mapping the input feature to the output. That is, the weight of the input feature can be adjusted by the learnable scaling factor S1, and then summed with the fifth feature and the sixth feature.

[0194] Optionally, the seventh feature can also be processed by the FFN (the fourth processing) to obtain the output feature of the multi-scale sub-module. It should be understood that the embodiments of the present application are not limited to the specific implementation and corresponding parameter setting of the FFN, and specific implementation can be referred to the related content in the prior art.

[0195] Optionally, the seventh feature can also be processed by the FFN (the fourth processing) to obtain the output feature of the multi-scale sub-module. It should be understood that the embodiments of the present application are not limited to the specific implementation and corresponding parameter setting of the FFN, and specific implementation can be referred to the related content in the prior art.

[0196] Optionally, the FFN can also be implemented based on the ResNet strategy.

[0197] For example, the input of the FFN can be directly mapped to the output. For example, the output feature of the FFN can be summed with the input feature of the FFN (or the input feature of the LN before the FNN) to obtain the output feature after residual processing.

[0198] For example, the information in the input feature of the FFN (or the input feature of the LN before the FNN) can be adjusted by a learnable scaling factor S2 in the process of mapping the input of the FFN to the output. That is, the weight of the input feature of the LN before the FNN can be adjusted by the learnable scaling factor S2, and then summed with the output feature after the FFN processing.

[0199] It should be understood that the specific structure and implementation of the multi-scale sub-module can also be referred to the related description of the HMB523 below, and the embodiments of the present application will not be repeated here.

[0200] In another possible implementation, the determining the seventh feature according to the input feature and the fifth feature can also be implemented based on a ResNet strategy.

[0201] For example, the fifth feature and the input feature can be summed to obtain the seventh feature processed by the residual.

[0202] For example, the input feature can be directly mapped to the output. For example, the fifth feature and the input feature can be summed to obtain the seventh feature processed by the residual.

[0203] Optionally, the seventh feature can also be processed by the FFN (the fourth processing) to obtain the output feature of the multi-scale sub-module.

[0204] Optionally, the seventh feature can also be processed by the FFN (the fourth processing) to obtain the output feature of the multi-scale sub-module.

[0205] Optionally, the FFN can also be implemented based on a ResNet strategy.

[0206] For example, the input of the FFN can be directly mapped to the output; or, in the process of mapping the input of the FFN to the output, the information in the input feature of the FFN (or the input feature of the LN before the FNN) can be adjusted by the learnable scaling factor S2. That is, the weight of the input feature of the LN before the FNN can be adjusted by the learnable scaling factor S2, and the output feature processed by the FFN is summed.

[0207] In another possible implementation, the determining the seventh feature according to the input feature and the third feature can be implemented based on a ResNet strategy.

[0208] Optionally, the seventh feature can also be processed by the FFN (the fourth processing) to obtain the output feature of the multi-scale sub-module.

[0209] Optionally, the seventh feature can also be processed by the FFN (the fourth processing) to obtain the output feature of the multi-scale sub-module.

[0210] Optionally, the FFN can also be implemented based on a ResNet strategy.

[0211] For example, the input of the FFN can be directly mapped to the output; or, in the process of mapping the input of the FFN to the output, the information in the input features of the FFN (or the input features of the LN before the FNN) can be adjusted by the learnable scaling factor S2. That is, the weight of the input features of the LN before the FNN can be adjusted by the learnable scaling factor S2, and summed with the output features after the FFN.

[0212] Optionally, each of the N2 groups of feature extraction modules is configured to: perform convolution on the output features of the N1th feature extraction module in each group of feature extraction modules to obtain an eighth feature; and perform residual processing on the input features of the first feature extraction module in each group of feature extraction modules and the eighth feature to obtain the output features of each group of feature extraction modules.

[0213] It should be understood that the output features are processed by convolution, which can realize the refinement of the features, and the residual structure allows the network to reuse the features learned by the previous layers in the deep layers, and directly transmits the shallow layer features to the deep layer through the connection of skipping some layers, so that the deep network can more easily learn effective feature representation, and alleviate the problem of gradient disappearance or gradient explosion.

[0214] Optionally, the first super-resolution network is further configured to: perform convolution on the output features of the N2th group of feature extraction modules in the first super-resolution network to obtain a ninth feature; and perform residual processing on the first feature and the ninth feature to obtain the second feature.

[0215] Similarly, the convolution processing can realize the refinement of the features, and the residual structure can enable the deep network to more easily learn effective feature representation, and alleviate the problem of gradient disappearance or gradient explosion.

[0216] S404: based on the second feature, reconstructing an output image, the resolution of the output image being higher than that of the input image.

[0217] Optionally, before the input image is obtained, the image to be processed can also be divided into blocks, and correspondingly, after the input image is reconstructed to obtain the output image, the plurality of input image blocks can be spliced into a target image.

[0218] It should be noted that the embodiments of the present application do not limit the implementation of the image reconstruction module, for example, the high-resolution frame reconstruction operation can be performed by cascading convolution and sub-pixel convolution, and specific descriptions can be referred to the related descriptions of the prior art.

[0219] It should be understood that, for the case that the input image is not blocked, the input image is the image to be processed itself, and the output image is the target image, i.e., the input image is the image to be processed without being segmented, and the output image does not need to be spliced. When the input image is a segmented image block of the image to be processed, the target image is spliced from the output image. That is, the input image can be a complete image to be processed, or an input image block after the image to be processed is segmented. The specific form of the input image is not limited in the embodiments of the present application.

[0220] FIG. 5 is a schematic block diagram of an image processing method 500 suitable for the embodiments of the present application.

[0221] As shown in FIG. 5, the method 500 (or also can be called Hi-Mamba) includes a shallow feature extraction module 510, a deep feature extraction module 520, and a deep feature extraction module 520, which are described in detail below.

[0222] The shallow feature extraction module 510 is configured to extract basic and low-level shallow features (feature #1) from the input low-resolution image, such as edges, corners, textures, etc., which are used for further processing to generate a high-resolution image.

[0223] Optionally, the shallow feature extraction module 510 can be implemented by at least one convolutional layer. The convolutional layer is a basic unit for processing image data in deep learning, and can extract local features of the image through a sliding window.

[0224] For example, in a super-resolution task, the shallow feature extraction module 510 can include a convolutional layer, which can use a small convolutional kernel (for example, a 3x3 convolutional kernel) to extract shallow features of the image. That is, the feature #1 (feature #1 e R H×W×C ), where H and W represent the height and width of the input image, and C is the number of channels.

[0225] The deep feature extraction module 520 is configured to extract deep features (feature #2) of the image based on the feature #1, which generally contains higher-level semantic information and more detailed texture details, and is crucial for generating a high-precision high-resolution (HR) image.

[0226] In a possible implementation, the deep feature extraction module 520 can include N2 Mamba groups 521 (N2 is a positive integer). That is, the feature #1 can be processed by the N2 Mamba groups 521 in turn to obtain the feature #2, where N2 determines the network depth of the deep feature extraction module 520.

[0227] Optionally, the deep feature extraction module 520 can further include a convolutional layer for refining the resultant feature #2 processed by the N2 Mamba groups 521. For example, the convolutional layer can use a smaller convolutional kernel (e.g., a 3x3 convolutional kernel).

[0228] Optionally, the deep feature extraction module 520 can be implemented based on a Residual Network (ResNet) strategy. The residual structure allows the network to reuse the features learned by the previous layers in the deep layers, and directly pass the shallow features to the deep layers through the connection of skipping some layers. Thus, the deep network can more easily learn effective feature representation, and alleviate the problem of gradient vanishing or gradient explosion.

[0229] For example, the input of the deep feature extraction module 520 can be directly mapped to the output. For example, the feature #1 and the feature #2 (or the feature #2 processed by the convolutional layer) can be summed. It should be understood that the implementation of the residual network is not limited in the embodiments of the present application, and can be referred to the related description in the prior art.

[0230] In a possible implementation, the Mamba group 521 can include N1 cascaded multi-directional multi-scale Mamba groups (DA-HMG) 522 (N1 is a positive integer). That is, the feature #1 is sequentially processed by the N1 DA-HMGs 522 to obtain the feature #3, wherein N1 determines the network depth of the Mamba group 521. The specific structure of the DA-HMG 522 will be described in detail below in FIG. 6, which will not be repeated here.

[0231] Optionally, the Mamba group 521 can further include a convolutional layer for refining the resultant feature #3 processed by the N1 DA-HMGs 522.

[0232] Optionally, in order to ensure the stability of the training, the Mamba group 521 can be implemented based on the ResNet strategy.

[0233] For example, the input of the Mamba group 521 can be directly mapped to the output. For example, the feature #1 and the feature #3 (or the feature #3 processed by the convolutional layer) can be summed to obtain the feature #2. It should be understood that the implementation of the residual network is not limited in the embodiments of the present application, and can be referred to the related description in the prior art.

[0234] Optionally, the DA-HMG 522 can be composed of a plurality of hierarchical Mamba blocks (HMB) 523 in different directions in cascade. It should be understood that the embodiments of the present application do not limit the number and direction of the HMB 523 contained in the DA-HMG 522. In addition, unlike the prior art which uses multi-directional scanning to process the input features, each HMB 523 in the embodiments of the present community only scans the input features in a single direction, thereby effectively reducing the computational overhead in the feature expansion process. For the specific structure of the HMB 523, it will be described in detail in FIG. 8 below, and the present application will not be repeated here.

[0235] The image reconstruction module 530 is used for image reconstruction based on the feature #2, so as to obtain an output high-resolution image (target image).

[0236] It should be noted that the embodiments of the present application do not limit the implementation of the image reconstruction module, for example, the high-resolution frame reconstruction operation can be performed by cascaded convolution and sub-pixel convolution, and the specific implementation can refer to the related description of the prior art.

[0237] Optionally, in order to further improve the quality of the image super-resolution output and further improve the computational efficiency, before the input image is processed by the method 500, the to-be-processed image can also be blocked to obtain a plurality of input blocks of the to-be-processed image. That is, the above-mentioned input image is any one of the plurality of input blocks of the to-be-processed image, and the input blocks are processed by the method 500 respectively to obtain a plurality of high-resolution output blocks (which can also be referred to as output blocks) corresponding to the input blocks.

[0238] Correspondingly, after the image processing of the plurality of blocks is performed by the method 500, the plurality of high-resolution output blocks also need to be spliced to obtain the target image. That is, by sequentially processing the plurality of input blocks of the to-be-processed image by the method 500, a plurality of high-resolution output blocks are obtained, and then the plurality of high-resolution output blocks are spliced into the target image.

[0239] Compared with processing the entire to-be-processed image, the block processing can reduce the amount of operation on the one hand, and can improve the quality of the output image on the other hand. Since the image has local correlation, for example, for a to-be-processed image including sky and flowers, if the to-be-processed image is blocked, the to-be-processed image can be divided into a plurality of input blocks with high internal correlation, for example, an input block only includes flowers or only includes sky, etc., which can to a certain extent make the sub-network in the method 500 use fewer operation layers to obtain more accurate results. For the complete to-be-processed image, the number of operation layers required by the sub-network in the method 500 depends more on the most complex content part of the to-be-processed image, so the number of operation layers required is more.

[0240] In addition, for a plurality of input patches of the image to be processed, the number of operation layers required by each patch can be different, for example, the number of operation layers required by a patch including flowers and plants is more, and the number of operation layers required by a patch including only the sky is less, so that the network structure can be more flexible while ensuring the accuracy.

[0241] It should be understood that, for the case where the input image is not blocked, the input image is the image to be processed itself, and the output image is the target image, that is, the input image is the image to be processed without being segmented, and the output image does not need to be spliced. When the input image is a segmented patch of the image to be processed, the target image is spliced from the output image. That is, the input image can be a complete image to be processed, or an input patch after the image to be processed is segmented. The specific form of the input image is not limited in the embodiments of the present application.

[0242] FIG. 6 is a structural schematic block diagram of a DA-HMG suitable for the embodiments of the present application.

[0243] For example, in order to facilitate understanding, for the specific structure of the DA-HMG 522 in FIG. 5, only the DA-HMG 522 including four HMBs 523 in different directions shown in FIG. 6 is taken as an example for description, which does not constitute a limitation on the embodiments of the present application.

[0244] As shown in FIG. 6, the DA-HMG 522 is composed of four HMBs 523 in different directions. Specifically, the four HMBs in different directions include a horizontal direction HMB (HMB-H), a vertical direction HMB (HMB-V), a reverse horizontal direction HMB (HMB-RH), and a reverse vertical direction HMB (HMB-RV). The four different directions will be described in detail below in combination with FIG. 7.

[0245] FIG. 7 is a schematic diagram of image unfolding modes suitable for the embodiments of the present application.

[0246] As shown in (a) of FIG. 7, horizontal direction unfolding indicates unfolding a sequence of images along the horizontal direction, that is, unfolding from the top left to the bottom right in the order of "1, 2, 3, 4"; thus, the HMB-H indicates that in the process of processing the input image features, it can include unfolding a sequence in the manner of unfolding along the horizontal direction.

[0247] As shown in (b) of FIG. 7, vertical direction unfolding indicates unfolding a sequence of images along the vertical direction, that is, unfolding from the top left to the bottom right in the order of "1, 3, 2, 4"; thus, the HMB-R indicates that in the process of processing the input image features, it can include unfolding a sequence in the manner of unfolding along the vertical direction.

[0248] As shown in Figure 7(c), the reverse horizontal expansion means expanding the image into a sequence in the reverse horizontal direction, that is, expanding it from the bottom right to the top left in the order of "4, 3, 2, 1"; thus, HMB-RH means that in the process of processing the input image features, it can be expanded into a sequence in the reverse horizontal direction.

[0249] As shown in Figure 7(d), reverse vertical expansion means expanding the image into a sequence along the reverse vertical direction, that is, expanding it from the bottom right to the top left in the order of "4, 2, 3, 1"; thus, HMB-RV means that in the process of processing the input image features, it can be expanded into a sequence in the reverse vertical direction.

[0250] It should be understood that DA-HMG 522 can cascade HMB 523 in different directions in different orders; or, it can cascade HMB 523 in other directions different from the four deployment methods mentioned above (e.g., deployed at intervals or rotated according to preset rules); or, it can cascade HMB 523 including repeated deployment directions (e.g., DA-HMG 522 is composed of cascaded HMB-H, HMB-R, HMB-RH, HMB-RV, and HMB-H). This application does not limit any of these possibilities.

[0251] It should be understood that in DA-HMG 522, the number of HMB 523 in multiple different directions will affect the SR performance of method 500.

[0252] In a preferred structure of DA-HMG 522, DA-HMG 522 is constructed by cascading four HMB 523s (HMB-H, HMB-V, HMB-RH and HMB-RV) in different directions as shown in Figure 6.

[0253] Specifically, ablation experiments were conducted to analyze the number of HMB 523 cells in different directions within DA-HMG 522 and their corresponding PSNR and SSIM performance. As shown in Table 1, the four-direction alternating model shown in Figure 6 exhibits the best performance. Compared to the unidirectional model containing only one deployment direction and the bidirectional alternating model containing two deployment directions, it has the highest PSNR and SSIM, with PSNR improved by 0.12 dB and 0.06 dB, respectively, and SSIM improved by 0.0003 and 0.0012, respectively.

[0254] It should be noted that, compared to parallel scanning measurements in existing technologies, cascaded alternating direction scanning does not incur additional computational and memory costs.

[0255] Table 1

[0256] FIG. 8 is a structural schematic block diagram of an HMB suitable for use in the present application.

[0257] As shown in FIG. 8, the HMB 523 can respectively perform deep separable convolutional attention (DSCA) processing, local SSM (L-SSM) processing and / or region SSM (R-SSM) processing on the input features, which will be described in detail below, and the present application will not be repeated here.

[0258] It should be understood that in the HMB 523, the DSCA processing, L-SSM processing and R-SSM processing will also affect the SR performance of the method 500.

[0259] In a preferred structure of the HMB 523, the DSCA processing, L-SSM processing and R-SSM processing as shown in FIG. 8 are included.

[0260] Specifically, the processing modes included in the HMB 523 and their corresponding SR performance are analyzed through ablation experiments. As shown in Table 2, compared with the HMB including only the L-SSM module, the R-SSM module only increases 0.02M parameters and 13G FLOPs, but brings 0.16dB PSNR and 0.0013 SSIM improvement; the introduction of the DSCA module further improves 0.03dB PSNR and 0.0004 SSIM, with the best SR performance.

[0261] In addition, Table 2 also shows the network structure including L-SSM, R-SSM, DSCA and DA-HMG, i.e. the DA-HMG 522 composed of four HMBs 523 in different directions as shown in FIG. 6 is cascaded in the image processing method 500 shown in FIG. 5, and each HMB 523 includes L-SSM, R-SSM and DSCA. The above-mentioned scheme further improves 0.13dB PSNR and 0.0012 SSIM, without bringing additional computational cost (the parameter amount and FLOPs remain unchanged).

[0262] Table 2

[0263] Optionally, before the input features are respectively processed by the three processes described above, the input features can also be processed by a layer normalization (LN) process. LN does not depend on batch statistics, can reduce overfitting, make the activation outputs of different layers have more stable distribution, help the network better learn feature representation, and improve the generalization ability of the model. In addition, LN also reduces the sensitivity of the network to the initialized weights, makes the network training more robust, and reduces the need for hyperparameter adjustment.

[0264] It should be understood that the specific process of LN processing can refer to the related description of the prior art, and the present application will not be repeated here.

[0265] It should be noted that the HMB 523 can process the input of the DSCA module 810, the L-SSM module 820 and the R-SSM module 830 through three LN modules respectively; or the DSCA module 810 and the L-SSM module 820 can also share the output of the same LN module, which is not limited by the present application.

[0266] In addition, in the HMB 523, the outputs of the L-SSM module 820 and the R-SSM module 830 can also be processed by the fusion module 840, which will be described in detail below, and the present application will not be repeated here.

[0267] Optionally, in order to ensure the stability of the training, the HMB 523 can also be implemented based on the ResNet strategy.

[0268] For example, the outputs of the DSCA module 810 and the fusion module 840 can be fused and summed with the input features of the HMB 523 to obtain the output features after residual processing.

[0269] For example, in the process of mapping the input of the HMB 523 to the output, the information in the input features of the HMB 523 can also be adjusted by a learnable scaling factor S1. That is, the weight of the input features of the HMB 523 can be adjusted by a learnable scaling factor S1, and summed with the features processed by the DSCA module 810 and the fusion module 840.

[0270] It should be understood that the present application does not limit the implementation of the residual network, and the related description of the prior art can be referred to.

[0271] Optionally, the FFN can also be included in the HMB 523. That is, for the output processed by the DSCA module 810, the L-SSM module 820 and the R-SSM module 830, the FFN can also be used to process the output. Specifically, the FFN can be implemented by a multi-layer perceptron (MLP). It should be understood that the embodiments of the present application do not limit the specific implementation of the FFN and the corresponding parameter settings, and specific reference can be made to related contents in the prior art.

[0272] Optionally, before the FFN processing of the output processed by the DSCA module 810, the L-SSM module 820 and the R-SSM module 830, the input features can also be processed by the LN. The specific process of the LN processing can be referred to the related description in the prior art, which will not be repeated here.

[0273] Optionally, the FFN can also be implemented based on the ResNet strategy. For example, the input of the FFN can be directly mapped to the output. For example, the output features of the FFN can be summed with the input features of the FFN (or the input features of the LN before the FNN), so as to obtain the output features processed by the residual.

[0274] Optionally, in the process of mapping the input of the FFN to the output, the information in the input features of the FFN (or the input features of the LN before the FNN) can also be adjusted by the learnable scaling factor S2. That is, the weight of the input features of the LN before the FNN can be adjusted by the learnable scaling factor S2, and then summed with the output features processed by the FFN.

[0275] It should be understood that the embodiments of the present application do not limit the implementation of the residual network, and specific reference can be made to the related description in the prior art.

[0276] The DSCA module 810 is implemented based on a DSCA network, which is used to extract the feature representation of the input features in the channel dimension. Specifically, the DSCA network combines the depth separable convolution and the attention mechanism. Compared with the classic channel attention module, the DSCA can reduce the model calculation to a certain extent.

[0277] In the depthwise separable convolution, a standard convolution operation is decomposed into a depthwise convolution and a pointwise convolution. The depthwise convolution applies a convolution kernel to each input channel independently, while the pointwise convolution combines the output of the depthwise convolution through a 1x1 convolution kernel to generate the final output feature map. The depthwise separable convolution can reduce the amount of calculation and the number of parameters while maintaining the representation ability of the network. The attention mechanism allows the model to dynamically focus on specific parts of the input data. By calculating the weight of the input features, the attention mechanism can enhance the model's perception of important features, thereby improving the performance of the model.

[0278] It should be understood that the specific process of DSCA processing can be referred to the related description of the prior art, which will not be repeated here.

[0279] The L-SSM module 820 is implemented based on the SSM architecture, and is used to extract the feature representation of the input features at the local pixel scale.

[0280] FIG. 9 is a structural schematic block diagram of an L-SSM suitable for the embodiments of the present application. As shown in FIG. 9, the L-SSM module 820 includes two branches.

[0281] For branch #1, the feature channels of the input features (input features e R H×W×C ) are first expanded to C1 through a linear layer; where C1 is a channel expansion factor, which can be determined according to a predefined manner. Then the input features are processed through a depthwise convolution layer (DWConv) and an activation function (for example, SiLU). Specifically, a plurality of convolution layers can be stacked to enable the model to learn the hierarchical feature representation of the data, and after the convolution operation, the activation function can be introduced to introduce nonlinearity, so that the model can learn more complex patterns.

[0282] The features processed by the activation function are further processed by the SSM module, and then the output features of the SSM module are processed by LN. Specifically, the SSM module is used to expand the input features according to one of the four expansion modes shown in FIG. 7 (i.e., processing the flattened feature map into a one-dimensional token sequence), and processing the expanded sequence according to the SSM algorithm.

[0283] For example, if the HMB 523 is HMB-H, in the HMB-H, the SSM module in the L-SSM can expand the input features into a sequence in a horizontal direction; if the HMB 523 is HMB-V, in the HMB-V, the SSM module in the L-SSM can expand the input features into a sequence in a vertical direction; if the HMB 523 is HMB-RH, in the HMB-RH, the SSM module in the L-SSM can expand the input features into a sequence in a reverse horizontal direction; if the HMB 523 is HMB-RV, in the HMB-RV, the SSM module in the L-SSM can expand the input features into a sequence in a reverse vertical direction.

[0284] In addition, for the specific implementation process of the SSM algorithm, reference can be made to the related description of the foregoing terms and prior art, which will not be repeated herein.

[0285] For branch #2, the feature channels of the input features (input features ∈ R H×W×C ) are expanded to C1 through a linear layer; where C1 is a channel expansion factor, which can be determined in a predefined manner. Then the input features are processed through an activation function (for example, SiLU).

[0286] In addition, the output features of branch #1 and branch #2 can also be aggregated, and the feature channels are projected back to C through a linear layer, so as to generate output features (output features of L-SSM ∈ R H×W×C ) having the same shape as the input features of the L-SSM.

[0287] Optionally, the output features of branch #1 and branch #2 can be aggregated by Hadamard product, that is, the element-wise multiplication of the output feature matrices of branch #1 and branch #2 is performed, so as to combine the features of the two branches; or other aggregation manners can also be used, which are not limited by the embodiments of the present application.

[0288] It should be noted that the embodiments of the present application only show the functions and connection relationships of various modules, and do not limit the number of layers, the number of neurons, the size of the convolution kernel and the specific type of the activation function of the deep convolutional layer.

[0289] The R-SSM module 830 is implemented based on the SSM architecture, and is used to extract the feature representation of the input features in a smaller region of pixel scale.

[0290] FIG. 10 is a structural schematic block diagram of an R-SSM suitable for the embodiments of the present application.

[0291] As shown in FIG. 10, the R-SSM module 830 also includes two branches, and the structures of the two branches are similar to those of the L-SSM module 820. For the specific structures of the two branches, reference can be made to the related description in the L-SSM module 820, which will not be repeated here.

[0292] It should be noted that, unlike the L-SSM module 820, the R-SSM module 830 can also map the input features through a linear layer and reduce the spatial dimension of the features using a down-sampling module before processing the input features through the two branches, so as to extract the feature representation of the input features in a smaller area of pixel scale. After processing by the down-sampling module, the input features (n is a positive integer).

[0293] Optionally, the input features can be down-sampled by using average pooling (Average Pooling); or other down-sampling methods such as maximum pooling (Max Pooling), minimum pooling (Min Pooling), etc. For the specific down-sampling method adopted by the down-sampling module and the implementation process thereof, reference can be made to the related description in the prior art, which will not be limited herein.

[0294] It should also be noted that, in the branch #3 and the branch #4, the linear layer expands the feature channels of the input features (the input features ) to C2, where C2 is a channel expansion factor, which can be determined according to a predefined manner. Correspondingly,

[0295] After aggregating the output features of the branch #3 and the branch #4, the feature channels are projected back to C through a linear layer, so as to generate output features (the output features of the R-SSM ) having the same shape as the input features of the branch #3 and the branch #4.

[0296] It should be understood that different sampling sizes (n) will generate features of different sizes, thereby affecting the final inference speed and model effect.

[0297] In a preferred structure of the R-SSM module 830, the down-sampling module processes the input features to a size of 4x4 pixel area (i.e., ).

[0298] Specifically, the ablation experiment is performed to analyze the SR performance of different sizes of pixel regions and their corresponding dimensions. As shown in Table 3, the feature size corresponding to the region size of 8x8 is large, and thus the corresponding SR performance is poor. The region size of 1x1 significantly increases the inference time (GPU (ms)) and FLOPs, but does not bring obvious improvement in other indicators, and the performance benefit is small. The pixel region size of 4x4, compared with the region size of 8x8, improves the 0.09dB PSNR while only increasing 3G FLOPs and 4ms latency.

[0299] Table 3

[0300] The fusion module 840 is configured to fuse the outputs of the L-SSM and the R-SSM, thereby improving the learning ability of the inter-pixel dependency in the image processing process.

[0301] The output feature A of the L-SSM is in R H×W×C , and the output feature B of the R-SSM is in R First, the output feature of the R-SSM is expanded into the same shape as the output feature of the L-SSM through two-dimensional repetition, to ensure that each region token is mapped to the corresponding local token (i.e., replacing each pixel value in the corresponding region (n x n) in feature A with each pixel value in feature B, so that feature B' is in R H×W×C ). Then, the weights of feature A and feature B' are adjusted through a learnable parameter S f , and the adjusted feature A and feature B' are fused to generate a new fusion feature C. The learnable parameter S f is an adaptive fusion weight obtained through training, the weight corresponding to feature A is (1-S f ), and the weight corresponding to feature B' is S f .

[0302] It should be noted that the fusion of the adjusted feature A and feature B' can be achieved by the add method as described in the foregoing term section, i.e., summing the pixel values at the corresponding positions in feature A and feature B', respectively; or other fusion methods can also be used. The fusion method of feature A and feature B' is not limited in the embodiments of the present application.

[0303] FIG. 11 is a structural schematic block diagram of a fusion module suitable for the embodiments of the present application.

[0304] As an example, as shown in FIG. 11, the output feature A of the L-SSM has a local pixel space of 4x4, while the output feature B of the corresponding R-SSM has a region pixel space of 2x2. The shape of the output feature B of the R-SSM is adjusted by means of two-dimensional Repeat, so as to be the same as the shape of the output feature A of the L-SSM. Specifically, the pixel #a in the output feature B of the R-SSM is used to replace the pixels #1, #2, #5 and #6 in the output feature A of the L-SSM; the pixel #b in the output feature B of the R-SSM is used to replace the pixels #3, #4, #7 and #8 in the output feature A of the L-SSM; the pixel #c in the output feature B of the R-SSM is used to replace the pixels #9, #10, #13 and #14 in the output feature A of the L-SSM; and the pixel #d in the output feature B of the R-SSM is used to replace the pixels #11, #12, #15 and #16 in the output feature A of the L-SSM, so as to obtain a new output feature B' of the R-SSM. The learnable parameters S f The weights of the features A and B' are adjusted respectively, and the adjusted features A and B' are fused to generate a new fused feature C.

[0305] Table 4 shows the performance comparison of the method 500 (Hi-Mamba) and a plurality of existing lightweight SR methods (for example, CARN, EDSR-baseline, LAPAR-A, MAFFSRN, RFDN, SAFMN, ESRT, N-Gram, SwinIR-Light, SPIN, DLGSANet-Light and MambaIR) on a plurality of public data sets (for example, Set5, Set14, BSD100, Urban100 and Manga109). Among them, in order to better compare the performance with the existing lightweight SR methods, the Hi-Mamba model adopts a smaller number of Mamba groups 521 and DA-HMG 522, that is, the values of N2 and N1 are smaller (for example, N2 = 6, N1 = 1), so that the Hi-Mamba model (corresponding to Hi-Mamba-S in Table 4) has a smaller network depth.

[0306] As shown in Table 4, taking the double SR method as an example, the performance of Hi-Mamba-S is significantly improved compared with the SOTA (state-of-the-art, a model used to describe the current optimal effect on a certain task in machine learning) method based on CNN (for example, CARN, EDSR-baseline, LAPAR-A, MAFFSRN, RFDN and SAFMN); compared with the existing method based on Transformer architecture (for example, ESRT, N-Gram, SwinIR-Light, SPIN and DLGSANet-Light), Hi-Mamba-S has higher accuracy and speed. Compared with the SOTA Mamba method MambaIR, Hi-Mamba-S reduces 303G FLOPs and achieves two times GPU acceleration, while the PSNR on the Urban100 dataset is improved by 0.06dB. In addition, the performance of Hi-Mamba-S on other multiple scales also achieves the best or sub-optimal effect.

[0307] Table 4

[0308] Table 5 shows the performance comparison of method 500 (Hi-Mamba) and multiple existing classic SR methods (for example, EDSR, RCAN, SAN, HAN, NLSN, ELAN, DLGSANet, SwinIR, EDT, SRFormer and MambaIR) on multiple public datasets (for example, Set5, Set14, BSD100, Urban100 and Manga109). Among them, in order to better compare the performance with the existing classic SR method, the Hi-Mamba model adopts a larger number of Mamba groups 521 and DA-HMG 522, that is, the values of N2 and N1 are larger (for example, N2=5, N1=2), so that the Hi-Mamba model (corresponding to Hi-Mamba-L in Table 5) has a larger network depth.

[0309] As shown in Table 5, the performance of Hi-Mamba-L is significantly improved compared with the SOTA method based on CNN (for example, EDSR, RCAN, SAN, HAN and NLSN); compared with the existing method based on Transformer architecture (for example, SRFormer), Hi-Mamba-L improves the PSNR on the Manga109 dataset of double SR by 0.32dB; and compared with the SOTA Mamba method MambaIR, Hi-Mamba-L improves the PSNR on the Set14 and Urban100 datasets of double SR by 0.09dB and 0.06dB, respectively.

[0310] Table 5

[0311] Figure 12 is a schematic diagram of the effect of the method of image processing suitable for embodiments of the application.

[0312] Figure 12 demonstrates the effect of the method shown in Figures 4 and 5 using Local Attribution Map (LAM). LAM is an explainability tool for super-resolution reconstruction tasks, aiming to find the pixels in the low-resolution input image that contribute most strongly to the network’s super-resolution result by calculating the contribution of each input pixel or patch to the final output. LAM tracks the information used by the model and, under the premise of specifying the local area of the super-resolution result, labels the pixels that contribute most to the super-resolution result. Specifically, this can be demonstrated by a heatmap or other visualization means.

[0313] In the context of LAM visualization, the Diffusion Index (DI) score is an indicator used to quantify the LAM result. The DI score is calculated by analyzing the diffusion pattern in the LAM map, reflecting the network’s information utilization on different image content. A higher DI score indicates that the network can extract information from a wider range of pixels, while a lower DI score indicates that the network only utilizes a limited area of the image for feature extraction.

[0314] As can be seen from Figure 12, LAM visualization analysis was performed on the method (Hi-Mamba) described in embodiments of the application and existing RCAN, SwinIR and MambaIR methods based on real-world pictures. The results show that, compared to existing image processing methods, Hi-Mamba has a higher DI score (10.6) and a larger pixel utilization range, thereby being able to capture more contextual information to restore the texture details of the image.

[0315] Figure 13 is a schematic diagram of the effect of the method of image processing suitable for embodiments of the application.

[0316] As shown in Figure 13, a visual result comparison was also performed on the method (Hi-Mamba-S) described in embodiments of the application and existing SR methods based on the 004 picture of the Urban100 dataset. In Figure 13, (a) is a high-resolution image, (j) is an image restored by Hi-Mamba-S, and (b) to (i) correspond to images restored by Bicubic, CARN, EDSR, SAFMN, ESRT, SwinIR-Light, N-Gram and MambaIR methods, respectively.

[0317] The evaluation results of different methods on the 004 picture of the Urban100 dataset show that the Hi-Mamba-S better explores the information of the LR image in the three dimensions of channel, local pixel and regional pixel by fusing the features of three different scales of DSCA, L-SSM and R-SSM, and thus more accurately and clearly restores the texture details of the image, has better robustness, and has greater application potential for regular texture data.

[0318] FIG. 14 is a schematic diagram of an effect of a method of image processing suitable for the embodiments of the present application.

[0319] As shown in FIG. 14, the method (Hi-Mamba-S) described in the embodiments of the present application and the existing SR methods are further compared in visual results based on the 078 picture of the Urban100 dataset. Among them, (a) in FIG. 14 is a high-resolution image, (j) in FIG. 14 is an image restored by the Hi-Mamba-S, and (b) to (i) in FIG. 14 correspond to images restored by the Bicubic, CARN, EDSR, SAFMN, ESRT, SwinIR-Light, N-Gram and MambaIR methods, respectively.

[0320] Similar to the results of FIG. 13, the evaluation results of different methods on the 078 picture of the Urban100 dataset also show that the image restored by the Hi-Mamba-S has more accurate and clear image texture details, and has the closest visual effect to the high-resolution image.

[0321] FIG. 15 is a schematic diagram of a model training method 1500 suitable for the first super-resolution network provided in the embodiments of the present application.

[0322] The method 1500 includes the following steps:

[0323] S1501: Obtain an image sample pair.

[0324] The obtaining unit is configured to obtain an image sample pair, the image sample pair including a first image and a second image corresponding to the first image, the resolution of the first image being lower than the resolution of the second image.

[0325] The training device can collect the image sample pair through a sensor (such as a camera, etc.), can obtain the image sample pair from a database, and can also receive the image sample pair sent by other devices, and the present application does not limit the way of obtaining the image sample pair.

[0326] When the first super-resolution network needs to be trained, the training apparatus can obtain a batch of training samples, i.e., image sample pairs used for training. Among them, the first image and the second image corresponding to the first image are known.

[0327] S1502: According to the image sample pair, the initial model of the first super-resolution network is trained to obtain the first super-resolution network.

[0328] In a possible implementation, according to the image sample pair, the initial model of the first super-resolution network is trained to obtain the first super-resolution network, including: performing the first feature extraction on the first image to obtain first image input features; inputting the first image features into the initial model of the first super-resolution network to obtain first image output features; based on the first image output features, reconstructing a third image; and adjusting parameters of the initial model of the first super-resolution network according to the third image and the second image to train the first super-resolution network.

[0329] The specific process of obtaining the first prediction result can refer to the related description in the foregoing FIG. 4 to FIG. 11, which will not be repeated here.

[0330] Optionally, adjusting the parameters of the initial model of the first super-resolution network according to the third image and the second image to train the first super-resolution network can include: after obtaining the third image, calculating the third image and the second image by using a preset target loss function to obtain a first target loss, the first target loss being used to indicate the difference between the third image and the second image. After obtaining the first target loss, the parameters of the initial model of the first super-resolution network can be updated based on the first target loss, and the initial model of the first super-resolution network with the updated parameters can be trained by using the next batch of training samples until a model training condition is met (for example, the first target loss reaches convergence, etc.), so as to obtain the trained first super-resolution network.

[0331] The first super-resolution network trained in this embodiment has the ability to improve the resolution of an input image. In the image processing process, the feature extraction is performed by using multiple cascaded multi-scale sub-modules with different scanning directions, which reduces the complexity of the system. The above scheme can effectively improve the peak signal-to-noise ratio without increasing the calculation amount and the parameter amount, and improves the accuracy of the high-resolution image reconstructed based on the low-resolution image.

[0332] In order to facilitate understanding of the above embodiments provided in the present application, the following points are explained.

[0333] (1) In embodiments of the present application, "indication" can include direct indication, indirect indication, display indication, and implicit indication. When describing that certain indication information is used to indicate A, it can be understood that the indication information carries A, directly indicates A, or indirectly indicates A.

[0334] In the present application, the information indicated by the indication information is referred to as to-be-indicated information. In the specific implementation process, there are many ways to indicate the to-be-indicated information, for example, but not limited to, the to-be-indicated information can be directly indicated, such as the to-be-indicated information itself or the index of the to-be-indicated information. The to-be-indicated information can also be indirectly indicated by indicating other information, where the other information and the to-be-indicated information have an association relationship. The to-be-indicated information can also be indicated only by a part, and the other part of the to-be-indicated information is known or agreed in advance. For example, the indication of a specific information can also be achieved by means of the arrangement order of each information agreed in advance (for example, the protocol stipulates), thereby reducing the indication overhead to a certain extent. In addition, the to-be-indicated information can be sent as a whole, or can be sent separately into multiple sub-information, and the sending period and / or sending time of these sub-information can be the same or different.

[0335] (2) In the present application, "sending" and "receiving" represent the direction of signal transmission. For example, "sending information to XX" can be understood as that the destination of the information is XX, which can include direct sending through the air interface, and also includes indirect sending through the air interface by other units or modules. "Receiving information from YY" can be understood as that the source of the information is YY, which can include direct receiving from YY through the air interface, and also includes indirect receiving from YY through the air interface from other units or modules. "Sending" can also be understood as the "output" of the chip interface, and "receiving" can also be understood as the "input" of the chip interface. In other words, sending and receiving can be carried out between devices, for example, between network devices and terminal devices, or can be carried out within a device, for example, between components, modules, chips, software modules or hardware modules within a device through a bus, wire or interface.

[0336] (3) In various embodiments of the present application, the terms and / or descriptions of different embodiments have consistency and can be mutually referenced if there is no special description and logical conflict. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0337] (4) In the present application, "first" and "second" are only for convenience of description, used to distinguish the objects, and not used to limit the scope of the embodiments of the present application. They are not used to describe the order or sequence of the characteristics. It should be understood that the objects thus described can be interchanged under appropriate circumstances, so as to describe schemes other than the embodiments of the present application.

[0338] (5) In this application, "predefined" can be realized by pre-storing corresponding codes, tables or other means for indicating relevant information in the device, and the specific implementation manner is not limited in this application.

[0339] (6) In this application, "example", "for example", "exemplarily", "as (another) example" and the like are used to represent as an example, illustration or description. Any embodiment or design scheme described as "example" in this application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes.

[0340] (7) In this application, "including", "containing", "having" and their variants mean "including but not limited to", unless otherwise specifically emphasized. "At least one" refers to one or more, and "multiple" refers to two or more.

[0341] (8) In this application, "and / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b and c can represent: a, or b, or c, or a and b, or a and c, or b and c, or a, b and c. Where a, b and c can be single or multiple.

[0342] (9) In various embodiments of the present application, the terms and / or descriptions of different embodiments are consistent and can be mutually referred to if there is no special description and no logical conflict. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0343] (10) Some optional features in various embodiments of the present application can not depend on other features in some scenarios, or can be combined with other features in some scenarios, without limitation.

[0344] The prediction method provided by the embodiments of the present application is described in detail above in combination with FIG. 1 to FIG. 15. The embodiments of the device of the present application will be described in detail below in combination with FIG. 16-FIG. 20. It should be understood that the description of the method embodiments and the description of the device embodiments correspond to each other, therefore, the parts not described in detail can be referred to the previous method embodiments.

[0345] FIG. 16 is a schematic block diagram of an image processing device 1600 provided by an embodiment of the present application.

[0346] The apparatus 1600 can be implemented by software, hardware or a combination of both. The apparatus 1600 provided in the embodiments of the present application can implement the method flow shown in FIG. 4 of the embodiments of the present application, and the apparatus 1600 comprises: an acquisition unit 1610, and a processing unit 1620.

[0347] The acquisition unit 1610 is configured to acquire an input image.

[0348] The processing unit 1620 is configured to perform the following operations: performing feature extraction on the input image to obtain first features; inputting the first features into a first super-resolution network to obtain second features; and reconstructing an output image based on the second features, wherein the resolution of the output image is higher than that of the input image; wherein the first super-resolution network comprises N2 groups of feature extraction modules, each group of feature extraction modules comprises N1 feature extraction modules, and each feature extraction module comprises N3 cascaded multi-scale sub-modules, the N3 cascaded multi-scale sub-modules are configured to sequentially perform feature extraction on input features according to different scanning directions, and N1, N2 and N3 are positive integers.

[0349] Optionally, the feature extraction module comprises four cascaded multi-scale sub-modules, and the four cascaded multi-scale sub-modules are configured to sequentially perform feature extraction on input features according to four different scanning directions; wherein the four different scanning directions comprise horizontal direction scanning, vertical direction scanning, reverse horizontal direction scanning and reverse vertical direction scanning.

[0350] Optionally, the multi-scale sub-module is configured to perform first processing on the input features based on a state space model (SSM) to obtain third features; perform downsampling and second processing on the input features based on the SSM to obtain fourth features; perform feature fusion on the third features and the fourth features to obtain fifth features; perform third processing on the input features based on a depth separable convolution attention (DSCA) to obtain sixth features; and determine the output features of the multi-scale sub-module according to the input features, the fifth features and the sixth features.

[0351] Optionally, the multi-scale sub-module is configured to: perform first processing on the input features based on a state space model (SSM) to obtain third features; perform downsampling and second processing on the input features based on the SSM to obtain fourth features; perform feature fusion on the third features and the fourth features to obtain fifth features; and determine the output features of the multi-scale sub-module according to the input features and the fifth features.

[0352] Optionally, the multi-scale sub-module is configured to: perform first processing based on a state space model (SSM) on the input feature to obtain a third feature; and determine the output feature of the multi-scale sub-module according to the input feature and the third feature.

[0353] Optionally, the determining the output feature of the multi-scale sub-module comprises: determining a seventh feature according to the input feature, the fifth feature and the sixth feature, and performing fourth processing based on a feed-forward network (FFN) on the seventh feature to obtain the output feature of the multi-scale sub-module; or determining a seventh feature according to the input feature and the fifth feature, and performing fourth processing based on a feed-forward network (FFN) on the seventh feature to obtain the output feature of the multi-scale sub-module; or determining a seventh feature according to the input feature and the third feature, and performing fourth processing based on a feed-forward network (FFN) on the seventh feature to obtain the output feature of the multi-scale sub-module.

[0354] Optionally, each feature extraction module group in the N2 feature extraction module groups is configured to: perform convolution on the output feature of the N1th feature extraction module in each feature extraction module group to obtain an eighth feature; and perform residual processing on the input feature of the first feature extraction module in each feature extraction module group and the eighth feature to obtain the output feature of each feature extraction module group.

[0355] Optionally, the first super-resolution network is configured to: perform convolution on the output feature of the N2th feature extraction module group in the first super-resolution network to obtain a ninth feature; and perform residual processing on the first feature and the ninth feature to obtain the second feature.

[0356] Optionally, the processing unit 1620 is further configured to divide a to-be-processed image into blocks to obtain the input image, the input image being any one of a plurality of input blocks obtained by dividing the to-be-processed image; after the output image is reconstructed, the method further comprises: splicing a plurality of output images corresponding to the plurality of input blocks into a target image.

[0357] FIG. 17 is a schematic block diagram of another image processing apparatus 1700 provided in an embodiment of the present application.

[0358] The apparatus 1700 can be implemented by software, hardware or a combination of both. The apparatus 1700 provided in an embodiment of the present application can implement the method flow shown in FIG. 15 of the present application, and the apparatus 1700 comprises: an acquisition unit 1710, and a training unit 1720.

[0359] The obtaining unit 1710 is configured to obtain an image sample pair, the image sample pair comprising a first image and a second image corresponding to the first image, the resolution of the first image being lower than the resolution of the second image.

[0360] The training unit 1720 is configured to train an initial model of a first super-resolution network to obtain the first super-resolution network according to the image sample pair.

[0361] Optionally, the training unit 1720 is further configured to train the initial model of the first super-resolution network to obtain the first super-resolution network according to the image sample pair, comprising: performing the first feature extraction on the first image to obtain first image input features; inputting the first image features into the initial model of the first super-resolution network to obtain first image output features; reconstructing a third image based on the first image output features; and adjusting parameters of the initial model of the first super-resolution network according to the third image and the second image to train the first super-resolution network.

[0362] The apparatus 1600 or the apparatus 1700 herein can be embodied in the form of functional modules. The term “unit” herein can be implemented in the form of software and / or hardware, and is not limited in a specific manner.

[0363] For example, the “unit” can be a software program, a hardware circuit, or a combination of both, which implements the above functions. The units in the apparatus 1600 will be described in detail below with reference to the apparatus 1800.

[0364] For example, the implementation of the obtaining unit 1610 in the apparatus 1600 will be described below. Similarly, the implementation of other units in the apparatus 1600, such as the processing unit 1620, can refer to the implementation of the obtaining unit 1610.

[0365] As an example of the software functional unit, the obtaining unit 1610 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the obtaining module 1010 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region (region), or can be distributed in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ), or can be distributed in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Generally, one region can include multiple AZs.

[0366] Likewise, the multiple hosts / virtual machines / containers for running the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Generally, one VPC is set in one region, and a communication gateway needs to be set in each VPC for cross-region communication between two VPCs in the same region or between VPCs in different regions, and the interconnection between VPCs is realized through the communication gateway.

[0367] The acquisition unit 1610 as an example of a hardware functional unit can include at least one computing device, such as a server, etc. Alternatively, the acquisition unit 1610 can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. The PLD can be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0368] The multiple computing devices included in the acquisition unit 1610 can be distributed in the same region or in different regions. The multiple computing devices included in the acquisition unit 1610 can be distributed in the same AZ or in different AZs. Likewise, the multiple computing devices included in the acquisition unit 1610 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs, etc.

[0369] Therefore, the modules of the various examples described in the embodiments of the present application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0370] It should be noted that the apparatus 1600 provided in the above examples is only used as an example for dividing the above functional units to execute the above method, and in actual application, the above functions can be completed by different functional units according to needs, that is, the internal structure of the apparatus 1600 is divided into different functional units to complete all or part of the above functions. For example, the acquisition unit 1610 can be used to execute any step in the above method, and the processing unit 1620 can be used to execute any step in the above method. The steps implemented by the acquisition unit 1610 and the processing unit 1620 can be specified as needed, and the functions of the above apparatus 1600 are implemented by the acquisition unit 1610 and the processing unit 1620 to implement different steps in the above method.

[0371] In addition, the apparatus and method embodiments provided in the above examples belong to the same concept, and the specific implementation process is described in the method embodiments above, which will not be repeated here.

[0372] The method provided in the embodiments of the present application can be executed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a processing unit, a memory, and a memory control unit, and the functions and structures of the hardware are described in detail later. The operating system is any one or more computer operating systems that implement business processing through processes, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a windows operating system. The application layer includes application programs such as a browser, an address book, word processing software, and instant messaging software. Optionally, the computer system is a handheld device such as a smartphone or a terminal device such as a personal computer, and the present application is not particularly limited as long as it can execute the method provided in the embodiments of the present application. The execution subject of the method provided in the embodiments of the present application can be a computing device, or a functional module in the computing device that can call and execute a program.

[0373] Next, a computing device provided in the embodiments of the present application is described in detail with reference to FIG. 18.

[0374] FIG. 18 is an architectural schematic diagram of a computing device 1800 provided in the embodiments of the present application.

[0375] The computing device 1800 can be a server or a computer or other device with computing capability. The computing device 1800 shown in FIG. 18 includes at least one processor 1810 and a memory 1820.

[0376] It should be understood that the present application does not limit the number of processors and memories in the computing device 1800.

[0377] The processor 1810 executes instructions in the memory 1820, so that the computing device 1800 implements the method provided by the present application. Alternatively, the processor 1810 executes instructions in the memory 1820, so that the computing device 1800 implements the functional modules provided by the present application, thereby implementing the method provided by the present application.

[0378] Optionally, the computing device 1800 further includes a communication interface 1830. The communication interface 1830 uses a transceiving module such as but not limited to a network interface card, a transceiver, to realize the communication between the computing device 1800 and other devices or communication networks.

[0379] Optionally, the computing device 1800 further includes a system bus 1840, wherein the processor 1810, the memory 1820 and the communication interface 1830 are connected with the system bus 1840 respectively. The processor 1810 can access the memory 1820 through the system bus 1840, for example, the processor 1810 can read and write data in the memory 1820 or execute code in the memory 1820 through the system bus 1840. The system bus 1840 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1840 is divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in FIG. 18, but it does not mean that there is only one bus or only one type of bus.

[0380] In one possible implementation, the function of the processor 1810 is mainly to interpret the instructions (or code) of the computer program and process the data in the computer software. The instructions of the computer program and the data in the computer software can be saved in the memory 1820 or the cache 1316.

[0381] Optionally, the processor 1810 is a chip that has a processing capability of signals. As an example but not limitation, the processor 1810 is a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Among them, the general processor is a microprocessor and the like. For example, the processor 1810 is a central processing unit (CPU).

[0382] Optionally, each processor 1810 includes at least one processing unit 1812 and a memory control unit 1814.

[0383] Optionally, the processing unit 1812 is also called core or kernel, which is the most important component of the processor. The processing unit 1812 is manufactured by single crystal silicon with a certain production process, and all the calculations, command reception, command storage and data processing of the processor are executed by the core. The processing units respectively independently run program instructions, and use the parallel computing capability to speed up the program running. Various processing units have fixed logic structures, for example, the processing unit includes logic units such as a first level cache, a second level cache, an execution unit, an instruction stage unit and a bus interface.

[0384] In one implementation, the memory control unit 1814 is used to control the data interaction between the memory 1820 and the processing unit 1812. Specifically, the memory control unit 1814 receives a memory access request from the processing unit 1812, and controls the access to the memory based on the memory access request. As an example but not limitation, the memory control unit is a memory management unit (MMU) and the like.

[0385] In one implementation, each memory control unit 1814 addresses the memory 1820 through a system bus. And an arbiter (not shown in Figure 18) is configured in the system bus, which is responsible for processing and coordinating the competitive access of multiple processing units 1812.

[0386] In one implementation, the processing unit 1812 and the memory control unit 1814 are communicatively connected through the internal connection line of the chip, such as the address line, so as to realize the communication between the processing unit 1812 and the memory control unit 1814.

[0387] Optionally, each processor 1810 also includes a cache 1816, which provides temporary storage to speed up the use of data. When the processor 1812 requires data, it first looks in the cache 1816 to find the required data. If the data is in the cache, the processor 1812 immediately accesses the data in the cache; otherwise, it accesses the data directly from memory. Because the cache 1816 is a smaller, faster version of memory, it speeds up the use of data by the processor 1812.

[0388] The memory 1820 can provide a space for processes in the computing device 1800 to run, for example, the memory 1820 stores a computer program (in particular, the code of the program) for generating a process. After the computer program is run by the processor to generate the process, the processor allocates a corresponding storage space in the memory 1820 for the process. Further, the above-mentioned storage space further includes a text segment, an initialized data segment, a bit initialized data segment, a stack segment, a heap segment, and the like. The memory 1820 stores data generated during the running of the process in the above-mentioned storage space of the process, for example, intermediate data, or process data, and the like.

[0389] Optionally, the memory is also referred to as the internal memory, which is used to temporarily store the data for operation in the processor 1810, and exchange the data with the external memory such as the hard disk. As long as the computer is running, the processor 1810 will call the data required for operation to the internal memory for operation, and the processing unit 1812 will transmit the result after the operation is completed.

[0390] By way of example, and not limitation, memory 1320 is volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. By way of example, and not limitation, nonvolatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. Volatile memory is random access memory (RAM), which acts as external cache. By way of example, and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). Proper selection of memory 1820 will depend on the particular system in which computer 1800 is used. Memory 1820 is intended to include, but is not limited to, these and any other suitable types of memory.

[0391] The structure of the computing device 1800 listed above is only an example, and the application is not limited thereto. The computing device 1300 of the embodiments of the application includes various hardware in the prior art computer system, for example, the computing device 1800 also includes other memories in addition to the memory 1820, such as disk storage and the like. Those skilled in the art should understand that the computing device 1800 can also include other devices necessary for normal operation. Meanwhile, according to specific needs, those skilled in the art should understand that the above-mentioned computing device 1800 can also include hardware devices for realizing other additional functions. In addition, those skilled in the art should understand that the above-mentioned computing device 1800 can also only include devices necessary for the embodiments of the application, and does not have to include all the devices shown in FIG. 18.

[0392] FIG. 19 is a schematic diagram of an architecture of a computing device cluster according to an embodiment of the application.

[0393] The computing device cluster includes at least one computing device. The computing device can be a server. In some embodiments, the computing device can also be a desktop computer, a notebook computer, or a terminal device such as a smart phone.

[0394] As shown in FIG. 19, the computing device cluster includes at least one computing device 1800. The same instructions for performing the above method can be stored in the memory 1820 of one or more computing devices 1800 in the computing device cluster.

[0395] In some possible implementations, the memory 1820 of one or more computing devices 1800 in the computing device cluster can also respectively store partial instructions for performing the above method. In other words, the combination of one or more computing devices 1800 can collectively execute the instructions of the above method.

[0396] It should be noted that the memories 1820 of different computing devices 1800 in the computing device cluster can store different instructions, respectively used to perform part of the functions of the above apparatus. That is, the instructions stored in the memories 1820 of different computing devices 1800 can implement the functions of one or more modules in the above apparatus.

[0397] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc.

[0398] FIG. 20 is a schematic diagram of the connection between computing devices 1800A and 1800B through a network according to an embodiment of the present application.

[0399] As shown in FIG. 20, two computing devices 1800A and 1800B are connected through a network. Specifically, the computing devices are connected to the network through the communication interfaces in the computing devices.

[0400] It should be understood that the functions of the computing device 1800A shown in FIG. 20 can also be completed by multiple computing devices 1800. Similarly, the functions of the computing device 1800B can also be completed by multiple computing devices 1800.

[0401] The present application also provides a computer program product containing instructions, which can be a software or program product containing instructions capable of running on a computing device or stored in any available medium. When it runs on the computing device, it causes the computing device to perform the above-provided method, or causes the computing device to implement the functions of the above-provided apparatus.

[0402] The application also provides a computer readable storage medium, which can be any available medium or data storage device including one or more available media that a computing device can store. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium includes instructions, which, when executed on a computing device, cause the computing device to perform the method provided above.

[0403] It should be understood that the size of the sequence number of each process described above in various embodiments of the application does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the application.

[0404] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0405] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0406] In several embodiments provided in the application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0407] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0408] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0409] The functions, if implemented in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various other media that can store program codes.

[0410] The above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which shall be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized by, The method comprises: obtaining an input image; performing feature extraction on the input image to obtain first features; inputting the first features into a first super-resolution network to obtain second features; reconstructing an output image based on the second features, the output image having a higher resolution than the input image; wherein the first super-resolution network comprises N2 groups of feature extraction modules, each group of feature extraction modules comprising N1 feature extraction modules, and each feature extraction module comprising N3 cascaded multi-scale sub-modules for sequentially performing feature extraction on input features in different scanning directions, N1, N2 and N3 being positive integers.

2. The method of claim 1, wherein, The feature extraction module comprises four cascaded multi-scale sub-modules for sequentially performing feature extraction on input features in four different scanning directions. The four different scanning directions comprise horizontal direction scanning, vertical direction scanning, reverse horizontal direction scanning and reverse vertical direction scanning.

3. The method according to claim 1 or 2, characterized in that, The multi-scale sub-module is configured to: perform first processing on the input features based on a state space model (SSM) to obtain third features; perform down-sampling and second processing on the input features based on the SSM to obtain fourth features; perform feature fusion on the third features and the fourth features to obtain fifth features; perform third processing on the input features based on a deep separable convolution attention (DSCA) to obtain sixth features; determine output features of the multi-scale sub-module based on the input features, the fifth features and the sixth features.

4. The method according to claim 1 or 2, characterized in that, The multi-scale sub-module is configured to: perform first processing on the input features based on a state space model (SSM) to obtain third features; perform down-sampling and second processing on the input features based on the SSM to obtain fourth features; perform feature fusion on the third features and the fourth features to obtain fifth features; determine output features of the multi-scale sub-module based on the input features and the fifth features.

5. The method according to claim 1 or 2, characterized in that, The multi-scale sub-module is configured to: perform first processing on the input features based on a state space model (SSM) to obtain third features; determine output features of the multi-scale sub-module based on the input features and the third features.

6. The method according to any one of claims 3 to 5, characterized in that, The determination of the output features of the multi-scale sub-module comprises: determining seventh features based on the input features, the fifth features and the sixth features, and performing fourth processing on the seventh features based on a feed-forward network (FFN) to obtain the output features of the multi-scale sub-module; or, determining seventh features based on the input features and the fifth features, and performing fourth processing on the seventh features based on a feed-forward network (FFN) to obtain the output features of the multi-scale sub-module; or, determining seventh features based on the input features and the third features, and performing fourth processing on the seventh features based on a feed-forward network (FFN) to obtain the output features of the multi-scale sub-module.

7. The method according to any one of claims 1 to 6, characterized in that, Each of the N2 feature extraction module groups is configured to perform convolution on an output feature of an N1th feature extraction module in each feature extraction module group to obtain an eighth feature. The input feature of the first feature extraction module in each feature extraction module group and the eighth feature are subjected to residual processing to obtain an output feature of each feature extraction module group.

8. The method according to any one of claims 1 to 7, characterized in that, The first super-resolution network is configured to perform convolution on an output feature of an N2th feature extraction module group in the first super-resolution network to obtain a ninth feature. The first feature and the ninth feature are subjected to residual processing to obtain the second feature.

9. The method of any one of claims 1 to 8, wherein, Before the input image is obtained, the method further includes: dividing a to-be-processed image to obtain the input image, the input image being any one of a plurality of input image patches obtained by dividing the to-be-processed image. After the output image is obtained, the method further includes: splicing a plurality of output images corresponding to the plurality of input image patches into a target image.

10. The method according to any one of claims 1 to 9, characterized in that, The method further includes: The first super-resolution network is trained according to an image sample pair, the image sample pair including a first image and a second image corresponding to the first image, the resolution of the first image being lower than the resolution of the second image.

11. The method of claim 10, wherein, The first super-resolution network is trained according to the following manner: The first feature extraction is performed on the first image to obtain a first image input feature; The first image input feature is input into an initial model of the first super-resolution network to obtain a first image output feature; Based on the first image output feature, a third image is reconstructed; Parameters of the initial model of the first super-resolution network are adjusted according to the third image and the second image to train the first super-resolution network.

12. An image processing apparatus characterized by comprising: includes: An acquisition unit is configured to acquire an input image; A processing unit is configured to perform the following operations: performing feature extraction on the input image to obtain a first feature; inputting the first feature into a first super-resolution network to obtain a second feature; based on the second feature, reconstructing an output image, the resolution of the output image being higher than the resolution of the input image; The first super-resolution network includes N2 feature extraction module groups, each feature extraction module group including N1 feature extraction modules, each feature extraction module including N3 cascaded multi-scale sub-modules, the N3 cascaded multi-scale sub-modules being configured to sequentially perform feature extraction on an input feature according to different scanning directions, N1, N2 and N3 being positive integers.

13. The apparatus of claim 12, wherein, The feature extraction module includes four cascaded multi-scale sub-modules, the four cascaded multi-scale sub-modules being configured to sequentially perform feature extraction on an input feature according to four different scanning directions. The four different scanning directions include horizontal direction scanning, vertical direction scanning, reverse horizontal direction scanning and reverse vertical direction scanning.

14. The apparatus of claim 13, wherein, The multi-scale sub-module is configured to: perform first processing based on a state space model (SSM) on the input feature to obtain a third feature; perform down-sampling and second processing based on the SSM on the input feature to obtain a fourth feature; characteristic fusion is performed on the third feature and the fourth feature to obtain a fifth feature; a third processing based on a deep separable convolution attention (DSCA) is performed on the input feature to obtain a sixth feature; an output feature of a multi-scale sub-module is determined according to the input feature, the fifth feature and the sixth feature.

15. An image processing apparatus characterized by comprising: comprising: a memory for storing a computer program; a processor for executing the computer program stored in the memory to enable the apparatus to perform the method according to any one of claims 1 to 11.

16. An image processing system, characterized by an image processing apparatus according to any one of claims 12 to 14.

17. A computer-readable storage medium, characterized in that, a computer program stored thereon, which, when executed by a computer, enables the method according to any one of claims 1 to 11 to be implemented.

18. A chip or chip system, characterized by comprising: circuitry for performing the method according to any one of claims 1 to 11.

19. A computer program product, characterised in that, the computer program product comprises computer program code which, when executed on a computer, enables the method according to any one of claims 1 to 11 to be implemented.

Images