Super-division model generation method, image super-division method and related equipment

Abstract

The embodiment of the invention provides a super-division model generation method, an image super-division method and related equipment, and belongs to the technical field of deep learning. The super-division model generation method comprises the following steps: acquiring a training data set and an initial image super-division network model; wherein the initial image super-division network model is established based on an over-parameterized row and column convolution module; training the initial image super-division network model by adopting the training data set; and carrying out structural re-parameterization processing on the trained image super-division network model to generate an image super-division network model for reasoning. The image super-division network model provided by the embodiment of the invention can efficiently run based on a small amount of calculation and memory resources in a reasoning stage, and further can run in real time on middle and low-end edge equipment with insufficient calculation and memory resources.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a method for generating a super-resolution model, an image super-resolution method, and related equipment. Background Art

[0002] With the rapid development of deep learning technology, image super-resolution technology based on deep learning has also been widely used in various fields. For example, in medical image processing, film and television production and other fields, image super-resolution network models are used to reconstruct low-resolution images into high-resolution images.

[0003] Early image super-resolution network models typically required transmitting data to a server for computational processing, and then returning the results to the user's edge device. As edge device hardware performance continues to improve, image super-resolution network models are also beginning to run on edge devices.

[0004] Existing image super-resolution network models are highly complex and require significant computational resources and memory space. They typically only run efficiently on high-end edge devices with ample computational and memory resources, but are inefficient on mid-range and low-end devices. For example, the naive convolution-based ECBSR model cannot run in real time on mid-range and low-end edge devices. Summary of the Invention

[0005] The embodiments of the present application provide a method for generating a super-resolution model, an image super-resolution method, and related devices, which can solve the technical problem of low efficiency of image super-resolution network models. To achieve the above objectives, the technical solutions provided by the embodiments of the present application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for generating an image super-resolution network model, the method comprising:

[0007] Obtaining a training data set and an initial image super-resolution network model; wherein the initial image super-resolution network model is constructed based on one or more over-parameterized row-column convolution modules, each of the over-parameterized row-column convolution modules including an over-parameterized row convolution module and an over-parameterized column convolution module;

[0008] Use the training data set to train the initial image super-resolution network model;

[0009] The trained image super-resolution network model is subjected to structural reparameterization processing to generate an image super-resolution network model for inference; wherein the row-column convolution module contained in the image super-resolution network model for inference and the corresponding over-parameterized row-column convolution module have equivalent convolution weights.

[0010] In a second aspect, an embodiment of the present application provides an image super-resolution reconstruction method, the reconstruction method comprising: inputting a low-resolution image into the image super-resolution network model for inference as described in the first aspect, to obtain a high-resolution image output by the image super-resolution network model;

[0011] The image super-resolution network model for inference includes one or more structurally reparameterized row-column convolution modules; each of the structurally reparameterized row-column convolution modules has an equivalent convolution weight to the corresponding over-parameterized row-column convolution module contained in the trained image super-resolution network model.

[0012] In a third aspect, an embodiment of the present application provides an electronic device comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method for generating an image super-resolution network model as described in the first aspect.

[0013] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the image super-resolution reconstruction method as described in the second aspect.

[0014] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executed by the method for generating an image super-resolution network model as described in the first aspect.

[0015] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the image super-resolution reconstruction method as described in the second aspect.

[0016] In an embodiment of the present application, during the training phase of the image super-resolution network model, an over-parameterized row-column convolution module is used. On the one hand, the row-column convolution module can reduce model parameters compared to the naive convolution module, thereby improving the model's operating efficiency; on the other hand, over-parameterization increases the number of model parameters, which can improve the model's characterization capabilities. During the inference phase of the image super-resolution network model, the trained image super-resolution network model can be structurally re-parameterized, the model's parameters can be equivalently converted, and the model structure can be simplified. In this way, the number of parameters and the amount of calculation during model inference can be reduced, thereby improving the model's inference efficiency. Furthermore, the image super-resolution network model provided in this embodiment can run efficiently based on a small amount of computing and memory resources during the inference phase, thereby achieving real-time operation on mid- and low-end edge devices with insufficient computing and memory resources.

[0017] In addition, when building an image super-resolution network model, the model complexity can be increased by increasing the number of convolution layers in the over-parameterized row and column convolution module, achieving model over-parameterization and thereby improving the model's representational capabilities. Furthermore, the over-parameterized row and column convolution module can adopt a multi-branch structure, with multiple convolution kernels used in parallel, allowing the model to introduce more nonlinear characteristics and further improving the model's representational capabilities. Optionally, multiple row and column convolution modules of different lengths can be stacked. Compared to stacking multiple naive convolution modules, this can reduce the model's computational complexity while achieving a larger receptive field. Optionally, before performing a convolution operation on the low-resolution image input to the model, it can be downsampled. By appropriately reducing the input image size, the computational complexity of subsequent convolution operations can be reduced while ensuring model performance. Optionally, regularization can be introduced in each row and column convolution module of the model to reduce the correlation between weight parameters between different channels, thereby allowing the model to extract richer feature information and enhancing the module's representational capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 A schematic diagram of a flow chart of a method for generating an image super-resolution network model provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a process for building an initial image super-resolution network model provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of the structure of an over-parameterized row-column convolution module provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of the structure of an image super-resolution network model provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of the structure of another image super-resolution network model provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of a convolution operation of two stacked naive convolution modules provided in an embodiment of the present application;

[0025] Figure 7A A schematic diagram of a convolution operation of two stacked row and column convolution modules provided in an embodiment of the present application;

[0026] Figure 7B A schematic diagram of another convolution operation of stacking two row and column convolution modules provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of the principle of converting an over-parameterized row-column convolution module into a single row-column convolution module provided by an embodiment of the present application;

[0028] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] Single-image super-resolution (SR) is the process of generating a high-resolution (HR) image from a single, degraded low-resolution (LR) image. There is no unique solution for reconstructing an HR image from an LR image, meaning that super-resolution is ill-posed. This ill-posed problem was initially addressed using interpolation methods. However, with the advent of deep learning, super-resolution is now commonly achieved using deep neural networks.

[0030] Image super-resolution assumes that LR images are obtained through two main degradation processes: blurring and downsampling. This can be expressed as: y = (x*k)↓s, where * represents the convolution operation between the LR image and the blur kernel, and ↓s is the s-fold downsampling operation with a downsampling factor of ×s. Most SR methods are built around the bicubic model with various scaling factors (e.g., ×2, ×3, ×4, ×8, etc.).

[0031] With the rapid development of deep learning technology, deep learning-based image super-resolution techniques have also been widely applied in various fields. For example, in the field of medical imaging, low-resolution X-rays can be reconstructed into high-resolution images based on deep neural network models for image super-resolution tasks (i.e., image super-resolution network models), facilitating more accurate diagnosis and treatment. Another example is that in the field of film and television production, image super-resolution networks can be used to improve the clarity and quality of captured images, thereby enhancing the user experience.

[0032] Advances in hardware technology have facilitated the training of larger and deeper neural networks for image super-resolution, significantly improving the performance of image super-resolution network models. However, these breakthroughs often also lead to the introduction of more complex network models, reducing network efficiency. Among them, the efficiency metrics of deep neural networks can include runtime, number of parameters, computational complexity (floating point operations, abbreviated as FLOPs), number of activation layers, and memory consumption. These metrics affect the deployment of deep neural networks in various ways. Among them, runtime is a relatively direct indicator of the efficiency of deep neural networks and can be used as the main indicator for evaluating the efficiency of deep neural networks. Since Shi et al. pioneered ESPCN (Efficient Sub-Pixel Convolutional Neural Network), designing efficient deep neural networks for single image super-resolution has become crucial.

[0033] Currently, advanced and efficient deep neural networks for super-resolution (SR) tasks include RFDN (residual feature distillation network), ECBSR, RLFN (residual local feature network), BSRN (blueprint separable residual network), and shuffleSR. Many advanced SR methods focus on reducing model parameters and FLOPs. For example, replacing naive convolution with depthwise separable convolution and blueprint separable convolution reduces model parameters and computation to a certain extent. However, in actual tests, these non-intensive convolution methods do not necessarily lead to faster execution speeds, and their performance is often inferior to that of intensive naive convolution. Furthermore, high-performing image super-resolution network models are often highly complex and require significant computational resources and memory space. As a result, these image super-resolution network models typically only run efficiently on high-end edge devices with ample computational and memory resources, and are inefficient on mid-range and low-end edge devices. For example, the ECBSR model uses a naive convolutional network structure that is friendly to edge deployment. However, it can generally only achieve real-time operation on advanced parallel computing units (such as GPUs and NPUs) on a few high-end flagship edge devices, and cannot run in real time on most mid-to-low-end edge devices. The SRLUT (Super-Resolution Using Look-Up Table) algorithm and the EdgeSR algorithm can also run in real time on some edge devices, but the performance is poor. Edge devices can be understood as user-end devices, such as personal computers, mobile phones, and other electronic devices.

[0034] Based on this, the embodiment of the present application proposes a method for generating an image super-resolution network model. The image super-resolution network model provided by this method is constructed using row-column convolution, which can reduce the number of parameters compared to naive convolution. In addition, the over-parameterization technology is applied in the model training stage, which can increase the number of parameters of the model and improve the representation ability of the model; and in the model reasoning stage, the structural re-parameterization technology is applied to perform equivalent conversion on the parameters of the model, simplify the model structure, and thereby reduce the number of parameters and the amount of calculation during model reasoning, thereby improving the efficiency of model reasoning. Therefore, the image super-resolution network model provided by the embodiment of the present application not only has good performance during reasoning, but can also run efficiently based on a small amount of computing and memory resources, thereby achieving real-time operation on mid- and low-end edge devices.

[0035] In order to make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

[0036] It should be clear that the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0037] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0038] In the description of this application, it should be understood that the terms "first" and "second" etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances. In addition, in the description of this application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0039] See also Figure 1 The method for generating an image super-resolution network model provided in this embodiment may specifically include the following steps:

[0040] S102, obtaining a training data set and an initial image super-resolution network model.

[0041] In implementation, an initial image super-resolution network model can be built based on an over-parameterized row-column convolution module, and a training dataset for training the image super-resolution network model can be constructed.

[0042] In one embodiment, the training dataset may be an image dataset such as Flickr2K, OST, and Div2k.

[0043] In another embodiment, in order to increase the number of training samples, a training dataset can also be reconstructed based on an existing image dataset. Accordingly, before step S102, the method for generating an image super-resolution network model can also include: obtaining a first image pair from a preset image dataset; wherein the first image pair includes a first image and a second image having different resolutions; based on corresponding cropping sizes and step sizes, cropping the first image and the second image at corresponding positions to obtain multiple second image pairs; and constructing a training dataset based on the multiple second image pairs.

[0044] In implementation, a high-resolution image (i.e., HR image) and a corresponding low-resolution image (i.e., LR image) can be first acquired from a preset image dataset to obtain an LR-HR image pair. For ease of description, the originally acquired LR-HR image can be referred to as the first image pair, and the LR-HR image constructed based on the first image pair can be referred to as the second image pair.

[0045] Taking the Div2k dataset as an example, the training images provided by the Div2k dataset include 800 HR images and their corresponding LR images. 800 LR-HR image pairs are too few, and directly training an image super-resolution network model based on this data will not yield good training results. To obtain an appropriate number of training samples, you can set appropriate cropping sizes and strides for both HR and LR images. For example, when training an image super-resolution network model that requires scaling the image height and width by a factor of 2, you can set the crop size of the HR images to 480×480 and the stride to 240 pixels. Similarly, you can set the crop size of the LR images to 240×240 and the stride to 120 pixels. This way, each HR image will be cropped into a 480×480 pixel image, cropped every 240 pixels, and each LR image will be cropped into a 240×240 pixel image, cropped every 120 pixels. After the above cropping steps, a larger number of LR-HR image pairs can be constructed for training the image super-resolution network model.

[0046] In the implementation, the image super-resolution network model can be built first to obtain the initial image super-resolution network model. Figure 2, build the initial image super-resolution network model, which may include the following steps:

[0047] S202, construct an over-parameterized row and column convolution module.

[0048] In implementation, the over-parameterized row-column convolution (OPRCConv) module can be composed of over-parameterized row convolution (OPRCConv) and over-parameterized column convolution (OPCConv). This application uses a row-column convolution module, which can reduce the number of model parameters and computational complexity compared to naive convolution.

[0049] It can be understood that in a naive convolution with a receptive field of k×k, each pixel in the input feature map is convolved with the convolution kernel k×k to obtain the output feature map. A row-column convolution with a receptive field of k×k can include a 1×k row convolution and a k×1 column convolution. During calculation, a one-dimensional row convolution is first performed on the input feature map to obtain an intermediate feature map, and then a one-dimensional column convolution is performed on the intermediate feature map to obtain the output feature map. Therefore, row-column convolution can be regarded as a dimensionality reduction process for the input feature map, thereby reducing the computational complexity of subsequent convolution operations.

[0050] In one embodiment, the computational cost of a naive convolution in FLOPs can be determined by referring to the following formula:

[0051] FLOPs = 2C in ×H in ×W in ×K 2 ×C out ÷S 2 (1)

[0052] The computational amount of FLOPs for a row-column convolution can be determined by referring to the following formula:

[0053] FLOPs = 2C in ×H in ×W in ×(K+K) ×C out ÷S 2 (2)

[0054] Among them, C in is the number of channels of the input feature map; H in is the height of the input feature map; W inis the width of the input feature map; K is the size of the convolution kernel; C out is the number of channels of the output feature map; S is the stride of the convolution.

[0055] Referring to formula (1) and formula (2), the ratio of the computational complexity of row-column convolution to that of naive convolution is 2 / K. Therefore, for naive convolution and row-column convolution with the same receptive field of k×k, the computational complexity of row-column convolution is only 2 / K of the computational complexity of naive convolution. k can also be selected as any value based on any one or more factors such as image size, complexity of super-resolution task, size of computing resources, etc., and this application does not impose any restrictions on this. Among them, k is often taken as 3, 5, 7, etc. At this time, 2 / K<1, and the computational complexity of row-column convolution is significantly reduced compared to naive convolution.

[0056] In one embodiment, the row convolution module and the column convolution module in the over-parameterized row-column convolution module respectively include multi-layer convolution processing.

[0057] In practice, the model representation capability can be improved by increasing the number of convolution layers in the over-parameterized row and column convolution modules. Specifically, multiple 1×1 convolutions are added to each row and column convolution module. In this way, each convolution layer can learn different feature representations, and the weights of each convolution layer can be regarded as learnable parameters, thereby enhancing the feature extraction capability of the convolution module. For example, Figure 3 The over-parameterized row convolution module and the over-parameterized column convolution module shown include three layers of convolution operations. Of course, the number of convolution layers can also be any value such as 2, 4, 5, etc., and this application does not impose any limitation on this.

[0058] In one embodiment, in the over-parameterized row and column convolution modules, the multi-layer convolution in the over-parameterized row convolution module and the over-parameterized column convolution module are constructed using a multi-branch structure. In which, multiple convolution kernels can be used in parallel. For example, see Figure 3 The over-parameterized row convolution module and the over-parameterized column convolution module each adopt a dual-branch structure, using a 1×1 convolution in parallel with a 1×k row convolution or a k×1 column convolution. This allows the model to introduce more nonlinear characteristics and further improve the model's representation ability. Of course, the number of convolution branches can also be any value such as 3, 4, or 5, and this application does not impose any restrictions on this.

[0059] It is worth mentioning that the row convolution layer and column convolution layer in the over-parameterized row-column convolution module can each be followed by an activation layer. Common activation functions can include PReLU (Parametric Rectified Linear Unit) activation function, GLU (Gated Linear Unit) activation function, ReLU (Rectified Linear Unit) activation function, etc. This application does not restrict the type of activation function.

[0060] In one embodiment, see Figure 3 , the initially constructed over-parameterized row convolution module can include 3 1×1 convolutions, 1 1×k convolution, and an activation function. Moreover, one of the 1×1 convolutions and the 1×k row convolution form a dual-branch optimization direction. When constructing the over-parameterized row convolution module, the dimension of the first 1×1 convolution in the module can be defined as (C mid ,C in ,1,1), define the dimension of the second 1×1 convolution as (C out ,C mid ,1,1), respectively define the dimensions of the 1×1 convolution branch as (C out ,C out ,1,1) and the dimension of the 1×3 row convolution branch is (C out ,C out ,1,3), and finally define the PReLU activation function.

[0061] Correspondingly, the initially constructed over-parameterized column convolution module can also use three 1×1 convolutions. When defining the initially constructed over-parameterized column convolution module, the dimension of the first 1×1 convolution in the module can be defined as (C mid ,C in ,1,1), define the dimension of the second 1×1 convolution as (C out ,C mid ,1,1), respectively define 1×1 convolution branches (C out ,C out ,1,1) and 3×1 column convolution branches (C out ,C out ,3,1), and finally define the PReLU activation function.

[0062] Among them, each of the above convolutions can be described as 4 dimensions, which represent the number of output channels (i.e. the number of convolution kernels), the number of input channels, and the height and width of the convolution kernel. In addition, the output of the previous convolution will be used as the input of the next convolution. For example, the number of output channels of the second convolution is C out , the number of input channels of the second convolution is equal to the number of output channels of the first convolution C mid, the size of the second convolution is 1×1.

[0063] It is worth mentioning that in Figure 3 In the illustrated embodiment, the value of k is generally any odd number, such as 3, 5, 7, 9, etc. The better the performance of the computing device, the larger the value can be.

[0064] S204: Use the over-parameterized row and column convolution module to build an initial image super-resolution network model.

[0065] In implementation, after defining the over-parameterized row-column convolution, the over-parameterized row-column convolution module can be used to build an image super-resolution network model. Figure 4 The low-resolution image input to the initially constructed image super-resolution network model can be sequentially passed through the first convolution module (denoted as conv first in the figure), the over-parameterized row-column convolution module (denoted as OPRCConv in the figure), and the last convolution module (denoted as conv last in the figure) to obtain a reconstructed feature layer. The reconstructed feature layer can then be upsampled by the pixel shuffle operation and added to the output of the nearest neighbor difference module (denoted as nearest in the figure) to obtain the final high-resolution output image. The upsampling factor in the pixel shuffle operation can determine the image magnification ratio. For example, an upsampling factor of 2 means that the height and width of the image are each expanded by 2 times, and the overall image is magnified by 4 times.

[0066] In one embodiment, the initial image super-resolution network model can be obtained by stacking multiple over-parameterized row-column convolution modules. By increasing the number of over-parameterized row-column convolution stacking layers, the model can better adapt to the changes and complexity in the input data, thereby improving the model's representation ability and generalization performance. Figure 5 The initial image super-resolution network model includes m over-parameterized row and column convolution modules. The value of m can be determined based on the computing power of the computing device. For example, if the model is applied to a computing device with average computing power for inference, the value of m can be preset to a smaller value such as 2 or 3. If the model is applied to a computing device with higher computing power for inference, the value of m can be preset to 5, 6, or a larger value. This application does not impose any restrictions on the value of m.

[0067] In one embodiment, the LR image input to the model can be directly convolved using various convolution modules, and then added to the upsampled image obtained through nearest neighbor interpolation after passing through the pixel shuffle module to obtain the final output HR image. If the input LR image has a height of H and a width of W, the final output HR image will have a size of rH × rW, where r is the magnification factor.

[0068] In another embodiment, a pixel unshuffle module may be provided before the first convolution module of the initial image super-resolution network model to downsample the input LR image. Accordingly, step S104 may specifically include: obtaining a first image for input to the image super-resolution network model from the training dataset, downsampling the first image; and training the initial image super-resolution network model based on the downsampled first image.

[0069] In implementation, see again Figure 5 , the LR image of the input image super-resolution network model can be downsampled based on the pixel unshuffle module, which can reduce the size of the input image so that subsequent convolution calculations can be calculated at a lower resolution, reducing the amount of model calculations. Afterwards, the first convolution module is used to extract and expand the feature layer obtained by downsampling, and the middle m OPRCConv modules are used to extract and refine the feature layer output by the previous convolution layer. The last convolution module and the pixel shuffle module are used together to reconstruct the residual features, and then the elements are added to the upsampled image obtained by nearest neighbor interpolation to obtain the final HR image. Taking the convolution kernel size of the first convolution module and the last convolution module as an example, both are 3×3, assuming that the input image is a single-channel grayscale image with a height of h and a width of w, the input image can be recorded as: I∈R 1×h×w In another embodiment, if the input image is a three-channel color image, it can be written as I∈R 3×h×w Image I∈R 1×h×w After the first layer of pixel unshuffle module, the image resolution can be reduced and the number of image channels can be increased to obtain the feature layer F(1,4,h1,w1). In the feature layer F(1,4,h1,w1), the first dimension is the batch size (i.e., batch size), that is, the number of data samples processed simultaneously. In the training phase, it can be any positive integer such as 1, 4, 16, and is fixed to 1 in the inference phase; the second dimension is the number of channels, which is equal to the square of the downsampling factor. For example, if 2x downsampling is used, the number of channels is 4 times the original number of channels; the third dimension is the height of the feature map; the third dimension is the width of the feature map. After the feature layer F(1,4,h1,w1) passes through the first convolution module (C f ,4,3,3), and then through m over-parameterized row and column convolution modules, the reconstructed feature layer F(1,C f ,h2,w2), and finally passes through the last convolution module (16,C f,3,3) to obtain the final feature layer F(1,16,h3,w3). Among them, the four dimensions in each convolution module represent the number of output channels, the number of input channels, and the height and width of the convolution kernel. In addition, the output of the previous convolution will be used as the input of the next convolution. The final feature layer F(1,16,h3,w3) is added to the upsampled image obtained by the nearest neighbor interpolation (denoted as nearest in the figure) after the pixel shuffle×4 module (i.e., 4 times upsampling) to obtain the final output image I out ∈R 1×hr×wr , where r is the magnification, which is 2 in this embodiment.

[0070] The magnification of the final HR image output is equal to the pixel unshuffle module's reduction factor multiplied by the pixel shuffle module's magnification factor. For example, when the pixel unshuffle module uses a 2x downsampling, the input model's LR image size becomes 1 / 2H × 1 / 2W. At this point, if you want to obtain an image super-resolution network model with a magnification factor of r, that is, to output an HR image of size rH × rW, the pixel shuffle module's magnification factor can be set to 2r. For example, if the pixel unshuffle module uses a 2x downsampling and the pixel shuffle module uses a 4x upsampling, the output image's magnification factor r compared to the input image is 2, meaning the output image is magnified to twice the input image in both height and width.

[0071] It should be noted that a larger downsampling factor in the pixel unshuffle module reduces the computational overhead of subsequent convolution operations, but may also result in greater information loss. Therefore, to balance computational overhead and model performance, you can choose whether to deploy the pixel unshuffle module and set an appropriate downsampling factor for it.

[0072] It is worth mentioning that the downsampling process will reduce the resolution of the feature map to a certain extent, and may lose some detail information that is important for reconstructing high-resolution images. In some embodiments, the pixel unshuffle module adopts 2x downsampling, combined with the over-parameterized row and column convolution module provided by this application, which can almost ensure information losslessness and significantly reduce the computational complexity of subsequent convolution operations, further improving the operating efficiency of the image super-resolution network model on edge devices. Therefore, setting the downsampling multiple of the pixel unshuffle module to 2 is a preferred implementation.

[0073] In one embodiment, when an image super-resolution network model includes multiple over-parameterized row-column convolution modules, each over-parameterized row-column convolution module can use row-column convolution with the same kernel size. When the number of convolution layers is the same, stacking multiple row-column convolutions can reduce the number of parameters while maintaining the same receptive field, compared to stacking multiple naive convolutions.

[0074] For example, taking the stacking of two convolutional layers as an example, for the stacking of two 3×3 naive convolutions, see Figure 6 , whose theoretical receptive field size is 5×5. Stacking two row-column convolutions with kernel sizes of 1×3 and 3×1, see Figure 7A Its theoretical receptive field is the same as that of a stack of two 3×3 naive convolutions, also 5×5. However, compared to a stack of two naive convolutions, the number of parameters in the former is 2 / 3, a reduction of 1 / 3. Furthermore, the computational complexity of convolution operations using multiple layers of row and column convolutions is also reduced accordingly.

[0075] In another embodiment, each over-parameterized row and column convolution module can also use row and column convolutions with different convolution kernel sizes. In other words, the image super-resolution network model provided in this embodiment can use a stack of multiple row and column convolutions with different convolution kernel sizes. In this way, when the number of convolution layers is the same, using multiple row and column convolution stacks can achieve a larger receptive field while reducing the amount of computation compared to using multiple naive convolution stacks.

[0076] For example, for a stack of two 3×3 naive convolutions, see again Figure 6 , its theoretical receptive field size is 5×5. And stacking a row-column convolution with kernel sizes of 1×3 and 5×1 and a row-column convolution with kernel sizes of 1×5 and 3×1, respectively, can be seen in Figure 7B , and its theoretical receptive field is 7×7. Clearly, the receptive field obtained by stacking a row-column convolution with kernel sizes of 1×3 and 5×1 and a row-column convolution with kernel sizes of 1×5 and 3×1 is larger than the receptive field obtained by stacking two 3×3 naive convolutions. Furthermore, the parameter count of the former is 8 / 9 of the parameter count of the latter, a reduction of 1 / 9. In other words, stacking multiple over-parameterized row-column convolutions with different kernel sizes can achieve a larger receptive field while reducing the number of parameters and computation compared to stacking multiple naive convolutions.

[0077] It is worth noting that the size of the convolution kernel is generally an odd number. Of course, in some embodiments, an even number may also be used, and this application does not limit this.

[0078] S104: Using the training data set to train the initial image super-resolution network model.

[0079] In practice, the initial image super-resolution network model can include an over-parameterized row-column convolution module. Input image data obtained from the training dataset, i.e., the LR image, is fed into the image super-resolution network model. The over-parameterized row-column convolution module can fully extract image features and efficiently reconstruct the HR image.

[0080] by Figure 3 For example, an over-parameterized row convolution module consists of three 1×1 convolutions, one 1×k convolution and an activation function PReLU. Its calculation process can be expressed as formula (3):

[0081] F out =σ((W 1×k +W 1×1 )*W 1×1 *W 1×1 *F in ) (3)

[0082] Among them, F in Represents the input feature layer of the row convolution module, F out represents the output feature layer, W 1x1 represents 1×1 convolution, W 1×k represents 1×k row convolution, and σ represents the activation function.

[0083] Similarly, the calculation process of over-parameterized column convolution can be expressed as formula (4):

[0084] F out =σ((W k×1 +W 1×1 )*W 1×1 *W 1x1 *F in ) (4)

[0085] Among them, F in represents the input feature layer of the column convolution module, F out represents the output feature layer, W 1x1 represents 1×1 convolution, W k×1 represents k×1 column convolution, and σ represents the activation function.

[0086] In this embodiment, the over-parameterized row-column convolution in the training phase has stronger representation ability than a single row-column convolution by learning excessive weight parameters and the optimization direction of the two branches, which can significantly improve the network performance.

[0087] In one embodiment, to enhance the representation capability of the module, regularization can be introduced to constrain the first 1×1 convolution in the over-parameterized row-column convolution. Accordingly, step S104 can specifically include: training the initial image super-resolution network model multiple times based on a loss function with a regularization term added.

[0088] In practice, regularization can be used to prevent model overfitting. Typically, a regularization term is added to the model's loss function to penalize model complexity, helping the model learn simpler, more generalizable features. For example, the introduction of orthogonal regularization allows the module to reduce the correlation between weight parameters across different numbers of channels, allowing it to extract richer feature information and enhance its representational capabilities.

[0089] In one embodiment, when orthogonal regularization is used to constrain the model, the calculation of the regularization term can refer to formula (5)

[0090]

[0091] Among them, W 1×1 represents 1×1 convolution, represents the transpose of 1×1 convolution, I represents the identity matrix, ‖·‖ F represents the F-norm.

[0092] Correspondingly, during the model training phase, the loss function uses L1 loss plus regularization loss L ortho The calculation of the total loss function can refer to formula (6):

[0093] L=L1+L ortho (6)

[0094] Among them, the L1 loss (Mean Absolute Error, MAE) minimizes the sum of the absolute differences between the target value and the predicted value. Of course, in practical applications, an appropriate loss function can be selected based on the specific task characteristics and data characteristics, such as the L2 loss function (Mean Squared Error, MSE), etc. This application does not limit this.

[0095] It is worth mentioning that in addition to using orthogonal regularization to constrain the model, regularization methods such as L1 regularization and L2 regularization can also be used to constrain the model, and this application does not impose any restrictions on this. Among them, L1 regularization reduces the size of parameters by adding the sum of the absolute values ​​of the parameters to the objective function as a regularization term, which can prompt the model to produce a sparse solution, that is, setting some parameters to zero, thereby achieving the effect of feature selection, reducing the complexity of the model, and preventing the model from overfitting. L2 regularization reduces the size of parameters by adding the sum of the squares of the parameters to the objective function as a regularization term, which can make the parameters of the model smoother and avoid overfitting problems.

[0096] It's important to note that while both L1 loss and L1 regularization involve calculating the L1 norm, and L2 loss and L2 regularization both involve calculating the L2 norm, they each play different roles in deep learning. L1 loss or L2 loss is primarily used to minimize prediction error during model training, enabling the model to better fit the data, while L1 regularization or L2 regularization is primarily used to control model complexity, prevent overfitting, and enhance the model's representational capabilities.

[0097] In one embodiment, the training strategy of the image super-resolution network model can adopt the optimizer Adam, in which the hyperparameters can be set to β1=0.9, β2=0.99, ε=10 -8 , the learning rate is set to 1e-4, and the number of iterations is 300,000. That is, when the model training times reaches the preset training times of 300,000, the training is terminated. At this time, the image super-resolution network model is used as the trained image super-resolution network model to reconstruct the LR image into the HR image.

[0098] Of course, it is also possible to determine when the image super-resolution network model has completed training based on other training strategies, and this application does not limit this. For example, the value of the loss function L is used to determine whether the model has completed training. When the value of L tends to be stable, that is, when the loss of the image super-resolution network model basically no longer decreases, the image super-resolution network model can be stopped from training. For another example, when the training time of the image super-resolution network model reaches a preset training time, the training is terminated.

[0099] In one embodiment, the training dataset already contains a large number of training samples. To increase data diversity without increasing the size of the training dataset, LR-HR image pairs can be read from the training sample set, and then a region of a certain size can be randomly cropped from the LR-HR image pair to train the model. Accordingly, step S104 can specifically include the following processing: reading a second image pair from the training dataset; randomly cropping a subregion from the second image pair based on a preset size, and using the subregion to train the initial image super-resolution network model.

[0100] In practice, after reading any LR-HR image pair from the training dataset, the LR-HR image pair can be cropped based on a pre-set random crop size. The ratio of the random crop size of the LR image to the random crop size of the HR image is equal to the ratio of the resolutions of the LR image to the HR image.

[0101] For example, taking the LR image size of 240×240 and the HR image size of 480×480 in the training dataset as an example, the height and width of the LR image are 1 / 2 of the height and width of the HR image. If the random cropping size of the LR image is set to 64×64, the random cropping size of the HR image can be enlarged by 2 times in both height and width, that is, set to 128×128. Based on this preset random cropping size, when a 240×240 LR image is read, a 64×64 sub-region (patch) can be randomly cropped from the LR image as the input training image for the image super-resolution network model, and a 128×128 sub-region can be cropped from the corresponding position in the HR image to train the image super-resolution network model.

[0102] S106, performing structural reparameterization processing on the trained image super-resolution network model to generate an image super-resolution network model for inference.

[0103] Among them, the row-column convolution module included in the image super-resolution network model for inference and the corresponding over-parameterized row-column convolution module have equivalent convolution weights.

[0104] In practice, after completing the training of the image super-resolution network model, an effective reparameterization strategy can be designed based on the current model structure and parameters. This reparameterization strategy can include changing the structure of the convolutional layer, modifying the activation function, or introducing new model components. This can reduce model complexity and improve computational efficiency while maintaining or improving model performance.

[0105] In one embodiment, the trained image super-resolution network model can be re-parameterized by simplifying the structure of the over-parameterized row-column convolution modules. Accordingly, step S106 may specifically include: converting each over-parameterized row-column convolution module into a single row-column convolution module.

[0106] In implementation, see Figure 8 In the model training phase, the over-parameterized row-column convolution module in the image super-resolution network model includes three 1×1 convolutions. After the model training is completed, the over-parameterized row-column convolution module can be equivalently transformed, and the over-parameterized row-column convolution module including three 1×1 convolutions used in the training phase can be replaced with a lightweight naive convolution module to simplify the image super-resolution network model. In other words, in the inference phase, through the multi-branch reparameterization technology, the over-parameterized row-column convolution can be converted into a single row-column convolution for inference. The calculation process of the inference phase can refer to formula (7):

[0107] F out =σ(W k×1 *σ(W 1×k *Fin )) (7)

[0108] Among them, for the simplified row-column convolution module, first use 1×k row convolution to input feature layer F in Perform convolution operation to obtain the row convolution result σ(W 1×k *F in ), and then use k×1 column convolution to perform convolution operation on the row convolution result to obtain the output feature layer F out .

[0109] In this way, by structurally reparameterizing the over-parameterized row-column convolutional module, we can obtain a row-column convolutional module with better performance. This, in turn, reduces the number of model parameters and computational complexity while improving the efficiency and performance of the model during the inference phase.

[0110] Further, see again Figure 5 When the image super-resolution network model contains multiple over-parameterized row and column convolution modules, the m over-parameterized row and column convolution modules in the model can be processed as follows: Figure 8 The equivalent conversion shown in FIG. The equivalent conversion may include: performing a linear transformation on multiple convolution weights of an over-parameterized row-column convolution module to generate a single convolution weight; and constructing a single row-column convolution module using the single convolution weight.

[0111] Specifically, in Figure 8In the illustrated embodiment, the over-parameterized row convolution module and the over-parameterized column convolution module in the training phase each include 4 different convolutions (3 layers of convolution and 2 branches), and accordingly, have 4 different convolution weights (i.e., learned parameters). The convolution calculation has a linear property, and there is no nonlinear activation function between the multiple convolutions of the over-parameterized row convolution module and the over-parameterized column convolution module, so these original convolution weights can be linearly transformed, such as weighted averaging or simple addition, to merge into an equivalent convolution weight. Among them, the multiple convolution weights in the over-parameterized row convolution module (i.e., the convolution weights of 3 1×1 convolutions and 1 1×k convolution) can be fused into a fixed 1×k convolution weight; the multiple convolution weights in the over-parameterized column convolution module (i.e., the convolution weights of 3 1×1 convolutions and 1 k×1 convolution) can be fused into a fixed k×1 convolution weight. The fused convolution weights can be used to construct new row-column convolution modules, which are functionally equivalent to the original over-parameterized row-column convolution modules. In this way, by performing equivalent conversions on m over-parameterized row-column convolution modules respectively, a lightweight image super-resolution network model consisting of m simple row-column convolution modules can be obtained. In other words, the re-parameterized row-column convolution modules contained in the image super-resolution network model used for inference have equivalent convolution weights to the corresponding over-parameterized row-column convolution modules contained in the trained initial image super-resolution network model. Furthermore, based on this lightweight image super-resolution network model, LR images can be efficiently inferred and HR images with better quality can be reconstructed.

[0112] It's worth noting that when converting a more complex network model structure to a simpler one, the parameters of the network model structure are equivalently converted, so the replacement of the two network model structures is equivalent. The simpler network model structure that has undergone equivalent conversion can be used directly in the inference phase, improving computational efficiency while maintaining the same performance as the complex network model structure before conversion.

[0113] It should be noted that due to space limitations, this application specification does not enumerate all optional implementation methods. After reading this application specification, those skilled in the art should be able to understand that as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0114] For example, in Figure 3 In one implementation of the embodiment shown, a technical feature is recorded: the over-parameterized row convolution module and the over-parameterized column convolution module each include three 1×1 convolutions. Figure 5In one implementation of the illustrated embodiment, another technical feature is described: the initial image super-resolution network model is obtained by stacking multiple over-parameterized row and column convolution modules. Since the above two technical features do not contradict each other, those skilled in the art should be able to think of the implementation method having both features after reading the specification of this application as an optional implementation method, that is, the initial image super-resolution network model includes multiple over-parameterized row and column convolution modules, and the row convolution module and the column convolution module in each over-parameterized row and column convolution module can respectively include multiple 1×1 convolutions.

[0115] It is worth mentioning that this scheme has a good performance on five baseline datasets, including Set5, Set14, B100, Urban100 and Div2k. Figure 5 The image super-resolution network model (denoted as Ours) provided in the illustrated embodiment is compared with other existing image super-resolution schemes, including FSRCNN, ECBSR_m4c16, ECBSR_m2c16, and ECBSR_m4c8. The model performance comparison results can be found in Table 1. Peak Signal to Noise Ratio (PSNR) and runtime are used as evaluation metrics for model performance. The test dataset uses the 100-image validation set of the baseline datasets Set5, Set14, B100, Urban100, and Div2k for the image super-resolution reconstruction task.

[0116] Table 1 Model performance comparison

[0117] model Set5 Set14 B100 Urban100 Div2k runtime FSRCNN 36.98 32.62 31.50 29.85 34.74 97ms ECBSR_m4c16 37.24 32.88 31.63 30.21 35.10 54ms ECBSR_m2c16 36.91 32.63 31.44 29.71 34.84 34ms ECBSR_m4c8 36.78 32.53 31.36 29.49 34.77 24ms Ours 37.01 32.66 31.50 29.84 34.94 24ms

[0118] For the runtime tests in Table 1, this solution first converted the neural network's PyTorch model into an ONNX model and then tested it on the CPU of an Apple M2 Pro chip. ECBSR[] is a leading edge-device-friendly image super-resolution algorithm, with m4c16, m2c16, and m4c8 representing different-sized versions. It can be seen that among the solutions that can achieve real-time execution (running time less than 33ms), namely ECBSR_m4c8 and this solution, this solution significantly outperforms ECBSR_m4c8. Furthermore, comparing PSNR values, this solution outperforms ECBSR_m4c8 by 0.23dB on the Set5 dataset, 0.13dB on Set14, 0.14dB on B100, 0.35dB on Urban100, and 0.17dB on Div2k.

[0119] As can be seen from the above, the image super-resolution network model provided by each embodiment of the present application can run efficiently based on a small amount of computing and memory resources in the inference stage, thereby ensuring real-time operation on low-end and mid-end edge devices with insufficient computing and memory resources. Compared with the existing advanced super-resolution algorithms for edge devices, each embodiment of the present application builds and trains an image super-resolution network model based on an over-parameterized row and column convolution module, and optimizes the trained image super-resolution network model through structural reparameterization processing, so that it can run in real time on low-end and mid-end edge devices and mobile devices, and can also run in real time on the CPU of some mobile devices, and its performance significantly surpasses other image super-resolution network models, such as ECBSR, SRLUT and EdgeSR.

[0120] Based on the same technical concept, an embodiment of the present application also provides an image super-resolution reconstruction method, which may include: inputting a low-resolution image into an image super-resolution network model for inference generated by any of the above-mentioned generation methods to obtain a high-resolution image output by the image super-resolution network model; the image super-resolution network model for inference includes one or more structurally reparameterized row-column convolution modules; each structurally reparameterized row-column convolution module has an equivalent convolution weight to the corresponding over-parameterized row-column convolution module contained in the trained image super-resolution network model.

[0121] The image super-resolution reconstruction method provided in this embodiment, its implementation principle and the technical effects to be achieved can be found in the embodiment of the method for generating an image super-resolution network model, which will not be repeated here.

[0122] Based on the same technical concept, the embodiment of the present application also provides an electronic device. Figure 9 , which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the method for generating an image super-resolution network model provided in any of the above embodiments or the image super-resolution reconstruction method provided in any of the above embodiments.

[0123] It should be noted that in actual applications, the methods for generating an image super-resolution network model and / or the methods for image super-resolution reconstruction provided in the above embodiments can be assigned to different functional modules as needed to complete each step, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the electronic device and the method for generating an image super-resolution network model provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiments and will not be repeated here.

[0124] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the existing technology can be embodied in the form of a software product. The software product of the image super-resolution network model generation method and / or the image super-resolution reconstruction method can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., including storing a number of instructions for enabling an electronic device to execute the methods described in each embodiment or certain parts of the embodiments.

[0125] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for generating an image super-resolution network model, characterized in that: The method comprises: Obtaining a training data set and an initial image super-resolution network model; wherein the initial image super-resolution network model is constructed based on one or more over-parameterized row-column convolution modules, each of the over-parameterized row-column convolution modules including an over-parameterized row convolution module and an over-parameterized column convolution module; Use the training data set to train the initial image super-resolution network model; The trained image super-resolution network model is subjected to structural reparameterization processing to generate an image super-resolution network model for inference; wherein the row-column convolution module contained in the image super-resolution network model for inference and the corresponding over-parameterized row-column convolution module have equivalent convolution weights.

2. The method according to claim 1, wherein The over-parameterized row convolution module and the over-parameterized column convolution module each include multi-layer convolution processing.

3. The method according to claim 2, wherein The multi-layer convolutions in the over-parameterized row convolution module and the over-parameterized column convolution module are respectively constructed using a multi-branch structure.

4. The method according to claim 2, wherein The structural reparameterization processing of the trained image super-resolution network model specifically includes: Each of the over-parameterized row-column convolution modules is equivalently converted into a single row-column convolution module.

5. The method according to claim 4, wherein The equivalent conversions include: Generate a single convolution weight by linearly transforming the multiple convolution weights of the over-parameterized row and column convolution module; The single row-column convolution module is constructed using the single convolution weight.

6. The method according to claim 1, wherein The training of the initial image super-resolution network model using the training data set specifically includes: Acquire an input image from the training data set, and perform downsampling processing on the input image; The initial image super-resolution network model is trained based on the input image that has been downsampled.

7. The method according to claim 1, wherein The training of the initial image super-resolution network model using the training data set specifically includes: The initial image super-resolution network model is trained iteratively multiple times based on a loss function with an added regularization term.

8. The method according to claim 7, wherein The regularization term is the orthogonal regularization loss L ortho , calculated using the following formula: Among them, W 1×1 represents 1×1 convolution, represents the transpose of 1×1 convolution, I represents the identity matrix, ‖·‖ F represents the F-norm.

9. A method for image super-resolution reconstruction, characterized in that: The reconstruction method comprises: Inputting a low-resolution image into the image super-resolution network model for inference according to any one of claims 1 to 8, to obtain a high-resolution image output by the image super-resolution network model; The image super-resolution network model for inference includes one or more structurally reparameterized row-column convolution modules; each of the structurally reparameterized row-column convolution modules has an equivalent convolution weight to the corresponding over-parameterized row-column convolution module contained in the trained image super-resolution network model.

10. An electronic device, characterized in that: The device includes: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method for generating an image super-resolution network model as described in any one of claims 1 to 8, or the image super-resolution reconstruction method as described in claim 9.

11. A computer-readable storage medium, characterized in that The computer storage medium stores multiple instructions, which are suitable for being loaded by a processor and executed by the method for generating an image super-resolution network model as described in any one of claims 1-8, or the image super-resolution reconstruction method as described in claim 9.

Images