Training method and device of image super-resolution reconstruction model

Abstract

The present disclosure provides a method and device for training an image super-resolution reconstruction model, which can be applied to the field of image processing technology. The method comprises: performing degradation processing on a first image to obtain a second image, the resolution of the second image being less than that of the first image; performing feature extraction on the second image using a multi-convolution branch to obtain a feature image, wherein the multi-convolution branch comprises a first multi-convolution branch and a second multi-convolution branch, and the first multi-convolution branch comprises an edge extraction operator and a denoising operator; performing up-sampling processing on the feature image to obtain a target image; adjusting parameters of an initial image super-resolution reconstruction model according to a loss value between the first image and the target image to obtain an intermediate image super-resolution reconstruction model, wherein the intermediate image super-resolution reconstruction model comprises the multi-convolution branch; and performing re-parameter processing on the intermediate image super-resolution reconstruction model to obtain a target image super-resolution reconstruction model.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically to a training method and apparatus for an image super-resolution reconstruction model. Background Technology

[0002] Image super-resolution reconstruction refers to obtaining a higher resolution image corresponding to the target image by employing appropriate reconstruction algorithms. Deep learning-based super-resolution methods have been proposed one after another; however, most of these methods suffer from excessive image parameters and complex network structures, making it difficult to implement super-resolution reconstruction tasks on edge devices or mobile devices.

[0003] To address the aforementioned issues, related technologies employ lightweight processing to reduce the number of model parameters, thereby reducing inference time. While this reduces model inference time, the reconstructed image lacks rich edges and textures, making it difficult to achieve both reduced model inference time and good visual effects in the reconstructed image. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a training method and apparatus for an image super-resolution reconstruction model.

[0005] According to a first aspect of this disclosure, a training method for an image super-resolution reconstruction model is provided, comprising: performing degradation processing on a first image to obtain a second image, wherein the resolution of the second image is less than that of the first image; performing feature extraction on the second image using multiple convolutional branches to obtain a feature image, wherein the multiple convolutional branches include a first multiple convolutional branch and a second multiple convolutional branch, the first multiple convolutional branch including an edge extraction operator and a denoising operator; performing upsampling processing on the feature image to obtain a target image; adjusting the parameters of an initial image super-resolution reconstruction model based on the loss value between the first image and the target image to obtain an intermediate image super-resolution reconstruction model, wherein the intermediate image super-resolution reconstruction model includes multiple convolutional branches; and performing reparameter reprocessing on the intermediate image super-resolution reconstruction model to obtain a target image super-resolution reconstruction model.

[0006] According to embodiments of this disclosure, feature extraction of a second image using multiple convolutional branches to obtain a feature image includes: extracting features of the second image using a first multiple convolutional branch to obtain a first feature image; extracting features of the first feature image using an enhanced residual block to obtain a second feature image, wherein the enhanced residual block includes a second multiple convolutional branch; and performing residual addition processing on the first feature image and the second feature image to obtain a feature image.

[0007] According to embodiments of this disclosure, reparameter processing is performed on an intermediate image super-resolution reconstruction model to obtain a target image super-resolution reconstruction model, including: padding each convolutional branch of the first multi-convolutional branch of the intermediate image super-resolution reconstruction model to obtain a first multi-convolutional branch of the target size; accumulating the first multi-convolutional branch of the target size to obtain a first single convolution; and determining the target image super-resolution reconstruction model based on the first single convolution.

[0008] According to embodiments of this disclosure, each convolutional branch includes at least one sub-convolution; each convolutional branch of the first multi-convolutional branch of the intermediate image super-resolution reconstruction model is padded to obtain a first multi-convolutional branch of the target size, including: when the convolutional branch of the first multi-convolutional branch includes at least two sub-convolutions, determining a channel expansion ratio; linearly recombining the kernel weights and biases of each sub-convolution according to the channel expansion ratio to obtain a first target kernel weight and a first target bias; and determining the first multi-convolutional branch of the target size according to the first target kernel weight and the first target bias.

[0009] According to an embodiment of this disclosure, accumulating the first multi-convolutional branches of the target size to obtain a first single convolution includes: determining the kernel weight and bias of each convolutional branch in the first multi-convolutional branches of the target size; adding the kernel weight and bias of each convolutional branch to obtain a second target convolutional kernel weight and a second target bias; and determining the first single convolution based on the second target convolutional kernel weight and the second target bias.

[0010] According to embodiments of this disclosure, the enhanced residual block further includes multiple initial single convolutions; the intermediate image super-resolution reconstruction model is subjected to reparameter processing to obtain the target image super-resolution reconstruction model, and the method further includes: padding each convolution branch of the second multi-convolution branch of the intermediate image super-resolution reconstruction model to obtain a second multi-convolution branch of the target size; accumulating the second multi-convolution branch of the target size to obtain a second single convolution; accumulating the second single convolution and multiple initial single convolutions to obtain a third single convolution; and determining the target image super-resolution reconstruction model based on the first single convolution and the third single convolution.

[0011] According to an embodiment of this disclosure, a second image is obtained by degrading a first image, including: adding noise to the first image to obtain a noisy image; blurring the noisy image to obtain a blurred image; and downsampling the blurred image to obtain the second image.

[0012] According to embodiments of this disclosure, adding noise to a first image to obtain a noisy image includes: performing Gaussian noise processing on the first image to obtain a noisy image, wherein the Gaussian noise processing includes first Gaussian noise processing, second Gaussian noise processing, and third Gaussian noise processing; the noisy image includes three channel images; wherein the Gaussian noise of each channel image in the noise image corresponding to the first Gaussian noise processing is inconsistent; the Gaussian noise of each channel image in the noise image corresponding to the second Gaussian noise processing is consistent; the Gaussian noise of each channel image in the noise image corresponding to the third Gaussian noise processing is inconsistent, and the Gaussian noise of each channel image is dependent on each other.

[0013] A second aspect of this disclosure provides an image super-resolution reconstruction method, comprising: acquiring an image to be reconstructed; and processing the image to be reconstructed using the aforementioned image super-resolution reconstruction model to obtain a reconstructed image.

[0014] A third aspect of this disclosure provides a training apparatus for an image super-resolution reconstruction model, comprising: a degradation module for performing degradation processing on a first image to obtain a second image, wherein the resolution of the second image is less than that of the first image; an extraction module for performing feature extraction on the second image using multiple convolutional branches to obtain a feature image, wherein the multiple convolutional branches include a first multiple convolutional branch and a second multiple convolutional branch, and the first multiple convolutional branch includes an edge extraction operator and a denoising operator; a sampling module for performing upsampling processing on the feature image to obtain a target image; an adjustment module for adjusting the parameters of an initial image super-resolution reconstruction model according to the loss value between the first image and the target image to obtain an intermediate image super-resolution reconstruction model, wherein the intermediate image super-resolution reconstruction model includes multiple convolutional branches; and a processing module for performing reparameter processing on the intermediate image super-resolution reconstruction model to obtain a target image super-resolution reconstruction model.

[0015] A fourth aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0016] The fifth aspect of this disclosure also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0017] A sixth aspect of this disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0018] According to embodiments of this disclosure, feature extraction is performed on the second image using multiple convolutional branches to obtain a feature image; the feature image is then upsampled to obtain a target image; based on the loss value between the first image and the target image, the parameters of the initial image super-resolution reconstruction model are adjusted to obtain an intermediate image super-resolution reconstruction model, and then reparameter processing is performed to obtain the target image super-resolution reconstruction model. By employing multiple convolutional branches and their edge extraction and denoising operators to extract features from the second image, the problem of insufficient extraction of image edges and textures is avoided, thus improving the visual effect of the reconstructed image. Simultaneously, the reparameter processing of the intermediate image super-resolution reconstruction model avoids the problem of excessively large model parameters, thereby reducing the model inference time. Attached Figure Description

[0019] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 The illustrations depict application scenarios of training methods, apparatuses, devices, media, and program products for image super-resolution reconstruction models according to embodiments of the present disclosure.

[0021] Figure 2 A flowchart illustrating a training method for an image super-resolution reconstruction model according to an embodiment of the present disclosure is shown schematically.

[0022] Figure 3 A schematic diagram of a first multi-branch convolution according to an embodiment of the present disclosure is shown;

[0023] Figure 4 A schematic diagram of an enhanced residual block according to an embodiment of the present disclosure is shown;

[0024] Figure 5 A schematic diagram of an image super-resolution reconstruction model according to an embodiment of the present disclosure is shown.

[0025] Figure 6 A schematic diagram illustrating reparameter processing according to an embodiment of the present disclosure is shown.

[0026] Figure 7 The diagram illustrates a training method for an image super-resolution reconstruction model according to an embodiment of the present disclosure.

[0027] Figure 8 A flowchart illustrating an image super-resolution reconstruction method according to an embodiment of the present disclosure is shown schematically.

[0028] Figure 9This schematic diagram illustrates a structural block diagram of a training apparatus for an image super-resolution reconstruction model according to an embodiment of the present disclosure;

[0029] Figure 10 A schematic diagram illustrating the structure of an image super-resolution reconstruction apparatus according to an embodiment of the present disclosure is shown; and

[0030] Figure 11 A block diagram of an electronic device suitable for implementing a training method for an image super-resolution reconstruction model according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0031] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0033] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0034] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0035] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0036] The embodiments of this disclosure provide a training method for an image super-resolution reconstruction model. A first image is degraded to obtain a second image, the resolution of which is lower than that of the first image. Features are extracted from the second image using multiple convolutional branches to obtain a feature image. The multiple convolutional branches include a first convolutional branch and a second convolutional branch, the first convolutional branch including an edge extraction operator and a denoising operator. The feature image is upsampled to obtain a target image. Based on the loss value between the first image and the target image, the parameters of the initial image super-resolution reconstruction model are adjusted to obtain an intermediate image super-resolution reconstruction model, the intermediate image super-resolution reconstruction model including multiple convolutional branches. The intermediate image super-resolution reconstruction model is then reparameterized to obtain the target image super-resolution reconstruction model.

[0037] Figure 1 The illustration schematically depicts an application scenario of a training method, apparatus, device, medium, and program product for an image super-resolution reconstruction model according to embodiments of the present disclosure.

[0038] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0039] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0040] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0041] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0042] It should be noted that the training method for the image super-resolution reconstruction model provided in this embodiment can generally be executed by server 105. Correspondingly, the training device for the image super-resolution reconstruction model provided in this embodiment can generally be located in server 105. The training method for the image super-resolution reconstruction model provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the training device for the image super-resolution reconstruction model provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0043] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0044] The following will be based on Figure 1 The described scene, through Figures 2-7 The training method of the image super-resolution reconstruction model of the disclosed embodiments is described in detail.

[0045] Figure 2 A flowchart illustrating a training method for an image super-resolution reconstruction model according to an embodiment of the present disclosure is shown.

[0046] like Figure 2 As shown, the training method of the image super-resolution reconstruction model in this embodiment includes operations S210 to S250.

[0047] In operation S210, the first image is degraded to obtain a second image, the resolution of which is less than that of the first image.

[0048] According to embodiments of this disclosure, the first image may be a high-resolution electronic endoscopy image obtained by extracting frames from a real electronic endoscopic surgical video that does not contain patient privacy and by using bicubic downsampling on the extracted frames to obtain a clean and unblurred image.

[0049] According to embodiments of this disclosure, the first image may also be a high-resolution image from the DIV2K dataset in the field of image restoration.

[0050] According to embodiments of this disclosure, degradation processing can be a process of reducing image resolution, whereby a low-resolution second image is obtained by degradation processing a high-resolution first image.

[0051] According to embodiments of this disclosure, the obtained second image can be randomly rotated by 90 degrees, 180 degrees, and 270 degrees for data augmentation.

[0052] In operation S220, feature extraction is performed on the second image using multiple convolutional branches to obtain a feature image. The multiple convolutional branches include a first multiple convolutional branch and a second multiple convolutional branch. The first multiple convolutional branch includes an edge extraction operator and a denoising operator.

[0053] According to embodiments of this disclosure, multi-convolutional branches enable multi-channel processing. Feature extraction is performed on the second image in each convolutional branch, and the results of each convolutional branch are then summed as the output to increase the number of model parameters and make model training more stable. Specifically, both the first and second multi-convolutional branches are used for feature extraction from the second image.

[0054] According to embodiments of this disclosure, an edge extraction operator is used to identify areas in an image where brightness changes significantly, which typically represent boundaries between different regions in the image.

[0055] According to embodiments of this disclosure, a denoising operator is used to reduce random variations (i.e., image noise) in an image while preserving the image details.

[0056] According to embodiments of this disclosure, edge extraction operators and denoising operators are added to the multi-convolutional branches, making the model more sensitive to edge information of the image.

[0057] In operation S230, the feature image is upsampled to obtain the target image.

[0058] According to embodiments of this disclosure, upsampling is performed through a third multi-convolutional branch, a PixelShuffle layer, and an activation function. The third multi-convolutional branch has the same structure as the first multi-convolutional branch. The number of input channels of the third multi-convolutional branch is 32, and the number of output channels is the number of input channels of the model multiplied by the square of the magnification factor. For example, if the number of input channels of the model is 3 and the magnification factor of the PixelShuffle layer is 2, then the number of output channels is 12.

[0059] According to embodiments of this disclosure, upsampling processing enlarges the size of the feature image to a higher resolution to obtain the target image.

[0060] In operation S240, the parameters of the initial image super-resolution reconstruction model are adjusted based on the loss value between the first image and the target image to obtain the intermediate image super-resolution reconstruction model, wherein the intermediate image super-resolution reconstruction model includes multiple convolutional branches.

[0061] According to embodiments of this disclosure, a loss value is obtained by determining the absolute difference between the first image and the target image.

[0062] According to embodiments of this disclosure, the parameters of the initial image super-resolution reconstruction model are adjusted based on the obtained loss value, wherein the batch size selected during training is 32, the patch size is 64, and the learning rate is... The training run consisted of 800 rounds. The Adam optimizer was used to calculate gradients and update model parameters so that the target image output by the intermediate image super-resolution reconstruction model would be closer to the first image.

[0063] In operation S250, the intermediate image super-resolution reconstruction model is reparameterized to obtain the target image super-resolution reconstruction model.

[0064] According to embodiments of this disclosure, reparameter processing can be performed on the intermediate image super-resolution reconstruction model to reduce the number of model parameters, thereby shortening the inference time of the target image super-resolution reconstruction model.

[0065] According to embodiments of this disclosure, feature extraction is performed on the second image using multiple convolutional branches to obtain a feature image; the feature image is then upsampled to obtain a target image; based on the loss value between the first image and the target image, the parameters of the initial image super-resolution reconstruction model are adjusted to obtain an intermediate image super-resolution reconstruction model, and then reparameter processing is performed to obtain the target image super-resolution reconstruction model. By employing multiple convolutional branches and their edge extraction and denoising operators to extract features from the second image, the problem of insufficient extraction of image edges and textures is avoided, thus improving the visual effect of the reconstructed image. Simultaneously, the reparameter processing of the intermediate image super-resolution reconstruction model avoids the problem of excessively large model parameters, thereby reducing the model inference time.

[0066] According to embodiments of this disclosure, feature extraction of a second image using multiple convolutional branches to obtain a feature image includes: extracting features of the second image using a first multiple convolutional branch to obtain a first feature image; extracting features of the first feature image using an enhanced residual block to obtain a second feature image, wherein the enhanced residual block includes a second multiple convolutional branch; and performing residual addition processing on the first feature image and the second feature image to obtain a feature image.

[0067] According to embodiments of this disclosure, the first multi-convolutional branch includes multiple convolutional branches, each convolutional branch including at least one sub-convolution, wherein multiple sub-convolutions are added in individual convolutional branches. Convolution is used to increase the number of channels by a factor of k. The value of k can be selected according to actual needs, with k=2 being the preferred value.

[0068] According to embodiments of this disclosure, since subsequent parameter reparameter processing is performed on the intermediate image super-resolution reconstruction model, it is necessary to convert multiple convolutional branches into a single convolutional branch. Therefore, after increasing the number of channels, it is necessary to restore the number of channels to the same level as other convolutional branches, and finally add... Convolution or Convolution achieves compression of convolution branch channels.

[0069] Figure 3 A schematic diagram of a first multi-branch convolution according to an embodiment of the present disclosure is shown.

[0070] According to embodiments of this disclosure, such as Figure 3 As shown, the first multi-convolutional branch 301 includes convolution, convolution, convolution, convolution, convolution- convolution, Convolution - Special Operators - Convolution, in addition, the first multi-convolutional branch 301 also includes residual connections, which can be viewed as special cases where the middle part is 1 and the rest are 0. Convolution, the input of the residual connection is related to the above. convolution, convolution, convolution, convolution, convolution- convolution, Convolution - Special Operators - The outputs of each convolution are connected. Each convolution branch performs feature extraction on the second image and inputs its output to the residual connection. The residual connection sums the outputs of each convolution branch and inputs the sum into the activation function (Parametric Rectified Linear Unit, PReLU) 302 to obtain the first feature image.

[0071] According to embodiments of this disclosure, the special operators include edge extraction operators (Sobel operators). , Laplacian operator Sharpening operator ) and denoising operators (Gaussian kernel operator) The specific parameters of the special operators are as follows: , , , , Furthermore, each special operator is accompanied by a scaling factor for the model to learn.

[0072] According to embodiments of this disclosure, each convolutional branch has 3 input channels, 32 output channels, and a stride of 1. The padding of the convolution is 0. The padding of convolution is , The padding of convolution is , The padding of convolution is . convolution- In convolution The convolution output has 64 channels and padding is... , The convolution input has 64 channels and padding is... . Convolution - Special Operators - The first in convolution The convolution output has 64 channels and padding is... The special operator uses depthwise convolution, meaning both the input and output channels are 64, and the padding is... The second one The convolution has 64 input channels and padding of [value missing]. .

[0073] According to embodiments of this disclosure, increasing the number of output channels can enhance the model's expressive power, enabling the capture of complex features. Convolution can achieve linear combination of features and dimensionality reduction or expansion without changing the spatial size of the input feature map.

[0074] Figure 4 A schematic diagram of an enhanced residual block according to an embodiment of the present disclosure is shown.

[0075] According to embodiments of this disclosure, there can be multiple reinforcing residual blocks, the number of which can be selected according to actual needs, preferably four. For example, Figure 4 As shown, the enhanced residual block includes a second multi-convolutional branch 402, and also includes multiple initial single convolutions 401, for example... Convolution, along with two skip connections 404 and an activation function 403.

[0076] According to embodiments of this disclosure, the second multi-convolution branch has a similar structure to the first multi-convolution branch, but the special operators of the second multi-convolution branch may not include the Gaussian kernel operator found in the first multi-convolution branch. The rest can be the same as the first multi-convolution branch, and will not be elaborated here.

[0077] According to embodiments of this disclosure, the initial single convolution can be two, concatenated with a second multi-convolution branch, wherein the first... Convolution is used to expand the number of channels; its input channels are 32 and its output channels are 64. The second multi-convolution branch has 64 input channels and 64 output channels. Convolution is used to compress the number of channels; it has 64 input channels and 32 output channels.

[0078] According to embodiments of this disclosure, skip connections allow the first feature image of the input to skip one or more layers and be directly passed from the input to subsequent layers, avoiding the gradient vanishing problem during model training, thereby enabling the model to train deep structures more effectively.

[0079] According to embodiments of this disclosure, feature extraction of the second image is performed using multiple convolutional branches and their edge extraction and denoising operators, which avoids the problem of insufficient extraction of image edges and textures, and achieves the technical effect of improving the visual effect of the reconstructed image.

[0080] Figure 5 A schematic diagram of an image super-resolution reconstruction model according to an embodiment of the present disclosure is shown.

[0081] According to embodiments of this disclosure, such as Figure 5 As shown, the second image 501 is input into the first multi-branch convolution 502 of the image super-resolution reconstruction model for feature extraction. The resulting first feature image is input into the enhancement residual block 503, and the second feature image is output. The first feature image is also input into the residual connection 505 together with the second feature image through the skip connection 504, and the feature image is output. After being sampled by the upsampling module 506, the target image 507 is obtained.

[0082] According to embodiments of this disclosure, reparameter processing is performed on an intermediate image super-resolution reconstruction model to obtain a target image super-resolution reconstruction model, including: padding each convolutional branch of the first multi-convolutional branch of the intermediate image super-resolution reconstruction model to obtain a first multi-convolutional branch of the target size; accumulating the first multi-convolutional branch of the target size to obtain a first single convolution; and determining the target image super-resolution reconstruction model based on the first single convolution.

[0083] According to embodiments of this disclosure, the filling process may involve filling items whose dimensions are not the target dimensions ( The convolution kernel weights are expanded to That is, in convolution, Convolution and The convolutional layers are padded with zeros to expand their size. .

[0084] According to embodiments of this disclosure, the accumulation process may involve accumulating multiple convolutional branches to obtain a single convolution. The first single convolution obtained has fewer parameters than the first multi-convolutional branch of the target size, which reduces the model inference time to a certain extent compared to the intermediate image super-resolution reconstruction model.

[0085] According to embodiments of this disclosure, each convolutional branch includes at least one sub-convolution; each convolutional branch of the first multi-convolutional branch of the intermediate image super-resolution reconstruction model is padded to obtain a first multi-convolutional branch of the target size, including: when the convolutional branch of the first multi-convolutional branch includes at least two sub-convolutions, determining a channel expansion ratio; linearly recombining the kernel weights and biases of each sub-convolution according to the channel expansion ratio to obtain a first target kernel weight and a first target bias; and determining the first multi-convolutional branch of the target size according to the first target kernel weight and the first target bias.

[0086] According to embodiments of this disclosure, the channel expansion ratio is used to increase the number of channels, thereby increasing the number of model parameters. It can be set according to actual needs, and preferably the channel expansion ratio k=2.

[0087] According to embodiments of this disclosure, linear recombination can be achieved by recombining the kernel weights and biases of multiple concatenated sub-convolutions to obtain first target kernel weights and first target biases, thereby converting multiple concatenated sub-convolutions into a first multi-convolution branch of a target size.

[0088] According to embodiments of this disclosure, for Concatenated convolution, assuming the number of input channels for this concatenated convolution is... The number of output channels is The channel expansion ratio is k. Convolution kernel weights The shape is bias The shape is ; Convolution kernel weights The shape is bias The shape is The converted one The first objective of the convolution is to set the kernel weights to K, the first objective bias to B, and the input data to I with shape I. , The convolution operation is shown in formula (1).

[0089] (1);

[0090] in, Convolution only performs linear combination of channels and not spatial aggregation, which can be achieved through linear recombination. The parameters in the convolution kernel are merged, which can be done by transposing them, that is, by... The first two dimensions are swapped to become Its shape is From this, we can deduce that: It can be seen that the shape of the weight K of the first target convolution kernel after merging is as follows: .

[0091] To obtain the first target bias B, since a broadcast operation is used when adding biases, it should be... The elements in the copy are expanded into Its shape is Therefore, there is It can be seen that the shape of the first target bias B after merging is... .

[0092] Based on the weights of the first target convolutional kernel First target bias Obtain the first multi-convolutional branch of the target size to achieve - Transformation of concatenated convolution.

[0093] According to embodiments of this disclosure, when the sub-convolution is a special operator, for example, -Special Operators- Concatenated convolution. Assume this concatenated convolution has the following input channels: The number of output channels is The channel expansion ratio is k. The first... Convolution kernel weights The shape is bias The shape is Special operators convolution kernel weights The shape is bias The shape is The weights of the second convolutional kernel The shape is bias The shape is One after conversion The first objective of the convolution is to set the kernel weights to K, the first objective bias to B, and the input data to I with shape I. , The convolution operation is shown in formula (2).

[0094] (2);

[0095] The first two concatenated convolutions (i.e.) - The special operator concatenated convolution is transformed into a single convolution, that is, the special operator concatenated convolution is transformed into a single convolution. -Special operators convert concatenated convolution into a single Convolution kernel weights and bias and the above - Concatenated convolution converted to The only difference with convolution is that the special operator convolution kernel needs to be copied and expanded. Its shape is Others also need to be The first two dimensions are swapped to become Its shape is , The elements in the copy are expanded into Its shape is Specifically, as shown in formula (3).

[0096] (3);

[0097] It can be seen that, - Convolution kernel after concatenation transformation by special operators Shape , The shape is In order to achieve -Special operators convert concatenated convolution into a single convolution.

[0098] Then - Concatenated convolution converted to Convolution, as shown in formula (4).

[0099] (4);

[0100] First to The first two dimensions are swapped to become Its shape is ,use Performing a convolution operation on it yields: Then on The first two dimensions are swapped to obtain the weights K of the first target convolutional kernel, which have the following shape: The first target bias B is determined by... right Perform convolution operation plus To obtain, that is, In order to achieve -Special Operators- Concatenated convolution converted to The first multi-convolutional branch of the target size.

[0101] According to an embodiment of this disclosure, accumulating the first multi-convolutional branches of the target size to obtain a first single convolution includes: determining the kernel weight and bias of each convolutional branch in the first multi-convolutional branches of the target size; adding the kernel weight and bias of each convolutional branch to obtain a second target convolutional kernel weight and a second target bias; and determining the first single convolution based on the second target convolutional kernel weight and the second target bias.

[0102] According to embodiments of this disclosure, the kernel weights and biases of each convolutional branch in the first multi-convolutional branch of the target size are determined. The kernel weights and biases of all branches are added together to obtain the second target kernel weights and the second target bias. Since the convolution calculation process is a linear calculation, the result obtained by calculating the input data with the kernel weights and biases of the first single convolution is consistent with the result obtained by calculating the weights and biases of the first multi-convolutional branch.

[0103] According to embodiments of this disclosure, assuming the number of convolutional branches is 3, the weights of each convolutional branch are k1, k2, k3, and the biases are b1, b2, b3, the weight of the second target convolutional kernel is K, the second target bias is B, and the input data is I. The above process can be described in detail as shown in formula (5).

[0104] (5);

[0105] According to embodiments of this disclosure, the enhanced residual block further includes multiple initial single convolutions; the intermediate image super-resolution reconstruction model is subjected to reparameter processing to obtain the target image super-resolution reconstruction model, and the method further includes: padding each convolution branch of the second multi-convolution branch of the intermediate image super-resolution reconstruction model to obtain a second multi-convolution branch of the target size; accumulating the second multi-convolution branch of the target size to obtain a second single convolution; accumulating the second single convolution and multiple initial single convolutions to obtain a third single convolution; and determining the target image super-resolution reconstruction model based on the first single convolution and the third single convolution.

[0106] According to embodiments of this disclosure, each convolutional branch of the second multi-convolutional branch is padded, which is the same as the padded process for each convolutional branch of the first multi-convolutional branch, and will not be described again here.

[0107] According to embodiments of this disclosure, the accumulation process for the second multi-convolutional branch of the target size is the same as the accumulation process for the first multi-convolutional branch of the target size, and will not be described again here.

[0108] According to embodiments of this disclosure, accumulating the second single convolution and multiple initial single convolutions can be achieved by filling the initial single convolutions with... After convolution, it is added to the second single convolution. Specifically, the kernel weights and biases of the initial single convolution and the second single convolution are added to obtain the third single convolution.

[0109] According to embodiments of this disclosure, residual connections can be viewed as special cases where the middle value is 1 and the others are 0. Convolution is used as a convolution branch for accumulation processing.

[0110] Figure 6 A schematic diagram illustrating reparameter processing according to an embodiment of the present disclosure is shown.

[0111] According to embodiments of this disclosure, such as Figure 6 As shown in Figure 601, the enhancement residual block of the intermediate image super-resolution reconstruction model is padded for each convolutional branch of its second multi-convolutional branch, so that each convolutional branch is as shown in Figure 602. Convolution; the second multi-convolutional branch of the target size is accumulated, and the multiple convolutional branches are merged into a second single convolution as shown in 603; subsequently, the initial single convolution and the second single convolution of the enhanced residual block are accumulated to obtain the result shown in 604. The convolution is performed, and finally the skip connections are removed to obtain the third single convolution as shown in 605.

[0112] According to an embodiment of this disclosure, tests were conducted on a test platform. Through reparameter processing, the number of model parameters was significantly reduced from 290260 to 60076, and the average time taken to infer and output a 1080p image was reduced from 0.14s to 30.10ms.

[0113] According to embodiments of this disclosure, by performing reparameter processing on the intermediate image super-resolution reconstruction model, the problem of excessive model parameters is avoided, thereby achieving the technical effect of reducing model inference time.

[0114] According to an embodiment of this disclosure, a second image is obtained by degrading a first image, including: adding noise to the first image to obtain a noisy image; blurring the noisy image to obtain a blurred image; and downsampling the blurred image to obtain the second image.

[0115] According to embodiments of this disclosure, the noise addition process can be performed by selecting multiple processing methods to add noise to the first image, such as Gaussian noise processing, Poisson noise processing, and JPEG compression noise processing. Each type of noise is randomly generated according to certain parameters. For example, for a certain number of first images, 40% of the first images are subjected to Gaussian noise processing, 30% of the first images are subjected to Poisson noise processing, and 30% of the first images are subjected to JPEG compression noise processing. These parameters can be adjusted according to actual needs.

[0116] According to an embodiment of this disclosure, Poisson noise processing adds noise to the first image using formula (6).

[0117] (6);

[0118] in, For Poisson noise processing, the Poisson distribution has the following parameters. It is the average number of random events occurring per unit of time. This represents the actual number of times a random event occurs per unit time, where the random event is the deviation caused by the image pixel value. For Poisson noise processing, It is the product of the pixel values ​​of the first image and a magnification parameter, so that the sampling of noise is more fluctuating and the generated noise is more obvious.

[0119] According to embodiments of this disclosure, each pixel in the three RGB channels of the first image is randomly sampled and divided by the magnification parameter. To ensure that pixel values ​​do not overflow, a three-channel Poisson noise with the same dimensions as the first image is generated, resulting in a noisy image.

[0120] According to embodiments of this disclosure, JPEG compression noise processing compresses a first image by randomly generating different quality factors, and then decompresses the compressed first image to obtain JPEG compression noise of different degrees. In the actual process of generating the noisy image, the quality factor is selected in the range of 30 to 90.

[0121] According to embodiments of this disclosure, fuzzing is divided into Gaussian isotropic fuzzing and Gaussian anisotropic fuzzing based on the size of the fuzzing kernel, wherein the size of the fuzzing kernel is selected from all odd numbers within the range of 7-15.

[0122] According to embodiments of this disclosure, downsampling processing includes nearest neighbor downsampling, bilinear downsampling, and bicubic downsampling.

[0123] According to embodiments of this disclosure, the first image is subjected to noise addition, blurring, and downsampling to reduce the resolution of the first image.

[0124] According to another embodiment of this disclosure, the order of the three degradation processing operations described above can be random. For example, the first image can be blurred first, noise can be added to the blurred image, and finally downsampling can be performed. The order of the three degradation processing operations can also be adjusted according to actual needs.

[0125] According to embodiments of this disclosure, adding noise to a first image to obtain a noisy image includes: performing Gaussian noise processing on the first image to obtain a noisy image, wherein the Gaussian noise processing includes first Gaussian noise processing, second Gaussian noise processing, and third Gaussian noise processing; the noisy image includes three channel images; wherein the Gaussian noise of each channel image in the noise image corresponding to the first Gaussian noise processing is inconsistent; the Gaussian noise of each channel image in the noise image corresponding to the second Gaussian noise processing is consistent; the Gaussian noise of each channel image in the noise image corresponding to the third Gaussian noise processing is inconsistent, and the Gaussian noise of each channel image is dependent on each other.

[0126] According to an embodiment of this disclosure, Gaussian noise processing can be performed by adding noise to the first image using formula (7).

[0127] (7);

[0128] in, Represents the mean, and n represents the dimension of the Gaussian noise. This represents the covariance matrix, due to the mean Both are set to zero, so based on n and Based on the differences, Gaussian noise can be divided into three types.

[0129] According to embodiments of this disclosure, the first Gaussian noise processing dimension is three-dimensional, and the covariance matrix... ,in It is a preset variance parameter, which is randomly selected within a preset variance range. for The identity matrix is ​​used to generate the corresponding Gaussian distribution function. The generated Gaussian noise is independent of each other in the three RGB channels, that is, the Gaussian noise in the three channels is inconsistent. According to this Gaussian distribution, each pixel in the three RGB channels of the first image is randomly sampled to generate three-channel Gaussian noise with the same dimension as the first image.

[0130] According to embodiments of this disclosure, the second Gaussian noise processing dimension is three-dimensional, and the covariance matrix... ,in It is a preset variance parameter, which is randomly selected within a preset variance range. , where all elements are 1 The matrix is ​​used to generate the corresponding Gaussian distribution function. The generated Gaussian noise is consistent in all three RGB channels. According to this Gaussian distribution, each pixel in the three RGB channels of the first image is randomly sampled to generate three-channel Gaussian noise with the same dimension as the first image.

[0131] According to embodiments of this disclosure, the third Gaussian noise processing dimension is three-dimensional, and the covariance matrix... ,in It is a preset variance parameter, which is randomly selected within a preset variance range. Randomly generated A real symmetric matrix is ​​used to generate the corresponding Gaussian distribution function. The generated Gaussian noise is dependent on the three RGB channels, that is, the three channels are inconsistent and the channels are correlated with each other. According to this Gaussian distribution, each pixel in the three RGB channels of the first image is randomly sampled to generate three-channel Gaussian noise with the same dimension as the first image.

[0132] According to embodiments of this disclosure, the preset variance range is {6 / 255, 7 / 255, ... 16 / 255}.

[0133] According to embodiments of this disclosure, the three types of Gaussian noise processing are randomly applied to the first image according to certain parameters. For example, for a certain number of first images, 30% of the first images are processed with first Gaussian noise, 40% of the first images are processed with second Gaussian noise, and 30% of the first images are processed with third Gaussian noise. These parameters can be adjusted according to actual needs.

[0134] Figure 7 The diagram illustrates a method for training an image super-resolution reconstruction model according to an embodiment of the present disclosure.

[0135] According to embodiments of this disclosure, such as Figure 7As shown, in operation S701, the first image is degraded to obtain the second image. In operation S702, features are extracted from the second image using a first multi-convolutional branch to obtain a first feature image. In operation S703, features are extracted from the first feature image using a second multi-convolutional branch to obtain a second feature image. In operation S704, a feature image is determined based on the first and second feature images. In operation S705, the feature image is upsampled to obtain the target image. In operation S706, the parameters of the initial image super-resolution reconstruction model are adjusted based on the target image to obtain an intermediate image super-resolution reconstruction model. In operation S707, the first multi-convolutional branch of the intermediate image super-resolution reconstruction model is padded to obtain a first multi-convolutional branch of the target size; in operation S708, the first multi-convolutional branch of the target size is accumulated to obtain a first single convolution. In operation S709, each convolutional branch of the second multi-convolutional branch of the intermediate image super-resolution reconstruction model is padded to obtain the second multi-convolutional branch of the target size. In operation S710, the second multi-convolutional branch of the target size is accumulated to obtain the second single convolution. In operation S711, the second single convolution and multiple initial single convolutions are accumulated to obtain the third single convolution. In operation S712, the target image super-resolution reconstruction model is determined based on the first single convolution determined in operation S708 and the third single convolution determined in operation S711.

[0136] Figure 8 A flowchart illustrating an image super-resolution reconstruction method according to an embodiment of the present disclosure is shown schematically.

[0137] The second aspect of this disclosure provides an image super-resolution reconstruction method, such as... Figure 8 As shown, the image super-resolution reconstruction method includes operations S810~S820.

[0138] The S810 is used to acquire the image to be reconstructed.

[0139] In operation S820, the image to be reconstructed is processed using the above-mentioned image super-resolution reconstruction model to obtain the reconstructed image.

[0140] According to embodiments of this disclosure, the above-described image super-resolution reconstruction model is used to process the image to be reconstructed, which reduces the model inference time while obtaining a reconstructed image with good visual effects.

[0141] Based on the training method of the image super-resolution reconstruction model described above, this disclosure also provides a training device for the image super-resolution reconstruction model. The following will combine... Figure 9 The device is described in detail.

[0142] Figure 9A schematic block diagram of a training apparatus for an image super-resolution reconstruction model according to an embodiment of the present disclosure is shown.

[0143] like Figure 9 As shown, the training device 900 for the image super-resolution reconstruction model in this embodiment includes a degradation module 910, an extraction module 920, a sampling module 930, an adjustment module 940, and a processing module 950.

[0144] The degradation module 910 is used to perform degradation processing on the first image to obtain a second image, the resolution of the second image being lower than the resolution of the first image. In one embodiment, the degradation module 910 can be used to perform the operation S210 described above, which will not be repeated here.

[0145] The extraction module 920 is used to extract features from the second image using multiple convolutional branches to obtain a feature image. The multiple convolutional branches include a first multi-convolutional branch and a second multi-convolutional branch. The first multi-convolutional branch includes an edge extraction operator and a denoising operator. In one embodiment, the extraction module 920 can be used to perform the operation S220 described above, which will not be repeated here.

[0146] The sampling module 930 is used to upsample the feature image to obtain the target image. In one embodiment, the sampling module 930 can be used to perform the operation S230 described above, which will not be repeated here.

[0147] The adjustment module 940 is used to adjust the parameters of the initial image super-resolution reconstruction model based on the loss value between the first image and the target image to obtain an intermediate image super-resolution reconstruction model, wherein the intermediate image super-resolution reconstruction model includes multiple convolutional branches. In one embodiment, the adjustment module 940 can be used to perform the operation S240 described above, which will not be repeated here.

[0148] The processing module 950 is used to perform reparameter processing on the intermediate image super-resolution reconstruction model to obtain the target image super-resolution reconstruction model. In one embodiment, the processing module 950 can be used to perform the operation S250 described above, which will not be repeated here.

[0149] According to embodiments of this disclosure, feature extraction is performed on the second image using multiple convolutional branches to obtain a feature image; the feature image is then upsampled to obtain a target image; based on the loss value between the first image and the target image, the parameters of the initial image super-resolution reconstruction model are adjusted to obtain an intermediate image super-resolution reconstruction model, and then reparameter processing is performed to obtain the target image super-resolution reconstruction model. By employing multiple convolutional branches and their edge extraction and denoising operators to extract features from the second image, the problem of insufficient extraction of image edges and textures is avoided, thus improving the visual effect of the reconstructed image. Simultaneously, the reparameter processing of the intermediate image super-resolution reconstruction model avoids the problem of excessively large model parameters, thereby reducing the model inference time.

[0150] According to embodiments of this disclosure, any plurality of modules among the degradation module 910, extraction module 920, sampling module 930, adjustment module 940, and processing module 950 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the degradation module 910, extraction module 920, sampling module 930, adjustment module 940, and processing module 950 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any one of the three implementation methods or a suitable combination of any of them. Alternatively, at least one of the degradation module 910, extraction module 920, sampling module 930, adjustment module 940 and processing module 950 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0151] According to embodiments of this disclosure, the extraction module 920 includes a first extraction submodule, a second extraction submodule, and a first processing submodule.

[0152] The first extraction submodule is used to extract features from the second image using the first multi-convolution branch to obtain the first feature image.

[0153] The second extraction submodule is used to extract features from the first feature image using an enhanced residual block to obtain a second feature image, wherein the enhanced residual block includes a second multi-convolutional branch.

[0154] The first processing submodule performs residual addition processing on the first feature image and the second feature image to obtain the feature image.

[0155] According to embodiments of this disclosure, the processing module 950 includes a first filling submodule, a first accumulation submodule, and a first determination submodule.

[0156] The first padding submodule is used to pad each convolutional branch of the first multi-convolutional branch of the intermediate image super-resolution reconstruction model to obtain the first multi-convolutional branch of the target size.

[0157] The first accumulation submodule is used to accumulate the first multi-convolutional branch of the target size to obtain the first single convolution.

[0158] The first determining submodule is used to determine the super-resolution reconstruction model of the target image based on the first single convolution.

[0159] According to embodiments of this disclosure, the first filling submodule includes a first determining unit, a combining unit, and a second determining unit.

[0160] The first determining unit is used to determine the channel expansion ratio.

[0161] The combination unit is used to linearly recombine the kernel weights and biases of each sub-convolution according to the channel expansion ratio to obtain the first target kernel weights and the first target bias.

[0162] The second determining unit is used to determine the first multi-convolutional branch of the target size based on the weights of the first target convolutional kernel and the first target bias.

[0163] According to embodiments of this disclosure, the first accumulation submodule includes a third determining unit, a first processing unit, and a fourth determining unit.

[0164] The third determining unit is used to determine the kernel weights and biases of each convolutional branch in the first multi-convolutional branch of the target size.

[0165] The first processing unit is used to add up the weights and biases of each convolution kernel to obtain the weights of the second target convolution kernel and the second target bias.

[0166] The fourth determining unit is used to determine the first single convolution based on the weights of the second target convolution kernel and the second target bias.

[0167] According to embodiments of this disclosure, the processing module 950 further includes a second filling submodule, a second accumulation submodule, a third accumulation submodule, and a second determination submodule.

[0168] The second padding submodule is used to pad each convolutional branch of the second multi-convolutional branch of the intermediate image super-resolution reconstruction model to obtain the second multi-convolutional branch of the target size.

[0169] The second accumulation submodule is used to accumulate the second multi-convolutional branch of the target size to obtain the second single convolution.

[0170] The third accumulation submodule is used to accumulate the second single convolution and multiple initial single convolutions to obtain the third single convolution.

[0171] The second determination submodule is used to determine the super-resolution reconstruction model of the target image based on the first and third single convolutions.

[0172] According to embodiments of this disclosure, the degradation module 910 includes a second processing submodule, a third processing submodule, and a fourth processing submodule.

[0173] The second processing submodule is used to add noise to the first image to obtain a noisy image.

[0174] The third processing submodule is used to blur the noisy image to obtain a blurred image.

[0175] The fourth processing submodule is used to downsample the blurred image to obtain the second image.

[0176] According to embodiments of this disclosure, the second processing submodule includes a second processing unit.

[0177] The second processing unit is used to perform Gaussian noise processing on the first image to obtain a noisy image. The Gaussian noise processing includes first Gaussian noise processing, second Gaussian noise processing, and third Gaussian noise processing. The noisy image includes three channel images. The Gaussian noise of each channel image in the noise image corresponding to the first Gaussian noise processing is inconsistent. The Gaussian noise of each channel image in the noise image corresponding to the second Gaussian noise processing is consistent. The Gaussian noise of each channel image in the noise image corresponding to the third Gaussian noise processing is inconsistent, and the Gaussian noise of each channel image is dependent on each other.

[0178] Figure 10 A schematic block diagram of an image super-resolution reconstruction apparatus according to an embodiment of the present disclosure is shown.

[0179] like Figure 10 As shown, the image super-resolution reconstruction apparatus 1000 includes an acquisition module 1010 and a reconstruction module 1020.

[0180] The acquisition module 1010 is used to acquire the image to be reconstructed.

[0181] The reconstruction module 1020 is used to process the image to be reconstructed using an image super-resolution reconstruction model to obtain the reconstructed image.

[0182] Figure 11 A block diagram of an electronic device suitable for implementing a training method for an image super-resolution reconstruction model according to an embodiment of the present disclosure is shown schematically.

[0183] like Figure 11 As shown, an electronic device 1100 according to an embodiment of the present disclosure includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage portion 1108 into a random access memory (RAM) 1103. The processor 1101 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0184] RAM 1103 stores various programs and data required for the operation of electronic device 1100. Processor 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Processor 1101 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1102 and / or RAM 1103. It should be noted that the programs may also be stored in one or more memories other than ROM 1102 and RAM 1103. Processor 1101 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0185] According to embodiments of this disclosure, the electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to a bus 1104. The electronic device 1100 may also include one or more of the following components connected to the input / output (I / O) interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the input / output (I / O) interface 1105 as needed. A removable medium 1111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1110 as needed so that computer programs read from it can be installed into the storage section 1108 as needed.

[0186] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0187] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 1102 and / or RAM 1103 and / or one or more memories other than ROM 1102 and RAM 1103 described above.

[0188] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the training method for the image super-resolution reconstruction model provided in the embodiments of this disclosure.

[0189] When the computer program is executed by the processor 1101, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0190] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1109, and / or installed from the removable medium 1111. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0191] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 1109, and / or installed from removable medium 1111. When the computer program is executed by processor 1101, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0192] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0193] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0194] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0195] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A training method for an image super-resolution reconstruction model, characterized in that, The method includes: The first image is degraded to obtain a second image, the resolution of which is smaller than that of the first image; Feature extraction is performed on the second image using multiple convolutional branches to obtain a feature image. The multiple convolutional branches include a first multiple convolutional branch and a second multiple convolutional branch. The first multiple convolutional branch includes an edge extraction operator and a denoising operator. The feature image is upsampled to obtain the target image; Based on the loss value between the first image and the target image, the parameters of the initial image super-resolution reconstruction model are adjusted to obtain an intermediate image super-resolution reconstruction model, wherein the intermediate image super-resolution reconstruction model includes the multi-convolutional branch; The intermediate image super-resolution reconstruction model is subjected to reparameter processing to obtain the target image super-resolution reconstruction model; The step of extracting features from the second image using multiple convolutional branches to obtain a feature image includes: The first feature image is obtained by extracting features from the second image using the first multi-convolution branch; The first feature image is extracted using an enhanced residual block to obtain a second feature image, wherein the enhanced residual block includes a second multi-convolutional branch; The first feature image and the second feature image are subjected to residual addition processing to obtain the feature image; The step of performing reparameter reprocessing on the intermediate image super-resolution reconstruction model to obtain the target image super-resolution reconstruction model includes: Each convolutional branch of the first multi-convolutional branch of the intermediate image super-resolution reconstruction model is padded to obtain the first multi-convolutional branch of the target size. The first multi-convolutional branch of the target size is accumulated to obtain the first single convolution; The target image super-resolution reconstruction model is determined based on the first single convolution; Each of the convolutional branches includes at least one sub-convolution; The process of padding each convolutional branch of the first multi-convolutional branch of the intermediate image super-resolution reconstruction model to obtain the first multi-convolutional branch of the target size includes: In the case where the convolutional branch of the first multi-convolutional branch includes at least two of the sub-convolutions... Determine the channel expansion ratio; Based on the channel expansion ratio, the kernel weights and biases of each sub-convolution are linearly recombinated to obtain the first target kernel weights and the first target bias. The first multi-convolutional branch of the target size is determined based on the weight of the first target convolutional kernel and the first target bias.

2. The method according to claim 1, characterized in that, The step of accumulating the first multi-convolutional branches of the target size to obtain the first single convolution includes: Determine the kernel weights and biases of each of the first multi-convolutional branches in the target size; Each of the convolution kernel weights and the biases is added together to obtain the second target convolution kernel weights and the second target biases; The first single convolution is determined based on the second target convolution kernel weights and the second target bias.

3. The method according to claim 1, characterized in that, The enhanced residual block also includes multiple initial single convolutions; The step of performing reparameter reprocessing on the intermediate image super-resolution reconstruction model to obtain the target image super-resolution reconstruction model further includes: Each convolutional branch of the second multi-convolutional branch of the intermediate image super-resolution reconstruction model is padded to obtain the second multi-convolutional branch of the target size. The second multi-convolutional branch of the target size is accumulated to obtain the second single convolution; The second single convolution and the plurality of the initial single convolutions are accumulated to obtain the third single convolution; The target image super-resolution reconstruction model is determined based on the first single convolution and the third single convolution.

4. The method according to claim 1, characterized in that, The process of degrading the first image to obtain the second image includes: The first image is subjected to noise addition processing to obtain a noisy image; The noisy image is blurred to obtain a blurred image; The blurred image is downsampled to obtain the second image.

5. The method according to claim 4, characterized in that, The step of adding noise to the first image to obtain a noisy image includes: The first image is subjected to Gaussian noise processing to obtain the noisy image, wherein the Gaussian noise processing includes first Gaussian noise processing, second Gaussian noise processing and third Gaussian noise processing; the noisy image includes three channels. Wherein, the Gaussian noise in each channel image of the noise image corresponding to the first Gaussian noise processing is inconsistent; The Gaussian noise in each channel image of the noise image corresponding to the second Gaussian noise processing is consistent. The Gaussian noise in each channel image of the noise image corresponding to the third Gaussian noise processing is inconsistent, and the Gaussian noise in each channel image is dependent on each other.

6. An image super-resolution reconstruction method, characterized in that, The method includes: Obtain the image to be reconstructed; The image to be reconstructed is processed using the image super-resolution reconstruction model as described in any one of claims 1-5 to obtain a reconstructed image.

7. A training device for an image super-resolution reconstruction model, characterized in that, The device includes: A degradation module is used to perform degradation processing on a first image to obtain a second image, wherein the resolution of the second image is less than the resolution of the first image; The extraction module is used to extract features from the second image using multiple convolutional branches to obtain a feature image. The multiple convolutional branches include a first multiple convolutional branch and a second multiple convolutional branch. The first multiple convolutional branch includes an edge extraction operator and a denoising operator. A sampling module is used to perform upsampling processing on the feature image to obtain the target image; An adjustment module is used to adjust the parameters of an initial image super-resolution reconstruction model based on the loss value between the first image and the target image to obtain an intermediate image super-resolution reconstruction model, wherein the intermediate image super-resolution reconstruction model includes the multi-convolutional branch; and The processing module is used to perform reparameter processing on the intermediate image super-resolution reconstruction model to obtain the target image super-resolution reconstruction model. The extraction module includes: The first extraction submodule is used to extract features from the second image using the first multi-convolutional branch to obtain the first feature image; The second extraction submodule is used to extract features from the first feature image using the enhanced residual block to obtain the second feature image, wherein the enhanced residual block includes a second multi-convolutional branch; The first processing submodule performs residual addition processing on the first feature image and the second feature image to obtain the feature image; The processing module includes: The first filling submodule is used to fill each convolutional branch of the first multi-convolutional branch of the intermediate image super-resolution reconstruction model to obtain the first multi-convolutional branch of the target size. The first accumulation submodule is used to accumulate the first multi-convolutional branch of the target size to obtain the first single convolution; The first determining submodule is used to determine the target image super-resolution reconstruction model based on the first single convolution; The first filling submodule includes: The first determining unit is used to determine the channel expansion ratio; The combination unit is used to linearly recombine the kernel weights and biases of each sub-convolution according to the channel expansion ratio to obtain the first target kernel weights and the first target bias. The second determining unit is used to determine the first multi-convolutional branch of the target size based on the weights of the first target convolutional kernel and the first target bias.

Images