Training method and device of image super-resolution reconstruction model

Feature extraction is performed through multi-convolution branch and edge extraction operator, combined with upsampling and re-parameter processing, and the problem of difficulty in implementing image super-resolution reconstruction method on edge devices and insufficient visual effects is solved, and efficient image reconstruction is achieved .

CN118941896AActive Publication Date: 2024-11-12TIANJIN UNIV
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411167965.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-12
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

Existing image super-resolution reconstruction methods are difficult to implement on edge devices or mobile devices, and the reconstructed image edges and textures are not rich enough, making it difficult to simultaneously reduce model inference time and improve visual effects.

Method used

Multi-convolution branches are used for feature extraction, including edge extraction operators and denoising operators. The model parameters are adjusted through upsampling and re-parameter processing to form the super-resolution reconstruction model of the target image.

Benefits of technology

The visual effect of the reconstruction image is improved, the amount of model parameters is reduced, the inference time is shortened, and the efficient image super-resolution reconstruction is achieved on edge devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941896A_ABST
    Figure CN118941896A_ABST
Patent Text Reader

Abstract

The invention provides a training method and device for an image super-resolution reconstruction model, and can be applied to the technical field of image processing. The method comprises the steps that degradation processing is carried out on a first image to obtain a second image, and the resolution ratio of the second image is smaller than that of the first image; performing feature extraction on the second image by using multi-volume integral branches to obtain a feature image, the multi-volume integral branches including a first multi-volume integral branch and a second multi-volume integral branch, and the first multi-volume integral branch including an edge extraction operator and a denoising operator; performing up-sampling processing on the feature image to obtain a target image; according to the loss value between the first image and the target image, parameters of the initial image super-resolution reconstruction model are adjusted, an intermediate image super-resolution reconstruction model is obtained, and the intermediate image super-resolution reconstruction model comprises a multi-convolution branch; and performing re-parameter processing on the intermediate image super-resolution reconstruction model to obtain a target image super-resolution reconstruction model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and more specifically to a training method and device for an image super-resolution reconstruction model. Background Art

[0002] Image super-resolution reconstruction method refers to obtaining a higher resolution image corresponding to the target image by adopting an appropriate reconstruction algorithm. Super-resolution methods based on deep learning have been proposed one after another, but most of them have the characteristics of too many image parameters and complex network structure, which makes it difficult to implement super-resolution reconstruction tasks on edge devices or mobile devices.

[0003] In order to solve the above problems, the related technology adopts lightweight processing to reduce the number of model parameters and thus reduce the inference time. Although the model inference time is reduced, the edges and textures of the reconstructed image are not rich enough, making it difficult to achieve the effect of both reducing the model inference time and achieving good visual effects of the reconstructed image. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a method and device for training an image super-resolution reconstruction model.

[0005] According to a first aspect of the present disclosure, a training method for an image super-resolution reconstruction model is provided, comprising: performing degradation processing on a first image to obtain a second image, wherein the resolution of the second image is smaller than that of the first image; performing feature extraction on the second image using a multi-convolution branch to obtain a feature image, wherein the multi-convolution branch comprises a first multi-convolution branch and a second multi-convolution branch, and the first multi-convolution branch comprises an edge extraction operator and a denoising operator; performing upsampling processing on the feature image to obtain a target image; adjusting parameters of an initial image super-resolution reconstruction model according to a loss value between the first image and the target image to obtain an intermediate image super-resolution reconstruction model, wherein the intermediate image super-resolution reconstruction model comprises a multi-convolution branch; and performing parameter re-processing on the intermediate image super-resolution reconstruction model to obtain a target image super-resolution reconstruction model.

[0006] According to an embodiment of the present disclosure, features of the second image are extracted using multiple convolution branches to obtain a feature image, including: extracting features of the second image using a first multiple convolution branch to obtain a first feature image; extracting features of the first feature image using an enhanced residual block to obtain a second feature image, wherein the enhanced residual block includes a second multiple convolution branch; performing residual addition processing on the first feature image and the second feature image to obtain a feature image.

[0007] According to an embodiment of the present disclosure, the intermediate image super-resolution reconstruction model is re-parameterized to obtain a target image super-resolution reconstruction model, including: performing padding processing on each convolution branch of the first multi-convolution branch of the intermediate image super-resolution reconstruction model to obtain a first multi-convolution branch of the target size; performing accumulation processing on the first multi-convolution branch of the target size to obtain a first single convolution; and determining the target image super-resolution reconstruction model based on the first single convolution.

[0008] According to an embodiment of the present disclosure, each convolution branch includes at least one sub-convolution; each convolution branch of the first multi-convolution branch of the intermediate image super-resolution reconstruction model is filled respectively to obtain a first multi-convolution branch of the target size, including: when the convolution branch of the first multi-convolution branch includes at least two sub-convolutions, determining the channel expansion ratio; according to the channel expansion ratio, linearly recombine the convolution kernel weights and bias of each sub-convolution respectively to obtain a first target convolution kernel weight and a first target bias; determine the first multi-convolution branch of the target size according to the first target convolution kernel weight and the first target bias.

[0009] According to an embodiment of the present disclosure, the first multi-convolution branches of the target size are accumulated to obtain a first single convolution, including: determining the convolution kernel weight and bias of each convolution branch in the first multi-convolution branches of the target size; adding each convolution kernel weight and bias respectively to obtain a second target convolution kernel weight and a second target bias; determining the first single convolution according to the second target convolution kernel weight and the second target bias.

[0010] According to an embodiment of the present disclosure, the enhanced residual block also includes multiple initial single convolutions; the intermediate image super-resolution reconstruction model is re-parameterized to obtain a target image super-resolution reconstruction model, and also includes: filling each convolution branch of the second multi-convolution branch of the intermediate image super-resolution reconstruction model respectively to obtain a second multi-convolution branch of the target size; accumulating the second multi-convolution branches of the target size to obtain a second single convolution; accumulating the second single convolution and multiple initial single convolutions to obtain a third single convolution; determining the target image super-resolution reconstruction model based on the first single convolution and the third single convolution.

[0011] According to an embodiment of the present disclosure, a first image is degraded to obtain a second image, including: adding noise to the first image to obtain a noisy image; blurring the noisy image to obtain a blurred image; and downsampling the blurred image to obtain the second image.

[0012] According to an embodiment of the present disclosure, a noise adding process is performed on a first image to obtain a noise image, including: performing Gaussian noise processing on the first image to obtain a noise image, wherein the Gaussian noise processing includes a first Gaussian noise processing, a second Gaussian noise processing and a third Gaussian noise processing; the noise image includes three channel images; wherein the Gaussian noise of each channel image in the noise image corresponding to the first Gaussian noise processing is inconsistent; the Gaussian noise of each channel image in the noise image corresponding to the second Gaussian noise processing is consistent; the Gaussian noise of each channel image in the noise image corresponding to the third Gaussian noise processing is inconsistent, and there is a dependency between the Gaussian noises of each channel image.

[0013] A second aspect of the present disclosure provides an image super-resolution reconstruction method, comprising: acquiring an image to be reconstructed; and processing the image to be reconstructed using the above-mentioned image super-resolution reconstruction model to obtain a reconstructed image.

[0014] The third aspect of the present disclosure provides a training device for an image super-resolution reconstruction model, including: a degradation module, used to perform degradation processing on a first image to obtain a second image, the resolution of the second image is smaller than the resolution of the first image; an extraction module, used to use multiple convolution branches to extract features of the second image to obtain a feature image, wherein the multiple convolution branches include a first multi-convolution branch and a second multi-convolution branch, and the first multi-convolution branch includes an edge extraction operator and a denoising operator; a sampling module, used to upsample the feature image to obtain a target image; an adjustment module, used to adjust the parameters of the initial image super-resolution reconstruction model according to the loss value between the first image and the target image to obtain an intermediate image super-resolution reconstruction model, wherein the intermediate image super-resolution reconstruction model includes multiple convolution branches; and a processing module, used to re-parameterize the intermediate image super-resolution reconstruction model to obtain a target image super-resolution reconstruction model.

[0015] A fourth aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0016] The fifth aspect of the present disclosure further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

[0017] The sixth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0018] According to the embodiments of the present disclosure, multiple convolution branches are used to extract features from the second image to obtain a feature image; the feature image is upsampled to obtain a target image; the parameters of the initial image super-resolution reconstruction model are adjusted according to the loss value between the first image and the target image to obtain an intermediate image super-resolution reconstruction model, and re-parameterization is performed to obtain a target image super-resolution reconstruction model. Since the feature extraction of the second image is performed using multiple convolution branches and their edge extraction operators and denoising operators, the problem of insufficient extraction of image edges and textures is avoided, and the technical effect of improving the visual effect of the reconstructed image is achieved. At the same time, since the intermediate image super-resolution reconstruction model is re-parameterized, the problem of excessive model parameters is avoided, and the technical effect of reducing the model inference time is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above contents and other purposes, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0020] Figure 1 A diagram schematically illustrates an application scenario of a training method, apparatus, device, medium, and program product for an image super-resolution reconstruction model according to an embodiment of the present disclosure;

[0021] Figure 2 A flowchart of a method for training an image super-resolution reconstruction model according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 3 A schematic diagram of a first multi-branch convolution according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 4 A schematic diagram of an enhanced residual block according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 5 A schematic diagram of an image super-resolution reconstruction model according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 6 A schematic diagram of heavy parameter processing according to an embodiment of the present disclosure is schematically shown;

[0026] Figure 7 A schematic diagram schematically shows a training method for an image super-resolution reconstruction model according to an embodiment of the present disclosure;

[0027] Figure 8 A flowchart of an image super-resolution reconstruction method according to an embodiment of the present disclosure is schematically shown;

[0028] Fig. 9The structure block diagram of the training device of the image super-resolution reconstruction model according to the embodiment of the present disclosure is schematically shown;

[0029] Fig.10 A structural block diagram of an image super-resolution reconstruction device according to an embodiment of the present disclosure is schematically shown; and

[0030] Fig.11 A block diagram of an electronic device suitable for implementing a training method for an image super-resolution reconstruction model according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0031] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0032] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0033] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0034] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0035] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0036] An embodiment of the present disclosure provides a training method for an image super-resolution reconstruction model, which performs degradation processing on a first image to obtain a second image, wherein the resolution of the second image is smaller than that of the first image; uses a multi-convolution branch to extract features of the second image to obtain a feature image, wherein the multi-convolution branch includes a first multi-convolution branch and a second multi-convolution branch, and the first multi-convolution branch includes an edge extraction operator and a denoising operator; performs upsampling processing on the feature image to obtain a target image; adjusts the parameters of an initial image super-resolution reconstruction model according to a loss value between the first image and the target image to obtain an intermediate image super-resolution reconstruction model, wherein the intermediate image super-resolution reconstruction model includes a multi-convolution branch; and performs parameter re-processing on the intermediate image super-resolution reconstruction model to obtain a target image super-resolution reconstruction model.

[0037] Figure 1 The application scenario diagram of the training method, apparatus, device, medium and program product of the image super-resolution reconstruction model according to the embodiments of the present disclosure is schematically shown.

[0038] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0039] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0040] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0041] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0042] It should be noted that the training method of the image super-resolution reconstruction model provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the training device of the image super-resolution reconstruction model provided in the embodiment of the present disclosure can generally be set in the server 105. The training method of the image super-resolution reconstruction model provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the training device of the image super-resolution reconstruction model provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0043] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0044] The following will be based on Figure 1 The scene described by Figure 2~Figure 7 The training method of the image super-resolution reconstruction model of the disclosed embodiment is described in detail.

[0045] Figure 2 The flowchart of the training method of the image super-resolution reconstruction model according to the embodiment of the present disclosure is schematically shown.

[0046] like Figure 2 As shown, the training method of the image super-resolution reconstruction model of this embodiment includes operations S210 to S250.

[0047] In operation S210, a degradation process is performed on the first image to obtain a second image, where a resolution of the second image is smaller than a resolution of the first image.

[0048] According to an embodiment of the present disclosure, the first image may be an image obtained by extracting frames from a real electronic endoscopic surgery video that does not contain patient privacy, and using bicubic downsampling on the extracted frames to obtain an image that is not cluttered or blurred, that is, a high-resolution electronic endoscopic image.

[0049] According to an embodiment of the present disclosure, the first image may also be a high-resolution image in a DIV2K dataset in the field of image restoration.

[0050] According to an embodiment of the present disclosure, the degradation process may be a process of reducing the image resolution, and a low-resolution second image is obtained by performing the degradation process on a high-resolution first image.

[0051] According to an embodiment of the present disclosure, the obtained second image may be randomly rotated by 90 degrees, 180 degrees, and 270 degrees for data enhancement.

[0052] In operation S220, features are extracted from the second image using a multi-convolution branch to obtain a feature image, wherein the multi-convolution branch includes a first multi-convolution branch and a second multi-convolution branch, and the first multi-convolution branch includes an edge extraction operator and a denoising operator.

[0053] According to an embodiment of the present disclosure, the multi-convolution branch realizes multi-channel processing, extracts features of the second image in each convolution branch, and then adds the results of each convolution branch as output to increase the number of model parameters and make model training more stable. The first multi-convolution branch and the second multi-convolution branch are both used to extract features of the second image.

[0054] According to an embodiment of the present disclosure, an edge extraction operator is used to identify places in an image where brightness changes significantly, and these places usually represent boundaries between different areas in the image.

[0055] According to an embodiment of the present disclosure, a denoising operator is used to reduce random variations in an image (ie, image noise) while retaining the details of the image.

[0056] According to an embodiment of the present disclosure, edge extraction operators and denoising operators are added to the multi-convolution branches, so that the model is more sensitive to the edge information of the image.

[0057] In operation S230, up-sampling is performed on the feature image to obtain a target image.

[0058] According to an embodiment of the present disclosure, upsampling processing is completed through a third multi-convolution branch, a PixelShuffle layer and an activation function, wherein the third multi-convolution branch has the same structure as the first multi-convolution branch, the number of input channels of the third multi-convolution branch is 32, and the number of output channels is the number of model input channels multiplied by the square of the magnification factor. For example, the number of model input channels is 3, the magnification factor of the PixelShuffle layer is 2, and the number of output channels is 12.

[0059] According to an embodiment of the present disclosure, the upsampling process enlarges the size of the feature image to a higher resolution to obtain a target image.

[0060] In operation S240, parameters of the initial image super-resolution reconstruction model are adjusted according to the loss value between the first image and the target image to obtain an intermediate image super-resolution reconstruction model, wherein the intermediate image super-resolution reconstruction model includes multiple convolution branches.

[0061] According to an embodiment of the present disclosure, the loss value is obtained by determining the absolute difference between the first image and the target image.

[0062] According to the embodiment of the present disclosure, the parameters of the initial image super-resolution reconstruction model are adjusted based on the obtained loss value, wherein the batch selected during training is 32, the patch is 64, and the learning rate is The training rounds are 800, and the Adam optimizer is used to calculate the gradient and update the model parameters so that the target image output by the intermediate image super-resolution reconstruction model is closer to the first image.

[0063] In operation S250, the intermediate image super-resolution reconstruction model is re-parameterized to obtain a target image super-resolution reconstruction model.

[0064] According to an embodiment of the present disclosure, the re-parameterization process may be to transform the intermediate image super-resolution reconstruction model to reduce the amount of model parameters, thereby shortening the inference time of the target image super-resolution reconstruction model.

[0065] According to the embodiments of the present disclosure, multiple convolution branches are used to extract features from the second image to obtain a feature image; the feature image is upsampled to obtain a target image; the parameters of the initial image super-resolution reconstruction model are adjusted according to the loss value between the first image and the target image to obtain an intermediate image super-resolution reconstruction model, and re-parameterization is performed to obtain a target image super-resolution reconstruction model. Since the feature extraction of the second image is performed using multiple convolution branches and their edge extraction operators and denoising operators, the problem of insufficient extraction of image edges and textures is avoided, and the technical effect of improving the visual effect of the reconstructed image is achieved. At the same time, since the intermediate image super-resolution reconstruction model is re-parameterized, the problem of excessive model parameters is avoided, and the technical effect of reducing the model inference time is achieved.

[0066] According to an embodiment of the present disclosure, features of the second image are extracted using multiple convolution branches to obtain a feature image, including: extracting features of the second image using a first multiple convolution branch to obtain a first feature image; extracting features of the first feature image using an enhanced residual block to obtain a second feature image, wherein the enhanced residual block includes a second multiple convolution branch; performing residual addition processing on the first feature image and the second feature image to obtain a feature image.

[0067] According to an embodiment of the present disclosure, the first multi-convolution branch includes a plurality of convolution branches, each convolution branch includes at least one sub-convolution, wherein a plurality of sub-convolution branches are added to the individual convolution branches. Convolution is used to increase the number of channels by a factor of k. The value of k can be selected according to actual needs, and k=2 is preferred.

[0068] According to the embodiments of the present disclosure, since the intermediate image super-resolution reconstruction model is subsequently re-parameterized, multiple convolution branches need to be converted into a single convolution branch, so it is necessary to restore the number of channels to the same as other convolution branches after the number of channels is increased, and finally add Convolution or Convolution realizes the compression of convolution branch channels.

[0069] Figure 3 A schematic diagram of a first multi-branch convolution according to an embodiment of the present disclosure is schematically shown.

[0070] According to the embodiments of the present disclosure, Figure 3 As shown, the first multi-convolution branch 301 includes convolution, convolution, convolution, convolution, convolution- convolution, Convolution - Special Operator - In addition, the first multi-convolution branch 301 also includes a residual connection, which can be regarded as a special case where the middle is 1 and the rest is 0. Convolution, the input end of the residual connection is connected to the above convolution, convolution, convolution, convolution, convolution- convolution, Convolution - Special Operator - The output ends of the convolutions are connected. Each convolution branch performs feature extraction on the second image, and inputs the output results of each convolution branch into the residual connection. The residual connection adds the output results of each convolution branch, and inputs the addition result into the activation function (Parametric Rectified Linear Unit, PReLU) 302 for processing to obtain the first feature image.

[0071] According to an embodiment of the present disclosure, the special operator includes an edge extraction operator (Sobel operator , , Laplacian operator , sharpening operator ) and denoising operator (Gaussian kernel operator ). Among them, the specific parameters of special operators are , , , , , and each special operator is accompanied by a scaling factor for the model to learn.

[0072] According to the embodiment of the present disclosure, the number of input channels of each convolution branch is 3, the number of output channels is 32, and the step length is 1. The padding of the convolution is 0, The padding of the convolution is , The padding of the convolution is , The padding of the convolution is . convolution- In convolution The number of convolution output channels is 64, and the padding is , The number of convolution input channels is 64, and the padding is . Convolution - Special Operator - The first convolution The number of output channels of the convolution is 64, and the padding is ; The special operator uses depth convolution, that is, the number of input channels and the number of output channels are both 64, and the padding is ; The second The number of input channels of the convolution is 64, and the padding is .

[0073] According to the embodiments of the present disclosure, the increase in the number of output channels can enhance the expressiveness of the model to capture complex features. Convolution can realize linear combination of features and dimensionality reduction or dimension increase without changing the spatial size of the input feature map.

[0074] Figure 4 A schematic diagram of an enhanced residual block according to an embodiment of the present disclosure is schematically shown.

[0075] According to the embodiment of the present disclosure, there may be multiple enhanced residual blocks, the number of which may be selected according to actual needs, preferably 4. Figure 4 As shown, the enhanced residual block includes a second multi-convolution branch 402 and also includes multiple initial single convolutions 401, such as convolution, and two skip connections 404 and an activation function 403.

[0076] According to an embodiment of the present disclosure, the second multi-convolution branch has a similar structure to the first multi-convolution branch, and the special operator of the second multi-convolution branch may not include the Gaussian kernel operator in the first multi-convolution branch. , the rest can be consistent with the first multi-convolution branch and will not be repeated here.

[0077] According to an embodiment of the present disclosure, the initial single convolution may be two, connected in series with the second multi-convolution branch, wherein the first Convolution is used to expand the number of channels. The number of input channels is 32 and the number of output channels is 64. The second multi-convolution branch has 64 input channels and 64 output channels. The second Convolution is used to compress the number of channels, with 64 input channels and 32 output channels.

[0078] According to an embodiment of the present disclosure, the jump connection can enable the input first feature image to skip one or more layers and be directly passed from the input end to the subsequent layers, avoiding the gradient vanishing problem during the model training process, thereby enabling the model to train deep structures more effectively.

[0079] According to the embodiments of the present disclosure, multiple convolution branches and their edge extraction operators and denoising operators are used to extract features from the second image, thereby avoiding the problem of insufficient extraction of image edges and textures, and achieving the technical effect of improving the visual effect of the reconstructed image.

[0080] Figure 5 A schematic diagram of an image super-resolution reconstruction model according to an embodiment of the present disclosure is schematically shown.

[0081] According to the embodiments of the present disclosure, Figure 5 As shown, the second image 501 is input into the first multi-branch convolution 502 of the image super-resolution reconstruction model for feature extraction, and the obtained first feature image is input into the enhanced residual block 503, and the second feature image is output; the first feature image is also input into the residual connection 505 together with the second feature image through the jump connection 504, and the feature image is output; after sampling by the upsampling module 506, the target image 507 is obtained.

[0082] According to an embodiment of the present disclosure, the intermediate image super-resolution reconstruction model is re-parameterized to obtain a target image super-resolution reconstruction model, including: performing padding processing on each convolution branch of the first multi-convolution branch of the intermediate image super-resolution reconstruction model to obtain a first multi-convolution branch of the target size; performing accumulation processing on the first multi-convolution branch of the target size to obtain a first single convolution; and determining the target image super-resolution reconstruction model based on the first single convolution.

[0083] According to an embodiment of the present disclosure, the filling process may be to fill the image with a size that is not the target size ( ) is expanded to , that is, in convolution, Convolution and The convolution is padded with 0 to expand its size to .

[0084] According to an embodiment of the present disclosure, the accumulation processing may be to accumulate multiple convolution branches to obtain a single convolution. The obtained first single convolution has a smaller parameter amount than the first multiple convolution branches of the target size, so that the target image super-resolution reconstruction model reduces the model inference time to a certain extent compared with the intermediate image super-resolution reconstruction model.

[0085] According to an embodiment of the present disclosure, each convolution branch includes at least one sub-convolution; each convolution branch of the first multi-convolution branch of the intermediate image super-resolution reconstruction model is filled respectively to obtain a first multi-convolution branch of the target size, including: when the convolution branch of the first multi-convolution branch includes at least two sub-convolutions, determining the channel expansion ratio; according to the channel expansion ratio, linearly recombine the convolution kernel weights and bias of each sub-convolution respectively to obtain a first target convolution kernel weight and a first target bias; determine the first multi-convolution branch of the target size according to the first target convolution kernel weight and the first target bias.

[0086] According to an embodiment of the present disclosure, the channel expansion ratio is used to amplify the number of channels to increase the amount of model parameters, and can be set according to actual needs. Preferably, the channel expansion ratio k=2.

[0087] According to an embodiment of the present disclosure, linear recombination can be to reorganize the convolution kernel weights and biases of multiple sub-convolutions in series respectively to obtain a first target convolution kernel weight and a first target bias, so as to realize the conversion of the multiple sub-convolutions in series into a first multi-convolution branch of a target size.

[0088] According to an embodiment of the present disclosure, for Serial convolution, assuming that the number of input channels of this serial convolution is , the number of output channels is , the channel expansion ratio is k. Convolution kernel weights The shape is , bias The shape is ; Convolution kernel weights The shape is , bias The shape is . After conversion, The first target convolution kernel weight of the convolution is K, the first target bias is B, the input data is I, and the shape is , represents the convolution operation, as shown in formula (1).

[0089] (1);

[0090] in, Convolution only performs linear combination of channels without performing spatial aggregation, which can be achieved by linearly recombining The parameters in the convolution kernel are combined, which can be done by transposing, that is, The first two dimensions of , whose shape is , from which we can infer: , it can be seen that the shape of the first target convolution kernel weight K after merging is .

[0091] For the acquisition of the first target bias B, since the broadcast operation is adopted when the bias is added, it should be The elements in are copied and expanded into , whose shape is , so there is , it can be seen that the shape of the merged first target offset B is .

[0092] Based on the first target convolution kernel weight , first target offset Get the first multi-convolution branch of the target size to achieve - Transformation of cascaded convolutions.

[0093] According to an embodiment of the present disclosure, when the subconvolution is a special operator, for example, -Special Operators- Cascade convolution. Assume that the number of input channels of this cascade convolution is , the number of output channels is , the channel expansion ratio is k. The first Convolution kernel weights The shape is , bias The shape is ; Special operator convolution kernel weight The shape is , bias The shape is ; The second convolution kernel weight The shape is , bias The shape is The converted one The first target convolution kernel weight of the convolution is K, the first target bias is B, the input data is I, and the shape is , represents the convolution operation, as shown in formula (2).

[0094] (2);

[0095] The first two concatenated convolutions (i.e. -Special operator series convolution) into a single convolution, that is, -Special operator concatenation converts convolution into a single Convolution kernel weights and bias , and the above - The concatenated convolution is converted into The only difference in convolution is that the special operator convolution kernel needs to be copied and expanded to , whose shape is , others also need to The first two dimensions of , whose shape is , The elements in are copied and expanded into , whose shape is , as shown in formula (3).

[0096] (3);

[0097] It can be seen that -Special operator concatenated convolution kernel after convolution transformation Shape , The shape is , in order to achieve -Special operator concatenation converts convolution into a single convolution.

[0098] Then - The cascaded convolution is transformed into Convolution, as shown in formula (4).

[0099] (4);

[0100] First The first two dimensions of , whose shape is ,use Performing convolution operation on it yields: Then The first two dimensions of are swapped to obtain the first target convolution kernel weight K, whose shape is , the first target bias B is given by right Perform convolution operation and add Get, that is, , to achieve -Special Operators- The concatenated convolution is converted into The first multi-convolution branch of target size.

[0101] According to an embodiment of the present disclosure, the first multi-convolution branches of the target size are accumulated to obtain a first single convolution, including: determining the convolution kernel weight and bias of each convolution branch in the first multi-convolution branches of the target size; adding each convolution kernel weight and bias respectively to obtain a second target convolution kernel weight and a second target bias; determining the first single convolution according to the second target convolution kernel weight and the second target bias.

[0102] According to an embodiment of the present disclosure, the convolution kernel weight and bias of each convolution branch in the first multi-convolution branch of the target size are determined, and the convolution kernel weights and biases on all branches are added together to obtain the second target convolution kernel weight and the second target bias. Because the convolution calculation process is a linear calculation, the result obtained by calculating the input data with the convolution kernel weight and bias of the first single convolution is consistent with the result obtained by calculating the weight and bias of the first multi-convolution branch.

[0103] According to an embodiment of the present disclosure, assuming that the number of convolution branches is 3, the weights of the convolution branches are k1, k2, k3, the biases are b1, b2, b3, the second target convolution kernel weight is K, the second target bias is B, the input data is I, represents the convolution operation. The above process can be specifically shown as formula (5).

[0104] (5);

[0105] According to an embodiment of the present disclosure, the enhanced residual block also includes multiple initial single convolutions; the intermediate image super-resolution reconstruction model is re-parameterized to obtain a target image super-resolution reconstruction model, and also includes: filling each convolution branch of the second multi-convolution branch of the intermediate image super-resolution reconstruction model respectively to obtain a second multi-convolution branch of the target size; accumulating the second multi-convolution branches of the target size to obtain a second single convolution; accumulating the second single convolution and multiple initial single convolutions to obtain a third single convolution; determining the target image super-resolution reconstruction model based on the first single convolution and the third single convolution.

[0106] According to an embodiment of the present disclosure, each convolution branch of the second multi-convolution branch is filled separately, which is the same as the filling process of each convolution branch of the first multi-convolution branch, and will not be repeated here.

[0107] According to an embodiment of the present disclosure, the accumulation processing performed on the second multi-convolution branch of the target size is the same as the accumulation processing performed on the first multi-convolution branch of the target size, and will not be repeated here.

[0108] According to an embodiment of the present disclosure, the second single convolution and multiple initial single convolutions are accumulated, and the initial single convolution can be filled with After the convolution, the convolution kernel weights and biases of the initial convolution kernel and the second convolution kernel are added to obtain the third convolution kernel.

[0109] According to the embodiments of the present disclosure, for the residual connection, it can be regarded as a special case where the middle is 1 and the rest are 0. Convolution, as a convolution branch, performs accumulation processing.

[0110] Figure 6 The figure schematically shows a schematic diagram of re-parameter processing according to an embodiment of the present disclosure.

[0111] According to the embodiments of the present disclosure, Figure 6 As shown, the enhanced residual block of the intermediate image super-resolution reconstruction model is shown in 601, and each convolution branch of its second multi-convolution branch is filled respectively, so that each convolution branch is as shown in 602 Convolution; performing accumulation processing on the second multi-convolution branches of the target size, merging the multiple convolution branches into a second single convolution as shown in 603; then, performing accumulation processing on the initial single convolution and the second single convolution of the enhanced residual block, obtaining a result as shown in 604 Convolution, and finally remove the skip connection to obtain the third single convolution as shown in 605.

[0112] According to the embodiments of the present disclosure, tests were conducted on a test platform. Through heavy parameter processing, the number of model parameters was significantly reduced from 290260 to 60076, and the average time used for inference output of a 1080p image was reduced from 0.14s to 30.10ms.

[0113] According to the embodiments of the present disclosure, by re-parameterizing the intermediate image super-resolution reconstruction model, the problem of excessive model parameters is avoided, thereby achieving the technical effect of reducing the model inference time.

[0114] According to an embodiment of the present disclosure, a first image is degraded to obtain a second image, including: adding noise to the first image to obtain a noisy image; blurring the noisy image to obtain a blurred image; and downsampling the blurred image to obtain the second image.

[0115] According to an embodiment of the present disclosure, row noise addition processing can be to select a plurality of processing methods to add noise to the first image, for example, Gaussian noise processing, Poisson noise processing and JPEG compression noise processing, wherein each type of noise is randomly generated according to certain parameters, for example, for a certain number of first images, 40% of the first images are subjected to Gaussian noise processing, 30% of the first images are subjected to Poisson noise processing, and 30% of the first images are subjected to JPEG compression noise processing, and the parameters can be adjusted according to actual needs.

[0116] According to an embodiment of the present disclosure, the Poisson noise processing adds noise to the first image through formula (6).

[0117] (6);

[0118] in, Poisson distribution with Poisson noise processing, parameters is the average number of random events per unit time. is the actual number of occurrences of a random event per unit time, and the random event is the deviation of the image pixel value. For Poisson noise processing, It is the product of the pixel value of the first image and an amplification parameter to make the noise sampling more fluctuating and the generated noise more obvious.

[0119] According to an embodiment of the present disclosure, each pixel in the three RGB channels of the first image is randomly sampled and divided by the magnification parameter Ensure that the pixel value does not overflow, thereby generating a three-channel Poisson noise consistent with the dimension of the first image to obtain a noisy image.

[0120] According to an embodiment of the present disclosure, JPEG compression noise processing compresses a first image by randomly generating different quality factors, and then decompresses the compressed first image to obtain different degrees of JPEG compression noise, wherein in the actual process of generating a noise image, the quality factor is selected in the range of 30 to 90.

[0121] According to an embodiment of the present disclosure, blur processing is divided into Gaussian isotropic blur processing and Gaussian anisotropic blur processing according to the size of the blur kernel, wherein the size of the blur kernel is selected from all odd numbers within the range of 7-15.

[0122] According to an embodiment of the present disclosure, the downsampling process includes nearest neighbor downsampling, bilinear downsampling and bicubic downsampling.

[0123] According to an embodiment of the present disclosure, noise adding processing, blurring processing and down-sampling processing are performed on the first image to reduce the resolution of the first image.

[0124] According to another embodiment of the present disclosure, the order of the above three degradation processing operations can be random, for example, blurring the first image first, adding noise to the blurred image, and finally downsampling. The order of the above three degradation processing operations can also be adjusted according to actual needs.

[0125] According to an embodiment of the present disclosure, a noise adding process is performed on a first image to obtain a noise image, including: performing Gaussian noise processing on the first image to obtain a noise image, wherein the Gaussian noise processing includes a first Gaussian noise processing, a second Gaussian noise processing and a third Gaussian noise processing; the noise image includes three channel images; wherein the Gaussian noise of each channel image in the noise image corresponding to the first Gaussian noise processing is inconsistent; the Gaussian noise of each channel image in the noise image corresponding to the second Gaussian noise processing is consistent; the Gaussian noise of each channel image in the noise image corresponding to the third Gaussian noise processing is inconsistent, and there is a dependency between the Gaussian noises of each channel image.

[0126] According to an embodiment of the present disclosure, Gaussian noise processing may be adding noise to the first image through formula (7).

[0127] (7);

[0128] in, represents the mean, n represents the number of dimensions of Gaussian noise, represents the covariance matrix, since the mean are all set to zero, so according to n and According to the difference in , it can be divided into three types of Gaussian noise.

[0129] According to an embodiment of the present disclosure, the first Gaussian noise processing dimension is three-dimensional, and the covariance matrix ,in is a preset variance parameter, which is randomly selected within the preset variance range. for The unit matrix of RGB is used to generate the corresponding Gaussian distribution function. The generated Gaussian noise is independent among the three RGB channels, that is, the Gaussian noise of the three channels is inconsistent. Each pixel point in the three RGB channels of the first image is randomly sampled according to this Gaussian distribution to generate three-channel Gaussian noise consistent with the dimension of the first image.

[0130] According to an embodiment of the present disclosure, the second Gaussian noise processing dimension is three-dimensional, and the covariance matrix ,in is a preset variance parameter, which is randomly selected within the preset variance range. , where all elements are 1 The matrix is ​​used to generate the corresponding Gaussian distribution function. The generated Gaussian noise is consistent in the three RGB channels. Each pixel point in the three RGB channels of the first image is randomly sampled according to the Gaussian distribution to generate three-channel Gaussian noise consistent with the dimension of the first image.

[0131] According to an embodiment of the present disclosure, the third Gaussian noise processing dimension is three-dimensional, and the covariance matrix ,in is a preset variance parameter, which is randomly selected within the preset variance range. Randomly generated A real symmetric matrix is ​​used to generate a corresponding Gaussian distribution function. The generated Gaussian noise has dependence among the three RGB channels, that is, the three channels are inconsistent and the channels are correlated with each other. Each pixel point in the three RGB channels of the first image is randomly sampled according to this Gaussian distribution to generate three-channel Gaussian noise consistent with the dimension of the first image.

[0132] According to an embodiment of the present disclosure, the above preset variance range is {6 / 255, 7 / 255, ... 16 / 255}.

[0133] According to an embodiment of the present disclosure, the above three types of Gaussian noise processing are randomly processed on the first images according to certain parameters. For example, for a certain number of first images, 30% of the first images are processed with the first Gaussian noise, 40% of the first images are processed with the second Gaussian noise, and 30% of the first images are processed with the third Gaussian noise. The parameters can be adjusted according to actual needs.

[0134] Figure 7 A schematic diagram schematically shows a method for training an image super-resolution reconstruction model according to an embodiment of the present disclosure.

[0135] According to the embodiments of the present disclosure, Figure 7As shown, in operation S701, the first image is degraded to obtain a second image. In operation S702, the first multi-convolution branch is used to extract features from the second image to obtain a first feature image. In operation S703, the second multi-convolution branch is used to extract features from the first feature image to obtain a second feature image. In operation S704, a feature image is determined based on the first feature image and the second feature image. In operation S705, the feature image is upsampled to obtain a target image. In operation S706, the parameters of the initial image super-resolution reconstruction model are adjusted according to the target image to obtain an intermediate image super-resolution reconstruction model. In operation S707, the first multi-convolution branch of the intermediate image super-resolution reconstruction model is padded to obtain a first multi-convolution branch of the target size; in operation S708, the first multi-convolution branch of the target size is accumulated to obtain a first single convolution. In operation S709, each convolution branch of the second multi-convolution branch of the intermediate image super-resolution reconstruction model is padded to obtain a second multi-convolution branch of the target size; in operation S710, the second multi-convolution branch of the target size is accumulated to obtain a second single convolution; in operation S711, the second single convolution and multiple initial single convolutions are accumulated to obtain a third single convolution. In operation S712, the target image super-resolution reconstruction model is determined according to the first single convolution determined in operation S708 and the third single convolution determined in operation S711.

[0136] Figure 8 The flowchart of the image super-resolution reconstruction method according to the embodiment of the present disclosure is schematically shown.

[0137] The second aspect of the present disclosure provides an image super-resolution reconstruction method, such as Figure 8 As shown, the image super-resolution reconstruction method includes operations S810 to S820.

[0138] In operation S810, an image to be reconstructed is acquired.

[0139] In operation S820, the image to be reconstructed is processed using the above-mentioned image super-resolution reconstruction model to obtain a reconstructed image.

[0140] According to the embodiments of the present disclosure, the above-mentioned image super-resolution reconstruction model is used to process the image to be reconstructed, which reduces the model inference time and obtains a reconstructed image with good visual effect.

[0141] Based on the above-mentioned training method of the image super-resolution reconstruction model, the present disclosure also provides a training device for the image super-resolution reconstruction model. Fig. 9 The device is described in detail.

[0142] Fig. 9The structural block diagram of a training device for an image super-resolution reconstruction model according to an embodiment of the present disclosure is schematically shown.

[0143] like Fig. 9 As shown, the training device 900 for the image super-resolution reconstruction model of this embodiment includes a degradation module 910 , an extraction module 920 , a sampling module 930 , an adjustment module 940 and a processing module 950 .

[0144] The degradation module 910 is used to perform degradation processing on the first image to obtain a second image, wherein the resolution of the second image is smaller than the resolution of the first image. In one embodiment, the degradation module 910 can be used to perform the operation S210 described above, which will not be described in detail here.

[0145] The extraction module 920 is used to extract features from the second image using a multi-convolution branch to obtain a feature image, wherein the multi-convolution branch includes a first multi-convolution branch and a second multi-convolution branch, and the first multi-convolution branch includes an edge extraction operator and a denoising operator. In one embodiment, the extraction module 920 can be used to perform the operation S220 described above, which will not be repeated here.

[0146] The sampling module 930 is used to perform up-sampling processing on the feature image to obtain the target image. In one embodiment, the sampling module 930 can be used to perform the operation S230 described above, which will not be described in detail here.

[0147] The adjustment module 940 is used to adjust the parameters of the initial image super-resolution reconstruction model according to the loss value between the first image and the target image to obtain an intermediate image super-resolution reconstruction model, wherein the intermediate image super-resolution reconstruction model includes multiple convolution branches. In one embodiment, the adjustment module 940 can be used to perform the operation S240 described above, which will not be repeated here.

[0148] The processing module 950 is used to perform parameter reprocessing on the intermediate image super-resolution reconstruction model to obtain the target image super-resolution reconstruction model. In one embodiment, the processing module 950 can be used to perform the operation S250 described above, which will not be described in detail here.

[0149] According to the embodiments of the present disclosure, multiple convolution branches are used to extract features from the second image to obtain a feature image; the feature image is upsampled to obtain a target image; the parameters of the initial image super-resolution reconstruction model are adjusted according to the loss value between the first image and the target image to obtain an intermediate image super-resolution reconstruction model, and re-parameterization is performed to obtain a target image super-resolution reconstruction model. Since the feature extraction of the second image is performed using multiple convolution branches and their edge extraction operators and denoising operators, the problem of insufficient extraction of image edges and textures is avoided, and the technical effect of improving the visual effect of the reconstructed image is achieved. At the same time, since the intermediate image super-resolution reconstruction model is re-parameterized, the problem of excessive model parameters is avoided, and the technical effect of reducing the model inference time is achieved.

[0150] According to an embodiment of the present disclosure, any multiple modules of the degradation module 910, the extraction module 920, the sampling module 930, the adjustment module 940 and the processing module 950 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the degradation module 910, the extraction module 920, the sampling module 930, the adjustment module 940 and the processing module 950 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware and firmware or in a suitable combination of any of them. Alternatively, at least one of the degradation module 910 , the extraction module 920 , the sampling module 930 , the adjustment module 940 , and the processing module 950 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.

[0151] According to an embodiment of the present disclosure, the extraction module 920 includes a first extraction submodule, a second extraction submodule and a first processing submodule.

[0152] The first extraction submodule is used to extract features from the second image using the first multi-convolution branch to obtain a first feature image.

[0153] The second extraction submodule is used to perform feature extraction on the first feature image using an enhanced residual block to obtain a second feature image, wherein the enhanced residual block includes a second multi-convolution branch.

[0154] The first processing submodule performs residual addition processing on the first feature image and the second feature image to obtain a feature image.

[0155] According to an embodiment of the present disclosure, the processing module 950 includes a first filling submodule, a first accumulating submodule and a first determining submodule.

[0156] The first filling submodule is used to perform filling processing on each convolution branch of the first multi-convolution branch of the intermediate image super-resolution reconstruction model to obtain the first multi-convolution branch of the target size.

[0157] The first accumulation submodule is used to perform accumulation processing on the first multi-convolution branch of the target size to obtain a first single convolution.

[0158] The first determination submodule is used to determine a target image super-resolution reconstruction model according to the first single convolution.

[0159] According to an embodiment of the present disclosure, the first filling submodule includes a first determining unit, a combining unit, and a second determining unit.

[0160] The first determining unit is used to determine the channel expansion ratio.

[0161] The combining unit is used to linearly recombine the convolution kernel weight and bias of each sub-convolution according to the channel expansion ratio to obtain a first target convolution kernel weight and a first target bias.

[0162] The second determining unit is used to determine a first multi-convolution branch of a target size according to a first target convolution kernel weight and a first target bias.

[0163] According to an embodiment of the present disclosure, the first accumulation submodule includes a third determination unit, a first processing unit and a fourth determination unit.

[0164] The third determination unit is used to determine the convolution kernel weight and bias of each convolution branch in the first multi-convolution branch of the target size.

[0165] The first processing unit is used to perform addition processing on each convolution kernel weight and bias respectively to obtain a second target convolution kernel weight and a second target bias.

[0166] The fourth determination unit is used to determine the first single convolution according to the second target convolution kernel weight and the second target bias.

[0167] According to an embodiment of the present disclosure, the processing module 950 further includes a second filling submodule, a second accumulating submodule, a third accumulating submodule and a second determining submodule.

[0168] The second filling submodule is used to fill each convolution branch of the second multi-convolution branch of the intermediate image super-resolution reconstruction model to obtain a second multi-convolution branch of the target size.

[0169] The second accumulation submodule is used to perform accumulation processing on the second multi-convolution branch of the target size to obtain a second single convolution.

[0170] The third accumulation submodule is used to accumulate the second single convolution and multiple initial single convolutions to obtain a third single convolution.

[0171] The second determination submodule is used to determine the target image super-resolution reconstruction model according to the first single convolution and the third single convolution.

[0172] According to an embodiment of the present disclosure, the degradation module 910 includes a second processing submodule, a third processing submodule and a fourth processing submodule.

[0173] The second processing submodule is used to perform noise addition processing on the first image to obtain a noisy image.

[0174] The third processing submodule is used to perform blur processing on the noisy image to obtain a blurred image.

[0175] The fourth processing submodule is used to perform down-sampling processing on the blurred image to obtain a second image.

[0176] According to an embodiment of the present disclosure, the second processing submodule includes a second processing unit.

[0177] The second processing unit is used to perform Gaussian noise processing on the first image to obtain a noise image, wherein the Gaussian noise processing includes a first Gaussian noise processing, a second Gaussian noise processing and a third Gaussian noise processing; the noise image includes three channel images; wherein the Gaussian noise of each channel image in the noise image corresponding to the first Gaussian noise processing is inconsistent; the Gaussian noise of each channel image in the noise image corresponding to the second Gaussian noise processing is consistent; the Gaussian noise of each channel image in the noise image corresponding to the third Gaussian noise processing is inconsistent, and the Gaussian noise of each channel image is dependent on each other.

[0178] Fig.10 The structural block diagram of an image super-resolution reconstruction device according to an embodiment of the present disclosure is schematically shown.

[0179] like Fig.10 As shown, the image super-resolution reconstruction device 1000 includes an acquisition module 1010 and a reconstruction module 1020 .

[0180] The acquisition module 1010 is used to acquire the image to be reconstructed.

[0181] The reconstruction module 1020 is used to process the image to be reconstructed using the image super-resolution reconstruction model to obtain a reconstructed image.

[0182] Fig.11 A block diagram of an electronic device suitable for implementing a training method for an image super-resolution reconstruction model according to an embodiment of the present disclosure is schematically shown.

[0183] like Fig.11 As shown, the electronic device 1100 according to an embodiment of the present disclosure includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage part 1108 to a random access memory (RAM) 1103. The processor 1101 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include an onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0184] In RAM 1103, various programs and data required for the operation of electronic device 1100 are stored. Processor 1101, ROM 1102 and RAM 1103 are connected to each other through bus 1104. Processor 1101 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1102 and / or RAM 1103. It should be noted that the program can also be stored in one or more memories other than ROM 1102 and RAM 1103. Processor 1101 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0185] According to an embodiment of the present disclosure, the electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to the bus 1104. The electronic device 1100 may further include one or more of the following components connected to the input / output (I / O) interface 1105: an input portion 1106 including a keyboard, a mouse, etc.; an output portion 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1108 including a hard disk, etc.; and a communication portion 1109 including a network interface card such as a LAN card, a modem, etc. The communication portion 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the input / output (I / O) interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed, so that a computer program read therefrom is installed into the storage portion 1108 as needed.

[0186] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0187] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1102 and / or RAM 1103 described above and / or one or more memories other than ROM 1102 and RAM 1103.

[0188] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the training method of the image super-resolution reconstruction model provided by the embodiment of the present disclosure.

[0189] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1101. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0190] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1109, and / or installed from a removable medium 1111. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0191] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1109, and / or installed from the removable medium 1111. When the computer program is executed by the processor 1101, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0192] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0193] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0194] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0195] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A training method for an image super-resolution reconstruction model, It is characterized in that The method comprises: Performing degradation processing on the first image to obtain a second image, wherein a resolution of the second image is smaller than a resolution of the first image; Performing feature extraction on the second image using a multi-convolution branch to obtain a feature image, wherein the multi-convolution branch includes a first multi-convolution branch and a second multi-convolution branch, and the first multi-convolution branch includes an edge extraction operator and a denoising operator; Performing upsampling processing on the feature image to obtain a target image; According to the loss value between the first image and the target image, adjusting the parameters of the initial image super-resolution reconstruction model to obtain an intermediate image super-resolution reconstruction model, wherein the intermediate image super-resolution reconstruction model includes the multi-convolution branch; The intermediate image super-resolution reconstruction model is re-parameterized to obtain a target image super-resolution reconstruction model.

2. The method according to claim 1, It is characterized in that The step of extracting features from the second image using multiple convolution branches to obtain a feature image includes: Using the first multi-convolution branch to extract features from the second image to obtain a first feature image; Using an enhanced residual block to extract features from the first feature image to obtain a second feature image, wherein the enhanced residual block includes the second multi-convolution branch; Perform residual addition processing on the first feature image and the second feature image to obtain the feature image.

3. The method according to claim 2, It is characterized in that The step of re-parameterizing the intermediate image super-resolution reconstruction model to obtain a target image super-resolution reconstruction model includes: Performing padding processing on each convolution branch of the first multi-convolution branch of the intermediate image super-resolution reconstruction model to obtain the first multi-convolution branch of the target size; Performing accumulation processing on the first multi-convolution branches of the target size to obtain a first single convolution; According to the first single convolution, the target image super-resolution reconstruction model is determined.

4. The method according to claim 3, It is characterized in that Each of the convolution branches includes at least one subconvolution; The step of performing padding processing on each of the first multiple convolution branches of the intermediate image super-resolution reconstruction model to obtain the first multiple convolution branches of the target size includes: In the case where the convolution branch of the first multi-convolution branch includes at least two of the sub-convolutions, Determine the channel expansion ratio; According to the channel expansion ratio, the convolution kernel weight and the bias of each of the sub-convolutions are linearly recombined to obtain a first target convolution kernel weight and a first target bias; Determine the first multi-convolution branch of the target size according to the first target convolution kernel weight and the first target bias.

5. The method according to claim 3, It is characterized in that The step of performing accumulation processing on the first multiple convolution branches of the target size to obtain a first single convolution comprises: Determine a convolution kernel weight and a bias of each of the convolution branches in the first multiple convolution branches of the target size; Each of the convolution kernel weights and the bias is added together to obtain a second target convolution kernel weight and a second target bias; Determine the first single convolution according to the second target convolution kernel weight and the second target bias.

6. The method according to claim 3, It is characterized in that The enhanced residual block also includes a plurality of initial single convolutions; The step of re-parameterizing the intermediate image super-resolution reconstruction model to obtain a target image super-resolution reconstruction model further includes: Performing padding processing on each convolution branch of the second multi-convolution branch of the intermediate image super-resolution reconstruction model to obtain the second multi-convolution branch of the target size; Performing accumulation processing on the second multi-convolution branches of the target size to obtain a second single convolution; Accumulate the second single convolution and the multiple initial single convolutions to obtain a third single convolution; According to the first single convolution and the third single convolution, the target image super-resolution reconstruction model is determined.

7. The method according to claim 1, It is characterized in that The step of performing degradation processing on the first image to obtain the second image includes: Performing noise addition processing on the first image to obtain a noise image; Performing blur processing on the noise image to obtain a blurred image; Down-sampling is performed on the blurred image to obtain the second image.

8. The method according to claim 7, It is characterized in that The step of performing noise addition processing on the first image to obtain a noise image includes: Performing Gaussian noise processing on the first image to obtain the noise image, wherein the Gaussian noise processing includes first Gaussian noise processing, second Gaussian noise processing and third Gaussian noise processing; the noise image includes three channel images; Wherein, the Gaussian noise of each channel image in the noise image corresponding to the first Gaussian noise processing is inconsistent; The Gaussian noise of each channel image in the noise image corresponding to the second Gaussian noise processing is consistent; The Gaussian noise of each channel image in the noise image corresponding to the third Gaussian noise processing is inconsistent, and the Gaussian noise of each channel image is dependent on each other.

9. A method for image super-resolution reconstruction, It is characterized in that The method comprises: Acquire an image to be reconstructed; The image to be reconstructed is processed using the image super-resolution reconstruction model according to any one of claims 1 to 8 to obtain a reconstructed image.

10. A training device for an image super-resolution reconstruction model, It is characterized in that The device comprises: A degradation module, configured to perform degradation processing on the first image to obtain a second image, wherein the resolution of the second image is smaller than the resolution of the first image; An extraction module, configured to perform feature extraction on the second image using a multi-convolution branch to obtain a feature image, wherein the multi-convolution branch includes a first multi-convolution branch and a second multi-convolution branch, and the first multi-convolution branch includes an edge extraction operator and a denoising operator; A sampling module, used for performing up-sampling processing on the feature image to obtain a target image; an adjustment module, configured to adjust parameters of an initial image super-resolution reconstruction model according to a loss value between the first image and the target image to obtain an intermediate image super-resolution reconstruction model, wherein the intermediate image super-resolution reconstruction model includes the multi-convolution branch; and The processing module is used to perform parameter reprocessing on the intermediate image super-resolution reconstruction model to obtain a target image super-resolution reconstruction model.

Citation Information

Patent Citations

  • Image super-resolution reconstruction method and system, storage medium and intelligent terminal

    CN116757934A

  • Face super-resolution reconstruction method and device based on lightweight network, and medium

    CN116883299A

  • Method for scaling binocular image at any scale and model training method

    CN117094896A

  • Super-resolution reconstruction method and device for low-resolution hyperspectral image and medium

    CN118505506A

  • Label-free adaptive CT super-resolution reconstruction method, system and device based on generative network

    US20240169610A1