Image super-resolution method and device for local motion blur

Abstract

The present invention discloses an image super-resolution method and device for local motion blur, which relates to the field of image processing, including: obtaining a low-resolution image to be reconstructed and inputting it into a trained image super-resolution model, inputting the low-resolution image into an image selection module, obtaining image grouping labels, the image grouping labels including positive labels and negative labels, the positive labels corresponding to blurred image blocks, and the negative labels corresponding to other image blocks; according to the image grouping labels, the blurred image blocks corresponding to the positive labels and the other image blocks corresponding to the negative labels are respectively input into a positive label feature extraction branch and a negative label feature extraction branch to obtain positive label features and negative label features, the positive label features and the negative label features are spliced ​​through a splicing layer to obtain image features; the image features are reconstructed through an image reconstruction module to obtain a corresponding high-resolution image. The present invention solves the problem of poor performance and low efficiency of local motion blurred images in super-resolution reconstruction.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image super-resolution method and device oriented to local motion blur. Background Art

[0002] Image super-resolution (SR) aims to generate a corresponding high-resolution (HR) image with rich details based on an existing low-resolution (LR) image. Since image SR can effectively restore valuable high-frequency information, it is widely used in various computer vision tasks, including video surveillance, medical imaging, remote sensing images, and cultural relics restoration.

[0003] In real-world photography, local motion blur occurs due to the mixing between moving objects and stationary backgrounds during the exposure process. Existing image deblurring methods mainly focus on global deblurring, but global deblurring inadvertently affects the background clarity of locally blurred images and wastes unnecessary computation on sharp pixels, especially for high-resolution images. In recent years, some local motion deblurring methods have also emerged. The Local Blur-Aware Gated Network (LBAG) solves this problem by detecting local blurred areas for target processing and uses a gate structure to mitigate the impact of deblurring on non-blurred areas, but since the entire image is processed by the network, it still involves unnecessary computation. In addition, the reliance on Convolutional Neural Networks (CNNs) leads to limitations. The interaction between images and convolution kernels is content-independent and cannot capture long-distance dependencies between pixels. Recently, transformers are good at long-distance pixel interactions and have been successfully used in advanced visual tasks such as image classification, object detection, and image segmentation. However, transformers often require a lot of memory and extend inference time, especially when processing high-resolution images. Therefore, how to better balance the performance and efficiency of the model becomes one of the key issues in solving local motion blur. Summary of the invention

[0004] The purpose of this application is to propose an image super-resolution method and device for local motion blur in response to the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides an image super-resolution method for local motion blur, comprising the following steps:

[0006] An image super-resolution model for local motion blur is constructed and trained to obtain a trained image super-resolution model, wherein the image super-resolution model includes an image selection module, a feature extraction module and an image reconstruction module, wherein the image selection module includes an image encoder, three blur confidence predictors and a Gumbel-Softmax layer, and the feature extraction module includes a positive label feature extraction branch, a negative label feature extraction branch and a splicing layer;

[0007] A low-resolution image to be reconstructed is obtained and input into a trained image super-resolution model. The low-resolution image is input into an image selection module. A first coding feature map, a second coding feature map and a third coding feature map of different scales are obtained through an image encoder. The first coding feature map, the second coding feature map and the third coding feature map are respectively passed through corresponding fuzzy confidence predictors and Gumbel-Softmax layers to obtain image grouping labels. The image grouping labels include positive labels and negative labels. The positive labels correspond to fuzzy image blocks, and the negative labels correspond to other image blocks. According to the image grouping labels, the fuzzy image blocks corresponding to the positive labels and the other image blocks corresponding to the negative labels are respectively input into the positive label feature extraction branch and the negative label feature extraction branch to obtain positive label features and negative label features. The positive label features and the negative label features are spliced ​​through a splicing layer to obtain image features. The image features are passed through an image reconstruction module to reconstruct the corresponding high-resolution image.

[0008] Preferably, the image encoder comprises an embedding layer, a first transformer layer, a second transformer layer, a third transformer layer and a fourth transformer layer connected in sequence; the low-resolution image is divided into a plurality of image blocks through the embedding layer and an initial feature is generated through the first transformer layer, and the initial feature is expressed as ,in , and C represent the height, width and number of channels of the low-resolution image, respectively, and p represents the size of the image block. Represents a set of real numbers; the initial features corresponding to each image block pass through the second transformer layer, the third transformer layer, and the fourth transformer layer in turn, and output the first encoding feature map, the second encoding feature map, and the third encoding feature map of different scales respectively. The first encoding feature map, the second encoding feature map, and the third encoding feature map are respectively expressed as:

[0009] ;

[0010] ;

[0011] ;

[0012] in, represents the first encoding feature map, represents the second encoding feature map, represents the third encoding feature map, represents the second transformer layer, represents the third transformer layer, Represents the fourth transformer layer.

[0013] Preferably, the fuzzy confidence predictor includes a multi-layer perceptron and a Softmax layer, and the first encoding feature map, the second encoding feature map and the third encoding feature map are respectively input into the corresponding fuzzy confidence predictor to obtain the corresponding feature embedding, as shown in the following formula:

[0014] ;

[0015] in, represents the feature embedding corresponding to the first encoding feature map, the second encoding feature map, or the third encoding feature map, represents a multi-layer perceptron, represents the first coding feature map, the second coding feature map or the third coding feature map, i=1,2,3;

[0016] The feature embedding is input into the Softmax layer and the fuzzy confidence is predicted as shown in the following formula:

[0017] ;

[0018] in, represents the fuzzy confidence corresponding to the first coding feature map, the second coding feature map or the third coding feature map, represents the Softmax function, so we get the probability distribution ;

[0019] The probability distribution Input to the Gumbel-Softmax layer for hard decision making to obtain the image grouping label, as shown in the following formula:

[0020] ;

[0021] Where s represents the image grouping probability. If s is greater than or equal to the threshold, the image grouping label is a positive label. If s is less than the threshold, the image grouping label is a negative label. Represents the Gumbel-Softmax function.

[0022] Preferably, the positive label feature extraction branch includes a first shallow feature extraction module and a deep feature extraction module, and the negative label feature extraction branch includes a second shallow feature extraction module; the first shallow feature extraction module and the second shallow feature extraction module both include a convolution layer with a convolution kernel size of 3×3, and other image blocks corresponding to the negative label are passed through the second shallow feature extraction module to obtain negative label features; the deep feature extraction module includes K residual transformer modules connected in sequence and a convolution layer with a convolution kernel size of 3×3, and the input features of the deep feature extraction module are added to the output features of the convolution layer with a convolution kernel size of 3×3. The output features of the deep feature extraction module are obtained; the residual transformer module includes a number of transformer layers and a convolution layer with a convolution kernel size of 1×1 connected in sequence, and the input features of the residual transformer module are added to the output features of the convolution layer with a convolution kernel size of 1×1 to obtain the output features of the residual transformer module; the blurred image block corresponding to the positive label passes through the first shallow feature extraction module to extract the shallow features, and the shallow features are input into the deep feature extraction module to extract the deep features, and the shallow features are added to the deep features to obtain the positive label features.

[0023] Preferably, the image reconstruction module comprises a convolution layer and a sub-pixel convolution layer with a convolution kernel size of 3×3 connected in sequence.

[0024] Preferably, the total loss function L used in the training process of the image super-resolution model is expressed as:

[0025] ;

[0026] in, represents the pruning loss, represents reconstruction losses;

[0027] The pruning loss is expressed as:

[0028] ;

[0029] in, represents the weight coefficient, Cross-Entropy represents the cross entropy loss function, Indicates the fuzzy confidence corresponding to the first coding feature map, the second coding feature map or the third coding feature map, i=1,2,3, Indicates that Down-Sample is used to perform fuzzy masking. A value obtained by downsampling that matches the resolution of the fuzzy confidence corresponding to the first coding feature map, the second coding feature map, or the third coding feature map; a fuzzy mask Pixel 1 is used to mark the blurred area in the low-resolution image, and pixel 0 is used to mark other areas in the low-resolution image;

[0030] The reconstruction loss is expressed as:

[0031] ;

[0032] ;

[0033] ;

[0034] in, and Represent the sharpened real high-resolution image and the reconstructed high-resolution image respectively. , SSIM and FFT represent loss, structural similarity loss and Fourier transform loss, represents the base reconstruction loss, represents the weight of the reconstruction loss, , and Represents the weight coefficient of the basic reconstruction loss.

[0035] In a second aspect, the present invention provides an image super-resolution device for local motion blur, comprising:

[0036] A model building module is configured to build and train an image super-resolution model for local motion blur to obtain a trained image super-resolution model, wherein the image super-resolution model includes an image selection module, a feature extraction module, and an image reconstruction module, wherein the image selection module includes an image encoder, three blur confidence predictors, and a Gumbel-Softmax layer, and the feature extraction module includes a positive label feature extraction branch, a negative label feature extraction branch, and a splicing layer;

[0037] The reconstruction module is configured to obtain a low-resolution image to be reconstructed and input it into a trained image super-resolution model. The low-resolution image is input into an image selection module. The first encoding feature map, the second encoding feature map and the third encoding feature map of different scales are obtained through an image encoder. The first encoding feature map, the second encoding feature map and the third encoding feature map are respectively passed through corresponding fuzzy confidence predictors and Gumbel-Softmax layers to obtain image grouping labels. The image grouping labels include positive labels and negative labels. The positive labels correspond to fuzzy image blocks, and the negative labels correspond to other image blocks. According to the image grouping labels, the fuzzy image blocks corresponding to the positive labels and the other image blocks corresponding to the negative labels are respectively input into the positive label feature extraction branch and the negative label feature extraction branch to obtain positive label features and negative label features. The positive label features and the negative label features are spliced ​​through a splicing layer to obtain image features. The image features are reconstructed through an image reconstruction module to obtain corresponding high-resolution images.

[0038] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0040] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] (1) The image super-resolution method for local motion blur proposed in the present invention accurately classifies the local motion blur area and other areas through an image selection module, and only performs deep feature extraction on the blurred area, avoiding redundant processing of the clear area, efficiently utilizing computing resources, and more effectively improving the quality of the reconstructed image.

[0043] (2) The image super-resolution method for local motion blur proposed in this paper extracts shallow features and deep features at multiple scales, combines local details with global dependencies, and significantly enhances the deblurring and super-resolution reconstruction capabilities.

[0044] (3) The image super-resolution method for local motion blur proposed in this invention introduces a blur confidence predictor as well as pruning loss and reconstruction loss. Through end-to-end training, the blurry area is accurately located, the misprocessing of the non-blurred area is avoided, and efficient high-resolution image restoration is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A schematic diagram of a flow chart of an image super-resolution method for local motion blur according to an embodiment of the present application;

[0047] Figure 2 A schematic diagram of the structure of an image super-resolution model of an image super-resolution method for local motion blur according to an embodiment of the present application;

[0048] Figure 3 A schematic diagram of an image encoder of an image super-resolution method for local motion blur according to an embodiment of the present application;

[0049] Figure 4 A schematic diagram of a deep feature extraction module of an image super-resolution method for local motion blur according to an embodiment of the present application;

[0050] Figure 5 A schematic diagram of a residual transformer module of an image super-resolution method for local motion blur according to an embodiment of the present application;

[0051] Figure 6 A schematic diagram of an image super-resolution device for local motion blur according to an embodiment of the present application;

[0052] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Figure 1An image super-resolution method for local motion blur provided by an embodiment of the present application is shown, comprising the following steps:

[0055] S1, construct and train an image super-resolution model for local motion blur to obtain a trained image super-resolution model, the image super-resolution model includes an image selection module, a feature extraction module and an image reconstruction module, the image selection module includes an image encoder, three blur confidence predictors and a Gumbel-Softmax layer, the feature extraction module includes a positive label feature extraction branch, a negative label feature extraction branch and a splicing layer.

[0056] Specifically, refer to Figure 2 The image super-resolution model for local motion blur constructed in the embodiment of the present application is composed of an image selection module, a feature extraction module and an image reconstruction module. The details of each module will be described one by one later.

[0057] In a specific embodiment, the total loss function L used in the training process of the image super-resolution model is expressed as:

[0058] ;

[0059] in, represents the pruning loss, represents reconstruction losses;

[0060] The pruning loss is expressed as:

[0061] ;

[0062] in, represents the weight coefficient, Cross-Entropy represents the cross entropy loss function, Indicates the fuzzy confidence corresponding to the first coding feature map, the second coding feature map or the third coding feature map, i=1,2,3, Indicates that Down-Sample is used to perform fuzzy masking. A value obtained by downsampling that matches the resolution of the fuzzy confidence corresponding to the first coding feature map, the second coding feature map, or the third coding feature map; a fuzzy mask Pixel 1 is used to mark the blurred area in the low-resolution image, and pixel 0 is used to mark other areas in the low-resolution image;

[0063] The reconstruction loss is expressed as:

[0064] ;

[0065] ;

[0066] ;

[0067] in, and Represent the sharpened real high-resolution image and the reconstructed high-resolution image respectively. , SSIM and FFT represent loss, structural similarity loss and Fourier transform loss, represents the base reconstruction loss, represents the weight of the reconstruction loss, , and Represents the weight coefficient of the basic reconstruction loss.

[0068] Specifically, during the training process of the image super-resolution model, a local motion blur dataset is obtained, a blur mask of the local motion blur dataset is annotated, the blurred area is marked with pixel 1, and other areas are marked with pixel 0.

[0069] Training parameters are set, and based on the training parameters, the image super-resolution model is trained end-to-end by reconstruction loss and pruning loss guided by the pre-annotated blur mask. In the pruning loss, the blur confidence generated by the blur confidence predictor is used, and the blur mask M is down-sampled to match the resolution of the blur confidence through down-sampling, so as to construct a cross entropy loss function. In one embodiment, the pruning loss , the reconstruction loss , , .

[0070] S2, obtain the low-resolution image to be reconstructed and input it into the trained image super-resolution model, the low-resolution image is input into the image selection module, and the first encoding feature map, the second encoding feature map and the third encoding feature map of different scales are obtained through the image encoder, the first encoding feature map, the second encoding feature map and the third encoding feature map are respectively passed through the corresponding fuzzy confidence predictor and the Gumbel-Softmax layer to obtain the image grouping label, the image grouping label includes a positive label and a negative label, the positive label corresponds to the fuzzy image block, and the negative label corresponds to other image blocks; according to the image grouping label, the fuzzy image block corresponding to the positive label and the other image blocks corresponding to the negative label are respectively input into the positive label feature extraction branch and the negative label feature extraction branch to obtain the positive label feature and the negative label feature, the positive label feature and the negative label feature are spliced ​​through the splicing layer to obtain the image feature; the image feature passes through the image reconstruction module to reconstruct the corresponding high-resolution image.

[0071] In a specific embodiment, the image encoder includes an embedding layer, a first transformer layer, a second transformer layer, a third transformer layer and a fourth transformer layer connected in sequence; the low-resolution image is divided into a plurality of image blocks through the embedding layer and the initial features are generated through the first transformer layer, and the initial features are expressed as ,in , and C represent the height, width and number of channels of the low-resolution image, respectively, and p represents the size of the image block. Represents a set of real numbers; the initial features corresponding to each image block pass through the second transformer layer, the third transformer layer, and the fourth transformer layer in turn, and output the first encoding feature map, the second encoding feature map, and the third encoding feature map of different scales respectively. The first encoding feature map, the second encoding feature map, and the third encoding feature map are respectively expressed as:

[0072] ;

[0073] ;

[0074] ;

[0075] in, represents the first encoding feature map, represents the second encoding feature map, represents the third encoding feature map, represents the second transformer layer, represents the third transformer layer, Represents the fourth transformer layer.

[0076] Specifically, refer to Figure 3 , the low-resolution image The input is sent to the image encoder, and the initial features corresponding to each image block are generated by the embedding layer and the first transformer layer; the three transformer layers , , The swin transformer structure is used to generate encoding feature maps at three different scales. The three different scales are 2p×2p, 4p×4p, and 8p×8p, that is, the output shape is H LR / 2p×W LR / 2p×2C、H LR / 4p×W LR / 4p×4C、H LR / 8p×WLR / 8p×8C encoding feature map.

[0077] In a specific embodiment, the fuzzy confidence predictor includes a multi-layer perceptron and a Softmax layer, and the first encoding feature map, the second encoding feature map, and the third encoding feature map are respectively input into the corresponding fuzzy confidence predictor to obtain corresponding feature embedding, as shown in the following formula:

[0078] ;

[0079] in, represents the feature embedding corresponding to the first encoding feature map, the second encoding feature map, or the third encoding feature map, represents a multi-layer perceptron, represents the first coding feature map, the second coding feature map or the third coding feature map, i=1,2,3;

[0080] The feature embedding is input into the Softmax layer and the fuzzy confidence is predicted as shown in the following formula:

[0081] ;

[0082] in, represents the fuzzy confidence corresponding to the first coding feature map, the second coding feature map or the third coding feature map, represents the Softmax function, so we get the probability distribution ;

[0083] The probability distribution Input to the Gumbel-Softmax layer for hard decision making to obtain the image grouping label, as shown in the following formula:

[0084] ;

[0085] Where s represents the image grouping probability. If s is greater than or equal to the threshold, the image grouping label is a positive label. If s is less than the threshold, the image grouping label is a negative label. Represents the Gumbel-Softmax function.

[0086] Specifically, the coded feature maps of different scales are input into the corresponding fuzzy confidence predictor to generate the corresponding fuzzy confidence. The fuzzy confidence corresponding to the coded feature maps of different scales constitutes a probability distribution. Gumbel-Softmax is used to make hard decisions, and the image blocks are divided into two groups. Positive labels are assigned to fuzzy image blocks, and negative labels are assigned to other image blocks. The specific process is as follows: the probability distribution is input into the Gumbel-Softmax layer, the image grouping probability is calculated by Gumbel noise and temperature coefficient, and the image grouping result is determined according to the image grouping probability.

[0087] In a specific embodiment, the positive label feature extraction branch includes a first shallow feature extraction module and a deep feature extraction module, and the negative label feature extraction branch includes a second shallow feature extraction module; the first shallow feature extraction module and the second shallow feature extraction module both include a convolution layer with a convolution kernel size of 3×3, and other image blocks corresponding to the negative label pass through the second shallow feature extraction module to obtain negative label features; the deep feature extraction module includes K residual transformer modules connected in sequence and a convolution layer with a convolution kernel size of 3×3, and the input features of the deep feature extraction module are consistent with the output features of the convolution layer with a convolution kernel size of 3×3. The output features of the deep feature extraction module are obtained by adding the input features of the residual transformer module and the output features of the convolution layer with a convolution kernel size of 1×1 to obtain the output features of the residual transformer module; the blurred image block corresponding to the positive label passes through the first shallow feature extraction module to extract the shallow features, and the shallow features are input to the deep feature extraction module to extract the deep features, and the shallow features are added to the deep features to obtain the positive label features.

[0088] Specifically, refer to Figure 4 , the first shallow feature extraction module is used to extract the shallow features of the blurred image block Then, a series of K residual transformer blocks (RTB) based on swin transformer and a convolution layer with a kernel size of 3×3 are used to extract the deep features of the blurred image blocks. , where a residual connection is formed between the input features of the deep feature extraction module and the output features of the convolution layer with a convolution kernel size of 3×3. The shallow features are added to the deep features to obtain the positive label features. The second shallow feature extraction module is used to extract shallow features of other image blocks. , the negative label feature can be obtained. The positive label feature and the negative label feature are concatenated through the concatenation layer to obtain the concatenated feature. In one embodiment, K=6. Figure 5 , the residual transformer module consists of multiple transformer layers and a convolution layer with a convolution kernel size of 1×1. A residual connection is formed between the input features of the residual transformer module and the output features of the convolution layer with a convolution kernel size of 1×1.

[0089] In a specific embodiment, the image reconstruction module includes a convolution layer with a convolution kernel size of 3×3 and a sub-pixel convolution layer connected in sequence.

[0090] Specifically, the splicing feature is input into the image reconstruction module, and passes through the convolution layer with a convolution kernel size of 3×3 and the sub-pixel convolution layer in sequence to reconstruct a high-resolution image.

[0091] Further references Figure 6 As an implementation of the methods shown in the above figures, the present application provides an embodiment of an image super-resolution device for local motion blur, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0092] The present application provides an image super-resolution device for local motion blur, including:

[0093] A model building module 1 is configured to build and train an image super-resolution model for local motion blur to obtain a trained image super-resolution model, wherein the image super-resolution model includes an image selection module, a feature extraction module, and an image reconstruction module, wherein the image selection module includes an image encoder, three blur confidence predictors, and a Gumbel-Softmax layer, and the feature extraction module includes a positive label feature extraction branch, a negative label feature extraction branch, and a splicing layer;

[0094] The reconstruction module 2 is configured to obtain a low-resolution image to be reconstructed and input it into a trained image super-resolution model. The low-resolution image is input into an image selection module, and a first encoding feature map, a second encoding feature map and a third encoding feature map of different scales are obtained through an image encoder. The first encoding feature map, the second encoding feature map and the third encoding feature map are respectively passed through corresponding fuzzy confidence predictors and Gumbel-Softmax layers to obtain image grouping labels. The image grouping labels include positive labels and negative labels. The positive labels correspond to fuzzy image blocks, and the negative labels correspond to other image blocks. According to the image grouping labels, the fuzzy image blocks corresponding to the positive labels and the other image blocks corresponding to the negative labels are respectively input into the positive label feature extraction branch and the negative label feature extraction branch to obtain positive label features and negative label features. The positive label features and the negative label features are spliced ​​through a splicing layer to obtain image features. The image features are reconstructed through an image reconstruction module to obtain corresponding high-resolution images.

[0095] Figure 7 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 7As shown, the electronic device of this embodiment includes: a processor 701 and a memory 702; wherein the memory 702 is used to store computer-executable instructions; the processor 701 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0096] Optionally, the memory 702 may be independent or integrated with the processor 701 .

[0097] When the memory 702 is independently provided, the electronic device further includes a bus 703 for connecting the memory 702 and the processor 701 .

[0098] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 701 executes the computer execution instructions, the above method is implemented.

[0099] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 701, the above method is implemented.

[0100] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0101] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0102] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0103] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 701 to perform some steps of the methods of various embodiments of the present application.

[0104] It should be understood that the processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 701 may be any conventional processor 701, etc. The steps of the method disclosed in the invention may be directly embodied in the execution of the hardware processor 701, or may be executed by a combination of hardware and software modules in the processor 701.

[0105] The memory 702 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0106] The bus 703 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 703 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 703 in the drawings of the present application is not limited to only one bus 703 or one type of bus 703.

[0107] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0108] An exemplary storage medium is coupled to the processor 701, so that the processor 701 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 701. The processor 701 and the storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as ASIC). Of course, the processor 701 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0109] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image super-resolution method for local motion blur, characterized in that: The following steps are involved: An image super-resolution model for local motion blur is constructed and trained to obtain a trained image super-resolution model, wherein the image super-resolution model includes an image selection module, a feature extraction module and an image reconstruction module, wherein the image selection module includes an image encoder, three blur confidence predictors and a Gumbel-Softmax layer, and the feature extraction module includes a positive label feature extraction branch, a negative label feature extraction branch and a splicing layer; A low-resolution image to be reconstructed is obtained and input into the trained image super-resolution model, the low-resolution image is input into the image selection module, and a first encoding feature map, a second encoding feature map and a third encoding feature map of different scales are obtained through an image encoder, the first encoding feature map, the second encoding feature map and the third encoding feature map are respectively passed through a corresponding fuzzy confidence predictor and a Gumbel-Softmax layer to obtain an image grouping label, the image grouping label includes a positive label and a negative label, the positive label corresponds to the fuzzy image block, and the negative label corresponds to other image blocks, the fuzzy confidence predictor includes a multi-layer perceptron and a Softmax layer, the first encoding feature map, the second encoding feature map and the third encoding feature map are respectively input into a corresponding fuzzy confidence predictor to obtain a corresponding feature embedding, as shown in the following formula: e i =MLP(r i ); Among them, e i represents the feature embedding corresponding to the first encoding feature map, the second encoding feature map, or the third encoding feature map, MLP represents a multi-layer perceptron, r i represents the first coding feature map, the second coding feature map or the third coding feature map, i=1,2,3; The feature embedding is input into the Softmax layer, and the fuzzy confidence is predicted as shown in the following formula: C i =Softmax(e i ); Among them, C i represents the fuzzy confidence corresponding to the first coding feature map, the second coding feature map or the third coding feature map, Softmax represents the Softmax function, so the probability distribution C = [C1, C2, C3] is obtained; The probability distribution C is input into the Gumbel-Softmax layer for hard decision making to obtain the image grouping label, as shown in the following formula: s = GumbelSoftmax(C); Wherein, s represents the image grouping probability, if s is greater than or equal to the threshold, the image grouping label is a positive label, if s is less than the threshold, the image grouping label is a negative label, and GumbelSoftmax represents the Gumbel-Softmax function; according to the image grouping label, the fuzzy image block corresponding to the positive label and the other image blocks corresponding to the negative label are respectively input into the positive label feature extraction branch and the negative label feature extraction branch to obtain positive label features and negative label features, and the positive label features and the negative label features are spliced ​​through the splicing layer to obtain image features; the image features are reconstructed through the image reconstruction module to obtain the corresponding high-resolution image; The total loss function L used in the training process of the image super-resolution model is expressed as: L=L p +L R ; Among them, L p represents the pruning loss, L R represents reconstruction losses; The pruning loss is expressed as: Among them, λ0 represents the weight coefficient, Cross-Entropy represents the cross entropy loss function, and C i Indicates the fuzzy confidence corresponding to the first coding feature map, the second coding feature map or the third coding feature map, i = 1, 2, 3, Down-Sample i (M) represents a value obtained by down-sampling the blur mask M through Down-Sample, which matches the resolution of the blur confidence corresponding to the first encoding feature map, the second encoding feature map, or the third encoding feature map; in the blur mask M, pixel 1 is used to mark the blur area in the low-resolution image, and pixel 0 is used to mark other areas in the low-resolution image; The reconstruction loss is expressed as: L R =wL' R (M·S',M·S)+(1-w)L' R ((1-M)·S',(1-M)·S); Among them, S and S' represent the sharpened real high-resolution image and the reconstructed high-resolution image, respectively. L1, SSIM and FFT represent L1 loss, structural similarity loss and Fourier transform loss, respectively. L' R represents the basic reconstruction loss, w represents the weight of the reconstruction loss, and λ1, λ2, and λ3 represent the weight coefficients of the basic reconstruction loss.

2. The image super-resolution method for local motion blur according to claim 1, characterized in that: The image encoder includes an embedding layer, a first transformer layer, a second transformer layer, a third transformer layer and a fourth transformer layer connected in sequence; the low-resolution image is divided into a plurality of image blocks through the embedding layer and an initial feature is generated through the first transformer layer, and the initial feature is represented as Among them, H LR , W LR and C represent the height, width and number of channels of the low-resolution image, respectively, and p represents the size of the image block. represents a set of real numbers; the initial features corresponding to each image block pass through the second transformer layer, the third transformer layer and the fourth transformer layer in sequence, and output the first encoding feature map, the second encoding feature map and the third encoding feature map of different scales respectively, and the first encoding feature map, the second encoding feature map and the third encoding feature map are respectively expressed as: r1=TL1(r0); r2=TL2(r1); r3=TL3(r2); in, represents the first encoding feature map, represents the second encoding feature map, represents the third encoded feature map, TL1 represents the second transformer layer, TL2 represents the third transformer layer, and TL3 represents the fourth transformer layer.

3. The image super-resolution method for local motion blur according to claim 1, characterized in that: The positive label feature extraction branch includes a first shallow feature extraction module and a deep feature extraction module, and the negative label feature extraction branch includes a second shallow feature extraction module; the first shallow feature extraction module and the second shallow feature extraction module both include a convolution layer with a convolution kernel size of 3×3, and other image blocks corresponding to the negative label are passed through the second shallow feature extraction module to obtain negative label features; the deep feature extraction module includes K residual transformer modules connected in sequence and a convolution layer with a convolution kernel size of 3×3, and the input features of the deep feature extraction module are added to the output features of the convolution layer with a convolution kernel size of 3×3 to obtain the deep feature. The output features of the feature extraction module; the residual transformer module includes a plurality of transformer layers and a convolution layer with a convolution kernel size of 1×1 connected in sequence, and the input features of the residual transformer module are added to the output features of the convolution layer with a convolution kernel size of 1×1 to obtain the output features of the residual transformer module; the blurred image block corresponding to the positive label passes through the first shallow feature extraction module to extract shallow features, and the shallow features are input to the deep feature extraction module to extract deep features, and the shallow features are added to the deep features to obtain the positive label features.

4. The image super-resolution method for local motion blur according to claim 1, characterized in that: The image reconstruction module includes a convolution layer with a convolution kernel size of 3×3 and a sub-pixel convolution layer connected in sequence.

5. An image super-resolution device for local motion blur, characterized in that: include: A model building module is configured to build and train an image super-resolution model for local motion blur to obtain a trained image super-resolution model, wherein the image super-resolution model includes an image selection module, a feature extraction module and an image reconstruction module, wherein the image selection module includes an image encoder, three blur confidence predictors and a Gumbel-Softmax layer, and the feature extraction module includes a positive label feature extraction branch, a negative label feature extraction branch and a splicing layer; The reconstruction module is configured to obtain a low-resolution image to be reconstructed and input it into the trained image super-resolution model, the low-resolution image is input into the image selection module, and the first encoding feature map, the second encoding feature map and the third encoding feature map of different scales are obtained through the image encoder, the first encoding feature map, the second encoding feature map and the third encoding feature map are respectively passed through the corresponding fuzzy confidence predictor and the Gumbel-Softmax layer to obtain the image grouping label, the image grouping label includes a positive label and a negative label, the positive label corresponds to the fuzzy image block, and the negative label corresponds to other image blocks, the fuzzy confidence predictor includes a multi-layer perceptron and a Softmax layer, the first encoding feature map, the second encoding feature map and the third encoding feature map are respectively input into the corresponding fuzzy confidence predictor to obtain the corresponding feature embedding, as shown in the following formula: e i =MLP(r i ); Among them, e i represents the feature embedding corresponding to the first encoding feature map, the second encoding feature map, or the third encoding feature map, MLP represents a multi-layer perceptron, r i represents the first coding feature map, the second coding feature map or the third coding feature map, i=1,2,3; The feature embedding is input into the Softmax layer, and the fuzzy confidence is predicted as shown in the following formula: C i =Softmax(e i ); Wherein, C0 represents the fuzzy confidence corresponding to the first coding feature map, the second coding feature map or the third coding feature map, and Softmax represents the Softmax function, so the probability distribution C = [C1, C2, C3] is obtained; The probability distribution C is input into the Gumbel-Softmax layer for hard decision making to obtain the image grouping label, as shown in the following formula: s = GumbelSoftmax(C); Wherein, s represents the image grouping probability, if s is greater than or equal to the threshold, the image grouping label is a positive label, if s is less than the threshold, the image grouping label is a negative label, and GumbelSoftmax represents the Gumbel-Softmax function; according to the image grouping label, the fuzzy image block corresponding to the positive label and the other image blocks corresponding to the negative label are respectively input into the positive label feature extraction branch and the negative label feature extraction branch to obtain positive label features and negative label features, and the positive label features and the negative label features are spliced ​​through the splicing layer to obtain image features; the image features are reconstructed through the image reconstruction module to obtain the corresponding high-resolution image; The total loss function L used in the training process of the image super-resolution model is expressed as: L=L p +L R ; Among them, L p represents the pruning loss, L R represents reconstruction losses; The pruning loss is expressed as: Among them, λ0 represents the weight coefficient, Cross-Entropy represents the cross entropy loss function, and c i Indicates the fuzzy confidence corresponding to the first coding feature map, the second coding feature map or the third coding feature map, i = 1, 2, 3, Down-Sample i (M) represents a value obtained by down-sampling the blur mask M through Down-Sample, which matches the resolution of the blur confidence corresponding to the first encoding feature map, the second encoding feature map, or the third encoding feature map; in the blur mask M, pixel 1 is used to mark the blur area in the low-resolution image, and pixel 0 is used to mark other areas in the low-resolution image; The reconstruction loss is expressed as: Among them, S and S' represent the sharpened real high-resolution image and the reconstructed high-resolution image, respectively. L1, SSIM and FFT represent L1 loss, structural similarity loss and Fourier transform loss, respectively. L' R represents the basic reconstruction loss, w represents the weight of the reconstruction loss, and λ1, λ2, and λ3 represent the weight coefficients of the basic reconstruction loss.

6. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images