A method, system and device for dynamic scene image deblurring

Abstract

This invention provides a method, system, and device for deblurring dynamic scene images. The specific steps are as follows: Constructing a dynamic scene image deblurring network, including a multi-scale dense feature extraction module connected between convolutional layers of a U-Net network, a ConvLSTM bidirectional connected structure inserted between adjacent skip connections in the U-Net network, and a U-Net network structure optimization strategy inserted into the U-Net network encoder and decoder; constructing a total loss function and training the dynamic scene image deblurring network to obtain a dynamic scene image deblurring network model; inputting the image to be processed into the dynamic scene image deblurring network model to obtain a deblurred image. This invention effectively reduces the loss of texture details during image restoration, suppresses noise to a certain extent, and prevents ringing artifacts, facilitating subsequent tasks and work.

Description

Technical Field

[0001] This invention belongs to the field of dynamic scene deblurring technology, specifically to a method, system and device for deblurring dynamic scene images. Background Technology

[0002] Images are a primary medium for people to acquire information and an important symbol for information exchange and cultural transmission across various industries in today's society. With the improvement of social life quality and the practical needs of daily work, image-based technologies are developing rapidly, such as object detection, image segmentation, and object tracking. However, during image acquisition, images often suffer from motion blur due to multiple interferences from the external environment. Image motion blur not only affects human perception but also severely reduces the processing efficiency of subsequent object detection and image recognition technologies. Therefore, how to remove image motion blur and restore clear images has gradually become an important research direction in the field of computer vision.

[0003] Dynamic scene deblurring methods typically employ constrained least squares filtering, regularization-based methods, and deep learning-based methods to accurately describe the original feature information of an image during the deblurring process, thereby improving the processing efficiency of subsequent target detection and image recognition techniques. In recent years, deep learning, with its powerful feature representation capabilities, has achieved a balance between time and quality in describing image features compared to traditional manual feature extraction methods. However, existing motion blur removal networks are prone to problems such as loss of texture details, inability to suppress noise, and the generation of ringing artifacts in some image restoration processes, posing significant challenges to subsequent tasks and research.

[0004] To address the problems of blurred object edges and poor visual perception caused by motion blur, many researchers have leveraged the U-Net structure's advantages of fusing high- and low-level semantic information and its fast training speed to develop a series of motion blur removal applications. While methods based on the U-Net encoder-decoder structure can achieve good image deblurring results, they can only capture a small portion of feature information during the feature extraction stage, failing to effectively obtain global contextual semantic information. Furthermore, the extracted features generally suffer from poor utilization efficiency and significant loss, resulting in image quality issues such as missing details and textures. Some scholars have used dense multi-receptor domain feature information extraction and aggregation methods to obtain image information at different scales, thereby increasing feature reuse efficiency and verifying the effectiveness of this application in improving the detail representation of blurred images. Although deep feature extraction and global feature aggregation based on dense connections effectively restore image sharpness, significantly improving image edges and details, the feature information extracted by the network during the encoding stage cannot be efficiently transmitted to the decoding stage, easily leading to the presence of a small amount of ringing artifacts in the image, making it difficult to distinguish text, slogans, and other information. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a method, system, and device for dynamic scene image deblurring, which can effectively reduce the loss of texture details during image restoration, suppress noise to a certain extent, and prevent ringing artifacts, thus facilitating subsequent tasks and work.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for deblurring dynamic scene images, the specific steps of which are as follows:

[0007] S1 constructs a dynamic scene image deblurring network, including a multi-scale dense feature extraction module connected between the convolutional layers of the U-Net network, a ConvLSTM bidirectional connected structure inserted between adjacent skip connections in the U-Net network, and a U-Net network structure optimization strategy inserted in the encoder and decoder of the U-Net network.

[0008] S2 constructs the total loss function and trains the dynamic scene image deblurring network to obtain the dynamic scene image deblurring network model;

[0009] S3 inputs the image to be processed into the dynamic scene image deblurring network model to obtain the deblurred image.

[0010] Furthermore, in S1, the U-Net network structure optimization strategy includes dilated convolution and subpixel convolution. Dilated convolution is used for downsampling in the encoder, and subpixel convolution is used for upsampling in the decoder.

[0011] Furthermore, in S1, the multi-scale dense feature extraction module includes a dense feature extraction network and a multi-level pyramid pooling branch (SPP). The dense feature extraction network is used to obtain deep features of the image, and the multi-level pyramid pooling branch (SPP) is used to obtain multi-scale local detail features. The deep features and multi-scale local detail features are fused to obtain global detail features used to optimize the sharpness of blurred images.

[0012] Furthermore, the dense feature extraction network includes convolutional layers in the original U-Net structure interconnected in a feedforward manner, and the convolutional layers closer to the input and output layers employ dense short connections; the dense feature extraction network in the... l t The input of each convolutional layer is t Given the number of subsequent layers in the densely connected structure, the output is:

[0013] (1)

[0014] in It consists of three consecutive operations: batch normalization, linear unit, and a 3×3 convolution.

[0015] Furthermore, the multi-level pyramid pooling branch SPP includes three parallel 3×3 dilated convolutional capture blocks with dilation rates of 2, 4, and 8, respectively.

[0016] Furthermore, the ConvLSTM bidirectional connectivity structure includes forward and reverse paths distributed between adjacent skip connections in the U-Net network. The forward path runs from bottom to top, and the reverse path runs from top to bottom. A convolutional long short-term memory (ConvLSTM) model is inserted in both the forward and reverse paths to extract multidimensional feature information from the context of adjacent features and fuse it with the current features in the decoding part.

[0017] Furthermore, the forward and reverse paths between adjacent hop connections in the U-Net network exist in pairs, with the path closer to the U-Net network encoder being the forward path and the path closer to the U-Net network decoder being the reverse path.

[0018] Furthermore, in S2, the total loss function include L 2. Loss function and L p Perceived loss, the formula is as follows:

[0019] (2)

[0020] Where λ represents the weight, which is usually taken as λ=0.01;

[0021] Assume there is a training set ( B i , I i ),Include For blurry / sharp images, L 2. The formula for the loss function is as follows:

[0022] (3)

[0023] in, For parameters, F Network for deblurring dynamic scene images;

[0024] L p The formula for perceived loss is as follows:

[0025] (4)

[0026] in: The image after deblurring; JThe corresponding clear original image; and They represent and J The corresponding feature map, C j , H j and W j They represent and The number of channels, height, and width of the feature map.

[0027] This invention provides a system for deblurring dynamic scene images, comprising:

[0028] The network building module is used to build a dynamic scene image deblurring network, including a multi-scale dense feature extraction module connected between the convolutional layers of the U-Net network, a ConvLSTM bidirectional connected structure inserted between adjacent skip connections in the U-Net network, and a U-Net network structure optimization strategy inserted in the U-Net network encoder and decoder.

[0029] The network training module is used to construct the total loss function and train the dynamic scene image deblurring network to obtain the dynamic scene image deblurring network model.

[0030] The image processing module is used to input the image to be processed into the dynamic scene image deblurring network model to obtain the deblurred image.

[0031] The present invention provides a terminal device, which includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method, or when the processor executes the computer program, it implements the functions of each module in the system.

[0032] Compared with the prior art, the present invention has at least the following beneficial effects:

[0033] This invention proposes a method for deblurring dynamic scene images. It constructs a multi-scale densely connected dynamic scene deblurring network improved with U-Net. The dilated convolution downsampling in the U-Net network effectively expands the receptive field, avoiding irreversible image damage without increasing the number of parameters. Sub-pixel convolution is used during upsampling to obtain clear image details with small kernels, reducing computational complexity. Secondly, a multi-scale dense feature extraction module is designed, strengthening deep feature extraction and reuse through densely connected convolutional layers. Spatial pyramid pooling (SPP) branches guide the transfer and fusion of multi-scale features, promoting effective preservation of image details and textures. Finally, a ConvLSTM bidirectional connected structure is used to non-linearly compensate for lost contextual features from simple cascading in the encoding path, promoting cross-stage interaction of deep features and weakening edge artifacts and noise interference. This method can improve the processing quality and efficiency of subsequent object detection and image recognition in various real-world scenarios.

[0034] Furthermore, in each branch of the Multi-Scale Dense Feature Extraction (MDFE) module, this invention employs short connections between convolutional layers that are close to the input and output layers. Convolutional networks can increase information exchange and feature description within the network through dense short connections, thereby effectively alleviating the gradient vanishing problem during network training while reducing feature loss and time consumption during feature transmission, and further enhancing the correlation between layers.

[0035] Furthermore, this invention uses 3×3 dilated convolutions with dilation rates of 2, 4, and 8 to capture multi-scale features of the block input and combines the output features with other features from the dense network. Meanwhile, in order to achieve local residual connections of feature information, 1×1 convolutions are used to compress the number of cascaded feature channels to avoid increasing parameter complexity.

[0036] Furthermore, the features processed by the bidirectional connected structure with ConvLSTM as the unit in this invention are passed to the next unit after being deleted, updated and saved. This can avoid the loss of information of the elements before the sequence and promote the fusion of feature information between adjacent skip branches of long sequence data, so as to make up for the deficiencies in the network structure and greatly improve the effect of image deblurring. Attached Figure Description

[0037] Figure 1 This is a diagram of the overall network structure of the present invention;

[0038] Figure 2 This is a network diagram of the Multi-Scale Dense Feature Extraction (MDFE) module of the present invention.

[0039] Figure 3 This is a diagram of the ConvLSTM bidirectional connected structure network of the present invention.

[0040] Figure 4 This is a map showing the receptive field range corresponding to dilated convolution in the U-Net network structure optimization strategy of this invention.

[0041] Figure 5 This diagram illustrates the subpixel convolution upsampling process in the U-Net network structure optimization strategy of this invention.

[0042] Figure 6 This is a graph showing the execution results of the present invention on the GOPRO dataset.

[0043] Figure 7 This is a comparison of the ablation experiment results of this invention on the GOPRO dataset. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0045] See Figure 1 This invention discloses a dynamic scene image deblurring method improved by dense connections and U-Net, comprising the following steps:

[0046] S1 constructs a multi-scale densely connected dynamic scene image deblurring network improved by U-Net. This includes a multi-scale dense feature extraction module connected between convolutional layers of the U-Net network, a ConvLSTM bidirectional connected structure inserted between adjacent skip connections in the U-Net network, and a U-Net network structure optimization strategy inserted into the U-Net encoder and decoder. The dilated convolution downsampling in the U-Net network structure optimization strategy effectively expands the receptive field, avoiding irreversible image damage without increasing the number of parameters. Sub-pixel convolutions are used to obtain clear image details with small kernels during upsampling, reducing computational complexity. In the U-Net network, deep feature extraction and reuse are enhanced by replacing the original U-Net structure with densely connected convolutional layers. An added multi-level pyramid pooling branch (SPP) guides the transfer and fusion of multi-scale features, promoting effective preservation of image detail and texture. Finally, a ConvLSTM bidirectional connected structure is inserted into the U-Net network to non-linearly compensate for lost contextual features from simple cascading in the encoding path, promoting cross-stage interaction of deep features and weakening edge artifacts and noise interference.

[0047] Specifically:

[0048] 1. Multi-scale Dense Feature Extraction Module (MDFE)

[0049] Many current deblurring algorithms, in pursuit of improved network performance, blindly pursue deeper and wider networks while neglecting the full utilization of features. Furthermore, as the network deepens, feature transmission gradually weakens due to changes in network structure, thus affecting the overall image quality improvement. To address these issues, this invention designs a method such as... Figure 2 The Multi-Scale Dense Feature Extraction (MDFE) module shown consists of a dense feature extraction network and a multi-level pyramid pooling branch (SPP). The dense feature extraction network is used to obtain deep features of the image, and the multi-level pyramid pooling branch (SPP) is used to obtain multi-scale local detail features. The deep features and multi-scale local detail features are fused to obtain global detail features used to optimize the sharpness of blurred images. The execution process of the MDFE module is as follows (see [link to documentation]). Figure 2 :

[0050] First, the design of the dense feature extraction network structure involves interconnecting the convolutional layers in the original U-Net structure using a feedforward approach. This adds inter-layer connections to the previously single-path feature transfer, building upon the existing connections between two convolutional layers and cross-layer connections. The outputs of all preceding layers serve as the input to this layer, and the output of this layer serves as the input to all subsequent layers. These added inter-layer connections enable the reuse of input features, ensuring maximum information flow. Each convolutional layer receives input data from all previous layers and reprocesses the input feature data, thereby enhancing feature reuse.

[0051] Secondly, the short connections between convolutional layers, close to the input and output layers, make the input and reception of feature information between adjacent convolutional layers efficient and time-saving. Thus, convolutional networks can increase information exchange and feature description within the network through dense short connections, effectively alleviating the gradient vanishing problem during network training and further enhancing the correlation between layers to obtain deeper features.

[0052] like Figure 2 As shown, the dense feature extraction network maximizes the information flow between the features extracted by each layer, directly connecting all layers together. Each layer receives input from all preceding layers and inputs its own output features into all subsequent layers. l t The input of the layer is t Given the number of subsequent layers in the densely connected structure, the output is:

[0053] (1)

[0054] in It consists of three consecutive operations: batch normalization, linear unit, and a 3×3 convolution.

[0055] Finally, to effectively optimize global detail features and ensure that the integrated feature information is effectively applied to the image sharpness reconstruction process, a multi-level pyramid pooling branch (SPP) was added to the multi-scale dense feature extraction module to learn multi-scale local detail features of the blurred image. This SPP branch contains three parallel modules: 3×3 dilated convolutional capture blocks with dilation rates of 2, 4, and 8. The scale-based local detail features output by this network structure are compressed using a 1×1 convolution and then fused with the deep features extracted from the dense feature extraction. This compensates for the loss of shallow detail information caused by the deepening of feature mining and enhances the global representation of local features in the image, resulting in a uniformly sharp reconstructed image after dynamic blur removal.

[0056] To achieve local residual connections of feature information and avoid information redundancy caused by multiple feature extraction methods and multi-level fusion, a 1×1 convolution is used to compress the number of cascaded feature channels, thus avoiding increasing parameter complexity.

[0057] 2. U-Net Network Structure Optimization Strategy

[0058] The U-Net structure mainly consists of an encoder and a decoder. In traditional deep neural networks, pooling operations of different lengths are generally used to downsample or upsample input features to obtain multi-scale contextual information. However, pooling operations reduce image resolution, and this effect is irreversible. It not only leads to the loss of spatial information in the reconstruction process of blurred images but may also prevent the restoration of blurred image edge details. Therefore, we use dilated convolution to replace pooling operations for downsampling. In dilated convolution, a k×k convolution kernel is enlarged to k+(k-1)(r-1) with a stride r, thus enhancing the effective aggregation of contextual information while maintaining the original resolution. The receptive field of the convolution operation increases with the dilated convolution rate, which not only avoids the loss of spatial information in the reconstruction process of blurred images but also effectively restores the edge details of blurred images. (See [link to dilated convolution operation]). Figure 4 .

[0059] Furthermore, this invention employs sub-pixel convolutional layers instead of traditional transposed convolutions. It achieves upsampling by rearranging feature pixels using a reverse sampling approach, restoring the global features obtained by the encoder to the original image size without increasing parameters or network complexity. (See [link to related documentation]). Figure 5 .

[0060] In the upsampling process of mapping from input (low-resolution LR) to output (high-resolution HR), sub-pixel convolution is used to enlarge the image. The interpolation function can be automatically learned in the previous convolution layer, thus learning a better and more complex mapping from LR to HR. Since the convolution operation is performed on the LR image, the pixel at the corresponding position in the low-resolution image and the corresponding upsampling filter weights jointly determine the value of the corresponding pixel in the high-resolution image. Convolution with small kernels can reduce computational complexity to a certain extent and effectively improve the efficiency of the entire process.

[0061] This invention uses skip connections between the encoding input part and the decoding output part, which can reduce the loss of features during the feature transfer process between the input feature map and the output feature map obtained by the decoder, thereby accelerating network convergence and improving image reconstruction quality.

[0062] 3. ConvLSTM bidirectional connectivity structure

[0063] See Figure 3 In the entire U-Net network architecture, the process from encoded input to decoded output is typically viewed as a simple skip connection. This connection replicates the pruning operation, extracting feature maps from the corresponding encoding path and the previously decoded convolutional layers, thus facilitating the transfer of spatial information from lower-level to higher-level layers. While this structure can transfer low-level edge details from the encoding to the decoding part through skip connections, the lack of transition information between low-level edge details and high-level semantic features leads to feature-semantic discrepancies throughout the model's learning process. Furthermore, as the network deepens, the sampling process gradually loses feature resolution, weakening the original role of skip connections in restoring image details.

[0064] Strengthening the effective utilization of features by connecting features from different layers is an important means to solve the semantic difference between features. This invention constructs a bidirectional connected structure with ConvLSTM as the unit in the skip connection of U-Net, and uses the memory characteristics of the Convolutional Long Short-Term Memory (ConvLSTM) model to extract multidimensional feature information from the contextual relationship of adjacent features, and realizes high-dimensional information input based on the data correlation in two directions.

[0065] The ConvLSTM bidirectional connectivity structure consists of four ConvLSTM convolutional long short-term memory models, which are distributed in pairs between adjacent skip connections in the U-Net network, with a bottom-up forward path and a top-down reverse path.

[0066] The execution process of the ConvLSTM bidirectional connected structure is as follows: Figure 3 As shown:

[0067] First, regarding the current features The process can be divided into two parts: one where the data flows directly to the decoding part, and the other where the data flows from the bottom up in a positive feature flow and is processed by ConvLSTM. This allows for the selection of deleted, updated, and saved features, resulting in the decoded part. The next ConvLSTM operation is completed in the reverse path.

[0068] Secondly, regarding features Then it will be in the jump connection with The extracted feature information is processed through a bottom-up positive feature information flow and ConvLSTM, which involves deletion, updating, and saving of selected features before fusion to obtain the decoded part. The next ConvLSTM operation on the feature information is completed in the neutralization and reverse path.

[0069] For features ,and The extracted features are deleted, updated, and saved through a bottom-up feature information flow and under the action of ConvLSTM, and then fused to obtain the output features of the decoding part. The feature flow in the reverse path completes the next ConvLSTM operation.

[0070] Finally, through characteristics The incoming top-down reverse feature information flow will, after undergoing the deletion, update, and saving of the extracted feature information through two ConvLSTMs respectively, continue to complete the aforementioned process. , After fusion, the output features are obtained separately. , The above processing avoids the loss of information from earlier elements in the sequence to a certain extent and improves the effective utilization rate of extracted features.

[0071] The ConvLSTM bidirectional connectivity structure of this invention can compensate for the loss of contextual information caused by the U-Net network structure. It can be applied to the feature information fusion between adjacent skip connections in long sequence U-Net network structures, and effectively expand the receptive field to avoid gradient explosion or gradient vanishing problems, thus greatly improving the effect of image deblurring.

[0072] To help the decoder accelerate network convergence and restore image edge details, this invention uses skip connections between the encoding input and decoding output sections. This directly aggregates shallow input feature information with the network's output image features, reducing transmission loss between the input feature map and the decoder's output feature map, thus effectively improving image reconstruction quality.

[0073] S2 constructs a loss function and trains a multi-scale densely connected dynamic scene deblurring network improved by U-Net to obtain the multi-scale densely connected dynamic scene deblurring network improved by U-Net.

[0074] The total loss function used in this invention during network training is derived from... L 2. Loss function and L p The components of the perceptual loss, as shown in formula (2), can be expressed as follows:

[0075] (2)

[0076] Where λ represents the weight, usually taken as λ=0.01;

[0077] Assume there is a training set ( B i , I i ),Include For blurry / sharp images, L 2. The loss function, as shown in formula (3), can be expressed as:

[0078] (3)

[0079] in, For parameters, F Network for deblurring dynamic scene images;

[0080] Perceptual loss is used to calculate the semantic difference before and after image deblurring. The specific method is shown in formula (4), which can be expressed as follows:

[0081] (4)

[0082] in: The image after deblurring; J The corresponding clear original image; and They represent and J The corresponding feature map, C j , H j and W j They represent and The number of channels, height, and width of the feature map.

[0083] The experimental portion utilizes the GoPro dataset, which contains 3214 blurred images and 3214 sharp images from various scenes. This experiment also uses this dataset, selecting 2103 pairs of sharp and blurred images as the training set, and testing on the remaining 1111 pairs.

[0084] This invention utilizes the PyTorch deep learning framework, with a GPU workstation configured as follows: Intel Core i7 CPU, 32GB RAM, and a 1080Ti graphics card with 12GB VRAM. The training phase employs 300 epochs, with a minimum batch size of 8, an initial decay rate of 0.0005, and a cosine learning rate decay, resulting in a multi-scale densely connected dynamic scene deblurring network improved by U-Net.

[0085] S3 inputs the remaining 1111 pairs of images into a multi-scale densely connected and U-Net-improved dynamic scene deblurring network model for detection, and compares the obtained experimental results on the GOPRO dataset with different state-of-the-art methods. Each row represents the experimental results obtained by different methods on various scene datasets, while different columns represent the experimental results obtained by different deblurring methods on the same scene.

[0086] Depend on Figure 6 Observations show that when images are severely blurred, the network proposed by Nah et al. results in artifacts and insufficient detail recovery in the restored image. As seen in Figure (a), the text details on the wall are severely lost. While the deblurred image by Zhang et al. has good visual effects, detail recovery is insufficient, specifically in the loss of detail in the headlights and license plate areas of the vehicle's outline. The method used by Gao et al. still suffers from ringing artifacts and texture blurring; as shown in Figure (d), the motorcycle is extremely difficult to distinguish, resulting in poor visual quality. In comparison, the method proposed in this invention provides clearer image restoration, effectively removing artifacts caused by blurring in areas such as the arms and face; and in Figure (e), the license plate number area is well restored. Furthermore, the algorithm of this invention not only outperforms other algorithms in restoring the details of the car's outline and body shape but also clearly presents even the smallest details on the vehicle body. In conclusion, the deblurring algorithm proposed in this invention has a superior performance advantage in image clarity restoration.

[0087] The method of this invention was quantitatively compared with advanced de-dynamic blurring algorithms on the GOPRO and HIDE datasets. The results are shown in Table 1.

[0088] Table 1 compares the performance of SSIM and PSNR on the GOPRO dataset.

[0089]

[0090] As can be seen from Table 1, compared with other advanced algorithms, the algorithm of this invention performs best on the GOPRO dataset and other advanced methods when quantitatively analyzed by PSNR and SSIM, and can achieve good quality improvement for blurred images in various scenes in the dataset.

[0091] Table 2. Quantitative experimental results of ablation experiments on the GOPRO dataset.

[0092]

[0093] To verify the effects of the proposed multi-scale dense feature extraction module MDFE, ConvLSTM bidirectional connected structure, and U-Net network structure optimization strategy on overall image quality improvement, four sets of ablation experiments were conducted on the GOPRO dataset. The network without the proposed modules was used as the baseline, and the proposed modules were sequentially added to the baseline to verify their performance improvement. Figure 7 As shown in Table 2, the objective results of the ablation experiments demonstrate that the network employing the Multi-Scale Dense Feature Extraction (MDFE) module improved PSNR and SSIM by 8.27 dB and 0.07 respectively on the GOPRO dataset. Although the blurriness of the original image was greatly improved, some distortion in facial colors remained, and blurred local edges made it difficult to clearly see the eyes and facial expressions. Adding the ConvLSTM bidirectional connected structure improved PSNR and SSIM by 1.38 dB and 0.04 respectively, validating the effectiveness of the ConvLSTM bidirectional connected structure. However, some artifacts remained in the eye and hair areas, failing to clearly restore the eye angles and some facial structures. After adopting the U-Net structure optimization strategy, PSNR and SSIM increased by 1.36 dB and 0.03 respectively, resulting in a significant improvement in image quality. The chin area was restored more realistically, and facial organs and expressions were clearly visible. This demonstrates that the multi-scale dense feature extraction module MDFE, the ConvLSTM bidirectional connected structure, and the U-Net network structure optimization strategy improve image quality from different aspects, and also proves the effectiveness of the proposed method. The method of this invention can effectively reduce and alleviate the problems of texture detail loss, inability to suppress noise, and ringing artifacts that occur in existing de-blurring networks during image restoration. Furthermore, dynamic scene deblurring improves the processing efficiency of subsequent target detection, image recognition, and other technologies.

[0094] This invention provides a system for deblurring dynamic scene images, comprising:

[0095] The network building module is used to build a dynamic scene image deblurring network, including a multi-scale dense feature extraction module connected between the convolutional layers of the U-Net network, a ConvLSTM bidirectional connected structure inserted between adjacent skip connections in the U-Net network, and a U-Net network structure optimization strategy inserted in the U-Net network encoder and decoder.

[0096] The network training module is used to construct the total loss function and train the dynamic scene image deblurring network to obtain the dynamic scene image deblurring network model.

[0097] The image processing module is used to input the image to be processed into the dynamic scene image deblurring network model to obtain the deblurred image.

[0098] The Multi-Scale Dense Feature Extraction (MDFE) module consists of a dense feature extraction network and an SPP feature pyramid. It enhances deep feature extraction and reuse through densely connected convolutional layers and uses spatial pyramid pooling branches to guide the transfer and fusion of multi-scale features, thereby promoting the effective preservation of image details and textures.

[0099] The ConvLSTM bidirectional connected structure consists of four ConvLSTM modules, divided into forward and reverse paths, and distributed in pairs between adjacent skip connections. It realizes high-dimensional information input based on the data correlation in the two directions, compensates for the lost context features from simple cascading in a non-linear manner from the encoding path, promotes the interaction of deep features across stages, and weakens edge artifacts and noise interference.

[0100] The U-Net network structure optimization strategy leverages dilated convolution downsampling to effectively expand the receptive field, avoiding irreversible image damage without increasing the number of parameters. Sub-pixel convolution is used during upsampling to obtain clear image details with small kernels, reducing computational complexity. Then, skip connections are used between the encoding input and decoding output to reduce the information difference between the input feature map and the decoder's output feature map, thereby improving convergence speed and image reconstruction quality.

[0101] One embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the above-described dynamic scene image deblurring method; or, when the processor executes the computer program, it implements the functions of each module in the above-described dynamic scene image deblurring system.

[0102] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention.

[0103] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0104] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0105] The memory can be used to store the computer program and / or module. The processor implements various functions of the terminal device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory.

[0106] If the modules / units integrated in the terminal device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0107] Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above-described dynamic scene image deblurring method. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form.

[0108] The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0109] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention. All or part of the process in the example method can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of the above-described dynamic scene image deblurring method. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form.

[0110] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for deblurring dynamic scene images, characterized in that, The specific steps are as follows: S1 constructs a dynamic scene image deblurring network, including a multi-scale dense feature extraction module connected between the convolutional layers of the U-Net network, a ConvLSTM bidirectional connected structure inserted between adjacent skip connections in the U-Net network, and a U-Net network structure optimization strategy inserted in the encoder and decoder of the U-Net network. S2 constructs the total loss function and trains the dynamic scene image deblurring network to obtain the dynamic scene image deblurring network model; S3 inputs the image to be processed into the dynamic scene image deblurring network model to obtain the deblurred image; In S1, the U-Net network structure optimization strategy includes dilated convolution and subpixel convolution. Dilated convolution is used for downsampling in the encoder, and subpixel convolution is used for upsampling in the decoder. In S1, the multi-scale dense feature extraction module includes a dense feature extraction network and a multi-level pyramid pooling branch SPP. The dense feature extraction network is used to obtain deep features of the image, and the multi-level pyramid pooling branch SPP is used to obtain multi-scale local detail features. The deep features and multi-scale local detail features are fused to obtain global detail features used to optimize the sharpness of blurred images.

2. The method for deblurring dynamic scene images according to claim 1, characterized in that, The dense feature extraction network includes convolutional layers from the original U-Net structure interconnected in a feedforward manner, with dense short connections between convolutional layers close to the input and output layers; the dense feature extraction network consists of the first... l t The input of each convolutional layer is t Given the number of subsequent layers in the densely connected structure, the output is: (1) in It consists of three consecutive operations: batch normalization, linear unit, and a 3×3 convolution.

3. The method for deblurring dynamic scene images according to claim 1, characterized in that, The multi-level pyramid pooling branch SPP includes three parallel 3×3 dilated convolutional capture blocks with dilation rates of 2, 4, and 8, respectively.

4. The method for deblurring dynamic scene images according to claim 1, characterized in that, The ConvLSTM bidirectional connectivity structure includes forward and reverse paths distributed between adjacent skip connections in the U-Net network. The forward path runs from bottom to top, and the reverse path runs from top to bottom. A convolutional long short-term memory (ConvLSTM) model is inserted in both the forward and reverse paths to extract multidimensional feature information from the context of adjacent features and fuse it with the current features in the decoding part.

5. The method for deblurring dynamic scene images according to claim 4, characterized in that, In a U-Net network, there are pairs of forward and reverse paths between adjacent hop connections. The path closer to the U-Net network encoder is the forward path, and the path closer to the U-Net network decoder is the reverse path.

6. The method for deblurring dynamic scene images according to claim 1, characterized in that, In S2, the total loss function include L 2. Loss function and L p Perceived loss, the formula is as follows: (2) Where λ represents the weight, which is usually taken as λ=0.01; Assume there is a training set ( B i , I i ),Include For blurry / sharp images, L 2. The formula for the loss function is as follows: (3) in, For parameters, F Network for deblurring dynamic scene images; L p The formula for perceived loss is as follows: (4) in: The image after deblurring; J The corresponding clear original image; and They represent and J The corresponding feature map, C j , H j and W j They represent and The number of channels, height, and width of the feature map.

7. A system for deblurring dynamic scene images using the method described in any one of claims 1 to 6, characterized in that, include: The network building module is used to build a dynamic scene image deblurring network, including a multi-scale dense feature extraction module connected between the convolutional layers of the U-Net network, a ConvLSTM bidirectional connected structure inserted between adjacent skip connections in the U-Net network, and a U-Net network structure optimization strategy inserted in the U-Net network encoder and decoder. The network training module is used to construct the total loss function and train the dynamic scene image deblurring network to obtain the dynamic scene image deblurring network model. The image processing module is used to input the image to be processed into the dynamic scene image deblurring network model to obtain the deblurred image.

8. A terminal device, characterized in that, The terminal device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method of any one of claims 1 to 6, or, when the processor executes the computer program, it implements the functions of each module in the system of claim 7.

Images