A video image motion blur removal method, system and experimental equipment

The object detection algorithm and the target recovery network directly repair the blurred area of ​​the video image, solving the problem of difficulty in estimating the fuzzy kernel, and achieving efficient video image demotion effect.

CN118096590BActive Publication Date: 2024-11-08CHANGCHUN UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410263088.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-08
Publication Date
2024-11-08
Estimated Expiration
2044-03-08

AI Technical Summary

Technical Problem

The fuzzy kernel estimation in the existing video image defuzzing method is difficult, resulting in poor restoration effect.

Method used

The object detection algorithm is used to identify the fuzzy frame of the moving target, separate the foreground and background, and use the target restoration network and multi-frame fusion method to generate clear images, and directly repair the fuzzy area bypassing the fuzzy kernel estimation.

Benefits of technology

It realizes efficient removal of motion blur of video images and generates clear video images, avoiding the difficulty of estimating blur kernels and improving the restoration effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118096590B_ABST
    Figure CN118096590B_ABST
Patent Text Reader

Abstract

The present invention belongs to the technical field of blurred image restoration technology, and in particular, is a method for de-motion blurring a video image, comprising the following steps: S1: constructing a target restoration network model; S2: pre-processing a blurred video image; S3: restoring the target area; S4: restoring the background; S5: fusing the image; S6: video output. The present invention constructs a target restoration network model, analyzes the 6D posture and scale transformation of the target in the blurred video, and then generates a high-definition image of the target, thereby bypassing the estimation of the blur kernel to complete the video image restoration task, thereby improving the restoration effect of the blurred image. In addition, the present invention also provides a video image de-motion blurring system, comprising: an acquisition module, a pre-processing module, a target foreground recovery module, a background recovery module, an image fusion module, and a video output module.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of blurred image restoration, and in particular to a method, system and experimental equipment for de-motion blurring of video images. Background Art

[0002] Video image deblurring technology is a very important image processing technology. It can restore video image blurring and distortion caused by a series of factors such as camera shake, slow shutter speed, object shaking, etc., which helps to improve image and video quality, provide more information, and enhance the accuracy of image analysis and recognition. This is very important for improving the effect of image processing and solving specific problems. At present, video deblurring technology has a wide range of applications in many fields, such as target recognition, object detection, medical imaging, etc.

[0003] Video image deblurring technology has made rapid progress since its development.

[0004] Depending on whether the blur kernel is known or not, commonly used image deblurring methods can be divided into two categories: one is when the blur kernel is known, called non-blind restoration; the other is when the blur kernel is unknown, called blind restoration.

[0005] Non-blind restoration is also known as traditional image restoration algorithm. This method performs deconvolution operations based on known blur kernels, such as inverse filtering, LR algorithm, Wiener filtering, etc. Since it is difficult to obtain an accurate blur kernel in practical applications, non-blind restoration performs poorly and cannot obtain a clear restored image.

[0006] The application scenarios of blind restoration in real scenes are more extensive. Most of the early studies used image priors, including total variation, heavy-tailed gradient priors, or super-Laplacian priors, which are usually applied to images in a coarse-to-fine manner.

[0007] Blur kernel is indispensable for both blind restoration and non-blind restoration. The estimation of blur kernel is difficult, and the quality of blur kernel estimation directly affects the effect of image restoration. In recent years, many blur kernel estimation methods have emerged, and the effect of restoring blurred images is also uneven.

[0008] Therefore, we propose a video image motion blur removal method to solve the above problem. Summary of the invention

[0009] 1. Technical issues to be solved

[0010] Aiming at the core problem of existing video image deblurring methods - the difficulty of blur kernel estimation. The present invention proposes a new method to restore blurred images, bypass blur kernel estimation, and directly repair blurred areas. The method first uses a target detection algorithm to identify the position of blurred frames of moving targets in the video and extract blurred areas, and then divides the blurred areas into foreground targets and backgrounds. The target restoration network and multi-frame fusion method are used to generate the foreground target and the corresponding background where the target is located, respectively, and finally the newly generated area is fused with the original blurred image to obtain the repaired image.

[0011] (II) Technical solution

[0012] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0013] A method for removing motion blur from a video image comprises the following steps:

[0014] S1: Build the target restoration network model, which consists of VGG16, RPN layer, deconvolution layer, mask layer and fully connected layer;

[0015] S2: Preprocessing of blurred video images: extracting the blurred area of ​​the blurred video image to be restored using the target detection algorithm, marking the temporal position of the blurred image in the video, and marking the image frame as a blurred frame;

[0016] S3: Restore the target area: Input the blurred area extracted in S2 into the target restoration network model in S1, and generate a clear image of the target with the correct posture and scale at the corresponding position. At this time, there is only the foreground target in the newly generated area, and there is no background in the original blurred area;

[0017] S4: Restoring the background: Use a multi-frame fusion method to combine the newly generated area in S3 with the nearest clear frame adjacent to the original blurred frame to repair the background area in the newly generated area;

[0018] S5: fused image: fuse the newly generated area after the background is repaired in S4 with the original blurred image to obtain a restored complete video image;

[0019] S6: Video output: All the repaired blurred images are replaced with corresponding video frames in the original video according to the time sequence positions marked in S2, and the video is output after completion.

[0020] Furthermore, in the S1 target restoration network model, the VGG16 layer is used to extract features of the entire image; the RPN layer is attached to the last convolutional layer of the VGG layer to output RoI; for each RoI, the corresponding features are extracted and used as inputs of three head branches; the three head branches are the box regression head branch, the segmentation head branch, and the pose head branch. The box regression head branch is used to output the location area of ​​the target; the segmentation head branch is used to segment the foreground target and the background; the pose head branch is used to output the pose of the target; the fully connected layer is used to regress the foreground target area and output a clear image of the target with the correct pose and scale at the corresponding position.

[0021] Furthermore, the temporal position in the video where the blurred image in S2 is located refers to the temporal position of the frame image in the process of sequentially playing the entire video.

[0022] Furthermore, the region newly generated by the target restoration network in S3 is specifically characterized in that a clear target with the same scale and posture has been generated at the target position of the original blurred region, while the pixel values ​​of other parts are zero.

[0023] Furthermore, the nearest clear frame adjacent to the blurred frame in S4 is obtained from the time sequence position of the blurred frame marked in S2. Specifically, it is the closest to the blurred frame in time sequence and is not marked as a blurred frame. And the background area in the newly generated area refers to the part of the background pixel value of zero in the newly generated area in S3.

[0024] In a second aspect, the present invention further provides a video image motion blur removal system, comprising:

[0025] An acquisition module, used for acquiring a target image data set;

[0026] A preprocessing module is used to preprocess the video image to be restored, extract the blurred area using a target detection algorithm, and record the temporal position of the blurred frame in the video;

[0027] The target foreground recovery module is used to generate a clear image of the target with the correct posture and size at the corresponding position;

[0028] A background restoration module, used to restore the missing background part of the area generated by the target foreground restoration module;

[0029] Image fusion module, used to fuse the complete newly generated area with the original blurred image;

[0030] The video output module is used to replace the corresponding video frames in the original video with all the repaired blurred images in the correct time sequence position, and output the video after completion.

[0031] In a third aspect, the present invention further provides an experimental device, comprising a digital camera and a computer. The digital camera is used to shoot a target image data set. The computer is used to store and execute the above-mentioned video image motion blur removal method.

[0032] (III) Beneficial effects

[0033] Compared with the prior art, the present invention provides a method, system and experimental equipment for de-motion blurring of video images, which have the following beneficial effects:

[0034] A target restoration network model was constructed, which consists of VGG16, RPN, deconvolution layer, and fully connected layer. It makes full use of various neural network structures to extract detail information and semantic information of blurred images, analyze the 6D posture and scale transformation of the target, and finally complete the restoration task of the foreground target.

[0035] The present invention removes motion blur by bypassing the estimation of blur kernels. A target restoration model is designed. The blurred area is input, and features are extracted by convolutional neural network. Then, RoI is output through RPN layer. The output RoI is used as input of three heads and finally output by fully connected layer to directly generate a clear image of the target at the corresponding position. Finally, the newly generated area is fused with the original blurred image to obtain a clear video image. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the process flow of the method for removing motion blur of video images of the present invention;

[0037] Figure 2 This is a schematic diagram of the target recovery network model structure of the present invention;

[0038] Figure 3 This is a flow chart of an embodiment of the method for de-motion blurring of a video image according to the present invention;

[0039] Figure 4 A structural schematic diagram of a video image motion blur removal method system provided by the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Example

[0042] like Figure 1As shown, an embodiment of the present invention provides a method for removing motion blur from a video image, and the method includes the following steps.

[0043] S1: Build the target recovery network model, such as Figure 2 As shown in the figure, the model consists of VGG16, RPN layer, deconvolution layer, mask layer and fully connected layer; in the S1 target restoration network model, the VGG16 layer is used to extract the features of the entire image; the RPN layer is attached to the last convolution layer of the VGG layer to output RoI; for each RoI, the corresponding features are extracted and used as the input of three head branches; the three head branches are box regression head branch, segmentation head branch and pose head branch; the box regression head branch is used to output the location area of ​​the target; the segmentation head branch is used to segment the foreground target and the background; the pose head branch is used to output the pose of the target; the fully connected layer is used to regress the foreground target area and output a clear image of the target with the correct pose and scale at the corresponding position.

[0044] After the target restoration network is built, the target dataset is prepared; the target dataset is constructed by the following method: 1,000 high-definition images of the target at various angles are prepared, and 73 sets of blur kernels are used to blur them, forming 73,000 pairs of blurred image pairs. Among them, the motion step size and motion angle of the blur kernel are set by the following parameters: the motion length range is discretized into 13 samples at equal intervals, the range is [1,25], and the step size is 2; the motion direction range is [0°,150°], and it is divided into 6 samples at equal intervals; when the motion length is 1, regardless of the motion direction, it is set as the identity kernel of the target. Finally, 73 blur kernels are generated with different combinations of motion lengths and directions; and the target dataset is split into a training set and a test set in a ratio of 7:3; the two sets of datasets are cropped, scaled, and rotated respectively, and the processed images are input into the training network.

[0045] S2: Preprocessing of blurred video images, using the target detection algorithm to extract the blurred area of ​​the blurred video image to be restored, marking the temporal position in the video where the blurred image is located, and marking the image frame as a blurred frame; the temporal position in the video where the blurred image is located in S2 refers to the temporal position of the frame image during the entire video sequential playback process. In this embodiment, the target detection algorithm used is the YOLOv5 algorithm. The network structure of YOLOv5 is mainly composed of an input end, a Backbone feature extraction network, a Neck feature fusion network, and a prediction output layer. The input end uses the image processing method of Mosaic data enhancement and black edge filling. The feature extraction network performs preliminary feature extraction on the input image. The feature fusion network uses the feature pyramid FPN structure and the path aggregation network PAN structure to fuse the position information extracted by the shallow network and the semantic information extracted by the deep network, so that the entire model can obtain richer feature information. After the collected real-time image data is transmitted to the input end of the detection network, the feature extraction network will extract features from its image. Feature fusion is then performed to enhance the feature expression of different scales and generate feature maps. Finally, the predicted feature maps of different scales are used to output information such as the target category, target number, target border, and confidence level that may exist in the image. Optionally, the target detection algorithm used in this embodiment includes but is not limited to SSD, YOLO series algorithms, RCNN, etc.

[0046] S3: Restore target area: Input the blurred area extracted in S2 into the target restoration network model in S1, and generate a clear target image with the correct posture and scale at the corresponding position. At this time, there is only the foreground target in the newly generated area, and there is no background in the original blurred area; the newly generated area in S3 through the target restoration network is specifically manifested as a clear target with the same scale and posture has been generated at the target position of the original blurred area, and the pixel values ​​of other parts are zero.

[0047] S4: Restore background: Use a multi-frame fusion method to fusion the newly generated area in S3 with the nearest clear frame adjacent to the original blurred frame to repair the background area in the newly generated area. The nearest clear frame adjacent to the blurred frame in S4 is obtained from the temporal position of the blurred frame marked in S2. Specifically, it is the closest to the blurred frame in time sequence and is not marked as a blurred frame. And the background area in the newly generated area refers to the part where the background pixel value is zero in the newly generated area in S3. The multi-frame fusion method refers to using the information of the previous frame and the next frame of the target frame to fuse and generate the image of the target frame. In this embodiment, it is specifically manifested as performing feature registration of the nearest frames before and after the blurred frame with the original blurred frame, aligning the two frames after the registration, and filling the missing background pixel values ​​in the area generated in S3 by averaging the corresponding background pixel values.

[0048] S5: Fused image: The newly generated area after the background is repaired in S4 is fused with the original blurred image to obtain a restored complete video image; the image fusion algorithm used in this embodiment is Laplacian pyramid fusion, which uses the Laplacian pyramid of the image for multi-resolution fusion. The image pyramid is composed of high-pass filter coefficients obtained by continuously downsampling (reducing) and upsampling (enlarging) the image. The advantage of Laplacian pyramid fusion is that it can fuse images at multiple scales, thereby retaining the detailed information of the image. This is very helpful to avoid excessive blur during the fusion process.

[0049] The following are the main steps of the Laplacian pyramid fusion used in this embodiment: 1. For each input image, first construct a Gaussian pyramid, which contains blurred versions of the image at different scales. The Gaussian pyramid is obtained by continuously downsampling the image, and the image at each level is half the size of the previous level. 2. Secondly, construct a Laplacian pyramid: wherein the Laplacian pyramid is obtained by subtracting the image of the previous level from the Gaussian pyramid. The Laplacian pyramid obtained in this way contains the detailed information of the image at different scales. 3. Finally, fuse the Laplacian pyramid: for each level of the Laplacian pyramid, the Laplacian pyramids of the two input images at the corresponding level are weighted added. The weights can be adjusted according to the needs of the application. Generally, the sum of the weights should be 1 to ensure that the fused image maintains the same brightness. 4. Reconstruct the fused image: By upsampling the fused Laplacian pyramid and adding the results of each level, the final fused image can be obtained.

[0050] S6: Video output: All the repaired blurred images are replaced with corresponding video frames in the original video according to the time sequence positions marked in S2, and the video is output after completion.

[0051] like Figure 3 As shown, it is a flow chart of an embodiment of the method for de-motion blurring of video images according to the present invention, wherein the symbol F represents a certain video frame, such as F1 represents the first frame.

[0052] A video image motion deblurring system provided by the present invention is described below. The video image motion deblurring system described below and the video image motion deblurring method described above can be referenced to each other.

[0053] like Figure 4 As shown, it is a structural schematic diagram of a video image motion blur removal system provided by the present invention, including: an acquisition module, a preprocessing module, a target foreground recovery module, a background recovery module, an image fusion module, and a video output module.

[0054] Among them: the acquisition module is used to acquire the target image data set; the preprocessing module is used to preprocess the video image to be restored, use the target detection algorithm to extract the blurred area and record the time sequence position of the blurred frame in the video; the target foreground recovery module is used to generate a clear target image with the correct posture and size at the corresponding position; the background recovery module is used to restore the missing background part of the area generated by the target foreground recovery module; the image fusion module is used to fuse the complete newly generated area with the original blurred image; the video output module is used to replace the corresponding video frames in the original video with all the repaired blurred images according to the correct time sequence position, and output the video after completion.

[0055] On the other hand, the present invention also provides an experimental device. The experimental device includes a digital camera and a computer terminal. The digital camera is used to shoot a target omnidirectional high-definition image data set for training a target foreground restoration network. The computer terminal is used to store and execute the above-mentioned video image deblurring method.

[0056] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for removing motion blur from a video image, characterized in that: The specific steps are: S1: Build the target restoration network model, which consists of VGG16, RPN layer, deconvolution layer, mask layer and fully connected layer; S2: Preprocessing of blurred video images: using the target detection algorithm to extract the blurred area of ​​the blurred video image to be restored, marking the temporal position of the blurred image in the video, and marking the image frame as a blurred frame; S3: Restore target area: Input the blurred area extracted in S2 into the target restoration network model in S1, and generate a clear image of the target with the correct posture and scale at the corresponding position; at this time, there is only the foreground target in the newly generated area, without the background in the original blurred area; S4: Restoring the background: Use a multi-frame fusion method to combine the newly generated area in S3 with the nearest clear frame adjacent to the original blurred frame to repair the background area in the newly generated area; S5: fused image: fuse the newly generated area after the background is repaired in S4 with the original blurred image to obtain a restored complete video image; S6: Video output: Replace the corresponding video frame in the original video with the restored complete video image according to the time sequence position marked in S2, and output the video after completion.

2. The method for de-motion blurring of a video image according to claim 1, characterized in that: In the S1 target restoration network model, the VGG16 layer is used to extract the features of the entire image; The RPN layer is attached to the last convolutional layer of the VGG layer to output the RoI. For each RoI, the corresponding features are extracted and used as the input of the three head branches. The three head branches are box regression head branch, segmentation head branch and pose head branch; The box regression head branch is used to output the location area of ​​the target; The segmentation head branch is used to segment the foreground target and the background; the pose head branch is used to output the pose of the target; the fully connected layer is used to regress the foreground target area and output a clear image of the target with the correct pose and scale at the corresponding position.

3. The method for de-motion blurring of a video image according to claim 1, characterized in that: The temporal position in the video where the blurred image in S2 is located refers to the temporal position of the frame image during the sequential playback of the entire video.

4. The method for de-motion blurring of a video image according to claim 1, characterized in that: The region newly generated by the target restoration network in S3 is specifically characterized in that a clear target with the same scale and posture has been generated at the target position of the original blurred region, while the pixel values ​​of other parts are zero.

5. The method for de-motion blurring of a video image according to claim 1, characterized in that: The nearest clear frame adjacent to the blurred frame in S4 is obtained from the time sequence position of the blurred frame marked in S2; Specifically, the region that is closest to the blurred frame in time sequence and is not marked as a blurred frame; and the background region in the newly generated region refers to the region in S3 where the background pixel value is zero.

6. A video image motion blur removal system, characterized in that: include: An acquisition module, used for acquiring a target image data set; A preprocessing module is used to preprocess the video image to be restored, extract the blurred area using a target detection algorithm, and record the temporal position of the blurred frame in the video; The target foreground recovery module is used to generate a clear image of the target with the correct posture and size at the corresponding position; A background restoration module, used to restore the missing background part of the area generated by the target foreground restoration module; An image fusion module is used to fuse the complete newly generated area with the original blurred image; a video output module is used to repair the complete video image and replace the corresponding video frame in the original video according to the correct time sequence position. After completion, the video is output, and a video image motion blur removal method as described in claims 1-5 is executed.

7. An experimental device, comprising a digital camera and a computer, characterized in that: The digital camera is used to capture a target image data set; the computer is used to store and execute a video image motion blur removal method as described in claims 1-5.