Image blur determination method, dataset construction method and deblurring method

By weighted fusion and blur degree calculation of continuous n-frame images, combined with the blur degree perception network, the problem of difficulty in distinguishing blur degree in different areas of the image is solved, and the image debum effect is improved.

CN113409203BActive Publication Date: 2024-11-26GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110649891.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-10
Publication Date
2024-11-26
Estimated Expiration
2041-06-10

AI Technical Summary

Technical Problem

The prior art cannot finely distinguish the degree of blurring in different areas of the image during the image debuffing process, resulting in poor debuffing effect.

Method used

By acquiring the original images of continuous n-frames, synthesize the target images and calculate the blur degree value of each pixel point, the data set is constructed and trained using the blur degree perception network to be constructed and trained, and the images to be processed are differentiated and defuzzed.

Benefits of technology

The fine blur degree evaluation and differentiated defuzzing processing of different areas of the image are realized, which improves the defuzzing effect and reduces the calculation cost.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113409203B_ABST
    Figure CN113409203B_ABST
Patent Text Reader

Abstract

The present disclosure provides a method for determining the blur degree of an image, a method for constructing a blurred image data set, an image deblurring method, a device, a storage medium and an electronic device, and relates to the field of image and video processing technology. The method for determining the blur degree of an image comprises: obtaining n consecutive frames of original images and a target image synthesized by weighted fusion of pixels at the same position in the n consecutive frames of original images, where n is a positive integer not less than 2; obtaining the pixel value of the pixel at the same position corresponding to each pixel point in the target image in each frame of the original image; and determining the blur degree data of the target image, including the blur degree value of each pixel point in the target image, according to the degree of deviation between the n pixel values ​​corresponding to each pixel point in the n frames of the original image. The present disclosure can represent the blur degree of different areas in an image, which is conducive to improving the image deblurring effect.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] During the image capture process, it is common that the image is blurred due to shaking, loss of focus or other reasons. When deblurring the image, it is usually necessary to first calculate the blur degree of the image so as to implement specific deblurring processing using a matching standard.

[0003] In the related art, most methods for calculating the degree of blur are to calculate an overall degree of blur for the entire image, which cannot reflect the difference in degree of blur between different regions of the image. Furthermore, the same standard is used for the entire image during deblurring, resulting in poor deblurring effect. Summary of the invention

[0004] The present disclosure provides a method for determining the blur degree of an image, a method for constructing a blurred image data set, an image deblurring method, an image blur degree determining device, a blurred image data set constructing device, an image deblurring device, a computer-readable storage medium and an electronic device, thereby at least to a certain extent solving the problems in the related art that the blur degree cannot be determined separately for different areas in an image and the image deblurring effect is poor.

[0005] According to a first aspect of the present disclosure, a method for determining the degree of blur of an image is provided, comprising: obtaining n consecutive frames of original images and a target image synthesized by weighted fusion of pixels at the same position in the n consecutive frames of original images, where n is a positive integer not less than 2; obtaining the pixel value of each pixel in the target image corresponding to the pixel at the same position in each frame of the original image; determining blur degree data of the target image according to the degree of deviation between n pixel values ​​corresponding to each pixel in the n frames of original images, the blur degree data of the target image including the blur degree value of each pixel in the target image.

[0006] According to a second aspect of the present disclosure, a method for constructing a blurred image dataset is provided, comprising: acquiring a target image and blur degree data of the target image determined according to the image blur degree determination method of the first aspect; taking the target image as a sample image and the blur degree data of the target image as a first label to construct a blurred image dataset; the blurred image dataset is used to train a blur degree perception network, and the blur degree perception network is used to determine the blur degree data of an image input into the blur degree perception network.

[0007] According to a third aspect of the present disclosure, there is provided an image deblurring method, comprising: acquiring an image to be processed; determining blur degree data of the image to be processed using the image to be processed as a target image according to the image blur degree determination method of the first aspect; and performing deblurring processing on the image to be processed based on the blur degree data of the image to be processed to obtain a deblurred image corresponding to the image to be processed.

[0008] According to a fourth aspect of the present disclosure, there is provided an image deblurring method, comprising: acquiring an image to be processed; processing the image to be processed using a blur degree perception network to obtain blur degree data of the image to be processed; based on the blur degree data of the image to be processed, deblurring the image to be processed to obtain a deblurred image corresponding to the image to be processed; wherein the blur degree perception network is trained using a blur image dataset constructed by the blur image dataset construction method of the second aspect above.

[0009] According to a fifth aspect of the present disclosure, a device for determining the degree of blurriness of an image is provided, comprising: an image acquisition module, configured to acquire n consecutive frames of original images and a target image synthesized by weighted fusion of pixels at the same position in the n consecutive frames of original images, where n is a positive integer not less than 2; a pixel value acquisition module, configured to acquire the pixel value of each pixel in the target image corresponding to the pixel at the same position in each frame of the original image; a blurriness data determination module, configured to determine the blurriness data of the target image according to the degree of deviation between the n pixel values ​​corresponding to each pixel in the n frames of original images, wherein the blurriness data of the target image includes the blurriness value of each pixel in the target image.

[0010] According to a sixth aspect of the present disclosure, a device for constructing a blurred image dataset is provided, comprising: a data acquisition module, configured to acquire a target image and blur degree data of the target image determined according to the image blur degree determination method of the first aspect above; a dataset construction module, configured to construct a blurred image dataset using the target image as a sample image and the blur degree data of the target image as a first label; the blurred image dataset is used to train a blur degree perception network, and the blur degree perception network is used to determine the blur degree data of an image input into the blur degree perception network.

[0011] According to the seventh aspect of the present disclosure, an image deblurring device is provided, comprising: an image acquisition module, configured to acquire an image to be processed; a blur degree data determination module, configured to determine the blur degree data of the image to be processed based on the image blur degree determination method of the first aspect, taking the image to be processed as the target image; and a deblurring processing module, configured to perform deblurring processing on the image to be processed based on the blur degree data of the image to be processed, so as to obtain a deblurred image corresponding to the image to be processed.

[0012] According to an eighth aspect of the present disclosure, an image deblurring device is provided, comprising: an image acquisition module, configured to acquire an image to be processed; a blur degree data determination module, configured to process the image to be processed using a blur degree perception network, and obtain blur degree data of the image to be processed; a deblurring processing module, configured to perform deblurring processing on the image to be processed based on the blur degree data of the image to be processed, and obtain a deblurred image corresponding to the image to be processed; wherein the blur degree perception network is trained using a blur image data set constructed by the blur image data set construction method of the second aspect above.

[0013] According to the ninth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for determining the degree of image blur of the first aspect, the method for constructing a blurred image data set of the second aspect, the image deblurring method of the third aspect, or the image deblurring method of the fourth aspect is implemented.

[0014] According to the tenth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the image blur degree determination method of the first aspect, the blurred image data set construction method of the second aspect, the image deblurring method of the third aspect, or the image deblurring method of the fourth aspect by executing the executable instructions.

[0015] The technical solution disclosed in this disclosure has the following beneficial effects:

[0016] The present disclosure provides a technical solution for determining the blur degree value of each pixel in an image. On the one hand, compared with the solution for calculating the overall blur degree value of an image in the related art, the present solution can more finely represent the blur degree of different regions in the image, and reflect the difference in blur degree between different regions, so as to facilitate further optimization by using appropriate methods and parameters for each region. For example, different regions in the image can be deblurred using differentiated parameters, which is conducive to improving the deblurring effect. On the other hand, the present solution determines the blur degree data based on the degree of pixel value deviation between the original images, and the calculation process is relatively simple, with a low implementation cost.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram showing a system architecture in this exemplary embodiment;

[0019] Figure 2 A schematic diagram showing the structure of an electronic device in this exemplary embodiment is shown;

[0020] Figure 3 A flow chart showing a method for determining the degree of image blur in this exemplary embodiment is shown;

[0021] Figure 4 A flow chart of synthesizing a target image in this exemplary embodiment is shown;

[0022] Figure 5 A schematic diagram showing a method of generating a target image and a blur degree image in this exemplary embodiment is shown;

[0023] Figure 6 A flowchart showing a method for constructing a blurred image data set in this exemplary embodiment is shown;

[0024] Figure 7 A schematic diagram of the structure of a fuzziness perception network in this exemplary embodiment is shown;

[0025] Figure 8 A schematic diagram showing the structure of a defuzzification network and a feature perception network in this exemplary embodiment is shown;

[0026] Fig. 9 A flow chart of an image deblurring method according to the exemplary embodiment is shown;

[0027] Fig.10 A flowchart showing another image deblurring method in this exemplary embodiment is shown;

[0028] Fig.11 A schematic flowchart of image deblurring in this exemplary embodiment is shown;

[0029] Fig.12 A flowchart of a training network in this exemplary embodiment is shown;

[0030] Fig.13 A schematic diagram showing the structure of a device for determining the degree of image blur in this exemplary embodiment is shown;

[0031] Fig.14A schematic diagram showing the structure of a device for constructing a blurred image data set in this exemplary embodiment is shown;

[0032] Fig.15 A schematic diagram showing the structure of an image deblurring device in this exemplary embodiment is shown;

[0033] Fig.16 A schematic structural diagram of another image deblurring device in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0034] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0035] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0037] In one solution of the related technology, the overall blur degree of the entire image is calculated, and then a corresponding blur kernel is generated, and the blur kernel is used to deblur the entire image. However, differences in scene depth and texture sparseness in different areas of the image will lead to differences in blur degrees in different areas. Using one blur kernel to deblur different areas cannot achieve optimal deblurring for each area, resulting in a large number of defects in the deblurred image, such as noise in some areas, blur in some areas, and even artifacts, ringing and other undesirable phenomena.

[0038] In view of the above problems, an exemplary embodiment of the present disclosure provides a method for determining the blur level of an image. The following first introduces the system architecture of the operating environment of the method for determining the blur level of an image.

[0039] Figure 1 The schematic diagram of the system architecture is shown, and the system architecture 100 may include a terminal 110 and a server 120. The terminal 110 may be a terminal device such as a desktop computer, a laptop computer, a smart phone, a tablet computer, etc., and the server 120 may be a server that provides image processing related services, or a cluster formed by multiple servers. The terminal 110 and the server 120 may be connected through a wired or wireless communication link to exchange data. The terminal 110 may shoot or receive or extract multiple frames of continuous original images in other ways. In one embodiment, the terminal 110 may send multiple frames of continuous original images to the server 120, and the server 120 may synthesize a target image based on the multiple frames of continuous original images, and output the blur degree data of the target image by executing the image blur degree determination method in this exemplary embodiment, and return the target image and the blur degree data to the terminal 110. In one embodiment, the terminal 110 may synthesize the target image from multiple frames of continuous original images, and send the original image and the target image to the server 120, and the server 120 may output the blur degree data of the target image by executing the image blur degree determination method in this exemplary embodiment, and return the blur degree data to the terminal 110. In one implementation, after synthesizing the target image from multiple consecutive frames of original images, the terminal 110 may continue to locally execute the image blur degree determination method in this exemplary implementation to obtain blur degree data of the target image.

[0040] As can be seen from the above, in this exemplary embodiment, the execution subject of the image blur degree determination method can be the above-mentioned terminal 110, or the above-mentioned server 120. The exemplary embodiment of the present disclosure also provides an electronic device for executing the image blur degree determination method, and the electronic device can be the terminal 110 or the server 120. Figure 2Taking the mobile terminal 200 in the example, the structure of the above electronic device is exemplarily described. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 2 The construction in can also be applied to fixed type equipment.

[0041] like Figure 2 As shown, the mobile terminal 200 may specifically include: a processor 210, an internal memory 221, an external memory interface 222, a USB (Universal Serial Bus) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a button 294 and a SIM (Subscriber Identification Module) card interface 295, etc.

[0042] The processor 210 may include one or more processing units, for example, the processor 210 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.

[0043] The encoder can encode (i.e. compress) an image or video, for example, encode the current image to obtain code stream data; the decoder can decode (i.e. decompress) the code stream data of the image or video to restore the image or video data. The mobile terminal 200 can support one or more encoders and decoders. In this way, the mobile terminal 200 can process images or videos in multiple encoding formats, such as: JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap) and other image formats, MPEG (Moving Picture Experts Group) 1, MPEG2, H.263, H.264, HEVC (High Efficiency Video Coding) and other video formats.

[0044] In one implementation, the processor 210 may include one or more interfaces, and may be connected to other components of the mobile terminal 200 via different interfaces.

[0045] The internal memory 221 may be used to store computer executable program codes, which include instructions. The internal memory 221 may include volatile memory and non-volatile memory. The processor 210 executes various functional applications and data processing of the mobile terminal 200 by running the instructions stored in the internal memory 221.

[0046] The external memory interface 222 can be used to connect an external memory, such as a Micro SD card, to expand the storage capacity of the mobile terminal 200. The external memory communicates with the processor 210 through the external memory interface 222 to implement data storage functions, such as storing images, videos and other files.

[0047] The USB interface 230 is an interface that complies with USB standard specifications and can be used to connect a charger to charge the mobile terminal 200, or to connect headphones or other electronic devices.

[0048] The charging management module 240 is used to receive charging input from a charger. While the charging management module 240 is charging the battery 242, it can also power the device through the power management module 241; the power management module 241 can also monitor the status of the battery.

[0049] The wireless communication function of the mobile terminal 200 can be implemented by antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modulation and demodulation processor and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 250 can provide 2G, 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 200. The wireless communication module 260 can provide WLAN (Wireless Local Area Networks, wireless local area network) (such as Wi-Fi (Wireless Fidelity) network), BT (Bluetooth), GNSS (Global Navigation Satellite System, Global Navigation Satellite System), FM (Frequency Modulation, frequency modulation), NFC (Near Field Communication, short-range wireless communication technology), IR (Infrared, infrared technology) and other wireless communication solutions applied to the mobile terminal 200.

[0050] The mobile terminal 200 can realize the display function and display the user interface through the GPU, the display screen 290 and the AP, etc. For example, when the user performs camera detection, the mobile terminal 200 can display the interface of the camera detection App (Application) on the display screen 290.

[0051] The mobile terminal 200 can realize the shooting function through the ISP, the camera module 291, the encoder, the decoder, the GPU, the display screen 290 and the AP, etc. For example, the user can turn on the image or video shooting function in the hidden camera detection App, and the image of the space to be detected can be collected through the camera module 291.

[0052] The mobile terminal 200 can implement audio functions through the audio module 270, the speaker 271, the receiver 272, the microphone 273, the earphone interface 274 and the AP.

[0053] The sensor module 280 may include a depth sensor 2801 , a pressure sensor 2802 , a gyroscope sensor 2803 , an air pressure sensor 2804 , etc., to implement corresponding sensing detection functions.

[0054] Indicator 292 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc. Motor 293 may generate vibration prompts, and may also be used for touch vibration feedback, etc. Buttons 294 include a power button, a volume button, etc.

[0055] The mobile terminal 200 may support one or more SIM card interfaces 295 for connecting to a SIM card to implement functions such as calls and mobile communications.

[0056] Figure 3 An exemplary process of the above-mentioned method for determining the blur level of an image is shown, which may include:

[0057] Step S310, obtaining n consecutive frames of original images and a target image synthesized by weighted fusion of pixels at the same position in the n consecutive frames of original images, where n is a positive integer not less than 2;

[0058] Step S320, obtaining the pixel value of the pixel point at the same position corresponding to each pixel point in the target image in each frame of the original image;

[0059] Step S330, determining blur degree data of the target image, including blur degree value of each pixel in the target image, according to the degree of deviation between n pixel values ​​corresponding to each pixel in the target image in n frames of original images.

[0060] Through the above method, a technical solution is provided to determine the blur degree value of each pixel in the image. On the one hand, compared with the solution of calculating the overall blur degree value of the image in the related art, this solution can more finely represent the blur degree of different areas in the image, and reflect the difference in blur degree between different areas, so as to facilitate further optimization by using appropriate methods and parameters for each area. For example, different parameters can be used to deblur different areas in the image, which is conducive to improving the deblurring effect. On the other hand, this solution determines the blur degree data based on the degree of pixel value deviation between the original images, and the calculation process is relatively simple, with a low implementation cost.

[0061] The following are Figure 3 Each step in the procedure is described in detail.

[0062] refer to Figure 3 In step S310, n consecutive frames of original images and a target image synthesized by weighted fusion of pixels at the same position in the n consecutive frames of original images are obtained, where n is a positive integer not less than 2.

[0063] The continuous n frames of original images may be n frames of images collected continuously during shooting, or may be n frames of images in a video, and the n frames of original images are images collected for the same shooting object or scene. In the continuous n frames of original images, there are generally moving objects, or the camera moves when shooting the continuous n frames of original images. Therefore, in the target image synthesized by the continuous n frames of original images, there is a certain degree of blur, that is, the target image can be regarded as a blurred image.

[0064] The number of pixels of n consecutive frames of original images can be the same. When synthesizing the target image, pixel-level synthesis can be used, that is, weighted fusion of pixels at the same position in n frames of original images is performed to obtain a pixel value, which is used as the pixel value of the corresponding position in the target image. Alternatively, when the number of pixels of n consecutive frames of original images is different, part of the original images can be upsampled or downsampled first to make the number of pixels of all n frames of original images the same, and then pixel-level synthesis can be performed.

[0065] In one embodiment, reference Figure 4 As shown, step S310 may include the following steps S410 and S420:

[0066] Step S410, obtaining n consecutive frames of original images.

[0067] Among them, n can be a fixed value or a variable value. Generally, the larger the value of n, that is, the more original images are involved in synthesizing the target image, the higher the degree of motion of the moving object, or the higher the degree of motion of the camera when shooting these original images, the more blurred the synthesized target image is. In one embodiment, the method for determining the degree of image blur may also include the following steps:

[0068] Before acquiring n consecutive frames of original images, the value of n is determined based on the expected global blur level value of the target image.

[0069] The global blur value is a quantitative parameter that characterizes the overall blur of an image. A custom parameter may be used, such as taking the average blur value of each pixel in an image as the global blur of the image. A commonly used blur parameter in the industry may also be used, such as a blur value based on the gradient of pixel values ​​in the image. In this exemplary embodiment, the expected global blur value is determined according to the actual application requirements of the target image, that is, the global blur value that the target image is expected to achieve.

[0070] It should be noted that the blur degree of the target image is positively correlated with the value of n, which mainly refers to the positive correlation between the global blur degree value and the value of n, not that the blur degree of each area in the target image is positively correlated with n. For example, the pure color background areas in multiple consecutive frames of original images are the same. As the value of n increases, the blur degree of the pure color background area in the synthesized target image does not change, then the blur degree of the pure color background area has nothing to do with n.

[0071] By pre-stating the global blur value of a large number of synthesized images and the number of original images involved in the synthesis, the corresponding relationship between the global blur value and n can be obtained, for example, the two parameters are in a linear relationship, etc. Therefore, after determining the expected global blur value of the target image, the corresponding value of n can be determined based on the corresponding relationship.

[0072] In one embodiment, n consecutive max Frame original image, n max is the maximum value that n can take, which can be determined based on experience or actual scenario requirements. After determining the value of n, max Select n consecutive frames of original images from the original frame, from the 1st frame to the nth frame. max -n+1 frames, that is, the selected consecutive n frames of original images can be located in the consecutive n max Any position in the original frame.

[0073] In one embodiment, it is considered that in the process of shooting n consecutive frames of original images, the camera may have a certain displacement or jitter, resulting in deviations in the shooting position and viewing angle corresponding to the n consecutive frames of original images. In order to facilitate the subsequent precise fusion, the n frames of original images can be registered, for example, the previous frame is sequentially registered with the next frame, or a reference frame is selected from the n frames of original images, and the other frames are all registered with the reference frame. The present disclosure does not limit the specific registration algorithm. For example, feature points can be extracted from the two frames of original images to be registered, and the feature points between the two frames of original images can be matched to obtain feature point matching pairs. The feature point matching pairs are then used to solve the posture transformation relationship between the two frames of original images, so as to transform any of the frames to achieve registration.

[0074] Step S420 , weighted fusion is performed on the pixels at the same position in the above n consecutive frames of original images, so as to synthesize the n consecutive frames of original images into a target image.

[0075] For example, the number of pixels of the continuous n frames of original images is W*H. Starting from a certain position, the pixel points at the position in each frame of the original image are selected and weighted fusion is performed. For example, starting from the position of the upper left corner pixel coordinate (1,1), the pixel point at the (1,1) position in each frame of the original image (referred to as the (1,1) pixel point) is selected to obtain n (1,1) pixel points, and the pixel values ​​are weighted fused to obtain the (1,1) fused pixel value. This exemplary embodiment can perform weighted fusion on the n pixels in each color channel respectively, for example, weighted fusion of the numerical values ​​of the R, G, and B channels respectively, or the pixel values ​​of the color modes such as RGB can be converted into gray values, and the gray values ​​can be weighted fused. The present disclosure does not limit the weight values ​​used for weighted fusion. For example, equal weights can be set for n frames of original images, and weighted fusion is equivalent to calculating the average pixel value for n pixels. Different weights can also be set for n frames of original images according to actual scene requirements, such as setting the weights of n frames of original images to increase or decrease according to the shooting time sequence of the n frames of original images. Different weights can also be set for the pixels at each position for the n frames of original images. For example, the gradient values ​​of the pixel (1,1) in the n frames of original images are calculated respectively, and the weight of the pixel (1,1) in the n frames of original images is set according to the magnitude relationship of the n gradient values. Generally, the larger the gradient value, the clearer the pixel (1,1) in the frame of original image is, and the larger its weight is. For example, the gradient value of the pixel (1,1) in the n frames of original images is normalized to obtain the weight of the pixel (1,1) in the n frames of original images. The pixel value weighted fusion method is adopted to traverse the pixel points at each position in a certain order, for example, starting from the position (1,1), traversing from left to right and from top to bottom to the position (W,H), and weighted fusion is performed on the n pixels at each position in the same manner as the above-mentioned pixel (1,1) to obtain the fused pixel value at each position. In this way, the fused pixel values ​​at all positions form a target image.

[0076] Continue to refer Figure 3 In step S320, the pixel value of each pixel in the target image corresponding to the pixel at the same position in each frame of the original image is obtained.

[0077] From the above, we can know that the target image is formed by the fusion of the pixels at each position in the n frames of the original image. Then, each pixel in the target image has a corresponding pixel at the same position in each frame of the original image. By obtaining the pixel values ​​of these pixels, we can obtain the n pixel values ​​corresponding to each pixel in the target image in the n frames of the original image. For example, for the (1,1) pixel in the target image, it corresponds to the pixel value at the (1,1) position in each frame of the original image, so we can obtain the n pixel values ​​corresponding to the (1,1) pixel. Based on the same method, we can obtain the n pixel values ​​corresponding to each pixel in the target image.

[0078] Continue to refer Figure 3 In step S330, the blur degree data of the target image is determined according to the degree of deviation between the n pixel values ​​corresponding to each pixel in the target image in the n frames of the original image, including the blur degree value of each pixel in the target image.

[0079] The degree of deviation between n pixel values ​​can be quantitatively expressed by the variance or standard deviation between the n pixel values. Generally, the higher the degree of deviation between the n pixel values ​​corresponding to a certain pixel point in the target image, the higher the blur value of the pixel point, and the two can satisfy a linear or nonlinear positive correlation. Exemplarily, the standard deviation between the n pixel values ​​corresponding to each pixel point can be used as the blur value of each pixel point, as shown below:

[0080]

[0081] Where (x, y) represents the position coordinates of the pixel point, D(x, y) represents the blur value of the pixel point (x, y) in the target image, and L i (x,y) represents the pixel value of the pixel point (x,y) in the i-th frame of the original image. Formula (1) represents the calculation of the standard deviation of the pixel value of the pixel point (x,y) in the n-frame original image as the blur value of the pixel point (x,y) in the target image. The larger the standard deviation, the greater the difference in image information about the point (x,y) in the n-frame original image. When synthesized into a target image, the more image information is lost at the point (x,y), so the blur value is higher.

[0082] It should be noted that formula (1) directly uses the standard deviation as the blur value, and the value of D(x, y) is related to L i The value of (x, y) is related to the size of L i When (x, y) is between 0 and 1, D(x, y) is generally also between 0 and 1. i When (x, y) is 0 to 255, D(x, y) is generally also within the range of 0 to 255. In one implementation, D(x, y) may be mapped according to a preset numerical range of the blur level value (related to actual scene requirements, such as 0 to 1, 0 to 100, etc.), so that the value of D(x, y) is converted to within the preset numerical range. For example, when the preset numerical range is 0 to 1, D(x, y) may be normalized.

[0083] When calculating the degree of deviation between pixel values, the pixel values ​​can be converted into grayscale values ​​and calculated, such as L in formula (1): i(x, y) can be the grayscale value of the pixel (x, y) in the original image of the i-th frame, or it can be calculated on each color channel separately, such as in formula (1), L i The values ​​of the R, G, and B channels of (x, y) are used to calculate D(x, y) of the three channels, and then the D(x, y) of the three channels are fused, such as calculating the average value or weighted average value, to obtain the blur value of the pixel point (x, y).

[0084] In one embodiment, the method for determining the image blur level may further include the following steps:

[0085] According to the blur degree data of the target image, a blur degree image corresponding to the target image is generated.

[0086] The pixel value of each pixel in the blur degree image is the blur degree value of each pixel in the target image. That is, the blur degree image represents the blur degree value of each pixel in the target image as a pixel value, and displays the blur degree data of the target image in a visual form. The blur degree image has the same number of pixels as the target image, which is convenient for storage and processing.

[0087] Figure 5 The schematic process of generating a target image and a blur degree image from n consecutive frames of original images is shown. In the n frames of original images, the pixels at the same position are extracted, the average pixel value is calculated, and the pixel value at the corresponding position in the target image is obtained. The standard deviation of the pixel value is calculated to obtain the pixel value at the corresponding position in the blur degree image.

[0088] An exemplary embodiment of the present disclosure also provides a method for constructing a blurred image dataset. Figure 6 An exemplary process of the method for constructing a fuzzy image dataset is shown, including the following steps S610 and S620:

[0089] Step S610, acquiring a target image and blur degree data of the target image determined according to the above-mentioned image blur degree determination method;

[0090] Step S620: Taking the target image as a sample image and the blur degree data of the target image as a first label, a blurred image data set is constructed.

[0091] For example, Figure 3 The target image obtained by the image blur degree determination method is regarded as a blurred image, which can be used as a sample image. The blur degree data of the target image is used as the corresponding label (Ground Truth). In order to distinguish other forms of labels, it is recorded as the first label here. The sample image-first label forms a training array. By obtaining a large number of training arrays, a blurred image dataset can be constructed.

[0092] The fuzzy image dataset can be used to train a fuzziness perception network, and the fuzziness perception network is used to determine the fuzziness data of an image input into the fuzziness perception network. The fuzziness perception network can be an end-to-end structured network. Figure 7 FIG. 1 shows a schematic structural diagram of a blur perception network, which may adopt a U-Net structure. For example, after a sample image is input into the blur perception network, the convolution layer 1 performs one or more convolution operations ( Figure 6 It is shown in the figure that the convolution layer 1 performs two convolution operations, and the present disclosure does not limit the specific number of convolution operations in each convolution layer), and then undergoes a pooling operation to obtain a feature image with a reduced size; the convolution layer 2 performs another round of convolution and pooling operations to obtain a feature image with a further reduced size; the convolution layer 3 performs another round of convolution and pooling operations to obtain a feature image with an even smaller size; a convolution operation is performed in the convolution layer 4, but no pooling operation is performed; then the transposed convolution layer 1 is entered, a transposed convolution operation is first performed, and then it is spliced ​​with the feature image in the convolution layer 3, and then one or more convolution operations are performed to obtain a feature image with an increased size; the transposed convolution layer 2 performs another round of transposed convolution operations, splicing with the feature image in the convolution layer 2, and a convolution operation to obtain a feature image with a further increased size; finally, the transposed convolution layer 3 performs another round of the above operations to output blur degree data. It should be noted that the present disclosure does not limit the number of convolutional layers and transposed convolutional layers in the blur perception network. According to the actual scenario requirements, other types of intermediate layers, such as Dropout layers, fully connected layers, etc., can also be added to the blur perception network.

[0093] According to the difference between the sample blur data (here, the blur data of the sample image is output, so it is called sample blur data) output by the blur perception network and the blur data as the first label, the loss function value is calculated. For example, a loss function can be established based on the MSE (Mean Square Error) between the sample blur data and the first label, and the loss function value is obtained after substituting the sample blur data and the first label. The parameters of the blur perception network are updated using the loss function value, such as by performing gradient descent update on the parameters of the network. Through multiple iterations, the network reaches a certain accuracy, or the loss function value converges, thereby completing the training of the blur perception network.

[0094] In one embodiment, the method for constructing a fuzzy image dataset may further include the following steps:

[0095] One original image among the n original images used to synthesize the target image is obtained, and the original image is used as a second label and added to the blurred image dataset.

[0096] The original image can be regarded as a clear image corresponding to the sample image (i.e., the target image), and it is used as another type of label corresponding to the sample image. In order to distinguish the first label mentioned above, it is recorded as the second label here. The second label is added to the blurred image dataset to form a binary training array with the sample image, or a ternary training array with the sample image and the first label.

[0097] In this exemplary embodiment, any original image in the above-mentioned continuous n frames of original images can be used as the second label, or after evaluating the blurriness of the original images, a frame of original images with the lowest blurriness value can be selected as the second label. For example, the blurriness perception network is used to process each frame of original images to obtain blurriness data of each frame of original images, and the average blurriness value of each frame of original images is calculated based on the blurriness data, and the frame of original images with the lowest average blurriness value is selected as the second label.

[0098] The blurred image data set containing the second label can also be used to train a deblurring network, where the deblurring network is used to deblur an image input into the deblurring network and output a deblurred image corresponding to the image.

[0099] In one embodiment, the deblurring network may be an end-to-end network, such as a U-Net network. Figure 7 The network structure of , in which the specific settings of convolutional layers, transposed convolutional layers or other types of intermediate layers can be Figure 7 According to the difference between the deblurred image output by the deblurring network and the second label, the parameters of the deblurring network are updated, thereby realizing the training of the deblurring network. It should be understood that in this exemplary embodiment, both the blur perception network and the deblurring network can adopt a network with a U-Net structure, but the detailed structure of the network can be different, and the parameters of the two networks are different after training, so that the functions realized by the two networks are different.

[0100] In one embodiment, the blur degree data may be integrated into the deblurring network. Generally, the blur degree data may be processed and then input into the middle layer of the deblurring network to be fused with the image information. Figure 8The schematic structure diagram of the deblurring network and the feature perception network is shown. The feature perception network is used to process the blur degree data, and its output is connected to the middle layer of the deblurring network. Exemplarily, the sample image is input into the deblurring network, and after the convolution and pooling operations in the convolution layers 1 to 3, a feature image with a decreasing size is obtained, and then enters the convolution layer 4. The blur degree data of the sample image (usually in the form of a blur degree image, which can be the blur degree data output by the first label or the blur degree perception network) is input into the feature perception network, and first passes through the convolution layer 1'( Figure 8 The convolution layer 1′ includes four convolution operations, which are not particularly limited in the present disclosure) to obtain a multi-channel feature ψ representing the blur degree feature; and then input ψ into multiple groups of different convolution layers, such as Figure 8 The convolutional layer 2' and convolutional layer 3' ( Figure 8 It is shown that both the convolution layer 2' and the convolution layer 3' include two convolution operations, which is not particularly limited in the present disclosure); the feature image output by the convolution layer 2' is input into the fully connected layer 1 to obtain the sample modulation parameter α, and the feature image output by the convolution layer 3' is input into the fully connected layer 2 to obtain the sample modulation parameter β (here α and β are the modulation parameters corresponding to the sample image, so they are called sample modulation parameters). Modulation refers to the fusion of blur degree information with image information. In one embodiment, modulation can be an affine transformation of the image, and α and β can be different types of transformation operation parameters. For example, affine transformation generally includes operations such as rotation, translation, and scaling. For example, α can be a scaling parameter, and β can be a translation parameter. The fully connected layers 1 and 2 can be connected to any intermediate layer in the deblurring network, for example, in Figure 8 It is connected to convolution layer 4, which means that the feature image in convolution layer 4 is affine transformed as follows:

[0101]

[0102] Among them, F is the feature image, and the dimensions of α and β are the same as F. is the feature image after affine transformation, Indicates element-by-element multiplication. The feature image after affine transformation is then processed by transposed convolution layer 1 to transposed convolution layer 3, and a sample deblurred image is output (the deblurred image here is the deblurred image corresponding to the sample image, so it is called the sample deblurred image).

[0103] exist Figure 8 In this paper, the feature perception network is used to represent the blur degree data as modulation parameters, and the feature image of the sample image is modulated, thereby realizing the fusion of the image and blur degree information, which is beneficial to improving the deblurring quality of the deblurring network.

[0104] It should be understood that Figure 8The feature perception network in the invention adopts the structure of SFT (Spatial Feature Transformation) layer, which is only schematic. In other embodiments, feature perception networks with other structures may be used, for example, a feature perception network with U-Net structure may be used. After the blur degree image is input into the feature perception network, it is processed by multiple convolutional layers and transposed convolutional layers, one or more of which may be connected to the corresponding convolutional layers or transposed convolutional layers in the deblurring network, so that the feature image in the feature perception network is input into the deblurring network, and spliced ​​with the feature image in the deblurring network, etc., which can also realize the fusion of the two kinds of information, image and blur degree.

[0105] In the structure of the above-mentioned deblurring network and feature perception network, a ternary training array such as sample image-first label-second label can be used for training, wherein the sample image and the first label are respectively input into two channels of the deblurring network, and after processing, the sample deblurred image is output, and the parameters of the deblurring network are updated according to the difference between the sample deblurred image and the second label (such as the loss function value such as MSE), or the parameters of the deblurring network and the feature perception network are updated at the same time to achieve training.

[0106] Existing public blurred image datasets (such as GoPro datasets, etc.) usually calculate the average value of multiple consecutive clear image frames to obtain blurred images. Using the image blur degree determination method in this exemplary embodiment, blur degree data can be determined for blurred images in a public blurred image dataset, thereby expanding the public blurred image dataset and constructing a more comprehensive blurred image dataset, so as to facilitate its application in the training of the blur degree perception network and the deblurring network.

[0107] An exemplary embodiment of the present disclosure also provides an image deblurring method. Fig. 9 An exemplary process of the image deblurring method is shown, including the following steps S910 to S930:

[0108] Step S910, obtaining an image to be processed that is synthesized by weighted fusion of pixels at the same position in n consecutive frames of original images, where n is a positive integer not less than 2.

[0109] Among them, the image to be processed is an image that needs to be deblurred, which is an image synthesized by continuous n frames of original images. For example, in some dynamic shooting modes, the camera continuously shoots multiple frames of original images in a short time, and combines them into one image for output. The image may have motion blur and can be used as the image to be processed here.

[0110] For a specific implementation of synthesizing the image to be processed from n consecutive frames of original images, please refer to the above Figure 4 Part of the content.

[0111] Step S920 , according to the above-mentioned method for determining the blur degree of an image, taking the image to be processed as the target image, determining blur degree data of the image to be processed.

[0112] Exemplarily, the image to be processed is used as the target image, and n consecutive frames of original images for synthesizing the image to be processed are obtained, and the following is performed: Figure 3 The method for determining the blur degree of an image is used to obtain blur degree data of the image to be processed, including the blur degree value of each pixel in the image to be processed.

[0113] Step S930: Deblurring the image to be processed based on the blur degree data of the image to be processed, to obtain a deblurred image corresponding to the image to be processed.

[0114] The blur degree data of the image to be processed can provide auxiliary or reference information for the deblurring process of the image to be processed. For example, based on the blur degree data of the image to be processed, different blur kernels can be set for different regions in the image to be processed, and then the blur kernels are used to deblur the different regions to obtain a deblurred image. Compared with using one blur kernel, different regions can be deblurred in a targeted manner to improve the deblurring effect.

[0115] In one implementation, step S930 may include the following steps:

[0116] The feature perception network is used to process the blur degree data of the image to be processed to obtain the modulation parameters;

[0117] The image to be processed is deblurred according to the modulation parameters to obtain a deblurred image corresponding to the image to be processed.

[0118] The feature perception network is used to convert the blur degree data into a specific form of modulation parameters, so as to facilitate fusion with the image information of the image to be processed. Figure 8 As shown, it can of course be in other forms, for example, the feature perception network is a network with a U-Net structure, and the modulation parameter is a feature image of the blur degree data. The present disclosure does not limit this.

[0119] According to the modulation parameters, the blur degree information can be fused with the image information, making the information dimension of the image to be processed more comprehensive, which facilitates the realization of high-quality image deblurring.

[0120] In one embodiment, the deblurring network can be used to perform deblurring. Exemplarily, the deblurring of the image to be processed according to the modulation parameters to obtain the deblurred image corresponding to the image to be processed can include the following steps:

[0121] The image to be processed is input into the input layer of the deblurring network, the modulation parameters are input into the middle layer of the deblurring network, and the deblurred image corresponding to the image to be processed is output through the deblurring network.

[0122] Among them, the specific processing process of the deblurring network can refer to the above Figure 8 Part of the content. Figure 8 The content of the first part is the process of the deblurring network processing the sample image and its blur degree data, that is, the training process. In the deblurring process of the image to be processed, the processing process of the deblurring network is the same, the difference is that the network has been fully trained and can directly output a high-quality deblurred image.

[0123] The exemplary embodiments of the present disclosure further provide another image deblurring method. Fig.10 An exemplary process of the image deblurring method is shown, including the following steps S1010 to S1030:

[0124] Step S1010, obtaining an image to be processed.

[0125] The image to be processed is an image that needs to be deblurred, and may be an image obtained by any means, such as a currently captured image or an image selected by a user. Compared with step S910, step S1010 does not need to limit the image to be processed to an image synthesized from n consecutive frames of original images. For example, the image to be processed may be a frame of image captured separately, and blur may exist in the image to be processed due to hand shaking or other reasons during shooting.

[0126] Step S1020, using a blur perception network to process the image to be processed to obtain blur degree data of the image to be processed.

[0127] The blur perception network is trained using the blur image dataset constructed by the blur image dataset construction method, which can be Figure 7 The network shown. For example, by Figure 6 The fuzzy image dataset construction method is used to construct a fuzzy image dataset, and the fuzzy degree perception network is trained using the dataset. In step S1020, the image to be processed is input into the trained fuzzy degree perception network, and the fuzzy degree data of the image to be processed is output. The fuzzy degree data of the image to be processed may be a fuzzy degree image corresponding to the image to be processed.

[0128] It should be noted that, through the blur perception network, the feature association between the target image and its blur data can be learned, and this feature association is common to blur images in all scenes. After training, the blur perception network can apply the learned feature association to blur images in any scene, that is, it can be used to process blur images in any scene. For this reason, this exemplary embodiment does not specifically limit the image to be processed, especially there is no need to limit the image to be processed to be an image synthesized from n consecutive frames of original images.

[0129] Step S1030 , based on the blur degree data of the image to be processed, deblurring the image to be processed to obtain a deblurred image corresponding to the image to be processed.

[0130] The blur degree data of the image to be processed can provide auxiliary or reference information for the deblurring process of the image to be processed. For example, based on the blur degree data of the image to be processed, different blur kernels can be set for different regions in the image to be processed, and then the blur kernels are used to deblur the different regions to obtain a deblurred image. Compared with using one blur kernel, different regions can be deblurred in a targeted manner to improve the deblurring effect.

[0131] In one implementation, step S1030 may include the following steps:

[0132] The feature perception network is used to process the blur degree data of the image to be processed to obtain the modulation parameters;

[0133] The image to be processed is deblurred according to the modulation parameters to obtain a deblurred image corresponding to the image to be processed.

[0134] The feature perception network is used to convert the blur degree data into a specific form of modulation parameters, so as to facilitate fusion with the image information of the image to be processed. Figure 8 As shown, it can of course be in other forms, for example, the feature perception network is a network with a U-Net structure, and the modulation parameter is a feature image of the blur degree data. The present disclosure does not limit this.

[0135] According to the modulation parameters, the blur degree information can be fused with the image information, making the information dimension of the image to be processed more comprehensive, which facilitates the realization of high-quality image deblurring.

[0136] In one embodiment, the deblurring network can be used to perform deblurring. Exemplarily, the deblurring of the image to be processed according to the modulation parameters to obtain the deblurred image corresponding to the image to be processed can include the following steps:

[0137] The image to be processed is input into the input layer of the deblurring network, the modulation parameters are input into the middle layer of the deblurring network, and the deblurred image corresponding to the image to be processed is output through the deblurring network.

[0138] Among them, the specific processing process of the deblurring network can refer to the above Figure 8 Part of the content. Figure 8 The content of the first part is the process of the deblurring network processing the sample image and its blur degree data, that is, the training process. In the deblurring process of the image to be processed, the processing process of the deblurring network is the same, the difference is that the network has been fully trained and can directly output a high-quality deblurred image.

[0139] Fig.11 The schematic diagram shows the deblurring of the image to be processed by the blur perception network, feature perception network and deblurring network. The image to be processed is first input into the blur perception network, which outputs the blur image corresponding to the image to be processed; the blur image is then input into the feature perception network, which outputs the modulation parameters; the image to be processed is input into the deblurring network, and the modulation parameters are input into the middle layer of the deblurring network, which outputs the deblurred image corresponding to the image to be processed. Fig.11 In the process, the image information of the image to be processed and the pixel-level blur degree information of the image are integrated, thereby achieving targeted deblurring of different areas of the image to be processed and obtaining a high-quality deblurred image.

[0140] In the above network training process, the blur perception network is trained using sample images-first labels, and the deblurring network is trained using sample images-first labels-second labels, or the deblurring network and feature perception network are trained. The two parts of training are performed separately. Fig.11 It can be seen that there is a connection relationship between the input and output of the three networks, so the blur perception network, deblurring network and feature perception network can be trained together. The following is an example of the complete training process. Fig.12 As shown, the image deblurring method may further include the following steps S1210 to S1240:

[0141] Step S1210, obtaining a blurred image data set, including a sample image, a first label and a second label; the sample image is a target image, the first label is blur degree data of the target image, and the second label is an original image of n consecutive original images for synthesizing the target image;

[0142] Step S1220, pre-training a blur perception network using the sample image and the first label;

[0143] Step S1230, pre-training the deblurring network using the sample image, the first label, and the second label, or pre-training the deblurring network and the feature perception network;

[0144] Step S1240, input the sample image into the blurriness perception network and the deblurring network respectively, input the sample blurriness data output by the blurriness perception network into the feature perception network, and input the sample modulation parameters output by the feature perception network into the middle layer of the deblurring network, and fine-tune the parameters of the blurriness perception network, the deblurring network and the feature perception network according to the difference between the sample deblurred image output by the deblurring network and the second label.

[0145] The whole training process is divided into two stages: pre-training and parameter fine-tuning. In pre-training, the blur perception network is trained separately, the deblur network is trained separately, or the deblur network and the feature perception network are trained separately as a whole. The network structure trained in each part is relatively small, and it is easy to quickly adjust the parameters in the network to obtain preliminary training results. In the parameter fine-tuning stage, the three networks are regarded as a large network structure. Specifically, the sample image is input into the blur perception network and the deblur network, and the blur perception network outputs the sample blur data. After being processed by the feature perception network, the sample modulation parameters are output and input into the middle layer of the deblur network. Finally, the deblur network outputs the sample deblurred image as the result of the output of the whole network structure. According to the difference between the second label and the second label, such as the MSE loss function value, the parameters of the three networks are updated together, thereby achieving global full training. On the basis of pre-training, the parameters of the three networks are adjusted more finely, so as to obtain a network with higher accuracy, which is conducive to further improving the image deblurring quality.

[0146] The exemplary embodiment of the present disclosure also provides an image blur level determination device. Fig.13 As shown, the image blur level determination device 1300 may include:

[0147] The image acquisition module 1310 is configured to acquire n consecutive frames of original images and a target image synthesized by weighted fusion of pixels at the same position in the n consecutive frames of original images, where n is a positive integer not less than 2;

[0148] The pixel value acquisition module 1320 is configured to acquire the pixel value of the pixel point at the same position corresponding to each pixel point in the target image in each frame of the original image;

[0149] The blur degree data determination module 1330 is configured to determine the blur degree data of the target image according to the degree of deviation between the n pixel values ​​corresponding to each pixel point in the target image in the n frames of original images. The blur degree data of the target image includes the blur degree value of each pixel point in the target image.

[0150] In one implementation, the blur level data determination module 1330 is further configured to:

[0151] According to the blur degree data of the target image, a blur degree image corresponding to the target image is generated, and the pixel value of each pixel point in the blur degree image is the blur degree value of each pixel point in the target image.

[0152] In one implementation, the image acquisition module 1310 is configured to:

[0153] Get n consecutive frames of original images;

[0154] The pixels at the same position in n consecutive frames of original images are weightedly fused to synthesize the n consecutive frames of original images into a target image.

[0155] In one implementation, the image acquisition module 1310 is further configured to:

[0156] Before acquiring n consecutive frames of original images, the value of n is determined based on the expected global blur level value of the target image.

[0157] The exemplary embodiment of the present disclosure also provides a device for constructing a fuzzy image dataset. Fig.14 As shown, the blurred image data set construction device 1400 may include:

[0158] The data acquisition module 1410 is configured to acquire a target image and blur degree data of the target image determined according to the above-mentioned image blur degree determination method;

[0159] The data set construction module 1420 is configured to construct a fuzzy image data set using the target image as a sample image and the blurriness data of the target image as a first label; the fuzzy image data set is used to train the blurriness perception network, and the blurriness perception network is used to determine the blurriness data of the image input into the blurriness perception network.

[0160] In one embodiment, the data acquisition module 1410 is further configured to acquire one frame of original image among the continuous n frames of original images used to synthesize the target image;

[0161] The data set construction module 1420 is also configured to add the above-mentioned one frame of original image as a second label to the blurred image data set; the blurred image data set is also used to train the deblurring network, and the deblurring network is used to deblur the image input to the deblurring network.

[0162] The exemplary embodiment of the present disclosure also provides an image deblurring determination device. Fig.15 As shown, the image deblurring device 1500 may include:

[0163] The image acquisition module 1510 is configured to acquire an image to be processed that is synthesized by weighted fusion of pixels at the same position in n consecutive frames of original images, where n is a positive integer not less than 2;

[0164] The blur degree data determination module 1520 is configured to determine the blur degree data of the image to be processed according to the above-mentioned image blur degree determination method, taking the image to be processed as the target image;

[0165] The deblurring processing module 1530 is configured to perform deblurring processing on the image to be processed based on the blur degree data of the image to be processed, and obtain a deblurred image corresponding to the image to be processed.

[0166] The exemplary embodiment of the present disclosure also provides another image deblurring determination device. Fig.16 As shown, the image deblurring device 1600 may include:

[0167] The image acquisition module 1610 is configured to acquire an image to be processed;

[0168] The blur degree data determination module 1620 is configured to process the image to be processed using a blur degree perception network to obtain blur degree data of the image to be processed;

[0169] The deblurring processing module 1630 is configured to perform deblurring processing on the image to be processed based on the blurring degree data of the image to be processed, so as to obtain a deblurred image corresponding to the image to be processed;

[0170] Among them, the blur degree perception network is trained by using the blur image dataset constructed by the above blur image dataset construction method.

[0171] In one implementation, the deblurring module 1630 is configured to:

[0172] The feature perception network is used to process the blur degree data of the image to be processed to obtain the modulation parameters;

[0173] The image to be processed is deblurred according to the modulation parameters to obtain a deblurred image corresponding to the image to be processed.

[0174] In one implementation, the deblurring module 1630 is configured to:

[0175] The image to be processed is input into the input layer of the deblurring network, the modulation parameters are input into the middle layer of the deblurring network, and the deblurred image is output through the deblurring network.

[0176] In one implementation, the image deblurring apparatus 1600 may further include a network training module configured to:

[0177] Obtain a blurred image data set, the blurred image data set comprising a sample image, a first label and a second label, the sample image is a target image, the first label is blur degree data of the target image, and the second label is an original image of n consecutive original images for synthesizing the target image;

[0178] Pre-training a blur perception network using the sample image and the first label;

[0179] Pre-training a deblurring network using the sample image, the first label, and the second label, or pre-training the deblurring network and the feature perception network;

[0180] The sample images are respectively input into the blur perception network and the deblurring network, the sample blur degree data output by the blur perception network is input into the feature perception network, and the sample modulation parameters output by the feature perception network are input into the middle layer of the deblurring network. According to the difference between the sample deblurred image output by the deblurring network and the second label, the parameters of the blur perception network, the deblurring network and the feature perception network are fine-tuned.

[0181] The details of each part of the above device have been described in detail in the method part implementation method, so they will not be repeated here.

[0182] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium, which can be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to cause the electronic device to perform the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. In one embodiment, the program product can be implemented as a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0183] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0184] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0185] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0186] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0187] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0188] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or implementations combining hardware and software aspects, which may be collectively referred to herein as "circuit", "module" or "system". Those skilled in the art will readily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0189] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for determining the blur degree of an image, It is characterized in that include: Obtaining n consecutive frames of original images and a target image synthesized by weighted fusion of pixels at the same position in the n consecutive frames of original images, where n is a positive integer not less than 2; Obtaining the pixel value of the pixel point at the same position corresponding to each pixel point in the target image in each frame of the original image; According to the degree of deviation between the n pixel values ​​corresponding to each pixel point in the n frames of the original image, the blur degree data of the target image is determined, and the blur degree data of the target image includes the blur degree value of each pixel point in the target image; wherein the blur degree value of each pixel point and the degree of deviation between the n pixel values ​​corresponding to the pixel point satisfy a linear or nonlinear positive correlation.

2. The method according to claim 1, It is characterized in that The method further comprises: A blur degree image corresponding to the target image is generated according to the blur degree data of the target image, wherein the pixel value of each pixel in the blur degree image is the blur degree value of each pixel in the target image.

3. The method according to claim 1, It is characterized in that The step of acquiring n consecutive frames of original images and a target image synthesized by weighted fusion of pixels at the same position in the n consecutive frames of original images comprises: Get n consecutive frames of original images; Weighted fusion is performed on the pixels at the same position in the consecutive n frames of original images to synthesize the consecutive n frames of original images into one target image.

4. The method according to claim 3, It is characterized in that Before acquiring n consecutive frames of original images, the method further includes: The value of n is determined based on the expected global blur value of the target image.

5. A method for constructing a fuzzy image dataset, It is characterized in that include: Acquire a target image and blur degree data of the target image determined by the image blur degree determination method according to any one of claims 1 to 4; The target image is used as a sample image and the blur degree data of the target image is used as a first label to construct a blur image dataset; the blur image dataset is used to train a blur degree perception network, and the blur degree perception network is used to determine the blur degree data of an image input into the blur degree perception network.

6. The method according to claim 5, It is characterized in that The method further comprises: A frame of original image among the continuous n frames of original images used to synthesize the target image is obtained, and the frame of original image is added to the blurred image dataset as a second label; the blurred image dataset is also used to train a deblurring network, and the deblurring network is used to deblur the image input into the deblurring network.

7. An image deblurring method, It is characterized in that include: Obtain an image to be processed that is synthesized by weighted fusion of pixels at the same position in n consecutive frames of original images, where n is a positive integer not less than 2; According to the image blur degree determination method according to any one of claims 1 to 4, the blur degree data of the image to be processed is determined by taking the image to be processed as the target image; Based on the blur degree data of the image to be processed, the image to be processed is deblurred to obtain a deblurred image corresponding to the image to be processed.

8. An image deblurring method, It is characterized in that include: Get the image to be processed; Processing the image to be processed by using a blur perception network to obtain blur degree data of the image to be processed; Based on the blur degree data of the image to be processed, deblurring the image to be processed to obtain a deblurred image corresponding to the image to be processed; Wherein, the blur degree perception network is trained by using the blur image dataset constructed by the blur image dataset construction method described in claim 5 or 6.

9. The method according to claim 8, It is characterized in that The step of performing deblurring processing on the image to be processed based on the blur degree data of the image to be processed to obtain a deblurred image corresponding to the image to be processed includes: Processing the blur degree data of the image to be processed by using a feature perception network to obtain a modulation parameter; Deblurring the image to be processed according to the modulation parameters to obtain a deblurred image corresponding to the image to be processed.

10. The method according to claim 9, It is characterized in that The deblurring the image to be processed according to the modulation parameters to obtain a deblurred image corresponding to the image to be processed includes: The image to be processed is input into the input layer of the deblurring network, the modulation parameters are input into the middle layer of the deblurring network, and the deblurred image is output through the deblurring network.

11. The method according to claim 10, It is characterized in that The method further comprises: Acquire a blurred image data set, the blurred image data set comprising a sample image, a first label and a second label, the sample image is the target image, the first label is blur degree data of the target image, and the second label is one frame of original image among n consecutive frames of original images synthesizing the target image; Pre-training the blur perception network using the sample image and the first label; Pre-training the deblurring network using the sample image, the first label, and the second label, or pre-training the deblurring network and the feature perception network; The sample image is respectively input into the blur degree perception network and the deblurring network, the sample blur degree data output by the blur degree perception network is input into the feature perception network, and the sample modulation parameter output by the feature perception network is input into the middle layer of the deblurring network, and according to the difference between the sample deblurred image output by the deblurring network and the second label, the parameters of the blur degree perception network, the deblurring network and the feature perception network are fine-tuned.

12. A device for determining the degree of image blur, It is characterized in that include: An image acquisition module is configured to acquire n consecutive frames of original images and a target image synthesized by weighted fusion of pixels at the same position in the n consecutive frames of original images, where n is a positive integer not less than 2; A pixel value acquisition module is configured to acquire the pixel value of each pixel in the target image corresponding to a pixel at the same position in each frame of the original image; The blur degree data determination module is configured to determine the blur degree data of the target image according to the degree of deviation between the n pixel values ​​corresponding to each pixel in the n frames of original images, wherein the blur degree data of the target image includes the blur degree value of each pixel in the target image; wherein the blur degree value of each pixel satisfies a linear or nonlinear positive correlation with the degree of deviation between the n pixel values ​​corresponding to the pixel.

13. A device for constructing a fuzzy image dataset, It is characterized in that include: A data acquisition module, configured to acquire a target image and blur degree data of the target image determined by the image blur degree determination method according to any one of claims 1 to 4; The data set construction module is configured to use the target image as a sample image and the blur degree data of the target image as a first label to construct a blur image data set; the blur image data set is used to train a blur degree perception network, and the blur degree perception network is used to determine the blur degree data of the image input into the blur degree perception network.

14. An image deblurring device, It is characterized in that include: An image acquisition module is configured to acquire an image to be processed that is synthesized by weighted fusion of pixels at the same position in n consecutive frames of original images, where n is a positive integer not less than 2; A blur degree data determination module, configured to determine blur degree data of the image to be processed according to the image blur degree determination method according to any one of claims 1 to 4, taking the image to be processed as a target image; The deblurring processing module is configured to perform deblurring processing on the image to be processed based on the blurring degree data of the image to be processed, so as to obtain a deblurred image corresponding to the image to be processed.

15. An image deblurring device, It is characterized in that include: An image acquisition module is configured to acquire an image to be processed; A blur degree data determination module is configured to process the image to be processed using a blur degree perception network to obtain blur degree data of the image to be processed; a deblurring processing module, configured to perform deblurring processing on the image to be processed based on the blur degree data of the image to be processed, so as to obtain a deblurred image corresponding to the image to be processed; Wherein, the blur degree perception network is trained by using the blur image dataset constructed by the blur image dataset construction method described in claim 5 or 6.

16. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

17. An electronic device, It is characterized in that include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 11 by executing the executable instructions.