Blurred image processing method and device, equipment, storage medium and program product

Abstract

The invention relates to a blurred image processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring a target blurred image, and acquiring a jitter parameter of the target blurred image; the jitter parameter is used for representing the blurring degree of the target blurred image; inputting the target blurred image and the jitter parameter into a target deblurring network model to obtain a target clear image output by the target deblurring network model; wherein the jitter parameter is related to the deblurring degree of the target deblurring network model on the target blurred image. By adopting the method, the flexibility of image deblurring processing can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a blurred image processing method, device, equipment, storage medium and program product. Background Art

[0002] When a user uses an electronic device to shoot, the image or video may be blurred due to hand shaking, etc. In order to remove or reduce the image blur caused by the movement of the shooting device, the electronic device is usually required to deblur the blurred image to obtain a clearer image.

[0003] However, the flexibility of image deblurring processing in the related art is low. Summary of the invention

[0004] Based on this, it is necessary to provide a blurred image processing method, device, equipment, storage medium and program product that can improve the flexibility of image deblurring processing in order to address the above technical problems.

[0005] In a first aspect, the present application provides a blurred image processing method. The method comprises:

[0006] Obtaining a target blurred image and obtaining a jitter parameter of the target blurred image; the jitter parameter is used to characterize the blur degree of the target blurred image;

[0007] Inputting the target blurred image and the jitter parameters into the target deblurring network model, and obtaining the target clear image output by the target deblurring network model;

[0008] Among them, the jitter parameter is related to the degree to which the target deblurring network model deblurs the target blurred image.

[0009] In a second aspect, the present application also provides a blurred image processing device. The device comprises:

[0010] A parameter acquisition module is used to acquire a target blurred image and obtain a jitter parameter of the target blurred image; the jitter parameter is used to characterize the blur degree of the target blurred image;

[0011] A blur processing module is used to input the target blur image and the jitter parameter into the target deblurring network model to obtain the target clear image output by the target deblurring network model;

[0012] Among them, the jitter parameter is related to the degree to which the target deblurring network model deblurs the target blurred image.

[0013] In a third aspect, the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps described in the first aspect when executing the computer program.

[0014] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect above.

[0015] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0016] The blurred image processing method, device, equipment, storage medium and program product described above obtain a target blurred image and a jitter parameter of the target blurred image; the jitter parameter is used to characterize the blur degree of the target blurred image; the target blurred image and the jitter parameter are input into a target deblurring network model to obtain a target clear image output by the target deblurring network model; wherein the jitter parameter is related to the degree to which the target deblurring network model deblurs the target blurred image. In this way, the target deblurring network can be used to deblur the target blurred image to a corresponding degree in combination with the blur degree characterized by the jitter parameter to obtain a target clear image, thereby improving the flexibility of image deblurring processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 is a schematic flow chart of a blurred image processing method in one embodiment;

[0019] Figure 2 A schematic diagram of a process for determining jitter parameters in one embodiment;

[0020] Figure 3 A schematic diagram of data flow of a fuzzy image processing method in one embodiment;

[0021] Figure 4 A schematic diagram of a process for obtaining an image training set in one embodiment;

[0022] Figure 5 A schematic diagram of a process for generating a reference blurred image in one embodiment;

[0023] Figure 6 A schematic diagram of a process for determining a convolution kernel in one embodiment;

[0024] Figure 7 A schematic diagram of a process for determining a motion blur kernel in one embodiment;

[0025] Figure 8 A schematic diagram of a preset coordinate system in one embodiment;

[0026] Fig. 9 A schematic diagram of a process of determining a motion blur kernel according to a mapped motion trajectory in one embodiment;

[0027] Fig.10 A schematic diagram of a reference trajectory range in an embodiment;

[0028] Fig.11 A schematic diagram of a process of determining a convolution kernel according to a motion blur kernel in one embodiment;

[0029] Fig.12 is a schematic diagram of a convolution image in one embodiment;

[0030] Fig.13 A schematic diagram of a process of outputting a clear target image by a target deblurring network model in one embodiment;

[0031] Fig.14 A schematic diagram of a process of outputting a clear target image by another target deblurring network model in one embodiment;

[0032] Fig.15 Schematic diagram of the process flow of the feature extraction module in one embodiment;

[0033] Fig.16 A schematic diagram of a feature extraction process in one embodiment;

[0034] Fig.17 A schematic diagram of the structure of a target defuzzification network model in one embodiment;

[0035] Fig.18 is a structural block diagram of a blurred image processing device in one embodiment;

[0036] Fig.19 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0038] When a user uses an electronic device to shoot, the image or video may be blurred due to hand shaking, etc. In order to remove or reduce the image blur caused by the movement of the shooting device, the electronic device is usually required to deblur the blurred image to obtain a clearer image.

[0039] At present, there are mainly artificial intelligence-based deblurring methods for deblurring single-frame images. In this method, the neural network model can directly learn the mapping from motion blur map to clear map based on an end-to-end learning strategy. Therefore, the model needs to implicitly detect the blur degree and blur direction of the input image during the deblurring process, and remove blur based on this implicit information. The need to implicitly detect the blur degree and blur direction increases the difficulty of network motion deblurring. When this method is used in video blur processing tasks, it can only improve the target frame to clarity, but cannot make the target frame consistent with the clarity of the previous and next frames, resulting in the deblurred video still having uneven video clarity when playing. Therefore, the flexibility of image deblurring processing in related technologies is relatively low.

[0040] In view of this, the embodiment of the present application provides a blurred image processing method, which can obtain a target blurred image and a jitter parameter of the target blurred image used to characterize the blur degree of the target blurred image; input the target blurred image and the jitter parameter into a target deblurring network model to obtain a target clear image output by the target deblurring network model; wherein the jitter parameter is related to the degree to which the target deblurring network model deblurs the target blurred image. In this way, the target deblurring network can be used to perform a corresponding degree of deblurring on the target blurred image in combination with the blur degree characterized by the jitter parameter to obtain a target clear image, thereby improving the flexibility of image deblurring processing.

[0041] In an exemplary embodiment, Figure 1 As shown, a blurred image processing method is provided, and the method is applied to an electronic device as an example for explanation, wherein the electronic device can be a device with an image processing function, and on this basis, the electronic device can also be a device with an image shooting function. Optionally, the electronic device can be a laptop, a smart phone, a tablet computer, an Internet of Things device, and a portable wearable device, etc., wherein the Internet of Things device can be a smart speaker, a smart TV, a smart car device, a projection device, etc., and the portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc., and the head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. Figure 1 As shown, the blurred image processing method may include the following steps:

[0042] Step 101: Obtain a target blurred image and obtain a jitter parameter of the target blurred image.

[0043] The target blurred image is the image to be deblurred.

[0044] In an optional embodiment of the present application, the target blurred image may be a first image pre-stored in the electronic device or a second image currently received or captured by the electronic device; optionally, the first image may be an image previously captured by the electronic device, or the first image may be an image stored in the electronic device.

[0045] In addition, the electronic device can also obtain the jitter parameter of the target blurred image. The jitter parameter is used to characterize the blur degree of the target blurred image. For example, when a user holds the electronic device and shoots the target blurred image, the target blurred image is blurred due to hand jitter. The motion data describing the hand jitter process is the jitter parameter.

[0046] The electronic device generates jitter parameters during the image capture process. For example, if the target blurred image is the first image, the electronic device generates jitter parameters when capturing the first image and stores the first image and the jitter parameters in correspondence, so that the jitter parameters of the first image can be acquired synchronously when the first image is acquired as the target blurred image. If the target blurred image is the second image currently being received, the jitter parameters can be received while the second image is being received.

[0047] Optionally, if no jitter parameter is received when receiving the second image, the electronic device may generate a jitter parameter for the second image. For example, the electronic device may identify the blurriness of the second image and generate a jitter parameter for it, or the electronic device uses a default jitter parameter as the jitter parameter of the second image.

[0048] In another optional embodiment of the present application, the target blurred image may be a first video frame in a first video pre-stored in the electronic device, or a second video frame in a second video currently received or being shot by the electronic device. Optionally, the first video frame may be any frame in the first video; the second video frame may be a newly acquired frame in the second video or a frame acquired historically.

[0049] Optionally, if no jitter parameters are received when receiving the second video, the electronic device may generate jitter parameters for each second video frame of the second video. For example, the target second video frame in the second video is used as a target blurred image, and the electronic device may identify the blurriness of the target second video frame and generate jitter parameters for it, or the electronic device uses the default jitter parameters as the jitter parameters of the target second video frame. Alternatively, if the jitter parameters of the target second video frame are missing, but the jitter parameters of the video frames before and after the target second video frame exist, the jitter parameters of the target second video frame may be determined based on the jitter parameters of the video frames before and after the target second video frame.

[0050] It can be seen that any image or video frame that needs to be deblurred can be used as a target blurred image. In addition, if there is a jitter parameter of the target blurred image, the target blurred image and its corresponding jitter parameter are obtained; if there is no jitter parameter of the target blurred image, the target blurred image is obtained and jitter parameters are generated for it.

[0051] Step 102, input the target blurred image and the jitter parameters into the target deblurring network model to obtain the target clear image output by the target deblurring network model.

[0052] A target defuzzification network model is pre-deployed in the electronic device.

[0053] The input of the target deblurring network model includes the blurred target image and jitter parameters, and the output is a clear target image.

[0054] In an optional embodiment of the present application, in addition to the target blurred image, other candidate blurred images related to the target blurred image can also be input into the target deblurring network model as reference information to improve the dimension of the deblurring processing of the target blurred image, thereby improving the accuracy of the deblurring processing.

[0055] Among them, the jitter parameter is related to the degree to which the target deblurring network model deblurs the target blurred image.

[0056] As mentioned above, the jitter parameter represents the blur degree of the target blurred image, and the deblurring degree of the target blurred image can be determined according to the blur degree, thereby instructing the target deblurring network model to perform deblurring processing on the target blurred image with a corresponding deblurring degree, rather than performing deblurring processing to a fixed degree.

[0057] In one possible scenario, the target blurred image is a target video frame. For example, a video contains multiple video frames, and the blur degree of each video frame is represented by a number. The video frame is described as: 3333383333, and each number represents the blur degree of each frame. 0 is a clear image, and the larger the number, the higher the blur degree. In order to ensure smooth viewing of the video, the video frames with a blur degree of 8 need to be deblurred to 3, not 0. Therefore, based on the jitter parameter as a guide for the degree and direction of image deblurring, the target deblurring network model deblurs the video frames in the video to a blur degree of 3, so that the clarity of the processed video is improved and the clarity is smooth. Of course, all video frames can also be deblurred to a blur degree of 0, that is, they are all at the highest clarity.

[0058] In another possible scenario, the target blurred image is a single image. Optionally, the single image can be processed to the highest definition or the target definition, which is not specifically limited here.

[0059] This allows for flexible deblurring of both video frames and single images.

[0060] The above-mentioned blurred image processing method can obtain a target blurred image and obtain a jitter parameter of the target blurred image; the jitter parameter is used to characterize the blur degree of the target blurred image; the target blurred image and the jitter parameter are input into the target deblurring network model to obtain a target clear image output by the target deblurring network model; wherein the jitter parameter is related to the degree to which the target deblurring network model deblurs the target blurred image. In this way, the target deblurring network can be used to perform a corresponding degree of deblurring on the target blurred image in combination with the blur degree characterized by the jitter parameter to obtain a target clear image, thereby improving the flexibility of image deblurring processing.

[0061] Please refer to Figure 2 In an optional embodiment of the present application, an optional technical process for determining jitter parameters is provided, such as Figure 2 As shown, the technical process includes the following steps:

[0062] Step 201: Acquire first jitter motion data when a shooting device of an electronic device shoots a blurred image of a target.

[0063] The photographing device is, for example, a camera.

[0064] The electronic device is also provided with a motion detection component. When the electronic device calls the shooting device to shoot a blurred image of a target, the motion detection component is called to generate first shaking motion data.

[0065] Exemplarily, the motion detection component is an IMU (Inertial Measurement Unit) component, which includes components such as a gyroscope and an accelerometer, and IMU data is obtained based on the IMU component as the first jitter motion data. The IMU data is, for example, the speed, acceleration, and direction of movement of the electronic device when shooting a blurred image of a target, which is not fully exemplified here.

[0066] Step 202: Determine a jitter parameter according to the first jitter motion data.

[0067] In a possible implementation manner, the first jitter motion data is substituted into a preset calculation formula to obtain a jitter parameter.

[0068] In another possible implementation, the first jitter motion data is input into a preset filtering model to perform anti-shake filtering processing to obtain jitter parameters.

[0069] In an optional embodiment of the present application, determining a jitter parameter according to the first jitter motion data includes: performing anti-shake filtering on the first jitter motion data to obtain an anti-shake compensation amount; and using the anti-shake compensation amount as the jitter parameter.

[0070] Exemplarily, the above-mentioned preset filtering model is implemented based on an OIS (Optical image stabilization) filtering algorithm. The OIS filtering algorithm is used to calculate the amount of motor movement required to drive the lens in the process of shooting a blurred target image, also known as the anti-shake compensation amount Target, and the anti-shake compensation amount is used as the jitter parameter.

[0071] In an optional embodiment of the present application, an OIS module is deployed in the electronic device, and the IMU component and the OIS algorithm are both deployed in the OIS module.

[0072] Based on this knowledge, the data flow of the fuzzy image processing method is as follows: Figure 3 As shown, an OIS module is deployed in the electronic device, and the IMU component obtains the first jitter motion data when the shooting device shoots the target blurred image, inputs the first jitter motion data into the OIS algorithm for anti-shake filtering, obtains the anti-shake compensation amount Target as the jitter parameter, and compares Target and the target blurred image I B Input the target deblurring network model deployed in the electronic device, and obtain the target clear image I output by the target deblurring network model E .

[0073] Compared with the current motion deblurring method, B As an input, the input of the target defuzzification network model in the embodiment of the present application is I B And shoot I BOIS calculates the Target value based on the first jitter motion data. In this way, the motion data obtained by the motion detection component in the electronic device is used as a guide for the degree and direction of image blur, which reduces the deblurring difficulty of the target deblurring network model and enhances the blur restoration effect of the target deblurring network model.

[0074] Since this method is compatible with the OIS module of the electronic device, it can report the position offset value caused by the jitter motion while deblurring, so it can be used in scenarios such as smooth processing of video clarity and de-motion blurring of captured images by electronic devices. Moreover, based on this, for the processing scenario of smooth video clarity, a certain degree of deblurring can be performed according to the input jitter parameters, so that the blur degree of the target frame is consistent with the previous and next frames, thereby improving the flexibility of video frame deblurring processing.

[0075] The generation of the target deblurring network model is described below.

[0076] In an exemplary embodiment, the method further includes: acquiring an image training set; and training an initial deblurring network model according to the image training set to obtain a target deblurring network model.

[0077] In a possible implementation manner, a plurality of reference blurred images and reference clear images corresponding to the plurality of reference blurred images are obtained to form an image training set.

[0078] In another possible implementation, a plurality of reference clear images are acquired, each reference clear image is blurred to obtain a reference blurred image corresponding to each reference clear image, and an image training set is formed based on each reference clear image and the reference blurred image.

[0079] In an optional embodiment of the present application, an initial deblurring network model is pre-constructed, and each reference blurred image is deblurred using the initial deblurring network model to obtain a first clear image corresponding to each reference blurred image, and the first clear image corresponding to each reference blurred image is compared with the corresponding reference clear image, and the initial deblurring network model is continuously adjusted according to the comparison result until the difference between the first clear image output by the initial deblurring network model and the corresponding reference clear image is less than a preset difference threshold, and the initial deblurring network model at this time is used as the target deblurring network model.

[0080] Please refer to Figure 4 In an optional embodiment of the present application, an optional technical process for obtaining an image training set is provided, and the technical process includes the following steps:

[0081] Step 401, obtaining a plurality of reference clear images.

[0082] In an optional embodiment of the present application, the reference clear image may be a clear image taken by the electronic device without shaking. Alternatively, it may be a clear image pre-stored in the electronic device after deblurring, or it may be a clear image from other sources received by the electronic device. As long as the image has a clarity greater than a preset clarity, the source of the reference clear image is not specifically limited.

[0083] Step 402: Generate a reference blurred image corresponding to each reference clear image according to each reference clear image.

[0084] In an optional embodiment of the present application, the jitter parameter is taken into consideration when deblurring the target blurred image. Correspondingly, the jitter motion data is also incorporated into the reference blurred image during the model training process, so that the target deblurring network model trained based on the reference blurred image can flexibly perform deblurring processing in combination with the deblurring degree indicated by the jitter parameter.

[0085] Therefore, for each reference clear image, the second shaking motion data corresponding to the reference clear image is determined, and blur processing is performed on the reference clear image based on the second shaking motion data to obtain a corresponding reference blurred image.

[0086] Step 403: construct an image training set according to each reference clear image and each reference blurred image.

[0087] In an embodiment of the present application, the initial deblurring network model can be supervised trained based on the constructed image training set including reference clear images and reference blurred images, so that the model can gradually realize the clear image prediction process "from coarse to fine", thereby improving the model training efficiency and model prediction accuracy.

[0088] Please refer to Figure 5 In an optional embodiment of the present application, an optional technical process for generating a reference blurred image is provided, and the technical process may include the following steps:

[0089] Step 501: for each reference clear image, determine the convolution kernel corresponding to the reference clear image.

[0090] Step 502 , performing convolution processing on the reference clear image according to the convolution kernel to obtain a reference blurred image corresponding to the reference clear image.

[0091] That is, it is necessary to generate convolution kernels corresponding to each reference clear image.

[0092] In an optional embodiment of the present application, a motion blur degradation model is established, and the motion blur degradation model can be expressed as: . is the reference blurred image, is the reference clear image, k is the convolution kernel, is the convolution operation.

[0093] Exemplarily, the convolution kernel is a convolution image, and the convolution image is convolved with the reference clear image to obtain a reference blurred image corresponding to the reference clear image. Optionally, the frame size of the convolution image is the same as the frame size of the reference clear image.

[0094] The convolution kernel may be determined based on a motion blur kernel of a reference clear image, and the motion blur kernel may be generated in combination with the motion of the electronic device during the exposure time of capturing the reference clear image. The motion blur kernel is a mathematical model used in the field of image processing and computer vision to describe the blur effect caused by the movement of a camera or an object during exposure.

[0095] In an embodiment of the present application, a corresponding convolution kernel is generated for each reference clear image, and the reference clear image is independently blurred based on the corresponding convolution kernel to obtain multiple reference blurred images with different blur degrees, thereby effectively enriching the image training set.

[0096] The process of determining the motion blur kernel and the convolution kernel is described below.

[0097] Please refer to Figure 6 In an optional embodiment of the present application, an optional technical process for determining a convolution kernel is provided, such as Figure 6 As shown, the technical process includes the following steps:

[0098] Step 601: Acquire second jitter motion data corresponding to a reference clear image.

[0099] In a possible implementation manner, the second jitter motion data corresponding to the reference clear image is generated based on a preset motion data generation algorithm.

[0100] In another possible implementation, when the electronic device captures the reference clear image, a motion detection component in the electronic device generates the second jitter motion data.

[0101] In an optional embodiment of the present application, the second jitter motion data includes a plurality of moving positions of the electronic device within an exposure time when the electronic device captures a reference clear image.

[0102] Exemplarily, the IMU component in the electronic device includes a gyroscope, and the gyroscope generates gyroscope data during the exposure time of shooting a reference clear image. The gyroscope data is integrated during the exposure time to obtain the movement of the electronic device during the exposure time; and the OIS module compensates for the movement of the electronic device during the shooting process, and the compensation amount is fed back through the Hall sensor, so the actual compensation amount of the motor can be obtained from the Hall sensor in the electronic device. The integral of the gyroscope data during the exposure time is subtracted from the compensation amount fed back by the Hall sensor to obtain the moving position of the electronic device during the shaking process.

[0103] In a possible implementation, the data can be integrated at different exposure times, and the motor compensation at the exposure time can be subtracted from the integrated data to obtain the moving position at the exposure time. In this way, multiple moving positions can be obtained within the exposure time, and the multiple moving positions are used as the second jitter motion data. It can be seen that the second jitter motion data describes the moving trajectory of the electronic device during the exposure time of capturing the reference clear image.

[0104] In another possible implementation, the gyroscope data is integrated within the complete exposure time, and the final motor compensation amount is subtracted from the integrated data to obtain the final moving position of the electronic device within the exposure time.

[0105] In an optional embodiment of the present application, each moving position may be a position in a U / V coordinate system, wherein U is the horizontal direction of the image and V is the vertical direction of the image. That is, the second jitter motion data describes a plurality of (u, v) positions of the electronic device moving linearly along the u / v direction during the exposure time of capturing the reference clear image.

[0106] Step 602: Determine a motion blur kernel corresponding to a reference clear image according to the second jitter motion data.

[0107] In an optional embodiment of the present application, a motion trajectory corresponding to a reference clear image is determined based on the second jitter motion data, and a motion blur kernel corresponding to the reference clear image is determined based on the motion trajectory.

[0108] Please refer to Figure 7 In an optional embodiment of the present application, an optional technical process for determining a motion blur kernel is provided, such as Figure 7 As shown, the technical process includes the following steps:

[0109] Step 701, mapping each moving position to a preset coordinate system.

[0110] Optionally, the preset coordinate system may be a standard coordinate system. Alternatively, it may be another coordinate system. For ease of understanding, the following description will be given by taking the preset coordinate system as a standard coordinate system as an example.

[0111] Step 702: Determine the mapping motion trajectory formed by each moving position in the preset coordinate system according to the mapping result.

[0112] In an optional embodiment of the present application, a plurality of block coordinates of preset areas are set in a preset coordinate system. In other words, the preset coordinate system is discontinuous, and adjacent coordinate points differ by one pixel value.

[0113] By mapping each moving position to a preset coordinate system, the candidate block coordinates that can construct the motion blur kernel are screened out according to the mapped motion trajectory. Figure 8 A schematic diagram of a preset coordinate system is exemplarily shown.

[0114] Optionally, the target slope u / v is determined according to the center of the circle of the standard coordinate system and the mapping position of the last moving position in the standard coordinate system, and the target slope is used as the mapping motion trajectory.

[0115] Optionally, a target slope u / v is determined according to the mapping position of each moving position in the standard coordinate system, and the target slope is used as the mapping motion trajectory. Exemplarily, the determined target slope may be a straight line.

[0116] Step 703: Determine the motion blur kernel corresponding to the reference clear image according to the mapped motion trajectory.

[0117] In a possible implementation, the block coordinates through which the motion trajectory is mapped are determined, and the motion blur kernel is determined according to the determined coordinate positions of each block coordinate.

[0118] For another possible implementation, see Fig. 9 , provides an optional technical process for determining a motion blur kernel based on a mapped motion trajectory, the technical process comprising the following steps:

[0119] Step 901 , traverse each block coordinate to determine whether the reference trajectory range planned with the block coordinate as the reference position covers the mapped motion trajectory; if so, use the block coordinate as a candidate block coordinate.

[0120] In an optional embodiment of the present application, for the currently traversed target block coordinates, the point coordinates of any position point in the target block coordinates are used as reference positions to plan the reference trajectory range corresponding to the target block coordinates, and then determine whether the reference trajectory range covers the mapped motion trajectory. Alternatively, the point coordinates of the central position point of the target block coordinates are used as reference positions to plan the reference trajectory range corresponding to the target block coordinates, and then determine whether the reference trajectory range covers the mapped motion trajectory.

[0121] For example, the reference position is the coordinate (i, j) in the standard coordinate system, and the planned reference trajectory range is [(jR) / (i+R), (j+R) / (iR)]. R can be a predetermined value. For example, if R is 0.5, the reference trajectory range is [(j-0.5) / (i+0.5), (j+0.5) / (i-0.5)].

[0122] If the reference trajectory range planned with the target block coordinates as the reference position covers the mapped motion trajectory, the target block coordinates are determined as candidate block coordinates.

[0123] For example, Fig.10 A reference trajectory range schematic diagram is shown. It can be understood that Fig.10 The reference trajectory range planned by the reference position (i, j) of the block coordinate currently traversed in does not cover the target slope, so the block coordinate is not used as a candidate block coordinate; in addition, Fig.10 A total of 7 candidate block coordinates are determined, including the origin block coordinates.

[0124] Step 902, determining the motion blur kernel according to the coordinate position of each candidate block coordinate.

[0125] In an optional embodiment of the present application, the coordinate positions of each candidate block coordinate together constitute a motion blur kernel. Exemplarily, {x[(0.5)~(1.5)], y[(0.5)~(1.5)]} is the coordinate position of a block coordinate.

[0126] Step 603: Determine a convolution kernel corresponding to the reference clear image according to the motion blur kernel corresponding to the reference clear image.

[0127] Please refer to Fig.11 In an optional embodiment of the present application, an optional technical process for determining a convolution kernel according to a motion blur kernel is provided, and the technical process includes the following steps:

[0128] Step 1101 , normalizing the coordinate position of each candidate block coordinate to obtain a first pixel block whose pixel values ​​are a first value.

[0129] That is, the coordinate position of each candidate block coordinate becomes each first pixel block, and the pixel value of each first pixel block is the first value.

[0130] In an optional embodiment of the present application, the sum of the pixel values ​​of each first pixel block is 1, that is, the sum of the first values ​​is 1.

[0131] Step 1102: Generate a convolution image including each first pixel block and a plurality of second pixel blocks according to each first pixel block.

[0132] That is, a plurality of first pixel blocks and a plurality of second pixel blocks are determined to jointly constitute a convolution image.

[0133] The pixel value of the second pixel block is the second value.

[0134] In an optional embodiment of the present application, the second value may be 0.

[0135] Step 1103, using the convolution image as the convolution kernel.

[0136] For example, Fig.12 A schematic diagram of a convolution image is shown, in which the motion blur kernel has a size of u = 40 and v = 20. Through this process, the motion process of the electronic device can be mapped into the shape of the motion blur kernel to generate a reference blurred image corresponding to the reference clear image based on this.

[0137] From the above description, we can know that the generation of convolution kernel is for application level, which is a process of mapping motion blur kernel to convolution kernel, so as to directly use the convolution kernel to convolve on the reference clear image to obtain the corresponding reference blur map. Based on the motion blur degradation model, a reference clear image-reference blur image pair can be generated: Bt , I Et}, which is the data used for network training. In addition, if I in the motion blur degradation model Et For non-full clarity, the network can also learn the mapping from the blur map to the target blur level image.

[0138] In an embodiment of the present application, since a reference blurred image is generated based on a convolution kernel and an initial deblurring network model is trained based on the reference blurred image, the model only needs to learn deblurring based on motion data, and has a low correlation with blurred scenes in a training data set, thereby reducing the model's dependence on the training data set.

[0139] In the process of training the initial deblurring network model according to the image training set, the process of the initial deblurring network model processing the reference blurred image in the image training set is the same as the process of the target deblurring network model processing the target blurred image below, and please refer to the description below.

[0140] The processing process of the target deblurring network model is described below.

[0141] Please refer to Fig.13 In an optional embodiment of the present application, an optional technical process for a target deblurring network model to output a target clear image is provided, and the technical process includes the following steps:

[0142] Step 1301 , down-sampling the target blurred image to obtain a plurality of candidate blurred images of different scales.

[0143] In an optional embodiment of the present application, the target blurred image is subjected to multiple different downsampling processes to obtain multiple candidate blurred images of different scales.

[0144] Taking two candidate blurred images as an example, illustratively, the target blurred image is of the first scale; the target blurred image is subjected to a first downsampling process to obtain a candidate blurred image of the second scale; and the target blurred image is subjected to a second downsampling process to obtain a candidate blurred image of the third scale, thereby obtaining multiple blurred images of different scales. Optionally, the first downsampling process is a 2-fold downsampling process, and the second downsampling process is a 3-fold downsampling process, then, correspondingly, the first scale is larger than the second scale, and the second scale is larger than the third scale.

[0145] It is understandable that the number of candidate blurred images can be determined according to training requirements, and is not specifically limited here.

[0146] Step 1302, input the target blurred image, multiple candidate blurred images and jitter parameters into the target deblurring network model to obtain the target clear image output by the target deblurring network model.

[0147] Since the scales of the target blurred image and multiple candidate blurred images are different, deblurring can be performed step by step based on blurred images of each scale. For example, the details of the blurred image can be gradually restored from small scale to large scale, realizing a "rough to fine" image restoration process, thereby ensuring that the target deblurring network model performs detailed deblurring on both the details and the whole of the target blurred image.

[0148] For ease of understanding, the following uses the first scale target blurred image I B1 , the candidate blurred image I of the second scale B2 , the candidate blurred image I of the third scale B3 The input target defuzzification network model is taken as an example for explanation.

[0149] In an exemplary embodiment, the target deblurring network model includes a feature extraction module, an encoder, and a decoder.

[0150] In an optional embodiment of the present application, a feature extraction module is used to extract features of blurred images of different scales according to jitter parameters to obtain feature information corresponding to blurred images of different scales, an encoder is used to encode each feature information, and a decoder is used to decode the encoding result to obtain a clear target image.

[0151] Please refer to Fig.14 In an optional embodiment of the present application, another optional technical process for outputting a clear target image by a target deblurring network model is provided, and the technical process includes the following steps:

[0152] Step 1401, taking each candidate blurred image and the target blurred image as input images, inputting the jitter parameters and each input image into a feature extraction module in groups, and obtaining first feature information corresponding to each input image output by the feature extraction module.

[0153] In one possible implementation, the target deblurring network model includes a single feature extraction module, which constructs a set of data with jitter parameters and each input image, and inputs each set of data into the feature extraction module in the order of the scale of the input image or in a random order, to obtain the first feature information corresponding to the input image output by the feature extraction module.

[0154] In another possible implementation, the target deblurring network model includes multiple feature extraction modules, wherein the number of feature extraction modules may be the same as the number of input images, so that the jitter parameters and each input image constitute a set of data, each set of data is input into a feature extraction module, and each feature extraction module outputs a first feature information corresponding to an input image.

[0155] Step 1402: Input each piece of first feature information into an encoder, so that the encoder performs feature extraction processing on each piece of first feature information to obtain a plurality of pieces of second feature information output by the encoder.

[0156] In an optional embodiment of the present application, the encoder includes a plurality of first sub-feature processing modules in cascade. Exemplarily, inputting each first feature information into the encoder includes: inputting each first feature information into first sub-feature processing modules at different levels. The smaller the scale of the input image corresponding to the first feature information, the later the level of the first sub-feature processing module that processes the first feature information.

[0157] Optionally, the number of first sub-feature processing modules is the same as the number of first feature information.

[0158] In this way, feature information of different scales can be processed step by step to gradually improve the image clarity.

[0159] Step 1403: input each second feature information into a decoder to obtain a target clear image output by the decoder.

[0160] In an optional embodiment of the present application, the decoder includes a plurality of second sub-feature processing modules in cascade. Exemplarily, inputting each second feature information into the decoder includes: inputting each second feature information into a second sub-feature processing module at a different level. Optionally, the later the level of the first sub-feature processing module that outputs the second feature information is, the earlier the level of the second sub-feature processing module that processes the second feature information is.

[0161] Optionally, the number of the second sub-feature processing modules is the same as the number of the second feature information.

[0162] In an optional embodiment of the present application, the feature extraction module is implemented based on the modulated deformable convolution module DCNv2, so that flexible feature sampling of the input image can be performed.

[0163] Please refer to Fig.15 In an optional embodiment of the present application, an optional technical process of the feature extraction module processing process is provided, and the technical process includes the following steps:

[0164] Step 1501: for each set of jitter parameters and input images, extract features of the jitter parameters to obtain sampling parameters.

[0165] Optionally, the sampling parameters include multiple sampling positions and weight information corresponding to each sampling position.

[0166] In this way, features can be extracted from corresponding positions of the input image according to the sampling positions. In this way, directional feature extraction is performed, which greatly improves the accuracy and effectiveness of feature extraction, thereby helping to improve the clarity of the target clear image.

[0167] Exemplarily, the feature extraction module includes a first convolution layer and a first data extraction layer, the jitter parameter is input into the first convolution layer for feature extraction, and the extracted information is processed by the first data extraction layer to obtain the sampling parameter.

[0168] Step 1502: Perform feature extraction processing on the input image according to the sampling parameters to obtain first feature information of the input image.

[0169] In an optional embodiment of the present application, according to the sampling parameters, feature extraction processing is performed on the input image to obtain the first feature information of the input image, including: according to multiple sampling positions, feature extraction is performed on the image area corresponding to each sampling position in the input image to obtain sub-feature information corresponding to each sampling position. Weighted processing is performed according to each sub-feature information and each weight information to obtain the first feature information of the input image.

[0170] Exemplarily, each sampling position is mapped to an input image, so as to determine each image region in the input image that can be used for feature extraction, and feature extraction is performed on each image region to obtain a plurality of sub-feature information.

[0171] Exemplarily, for sub-feature information corresponding to each sampling position, the sub-feature information is multiplied by the weight of the sampling position to obtain intermediate feature information, and each intermediate feature information is added to obtain the first feature information of the input image.

[0172] For ease of understanding, the following describes the process of extracting features from a single input image using DCNv2 as the feature extraction module.

[0173] Please refer to Fig.16 The feature extraction process diagram shown in the figure shows that the network uses DCNv2 to process the target deblurring degree. The jitter parameter Target first passes through a simple first convolutional layer conv to extract information, and then uses the first data extraction layer 3K to determine multiple sampling positions △P k and weight information △m k , where 3k is divided into [2k, k], lastk represents the second half, which outputs weight information; first2k represents the first half, which outputs sampling positions. Optionally, the weight information can be determined based on the sigmoid function.

[0174] Understandably, Fig.16 In this way, the input image is subjected to feature extraction based on each sampling position, and weighted processing is performed in combination with the weight information to obtain the first feature information. Optionally, the first feature information may be a feature map after feature extraction.

[0175] In the embodiment of the present application, the structure of DCNv2 can select the sampling position more purposefully according to the input Target, flexibly transform and weight the feature position, and meet the requirements of the image deblurring processing in the embodiment of the present application, compared with the basic deblurring processing. Compared with the basic deblurring processing, the movement of the electronic device can be collected by the sensor (IMU). Considering that one of the biggest difficulties in deblurring processing is to identify the degree of blur in the blur image, we can directly collect it through the sensor in this step. Taking this advantage as a guide for the degree and direction of image blur, the deblurring difficulty of the network is reduced and the network blur restoration effect is enhanced.

[0176] In an exemplary embodiment, each second feature information is input into a decoder to obtain a target clear image output by the decoder, including: inputting each second feature information into a decoder to obtain a plurality of candidate clear images of different scales output by the decoder; and taking the candidate clear image with the largest scale as the target clear image.

[0177] Exemplarily, each second sub-feature processing module outputs candidate clear images of different scales respectively.

[0178] For example, the scale of the candidate clear image output by the second sub-feature processing module at a later level is larger.

[0179] It should be noted that when deblurring the target blurred image, only the candidate clear image with the largest scale needs to be retained as the target clear image. During the training of the initial deblurring network model, the candidate clear images of various scales can be used to adjust the parameters of the initial deblurring network model.

[0180] In a possible implementation, during the training process, a loss function value is calculated based on a plurality of candidate clear images of different scales, and the model parameters are adjusted based on the loss function value.

[0181] For example, the candidate clear images of three scales from small to large output by the model are: I E↓4 ,I E↓2 ,I E The loss function used in model training is the L1 loss function, and the formula is: Loss = ||II E ||1 + ||I ↓2 -I E↓2 || + ||I ↓4 -I E↓4 ||. Among them, I, I↓2, and I↓4 are the reference clear images without downsampling, downsampling once, and downsampling twice, respectively.

[0182] In an optional embodiment of the present application, the encoder and decoder may be implemented based on a multi-scale U-Net structure. For ease of understanding, the structure of the target deblurring network model is described below using this as an example.

[0183] like Fig.17 As shown in the figure, the first feature information output by each DCNv2 is respectively input into the first sub-feature processing module at different levels, and the second feature information output by each first sub-feature processing module reaches the corresponding second sub-feature processing module and the first sub-feature processing module connected at the next level. Thus, the image is gradually restored based on feature information of different scales. In addition, the model has three scales of fuzzy image input and three scales of clear image output.

[0184] From the above description, it can be seen that the blurred image processing method provided in the embodiment of the present application can perform a corresponding degree of deblurring on the target blurred image based on the jitter parameters output by the sensor of the electronic device, and can achieve the effect of removing a specific degree of blurring on blurred images of different scales based on the jitter parameters by using DCNv2 and performing flexible feature sampling on blurred images of different scales, thereby improving the flexibility of deblurring. In addition, the method can also expand the compensation angle of OIS of electronic devices, effectively reducing the cost and volume of electronic devices.

[0185] In the optional embodiment of the present application, since the method reasonably utilizes sensor data to assist in deblurring, the solution is decoupled from the specific network structure used, so the required network structure can be used according to the needs. For example, in the scenario of deblurring images in the album of an electronic device, a relatively large deblurring network model with good basic effect can be selected; when deblurring is performed in the process of video / photography, a relatively lightweight model can be selected to reduce the time required for deblurring.

[0186] It is understandable that the fuzzy image processing method can be applied to many image restoration tasks. In addition, in image generation scenarios (such as motion simulation), IMU data can also be used to guide electronic devices to generate from clear images to fuzzy images.

[0187] For ease of understanding, the blurred image processing method provided by the present application is described below with a complete embodiment.

[0188] The first jitter motion data when the shooting device of the electronic device shoots a blurred image of a target is obtained; the first jitter motion data is subjected to anti-shake filtering processing to obtain an anti-shake compensation amount; and the anti-shake compensation amount is used as a jitter parameter.

[0189] Obtain a target blurred image, perform downsampling on the target blurred image, and obtain multiple candidate blurred images of different scales. Input the target blurred image, multiple candidate blurred images, and jitter parameters into a target deblurring network model, and obtain a target clear image output by the target deblurring network model.

[0190] The target deblurring network model includes a feature extraction module, an encoder and a decoder. Each candidate blurred image and the target blurred image are used as input images. The jitter parameters and each input image are grouped and input into the feature extraction module to obtain the first feature information corresponding to each input image output by the feature extraction module; each first feature information is input into the encoder so that the encoder performs feature extraction processing on each first feature information to obtain multiple second feature information output by the encoder; each second feature information is input into the decoder to obtain the target clear image output by the decoder. Multiple candidate clear images of different scales can be output by the decoder; the candidate clear image with the largest scale is used as the target clear image.

[0191] In addition, an image training set needs to be constructed during the training process of the target deblurring network model. First, multiple reference clear images and second jitter motion data corresponding to the reference clear images are obtained, and the second jitter motion data includes multiple moving positions of the electronic device during the exposure time when the electronic device shoots the reference clear image. Each moving position is mapped to a preset coordinate system, and according to the mapping result, a mapping motion trajectory formed by each moving position in the preset coordinate system is determined; a plurality of block coordinates of preset areas are set in the preset coordinate system; each block coordinate is traversed to determine whether the reference trajectory range planned with the block coordinate as the reference position covers the mapping motion trajectory; if so, the block coordinate is used as a candidate block coordinate; according to the coordinate position of each candidate block coordinate, a motion blur kernel is determined. The coordinate position of each candidate block coordinate is normalized to obtain a first pixel block with a plurality of pixel values ​​of a first value; according to each first pixel block, a convolution image including each first pixel block and a plurality of second pixel blocks is generated; wherein the pixel value of the second pixel block is a second value; and the convolution image is used as a convolution kernel. The reference clear image is convolved according to the convolution kernel to obtain a reference blurred image corresponding to the reference clear image.

[0192] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0193] Based on the same inventive concept, the embodiment of the present application also provides a blurred image processing device for implementing the blurred image processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more blurred image processing device embodiments provided below can refer to the limitations of the blurred image processing method above, and will not be repeated here.

[0194] In one embodiment, Fig.18 As shown, a blurred image processing device 1800 is provided, comprising: a parameter acquisition module 1801 and a blur processing module 1802, wherein:

[0195] The parameter acquisition module 1801 is used to acquire a target blurred image and obtain a jitter parameter of the target blurred image; the jitter parameter is used to characterize the blur degree of the target blurred image;

[0196] The blur processing module 1802 is used to input the target blur image and the jitter parameter into the target deblurring network model to obtain the target clear image output by the target deblurring network model;

[0197] Among them, the jitter parameter is related to the degree to which the target deblurring network model deblurs the target blurred image.

[0198] In an optional embodiment of the present application, the parameter acquisition module 1801 is specifically used to: obtain first jitter motion data when a shooting device of an electronic device shoots a blurred image of a target; and determine a jitter parameter according to the first jitter motion data.

[0199] In an optional embodiment of the present application, the parameter acquisition module 1801 is specifically used to: perform anti-shake filtering processing on the first jitter motion data to obtain an anti-shake compensation amount; and use the anti-shake compensation amount as a jitter parameter.

[0200] In an optional embodiment of the present application, the blur processing module 1802 is specifically used to: downsample the target blurred image to obtain multiple candidate blurred images of different scales; input the target blurred image, multiple candidate blurred images and jitter parameters into the target deblurring network model to obtain the target clear image output by the target deblurring network model.

[0201] In an optional embodiment of the present application, the target deblurring network model includes a feature extraction module, an encoder and a decoder; the blur processing module 1802 is specifically used to: take each candidate blurred image and the target blurred image as input images, input the jitter parameters and each input image into the feature extraction module in groups, and obtain the first feature information corresponding to each input image output by the feature extraction module; input each first feature information into the encoder, so that the encoder performs feature extraction processing on each first feature information, and obtain multiple second feature information output by the encoder; input each second feature information into the decoder, and obtain a target clear image output by the decoder.

[0202] In an optional embodiment of the present application, the blur processing module 1802 is specifically used to: for each group of jitter parameters and input images, perform feature extraction on the jitter parameters to obtain sampling parameters; and perform feature extraction processing on the input image according to the sampling parameters to obtain first feature information of the input image.

[0203] In an optional embodiment of the present application, the sampling parameters include multiple sampling positions and weight information corresponding to each sampling position; the fuzzy processing module 1802 is specifically used to: according to the multiple sampling positions, extract features of the image area corresponding to each sampling position in the input image to obtain sub-feature information corresponding to each sampling position; perform weighted processing according to each sub-feature information and each weight information to obtain the first feature information of the input image.

[0204] In an optional embodiment of the present application, the blur processing module 1802 is specifically used to: input each second feature information into a decoder to obtain a plurality of candidate clear images of different scales output by the decoder; and use the candidate clear image with the largest scale as the target clear image.

[0205] In an optional embodiment of the present application, the device also includes a training module, which is used to: obtain an image training set; train an initial deblurring network model according to the image training set to obtain a target deblurring network model.

[0206] In an optional embodiment of the present application, the training module is specifically used to: obtain multiple reference clear images; generate reference blurred images corresponding to each reference clear image based on each reference clear image; and construct an image training set based on each reference clear image and each reference blurred image.

[0207] In an optional embodiment of the present application, the training module is specifically used to: for each reference clear image, determine a convolution kernel corresponding to the reference clear image; and perform convolution processing on the reference clear image according to the convolution kernel to obtain a reference blurred image corresponding to the reference clear image.

[0208] In an optional embodiment of the present application, the training module is specifically used to: obtain second jitter motion data corresponding to a reference clear image; determine a motion blur kernel corresponding to the reference clear image based on the second jitter motion data; and determine a convolution kernel corresponding to the reference clear image based on the motion blur kernel corresponding to the reference clear image.

[0209] In an optional embodiment of the present application, the training module is specifically used to: map each moving position to a preset coordinate system; determine the mapping motion trajectory formed by each moving position in the preset coordinate system based on the mapping result; and determine the motion blur kernel corresponding to the reference clear image based on the mapping motion trajectory.

[0210] In an optional embodiment of the present application, the training module is specifically used to: traverse each block coordinate to determine whether the reference trajectory range planned with the block coordinate as the reference position covers the mapped motion trajectory; if so, use the block coordinate as the candidate block coordinate; and determine the motion blur kernel according to the coordinate position of each candidate block coordinate.

[0211] In an optional embodiment of the present application, the training module is specifically used to: normalize the coordinate position of each candidate block coordinate to obtain a first pixel block whose pixel values ​​are the first value; generate a convolution image including each first pixel block and a plurality of second pixel blocks according to each first pixel block; wherein the pixel value of the second pixel block is the second value; and use the convolution image as the convolution kernel.

[0212] Each module in the blurred image processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0213] In an exemplary embodiment, an electronic device is provided, the internal structure of which can be as shown in FIG. Fig.19 As shown. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a blurred image processing method is implemented. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device housing, or an external keyboard, touchpad or mouse.

[0214] Those skilled in the art will understand that Fig.19 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0215] In an exemplary embodiment, an electronic device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0216] Obtain a target blurred image and a jitter parameter of the target blurred image; the jitter parameter is used to characterize the blur degree of the target blurred image; the target blurred image and the jitter parameter are input into a target deblurring network model to obtain a target clear image output by the target deblurring network model; wherein the jitter parameter is related to the degree to which the target deblurring network model deblurs the target blurred image.

[0217] In an exemplary embodiment, the processor further implements the following steps when executing the computer program:

[0218] The first shaking motion data when the shooting device of the electronic device shoots a blurred image of a target is obtained; and the shaking parameter is determined according to the first shaking motion data.

[0219] In an exemplary embodiment, the processor further implements the following steps when executing the computer program:

[0220] Perform anti-shake filtering processing on the first jitter motion data to obtain an anti-shake compensation amount; and use the anti-shake compensation amount as a jitter parameter.

[0221] In an exemplary embodiment, the processor further implements the following steps when executing the computer program:

[0222] The target blurred image is downsampled to obtain multiple candidate blurred images of different scales; the target blurred image, multiple candidate blurred images and jitter parameters are input into the target deblurring network model to obtain the target clear image output by the target deblurring network model.

[0223] In an exemplary embodiment, the processor further implements the following steps when executing the computer program:

[0224] Each candidate blurred image and the target blurred image are taken as input images, and the jitter parameters and each input image are respectively input into the feature extraction module in groups to obtain the first feature information corresponding to each input image output by the feature extraction module; each first feature information is input into the encoder so that the encoder performs feature extraction processing on each first feature information to obtain multiple second feature information output by the encoder; each second feature information is input into the decoder to obtain the target clear image output by the decoder.

[0225] In an exemplary embodiment, the processor further implements the following steps when executing the computer program:

[0226] For each set of jitter parameters and input images, feature extraction is performed on the jitter parameters to obtain sampling parameters; and feature extraction processing is performed on the input image according to the sampling parameters to obtain first feature information of the input image.

[0227] In an exemplary embodiment, the processor further implements the following steps when executing the computer program:

[0228] According to multiple sampling positions, feature extraction is performed on image regions corresponding to each sampling position in the input image to obtain sub-feature information corresponding to each sampling position; weighted processing is performed according to each sub-feature information and each weight information to obtain first feature information of the input image.

[0229] In an exemplary embodiment, the processor further implements the following steps when executing the computer program:

[0230] Each second feature information is input into a decoder to obtain a plurality of candidate clear images of different scales output by the decoder; and the candidate clear image with the largest scale is used as a target clear image.

[0231] In an exemplary embodiment, the processor further implements the following steps when executing the computer program:

[0232] Obtain an image training set; train an initial deblurring network model according to the image training set to obtain a target deblurring network model.

[0233] In an exemplary embodiment, the processor further implements the following steps when executing the computer program:

[0234] Acquire multiple reference clear images; generate reference blurred images corresponding to each reference clear image according to each reference clear image; and construct an image training set according to each reference clear image and each reference blurred image.

[0235] In an exemplary embodiment, the processor further implements the following steps when executing the computer program:

[0236] For each reference clear image, a convolution kernel corresponding to the reference clear image is determined; and the reference clear image is convolved according to the convolution kernel to obtain a reference blurred image corresponding to the reference clear image.

[0237] In an exemplary embodiment, the processor further implements the following steps when executing the computer program:

[0238] Acquire second jitter motion data corresponding to the reference clear image; determine a motion blur kernel corresponding to the reference clear image according to the second jitter motion data; and determine a convolution kernel corresponding to the reference clear image according to the motion blur kernel corresponding to the reference clear image.

[0239] In an exemplary embodiment, the processor further implements the following steps when executing the computer program:

[0240] Map each moving position to a preset coordinate system; determine a mapping motion trajectory formed by each moving position in the preset coordinate system according to the mapping result; and determine a motion blur kernel corresponding to a reference clear image according to the mapping motion trajectory.

[0241] In an exemplary embodiment, the processor further implements the following steps when executing the computer program:

[0242] Traverse each block coordinate to determine whether the reference trajectory range planned with the block coordinate as the reference position covers the mapped motion trajectory; if so, use the block coordinate as the candidate block coordinate; determine the motion blur kernel according to the coordinate position of each candidate block coordinate.

[0243] In an exemplary embodiment, the processor further implements the following steps when executing the computer program:

[0244] The coordinate position of each candidate block coordinate is normalized to obtain a plurality of first pixel blocks whose pixel values ​​are the first value; based on each first pixel block, a convolution image including each first pixel block and a plurality of second pixel blocks is generated; wherein the pixel value of the second pixel block is the second value; and the convolution image is used as the convolution kernel.

[0245] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0246] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0247] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0248] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0249] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0250] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A blurred image processing method, characterized in that: The method comprises: Acquire a target blurred image, and acquire a jitter parameter of the target blurred image; the jitter parameter is used to characterize the blur degree of the target blurred image; Inputting the target blurred image and the jitter parameter into a target deblurring network model to obtain a target clear image output by the target deblurring network model; The jitter parameter is related to the degree to which the target deblurred image is deblurred by the target deblurring network model.

2. The method according to claim 1, characterized in that: Obtaining the jitter parameters of the target blurred image includes: Acquire first jitter motion data when a shooting device of the electronic device shoots the blurred image of the target; The jitter parameter is determined according to the first jitter motion data.

3. The method according to claim 2, characterized in that The determining the jitter parameter according to the first jitter motion data includes: Performing anti-shake filtering processing on the first shaking motion data to obtain an anti-shake compensation amount; The anti-shake compensation amount is used as the shake parameter.

4. The method according to claim 1, characterized in that: The step of inputting the target blurred image and the jitter parameter into a target deblurring network model to obtain a target clear image output by the target deblurring network model comprises: Downsampling the target blurred image to obtain a plurality of candidate blurred images of different scales; The target blurred image, the plurality of candidate blurred images and the jitter parameter are input into the target deblurring network model to obtain the target clear image output by the target deblurring network model.

5. The method according to claim 4, characterized in that The target deblurring network model includes a feature extraction module, an encoder and a decoder; the target blurred image, a plurality of candidate blurred images and the jitter parameter are input into the target deblurring network model to obtain the target clear image output by the target deblurring network model, including: Taking each of the candidate blurred images and the target blurred image as input images, inputting the jitter parameters and each of the input images into the feature extraction module in groups, and obtaining first feature information corresponding to each of the input images output by the feature extraction module; Input each of the first feature information into the encoder, so that the encoder performs feature extraction processing on each of the first feature information to obtain a plurality of second feature information output by the encoder; Each of the second feature information is input into the decoder to obtain the target clear image output by the decoder.

6. The method according to claim 5, characterized in that The processing process of the feature extraction module includes: For each group of the jitter parameters and the input image, extract features of the jitter parameters to obtain sampling parameters; According to the sampling parameters, feature extraction processing is performed on the input image to obtain first feature information of the input image.

7. The method according to claim 6, characterized in that The sampling parameters include a plurality of sampling positions and weight information corresponding to each of the sampling positions; and the feature extraction process is performed on the input image according to the sampling parameters to obtain the first feature information of the input image, including: According to the multiple sampling positions, feature extraction is performed on the image area corresponding to each sampling position in the input image to obtain sub-feature information corresponding to each sampling position; A weighted process is performed according to each of the sub-feature information and each of the weight information to obtain the first feature information of the input image.

8. The method according to claim 5, characterized in that The step of inputting each of the second feature information into the decoder to obtain the target clear image output by the decoder includes: Inputting each of the second feature information into the decoder to obtain a plurality of candidate clear images of different scales output by the decoder; The candidate clear image with the largest scale is used as the target clear image.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: Get the image training set; The initial deblurring network model is trained according to the image training set to obtain the target deblurring network model.

10. The method according to claim 9, characterized in that The step of obtaining an image training set comprises: Acquire multiple reference clear images; Generate a reference blurred image corresponding to each reference clear image according to each reference clear image; The image training set is constructed according to each of the reference clear images and each of the reference blurred images.

11. The method according to claim 10, characterized in that The step of generating a reference blurred image corresponding to each of the reference clear images according to each of the reference clear images comprises: For each of the reference clear images, determining a convolution kernel corresponding to the reference clear image; The reference clear image is convolved according to the convolution kernel to obtain the reference blurred image corresponding to the reference clear image.

12. The method according to claim 11, characterized in that The determining of the convolution kernel corresponding to the reference clear image includes: Acquire second jitter motion data corresponding to the reference clear image; Determining a motion blur kernel corresponding to the reference clear image according to the second jitter motion data; According to the motion blur kernel corresponding to the reference clear image, a convolution kernel corresponding to the reference clear image is determined.

13. The method according to claim 12, characterized in that The second jitter motion data includes a plurality of moving positions of the electronic device within an exposure time when the electronic device captures the reference clear image; and determining the motion blur kernel corresponding to the reference clear image according to the second jitter motion data includes: Mapping each of the moving positions to a preset coordinate system; Determine, according to the mapping result, a mapping motion trajectory formed by each of the moving positions in the preset coordinate system; According to the mapped motion trajectory, a motion blur kernel corresponding to the reference clear image is determined.

14. The method according to claim 13, characterized in that The preset coordinate system is provided with a plurality of block coordinates of preset areas; and the motion blur kernel corresponding to the reference clear image is determined according to the mapped motion trajectory, including: Traversing each of the block coordinates, determining whether a reference trajectory range planned with the block coordinate as a reference position covers the mapped motion trajectory; if so, taking the block coordinate as a candidate block coordinate; The motion blur kernel is determined according to the coordinate position of each candidate block coordinate.

15. The method according to claim 14, characterized in that The step of determining the convolution kernel corresponding to the reference clear image according to the motion blur kernel corresponding to the reference clear image comprises: Normalizing the coordinate position of each candidate block coordinate to obtain a first pixel block having a plurality of pixel values ​​having a first value; Generate a convolution image including each of the first pixel blocks and a plurality of second pixel blocks according to each of the first pixel blocks; wherein the pixel value of each of the second pixel blocks is a second value; The convolution image is used as the convolution kernel.

16. A blurred image processing device, characterized in that: The device comprises: A parameter acquisition module, used to acquire a target blurred image and acquire a jitter parameter of the target blurred image; the jitter parameter is used to characterize the blur degree of the target blurred image; A blur processing module, used for inputting the target blurred image and the jitter parameter into a target deblurring network model to obtain a target clear image output by the target deblurring network model; The jitter parameter is related to the degree to which the target deblurred image is deblurred by the target deblurring network model.

17. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 15 are implemented.

18. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images