Image processing method, device, electronic device, and computer-readable storage medium

Abstract

The present application relates to an image processing method, apparatus, computer device, storage medium, and computer program product. The method comprises: obtaining initial RAW image data and corresponding gyroscope data; determining, based on the positional relationship between the gyroscope and the image sensor, the jitter trajectory of the image sensor when acquiring the initial RAW image data according to the gyroscope data; determining a blur kernel corresponding to the image sensor according to the jitter trajectory of the image sensor; and deblurring the initial RAW image data according to the blur kernel to obtain a target RAW image. This method can avoid the influence of other nonlinear processing processes in the image processing process on the initial RAW image data, and provides an accurate blur kernel for deblurring the image, resulting in a better deblurring effect and a target RAW image with higher clarity.

Description

Technical Field

[0001] The present application relates to the field of imaging technology, and in particular to an image processing method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the advancement of technology, electronic devices such as mobile phones and tablets are increasingly used in people's work, study, and daily communication. Electronic devices are often equipped with cameras for both photo and video recording. However, due to the small size of the camera and its small photosensitive area, the user's hand may shake during the photo-taking process, resulting in blurred image data. This is especially true in low-light conditions at night. To increase the amount of light entering the sensor, the exposure time is extended, resulting in even more blurred images. This is because the range of hand shake exceeds the size of a single sensor pixel within the shutter time.

[0003] In order to remove image blur caused by hand shaking, a common method is to use a corresponding deblurring algorithm to deblur the image blurred by hand shaking, but the deblurring effect is poor. Summary of the Invention

[0004] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can effectively remove image blur caused by jitter.

[0005] In a first aspect, the present application provides an image processing method. The method comprises:

[0006] Get the initial RAW image data and the corresponding gyroscope data;

[0007] determining, based on the positional relationship between the gyroscope and the image sensor, a jitter trajectory of the image sensor when acquiring the initial RAW image data according to the gyroscope data;

[0008] determining a blur kernel corresponding to the image sensor according to a jitter trajectory of the image sensor;

[0009] Deblurring is performed on the initial RAW image data according to the blur kernel to obtain a target RAW image.

[0010] In a second aspect, the present application further provides an image processing device. The device comprises:

[0011] An initial data acquisition module, used to acquire initial RAW image data and corresponding gyroscope data;

[0012] a jitter trajectory determining module, configured to determine, based on a positional relationship between the gyroscope and the image sensor and according to the gyroscope data, a jitter trajectory of the image sensor when acquiring the initial RAW image data;

[0013] a blur kernel determining module, configured to determine a blur kernel corresponding to the image sensor according to a jitter trajectory of the image sensor;

[0014] The dithering and blurring processing module is configured to perform a deblurring process on the initial RAW image data according to the blur kernel to obtain a target RAW image.

[0015] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0016] Get the initial RAW image data and the corresponding gyroscope data;

[0017] determining, based on the positional relationship between the gyroscope and the image sensor, a jitter trajectory of the image sensor when acquiring the initial RAW image data according to the gyroscope data;

[0018] determining a blur kernel corresponding to the image sensor according to a jitter trajectory of the image sensor;

[0019] Deblurring is performed on the initial RAW image data according to the blur kernel to obtain a target RAW image.

[0020] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0021] Get the initial RAW image data and the corresponding gyroscope data;

[0022] determining, based on the positional relationship between the gyroscope and the image sensor, a jitter trajectory of the image sensor when acquiring the initial RAW image data according to the gyroscope data;

[0023] determining a blur kernel corresponding to the image sensor according to a jitter trajectory of the image sensor;

[0024] Deblurring is performed on the initial RAW image data according to the blur kernel to obtain a target RAW image.

[0025] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0026] Get the initial RAW image data and the corresponding gyroscope data;

[0027] determining, based on the positional relationship between the gyroscope and the image sensor, a jitter trajectory of the image sensor when acquiring the initial RAW image data according to the gyroscope data;

[0028] determining a blur kernel corresponding to the image sensor according to a jitter trajectory of the image sensor;

[0029] Deblurring is performed on the initial RAW image data according to the blur kernel to obtain a target RAW image.

[0030] The above-mentioned image processing method, device, electronic device, computer-readable storage medium and computer program product determine the jitter trajectory of the image sensor when collecting initial RAW image data based on gyroscope data, accurately determine the blur kernel of the image sensor based on the jitter trajectory, and deblur the initial RAW image data based on the blur kernel. Since the deblurring processing is performed on the initial RAW image data, the influence of other nonlinear processing processes in the image processing process on the initial RAW image data can be avoided, and the deblurring effect is better. At the same time, using an accurate blur kernel to deblur the image can obtain a target RAW image with higher clarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0032] Figure 1 A diagram showing an application environment of an image processing method in one embodiment;

[0033] Figure 2 is a flowchart of an image processing method in one embodiment;

[0034] Figure 3 is a flow chart of step 204 in one embodiment;

[0035] Figure 4 is a flow chart of step 302 in one embodiment;

[0036] Figure 5 is a flow chart of step 402 in one embodiment;

[0037] Figure 6 is a schematic diagram of a spatial rectangular coordinate system established with a gyroscope as the center in one embodiment;

[0038] Figure 7 is a schematic diagram of a polar coordinate system established with a gyroscope as the center in one embodiment;

[0039] Figure 8 is a flow chart of step 206 in one embodiment;

[0040] Figure 9 Schematic diagram of an electronic device generating a diffuse circular spot by shaking in a direction perpendicular to the image sensor plane in one embodiment;

[0041] Figure 10 A schematic diagram of a diffuse circular spot generated during movement of a target position in an image sensor in one embodiment;

[0042] Figure 11 Schematic diagram of a method for calculating a first blur kernel corresponding to a target position in one embodiment;

[0043] Figure 12 1 is a schematic diagram of a process for training a convolutional neural network based on a fuzzy kernel in one embodiment;

[0044] Figure 13 is a schematic diagram of downsampling initial RAW image data and a blur kernel in one embodiment;

[0045] Figure 14 is a flowchart of an image processing method in another embodiment;

[0046] Figure 15 is a flowchart of an image processing method in another embodiment;

[0047] Figure 16 is a structural block diagram of an image processing device in one embodiment;

[0048] Figure 17 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0050] When using a camera to take images, in order to increase the amount of light entering the camera, when the exposure time is too long, hand shaking will cause the acquired image to become blurred. This is because the range of hand shaking within the shutter time exceeds the size of a single pixel of the sensor.

[0051] To address the blur caused by hand shaking, one solution in traditional technology is to add OIS (Optical Image Stabilizer) and use OIS to control the movement of the lens or image sensor to compensate for hand shaking. This solution is expensive, and as the sensor becomes larger, the size of the lens and the corresponding OIS will also become larger, and the weight will also increase. Another solution is to use a deblurring algorithm to eliminate image blur, and obtain a clear image by obtaining the blur kernel caused by hand shaking, but the clarity of the obtained image is low. To address the above problems, this application proposes an image processing method.

[0052] The image processing method provided in the embodiment of the present application is described by taking an electronic device as an example. Figure 1 As shown, the electronic device 100 can be a terminal or a server. The terminal can be, but is not limited to, various personal computers, cameras, scanners, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented as an independent server or a server cluster consisting of multiple servers. It is understandable that the method can also be applied to servers, and can also be applied to systems including terminals and servers, and can be implemented through the interaction between the terminals and servers.

[0053] In one embodiment, an electronic device includes components such as a camera and a gyroscope. The operating system of the electronic device may include Android, IOS (Input Output System), Windows Phone, Windows, etc., and can support the operation of multiple applications, such as map applications and game applications. A camera is an image data acquisition device that can be used for shooting. It can be front-mounted or rear-mounted. A camera generally includes components such as a lens, a base, an infrared filter, an image sensor, and a circuit board. A gyroscope can measure angular velocity and acceleration and has high dynamic characteristics. It is understandable that a gyroscope is not a sensor installed in the camera, but rather a general sensor in the electronic device that can be simultaneously called by multiple applications to perform related business operations. For example, a camera application can call on the gyroscope for image processing, a map application can call on the gyroscope for navigation, and a game application can call on the gyroscope for AR (Augmented Reality) three-dimensional modeling.

[0054] In one embodiment, Figure 2 As shown, an image processing method is provided, including the following steps 202 to 208.

[0055] Step 202: Acquire initial RAW image data and corresponding gyroscope data.

[0056] Initial RAW image data refers to the original image file acquired by the image sensor that has not been processed, printed or edited.

[0057] In this embodiment, when the electronic device receives a camera startup instruction, it can call the camera to collect initial RAW image data within the exposure time, and at the same time, call the gyroscope to measure corresponding gyroscope data, such as angular velocity data and acceleration data. The camera startup instruction refers to an instruction to start the camera, which can be triggered by clicking the camera application or pressing a designated button. When the electronic device shakes, the gyroscope can measure gyroscope data before and after the shake or within a specified time. Optionally, the electronic device can obtain the initial RAW image data and gyroscope data collected within the exposure time.

[0058] In an optional embodiment, the raw data output by the image sensor undergoes bad pixel correction (BPC), black level correction (BLC), and lens shadow correction (LSC) to obtain initial RAW image data. Bad pixel correction refers to the correction of defective pixels caused by defects in the array process forming each light-collecting point on the sensor, or errors in the light signal conversion process, resulting in incorrect information for some pixels in the image. Black level refers to the signal level corresponding to when the image data is 0. In actual situations, dark current exists in the image sensor circuit, resulting in a certain output voltage even when there is no light input. This output value needs to be removed through correction so that the output voltage of the sensor circuit is proportional to the input brightness. Due to the optical characteristics of the lens, the edge areas of the sensor image area receive less light intensity than the center, resulting in inconsistent brightness between the center and the periphery. Therefore, LSC is required to restore the energy attenuated by the lens to its pre-attenuation state.

[0059] Step 204 : Based on the positional relationship between the gyroscope and the image sensor, determine the jitter trajectory of the image sensor when acquiring the initial RAW image data according to the gyroscope data.

[0060] Because gyroscopes and image sensors are typically installed at different locations on electronic devices, after acquiring gyroscope data, it is necessary to determine the jitter trajectory of the image sensor when acquiring the initial RAW image data based on the positional relationship between the gyroscope and the image sensor.

[0061] Alternatively, the gyroscope and the image sensor can be established in the same coordinate system, with the gyroscope's coordinates representing the image sensor's coordinates. For example, a spatial rectangular coordinate system and a corresponding polar coordinate system can be established with the gyroscope as the origin. The coordinates of the image sensor in the corresponding spatial rectangular coordinate system can be obtained using the acquired gyroscope data or the gyroscope's polar coordinate values, based on the conversion relationship between polar coordinates and spatial rectangular coordinates.

[0062] Since gyroscope data is constantly changing when an electronic device is shaken, a preset sampling duration can be set. The exposure duration for acquiring the initial RAW image data can include multiple preset sampling durations. Gyroscope data is acquired once every preset sampling duration. Each acquisition of gyroscope data corresponds to a gyroscope polar coordinate. Based on the conversion relationship between polar coordinates and spatial rectangular coordinates, the coordinates of the corresponding image sensor in the spatial rectangular coordinate system are obtained. Based on the coordinates of the image sensor corresponding to each acquired gyroscope data, the image sensor's shake trajectory can be obtained. For example, the image sensor coordinates can be connected in a predetermined order to form the image sensor's shake trajectory.

[0063] Optionally, the image sensor can be divided into multiple areas, each area includes at least one target position, and the jitter trajectory of each target position when collecting initial RAW image data is determined based on the gyroscope data, thereby obtaining the jitter trajectory of each area when collecting initial RAW image data, and finally obtaining the jitter trajectory of the image sensor when collecting initial RAW image data.

[0064] Step 206 : Determine a blur kernel corresponding to the image sensor according to the jitter trajectory of the image sensor.

[0065] The jitter trajectory of an image sensor is typically characterized by the jitter trajectory of at least one target position on the image sensor. For the jitter trajectory of a target position, the coordinates corresponding to the jitter trajectory of the target position can be directly used as the corresponding point spread function (PSF). However, for lenses with a shallow depth of field, when the coordinates perpendicular to the image plane of the image sensor change, the size of the diffuse spot in the corresponding direction will change. Therefore, it is necessary to determine the point spread function corresponding to the target position at each coordinate based on the coordinates corresponding to the jitter trajectory of the target position and the size of the corresponding diffuse spot, and further determine the blur kernel (Blur Kernel) corresponding to the target position based on the point spread function.

[0066] The diffuse spot is the light intensity distribution of the diffraction image formed by a point light source (i.e., a star) passing through the optical system at different sections in front of and behind the image sensor image plane. A diffuse circular spot is a circular diffuse spot. The point spread function is used to characterize the light field distribution of the output image when the input object is a point light source. The blur kernel is actually a matrix. The convolution of a clear image with the blur kernel causes the image to become blurred, hence the name blur kernel. The blur kernel is a type of convolution kernel, and the essence of image convolution is matrix convolution. In other words, image blurring can be viewed as the process of convolving a clear image with a blur kernel to obtain a blurred image.

[0067] Optionally, the electronic device may determine the blur kernel corresponding to the target position according to the coordinates and diffuse circular spot corresponding to the jitter trajectory of the target position, and may determine the blur kernel corresponding to the image sensor based on the blur kernels corresponding to each target position.

[0068] Step 208 : Deblurring the initial RAW image data according to the blur kernel to obtain a target RAW image.

[0069] In this embodiment, the initial RAW image data is collected by an electronic device and is usually blurred by jitter. According to the definition of the blur kernel, the initial RAW image data is deblurred according to the blur kernel to obtain a clear target RAW image.

[0070] Optionally, the initial RAW image data may be deconvolved by a blur kernel to obtain a target RAW image.

[0071] Optionally, based on the blur kernel, the existing clear image can be convolved into a blurred image, and the convolutional neural network is trained by an image pair consisting of a clear image and a blurred image to obtain a target convolutional neural network. The initial RAW image data is input into the target neural network to obtain a target RAW image.

[0072] In the above-mentioned image processing method, the jitter trajectory of the image sensor when acquiring the initial RAW image data is determined based on gyroscope data. Based on the jitter trajectory, the image sensor's blur kernel is accurately determined. The initial RAW image data is then deblurred based on this blur kernel. Since the deblurring is performed on the initial RAW image data, the impact of other nonlinear processing steps in the image processing process on the initial RAW image data can be avoided, resulting in a better deblurring effect and a higher-definition target RAW image. In addition, the electronic device can use the existing gyroscope to assist in deblurring the initial RAW image data, which can save hardware costs compared to traditional OIS methods.

[0073] In one embodiment, Figure 3As shown, the image sensor includes multiple areas; in step 204 , determining the jitter trajectory of the image sensor when acquiring the initial RAW image data according to the gyroscope data includes the following steps 302 to 304 .

[0074] Step 302 : For each area of ​​the image sensor, determine a first jitter trajectory of each area when acquiring initial RAW image data according to gyroscope data.

[0075] In this embodiment, the image sensor includes multiple regions, and the specific division method of the image sensor into the multiple regions can be set as needed. For example, when the aspect ratio of the image sensor is 4:3, the image sensor is evenly divided into 16*12 regions.

[0076] It can be understood that the areas in the image sensor are used for spatially non-uniform deblurring. Each area has a different first jitter trajectory when collecting the initial RAW image data, and the corresponding blur kernel is also different. Each area in the initial RAW image data uses the corresponding blur kernel for deblurring.

[0077] Optionally, each area of ​​the image sensor includes at least one target position, and a second jitter trajectory of the target position when collecting initial RAW image data is determined based on the gyroscope data. Based on the second jitter trajectory corresponding to each target position in the area, a first jitter trajectory of the area when collecting initial RAW image data is determined.

[0078] Step 304 : Determine the jitter trajectory of the image sensor when acquiring the initial RAW image data according to the first jitter trajectory corresponding to each area.

[0079] Optionally, the first jitter traces corresponding to all areas of the image sensor are used as the jitter traces of the image sensor when acquiring the initial RAW image data. For example, the image sensor includes areas A, B, and C. The first jitter trace corresponding to area A of the image sensor is used as the jitter trace of area A of the image sensor when acquiring the initial RAW image data. The first jitter trace corresponding to area B of the image sensor is used as the jitter trace of area B of the image sensor when acquiring the initial RAW image data. The first jitter trace corresponding to area C of the image sensor is used as the jitter trace of area C of the image sensor when acquiring the initial RAW image data.

[0080] In this embodiment, for each area of ​​the image sensor, a first jitter trajectory of each area on the image sensor can be determined based on the gyroscope data. Based on the first jitter trajectory of each area, a blur kernel of the corresponding area can be determined, so that deblurring processing can be performed on the corresponding area in the initial RAW image data to obtain a clearer target RAW image for each area.

[0081] In one embodiment, Figure 4 As shown, each area includes at least one target position; for each area of ​​the image sensor, step 302 of determining the first jitter trajectory of each area when acquiring the initial RAW image data according to the gyroscope data includes the following steps 402 to 406.

[0082] Step 402: for each target position, determine an affine matrix for transforming the target position from a first coordinate to a second coordinate based on the gyroscope data; wherein a first sampling moment corresponding to the first coordinate and a second sampling moment corresponding to the second coordinate are separated by a preset sampling time.

[0083] In this embodiment, each region on the image sensor includes at least one target location. The specific location and number of target locations in each region can be set as needed. For example, the target location in a region can be the center point or edge point of the region.

[0084] Optionally, within the exposure duration, gyroscope data is collected at intervals of a preset sampling duration, and an affine matrix for transforming the target position from a first coordinate to a second coordinate is determined based on the gyroscope data collected at each time, wherein a first sampling moment corresponding to the first coordinate and a second sampling moment corresponding to the second coordinate are separated by a preset sampling duration. For example, at time t1, the target position corresponds to the first coordinate. After an interval of a preset duration Δt from time t1 to time t2, gyroscope data is collected at time t2, and the target position at time t2 corresponds to the second coordinate. When the second coordinate of the target position is to be determined, the affine matrix for transforming the first coordinate to the second coordinate can be determined based on the gyroscope data between times t1 and t2. The preset sampling duration can be set as needed.

[0085] Step 404 : determining a second shaking trajectory of the target position when acquiring the initial RAW image data according to the affine matrix.

[0086] Based on the gyroscope data collected each time, the affine matrix that transforms the target position from the first coordinate to the second coordinate is determined. Thus, the second coordinate after the position change is obtained based on the first coordinate and the affine matrix. When the coordinates corresponding to the target position at all sampling times are obtained, the corresponding coordinates are connected in order from small to large or from large to small according to the sampling times. This can obtain the second jitter trajectory of the target position when the initial RAW image data is collected.

[0087] Step 406 : Determine the first shaking trajectory of the region when acquiring the initial RAW image data according to the second shaking trajectory corresponding to each target position on the region.

[0088] Optionally, when the region includes one target location, the second jitter trajectory corresponding to the target location is used as the first jitter trajectory for the region when acquiring the initial RAW image data. When the region includes at least two target locations, the average of the second jitter trajectories corresponding to the respective target locations is used as the first jitter trajectory for the region when acquiring the initial RAW image data.

[0089] It should be noted that if a region includes at least two target locations, other methods may be used to determine the first jitter trajectory for the corresponding region, such as weighted averaging or filtering, which are not limited here. It is understood that when using filtering to determine the first jitter trajectory for a region, the second jitter trajectory corresponding to one target location may be selected from the second jitter trajectories corresponding to the various target locations according to a preset filtering condition to serve as the first jitter trajectory for the region.

[0090] In this embodiment, the first jitter trajectory of the region when acquiring the initial RAW image data is determined based on the second jitter trajectory corresponding to the target position in the region. This can more accurately determine the first jitter trajectory corresponding to each region in the image sensor, and determine the blur kernel corresponding to the region based on the first jitter trajectory, ultimately obtaining a clearer target RAW image of each region.

[0091] In one embodiment, Figure 5 As shown, the gyroscope data includes acceleration data and angular velocity data; in step 402 , the affine matrix for transforming the target position from the first coordinate to the second coordinate is determined according to the gyroscope data, including the following steps 502 to 506 .

[0092] Step 502: Determine the translation distance of the gyroscope from the first sampling moment to the second sampling moment based on the acceleration data; and determine the rotation angle of the gyroscope from the first sampling moment to the second sampling moment based on the angular velocity data.

[0093] Gyroscope data includes acceleration and angular velocity data, with acceleration data including linear acceleration data. Performing a quadratic integration on the linear acceleration data yields the gyroscope's translational distance from the first sampling moment to the second sampling moment, i.e., the translational distance from the first coordinate to the second coordinate along each coordinate axis. Performing a quadratic integration on the angular velocity data yields the gyroscope's rotation angle from the first sampling moment to the second sampling moment, i.e., the rotation angle from the first coordinate to the second coordinate along each coordinate axis.

[0094] Step 504 : determining the translation amount of the target position from the first coordinate to the second coordinate according to the translation distance; and determining the rotation amount of the target position from the first coordinate to the second coordinate according to the rotation angle.

[0095] Optionally, the electronic device obtains the positional relationship between the gyroscope and the image sensor, and establishes a coordinate system centered on the gyroscope based on the positional relationship. When the translation distance and rotation angle corresponding to the gyroscope from the first sampling moment to the second sampling moment are determined, the corresponding translation amount and rotation amount of the target position in the image sensor from the first sampling moment to the second sampling moment can be determined.

[0096] Step 506: Determine an affine matrix for transforming the target position from the first coordinate to the second coordinate based on the translation amount and the rotation amount.

[0097] Optionally, a first translation coordinate is obtained based on the first coordinate and the translation amount, and a first rotation coordinate is obtained based on the first coordinate and the rotation amount; a second coordinate can be obtained based on the first translation coordinate and the first rotation coordinate. For example, the first translation coordinate and the second rotation coordinate can be added to obtain the second coordinate.

[0098] The step 404 of determining a second jitter trajectory of the target position when acquiring the initial RAW image data according to the affine matrix includes:

[0099] The coordinates of the target position corresponding to each sampling moment are determined according to the affine matrix corresponding to each preset sampling time length; wherein the coordinates corresponding to each sampling moment are used to represent the second shaking trajectory.

[0100] After determining the second coordinate corresponding to the affine matrix of the preset sampling time, the electronic device can determine the coordinates corresponding to the target position at each sampling moment based on the affine matrix corresponding to each preset sampling time. The coordinates corresponding to the target position at each sampling moment are the second jitter trajectory that represents the target position.

[0101] In one example, if Figure 6 As shown, a spatial rectangular coordinate system is established with the gyroscope as the center. Then, the coordinates of the gyroscope are (0,0,0), the center point of the image sensor is selected as the target position, the initial coordinates of the target position are (x0,y0,z0) as the first coordinates, and the coordinates of the edge point of the image sensor are (x i ,y i ,z i ).

[0102] During the exposure time T, the electronic device collects gyroscope data once every preset sampling time Δt, with the number of sampling times n = T / Δt. The gyroscope data collected each time is temporally integrated twice to obtain the gyroscope's translation distance and rotation angle in the X, Y, and Z directions, respectively. The gyroscope's translation distance and rotation angle in the X, Y, and Z directions are then converted into the image sensor's translation and rotation amounts in the X, Y, and Z directions, respectively.

[0103] The electronic device adds the translation amount of the target position in the X direction, Y direction and Z direction to the initial coordinates of the corresponding directions to obtain the coordinate values ​​of the target position in the X direction, Y direction and Z direction after the preset sampling time, as shown in the following formula (1).

[0104]

[0105] Where x0, y0, and z0 are the initial coordinates of the image sensor in the X, Y, and Z directions, respectively. Δx, Δy, and Δz are the translation amounts of the target position in the X, Y, and Z directions after the preset sampling time, respectively. They are the coordinate values ​​of the target position in the X, Y and Z directions corresponding to the preset sampling time. is the first translation coordinate.

[0106] For the rotation on the axis, the corresponding rotation amount in each direction can be obtained through the mapping relationship between the polar coordinate system and the spatial rectangular coordinate system. Figure 7 The polar coordinate system is shown with the gyroscope as the center. The coordinates of the gyroscope are (0, 0) and the image sensor rotates Δω on the Z axis. z If the initial coordinate of the target position in the polar coordinate system is (x0, y0, z0) as the first coordinate, the coordinate after rotation is (x0′, y0′, z0′).

[0107] During the preset sampling time, when the Z axis rotates, its projection on the XY plane will undergo a position change. After polar coordinate transformation, the first rotation coordinate corresponding to the Z axis rotation can be obtained, as shown in formula (2).

[0108]

[0109] Similarly, for the rotation of the X-axis and Y-axis, the corresponding first rotation coordinates can also be obtained. When the X-axis rotates, the corresponding first rotation coordinates are shown in formula (3):

[0110]

[0111] When rotating about the Y axis, the corresponding first rotation coordinate is shown in formula (4):

[0112]

[0113] Among them, Δω z , Δω x and Δω y They are the rotation amounts on the Z axis, X axis, and Y axis respectively. It can be understood that if the target position is rotated on the axis, the coordinates in the corresponding direction remain unchanged. For example, if it rotates on the Z axis, the coordinates in the Z direction remain unchanged.

[0114] In actual applications, the target position may rotate about at least one of the X-axis, Y-axis, and Z-axis within the preset sampling time, or may not rotate at all. In this case, only translation occurs, that is, only a translation amount is generated.

[0115] The first rotation coordinate is obtained by adding up the values ​​of the rotation coordinates corresponding to the rotation direction. For example, when the X-axis and the Z-axis rotate, the X-axis is rotated by the corresponding Corresponding to the Z-axis rotation Add together and rotate the X axis by the corresponding Corresponding to the Z-axis rotation Add together and rotate the X axis by the corresponding Corresponding to the Z-axis rotation Add them together to get the first rotated coordinate.

[0116] The coordinate values ​​corresponding to the first translation coordinate and the first rotation coordinate are added to obtain the second coordinate.

[0117] It should be noted that the above formulas (1) to (4) are all affine matrices for transforming the target position from the first coordinate to the second coordinate. The corresponding affine matrix is ​​selected according to the coordinate axis that rotates in the actual application. For example, when only the Z axis rotates, the affine matrix is ​​formula (1) and formula (2).

[0118] Optionally, the electronic device combines the second coordinates obtained at each preset sampling time interval of the target position on the image sensor to obtain a second jitter trajectory of the target position. For example, the initial coordinates of the target position are (x0, y0, z0), and the second coordinates obtained at each preset sampling time interval are Where n is a positive integer, then the second jitter trajectory of the target position is The second shaking tracks at other positions on the image sensor can also be calculated using the same method as above, which will not be described in detail here.

[0119] In this embodiment, the electronic device determines the translation distance and rotation angle of the gyroscope within the preset sampling time based on the acceleration data and angular velocity data within the preset sampling time, and then obtains the translation and rotation corresponding to the target position on the image sensor. The corresponding affine matrix is ​​determined based on the translation and rotation. The second coordinate of the target position corresponding to each preset sampling time interval can be determined based on the affine matrix, thereby accurately determining the second jitter trajectory corresponding to the target position.

[0120] In one embodiment, Figure 8 As shown, step 206 of determining the blur kernel corresponding to the image sensor according to the jitter trajectory of the image sensor includes steps 802 to 808 .

[0121] Step 802 : determining the point spread function of the coordinates of the target position at each sampling moment according to the coordinates of the target position at each sampling moment.

[0122] Optionally, for the jitter occurring in the image plane direction of the image sensor, only the change in the position where the corresponding jitter occurs needs to be considered, and the size of the diffuse spot caused by the jitter will not change. That is, the coordinates corresponding to the target position at each sampling moment can be used as the point spread function corresponding to the target position at each sampling moment.

[0123] When jitter occurs in a direction perpendicular to the image sensor's image plane, the size of the diffuse spot caused by the jitter will change. Based on the coordinates of the target position at each sampling moment and the diffuse spot size, the point spread function of the target position at that sampling moment is determined. For example, if the image sensor's image plane is in the X and Y directions, the direction perpendicular to the image sensor's image plane is the Z direction.

[0124] Step 804: Determine a first blur kernel corresponding to the target position according to the point spread function.

[0125] Optionally, the point spread function of the target position's coordinates at each sampling moment may be integrated to obtain a first blur kernel corresponding to the target position. Furthermore, the point spread function of the target position's coordinates at each sampling moment may be summed and averaged to obtain the first blur kernel corresponding to the target position.

[0126] Step 806 : Obtain a second blur kernel corresponding to the region based on the first blur kernel corresponding to each target position.

[0127] Optionally, the electronic device integrates the first blur kernels corresponding to each target location in the region to obtain a second blur kernel corresponding to the region. Furthermore, the electronic device may add and average the first blur kernels corresponding to each target location to obtain the second blur kernel corresponding to the region. Alternatively, the electronic device may perform a weighted average of the first blur kernels corresponding to each target location to obtain the second blur kernel corresponding to the region. For example, the weights corresponding to each target location may be determined based on the specific location of each target location in the region. Alternatively, the electronic device may use the median value of the first blur kernels corresponding to each target location as the second blur kernel corresponding to the region.

[0128] Step 808 : Determine the blur kernel corresponding to the image sensor according to the second blur kernel corresponding to each area.

[0129] The image sensor includes multiple areas, each area corresponds to a second blur kernel, and after the second blur kernels corresponding to all areas are obtained, the blur kernel corresponding to the image sensor is obtained.

[0130] In this embodiment, the electronic device determines the corresponding first blur kernel through the point spread function corresponding to the target position at each coordinate, and then determines the second blur kernel corresponding to the area based on the first blur kernel corresponding to each target position in the area, thereby obtaining the blur kernel corresponding to the image sensor. The blur kernel of the image sensor can be accurately determined to avoid errors caused by the blur kernel data of a single position.

[0131] In some embodiments, step 802 of determining a point spread function of the coordinates of the target position at each sampling moment according to the coordinates of the target position at each sampling moment includes:

[0132] According to the coordinates of the target position at each sampling moment and the photographing parameters, the point spread function of the coordinates of the target position at each sampling moment is determined.

[0133] Optionally, when the electronic device shakes in a direction perpendicular to the image plane of the image sensor, the ideal imaging point at the target position will diffuse into a larger circular spot. Figure 9 As shown in the figure, when an electronic device experiences jitter perpendicular to the image sensor's image plane during capture, the subject's relative position shifts from point A to point B, generating a diffuse circular spot on the image sensor's image plane. Although jitter perpendicular to the image sensor's image plane doesn't cause the blur kernel to move, it does change its size.

[0134] The electronic device determines the point spread function of the target position at each sampling moment based on the coordinates and diffuse spots corresponding to the target position at each sampling moment. Optionally, the electronic device obtains the coordinates of the target position at each sampling moment and the diffuse spots corresponding to each coordinate; adjusts each coordinate to the size of the corresponding diffuse spot to obtain the point spread function corresponding to each coordinate. The size of the diffuse spot includes information such as the diameter, radius, circumference, and area of ​​the diffuse spot. The size of the diffuse spot can reflect the size of the corresponding blur kernel. The size of the diffuse spot corresponding to each coordinate can be the same or different.

[0135] In this embodiment, the photographing parameters refer to parameters used when capturing an image, including focal length, lens diameter, distance from the position of the photographed object to the lens, image distance, etc. The electronic device can obtain the diffuse circular spot on the image sensor by determining the size of the diffuse circular spot.

[0136] In one example, combining Figure 9 It can be shown that the size of the diffuse circular spot can be calculated by the following formula (5).

[0137]

[0138] Where δ is the diameter of the diffuse spot, μ B is the distance from point B to the lens, μA is the distance from point A to the lens, f is the focal length, D is the diameter of the lens, and Δz is the moving distance in the direction perpendicular to the image plane of the image sensor. A The parameters are known, and the moving distance Δz in the direction perpendicular to the image plane of the image sensor is obtained, then μ can be calculated. B Parameters, and then obtain the focal length f and the diameter D of the lens, the diameter of the diffuse circular spot can be calculated.

[0139] For the target position in each area of ​​the image sensor, the corresponding point spread function can be obtained by using the above method. Figure 10 As shown, at time t1 within the exposure time, the coordinates of the target position in a certain area are (x1, y1, z1), and the diameter of the generated diffuse circular spot is 1 pixel; at time t2, the coordinates of the target position are (x2, y2, z2), that is, the target position moves a distance of (x2-x1) in the x direction and a distance of (y2-y1) in the y direction, and based on the movement distance (z2-z1) of the image sensor in the z direction, the diameter of the diffuse circular spot on the imaging surface of the image sensor is determined; at time t3, the coordinates of the target position are (x3, y3, z3), that is, the target position moves a distance of (x3-x2) in the x direction and a distance of (y3-y2) in the y direction, and based on the movement distance (z3-z2) of the image sensor in the z direction, the diameter of the diffuse circular spot on the imaging surface of the image sensor is determined. Among them, (x2-x1), (y2-y1), (z2-z1), (x3-x2), (y3-y2), and (z3-z2) are all used to represent the difference between the front and back coordinate values.

[0140] In this embodiment, the point spread function corresponding to each coordinate is determined based on the coordinates corresponding to the target position at each sampling moment and the shooting parameters. The point spread function not only takes into account the coordinates corresponding to the target position at each moment, namely the second jitter trajectory, but also takes into account the diffuse circular spot generated by the electronic device moving in a direction perpendicular to the image plane of the image sensor. It can more accurately characterize the blur kernel generated by the electronic device moving in the corresponding direction.

[0141] In some embodiments, determining a first blur kernel corresponding to the target position according to a point spread function includes calculating an average value of each point spread function corresponding to the target position and using the average value as the first blur kernel corresponding to the target position.

[0142] Optionally, the electronic device performs integration processing on the second jitter trajectory and the diffuse circular spot of the target position to obtain a first blur kernel corresponding to the target position. Figure 11As shown, the electronic device obtains the second movement trajectory and the diffuse circular spot from time t1, time t2, time t3 to time tn within the exposure time, integrates them over time, for example, first sums them and then takes the average, to obtain the first blur kernel corresponding to the target position.

[0143] In one embodiment, step 208 of deblurring the initial RAW image data according to the blur kernel to obtain a target RAW image includes: generating a blurred RAW image according to the blur kernel, and training a convolutional neural network based on the blurred RAW image to obtain a target convolutional neural network; and inputting the initial RAW image data into the target convolutional neural network to obtain the target RAW image.

[0144] Optionally, after obtaining the first blur kernel corresponding to each area in the image sensor, the corresponding area of ​​the clear image is blurred according to each first blur kernel, which can be to convolve the corresponding area of ​​the clear image with the corresponding first blur kernel to obtain a blurred image of the corresponding area, splice the blurred images of all areas to obtain a blurred image corresponding to the clear image, and train the convolutional neural network based on the image pair consisting of the clear image and the blurred image corresponding to the clear image. When the training conditions are met, the target convolutional neural network is obtained. The clear image can be an image that meets the image clarity requirements, and the specific method of obtaining the clear image is not specifically limited. The training conditions can be set according to the specific application scenario, for example, it can be to reach a preset number of training times or the loss function value is less than a preset value. In this embodiment, the initial RAW image data is input into the target convolutional neural network to obtain a clear target RAW image.

[0145] In one example, if Figure 12 As shown, the process of training a convolutional neural network based on a blur kernel includes: performing Gamma (gamma correction), CCM (Color Correction), AWB (Automatic White Balance) and other processing on the acquired RGB clear image to obtain a first image, performing convolution on the first image using a blur kernel, and performing noise addition and sampling processing on the convolution result to obtain a blurred RAW image; performing sampling processing on the first image to obtain a clear RAW image; forming an image pair of the blurred RAW image and the clear RAW image, and training the convolutional neural network to obtain a target convolutional neural network. The sampling processing refers to extracting pixels at specific positions from the first channel image to form pixels of the second channel, and the number of the first channel is greater than the number of the second channel. For example, pixels at specific positions are sampled from a three-channel JPG (Joint Photographic Experts Group, JPEG) format image to form a single-channel RAW image.

[0146] In this embodiment, a blurred image is generated based on the blur kernel, and a convolutional neural network is trained based on the clear image and the blurred image to obtain a target convolutional neural network. By inputting the blurred initial RAW image data into the target convolutional neural network, a target RAW image with higher clarity can be obtained conveniently and quickly.

[0147] In one embodiment, step 208 of deblurring the initial RAW image data according to the blur kernel to obtain the target RAW image includes: downsampling the initial RAW image data according to the color filter array of the image sensor to obtain a first RAW image corresponding to each color channel; deblurring the first RAW image using a sub-blur kernel at a position corresponding to the first RAW image to obtain a second RAW image; and determining the target RAW image based on the second RAW image.

[0148] The color filter array (CFA) is a mosaic coating above the image sensor. Since the image sensor can only sense the intensity of light and cannot distinguish the wavelength (color) of light, the image sensor needs to obtain the color information of the pixel through color filtering. Depending on the configuration type, CFA usually includes RGGB, RYYB, RGBW, etc. Among them, RGGB uses a red R, a blue B and two green G as a group of color filter arrays, also known as the Bayer array; RYYB is based on the RGGB array, and the two G filters are changed to yellow Y filters; RGBW is based on the RGGB array and one of the G filters in RGGB is replaced with W (white light). In this embodiment, according to the CFA of the image sensor, the initial RAW image data is downsampled to obtain the first RAW image corresponding to each color channel. For example, for RG i G r The initial RAW image in B format is divided into R channel, G i Channel, G r The pixels of the A channel and the B channel are extracted to obtain the corresponding first RAW image, such as Figure 13 shown.

[0149] Correspondingly, the blur kernel is also downsampled to obtain sub-blur kernels at corresponding positions in each first RAW image. These sub-blur kernels are then selected to perform deblurring on the first RAW image corresponding to each color channel, yielding a second RAW image. A sub-blur kernel can be understood as a portion of the blur kernel. Furthermore, the sub-blur kernel at the corresponding position is a portion of the first blur kernel for the corresponding region. For example, for the first pixel in the R channel, the sub-blur kernel at the corresponding position on the image sensor is selected for deblurring. This is equivalent to locking the deblurring process to the smallest unit pixel in each color channel. The target RAW image is then obtained based on the second RAW image corresponding to each color channel.

[0150] In this embodiment, the electronic device downsamples the initial RAW image into first RAW images corresponding to each color channel, and deblurs the pixels in each color channel using the sub-blur kernel at the corresponding position, to obtain a precisely deblurred second RAW image, thereby obtaining a clearer target RAW image.

[0151] In one embodiment, deblurring the first RAW image using a sub-blur kernel at a position corresponding to the first RAW image to obtain a second RAW image includes: deconvolving the first RAW image using the sub-blur kernel at a position corresponding to the first RAW image to obtain the second RAW image;

[0152] Determining a target RAW image according to the second RAW images includes: stitching the second RAW images together to obtain the target RAW image.

[0153] Optionally, deconvolution is performed on the first RAW image using a subblur kernel at a position corresponding to the first RAW image to obtain a second RAW image, and the second RAW images are concatenated using a color filter array to obtain the target RAW image. Deconvolution can use a deconvolution algorithm such as Wiener filtering, Patch-wise, or Richard-Lucy.

[0154] In one embodiment, the image processing flow is as follows Figure 14 As shown in the figure, the original RAW image data output by the image sensor is corrected for bad pixels, black level, and lens vignetting to obtain the initial RAW image data. The initial RAW image data is deblurred to obtain the target RAW image. The target RAW image is subjected to noise reduction and AWB processing and then converted to an RGB image through interpolation. The RGM is converted to a YUV image after CCM and Gamma correction. The YUV image is subjected to noise reduction, sharpening, and other processing to obtain the target image.

[0155] In this embodiment, by deblurring the RAW format data, the processing processes performed before deblurring are all linear processing, and non-linear processing processes such as noise reduction and sharpening are all performed after obtaining the target RAW data. This can make the restoration of images blurred due to shaking more accurate, that is, the obtained target image has higher clarity.

[0156] In one embodiment, Figure 15 As shown, the image processing process includes: opening the camera shutter for exposure, and collecting initial RAW image data and corresponding gyroscope data within the exposure time; establishing a corresponding coordinate system based on the positional relationship between the image sensor and the gyroscope, and using a unified standard to represent the gyroscope position and the image sensor position, and determining the affine matrix corresponding to the coordinate change of the image sensor when collecting the initial RAW image data based on the gyroscope data; determining the jitter trajectory of the image sensor based on each affine matrix, and determining the blur kernel of the image sensor within the exposure time based on the jitter trajectory of the image sensor; downsampling the initial RAW image data based on the color filter array of the image sensor to obtain a first RAW image corresponding to each color channel, using a sub-blur kernel at a position corresponding to the first RAW image to deblur the first RAW image to obtain a second RAW image, and obtaining a target RAW image based on each second RAW image; performing other image processing processes on the target RAW image to obtain a clear target image.

[0157] In this embodiment, by downsampling the initial RAW image data, each color channel is deblurred separately to achieve a better deblurring effect. At the same time, the influence of other nonlinear processing processes in the image processing process on the initial RAW image data can be avoided, and a target image with higher clarity can be obtained.

[0158] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0159] Based on the same inventive concept, embodiments of the present application also provide an image processing device for implementing the aforementioned image processing method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more of the following image processing device embodiments can be found in the above-described limitations on the image processing method and will not be further elaborated here.

[0160] In one embodiment, Figure 16 As shown, an image processing device is provided, including: an initial data acquisition module 1602, a jitter trajectory determination module 1604, a blur kernel determination module 1606 and a jitter blur processing module 1608, wherein:

[0161] Initial data acquisition module 1602, for acquiring initial RAW image data and corresponding gyroscope data;

[0162] a jitter trajectory determining module 1604 for determining a jitter trajectory of the image sensor when acquiring the initial RAW image data based on the positional relationship between the gyroscope and the image sensor and according to the gyroscope data;

[0163] A blur kernel determining module 1606 is configured to determine a blur kernel corresponding to the image sensor according to a jitter trajectory of the image sensor;

[0164] The dithering and blurring processing module 1608 is configured to perform a deblurring process on the initial RAW image data according to the blur kernel to obtain a target RAW image.

[0165] In one embodiment, the image sensor includes multiple regions; the jitter trajectory determination module 1604 is further used to: determine, for each region of the image sensor, a first jitter trajectory of each region when acquiring initial RAW image data based on gyroscope data; and determine, based on the first jitter trajectory corresponding to each region, a jitter trajectory of the image sensor when acquiring initial RAW image data.

[0166] In one embodiment, each region includes at least one target position; the jitter trajectory determination module 1604 is further used to: for each target position, determine an affine matrix for transforming the target position from a first coordinate to a second coordinate based on gyroscope data; wherein a first sampling moment corresponding to the first coordinate and a second sampling moment corresponding to the second coordinate are separated by a preset sampling duration; based on the affine matrix, determine a second jitter trajectory of the target position when collecting initial RAW image data; based on the second jitter trajectory corresponding to each target position on the region, determine a first jitter trajectory of the region when collecting initial RAW image data.

[0167] In one embodiment, the gyroscope data includes acceleration data and angular velocity data; the jitter trajectory determination module 1604 is further used to: determine the translation distance of the gyroscope from the first sampling moment to the second sampling moment based on the acceleration data; and determine the rotation angle of the gyroscope from the first sampling moment to the second sampling moment based on the angular velocity data; determine the translation amount of the target position from the first coordinate to the second coordinate based on the translation distance; and determine the rotation amount of the target position from the first coordinate to the second coordinate based on the rotation angle; determine the affine matrix of the target position from the first coordinate to the second coordinate based on the translation amount and the rotation amount. Determine the coordinates corresponding to the target position at each sampling moment based on the affine matrix corresponding to each preset sampling time length; the coordinates corresponding to each sampling moment are used to characterize the second jitter trajectory.

[0168] In one embodiment, the blur kernel determination module 1606 is further used to: determine the point spread function of the coordinates corresponding to the target position at each sampling moment according to the coordinates corresponding to the target position at each sampling moment; determine the first blur kernel corresponding to the target position according to the point spread function; obtain the second blur kernel corresponding to the area according to the first blur kernel corresponding to each of the target positions; and determine the blur kernel corresponding to the image sensor according to the second blur kernel corresponding to each of the areas.

[0169] In one embodiment, the blur kernel determination module 1606 is further configured to determine a point spread function of the coordinates corresponding to the target position at each sampling moment according to the coordinates corresponding to the target position at each sampling moment and the photographing parameters.

[0170] In one embodiment, the blur kernel determination module 1606 is further configured to calculate an average value of each point spread function corresponding to the target position, and use the average value as a first blur kernel corresponding to the target position.

[0171] In one embodiment, the jitter blur processing module 1608 is further used to: generate a blurred RAW image based on the blur kernel, and train a convolutional neural network based on the blurred RAW image to obtain a target convolutional neural network; input the initial RAW image data into the target convolutional neural network to obtain the target RAW image.

[0172] In one embodiment, the dither blur processing module 1608 is further configured to: downsample the initial RAW image data according to the color filter array of the image sensor to obtain a first RAW image corresponding to each color channel; deblur the first RAW image using a sub-blur kernel at a position corresponding to the first RAW image to obtain a second RAW image; and determine the target RAW image based on the second RAW image.

[0173] In one embodiment, the jitter blur processing module 1608 is further configured to: deconvolve the first RAW image using a sub-blur kernel at a position corresponding to the first RAW image to obtain the second RAW image; and stitch the second RAW images together to obtain the target RAW image.

[0174] Each module in the above-mentioned image processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0175] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 17 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image processing method is implemented.

[0176] Those skilled in the art will understand that Figure 17 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0177] The present application also provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, cause the processors to perform the steps of the image processing method.

[0178] An embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the image processing method.

[0179] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0180] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0181] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0182] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An image processing method, characterized in that: The image sensor includes a plurality of areas, each of the areas including at least one target location; including: Get the initial RAW image data and the corresponding gyroscope data; determining, based on the positional relationship between the gyroscope and the image sensor, a first jitter trajectory of each of the regions when acquiring the initial RAW image data according to the gyroscope data; determining, according to the first jitter trajectories corresponding to the respective regions, jitter trajectories of the image sensor when acquiring the initial RAW image data; Calculate the size of the diffuse circular spot based on the photographic parameters; Determining a point spread function of the coordinates corresponding to the target position at each sampling moment according to the coordinates corresponding to the target position at each sampling moment and the size of the diffuse circular spot; Calculating an average value of each point spread function corresponding to the target position, and using the average value as a first blur kernel corresponding to the target position; Obtaining a second blur kernel corresponding to the region according to the first blur kernel corresponding to each target position; determining a blur kernel corresponding to the image sensor according to the second blur kernel corresponding to each of the regions; The initial RAW image data is deblurred according to a blur kernel corresponding to the image sensor to obtain a target RAW image.

2. The method according to claim 1, characterized in that The determining, according to the gyroscope data, a first jitter trajectory of each of the regions when acquiring the initial RAW image data comprises: For each target position, determining an affine matrix for transforming the target position from a first coordinate to a second coordinate based on the gyroscope data; a first sampling moment corresponding to the first coordinate and a second sampling moment corresponding to the second coordinate are separated by a preset sampling time; determining, according to the affine matrix, a second jitter trajectory of the target position when acquiring the initial RAW image data; According to the second shaking trajectory corresponding to each target position on the area, a first shaking trajectory of the area when acquiring the initial RAW image data is determined.

3. The method according to claim 2, characterized in that The gyroscope data includes acceleration data and angular velocity data; and determining an affine matrix for transforming the target position from a first coordinate to a second coordinate based on the gyroscope data includes: Determining, based on the acceleration data, a translation distance of the gyroscope from the first sampling moment to the second sampling moment; and determining, based on the angular velocity data, a rotation angle of the gyroscope from the first sampling moment to the second sampling moment; determining a translation amount of the target position from the first coordinate to the second coordinate according to the translation distance; and determining a rotation amount of the target position from the first coordinate to the second coordinate according to the rotation angle; Determining an affine matrix for transforming the target position from the first coordinate to the second coordinate based on the translation amount and the rotation amount; The determining, according to the affine matrix, a second jitter trajectory of the target position when acquiring the initial RAW image data includes: The coordinates of the target position at each sampling moment are determined according to the affine matrix corresponding to each of the preset sampling durations; the coordinates corresponding to each sampling moment are used to represent the second jitter trajectory.

4. The method according to claim 1, wherein The deblurring the initial RAW image data according to the blur kernel corresponding to the image sensor to obtain a target RAW image includes: Generating a blurred RAW image according to a blur kernel corresponding to the image sensor, and training a convolutional neural network according to the blurred RAW image to obtain a target convolutional neural network; The initial RAW image data is input into the target convolutional neural network to obtain the target RAW image.

5. The method according to any one of claims 1 to 4, characterized in that The deblurring the initial RAW image data according to the blur kernel corresponding to the image sensor to obtain a target RAW image includes: downsampling the initial RAW image data according to a color filter array of the image sensor to obtain a first RAW image corresponding to each color channel; Deblurring the first RAW image using a sub-blur kernel at a position corresponding to the first RAW image to obtain a second RAW image; The target RAW image is determined according to the second RAW image.

6. The method according to claim 5, characterized in that Deblurring the first RAW image using a sub-blur kernel at a position corresponding to the first RAW image to obtain a second RAW image, including: Deconvolving the first RAW image using a sub-blur kernel at a position corresponding to the first RAW image to obtain a second RAW image; Determining the target RAW image according to the second RAW image includes: The second RAW images are stitched together to obtain the target RAW image.

7. The method according to claim 1, characterized in that The obtaining of initial RAW image data includes: Defect pixel correction, black level correction and lens vignetting correction are performed on the original RAW image data output by the image sensor to obtain initial RAW image data.

8. An image processing device, characterized in that: The image sensor includes a plurality of areas, each of the areas including at least one target location; including: An initial data acquisition module, used to acquire initial RAW image data and corresponding gyroscope data; a jitter trajectory determination module, configured to determine, based on a positional relationship between the gyroscope and the image sensor and according to the gyroscope data, a first jitter trajectory for each of the regions when acquiring the initial RAW image data; and determine, based on the first jitter trajectory corresponding to each region, a jitter trajectory of the image sensor when acquiring the initial RAW image data; a blur kernel determination module, configured to calculate the size of the diffuse circular spot based on the photographing parameters; determine the point spread function of the coordinates of the target position at each sampling moment based on the coordinates corresponding to the target position at each sampling moment and the size of the diffuse circular spot; calculate the average value of each point spread function corresponding to the target position, and use the average value as the first blur kernel corresponding to the target position; obtain the second blur kernel corresponding to the region based on the first blur kernel corresponding to each target position; and determine the blur kernel corresponding to the image sensor based on the second blur kernel corresponding to each region; The dither blur processing module is used to perform deblurring processing on the initial RAW image data according to a blur kernel corresponding to the image sensor to obtain a target RAW image.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images