Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The application relates to an image processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring initial RAW image data and corresponding gyroscope data; converting the initial RAW image data into initial YUV image data; determining a shaking track of an image sensor when collecting the initial RAW image data according to the gyroscope data based on a positional relationship between a gyroscope and the image sensor; determining a blur kernel corresponding to the image sensor according to the shaking track of the image sensor; and performing deblurring processing on Y channel data in the initial YUV image data according to the blur kernel to obtain a target YUV image. The method can reduce calculation redundancy, improve operation speed, and obtain a target YUV image with higher definition.

Description

Technical Field

[0001] This application relates to the field of imaging technology, and in particular to an image processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the development of technology, electronic devices such as mobile phones and tablets are increasingly used in people's work, study, and daily communication. These devices are typically equipped with cameras, enabling them to take photos and videos. Because camera components are small and their photosensitive area is also small, hand tremors during photo taking can lead to blurry image data, especially in low-light environments at night. To increase the amount of light entering the sensor, the exposure time is often longer, resulting in a more blurred image. This is because the range of hand tremors within the shutter speed exceeds the size of a single pixel on the sensor.

[0003] To remove image blur caused by hand tremors, a common method is to use a corresponding deblurring algorithm to deblur the image, but the computational redundancy of the corresponding algorithm is too large. Summary of the Invention

[0004] This application provides an image processing method, apparatus, electronic device, and computer-readable storage medium that can effectively remove image blur while reducing computational redundancy in deblurring algorithms.

[0005] Firstly, this application provides an image processing method. The method includes:

[0006] Acquire initial RAW image data and corresponding gyroscope data;

[0007] Convert the initial RAW image data into initial YUV image data;

[0008] Based on the positional relationship between the gyroscope and the image sensor, the jitter trajectory of the image sensor when acquiring the initial RAW image data is determined according to the gyroscope data.

[0009] The blur kernel corresponding to the image sensor is determined based on the jitter trajectory of the image sensor;

[0010] The Y channel data in the initial YUV image data is deblurred according to the blur kernel to obtain the target YUV image.

[0011] Secondly, this application also provides an image processing apparatus. The apparatus includes:

[0012] The initial data acquisition module is used to acquire initial RAW image data and corresponding gyroscope data;

[0013] An image format conversion module is used to convert the initial RAW image data into initial YUV image data;

[0014] A jitter trajectory determination module is used to determine the jitter trajectory of the image sensor when acquiring the initial RAW image data, based on the positional relationship between the gyroscope and the image sensor and according to the gyroscope data.

[0015] The blur kernel determination module is used to determine the blur kernel corresponding to the image sensor based on the jitter trajectory of the image sensor;

[0016] The jitter blurring module is used to deblur the Y channel data in the initial YUV image data according to the blur kernel to obtain the target YUV image.

[0017] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0018] Acquire initial RAW image data and corresponding gyroscope data;

[0019] Convert the initial RAW image data into initial YUV image data;

[0020] Based on the positional relationship between the gyroscope and the image sensor, the jitter trajectory of the image sensor when acquiring the initial RAW image data is determined according to the gyroscope data.

[0021] The blur kernel corresponding to the image sensor is determined based on the jitter trajectory of the image sensor;

[0022] The Y channel data in the initial YUV image data is deblurred according to the blur kernel to obtain the target YUV image.

[0023] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0024] Acquire initial RAW image data and corresponding gyroscope data;

[0025] Convert the initial RAW image data into initial YUV image data;

[0026] Based on the positional relationship between the gyroscope and the image sensor, the jitter trajectory of the image sensor when acquiring the initial RAW image data is determined according to the gyroscope data.

[0027] The blur kernel corresponding to the image sensor is determined based on the jitter trajectory of the image sensor;

[0028] The Y channel data in the initial YUV image data is deblurred according to the blur kernel to obtain the target YUV image.

[0029] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0030] Acquire initial RAW image data and corresponding gyroscope data;

[0031] Convert the initial RAW image data into initial YUV image data;

[0032] Based on the positional relationship between the gyroscope and the image sensor, the jitter trajectory of the image sensor when acquiring the initial RAW image data is determined according to the gyroscope data.

[0033] The blur kernel corresponding to the image sensor is determined based on the jitter trajectory of the image sensor;

[0034] The Y channel data in the initial YUV image data is deblurred according to the blur kernel to obtain the target YUV image.

[0035] The aforementioned image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product determine the jitter trajectory of the image sensor when acquiring initial RAW image data based on gyroscope data. The jitter trajectory accurately determines the blur kernel of the image sensor. Based on the blur kernel, the Y channel data in the initial YUV image data is deblurred. Since the degree of blurring caused by jitter is the same for all color channels, deblurring the Y channel data, which has more image details, can significantly reduce the computational load of the algorithm, avoid unnecessary computational redundancy, and improve the processing speed. Furthermore, a more accurate blur kernel can achieve better deblurring results, resulting in a clearer target YUV image. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is an application environment diagram of an image processing method in one embodiment;

[0038] Figure 2 This is a flowchart of an image processing method in one embodiment;

[0039] Figure 3 Here is a flowchart of step 206 in one embodiment;

[0040] Figure 4 Here is a flowchart of step 302 in one embodiment;

[0041] Figure 5 Here is a flowchart of step 402 in one embodiment;

[0042] Figure 6 This is a schematic diagram of a spatial rectangular coordinate system established with the gyroscope as the center in one embodiment;

[0043] Figure 7 This is a schematic diagram of a polar coordinate system established with the gyroscope as the center in one embodiment;

[0044] Figure 8 Here is a flowchart of step 208 in one embodiment;

[0045] Figure 9 This is a schematic diagram of an electronic device generating a diffuse circular spot by jittering in a direction perpendicular to the image sensor phase plane in one embodiment;

[0046] Figure 10 This is a schematic diagram illustrating the generation of diffuse circular spots at the target location in an image sensor during movement, as shown in one embodiment.

[0047] Figure 11 This is a schematic diagram illustrating the calculation method of the first fuzzy kernel corresponding to the target position in one embodiment;

[0048] Figure 12 This is a schematic diagram of the process of training a convolutional neural network based on a fuzzy kernel in one embodiment;

[0049] Figure 13 This is a flowchart illustrating the image processing method in another embodiment;

[0050] Figure 14 This is a flowchart illustrating the image processing method in another embodiment;

[0051] Figure 15 This is a structural block diagram of an image processing device in one embodiment;

[0052] Figure 16 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] The image processing method provided in this application is illustrated using an example running on an electronic device. Figure 1 As shown, electronic device 100 can be either a terminal or a server. The terminal can be, but is not limited to, various personal computers, cameras, scanners, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. The server can be implemented using a standalone server or a server cluster composed of multiple servers. It is understood that this method can also be applied to servers, and further to systems including both terminals and servers, and is implemented through the interaction between the terminal and the server.

[0055] In one embodiment, the electronic device includes components such as a camera and a gyroscope. The operating system of the electronic device may include Android, iOS (Input Output System), Windows Phone, Windows, etc., and can support the operation of various applications, such as map applications and game applications. A camera is an image data acquisition device used for taking pictures; it can be front-facing or rear-facing. A camera generally includes components such as a lens, base, infrared filter, image sensor, and circuit board. A gyroscope can measure angular velocity and acceleration and has high dynamic characteristics. It is understood that the gyroscope is not a sensor installed in the camera, but rather a general-purpose sensor in the electronic device, which can be simultaneously accessed by multiple applications for related business operations. For example, a camera application can use the gyroscope for image processing, a map application can use the gyroscope for navigation, and a game application can use the gyroscope for AR (Augmented Reality) 3D modeling, etc.

[0056] In one embodiment, such as Figure 2 As shown, an image processing method is provided, including the following steps 202 to 210.

[0057] Step 202: Obtain the initial RAW image data and the corresponding gyroscope data.

[0058] Initial RAW image data refers to the raw image files acquired by the image sensor that have not yet been processed, printed, or edited.

[0059] In this embodiment, when the electronic device receives a camera activation command, it can call the camera to acquire initial RAW image data within the exposure time. Simultaneously, it can call the gyroscope to measure corresponding gyroscope data, such as angular velocity and acceleration data. The camera activation command is an instruction to start the camera, which can be triggered by clicking the camera application or pressing a designated button. When the electronic device experiences shaking, the gyroscope can measure gyroscope data before and after the shaking or within a specified time period. Optionally, the electronic device can acquire the initial RAW image data and gyroscope data collected within the exposure time.

[0060] In an optional embodiment, the raw data output by the image sensor undergoes Bad Pixel Correction (BPC), Black Level Correction (BLC), Lens Shadow Correction (LSC), noise reduction, and Automatic White Balance (AWB) to obtain initial RAW image data. Bad Pixel Correction refers to correcting defective pixels caused by flaws in the array forming each light-collecting point on the image sensor, or errors during light signal conversion. Black Level refers to the signal level corresponding to zero image data. In practice, dark current exists in the image sensor circuit, resulting in a certain output voltage even without light input. This output value needs to be removed through correction to ensure the sensor circuit's output voltage is proportional to the input brightness. Due to the optical characteristics of the lens, the light intensity received at the edges of the sensor's image area is lower than that at the center, causing a brightness inconsistency between the center and the periphery. Therefore, LSC is needed to restore the energy attenuated through the lens to its pre-attenuation state.

[0061] Step 204: Convert the initial RAW image data into initial YUV image data.

[0062] Electronic devices can convert initial RAW image data into initial YUV image data. Optionally, electronic devices can convert initial RAW image data into RGB (red, green, blue) image data, and then convert the RGB image data into corresponding initial YUV image data. Here, Y represents luminance (or Luma), i.e., grayscale value, while U and V represent chrominance (or Chroma), describing the color and saturation of the image.

[0063] Step 206: Based on the positional relationship between the gyroscope and the image sensor, determine the jitter trajectory of the image sensor when acquiring initial RAW image data according to the gyroscope data.

[0064] Since gyroscopes and image sensors are usually installed in different locations on electronic devices, after acquiring gyroscope data, it is necessary to determine the jitter trajectory of the image sensor when acquiring initial RAW image data based on the positional relationship between the gyroscope and the image sensor.

[0065] Optionally, the gyroscope and image sensor can be established in the same coordinate system, and the coordinates of the gyroscope can be used to characterize the coordinates of the image sensor. For example, a spatial rectangular coordinate system and a corresponding polar coordinate system can be established with the gyroscope as the origin. By using the acquired gyroscope data or the polar coordinate values ​​of the gyroscope, and based on the conversion relationship between polar coordinates and spatial rectangular coordinates, the coordinate values ​​of the image sensor in the corresponding spatial rectangular coordinate system can be obtained.

[0066] Since gyroscope data changes continuously when electronic devices vibrate, a preset sampling duration can be set. The exposure duration for acquiring initial RAW image data can include multiple preset sampling durations. Gyroscope data is acquired once at each preset sampling duration interval. Each acquired gyroscope data corresponds to a polar coordinate of the gyroscope. Based on the conversion relationship between polar coordinates and spatial rectangular coordinates, the coordinates of the corresponding image sensor in the spatial rectangular coordinate system are obtained. Based on the coordinates of the image sensor corresponding to each acquired gyroscope data, the coordinates of the image sensor are connected in a preset order to obtain the vibration trajectory of the image sensor.

[0067] Optionally, the image sensor can be divided into multiple regions, each region including at least one target position. The jitter trajectory of each target position when acquiring initial RAW image data is determined based on gyroscope data, thereby obtaining the jitter trajectory of each region when acquiring initial RAW image data, and finally obtaining the jitter trajectory of the image sensor when acquiring initial RAW image data.

[0068] Step 208: Determine the blur kernel corresponding to the image sensor based on the jitter trajectory of the image sensor.

[0069] Image sensor jitter is typically characterized by the jitter trajectory at at least one target location on the image sensor. For a single target location's jitter trajectory, the coordinates corresponding to that trajectory can be directly used as the corresponding point spread function (PSF). However, for lenses with shallow depth of field, changes in the coordinates perpendicular to the image sensor's image plane will cause changes in the size of the blur spot in that direction. Therefore, based on the coordinates of the target location's jitter trajectory and the size of the corresponding blur spot, the PSF corresponding to each coordinate can be determined. Furthermore, the blur kernel corresponding to that target location can be determined based on the PSF.

[0070] In an image sensor, the intensity distribution of the diffracted image formed by a point light source (i.e., a star) on different cross sections before and after the image sensor image is formed after passing through an optical system is called the diffusion spot. A circular diffusion spot is a diffusion spot with a circular shape. The point spread function is used to characterize the light field distribution of the output image when the input is a point light source. The blur kernel is actually a matrix; convolving a sharp image with a blur kernel results in a blurred image, hence the name "blur kernel." The blur kernel is a type of convolution kernel; the essence of image convolution is matrix convolution. In other words, image blurring can be viewed as the process of convolving a sharp image with a blur kernel to obtain a blurred image.

[0071] Optionally, the electronic device can determine the blur kernel corresponding to the target position based on the coordinates of the jitter trajectory and the blur circle at the target position, and the blur kernel corresponding to the image sensor can be determined based on the blur kernel corresponding to each target position.

[0072] Step 210: Deblur the Y channel data in the initial YUV image data according to the blur kernel to obtain the target YUV image.

[0073] Understandably, the initial RAW image data is acquired by an electronic device and has jitter blur; correspondingly, the initial YUV image data also has jitter blur. According to the definition of a blur kernel, by deblurring the Y channel data in the initial YUV image data using a blur kernel, a clear target YUV image can be obtained.

[0074] Optionally, the Y channel data in the initial YUV image data can be deconvolved using a blur kernel to obtain the target YUV image.

[0075] Optionally, based on a blur kernel, an existing clear image can be convolved to obtain a corresponding blurred image. The convolutional neural network can be trained using the image pair composed of the clear and blurred images to obtain the target convolutional neural network. The Y channel data in the initial YUV image data can be input into the target convolutional neural network to obtain the target YUV image.

[0076] In the image processing method described above, the jitter trajectory of the image sensor during the acquisition of initial RAW image data is determined based on gyroscope data. This jitter trajectory allows for accurate determination of the image sensor's blur kernel. The Y-channel data in the initial YUV image data is then deblurred based on this kernel. Since the degree of blurring caused by jitter is the same across all color channels, deblurring the Y-channel data, which contains more image details, significantly reduces the computational load, avoids unnecessary computational redundancy, and improves processing speed. Furthermore, a more accurate blur kernel achieves better deblurring results, yielding a clearer target YUV image. Additionally, the electronic device can utilize its existing gyroscope to assist in deblurring the initial RAW image data, saving hardware costs compared to the traditional OIS (Optical Image Stabilizer) method.

[0077] In one embodiment, such as Figure 3 As shown, the image sensor includes multiple regions; step 206 determines the jitter trajectory of the image sensor when acquiring initial RAW image data based on gyroscope data, including steps 302 to 304.

[0078] Step 302: For each region of the image sensor, determine the first jitter trajectory of each region when acquiring the initial RAW image data based on the gyroscope data.

[0079] In this embodiment, the image sensor includes multiple regions, and the specific method of dividing the image sensor into multiple regions can be set as needed. For example, when the aspect ratio of the image sensor is 4:3, the image sensor can be evenly divided into 16*12 regions.

[0080] Understandably, the regions in the image sensor are used for spatially non-uniform deblurring. Each region has a different initial jitter trajectory when acquiring the initial RAW image data, and the corresponding blur kernel is also different. Each region in the initial RAW image data is deblurred using the corresponding blur kernel.

[0081] Optionally, each region of the image sensor includes at least one target location. A second jitter trajectory of the target location during the acquisition of initial RAW image data is determined based on gyroscope data. A first jitter trajectory of the region during the acquisition of initial RAW image data is determined based on the second jitter trajectory corresponding to each target location in the region.

[0082] Step 304: Determine the jitter trajectory of the image sensor when acquiring initial RAW image data based on the first jitter trajectory corresponding to each region.

[0083] Optionally, the first jitter trajectory corresponding to all regions in the image sensor is used as the jitter trajectory of the image sensor when acquiring initial RAW image data. For example, the image sensor includes regions A, B, and C. The first jitter trajectory corresponding to region A in the image sensor is used as the jitter trajectory of region A in the image sensor when acquiring initial RAW image data; the first jitter trajectory corresponding to region B in the image sensor is used as the jitter trajectory of region B in the image sensor when acquiring initial RAW image data; and the first jitter trajectory corresponding to region C in the image sensor is used as the jitter trajectory of region C in the image sensor when acquiring initial RAW image data.

[0084] In this embodiment, for each region of the image sensor, the first jitter trajectory of each region on the image sensor can be determined based on the gyroscope data. Based on the first jitter trajectory of each region, the blur kernel of the corresponding region can be determined, thereby enabling deblurring processing of the corresponding region in the initial RAW image data to obtain a clearer target RAW image for each region.

[0085] In one embodiment, such as Figure 4 As shown, each region includes at least one target location; for each region of the image sensor, step 302, which determines the first jitter trajectory of each region when acquiring the initial RAW image data based on gyroscope data, includes steps 402 to 406.

[0086] Step 402: For each target position, determine the affine matrix that transforms the target position from the first coordinate to the second coordinate based on the gyroscope data; wherein the interval between the first sampling time corresponding to the first coordinate and the second sampling time corresponding to the second coordinate is a preset sampling time.

[0087] In this embodiment, each region on the image sensor includes at least one target location. The specific location and number of target locations in each region can be set as needed. For example, a target location in a region can be the center point or an edge point of that region, etc.

[0088] Optionally, within the exposure duration, gyroscope data is collected at preset sampling intervals. Based on each collected gyroscope data, an affine matrix transforming the target position from the first coordinate to the second coordinate is determined. The first sampling time corresponding to the first coordinate and the second sampling time corresponding to the second coordinate are separated by a preset sampling interval. For example, the target position at time t1 corresponds to the first coordinate. Starting from time t1, a preset interval Δt is elapsed until time t2. Gyroscope data is collected at time t2, and the target position at time t2 corresponds to the second coordinate. When determining the second coordinate of the target position, the affine matrix transforming the first coordinate to the second coordinate can be determined based on the gyroscope data between times t1 and t2. The preset sampling interval can be set as needed.

[0089] Step 404: Determine the second jitter trajectory of the target position when acquiring the initial RAW image data based on the affine matrix.

[0090] Based on the gyroscope data collected each time, the affine matrix that transforms the target position from the first coordinate to the second coordinate is determined. Then, the second coordinate after the position change is obtained based on the first coordinate and the affine matrix. When the coordinates of the target position at all sampling times are obtained, the corresponding coordinates are connected according to the sampling times in ascending or descending order to obtain the second jitter trajectory of the target position when the initial RAW image data is collected.

[0091] Step 406: Determine the first jitter trajectory of the region when acquiring initial RAW image data based on the second jitter trajectory corresponding to each target position in the region.

[0092] Optionally, when the region includes a target location, the second jitter trajectory corresponding to that target location is used as the first jitter trajectory of that region when acquiring the initial RAW image data. When the region includes at least two target locations, the average value of the second jitter trajectories corresponding to each target location can be used as the first jitter trajectory of that region when acquiring the initial RAW image data.

[0093] It should be noted that if the region includes at least two target locations, other methods can be used to determine the first jitter trajectory of the corresponding region, such as weighted averaging or filtering, etc., which are not limited here. Understandably, when using filtering to determine the first jitter trajectory of a region, a second jitter trajectory corresponding to one target location can be selected from the second jitter trajectories corresponding to each target location according to preset filtering conditions, and used as the first jitter trajectory corresponding to that region.

[0094] In this embodiment, the first jitter trajectory of the region when acquiring the initial RAW image data is determined based on the second jitter trajectory corresponding to the target position in the region. This can more accurately determine the first jitter trajectory corresponding to each region in the image sensor. Based on the first jitter trajectory, the blur kernel corresponding to the region is determined, and finally, a clearer target RAW image of each region is obtained.

[0095] In one embodiment, such as Figure 5 As shown, the gyroscope data includes acceleration data and angular velocity data; step 402 determines the affine matrix that transforms the target position from the first coordinate to the second coordinate based on the gyroscope data, including steps 502 to 506.

[0096] Step 502: Based on the acceleration data, determine the translation distance of the gyroscope from the first sampling time to the second sampling time; and based on the angular velocity data, determine the rotation angle of the gyroscope from the first sampling time to the second sampling time.

[0097] Gyroscope data includes acceleration data and angular velocity data, with the acceleration data including linear acceleration data. By performing a double integration on the linear acceleration data, the translational distance of the gyroscope from the first sampling time to the second sampling time can be obtained, that is, the translational distance from the first coordinate to the second coordinate along each coordinate axis. By performing a double integration on the angular velocity, the rotational angle of the gyroscope from the first sampling time to the second sampling time can be obtained, that is, the rotational angle from the first coordinate to the second coordinate along each corresponding coordinate axis.

[0098] Step 504: Determine the translation amount of the target position from the first coordinate to the second coordinate based on the translation distance; and determine the rotation amount of the target position from the first coordinate to the second coordinate based on the rotation angle.

[0099] Optionally, the electronic device acquires the positional relationship between the gyroscope and the image sensor, and establishes a coordinate system centered on the gyroscope based on the positional relationship. When the translational distance and rotational angle of the gyroscope from the first sampling time to the second sampling time are determined, the translational amount and rotational amount of the target position in the image sensor from the first sampling time to the second sampling time can be determined accordingly.

[0100] Step 506: Determine the affine matrix that transforms the target position from the first coordinate to the second coordinate based on the translation and rotation amounts.

[0101] Optionally, a first translation coordinate is obtained based on the first coordinate and the translation amount, and a first rotation coordinate is obtained based on the first coordinate and the rotation amount; a second coordinate can be obtained based on the first translation coordinate and the first rotation coordinate. For example, the first translation coordinate and the second rotation coordinate can be added together to obtain the second coordinate.

[0102] Step 404, which determines the second jitter trajectory of the target position during the acquisition of the initial RAW image data based on the affine matrix, includes:

[0103] Based on the affine matrix corresponding to each preset sampling duration, the coordinates of the target position at each sampling time are determined; wherein, the coordinates corresponding to each sampling time are used to characterize the second jitter trajectory.

[0104] After determining the second coordinates corresponding to the affine matrix of the preset sampling duration, the electronic device can determine the coordinates of the target position at each sampling time according to the affine matrix corresponding to each preset sampling duration. The coordinates of the target position at each sampling time are the second jitter trajectory that represents the target position.

[0105] In one example, such as Figure 6 As shown, a spatial rectangular coordinate system is established with the gyroscope as the center. The coordinates of the gyroscope are (0,0,0). The center point of the image sensor is selected as the target position, and the initial coordinates of the target position are (x0,y0,z0). The coordinates of the edge points of the image sensor are (x0,y0,z0). i ,y i ,z i ).

[0106] Within the exposure time T, the electronic device collects gyroscope data once every preset sampling time Δt, with the number of samplings n = T / Δt. The gyroscope data collected each time is integrated twice over time to obtain the translation distance and rotation angle of the gyroscope in the X, Y, and Z directions, respectively. Then, the translation distance and rotation angle of the gyroscope in the X, Y, and Z directions are converted into the translation and rotation amounts of the image sensor in the X, Y, and Z directions, respectively.

[0107] The electronic device adds the translation amount of the target position in the X, Y and Z directions and the initial coordinates in the corresponding directions to obtain the coordinate values ​​of the target position in the X, Y and Z directions after a preset sampling time, as shown in the following formula (1).

[0108]

[0109] Where x0, y0, and z0 are the initial coordinates of the image sensor in the X, Y, and Z directions, respectively, and Δx, Δy, and Δz are the translations of the target position in the X, Y, and Z directions after a preset sampling time, respectively. These are the coordinates of the target location in the X, Y, and Z directions after a preset sampling time. It is the first translation coordinate.

[0110] For rotations along an axis, the rotation amount in each direction can be obtained through the mapping relationship between the polar coordinate system and the spatial rectangular coordinate system. For example... Figure 7 The diagram shows a polar coordinate system centered on the gyroscope, with the gyroscope's coordinates at (0, 0). The image sensor rotates Δω along the Z-axis. z If the initial coordinates of the target position in the polar coordinate system are (x0, y0, z0), the rotated coordinates are (x0′, y0′, z0′).

[0111] Within the preset sampling time, when the Z-axis rotates, its projection on the XY plane will undergo a position transformation. After polar coordinate transformation, the first rotation coordinate corresponding to the Z-axis rotation can be obtained, as shown in formula (2).

[0112]

[0113] Similarly, the first rotation coordinates can be obtained for rotations along the X and Y axes. For rotations along the X-axis, the corresponding first rotation coordinates are shown in formula (3):

[0114]

[0115] For rotation along the Y-axis, the corresponding first rotation coordinate is shown in formula (4):

[0116]

[0117] Where, Δω z ,Δω x and Δω y These are the rotation amounts on the Z-axis, X-axis, and Y-axis, respectively. It can be understood that if the target position rotates on an axis, the coordinates in the corresponding direction remain unchanged. For example, if it rotates on the Z-axis, the coordinates in the Z-direction remain unchanged.

[0118] In practical applications, the target position may rotate on at least one of the X, Y, and Z axes within the preset sampling time, or it may not rotate at all. In this case, only translation will occur, that is, only the amount of translation will be generated.

[0119] The first rotation coordinate is obtained by adding the rotation coordinates corresponding to the rotation direction when rotation occurs. For example, when the X-axis and Z-axis rotate, the first rotation coordinate is obtained by adding the rotation coordinates corresponding to the X-axis rotation direction. Corresponding to Z-axis rotation Add, rotate the X-axis accordingly Corresponding to Z-axis rotation Add, rotate the X-axis accordingly Corresponding to Z-axis rotation Add them together to get the first rotated coordinates.

[0120] Add the coordinate values ​​corresponding to the first translation coordinate and the first rotation coordinate to obtain the second coordinate.

[0121] It should be noted that formulas (1) to (4) above are all affine matrices for transforming the target position from the first coordinate to the second coordinate. The corresponding affine matrix is ​​selected according to the coordinate axis that is rotated in the actual application. For example, when only the Z-axis is rotated, the affine matrix is ​​formula (1) and formula (2).

[0122] Optionally, the electronic device combines the second coordinates obtained at preset sampling intervals on the target position from the image sensor to obtain a second jitter trajectory of the target position. For example, the initial coordinates of the target position are (x0, y0, z0), and the second coordinates obtained at preset sampling intervals are... Where n is a positive integer, then the second jitter trajectory at the target position is... The second jitter trajectory at other locations on the image sensor can also be calculated using the same method described above, and will not be repeated here.

[0123] In this embodiment, the electronic device determines the translation distance and rotation angle of the gyroscope within the preset sampling time based on the acceleration data and angular velocity data within the preset sampling time, thereby obtaining the translation and rotation amount corresponding to the target position on the image sensor. The corresponding affine matrix is ​​determined based on the translation and rotation amount. The second coordinate corresponding to the target position at each preset sampling time interval can be determined based on the affine matrix, thereby accurately determining the second jitter trajectory corresponding to the target position.

[0124] In one embodiment, such as Figure 8 As shown, step 208, which determines the blur kernel corresponding to the image sensor based on the jitter trajectory of the image sensor, includes steps 802 to 808.

[0125] Step 802: Determine the point spread function of the target position at each sampling time based on the coordinates of the target position at each sampling time.

[0126] Optionally, for jitter occurring in the image sensor's image plane direction, only the positional change of the corresponding jitter needs to be considered, without causing a change in the size of the blur spot caused by the jitter. That is, the coordinates of the target position at each sampling time can be used as the point spread function of the target position at each sampling time.

[0127] When jitter occurs in a direction perpendicular to the image sensor's image plane, the size of the blur spot caused by the jitter changes. Therefore, the point spread function of the target position at each sampling time is determined based on the target position's coordinates and the blur spot size. For example, if the image sensor's image plane is located in the X and Y directions, then the direction perpendicular to the image sensor's image plane is the Z direction.

[0128] Step 804: Determine the first fuzzy kernel corresponding to the target position based on the point spread function.

[0129] Optionally, the point spread function of the target position at each sampling time can be integrated to obtain the first blur kernel corresponding to the target position. Further, the point spread functions of the target position at each sampling time can be summed and their average value calculated to obtain the first blur kernel corresponding to the target position.

[0130] Step 806: Based on the first fuzzy kernel corresponding to each target location, obtain the second fuzzy kernel corresponding to the region.

[0131] Optionally, the electronic device integrates the first fuzzy kernels corresponding to each target location in the region to obtain a second fuzzy kernel for that region. Further, the electronic device can sum the first fuzzy kernels corresponding to each target location and calculate the average to obtain the second fuzzy kernel for that region. Alternatively, the electronic device can perform a weighted average of the first fuzzy kernels corresponding to each target location to obtain the second fuzzy kernel for that region; for example, the weight of each target location can be determined based on its specific location within the region. Or, the electronic device can use the median value of the first fuzzy kernels corresponding to each target location as the second fuzzy kernel for that region.

[0132] Step 808: Determine the blur kernel corresponding to the image sensor based on the second blur kernel corresponding to each region.

[0133] The image sensor includes multiple regions, each corresponding to a second blur kernel. After obtaining the second blur kernels corresponding to all regions, the blur kernel corresponding to the image sensor is obtained.

[0134] In this embodiment, the electronic device determines the corresponding first blur kernel by the point spread function corresponding to each coordinate of the target position, and then determines the corresponding second blur kernel of the region based on the first blur kernel corresponding to each target position in the region, thereby obtaining the blur kernel corresponding to the image sensor. This can accurately determine the blur kernel of the image sensor and avoid errors caused by blur kernel data at a single position.

[0135] In some embodiments, step 802, which determines the point spread function of the target position at each sampling time based on the coordinates of the target position at each sampling time, includes:

[0136] Based on the target location's coordinates at each sampling time and the image capture parameters, determine the point spread function of the target location's coordinates at each sampling time.

[0137] Optionally, when the electronic device jitters in a direction perpendicular to the image plane of the image sensor, the ideal imaging point at the target location will diffuse into a large circular spot. For example... Figure 9 As shown, if the electronic device experiences shaking in a direction perpendicular to the image sensor's image plane during shooting, causing the relative position of the subject to move from point A to point B, then a blurred circular spot will be generated on the image sensor's image plane. Although shaking in a direction perpendicular to the image sensor's image plane does not cause the blur kernel to move, it will change the size of the blur kernel.

[0138] The electronic device determines the point spread function (PFD) of the target location at each sampling time based on the target location's coordinates and the blur circle at each sampling time. Optionally, the electronic device acquires the target location's coordinates at each sampling time and the blur circle corresponding to each coordinate; it then adjusts each coordinate to the size of the corresponding blur circle to obtain the PFD for each coordinate. The size of the blur circle includes information such as its diameter, radius, perimeter, and area. The size of the blur circle reflects the size of the corresponding blur kernel. The size of the blur circle corresponding to each coordinate can be the same or different.

[0139] In this embodiment, the imaging parameters refer to the parameters used when capturing an image, including focal length, lens diameter, distance from the subject to the lens, image distance, etc. The electronic device can obtain the circle of confusion on the image sensor by determining the size of the circle of confusion.

[0140] In one example, combined Figure 9 The size of the diffuse circular spot can be calculated using the following formula (5).

[0141]

[0142] Where δ is the diameter of the diffuse circular spot, μ B It is the distance from point B to the lens, μ A Let f be the distance from point A to the lens, f be the focal length, D be the diameter of the lens, and Δz be the distance moved in the direction perpendicular to the image plane of the image sensor. In this context, μ represents the focal length of a fixed-focus lens in an electronic device. A Given the parameters, and obtaining the movement distance Δz in the direction perpendicular to the image plane of the image sensor, μ can be calculated. B By obtaining the parameters, the focal length f, and the lens diameter D, the diameter of the blurred circle can be calculated.

[0143] For each target location in each region of the image sensor, the corresponding point spread function can be obtained using the method described above. For example... Figure 10 As shown, at time t1 within the exposure time, the coordinates of the target position in a certain region are (x1, y1, z1), and the diameter of the resulting blur circle is 1 pixel. At time t2, the coordinates of the target position are (x2, y2, z2), which means the target position moves (x2-x1) in the x direction and (y2-y1) in the y direction. Based on the distance the image sensor moves in the z direction (z2-z1), the diameter of the blur circle on the image sensor's imaging surface is determined. At time t3, the coordinates of the target position are (x3, y3, z3), which means the target position moves (x3-x2) in the x direction and (y3-y2) in the y direction. Based on the distance the image sensor moves in the z direction (z3-z2), the diameter of the blur circle on the image sensor's imaging surface is determined. Among them, (x2-x1), (y2-y1), (z2-z1), (x3-x2), (y3-y2), and (z3-z2) are all used to represent the difference between the coordinate values ​​before and after.

[0144] In this embodiment, the point spread function corresponding to each coordinate is determined based on the coordinates of the target position at each sampling time and the imaging parameters. The point spread function not only considers the coordinates of the target position at each time, i.e. the second jitter trajectory, but also considers the diffuse circular spots generated by the movement of the electronic device in the direction perpendicular to the image plane of the image sensor. It can more accurately characterize the blur kernel generated by the movement of the electronic device in the corresponding direction.

[0145] In some embodiments, determining the first fuzzy kernel corresponding to the target location based on the point spread function includes: calculating the average value of each point spread function corresponding to the target location, and using the average value as the first fuzzy kernel corresponding to the target location.

[0146] Optionally, the electronic device integrates the second jitter trajectory and the diffuse circular spot at the target location to obtain the first blur kernel corresponding to the target location. In one example, such as Figure 11 As shown, the electronic device acquires the second moving trajectory and diffuse circular spot at times t1, t2, t3 up to tn within the exposure time, integrates them over time, for example, by summing and then taking the average value, to obtain the first blur kernel corresponding to the target position.

[0147] In one embodiment, step 210, which involves deblurring the Y-channel data in the initial YUV image data according to a blur kernel to obtain a target YUV image, includes: generating a blurred Y-channel image based on the blur kernel, training a convolutional neural network based on the blurred Y-channel image to obtain a target convolutional neural network; and inputting the Y-channel data in the initial YUV image data into the target convolutional neural network to obtain the target YUV image.

[0148] Optionally, after obtaining the first blur kernel corresponding to each region in the image sensor, the corresponding region of the clear image is blurred according to each first blur kernel. This can be done by convolving the corresponding region of the clear image with the corresponding first blur kernel to obtain the blurred image of the corresponding region, and then stitching the blurred images of all regions together to obtain the blurred image corresponding to the clear image.

[0149] A convolutional neural network is trained using image pairs consisting of the Y-channel images from a clear image and the Y-channel images from a corresponding blurred image. When preset training conditions are met, a target convolutional neural network is obtained. The clear image can be any image that meets the required image sharpness; the specific method for obtaining the clear image is not limited. Training conditions can be set according to the specific application scenario, such as reaching a preset number of training iterations or a loss function value being less than a preset value. In this embodiment, the Y-channel data from the initial YUV image data is input into the target convolutional neural network to obtain the corresponding deblurred Y-channel data. A clear target YUV image can then be obtained based on the deblurred Y-channel data.

[0150] In one example, a flowchart illustrating the training process of a convolutional neural network based on a fuzzy kernel is shown below. Figure 12 As shown, the acquired clear RGB image is processed by Gamma (Gamma correction), CCM (Color Correction), and AWB (Automatic White Balance) to obtain the first image. The first image is then convolved using a blur kernel, and the convolution result is noise-enhanced to obtain a blurred RAW image. The first image is then used as the clear RAW image. The blurred RAW image and the clear RAW image are then processed again by Gamma, CCM, and AWB to convert them into corresponding blurred RGB images and clear RGB images. The blurred RGB images and the clear RGB images are then converted into corresponding blurred YUV images and clear YUV images. Based on the image pairs formed by the blurred Y channel images in the blurred YUV images and the clear Y channel images in the clear YUV images, the convolutional neural network is trained. After meeting the preset training conditions, the target convolutional neural network is obtained.

[0151] In this embodiment, a blurred Y-channel image is generated based on a blur kernel. A convolutional neural network is trained based on the clear Y-channel image in the clear image and the blurred Y-channel image in the blurred image to obtain a target convolutional neural network. By inputting the Y-channel data from the blurred initial YUV image data into the target convolutional neural network, a target YUV image with high clarity can be obtained conveniently and quickly.

[0152] In one embodiment, step 210, which involves deblurring the Y channel data in the initial YUV image data according to a blur kernel to obtain a target YUV image, includes: using a blur kernel corresponding to the position of the Y channel data to deconvolve the Y channel data to obtain a first image; and stitching the first images together to obtain the target YUV image.

[0153] Optionally, based on the region in the image sensor corresponding to the Y-channel data, a second blur kernel corresponding to that region is used to deconvolve the corresponding Y-channel data to obtain the first image. For example, if the Y-channel data corresponds to the first region in the image sensor, then the second blur kernel corresponding to the first region is used to deconvolve the Y-channel data to obtain the first image. The deconvolution can utilize Wiener filtering, patch-wise, Richard-Lucy, or other deconvolution algorithms.

[0154] In one example, the corresponding Y-channel data is deconvolved using the second blur kernel corresponding to the corresponding region, while the U and V channels corresponding to the Y channel are left unprocessed. Then, based on the deconvolved Y-channel data and the U and V channel data from the initial YUV image data, a first image is obtained. The first images corresponding to each region are then stitched together to obtain the target YUV image.

[0155] In this embodiment, a blur kernel corresponding to the position of the Y channel data is used to deconvolve the Y channel to obtain the first image, and then the target YUV image is obtained. This can make the Y channel data after deconvolution clearer, thereby obtaining a clearer target YUV image.

[0156] In one embodiment, the image processing method flows as follows: Figure 13 As shown, the raw RAW image data output by the image sensor undergoes bad pixel correction, black level correction, lens vignetting correction, noise reduction, and AWB to obtain the initial RAW image data. The initial RAW image data is then converted to initial RGB image data, for example, through interpolation. The initial RGB image data is then converted to initial YUV image data after gamma correction and CCM processing. The Y channel data in the initial YUV image data is then deblurred to obtain the target YUV image. Finally, the target YUV image is processed by noise reduction, sharpening, and other methods to obtain the target image.

[0157] In this embodiment, by deblurring the Y channel data in the initial YUV image data, all processing steps before deblurring are linear. Non-linear processing steps such as noise reduction and sharpening are performed after obtaining the target YUV data. This makes the recovery of the image blurred by jitter more accurate, that is, the clarity of the target image is higher. At the same time, it reduces the computational load of the deblurring algorithm, avoids multi-channel computational redundancy, and improves the operation speed.

[0158] In one embodiment, the image processing method flows as follows: Figure 14 As shown, the camera shutter is opened for exposure, and initial RAW image data and corresponding gyroscope data are acquired within the exposure time. Based on the positional relationship between the image sensor and the gyroscope, a corresponding coordinate system is established, using a unified standard to represent the gyroscope and image sensor positions. The affine matrix corresponding to the coordinate changes of the image sensor during the acquisition of the initial RAW image data is determined based on the gyroscope data. The image sensor's jitter trajectory is determined based on each affine matrix, and the blur kernel for the image sensor within the exposure time is determined based on the image sensor's jitter trajectory. After processing the initial RAW image data through the first part of the image processing flow, initial YUV image data is obtained. The Y channel data in the initial YUV image data is deblurred according to the blur kernel to obtain the target YUV image. Further, the target YUV image undergoes the second part of the image processing flow to obtain a clear target image. The first part of the image processing flow may include, for example, bad pixel correction, black level correction, lens vignetting correction, noise reduction, and format conversion. The second part of the image processing flow may include, for example, noise reduction and sharpening.

[0159] In this embodiment, by deblurring the Y channel data with more image details, the computational load of the algorithm can be greatly reduced and the operation speed can be improved. At the same time, a better deblurring effect can be achieved through an accurate blur kernel, resulting in a clearer target YUV image. This removes image blur while reducing computational redundancy in the blur algorithm.

[0160] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0161] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0162] In one embodiment, such as Figure 15 As shown, an image processing apparatus is provided, including: an initial data acquisition module 1502, an image format conversion module 1504, a jitter trajectory determination module 1506, a blur kernel determination module 1508, and a jitter blur processing module 1510, wherein:

[0163] The initial data acquisition module 1502 is used to acquire initial RAW image data and corresponding gyroscope data;

[0164] Image format conversion module 1504 is used to convert the initial RAW image data into initial YUV image data;

[0165] The jitter trajectory determination module 1506 is used to determine the jitter trajectory of the image sensor when acquiring the initial RAW image data based on the positional relationship between the gyroscope and the image sensor and according to the gyroscope data.

[0166] The blur kernel determination module 1508 is used to determine the blur kernel corresponding to the image sensor based on the jitter trajectory of the image sensor;

[0167] The jitter blurring module 1510 is used to perform deblurring processing on the Y channel data in the initial YUV image data according to the blur kernel to obtain the target YUV image.

[0168] In one embodiment, the image sensor includes multiple regions; the jitter trajectory determination module 1506 is further configured to: determine a first jitter trajectory for each region of the image sensor when acquiring initial RAW image data based on gyroscope data; and determine the jitter trajectory of the image sensor when acquiring initial RAW image data based on the first jitter trajectory corresponding to each region.

[0169] In one embodiment, each region includes at least one target location; the jitter trajectory determination module 1506 is further configured to: for each target location, determine an affine matrix transforming the target location from the first coordinate to the second coordinate based on gyroscope data; wherein the first sampling time corresponding to the first coordinate and the second sampling time corresponding to the second coordinate are spaced apart by a preset sampling duration; determine the second jitter trajectory of the target location when acquiring initial RAW image data based on the affine matrix; and determine the first jitter trajectory of the region when acquiring initial RAW image data based on the second jitter trajectories corresponding to each target location in the region.

[0170] In one embodiment, the gyroscope data includes acceleration data and angular velocity data; the jitter trajectory determination module 1506 is further configured to: determine the translation distance of the gyroscope from the first sampling time to the second sampling time based on the acceleration data; and determine the rotation angle of the gyroscope from the first sampling time to the second sampling time based on the angular velocity data; determine the translation amount of the target position from the first coordinate to the second coordinate based on the translation distance; and determine the rotation amount of the target position from the first coordinate to the second coordinate based on the rotation angle; and determine the affine matrix of the target position from the first coordinate to the second coordinate based on the translation amount and the rotation amount. Based on the affine matrix corresponding to each preset sampling duration, the coordinates of the target position at each sampling time are determined; the coordinates corresponding to each sampling time are used to characterize the second jitter trajectory.

[0171] In one embodiment, the blur kernel determination module 1508 is further configured to: determine the point spread function of the target position at each sampling time based on the coordinates of the target position at each sampling time; determine the first blur kernel corresponding to the target position based on the point spread function; obtain the second blur kernel corresponding to the region based on the first blur kernel corresponding to each target position; and determine the blur kernel corresponding to the image sensor based on the second blur kernel corresponding to each region.

[0172] In one embodiment, the fuzzy kernel determination module 1508 is further configured to: determine the point spread function of the target position at each sampling time based on the coordinates of the target position at each sampling time and the image capture parameters.

[0173] In one embodiment, the fuzzy kernel determination module 1508 is further configured to: calculate the average value of the diffusion function of each point corresponding to the target position, and use the average value as the first fuzzy kernel corresponding to the target position.

[0174] In one embodiment, the jitter blur processing module 1510 is further configured to: generate a blurred Y-channel image based on the blur kernel, and train a convolutional neural network based on the blurred Y-channel image to obtain a target convolutional neural network; input the Y-channel data in the initial YUV image data into the target convolutional neural network to obtain the target YUV image.

[0175] In one embodiment, the jitter blur processing module 1510 is further configured to: deconvolve the Y channel data using a blur kernel corresponding to the position of the Y channel data to obtain a first image; and stitch the first images together to obtain the target YUV image.

[0176] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0177] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 16 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an image processing method.

[0178] Those skilled in the art will understand that Figure 16 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0179] This application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, cause the processors to perform the steps of an image processing method.

[0180] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform an image processing method.

[0181] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0182] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0183] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0184] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, The image sensor includes multiple regions, each region including at least one target location; including: Acquire initial RAW image data and corresponding gyroscope data; Convert the initial RAW image data into initial YUV image data; Based on the positional relationship between the gyroscope and the image sensor, the first jitter trajectory of each region during the acquisition of the initial RAW image data is determined according to the gyroscope data; Based on the first jitter trajectory corresponding to each of the regions, the jitter trajectory of the image sensor when acquiring the initial RAW image data is determined; Based on the coordinates of the target location at each sampling time and the size of the diffused circular spot, the point spread function of the target location at each sampling time is determined. Calculate the average value of the spread function of each point corresponding to the target location, and use the average value as the first blur kernel corresponding to the target location; Based on the first fuzzy kernel corresponding to each of the target locations, the second fuzzy kernel corresponding to the region is obtained; The blur kernel corresponding to the image sensor is determined based on the second blur kernel corresponding to each of the regions. The Y channel data in the initial YUV image data is deblurred according to the blur kernel corresponding to the image sensor to obtain the target YUV image.

2. The method according to claim 1, characterized in that, The step of determining the first jitter trajectory of each region during the acquisition of the initial RAW image data based on the gyroscope data includes: For each target position, an affine matrix is ​​determined based on the gyroscope data to transform the target position from the first coordinate to the second coordinate; the first sampling time corresponding to the first coordinate and the second sampling time corresponding to the second coordinate are spaced apart by a preset sampling time. Based on the affine matrix, determine the second jitter trajectory of the target position when acquiring the initial RAW image data; Based on the second jitter trajectory corresponding to each target position in the region, the first jitter trajectory of the region during the acquisition of the initial RAW image data is determined.

3. The method according to claim 2, characterized in that, The gyroscope data includes acceleration data and angular velocity data; determining the affine matrix transforming the target position from the first coordinate to the second coordinate based on the gyroscope data includes: Based on the acceleration data, determine the translational distance of the gyroscope from the first sampling time to the second sampling time; and based on the angular velocity data, determine the rotational angle of the gyroscope from the first sampling time to the second sampling time. Based on the translation distance, determine the amount of translation of the target position from the first coordinate to the second coordinate; and based on the rotation angle, determine the amount of rotation of the target position from the first coordinate to the second coordinate. Based on the translation and rotation amounts, determine the affine matrix that transforms the target position from the first coordinate to the second coordinate; Determining the second jitter trajectory of the target position during the acquisition of the initial RAW image data based on the affine matrix includes: Based on the affine matrix corresponding to each preset sampling duration, the coordinates of the target position at each sampling time are determined; the coordinates corresponding to each sampling time are used to characterize the second jitter trajectory.

4. The method according to claim 1, characterized in that, The step of deblurring the Y channel data in the initial YUV image data according to the blur kernel corresponding to the image sensor to obtain the target YUV image includes: A blurred Y-channel image is generated based on the blur kernel corresponding to the image sensor, and a convolutional neural network is trained based on the blurred Y-channel image to obtain the target convolutional neural network. The Y channel data from the initial YUV image data is input into the target convolutional neural network to obtain the target YUV image.

5. The method according to claim 1, characterized in that, The step of deblurring the Y channel data in the initial YUV image data according to the blur kernel corresponding to the image sensor to obtain the target YUV image includes: The Y-channel data is deconvolved using a blur kernel corresponding to the position of the Y-channel data to obtain the first image; The individual first images are stitched together to obtain the target YUV image.

6. The method according to claim 1, characterized in that, The acquisition of initial RAW image data includes: The raw RAW image data output by the image sensor is subjected to bad pixel correction, black level correction, lens vignetting correction, noise reduction, and automatic white balance correction to obtain the initial RAW image data.

7. An image processing apparatus, characterized in that, The image sensor includes multiple regions, each region including at least one target location; including: The initial data acquisition module is used to acquire initial RAW image data and corresponding gyroscope data; An image format conversion module is used to convert the initial RAW image data into initial YUV image data; The jitter trajectory determination module is used to determine the first jitter trajectory of each region when acquiring the initial RAW image data based on the positional relationship between the gyroscope and the image sensor, according to the gyroscope data; and to determine the jitter trajectory of the image sensor when acquiring the initial RAW image data based on the first jitter trajectory corresponding to each region. The blur kernel determination module is used to determine the point spread function of the target position at each sampling time based on the coordinates of the target position at each sampling time and the size of the blur circle; calculate the average value of each point spread function corresponding to the target position, and use the average value as the first blur kernel corresponding to the target position; obtain the second blur kernel corresponding to the region based on the first blur kernel corresponding to each target position; and determine the blur kernel corresponding to the image sensor based on the second blur kernel corresponding to each region. The jitter blurring module is used to deblur the Y channel data in the initial YUV image data according to the blur kernel corresponding to the image sensor to obtain the target YUV image.

8. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images