Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The application relates to an image processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring an initial image and corresponding gyroscope data; determining a target blur kernel corresponding to an image sensor according to the gyroscope data based on the positional relationship between the gyroscope and the image sensor; acquiring phase difference data of each pixel in the initial image, and determining depth information corresponding to the initial image according to the phase difference data; performing scale transformation on the target blur kernel according to the depth information to obtain a target scale blur kernel; and performing deblurring processing on the initial image according to the depth information and the target scale blur kernel to obtain a target image. The method uses corresponding target scale blur kernels to perform deblurring processing on initial images corresponding to different depth information, so that image restoration is more accurate, and the obtained target image is clearer.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to an image processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] When taking pictures, hand tremors can cause the images to become blurry. To make blurry images clear, they need to be deblurred.

[0003] In traditional techniques, to remove image blur caused by hand tremors, a common method is to use corresponding deblurring algorithms to process the blurred image. However, in real-world scenarios, the objects being photographed are three-dimensional. Due to the depth-of-field limitations of the camera lens, when photographing multiple objects at different depths, the resulting blur kernels are often different. Therefore, using traditional deblurring algorithms to process blurred images results in relatively low accuracy. Summary of the Invention

[0004] This application provides an image processing method, apparatus, electronic device, and computer-readable storage medium that can achieve accurate deblurring.

[0005] Firstly, this application provides an image processing method. The method includes:

[0006] Acquire the initial image and the corresponding gyroscope data;

[0007] Based on the positional relationship between the gyroscope and the image sensor, the target blur kernel corresponding to the image sensor is determined according to the gyroscope data;

[0008] The phase difference data of each pixel in the initial image is obtained, and the depth information corresponding to the initial image is determined based on the phase difference data;

[0009] The target fuzzy kernel is scaled based on the depth information to obtain the target fuzzy kernel.

[0010] Based on the depth information and the target scale blur kernel, the initial image is deblurred to obtain the target image.

[0011] Secondly, this application also provides an image processing apparatus. The apparatus includes:

[0012] The initial data acquisition module is used to acquire the initial image and the corresponding gyroscope data;

[0013] The fuzzy kernel determination module is used to determine the target fuzzy kernel corresponding to the image sensor based on the positional relationship between the gyroscope and the image sensor, according to the gyroscope data.

[0014] The depth information determination module is used to acquire the phase difference data of each pixel in the initial image, and determine the depth information corresponding to the initial image based on the phase difference data;

[0015] The fuzzy kernel scaling module is used to perform scaling transformation on the target fuzzy kernel based on the depth information to obtain the target scale fuzzy kernel;

[0016] The deblurring module is used to deblur the initial image based on the depth information and the target scale blur kernel to obtain the target image.

[0017] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0018] Acquire the initial image and the corresponding gyroscope data;

[0019] Based on the positional relationship between the gyroscope and the image sensor, the target blur kernel corresponding to the image sensor is determined according to the gyroscope data;

[0020] The phase difference data of each pixel in the initial image is obtained, and the depth information corresponding to the initial image is determined based on the phase difference data;

[0021] The target fuzzy kernel is scaled based on the depth information to obtain the target fuzzy kernel.

[0022] Based on the depth information and the target scale blur kernel, the initial image is deblurred to obtain the target image.

[0023] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0024] Acquire the initial image and the corresponding gyroscope data;

[0025] Based on the positional relationship between the gyroscope and the image sensor, the target blur kernel corresponding to the image sensor is determined according to the gyroscope data;

[0026] The phase difference data of each pixel in the initial image is obtained, and the depth information corresponding to the initial image is determined based on the phase difference data;

[0027] The target fuzzy kernel is scaled based on the depth information to obtain the target fuzzy kernel.

[0028] Based on the depth information and the target scale blur kernel, the initial image is deblurred to obtain the target image.

[0029] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0030] Acquire the initial image and the corresponding gyroscope data;

[0031] Based on the positional relationship between the gyroscope and the image sensor, the target blur kernel corresponding to the image sensor is determined according to the gyroscope data;

[0032] The phase difference data of each pixel in the initial image is obtained, and the depth information corresponding to the initial image is determined based on the phase difference data;

[0033] The target fuzzy kernel is scaled based on the depth information to obtain the target fuzzy kernel.

[0034] Based on the depth information and the target scale blur kernel, the initial image is deblurred to obtain the target image.

[0035] The aforementioned image processing methods, apparatuses, electronic devices, readable storage media, and computer program products acquire an initial image and corresponding gyroscope data. Based on the positional relationship between the gyroscope and the image sensor, they determine the target blur kernel corresponding to the image sensor using the gyroscope data. They then determine the depth information of the initial image based on the phase difference data of each pixel in the initial image. Based on the depth information, they scale-transform the target blur kernel to obtain a target scale blur kernel. Finally, they deblur the initial image using the depth information and the target scale blur kernel to obtain the target image. This application, by scaling the target blur kernel based on the depth information, can obtain a corresponding target scale blur kernel. The blur kernel for the out-of-focus area can be determined using the blur kernel for the focused area. Different blur kernels are used for deblurring the focused and out-of-focus areas. Initial images corresponding to different depth information are deblurred using the corresponding target scale blur kernel. This better matches actual jitter conditions, resulting in more accurate image recovery and a clearer target image. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart of an image processing method in one embodiment;

[0038] Figure 2A This is a schematic diagram of the imaging of an object in one embodiment;

[0039] Figure 2B This is a schematic diagram of the imaging process when the image is in focus, as shown in one embodiment.

[0040] Figure 3 This is a schematic diagram of the depth information corresponding to the initial image in one embodiment;

[0041] Figure 4 This is a schematic diagram illustrating the target scale fuzzy kernel obtained by scaling the target fuzzy kernel in one embodiment.

[0042] Figure 5A This is a schematic diagram of the mask corresponding to a 0.5µm phase difference gradient value in one embodiment;

[0043] Figure 5B This is a schematic diagram of the mask corresponding to a 1µm phase difference gradient value in one embodiment;

[0044] Figure 6 Here is a flowchart of step 104 in one embodiment;

[0045] Figure 7 Here is a flowchart of step 602 in one embodiment;

[0046] Figure 8 This is a schematic diagram of a spatial rectangular coordinate system established with the gyroscope as the center in one embodiment;

[0047] Figure 9 This is a schematic diagram of a polar coordinate system established with the gyroscope as the center in one embodiment;

[0048] Figure 10 This is a schematic diagram of a preset position generating a diffuse circular spot during movement in one embodiment;

[0049] Figure 11 A flowchart of an image processing method in another embodiment;

[0050] Figure 12 This is a structural block diagram of an image processing device in one embodiment;

[0051] Figure 13 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] When taking pictures with a camera, in order to increase the amount of light entering the camera, if the exposure time is too long, hand tremors will cause the acquired image to become blurry. This is because the range of hand tremors during the shutter speed exceeds the size of a single pixel on the sensor.

[0054] To address blurring caused by hand tremors, a traditional approach involves incorporating an OIS (Optical Image Stabilizer) to control the movement of the lens or image sensor, thus compensating for hand tremors. This approach is costly, and as sensors become larger, the size and weight of the lens and corresponding OIS also increase. Another approach is to use deblurring algorithms to eliminate image blur by identifying the blur kernel caused by hand tremors, thereby obtaining a clear image. However, the resulting image has lower sharpness. To address these issues, this application proposes an image processing method.

[0055] The image processing method provided in this application is illustrated using an electronic device as an example. The electronic device can be a terminal or a server. The terminal can be, but is not limited to, various personal computers, cameras, scanners, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. The server can be implemented using a standalone server or a server cluster composed of multiple servers. It is understood that this method can also be applied to servers, and can also be applied to systems including terminals and servers, and implemented through the interaction between the terminal and the server.

[0056] In one embodiment, such as Figure 1 As shown, an image processing method is provided, which includes the following steps 102 to 110.

[0057] Step 102: Obtain the initial image and the corresponding gyroscope data.

[0058] The electronic device acquires the initial image from the image sensor and the corresponding gyroscope data from the gyroscope. The initial image can be any of the following: RAW image, YUV image, HSV image (Hue, Saturation, Value), or RGB image (Red, Green, Blue). In a YUV image, the Y component represents luminance (Luminance), and the U and V components represent chrominance (Chrominance). The initial image is often blurred due to camera shake; therefore, it needs to be deblurred to obtain a clearer target image.

[0059] In one optional embodiment, the electronic device includes components such as a camera and a gyroscope. A camera is an image data acquisition device used for taking pictures; it can be front-facing or rear-facing. A camera typically includes components such as a lens, base, infrared filter, image sensor, and circuit board. A gyroscope measures angular velocity and acceleration and has high dynamic characteristics. The camera and gyroscope are usually installed in different locations on the electronic device; therefore, when the electronic device vibrates, the gyroscope's vibration trajectory is not exactly the same as the vibration trajectory of the camera's image sensor.

[0060] Optionally, when the electronic device receives a camera activation command, it can invoke the camera to acquire an initial image via the image sensor within the exposure time. Simultaneously, it can invoke the gyroscope to measure gyroscope data within the exposure time; this gyroscope data could be, for example, angular acceleration data and linear acceleration data. The camera activation command refers to an instruction to start the camera, which can be triggered by clicking the camera application or pressing a designated button. Understandably, the gyroscope can acquire gyroscope data within a specified time period as needed.

[0061] Step 104: Based on the positional relationship between the gyroscope and the image sensor, determine the target blur kernel corresponding to the image sensor according to the gyroscope data.

[0062] Optionally, based on the positional relationship between the gyroscope and the image sensor, the electronic device determines the jitter trajectory of the image sensor when acquiring the initial image based on the gyroscope data, and then determines the target blur kernel corresponding to the image sensor based on the jitter trajectory.

[0063] Optionally, the gyroscope and image sensor can be established in the same coordinate system, and the coordinates of the image sensor can be represented by the coordinates of the gyroscope. For example, a spatial rectangular coordinate system and a corresponding polar coordinate system can be established with the gyroscope as the origin. The coordinates of the image sensor in the corresponding spatial rectangular coordinate system can be obtained by using the acquired gyroscope data or the polar coordinate values ​​of the gyroscope, according to the conversion relationship between polar coordinates and spatial rectangular coordinates.

[0064] When an electronic device vibrates, the gyroscope data constantly changes. By setting a preset sampling duration, gyroscope data is collected at preset intervals. The exposure duration can include multiple preset sampling durations. Each collected gyroscope data corresponds to a gyroscope polar coordinate. Based on the conversion relationship between polar coordinates and Cartesian coordinates, the coordinates of the corresponding image sensor in the Cartesian coordinate system can be obtained. Based on the coordinates of the image sensor corresponding to each collected gyroscope data, the jitter trajectory of the image sensor can be obtained. For example, the image sensor coordinates can be connected in a predetermined order to form the jitter trajectory of the image sensor.

[0065] A blur kernel is actually a matrix. Convolving a sharp image with a blur kernel results in a blurred image, hence the name "blur kernel." A blur kernel is a type of convolution kernel, and the essence of image convolution is matrix convolution. In other words, image blurring can be viewed as the process of convolving a sharp image with a blur kernel to obtain a blurred image.

[0066] The jitter trajectory corresponding to an image sensor is typically characterized by the jitter trajectory at at least one preset position on the image sensor. For lenses with shallow depth of field, changes in the coordinates perpendicular to the image sensor's image plane will cause changes in the size of the blur spot in that direction. Therefore, it is necessary to determine the point spread function (PSF) corresponding to each coordinate at the preset position based on the coordinates of the jitter trajectory at that preset position and the size of the corresponding blur spot. The blur spot is the light intensity distribution of the diffraction image formed by a point light source (i.e., a star) on different cross sections before and after the image sensor's image plane after passing through the optical system. A circular blur spot is a blur spot with a circular shape. The point spread function is used to characterize the light field distribution of the output image when the input object is a point light source. In this embodiment, the point spread function is used to characterize the corresponding blur kernel. For a jitter trajectory at a preset position, the corresponding blur kernel can be determined by the coordinates of the jitter trajectory at that preset position and the size of the corresponding circular blur spot. For example, during focusing, the scale of the blur circle corresponding to the jitter trajectory at a preset position is the base scale, for example, one pixel in size. Then, the sub-blur kernel corresponding to a certain coordinate at the preset position is a one-pixel blur kernel at that coordinate position. This blur kernel at the preset position can be obtained by weighted averaging of the sub-blur kernels corresponding to each coordinate of the jitter trajectory. By stitching together the blur kernels at each preset position, the target blur kernel corresponding to the image sensor can be obtained.

[0067] Step 106: Obtain the phase difference data of each pixel in the initial image, and determine the depth information corresponding to the initial image based on the phase difference data.

[0068] Based on the principle of Phase Detection Auto Focus (PDAF), the difference in the imaging positions of light rays entering the lens from different directions within the image sensor can be obtained. Based on this difference and the geometric relationship between the lens and the image sensor, the defocus distance is calculated. The defocus distance refers to the distance between the current position of the image sensor and its expected position when in focus. The imaging device can then focus based on this defocus distance. Therefore, a phase difference of 0 corresponds to a focus position; a smaller phase difference indicates a closer distance to the focal point, and vice versa.

[0069] When the image sensor uses full-pixel autofocus, meaning every pixel on the image sensor can participate in focusing, the phase difference of each pixel in the initial image can be obtained. Here, phase difference refers to the difference in imaging position caused by imaging light rays entering from different directions. For example, as... Figure 2AAs shown, the distance between the same position in the image formed by the imaging rays from both the left and right sides of the same pixel is the phase difference of that pixel. When the same pixel appears at the same position after being imaged by the imaging rays from both sides, it means that the phase difference of that pixel is 0, and it is in focus. A schematic diagram of the image when in focus is shown below. Figure 2B As shown.

[0070] Since the focusing process mainly involves adjusting the distance between the imaging object and the lens, the phase difference data of a pixel can be used to characterize the depth information corresponding to that pixel. For example, if the phase difference of a pixel is 1 pixel, it means that the pixel needs to be adjusted by 1 pixel in the depth direction to achieve focus. In this embodiment, the phase difference of each pixel in the initial image is obtained, and the phase differences of all pixels or a portion of the pixels constitute the depth information corresponding to the initial image. Optionally, the depth information corresponding to the initial image may include the phase difference of each pixel in the initial image, and the phase difference corresponding to each pixel position constitutes the depth information distribution of the initial image.

[0071] Step 108: Perform a scale transformation on the target fuzzy kernel based on the depth information to obtain the target scale fuzzy kernel.

[0072] In this embodiment, the electronic device performs a scale transformation on the target blur kernel based on the depth information of the initial image to obtain the target scale blur kernel. The same depth information in the initial image corresponds to one target scale blur kernel, and different depth information corresponds to different target scale blur kernels. For example, if the initial image includes four depth information values ​​of 1µm, 2µm, 3µm, and 4µm, then the depths of 1µm, 2µm, 3µm, and 4µm correspond to four target scale blur kernels respectively.

[0073] Optionally, the target blur kernel when in focus is scaled based on the depth information to obtain the target scale blur kernel when out of focus. For example, the reference scale of the target blur kernel when in focus is scaled based on the depth information to obtain the target scale blur kernel. For instance, the reference scale of the target blur kernel can be added to the depth information to obtain the target scale corresponding to the target scale blur kernel.

[0074] Step 110: Deblur the initial image based on the depth information and the target scale blur kernel to obtain the target image.

[0075] Optionally, the initial image is deconvolved based on the depth information and the target scale blur kernel to obtain the target image.

[0076] In this embodiment, pixels at the corresponding depth information positions in the initial image are deconvolved based on the same depth information and the target scale blur kernel corresponding to that depth information to obtain sub-images corresponding to that depth information. All sub-images corresponding to depth information are then fused to obtain the target image. For example, pixels in the initial image corresponding to a depth information position of 0.5µm are multiplied by the target scale blur kernel corresponding to 0.5µm to obtain a sub-image corresponding to 0.5µm. Sub-images corresponding to other depth information are obtained using the same method. All sub-images corresponding to depth information are then fused to obtain the target image.

[0077] Alternatively, the initial image can be deconvolved using different target scale blur kernels to obtain corresponding deconvolved images. Pixels at the corresponding depth information locations in each deconvolved image can then be extracted and fused to obtain the target image.

[0078] The image processing method described above performs a scale transformation on the target blur kernel based on the depth information of the initial image to obtain a corresponding target scale blur kernel. In other words, the blur kernel for the out-of-focus area can be determined by the blur kernel for the focused area. Different blur kernels are used for deblurring the focused and out-of-focus areas. Initial images with different depth information are deblurred using the corresponding target scale blur kernel. This better matches the actual shaking situation, resulting in more accurate image restoration and a clearer target image. Furthermore, the electronic device can use its existing gyroscope to assist in deblurring the initial image, saving hardware costs compared to traditional OIS methods.

[0079] In one embodiment, step 106, which determines the depth information corresponding to the initial image based on the phase difference data, includes:

[0080] Based on the phase difference interval corresponding to the phase difference data and the preset number of gradients, determine the phase difference gradient value of the preset number of gradients; map the phase difference data into the corresponding phase difference gradient value, and determine the depth information corresponding to the initial image based on the mapped phase difference gradient value.

[0081] The electronic device obtains the phase difference interval corresponding to the phase difference data based on the phase difference data of each pixel. The phase difference interval is a range that represents the phase difference data. For example, by determining the minimum and maximum phase difference in the phase difference data, the corresponding phase difference interval can be obtained. The preset gradient number can be set according to specific circumstances. For example, a preset gradient number of 10 can be understood to include the number of phase difference types of all pixels in the initial image. For instance, if there are 20 types of phase difference data, then the preset gradient number can be set to 20.

[0082] Optionally, the phase gradient values ​​of the preset gradient number can be determined uniformly or non-uniformly based on the phase difference interval corresponding to the phase difference data and the preset gradient number. Uniformity means that the intervals between phase gradient values ​​are equal, while non-uniformity means that the intervals between phase gradient values ​​are not equal.

[0083] In one example, the maximum phase difference is 5um, the minimum phase difference is 0, and the preset number of gradients is 11. Then, the corresponding phase difference interval is [0, 5]. Based on the phase difference interval and the preset number of gradients, the 11 phase gradient values ​​are uniformly determined as follows: 0, 0.5um, 1um, 1.5um, 2um, 2.5um, 3um, 3.5um, 4um, 4.5um, and 5um.

[0084] Optionally, the phase difference data can be mapped to corresponding phase difference gradient values ​​using methods such as rounding or rounding up / down. The depth information corresponding to the initial image can then be determined based on the mapped phase difference gradient values. Mapping can be understood as a data processing method that converts phase difference data into corresponding phase difference gradient values.

[0085] In one example, using the phase gradient values ​​obtained in the previous example, if the phase difference at a certain pixel location is 0.3µm, since 0.3µm is between 0 and 0.5µm, it can be rounded up to 0 or down to 0.5µm to map the phase difference to the corresponding phase difference gradient value. The phase difference at the corresponding pixel location is then replaced by the mapped phase difference gradient value. Each pixel is processed in the same way; that is, the depth information corresponding to the initial image can be obtained based on the mapped phase difference gradient value. A schematic diagram of the depth information corresponding to the initial image in one example is shown below. Figure 3 As shown.

[0086] In this embodiment, by determining the preset number of phase difference gradient values ​​based on the phase difference interval corresponding to the phase difference data and the preset number of gradients, the phase difference data is mapped to the corresponding phase difference gradient values. The depth information corresponding to the initial image is determined based on the mapped phase difference gradient values. This reduces the types of phase difference data to a certain extent, retaining only the preset number of phase difference gradient values. By determining the depth information of the initial image based on the phase difference gradient values, more layered depth information corresponding to the initial image can be obtained, resulting in a more discriminative target scale blur kernel, thereby obtaining a clearer target image.

[0087] In one embodiment, scaling the target blur kernel based on depth information to obtain the target scale blur kernel includes:

[0088] Based on the reference scale and phase difference gradient value of the target blur kernel during focusing, the target scale corresponding to the target scale blur kernel is determined; based on the target scale, the target blur kernel is scaled to obtain the target scale blur kernel.

[0089] Optionally, the target scale corresponding to the target scale blur kernel can be obtained by summing the reference scale of the target blur kernel and the phase difference gradient value during focusing. Then, the target blur kernel is scaled according to the target scale corresponding to the target scale blur kernel to obtain the target scale blur kernel. For a target blur kernel at a preset position, the sub-blur kernels corresponding to each target coordinate of the target blur kernel can be scaled separately to obtain the corresponding target scale sub-blur kernels. The weighted average of the target scale sub-blur kernels is then obtained to obtain the target scale blur kernel corresponding to the preset position. Here, the reference scale of the target blur kernel during focusing refers to the size of the target blur kernel during focusing. The reference scale is related to the shooting parameters and can be calculated according to existing technology, which will not be elaborated here.

[0090] In one example, such as Figure 4 As shown, the base scale of the target blur kernel at a certain preset position is 1 pixel for each target coordinate corresponding to the sub-blur kernel. Therefore, when performing scale transformation, such as for a phase difference gradient value of 0.5 pixels, the target scale is 1.5 pixels, and the coordinates remain unchanged. Then, the scale of the target scale sub-blur kernels corresponding to each coordinate is transformed to 1.5 pixels, and the target scale sub-blur kernels are weighted and averaged to obtain the target scale blur kernel corresponding to the preset position.

[0091] Optionally, the scale of the target blur kernel is transformed from the reference scale to the target scale to obtain the target scale blur kernel. The blur spot corresponding to the target scale blur kernel can be circular, square, or other shapes, without specific limitations. For example, after determining the target scale, the blur spot corresponding to the target scale blur kernel can be a circle with the target scale as the radius, or a square with the target scale as the side length, etc.

[0092] In one example, assuming the baseline size of the target blur kernel during focusing is 1 pixel, for a target scale blur kernel with a phase difference gradient value of 0.5 pixels, its target scale is 1.5 pixels, that is, the scale is adjusted from the original 1 pixel to 1.5 pixels; for a target scale blur kernel with a phase difference gradient value of 1 pixel, its target scale is 2 pixels, that is, the scale is adjusted from the original 1 pixel to 2 pixels. For target scale blur kernels corresponding to other phase difference gradient values, the same method is used, which will not be repeated here.

[0093] It should be noted that the corresponding target scale blur kernel can be determined based on the reference scale of the target blur kernel during focusing and all phase difference gradient values; alternatively, it can be determined based on the reference scale of the target blur kernel during focusing and some phase difference gradient values. Each phase difference gradient value corresponds to one target scale blur kernel. For the same phase difference gradient value, this gradient value is used to transform the reference scale corresponding to all target blur kernels to obtain the corresponding target scale blur kernel.

[0094] This embodiment determines the target scale corresponding to the target scale blur kernel by using the reference scale of the target blur kernel during focusing and the phase difference gradient value. Based on the target scale, a scale transformation is performed on the target blur kernel to obtain the target scale blur kernel. That is, using the scale of the target blur kernel during focusing as a reference, the target scale corresponding to the target scale blur kernel when out of focus can be accurately obtained. The greater the defocus distance, the larger the corresponding target scale, and thus the more blurred the image. By deblurring the initial image based on the accurate target scale blur kernel corresponding to different target scales, a clearer target image can be obtained.

[0095] In one embodiment, the initial image is deblurred based on depth information and a target scale blur kernel to obtain the target image, including:

[0096] Based on the depth information, the mask corresponding to each phase difference data in the depth information is obtained; based on the mask and the target scale blur kernel, the initial image is deblurred to obtain the target image.

[0097] In this embodiment, the depth information corresponding to the initial image represents the distribution of phase difference gradient values ​​mapped from the phase difference of each pixel in the initial image. Therefore, the same phase difference gradient value corresponds to a mask, and different phase difference gradient values ​​correspond to different masks. The mask is used to represent the corresponding position of the corresponding phase difference gradient value in the initial image. In one example, a schematic diagram of the mask corresponding to a 0.5µm phase difference gradient value in the depth information is shown below. Figure 5A As shown, and the schematic diagram of the mask corresponding to the 1µm phase difference gradient value is as follows. Figure 5B As shown, the masks corresponding to other phase difference gradient values ​​are obtained using the same method, and will not be described in detail here.

[0098] The initial image is deblurred based on the mask and the target scale blur kernel to obtain the target image. The pixels corresponding to each phase difference gradient value in the initial image can be extracted using the mask. Then, the target scale blur kernel corresponding to each phase difference gradient value is used for deblurring, resulting in deblurred images corresponding to each mask. These deblurred images are then fused to obtain the target image. The deblurring process includes deconvolution, and deconvolution algorithms can include Wiener filtering, patch-wise algorithms, Richard-Lucy algorithms, etc.

[0099] Optionally, based on a preset phase difference, masks and target scale blur kernels corresponding to phase difference gradient values ​​that satisfy preset phase conditions are selected, and the initial image is deblurred; phase difference gradient values ​​that do not meet the preset phase conditions are discarded. For example, phase difference gradient values ​​not greater than the preset phase difference are selected as target phase difference gradient values, and the initial image is deblurred based on the mask and target scale blur kernel corresponding to the target phase difference gradient values ​​to obtain the target image; phase difference gradient values ​​greater than the preset phase difference are directly discarded, and the corresponding mask and target scale blur kernel are not calculated.

[0100] In this embodiment, the initial image is deblurred based on the mask corresponding to each phase difference data in the depth information and the target scale blur kernel. That is, the images with different depth information are extracted using the corresponding mask and deblurred using the corresponding target scale blur kernel to obtain clear images corresponding to different depth information. The clear images corresponding to different depth information are then fused to obtain a clear target image.

[0101] In some embodiments, the initial image is deblurred according to a mask and a target scale blur kernel to obtain a target image, including:

[0102] Based on the initial image and the mask, multiple first images are obtained; for each first image, the first image is deconvolved according to the target scale blur kernel to obtain the corresponding second image; based on the multiple second images, the target image is obtained.

[0103] Optionally, using masks corresponding to different phase difference gradient values, pixels at corresponding positions in the initial image are extracted to obtain first images corresponding to different phase difference gradient values. For example, the masks corresponding to different phase difference gradient values ​​are multiplied by the initial image to obtain first images corresponding to different phase difference gradient values. For each first image, the target scale blur kernel corresponding to the corresponding phase difference gradient value is used to deconvolve the first image to obtain a second image corresponding to that phase difference gradient value. The second images corresponding to each phase difference gradient value are then stitched together or fused to obtain the target image.

[0104] In this embodiment, pixels at corresponding positions in the initial image are first extracted using a mask, then deblurred using the corresponding target scale blur kernel, and finally the deblurred pixels are fused into the target image. By using different blur kernels to process different depth information in the initial image, and because the processing is pixel-specific, the computational load can be reduced and the processing speed can be improved.

[0105] In some embodiments, the initial image is deblurred according to a mask and a target scale blur kernel to obtain a target image, including:

[0106] The initial image is deconvolved using the target scale blur kernel to obtain multiple third images; for each third image, a corresponding fourth image is obtained based on the third image and the corresponding mask; and the target image is obtained based on the multiple fourth images.

[0107] Optionally, the initial image is deconvolved using the target scale blur kernel corresponding to each phase difference gradient value to obtain the deconvolved third image corresponding to each phase difference gradient value. For each third image, the third image is multiplied by the mask corresponding to the corresponding phase difference gradient value to obtain the fourth image corresponding to that phase difference gradient value. The fourth image represents the pixel at the corresponding position of the mask in the target image. The target image is obtained by fusing the fourth images corresponding to each phase difference gradient value.

[0108] In this embodiment, the initial image is first deconvolved using a target scale blur kernel to obtain a corresponding deconvolved image. Then, the corresponding pixels in the deconvolved image are extracted using a mask corresponding to each phase difference gradient value and fused to obtain the target image. This allows for accurate deconvolution of the initial image, resulting in a clearer target image.

[0109] In one embodiment, such as Figure 6 As shown, the image sensor includes multiple preset areas; step 104, which determines the target blur kernel corresponding to the image sensor based on gyroscope data, includes steps 602 to 606.

[0110] Step 602: For each preset area, determine the first movement trajectory corresponding to the preset area based on the gyroscope data.

[0111] In this embodiment, the image sensor includes multiple preset regions. The specific method for dividing the image sensor into these preset regions can be set as needed. For example, when the aspect ratio of the image sensor is 4:3, the image sensor can be evenly divided into 16*12 preset regions.

[0112] Understandably, the preset regions in the image sensor are used for spatially non-uniform deblurring. Each preset region has a different initial movement trajectory when the initial image is acquired, and the corresponding blur kernel is also different. Each region in the initial image is deblurred using the corresponding blur kernel.

[0113] Step 604: Determine the first fuzzy kernel corresponding to the preset region based on the first movement trajectory corresponding to the preset region.

[0114] Optionally, a first blur kernel corresponding to the preset region is determined based on the coordinates on the first moving trajectory corresponding to the preset region and the size of the corresponding imaging blur circle. For example, a weighted average of the coordinates on the first moving trajectory and the size of the corresponding blur circle is used to obtain the first blur kernel corresponding to the preset region. The first blur kernel may include multiple second blur kernels corresponding to different jitter positions.

[0115] Step 606: Obtain the target blur kernel corresponding to the image sensor based on the first blur kernel corresponding to each preset region.

[0116] Optionally, in this embodiment, the image sensor includes multiple preset regions, each preset region corresponding to a first blur kernel. The first blur kernels corresponding to all preset regions in the image sensor can be used as the target blur kernel of the image sensor when acquiring the initial image.

[0117] In this embodiment, the image sensor is divided into multiple preset regions. Based on the corresponding gyroscope data, the first movement trajectory corresponding to each preset region can be determined, and the first blur kernel can be determined based on the first movement trajectory. Since the first movement trajectories corresponding to different preset regions on the image sensor surface are different, the corresponding first blur kernels are also different, thereby obtaining a more accurate target blur kernel. Using each first blur kernel to perform deblurring processing on the corresponding preset region in the initial image will result in higher deblurring accuracy.

[0118] In one embodiment, such as Figure 7 As shown, each preset area includes at least one preset position; step 602, which determines the first movement trajectory corresponding to the preset area based on gyroscope data, includes steps 702 to 708.

[0119] Step 702: Determine the translation and rotation of the gyroscope based on the gyroscope data within the preset sampling time.

[0120] In this embodiment, each preset region on the image sensor includes at least one preset position. The specific location and number of preset positions within each preset region can be set as needed. For example, a preset position within a preset region can be the center point or an edge point of that preset region, etc.

[0121] Optionally, within the exposure time, gyroscope data is collected at preset sampling intervals. The translation and rotation of the gyroscope within the preset sampling time are determined based on the collected gyroscope data. The gyroscope data includes acceleration data and angular velocity data, where the acceleration data includes linear acceleration data. By performing a second integral on the linear acceleration data, the translation of the gyroscope within the preset sampling time can be obtained, where the translation includes the translation of the gyroscope in each coordinate axis direction. The angular velocity data includes angular acceleration. By performing a second integral on the angular acceleration, the rotation of the gyroscope within the preset sampling time can be obtained, where the rotation includes the rotation angle of the gyroscope on each coordinate axis.

[0122] Step 704: Determine the target coordinates of the preset position in the image sensor based on the translation and rotation of the gyroscope.

[0123] Optionally, based on the positional relationship between the gyroscope and the image sensor, and according to the transformation relationship between the polar coordinate system where the gyroscope is located and the spatial rectangular coordinate system where the image sensor is located, the target coordinates of a preset position in the image sensor can be obtained according to the translation and rotation of the gyroscope.

[0124] Step 706: Determine the second movement trajectory at the preset position based on the target coordinates.

[0125] At preset sampling intervals, the gyroscope performs a data acquisition to obtain corresponding gyroscope data. Based on the gyroscope data, the corresponding translation and rotation amounts of the gyroscope are obtained. These translation and rotation amounts are then used to determine the target coordinates of the image sensor at a preset position. Therefore, multiple preset sampling intervals within the exposure time correspond to multiple target coordinates at preset positions. Optionally, these multiple target coordinates can be connected in chronological order to form a second movement trajectory at the preset position.

[0126] Step 708: Determine the first movement trajectory corresponding to the preset area based on the second movement trajectory.

[0127] Optionally, when the preset area includes only one preset position, the second movement trajectory corresponding to that preset position is used as the first movement trajectory of the preset area when acquiring the initial image. When the preset area includes at least two preset positions, the average value of the second movement trajectories of each preset position can be used as the first movement trajectory of the preset area when acquiring the initial image.

[0128] It should be noted that if the preset area includes at least two preset positions, other methods can be used to determine the first movement trajectory of the corresponding preset area, such as weighted average, filtering, etc., which are not limited here. Understandably, when using filtering to determine the first movement trajectory of the preset area, a second movement trajectory corresponding to a preset position can be selected from the second movement trajectories corresponding to each preset position according to preset filtering conditions, and used as the first movement trajectory corresponding to that preset area.

[0129] In one example, such as Figure 8 As shown, a spatial rectangular coordinate system is established with the gyroscope as the center. The coordinates of the gyroscope are (0,0,0). The center point of the image sensor is selected as the preset position, with initial coordinates of (x0,y0,z0). The coordinates of the edge points of the image sensor are (x0,y0,z0). i ,y i ,z i ).

[0130] Within the exposure time T, the electronic device collects gyroscope data once every preset sampling time Δt, with the number of samplings n = T / Δt. The gyroscope data collected each time is integrated twice over time to obtain the translation and rotation of the gyroscope in the X, Y, and Z directions, respectively. Then, the translation and rotation of the gyroscope in the X, Y, and Z directions are converted into the translation and rotation of the image sensor in the X, Y, and Z directions, respectively.

[0131] The electronic device adds the translation amount of the preset position in the X, Y and Z directions and the initial coordinates of the corresponding directions to obtain the coordinate values ​​of the preset position in the X, Y and Z directions after the preset sampling time, as shown in the following formula (1).

[0132]

[0133] Where x0, y0, and z0 are the initial coordinates of the image sensor in the X, Y, and Z directions, respectively, and Δx, Δy, and Δz are the translation amounts in the X, Y, and Z directions after a preset sampling time at the preset position. These are the coordinate values ​​in the X, Y, and Z directions corresponding to the preset position after a preset sampling time. The first coordinate is obtained based on the translation.

[0134] For rotations along an axis, the rotation amount in each direction can be obtained through the mapping relationship between the polar coordinate system and the spatial rectangular coordinate system. For example... Figure 9 The diagram shows a polar coordinate system centered on the gyroscope, with the gyroscope's coordinates at (0, 0, 0). The image sensor rotates Δω along the Z-axis. zIf the initial coordinates of the preset position in the polar coordinate system are (x0, y0, z0), the coordinates after rotation are (x′0, y′0, z′0).

[0135] Within the preset sampling time, when the Z-axis rotates, its projection on the XY plane will undergo a position change. After polar coordinate transformation, the second coordinate corresponding to the Z-axis rotation can be obtained, as shown in formula (2).

[0136]

[0137] Similarly, the second coordinates can also be obtained by rotating the X and Y axes. For rotation of the X-axis, the corresponding second coordinates are shown in formula (3):

[0138]

[0139] When rotating along the Y-axis, the corresponding second coordinate is shown in formula (4):

[0140]

[0141] Where, Δω z ,Δω x and Δω y These are the rotation amounts on the Z-axis, X-axis, and Y-axis, respectively. It can be understood that if the preset position is rotated on the axis, then the coordinates in the corresponding direction remain unchanged. For example, if it is rotated on the Z-axis, then the coordinates in the Z direction remain unchanged.

[0142] In practical applications, the preset position may rotate on at least one of the X, Y, and Z axes within the preset sampling time, or it may not rotate at all. In this case, only translation will occur, that is, only the translation amount will be generated.

[0143] The second coordinate is obtained by adding the rotation coordinates corresponding to the rotation direction when rotation occurs. For example, when the X-axis and Z-axis rotate, the coordinates corresponding to the rotation of the X-axis are added together. Corresponding to Z-axis rotation Add, rotate the X-axis accordingly Corresponding to Z-axis rotation Add, rotate the X-axis accordingly Corresponding to Z-axis rotation Add them together to get the second coordinate.

[0144] Add the coordinate values ​​corresponding to the first and second coordinates to obtain the target coordinates.

[0145] Optionally, the electronic device combines the target coordinates obtained at preset sampling intervals at a preset position on the image sensor to obtain a second jitter trajectory at the preset position. For example, the initial coordinates of the preset position are (x0, y0, z0), and the target coordinates obtained at preset sampling intervals are... Where n is a positive integer, then the second jitter trajectory at the preset position is... The second jitter trajectory at other preset positions on the image sensor can also be calculated using the same method described above, and will not be repeated here.

[0146] Step 604, which determines the first fuzzy kernel corresponding to the preset region based on the first movement trajectory corresponding to the preset region, includes: determining the second fuzzy kernel corresponding to the preset position based on the second movement trajectory corresponding to the preset position; and determining the first fuzzy kernel corresponding to the preset region based on the second fuzzy kernel.

[0147] Optionally, the second blur kernel corresponding to the preset position can be determined based on the coordinates of each target on the second moving trajectory corresponding to the preset position and the scale of the corresponding blur spot. For example, the electronic device acquires the target coordinates corresponding to the preset position at each sampling time and the blur circle corresponding to each target coordinate; it adjusts each target coordinate to the scale of the corresponding blur circle to obtain the point spread function corresponding to each target coordinate, which represents the blur kernel. The scale of the blur circle includes information such as the diameter, radius, perimeter, and area of ​​the blur circle. The scale of the blur circle can reflect the scale of the corresponding sub-blur kernel. The scale of the blur circle corresponding to each target coordinate can be the same or different. Optionally, the scale of the blur circle can be calculated based on the imaging parameters, where the imaging parameters refer to the parameters when capturing the image, including focal length, lens diameter, distance from the object to the lens, image distance, etc. The electronic device can obtain the blur circle on the image sensor by determining the scale of the blur circle.

[0148] Optionally, the electronic device integrates the second fuzzy kernels corresponding to each preset position in the preset region to obtain a first fuzzy kernel corresponding to the preset region. Further, the electronic device can sum the second fuzzy kernels corresponding to each preset position and calculate the average to obtain the second fuzzy kernel corresponding to the preset region. Alternatively, the electronic device can perform a weighted average of the second fuzzy kernels corresponding to each preset position to obtain the first fuzzy kernel corresponding to the preset region; for example, the weight of the second fuzzy kernel corresponding to each preset position can be determined based on the specific location of each preset position in the preset region. Or, the electronic device can use the median value of the second fuzzy kernels corresponding to each preset position as the first fuzzy kernel corresponding to the preset region.

[0149] In this embodiment, the first movement trajectory of the preset region when acquiring the initial image is determined according to the second movement trajectory corresponding to the preset position in the preset region. This can more accurately determine the first movement trajectory corresponding to each preset region in the image sensor. The first blur kernel corresponding to the preset region is determined according to the first movement trajectory, thereby obtaining a more accurate target blur kernel.

[0150] In some embodiments, determining a second fuzzy kernel corresponding to a preset position based on a second movement trajectory corresponding to a preset position includes:

[0151] Based on the target coordinates in the second movement trajectory, determine the sub-fuzzy kernel corresponding to each target coordinate; based on the sub-fuzzy kernel corresponding to each target coordinate, determine the second fuzzy kernel corresponding to the preset position.

[0152] Optionally, the first movement trajectory corresponding to the preset position includes multiple target coordinates. Based on each target coordinate and the scale of the corresponding blur spot at the target coordinate, a sub-blur kernel at the target coordinate is determined. The sub-blur kernels are then weighted and averaged to obtain the second blur kernel corresponding to the preset position.

[0153] In one example, the process of a preset position generating diffuse circular spots during movement is as follows: Figure 10 As shown, at time t1 within the exposure time, the coordinates of a preset position in a preset area are (x1, y1, z1), and the diameter of the resulting diffuse circle is 1 pixel. At time t2, the coordinates of the preset position are (x2, y2, z2), which is the distance the preset position moves in the x direction (x2-x1) and in the y direction (y2-y1). Based on the distance the image sensor moves in the z direction (z2-z1), the diameter of the diffuse circle on the image sensor's imaging surface is determined. At time t3, the coordinates of the preset position are (x3, y3, z3), which is the distance the preset position moves in the x direction (x3-x2) and in the y direction (y3-y2). Based on the distance the image sensor moves in the z direction (z3-z2), the diameter of the diffuse circle on the image sensor's imaging surface is determined. Among them, (x2-x1), (y2-y1), (z2-z1), (x3-x2), (y3-y2), and (z3-z2) are all used to represent the difference between the coordinate values ​​before and after.

[0154] The electronic device acquires the target coordinates and blur circles at times t1, t2, t3 and up to tn within the exposure time, and integrates them over time, that is, first sums and then takes the average value, to obtain the second blur kernel corresponding to the preset position.

[0155] In one example, if the size of the diffuse circle remains unchanged during the movement of the preset position, then the scale of each sub-blur kernel is the same, only the coordinates change, and the scale of the second blur kernel obtained from each sub-blur kernel is the same as the scale of any one of the sub-blur kernels.

[0156] In the above embodiments, based on the target coordinates in the second movement trajectory, a sub-fuzzy kernel corresponding to each target coordinate is determined; based on the sub-fuzzy kernel corresponding to each target coordinate, a second fuzzy kernel corresponding to the preset position is determined, which can more accurately determine the second fuzzy kernel of the preset position.

[0157] In one embodiment, a flowchart of the image processing method is shown below. Figure 11 As shown, the camera shutter is opened for exposure. During the exposure time, an initial image and corresponding gyroscope data are acquired. Based on the positional relationship between the image sensor and the gyroscope, a corresponding coordinate system is established. A unified standard can be used to represent the positions of the gyroscope and the image sensor. The movement trajectory of the image sensor during the initial image acquisition is determined based on the gyroscope data. The target blur kernel of the image sensor during the exposure time is determined based on the movement trajectory of the image sensor. The phase difference data of each pixel in the initial image is acquired. The depth information of the initial image is determined based on the phase difference data, and the depth information is divided into multiple phase difference gradient values. The target blur kernel is scaled based on the phase difference gradient values ​​to obtain the target scale blur kernel. Simultaneously, a mask corresponding to each depth gradient value is obtained based on the depth information. The initial image is deconvolved based on the mask and the target scale blur kernel to obtain the target image. Optionally, the initial image is deconvolved based on the mask and the target scale blur kernel corresponding to a preset region to obtain a target sub-image. Target sub-images corresponding to other preset regions are obtained using the same method. The target sub-images corresponding to each preset region can be stitched together to obtain the target image.

[0158] The image processing method in this embodiment divides the depth information of the initial image into different depth gradient values, performs scale transformation on the target blur kernel based on the depth gradient values ​​to obtain the target scale blur kernel, and simultaneously obtains the mask corresponding to the depth gradient values ​​in the initial image. Based on the mask and the target scale blur kernel, the initial image is deconvolved into regions. Since the blur kernels corresponding to different preset regions on the image sensor surface are different, and the blur kernel scales corresponding to different depth information are different, it is equivalent to simultaneously considering the influence of different regions on the image sensor surface and the change in the focal length direction of the image sensor on the blur kernel, thereby obtaining a target image with high clarity.

[0159] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0160] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0161] In one embodiment, such as Figure 12 As shown, an image processing apparatus is provided, including: an initial data acquisition module 1202, a blur kernel determination module 1204, a depth information determination module 1206, a blur kernel scaling transformation module 1208, and a deblurring processing module 1210, wherein:

[0162] The initial data acquisition module 1202 is used to acquire the initial image and the corresponding gyroscope data;

[0163] The fuzzy kernel determination module 1204 is used to determine the target fuzzy kernel corresponding to the image sensor based on the positional relationship between the gyroscope and the image sensor, according to the gyroscope data.

[0164] The depth information determination module 1206 is used to acquire the phase difference data of each pixel in the initial image and determine the depth information corresponding to the initial image based on the phase difference data.

[0165] The fuzzy kernel scaling transformation module 1208 is used to perform scaling transformation on the target fuzzy kernel according to the depth information to obtain the target scale fuzzy kernel;

[0166] The deblurring module 1210 is used to deblur the initial image based on the depth information and the target scale blur kernel to obtain the target image.

[0167] In one embodiment, the depth information determination module 1206 is further configured to:

[0168] Based on the phase difference interval corresponding to the phase difference data and the preset number of gradients, determine the phase difference gradient value of the preset number of gradients; map the phase difference data into the corresponding phase difference gradient value, and determine the depth information corresponding to the initial image based on the mapped phase difference gradient value.

[0169] In one embodiment, the fuzzy kernel scaling module 1208 is further configured to:

[0170] Based on the reference scale of the target blur kernel during focusing and the phase difference gradient value, the target scale corresponding to the target scale blur kernel is determined; based on the target scale, the target blur kernel is scaled to obtain the target scale blur kernel.

[0171] In one embodiment, the deblurring module 1210 is further configured to:

[0172] Based on the depth information, a mask corresponding to each phase difference data in the depth information is obtained; based on the mask and the target scale blur kernel, the initial image is deblurred to obtain the target image.

[0173] In one embodiment, the deblurring module 1210 is further configured to:

[0174] Based on the initial image and the mask, a plurality of first images are obtained; for each first image, the first image is deconvolved according to the target scale blur kernel to obtain a corresponding second image; based on the plurality of second images, the target image is obtained.

[0175] In one embodiment, the deblurring module 1210 is further configured to:

[0176] The initial image is deconvolved using the target scale blur kernel to obtain multiple third images; for each third image, a corresponding fourth image is obtained based on the third image and the corresponding mask; and the target image is obtained based on the multiple fourth images.

[0177] In one embodiment, the image sensor includes multiple preset regions; the blur kernel determination module 1204 is further configured to:

[0178] For each preset region, a first motion trajectory corresponding to the preset region is determined based on the gyroscope data; a first blur kernel corresponding to the preset region is determined based on the first motion trajectory corresponding to the preset region; and a target blur kernel corresponding to the image sensor is obtained based on the first blur kernel corresponding to each preset region.

[0179] In one embodiment, each preset region includes at least one preset location; the fuzzy kernel determination module 1204 is further configured to:

[0180] The translation and rotation of the gyroscope are determined based on gyroscope data within a preset sampling time. The target coordinates of a preset position in the image sensor are determined based on the translation and rotation of the gyroscope. A second movement trajectory for the preset position is determined based on the target coordinates. A first movement trajectory corresponding to the preset region is determined based on the second movement trajectory. A second blur kernel corresponding to the preset position is determined based on the second movement trajectory. A first blur kernel corresponding to the preset region is determined based on the second blur kernel.

[0181] In one embodiment, the fuzzy kernel determination module 1204 based on the second movement trajectory corresponding to the preset position is further configured to:

[0182] Based on the target coordinates in the second movement trajectory, determine the sub-fuzzy kernel corresponding to each target coordinate; based on the sub-fuzzy kernel corresponding to each target coordinate, determine the second fuzzy kernel corresponding to the preset position.

[0183] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0184] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 13 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an image processing method.

[0185] Those skilled in the art will understand that Figure 13The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0186] This application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, cause the processors to perform the steps of an image processing method.

[0187] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform an image processing method.

[0188] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0189] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0190] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0191] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, include: Acquire the initial image and the corresponding gyroscope data; Based on the positional relationship between the gyroscope and the image sensor, the target blur kernel corresponding to the image sensor is determined according to the gyroscope data; The phase difference data of each pixel in the initial image is obtained, and the phase difference gradient value of the preset gradient number is determined according to the phase difference interval corresponding to the phase difference data and the preset gradient number. The phase difference data is mapped into corresponding phase difference gradient values, and the depth information corresponding to the initial image is determined based on the mapped phase difference gradient values. The target scale corresponding to the target scale blur kernel is determined based on the reference scale of the target blur kernel during focusing and the phase difference gradient value; wherein, the reference scale of the target blur kernel during focusing refers to the size of the target blur kernel during focusing; Based on the target scale, the target fuzzy kernel is scaled to obtain the target scale fuzzy kernel; Based on the depth information and the target scale blur kernel, the initial image is deblurred to obtain the target image.

2. The method according to claim 1, characterized in that, The step of deblurring the initial image based on the depth information and the target scale blur kernel to obtain the target image includes: Based on the depth information, the mask corresponding to each phase difference data in the depth information is obtained; The initial image is deblurred based on the mask and the target scale blur kernel to obtain the target image.

3. The method according to claim 2, characterized in that, The step of deblurring the initial image based on the mask and the target scale blur kernel to obtain the target image includes: Based on the initial image and the mask, multiple first images are obtained; For each of the first images, the first image is deconvolved according to the target scale blur kernel to obtain the corresponding second image; The target image is obtained based on multiple second images.

4. The method according to claim 2, characterized in that, The step of deblurring the initial image based on the mask and the target scale blur kernel to obtain the target image includes: The initial image is deconvolved according to the target scale blur kernel to obtain multiple third images; For each of the third images, a corresponding fourth image is obtained based on the third image and the corresponding mask; The target image is obtained based on multiple fourth images.

5. The method according to claim 1, characterized in that, The image sensor includes multiple preset regions; determining the target blur kernel corresponding to the image sensor based on the gyroscope data includes: For each preset area, a first movement trajectory corresponding to the preset area is determined based on the gyroscope data; Based on the first movement trajectory corresponding to the preset region, determine the first fuzzy kernel corresponding to the preset region; The target blur kernel corresponding to the image sensor is obtained based on the first blur kernel corresponding to each of the preset regions.

6. The method according to claim 5, characterized in that, Each preset region includes at least one preset position; determining the first movement trajectory corresponding to the preset region based on the gyroscope data includes: The translation and rotation of the gyroscope are determined based on the gyroscope data within a preset sampling period; The target coordinates of the preset position in the image sensor are determined based on the translation and rotation of the gyroscope. Based on the target coordinates, determine the second movement trajectory at the preset position; Based on the second movement trajectory, determine the first movement trajectory corresponding to the preset area; The step of determining the first fuzzy kernel corresponding to the preset region based on the first movement trajectory corresponding to the preset region includes: Based on the second movement trajectory corresponding to the preset position, determine the second fuzzy kernel corresponding to the preset position; Based on the second fuzzy kernel, the first fuzzy kernel corresponding to the preset region is determined.

7. The method according to claim 6, characterized in that, The step of determining the second fuzzy kernel corresponding to the preset position based on the second movement trajectory corresponding to the preset position includes: Based on the target coordinates in the second movement trajectory, determine the sub-fuzzy kernel corresponding to each target coordinate; Based on the sub-fuzzy kernels corresponding to each target coordinate, determine the second fuzzy kernel corresponding to the preset position.

8. An image processing apparatus, characterized in that, include: The initial data acquisition module is used to acquire the initial image and the corresponding gyroscope data; The fuzzy kernel determination module is used to determine the target fuzzy kernel corresponding to the image sensor based on the positional relationship between the gyroscope and the image sensor, according to the gyroscope data. The depth information determination module is used to acquire phase difference data of each pixel in the initial image, and determine the phase difference gradient value of the preset gradient number according to the phase difference interval corresponding to the phase difference data and the preset gradient number; map the phase difference data into the corresponding phase difference gradient value, and determine the depth information corresponding to the initial image according to the mapped phase difference gradient value; The blur kernel scale transformation module is used to determine the target scale corresponding to the target scale blur kernel based on the reference scale of the target blur kernel during focusing and the phase difference gradient value; Based on the target scale, the target blur kernel is scaled to obtain the target scale blur kernel, wherein the reference scale of the target blur kernel during focusing refers to the size of the target blur kernel during focusing; The deblurring module is used to deblur the initial image based on the depth information and the target scale blur kernel to obtain the target image.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images