Image processing methods and apparatus, calibration methods and apparatus, electronic equipment

By generating a motion blur kernel through the conversion relationship between camera module and gyroscope data, the problem of high hardware cost in traditional image processing is solved, and a clearer image de-motion blur effect is achieved.

CN117934325BActive Publication Date: 2026-11-17GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202211239981.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-10-11
Publication Date
2026-11-17
Estimated Expiration
2042-10-11

AI Technical Summary

Technical Problem

Traditional image processing methods suffer from high hardware costs, especially in low-light shooting where image blur caused by electronic device shake is difficult to effectively solve.

Method used

Data is acquired by the camera module and target gyroscope of the electronic device to determine motion information, and a motion blur kernel is generated based on the transformation relationship for de-motion blurring, reducing the reliance on optical image stabilization motors and image sensors.

Benefits of technology

It reduces the hardware cost of image processing while improving image clarity and achieving more accurate motion blur removal.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117934325B_ABST
    Figure CN117934325B_ABST
Patent Text Reader

Abstract

This application relates to an image processing method, apparatus, electronic device, storage medium, and computer program product. The method includes: acquiring a first image through a camera module and obtaining target gyroscope data through a target gyroscope; determining the motion information of the target gyroscope based on the target gyroscope data; converting the motion information of the target gyroscope into motion information of the camera module according to the target conversion relationship between the target gyroscope and the camera module; the target conversion relationship is obtained by the electronic device during calibration based on gyroscope data from a first gyroscope and a second gyroscope, where the first gyroscope is located at the target gyroscope position within the electronic device, and the second gyroscope is located at the camera module position within the electronic device; generating a motion blur kernel based on the motion information of the camera module; and using the motion blur kernel to deblur the first image to obtain the target image. This method can reduce the hardware cost of image processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of imaging technology, and in particular to an image processing method and apparatus, a calibration method and apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In everyday shooting, especially in low-light conditions, the exposure time increases significantly. At this time, camera offset due to electronic device shake exceeds the size of the imaging pixels, resulting in blurred images. Therefore, image deblurring is necessary. Image deblurring technology is a technique that restores blurred images using information related to the blur and necessary prior image knowledge.

[0003] Traditional image processing methods typically involve increasing the size of the image sensor and optical image stabilization motor, increasing the light energy collected by the camera module, improving the signal-to-noise ratio, and using better lenses to achieve higher resolution, thereby reducing image blur caused by electronic device shake.

[0004] However, this traditional image processing method suffers from high hardware costs. Summary of the Invention

[0005] This application provides an image processing method and apparatus, a calibration method and apparatus, an electronic device, and a computer-readable storage medium, which can reduce the hardware cost of image processing.

[0006] Firstly, this application provides an image processing method. The method includes:

[0007] The first image is acquired by the camera module of the electronic device, and the target gyroscope data is obtained by the target gyroscope of the electronic device during the acquisition of the first image.

[0008] Based on the target gyroscope data, determine the motion information of the target gyroscope;

[0009] Based on the target conversion relationship between the target gyroscope and the camera module, the motion information of the target gyroscope is converted into the motion information of the camera module; the target conversion relationship is obtained by the electronic device during the calibration process based on the gyroscope data of the first gyroscope and the gyroscope data of the second gyroscope, wherein the first gyroscope is located at the target gyroscope position in the electronic device, and the second gyroscope is located at the camera module position in the electronic device;

[0010] Based on the motion information of the camera module, a motion blur kernel is generated;

[0011] The motion blur kernel is used to deblur the first image to obtain the target image.

[0012] Secondly, this application also provides an image processing apparatus. The apparatus includes:

[0013] The data acquisition module is used to acquire a first image through the camera module of the electronic device, and to acquire target gyroscope data through the target gyroscope of the electronic device during the acquisition of the first image.

[0014] The motion information determination module is used to determine the motion information of the target gyroscope based on the target gyroscope data;

[0015] The conversion module is used to convert the motion information of the target gyroscope into the motion information of the camera module according to the target conversion relationship between the target gyroscope and the camera module; the target conversion relationship is obtained by the electronic device during the calibration process based on the gyroscope data of the first gyroscope and the gyroscope data of the second gyroscope, wherein the first gyroscope is located at the target gyroscope position in the electronic device, and the second gyroscope is located at the camera module position in the electronic device;

[0016] The blur kernel generation module is used to generate a motion blur kernel based on the motion information of the camera module;

[0017] The deblurring module is used to deblur the first image using the motion blur kernel to obtain the target image.

[0018] Thirdly, this application also provides an electronic device. The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0019] The first image is acquired by the camera module of the electronic device, and the target gyroscope data is obtained by the target gyroscope of the electronic device during the acquisition of the first image.

[0020] Based on the target gyroscope data, determine the motion information of the target gyroscope;

[0021] Based on the target conversion relationship between the target gyroscope and the camera module, the motion information of the target gyroscope is converted into the motion information of the camera module; the target conversion relationship is obtained by the electronic device during the calibration process based on the gyroscope data of the first gyroscope and the gyroscope data of the second gyroscope, wherein the first gyroscope is located at the target gyroscope position in the electronic device, and the second gyroscope is located at the camera module position in the electronic device;

[0022] Based on the motion information of the camera module, a motion blur kernel is generated;

[0023] The motion blur kernel is used to deblur the first image to obtain the target image.

[0024] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0025] The first image is acquired by the camera module of the electronic device, and the target gyroscope data is obtained by the target gyroscope of the electronic device during the acquisition of the first image.

[0026] Based on the target gyroscope data, determine the motion information of the target gyroscope;

[0027] Based on the target conversion relationship between the target gyroscope and the camera module, the motion information of the target gyroscope is converted into the motion information of the camera module; the target conversion relationship is obtained by the electronic device during the calibration process based on the gyroscope data of the first gyroscope and the gyroscope data of the second gyroscope, wherein the first gyroscope is located at the target gyroscope position in the electronic device, and the second gyroscope is located at the camera module position in the electronic device;

[0028] Based on the motion information of the camera module, a motion blur kernel is generated;

[0029] The motion blur kernel is used to deblur the first image to obtain the target image.

[0030] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0031] The first image is acquired by the camera module of the electronic device, and the target gyroscope data is obtained by the target gyroscope of the electronic device during the acquisition of the first image.

[0032] Based on the target gyroscope data, determine the motion information of the target gyroscope;

[0033] Based on the target conversion relationship between the target gyroscope and the camera module, the motion information of the target gyroscope is converted into the motion information of the camera module; the target conversion relationship is obtained by the electronic device during the calibration process based on the gyroscope data of the first gyroscope and the gyroscope data of the second gyroscope, wherein the first gyroscope is located at the target gyroscope position in the electronic device, and the second gyroscope is located at the camera module position in the electronic device;

[0034] Based on the motion information of the camera module, a motion blur kernel is generated;

[0035] The motion blur kernel is used to deblur the first image to obtain the target image.

[0036] The aforementioned image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product acquire a first image through the camera module of the electronic device and obtain target gyroscope data during the acquisition of the first image through the target gyroscope of the electronic device; determine the motion information of the target gyroscope based on the target gyroscope data; then, according to the target conversion relationship between the target gyroscope and the camera module, the motion information of the target gyroscope can be converted into the motion information of the camera module, and a motion blur kernel can be generated based on the motion information of the camera module to perform motion blur removal on the first image. This can reduce the hardware of the optical image stabilization motor and avoid increasing the size of the image sensor, thereby reducing the hardware cost of image motion blur removal. Meanwhile, during the calibration process, the first gyroscope is located at the target gyroscope position within the electronic device, and the second gyroscope is located at the camera module position. Therefore, by using the gyroscope data from the first and second gyroscopes, the conversion relationship between the first and second gyroscopes can be accurately obtained, which is the target conversion relationship between the target gyroscope and the camera module. Based on this target conversion relationship, the first image can be more accurately deblurred, resulting in a clearer target image.

[0037] Sixthly, this application provides a calibration method applied to an electronic device. During the calibration process, a first gyroscope is located at a target gyroscope position within the electronic device, and a second gyroscope is located at a camera module position within the electronic device. The method includes:

[0038] Acquire the first gyroscope data collected by the first gyroscope and the second gyroscope data collected by the second gyroscope when the electronic device is in motion;

[0039] Based on the first gyroscope data and the second gyroscope data, a conversion relationship between the first gyroscope and the second gyroscope is obtained; the conversion relationship is used by the electronic device to convert the motion information of the target gyroscope into the motion information of the camera module during use.

[0040] Seventhly, this application also provides a calibration device applied to an electronic device. During the calibration process of the electronic device, a first gyroscope is provided at the target gyroscope position in the electronic device, and a second gyroscope is provided at the camera module position in the electronic device. The device includes:

[0041] The data acquisition module is used to acquire the first gyroscope data collected by the first gyroscope and the second gyroscope data collected by the second gyroscope when the electronic device is in motion.

[0042] The calibration module is used to calibrate the conversion relationship between the first gyroscope and the second gyroscope based on the first gyroscope data and the second gyroscope data; the conversion relationship is used by the electronic device to convert the motion information of the target gyroscope into the motion information of the camera module during use.

[0043] Eighthly, this application also provides an electronic device in which, during the calibration process, a first gyroscope is provided at the target gyroscope position and a second gyroscope is provided at the camera module position. The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0044] Acquire the first gyroscope data collected by the first gyroscope and the second gyroscope data collected by the second gyroscope when the electronic device is in motion;

[0045] Based on the first gyroscope data and the second gyroscope data, a conversion relationship between the first gyroscope and the second gyroscope is obtained; the conversion relationship is used by the electronic device to convert the motion information of the target gyroscope into the motion information of the camera module during use.

[0046] Ninthly, this application also provides a computer-readable storage medium applied to an electronic device, wherein during the calibration process of the electronic device, a first gyroscope is provided at the target gyroscope position in the electronic device, and a second gyroscope is provided at the camera module position in the electronic device. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0047] Acquire the first gyroscope data collected by the first gyroscope and the second gyroscope data collected by the second gyroscope when the electronic device is in motion;

[0048] Based on the first gyroscope data and the second gyroscope data, a conversion relationship between the first gyroscope and the second gyroscope is obtained; the conversion relationship is used by the electronic device to convert the motion information of the target gyroscope into the motion information of the camera module during use.

[0049] Tenthly, this application also provides a computer program product applied to an electronic device, wherein during the calibration process of the electronic device, a first gyroscope is provided at the target gyroscope position in the electronic device, and a second gyroscope is provided at the camera module position in the electronic device. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0050] Acquire the first gyroscope data collected by the first gyroscope and the second gyroscope data collected by the second gyroscope when the electronic device is in motion;

[0051] Based on the first gyroscope data and the second gyroscope data, a conversion relationship between the first gyroscope and the second gyroscope is obtained; the conversion relationship is used by the electronic device to convert the motion information of the target gyroscope into the motion information of the camera module during use.

[0052] In the aforementioned calibration method, apparatus, electronic device, computer-readable storage medium, and computer program product, during the calibration process, the first gyroscope is located at the target gyroscope position within the electronic device, and the second gyroscope is located at the camera module position within the electronic device. Therefore, by using the gyroscope data collected by the first gyroscope and the second gyroscope during the movement of the electronic device, the conversion relationship between the first and second gyroscopes can be accurately obtained, that is, the target conversion relationship between the target gyroscope and the camera module. Based on this target conversion relationship, the first image can be more accurately deblurred, resulting in a clearer target image. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart of an image processing method in one embodiment;

[0055] Figure 2 This is a schematic diagram of an electronic device in use during one embodiment;

[0056] Figure 3 A flowchart of an image processing method in another embodiment;

[0057] Figure 4 This is a schematic diagram of a uniform block division method in one embodiment;

[0058] Figure 5 This is a schematic diagram of a centrally located non-uniformly partitioned block method in one embodiment;

[0059] Figure 6 This is a schematic diagram of a main body non-uniform block division method in one embodiment;

[0060] Figure 7 This is a schematic diagram of the image sensor of the camera module rotating around the Y-axis in one embodiment;

[0061] Figure 8 This is a schematic diagram of generating a sub-motion blur kernel for a preset region in one embodiment;

[0062] Figure 9 This is a schematic diagram illustrating the generation of a motion blur kernel in one embodiment;

[0063] Figure 10 Here is a flowchart of the calibration method in one embodiment;

[0064] Figure 11 This is a schematic diagram of an electronic device being calibrated in one embodiment;

[0065] Figure 12 This is a structural block diagram of an image processing device in one embodiment;

[0066] Figure 13 This is a structural block diagram of the calibration device in one embodiment;

[0067] Figure 14 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0069] In one embodiment, such as Figure 1 As shown, an image processing method is provided. This embodiment illustrates the application of this method to an electronic device, which can be a terminal or a server. It is understood that the method can also be applied to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, smart cars, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. The server can be a standalone server or a server cluster consisting of multiple servers.

[0070] In this embodiment, the image processing method includes the following steps 102 to 110:

[0071] Step 102: Acquire the first image through the camera module of the electronic device, and obtain target gyroscope data through the target gyroscope of the electronic device during the acquisition of the first image.

[0072] A camera module is a module containing multiple camera components used for image processing and transmission. These camera components may specifically include a lens for imaging, an AF (Auto Focus) motor for focusing during lens movement, and an image sensor. They may also include filters, circuit boards, etc. The image sensor converts the light signals entering the electronic device into electrical signals and generates an image. The gyroscope acquires gyroscope data, including angular acceleration and linear acceleration. Optionally, the camera module can be a telephoto camera module, or other types of camera modules, such as a wide-angle camera module, etc., and is not limited to these.

[0073] The target gyroscope data includes angular acceleration and linear acceleration, and may also include translational acceleration.

[0074] like Figure 2 As shown, in the three-dimensional coordinate system XYZ, the electronic device includes a target gyroscope and a camera module, both of which can move in the X, Y, and Z directions.

[0075] Step 104: Determine the motion information of the target gyroscope based on the target gyroscope data.

[0076] The motion information of the target gyroscope includes the gyroscope's rotation angle and displacement.

[0077] Optionally, during the capture time of the first image, the electronic device acquires multiple sets of target gyroscope data. Each set of target gyroscope data includes the angular acceleration and translational acceleration of the target gyroscope. The rotation angle of the target gyroscope is obtained by integrating the angular acceleration twice, and the displacement of the target gyroscope is obtained by integrating the translational acceleration twice. Based on the rotation angle and displacement of the target gyroscope each time, the coordinate position of the target gyroscope at each time can be determined.

[0078] Optionally, determining the motion information of the target gyroscope based on the target gyroscope data includes: filtering the target gyroscope data and obtaining the motion information of the target gyroscope based on the filtered target gyroscope data; wherein the filtering process includes at least one of noise reduction processing and bias adjustment processing.

[0079] Understandably, the target gyroscope data output by the target gyroscope may differ in format from the data used by the algorithm. For example, the acceleration information output by the target gyroscope may have positive or negative values ​​depending on the direction, while the algorithm's input may be a non-zero data point. Therefore, bias processing of the target gyroscope data is necessary. This could involve adding a bias to the target gyroscope data to convert negative values ​​to positive values, or taking the absolute value. Filtering the target gyroscope data in electronic devices is a preprocessing step for the raw data.

[0080] Electronic devices can filter the target gyroscope data to obtain more accurate filtered target gyroscope data, thereby obtaining more accurate motion information of the target gyroscope.

[0081] Since the clock frequency of the target gyroscope is relatively high, in order to further improve the processing efficiency of the image processing algorithm, the raw gyroscope data output by the target gyroscope can be downsampled in time, and the raw gyroscope data output at multiple times can be averaged into a single target gyroscope data, and then the target gyroscope data can be filtered.

[0082] Step 106: Based on the target conversion relationship between the target gyroscope and the camera module, the motion information of the target gyroscope is converted into the motion information of the camera module. The target conversion relationship is obtained by the electronic device during the calibration process based on the gyroscope data of the first gyroscope and the gyroscope data of the second gyroscope. The first gyroscope is located at the target gyroscope position in the electronic device, and the second gyroscope is located at the camera module position in the electronic device.

[0083] It is understandable that electronic devices include a target gyroscope and a camera module, which are located in different positions, and therefore there is a motion information conversion relationship between the target gyroscope and the camera module.

[0084] Optionally, the target transformation relationship can be a transformation matrix. The electronic device obtains the transformation matrix between the pre-calibrated target gyroscope and the camera module, multiplies the motion information of the target gyroscope by the transformation matrix, and obtains the motion information of the camera module.

[0085] It should be noted that the first gyroscope and the target gyroscope can be different gyroscopes or the same gyroscope. That is, during the calibration process of the electronic device, the camera module is replaced with the second gyroscope, and the target gyroscope is used as the first gyroscope.

[0086] Step 108: Generate a motion blur kernel based on the motion information of the camera module.

[0087] A blur kernel is actually a matrix. Convolving a sharp image with a blur kernel results in a blurred image, hence the name "blur kernel." It's a type of convolution kernel, and image convolution is essentially matrix convolution. In other words, image blurring can be viewed as the process of convolving a sharp image with a blur kernel to obtain a blurred image.

[0088] The point spread function (PSF) describes the response of an optical system to a point impulse function during the imaging process in the spatial domain. It can be used to describe the imaging effect, which is also known as the blur kernel.

[0089] Optionally, the electronic device determines the motion trajectory of the camera module based on the motion information of the camera module; and uses the motion trajectory of the camera module as the motion fuzzy kernel.

[0090] It is understandable that there is a certain amount of motion during the process of the camera module capturing the first image, which causes the first image to be blurred. This blur is called motion blur, and the motion blur in the first image can be characterized by the motion blur kernel.

[0091] Optionally, the electronic device acquires motion information of the camera module at preset intervals, and determines the position of the camera module at preset intervals based on the motion information at preset intervals; wherein, the motion between adjacent moments is considered to be unidirectional motion, and the positions are connected in chronological order to obtain the motion trajectory of the camera module.

[0092] Step 110: Use a motion blur kernel to deblur the first image to obtain the target image.

[0093] Optionally, the electronic device uses a patch-wise deconvolution algorithm, employing a motion blur kernel to deconvolve the first image to obtain the target image. The target image has higher sharpness than the first image.

[0094] The electronic device performs motion blur removal processing on the first image, which may be deconvolution processing or other processing methods, and is not limited here.

[0095] The aforementioned image processing method involves acquiring a first image through the camera module of an electronic device and obtaining target gyroscope data during the acquisition process using the target gyroscope of the electronic device. Based on this target gyroscope data, the motion information of the target gyroscope is determined. Then, according to the target conversion relationship between the target gyroscope and the camera module, the motion information of the target gyroscope can be converted into the motion information of the camera module. Based on this motion information of the camera module, a motion blur kernel can be generated to deblur the first image. This reduces the hardware requirements of the optical image stabilization motor and avoids increasing the size of the image sensor, thus lowering the hardware cost of image deblurring. Simultaneously, during the calibration process of the electronic device, the first gyroscope is located at the target gyroscope position within the electronic device, and the second gyroscope is located at the camera module position. Therefore, by using the gyroscope data from the first and second gyroscopes, the conversion relationship between the first and second gyroscopes—that is, the target conversion relationship between the target gyroscope and the camera module—can be accurately obtained. Based on this target conversion relationship, the first image can be deblurred more accurately, resulting in a clearer target image.

[0096] In one embodiment, such as Figure 3 As shown, a preview image is displayed on the user interface of the electronic device screen. After the user issues a shooting command through the user interface, generating a shutter signal, the camera module performs the exposure process based on the shutter signal, converting the light signal collected by the lens into an electrical signal and outputting it as raw image data (RAW file). This raw image data is processed by the image processing chip and output as a first image containing motion blur. Simultaneously, the target gyroscope collects raw gyroscope data based on the shutter signal. This raw gyroscope data is filtered to obtain target gyroscope data, which is then passed to the blur kernel generator to generate a motion blur kernel. If the output frequency of the target gyroscope is high and the amount of data collected is large, downsampling can be performed in time. The electronic device uses the motion blur kernel to remove motion blur from the first image, obtaining the target image, which is then displayed on the user interface.

[0097] In one embodiment, the imaging surface of the image sensor in the camera module is divided into at least two preset regions; based on the motion information of the camera module, a motion blur kernel is generated, including: for each preset region, based on the motion information of the camera module, determining the motion trajectory of a preset imaging point in the preset region, and generating a sub-motion blur kernel for the preset region based on the motion trajectory of the preset imaging point; and generating a motion blur kernel based on the sub-motion blur kernels of each preset region.

[0098] Understandably, for image sensors in camera modules, due to their large size, using the same motion blur kernel to describe and process motion blur across the entire imaging surface can easily lead to artifacts in the processed image. Therefore, the imaging surface of the image sensor can be segmented. For example, if the pixel resolution of the image sensor's imaging surface is 4000*3000, it can be divided into 40*30 preset regions. For each 100*100 pixel preset region, a motion blur kernel with a preset imaging point (such as the center point of the preset region) can be used as the motion blur kernel for that region. Correspondingly, if the pixel resolution of the image generated by the image sensor is 4000*3000, it can be divided into 40*30 blocks. For each 100*100 pixel block, a motion blur kernel with a feature point (such as the center point of the block) can be used as the convolution kernel required for the deblurring process. Therefore, the density of the motion blur kernel is lower than the image resolution, saving time in motion blur kernel computation.

[0099] The division method of the image sensor can be set as needed, and is not limited here.

[0100] In one alternative implementation, such as Figure 4 The diagram shows a uniform segmentation method, in which the electronic device can uniformly divide the imaging surface of the image sensor to obtain at least two preset regions.

[0101] In one alternative implementation, the electronic device can divide the region of interest in the imaging surface of the image sensor with a first density and divide the region of non-interest in the imaging surface of the image sensor with a second density to obtain at least two preset regions; the first density is higher than the second density.

[0102] The region of interest and the region of non-interest can both be set as needed. In one embodiment, such as... Figure 5 The diagram illustrates a centrally-oriented non-uniform segmentation method. The region of interest (ROI) is the central region of the image sensor's imaging surface, while the non-ROI regions are the edge regions of the image sensor's imaging surface. It's understandable that people often place the subject of interest in the center of the frame when shooting; therefore, the central region of the image sensor's imaging surface is the ROI. By dividing it with a high initial density, each preset region obtained from the central region can yield a sub-motion blur kernel with high precision.

[0103] In another embodiment, such as Figure 6 The diagram shows a non-uniform segmentation method, where the region of interest is the area where the subject is located, and the other areas are not of interest.

[0104] In other embodiments, the region of interest may be the area on the image sensor's imaging surface corresponding to the area where the user touches the screen, and the region of non-interest is the area on the screen that is not touched and corresponds to the area on the image sensor's imaging surface.

[0105] It is understandable that electronic devices can use a higher first density to divide the region of interest into blocks, which can obtain a sub-motion blur kernel with higher accuracy for each preset region, thereby reducing the occurrence of artifacts in the image region corresponding to the region of interest. On the other hand, using a lower density to divide the non-region of interest into blocks can improve the efficiency of the algorithm.

[0106] Optionally, for each preset region, the electronic device determines the motion trajectory of at least one preset imaging point within the preset region based on the motion information of the camera module, and generates a sub-motion blur kernel for the preset region based on the motion trajectory of the at least one preset imaging point; the sub-motion blur kernels of each preset region are then stitched together to obtain the motion blur kernel of the first image. The position of the preset imaging point within the preset region can be set as needed. For example, the preset imaging point can be the center point or an edge point within the preset region, and is not limited thereto.

[0107] Wherein, if the preset region includes one preset imaging point, the motion trajectory of the preset imaging point is used as the sub-motion blur kernel of the preset region; if the preset region includes at least two preset imaging points, the sub-motion blur kernel of the preset region is generated based on the motion trajectories of the at least two preset imaging points. Optionally, the electronic device may use the motion trajectory of any one of the at least two preset imaging points as the sub-motion blur kernel of the preset region, or it may average the motion trajectories of the at least two preset imaging points and use the averaged motion trajectory as the sub-motion blur kernel of the preset region, or it may use other methods to determine the sub-motion blur kernel of the preset region, which is not limited here.

[0108] The process of applying a motion blur kernel to a first image to obtain a target image includes: applying a sub-motion blur kernel to a corresponding image region in the first image to obtain a motion-blurred image region; and stitching together the motion-blurred image regions to obtain the target image.

[0109] In this embodiment, for each preset area, the electronic device determines the motion trajectory of a preset imaging point in the preset area based on the motion information of the camera module, and generates a sub-motion blur kernel for the preset area based on the motion trajectory of the preset imaging point. The sub-motion blur kernel of each preset area can more accurately characterize the motion blur of the image area obtained in the preset area, thereby obtaining a more accurate motion blur kernel.

[0110] In one embodiment, the motion information of the camera module is the motion information of the reference imaging point on the imaging surface of the image sensor in the camera module; for each preset area, based on the motion information of the camera module, the motion trajectory of the preset imaging point in the preset area is determined, including: for each preset area, determining the positional relationship between the preset imaging point and the reference imaging point in the preset area; based on the motion information and positional relationship of the reference imaging point, determining the position of the preset imaging point at preset time intervals; and generating the motion trajectory of the preset imaging point based on each position.

[0111] The reference imaging point can be set as needed. For example, the reference imaging point can be the center point of the image sensor's imaging surface, the edge point of the image sensor's imaging surface, or other points that can be set, without limitation here.

[0112] Optionally, for each preset area, based on the motion information and positional relationship of the reference imaging point, the motion information of the preset imaging point at each preset time interval is determined; based on the current position of the preset imaging point and the motion information after the preset time interval, the position at the next time interval is determined; and the positions at each preset time interval are connected to obtain the motion trajectory of the preset imaging point.

[0113] It is understandable that during the movement of an electronic device, the motion information of a preset imaging point can be mapped based on the motion information of a reference imaging point and the positional relationship between the reference imaging point and the preset imaging point.

[0114] like Figure 7 and Figure 8 As shown, taking the rotation angle θ of an image sensor around the Y-axis within a preset time Δt in an electronic device as an example, the reference imaging point in the imaging surface of the image sensor is the center point, which is located on the Y-axis. Therefore, the motion information of the center point is the rotation angle θ around the Y-axis. The positional relationship between the preset imaging point 1 and the center point in the preset region 1 of the image sensor imaging surface is: the distance in the X direction is Δx1, and the distance in the Y direction is Δy1. The positional relationship between the preset imaging point 2 and the center point in the preset region 2 of the image sensor imaging surface is: the distance in the X direction is Δx2, and the distance in the Y direction is Δy2. Therefore, the motion information of the preset imaging point 1 is: ΔL1 = Δx1 - Δx1·sinθ. The position of the preset imaging point 1 at the next moment is determined by moving ΔL1 from its current position. The two positions are then connected to obtain the motion trajectory of the preset imaging point 1, and this motion trajectory is used as the sub-motion blur kernel 1 of the preset region where the preset imaging point 1 is located. Motion information of preset imaging point 2: ΔL2=Δx2-Δx2·sinθ. The position of the preset imaging point 2 at the next moment is determined by moving ΔL2 from the current position. The two positions are then connected to obtain the motion trajectory of the preset imaging point 2. The motion trajectory of the preset imaging point 2 is used as the sub-motion blur kernel 2 of the preset region where the preset imaging point 2 is located.

[0115] It should be noted that the preset duration Δt is less than the shutter speed. The shutter speed is the exposure time for capturing the first image, which can ensure that a sufficiently fine motion blur kernel is recorded, and the first image can be deblurred more accurately.

[0116] In this embodiment, for each preset region, the positional relationship between the preset imaging point and the reference imaging point in the preset region is determined; based on the motion information and positional relationship of the reference imaging point, the position of the preset imaging point at each preset time interval can be determined; based on each position, the motion trajectory of the preset imaging point is generated, thereby accurately determining the motion trajectory of each preset imaging point on the imaging surface of the image sensor, that is, the sub-motion blur kernel of each preset region.

[0117] In one embodiment, generating a motion blur kernel based on the motion information of the camera module includes: generating a motion blur kernel component in the X direction based on the translation information, rotation information around the Y direction, and rotation information around the Z direction of the image sensor in the camera module; the planes containing the X and Y directions are parallel to the imaging surface of the image sensor, and the Z direction is perpendicular to the imaging surface of the image sensor; generating a motion blur kernel component in the Y direction based on the translation information, rotation information around the X direction, and rotation information around the Z direction of the image sensor in the camera module; and generating a motion blur kernel based on the motion blur kernel components in the X and Y directions.

[0118] Among them, the motion blur kernel component in the X direction and the motion blur kernel component in the Y direction are the motion blur generated when the image sensor moves laterally. That is, the planes in the X and Y directions are parallel to the imaging surface of the image sensor in the camera module, while the Z direction is perpendicular to the imaging surface of the image sensor.

[0119] It is understandable that rotation around the Y-axis (vertical direction) will cause the image point to move in the horizontal direction (X direction) orthogonal to it, while rotation around the X-axis will cause the image to move in the Y direction. Rotation around the Z-axis will cause movement in both the X and Y directions simultaneously. Therefore, in the three-dimensional coordinate system XYZ, based on the translation information in the X direction, the rotation information around the Y direction, and the rotation information around the Z direction of the image sensor in the camera module, the position of the camera module in the X direction at preset time intervals can be determined. Connecting the positions of the camera module in the X direction at preset time intervals yields the motion blur kernel component in the X direction.

[0120] Similarly, the translation, rotation around the X and Z directions of the image sensor in the Y direction can all affect the motion blur kernel component in the Y direction. Based on the translation information, rotation information around the X and Z directions of the image sensor in the camera module, the position of the camera module in the Y direction at each preset time interval can be determined. By connecting the positions of the camera module in the Y direction at each preset time interval, the motion blur kernel component in the Y direction can be obtained.

[0121] Electronic devices can generate a motion blur kernel by adding the motion blur kernel components in the X direction and the Y direction.

[0122] It is understandable that the defocus blur inherent in the image itself or the defocus blur caused by focusing (image blurring) is usually the image blur that is desired to be retained. The motion blur kernel component in the Z direction includes both defocus blur caused by motion and motion blur, with the defocus blur caused by motion being dominant. Since the depth of field of the camera module is large, the rotation and translation of the image sensor during the shooting process are unlikely to cause large defocus blur. Therefore, the electronic device can determine the motion blur kernel components in the X direction and the Y direction respectively, generate a motion blur kernel, and use this motion blur kernel to deblur the first image to obtain the target image after motion blur removal. This target image contains the defocus blur of the image itself that the user wants to retain, thus preserving the image blurring effect and improving the accuracy of image processing.

[0123] In another embodiment, generating a motion blur kernel based on the motion information of the camera module includes: generating an X-direction motion blur kernel component based on the translation information, rotation information around the Y-direction, and rotation information around the Z-direction of the image sensor in the camera module; the planes containing the X and Y directions are parallel to the imaging surface of the image sensor, and the Z direction is perpendicular to the imaging surface of the image sensor; generating a Y-direction motion blur kernel component based on the translation information, rotation information around the X-direction, and rotation information around the Z-direction of the image sensor in the camera module; generating a Z-direction motion blur kernel component based on the translation information, rotation information around the X-direction, and rotation information around the Y-direction of the image sensor in the camera module, as well as the change in scene depth or the change in focus position; and generating a motion blur kernel based on the X-direction motion blur kernel component, the Y-direction motion blur kernel component, and the Z-direction motion blur kernel component.

[0124] It is understandable that translation in the Z direction, rotation around the X direction, and rotation around the Y direction of the image sensor can all affect the motion blur kernel component in the Z direction. Furthermore, changes in scene depth or focus position can also affect the motion blur kernel component in the Z direction. Specifically, changes in scene depth refer to changes in the distance between the subject and the camera module, while changes in focus position refer to changes in the focus position of the camera module in the Z direction.

[0125] Therefore, based on the translation information in the Z direction, the rotation information around the X direction, and the rotation information around the Y direction of the image sensor in the camera module, as well as the change in scene depth or the change in focus position, the position of the camera module in the Z direction at each preset time interval can be determined. By connecting the positions of the camera module in the Z direction at each preset time interval, the motion blur kernel component in the Z direction can be obtained.

[0126] Electronic devices can generate a motion blur kernel by adding the motion blur kernel components in the X direction, Y direction, and Z direction.

[0127] In one embodiment, such as Figure 9 As shown, the electronic device acquires target gyroscope data through the target gyroscope. The target gyroscope data includes angular acceleration and translational acceleration. The angular acceleration is integrated once to obtain the angular velocity, and then the angular velocity is integrated again to obtain the rotation around the X-axis, the rotation around the Y-axis, and the rotation around the Z-axis. The translational acceleration is integrated once to obtain the velocity, and then the velocity is integrated again to obtain the translation along the X-axis, the translation along the Y-axis, and the translation along the Z-axis. The electronic device converts the motion information of the target gyroscope into the motion information of the camera module based on the target transformation relationship between the target gyroscope and the camera module. That is, it performs coordinate system transformation to obtain the rotation of the image sensor in the camera module around the X-axis, Y-axis, and Z-axis, as well as the translation along the X-axis, Y-axis, and Z-axis. Based on the translation information in the Z-direction, rotation information around the X-direction, and rotation information around the Y-direction of the image sensor in the camera module, as well as the changes in scene depth or focus position, the position of the camera module in the Z-direction at preset time intervals can be determined. Connecting the positions of the camera module in the Z-direction at preset time intervals yields the motion blur kernel component in the Z-direction. Adding the motion blur kernel components in the X-direction, Y-direction, and Z-direction vectors together generates the motion blur kernel.

[0128] In one embodiment, de-blurring a first image using a motion blur kernel to obtain a target image includes: processing the first image using a preset processing method to obtain a second image; convolving the second image using a motion blur kernel to obtain a third image; determining the difference between the third image and the first image; if the difference is greater than a preset threshold, adjusting the second image to obtain a new second image, and iteratively performing the step of convolving the second image using a motion blur kernel on the new second image until the difference is less than or equal to the preset threshold, and using the latest second image as the target image.

[0129] The preset processing method can be set as needed and is not limited here. The preset difference can also be set as needed and is not limited here. For example, the preset difference can be 0.

[0130] Optionally, the electronic device superimposes a random signal onto the first image to obtain a second image. It is understood that superimposing a random signal onto the first image makes the second image different from the first image, allowing the third image after the first iteration to have a significant difference from the original blurred image, thereby making some areas closer to the desired target image relative to the first image.

[0131] Alternatively, the electronic device may generate a second image through other preset processing methods, such as using some simple neural networks to process the first image to generate a deblurred second image.

[0132] The electronic device determines the difference between the third image and the first image. If the difference is greater than a preset threshold, the second image is adjusted to obtain a new second image. The new second image is then iteratively processed by convolving it with a motion blur kernel until the difference is less than or equal to the preset threshold. The latest second image is then used as the target image.

[0133] During the iteration process, which is the process of minimizing the difference, the second image obtained in the previous iteration will be adjusted in each iteration. For example, if the difference becomes smaller after increasing the pixel value in a certain block of the second image, the pixel value in that block will continue to increase in the next iteration.

[0134] In one embodiment, without considering noise, the image degradation model for deblurring the first image by the electronic device is as follows:

[0135] I blurred = ideal *SF motion

[0136] Where I ideal This refers to an ideal image without motion blur, which is also the target image desired after deblurring; I blurr=edThis refers to the first image, i.e., the image containing motion blur; PSF motion It refers to the point spread function that describes motion blur, which is the motion blur kernel generated from the target gyroscope data.

[0137] In actual shooting, the first image captured by the camera module is generated by convolving an ideal image with a motion blur kernel function. Therefore, for the convolution operation, an optimization method is used to solve the problem in reverse.

[0138] The electronic device convolves the second image with a motion blur kernel to obtain a third image, and determines the difference between the third image and the first image:

[0139] ΔI=I generated *SF motion -blurred

[0140] Among them, I generated It is the second image, I generated *SF motion It is the third image, I blurred ΔI is the first image, and ΔI is the difference between the third image and the first image.

[0141] By minimizing the difference until it is less than or equal to a preset threshold, the latest second image becomes the target image, thus eliminating motion blur in the first image.

[0142] In this embodiment, the electronic device processes the first image using a preset processing method to obtain the second image; it then performs convolution processing on the second image using a motion blur kernel to obtain the third image; it determines the difference between the third image and the first image, and if the difference is greater than a preset threshold, it adjusts the second image to obtain a new second image, and iteratively performs the step of convolution processing on the second image using a motion blur kernel on the new second image until the difference is less than or equal to the preset threshold. Then, the latest second image, which is also the target image, can accurately obtain the target image with motion blur eliminated, and the target image retains the out-of-focus blur, thus preserving the bokeh effect of the image.

[0143] In one embodiment, an image processing method is also provided, comprising the following steps:

[0144] Step A1: Acquire the first image through the camera module of the electronic device, and obtain target gyroscope data through the target gyroscope of the electronic device during the acquisition of the first image.

[0145] Step A2: Determine the motion information of the target gyroscope based on the target gyroscope data.

[0146] Step A3: Based on the target conversion relationship between the target gyroscope and the camera module, the motion information of the target gyroscope is converted into the motion information of the reference imaging point of the image sensor imaging surface in the camera module. The target conversion relationship is obtained by the electronic device during the calibration process based on the gyroscope data of the first gyroscope and the gyroscope data of the second gyroscope. The first gyroscope is located at the target gyroscope position in the electronic device, and the second gyroscope is located at the camera module position in the electronic device.

[0147] Step A4: The imaging surface of the image sensor in the camera module is divided into at least two preset regions; for each preset region, the positional relationship between a preset imaging point and a reference imaging point in the preset region is determined; based on the translation information, rotation information around the Y direction, and rotation information around the Z direction of the preset imaging point, and the positional relationship, the position of the preset imaging point in the X direction at preset time intervals is generated; based on the position of the preset imaging point in the X direction at preset time intervals, the motion blur kernel component of the preset imaging point in the X direction at preset time intervals is generated; based on the translation information, rotation information around the X direction, and rotation information around the Z direction of the preset imaging point, and the positional relationship, The system generates the position of a preset imaging point in the Y direction at preset time intervals; based on the position of the preset imaging point in the Y direction at preset time intervals, it generates motion blur kernel components in the Y direction at preset time intervals; based on the motion blur kernel components in the X and Y directions at preset time intervals, it generates motion blur kernels for the preset imaging point at preset time intervals; it integrates the motion blur kernels for the preset imaging point at preset time intervals over time to generate sub-motion blur kernels for preset regions; and it generates motion blur kernels based on the sub-motion blur kernels for each preset region; wherein the planes containing the X and Y directions are parallel to the imaging plane of the image sensor, and the Z direction is perpendicular to the imaging plane of the image sensor.

[0148] Step A5: Process the first image using a preset processing method to obtain the second image; perform convolution processing on the second image using a motion blur kernel to obtain the third image; determine the difference between the third image and the first image; if the difference is greater than a preset threshold, adjust the second image to obtain a new second image, and iteratively perform the step of convolution processing on the second image using a motion blur kernel on the new second image until the difference is less than or equal to the preset threshold, and use the latest second image as the target image.

[0149] In one embodiment, such as Figure 10 As shown, a calibration method is provided. This embodiment illustrates the application of this method to an electronic device, which can be a terminal or a server. It is understood that the method can also be applied to a system that includes a terminal and a server, and is implemented through the interaction between the terminal and the server.

[0150] In this embodiment, during the calibration process of the electronic device, a first gyroscope is provided at the target gyroscope position in the electronic device, and a second gyroscope is provided at the camera module position in the electronic device; the image processing method includes the following steps 1002 to 1004:

[0151] Step 1002: Acquire the first gyroscope data collected by the first gyroscope and the second gyroscope data collected by the second gyroscope when the electronic device is in motion.

[0152] Step 1004: Based on the data from the first gyroscope and the second gyroscope, obtain the conversion relationship between the first gyroscope and the second gyroscope; the conversion relationship is used by the electronic device to convert the motion information of the target gyroscope into the motion information of the camera module during use.

[0153] like Figure 11 As shown, during the calibration process, the electronic device is fixed on a motion platform capable of three-axis rotation and three-axis translation. A first gyroscope is installed at the target gyroscope position in the electronic device, and a second gyroscope is installed at the camera module position. By measuring the first gyroscope data and the second gyroscope data of the first gyroscope and the second gyroscope data respectively; the first gyroscope data includes the translation data of the first gyroscope on the X-axis, Y-axis, and Z-axis, as well as the rotation data around the X-axis, Y-axis, and Z-axis; the second gyroscope data includes the translation data of the second gyroscope on the X-axis, Y-axis, and Z-axis, as well as the rotation data around the X-axis, Y-axis, and Z-axis; based on the first and second gyroscope data, 36 linearly independent equations are established, and a transformation matrix R with 36 elements can be solved. This transformation matrix represents the transformation relationship between the first and second gyroscopes, and is used as the target transformation relationship between the target gyroscope and the camera module.

[0154] The electronic device can multiply the motion information of the target gyroscope by the transformation matrix R to obtain the motion information of the camera module:

[0155]

[0156] in, It is the transformation matrix R. It is the motion information of the target gyroscope. It is the motion information of the camera module.

[0157] Optionally, during the calibration of the electronic device, the rotating axis passes through the first gyroscope.

[0158] It is understandable that if the rotation axis passes through the first gyroscope, the input matrix composed of the data from the first gyroscope and the data from the second gyroscope will have fewer non-zero terms, which can simplify the solution of the matrix and determine the transformation matrix more quickly.

[0159] In the above calibration method, during the calibration process, the first gyroscope is located at the target gyroscope position within the electronic device, and the second gyroscope is located at the camera module position within the electronic device. Therefore, by using the gyroscope data collected by the first gyroscope and the second gyroscope during the movement of the electronic device, the conversion relationship between the first gyroscope and the second gyroscope can be accurately obtained, that is, the target conversion relationship between the target gyroscope and the camera module. Based on this target conversion relationship, the first image can be more accurately deblurred, resulting in a clearer target image.

[0160] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0161] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0162] In one embodiment, such as Figure 12 As shown, an image processing device is provided, including: a data acquisition module 1202, a motion information determination module 1204, a conversion module 1206, a blur kernel generation module 1208, and a deblurring module 1210, wherein:

[0163] The data acquisition module 1202 is used to acquire a first image through the camera module of the electronic device and to acquire target gyroscope data through the target gyroscope of the electronic device during the acquisition of the first image.

[0164] The motion information determination module 1204 is used to determine the motion information of the target gyroscope based on the target gyroscope data.

[0165] The conversion module 1206 is used to convert the motion information of the target gyroscope into the motion information of the camera module according to the target conversion relationship between the target gyroscope and the camera module. The target conversion relationship is obtained by the electronic device during the calibration process based on the gyroscope data of the first gyroscope and the gyroscope data of the second gyroscope. The first gyroscope is located at the target gyroscope position in the electronic device, and the second gyroscope is located at the camera module position in the electronic device.

[0166] The blur kernel generation module 1208 is used to generate a motion blur kernel based on the motion information of the camera module.

[0167] The deblurring module 1210 is used to perform motion blur removal on the first image using a motion blur kernel to obtain the target image.

[0168] The aforementioned image processing device acquires a first image through the camera module of an electronic device and obtains target gyroscope data during the acquisition process using the target gyroscope of the electronic device. Based on this target gyroscope data, the motion information of the target gyroscope is determined. Then, according to the target conversion relationship between the target gyroscope and the camera module, the motion information of the target gyroscope can be converted into the motion information of the camera module. Based on this motion information of the camera module, a motion blur kernel can be generated to deblur the first image. This reduces the hardware requirements of the optical image stabilization motor and avoids increasing the size of the image sensor, thus lowering the hardware cost of image deblurring. Simultaneously, during the calibration process of the electronic device, the first gyroscope is located at the target gyroscope position within the electronic device, and the second gyroscope is located at the camera module position. Therefore, by using the gyroscope data from the first and second gyroscopes, the conversion relationship between the first and second gyroscopes can be accurately obtained, i.e., the target conversion relationship between the target gyroscope and the camera module. Based on this target conversion relationship, the first image can be deblurred more accurately, resulting in a clearer target image.

[0169] In one embodiment, the imaging surface of the image sensor in the camera module is divided into at least two preset regions; the aforementioned blur kernel generation module 1208 is further configured to, for each preset region, determine the motion trajectory of a preset imaging point in the preset region based on the motion information of the camera module, and generate a sub-motion blur kernel for the preset region based on the motion trajectory of the preset imaging point; and generate a motion blur kernel based on the sub-motion blur kernels of each preset region.

[0170] In one embodiment, the motion information of the camera module is the motion information of the reference imaging point of the image sensor imaging surface in the camera module; the aforementioned blur kernel generation module 1208 is further configured to determine the positional relationship between the preset imaging point and the reference imaging point in each preset area; determine the position of the preset imaging point at each preset time interval based on the motion information and positional relationship of the reference imaging point; and generate the motion trajectory of the preset imaging point based on each position.

[0171] In one embodiment, the aforementioned blur kernel generation module 1208 is further configured to generate a motion blur kernel component in the X direction based on the translation information, rotation information around the Y direction, and rotation information around the Z direction of the image sensor in the camera module; the plane containing the X and Y directions is parallel to the imaging surface of the image sensor, and the Z direction is perpendicular to the imaging surface of the image sensor; a motion blur kernel component in the Y direction is generated based on the translation information, rotation information around the X direction, and rotation information around the Z direction of the image sensor in the camera module; and a motion blur kernel is generated based on the motion blur kernel component in the X direction and the motion blur kernel component in the Y direction.

[0172] In one embodiment, the deblurring module 1210 is further configured to process the first image using a preset processing method to obtain a second image; perform convolution processing on the second image using a motion blur kernel to obtain a third image; determine the difference between the third image and the first image; if the difference is greater than a preset threshold, adjust the second image to obtain a new second image, and iteratively perform the step of convolution processing on the second image using a motion blur kernel on the new second image until the difference is less than or equal to the preset threshold, and use the latest second image as the target image.

[0173] Based on the same inventive concept, this application also provides a calibration apparatus for implementing the calibration method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more calibration apparatus embodiments provided below can be found in the limitations of the calibration method described above, and will not be repeated here.

[0174] In one embodiment, such as Figure 13 As shown, a calibration device is provided for use in electronic devices. During the calibration process of the electronic device, a first gyroscope is installed at the target gyroscope position in the electronic device, and a second gyroscope is installed at the camera module position in the electronic device. The calibration device includes: a data acquisition module 1302 and a calibration module 1304, wherein:

[0175] The data acquisition module 1302 is used to acquire the first gyroscope data collected by the first gyroscope and the second gyroscope data collected by the second gyroscope when the electronic device is in motion.

[0176] The calibration module 1304 is used to calibrate the conversion relationship between the first gyroscope and the second gyroscope based on the data from the first gyroscope and the second gyroscope. The conversion relationship is used by the electronic device to convert the motion information of the target gyroscope into the motion information of the camera module during use.

[0177] In the aforementioned calibration device, during the calibration process of the electronic device, the first gyroscope is located at the target gyroscope position within the electronic device, and the second gyroscope is located at the camera module position within the electronic device. Therefore, by using the gyroscope data collected by the first gyroscope and the second gyroscope when the electronic device is in motion, the conversion relationship between the first gyroscope and the second gyroscope can be accurately obtained, that is, the target conversion relationship between the target gyroscope and the camera module. Based on this target conversion relationship, the first image can be more accurately deblurred to obtain a clearer target image.

[0178] In one embodiment, during the calibration of the electronic device, a rotating axis passes through a first gyroscope.

[0179] Each module in the aforementioned image processing and calibration devices can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the electronic device, or stored in the memory of the electronic device as software, so that the processor can call and execute the corresponding operations of each module.

[0180] In one embodiment, an electronic device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 14As shown, this electronic device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image processing method or calibration method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the electronic device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the electronic device, or external keyboards, touchpads, or mice, etc.

[0181] Those skilled in the art will understand that Figure 14 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0182] This application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, cause the processors to perform steps of an image processing method or steps of a calibration method.

[0183] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform an image processing method or a calibration method.

[0184] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0185] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0186] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0187] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, Applied to electronic devices, the method includes: The first image is acquired by the camera module of the electronic device, and the target gyroscope data is obtained by the target gyroscope of the electronic device during the acquisition of the first image. Based on the target gyroscope data, determine the motion information of the target gyroscope; Based on the target conversion relationship between the target gyroscope and the camera module, the motion information of the target gyroscope is converted into the motion information of the camera module. This target conversion relationship is obtained by the electronic device during calibration based on gyroscope data from a first gyroscope and a second gyroscope. The first gyroscope is located at the target gyroscope position within the electronic device, and the second gyroscope is located at the camera module position within the electronic device. The image sensor imaging surface in the camera module is divided into at least two preset regions. Each preset region includes a region of interest (ROI) and a region of non-ROI, with the ROI having a higher division density than the non-ROI. For each preset region, based on the motion information of the camera module, the motion trajectory of a preset imaging point in the preset region is determined, and based on the motion trajectory of the preset imaging point, a sub-motion blur kernel for the preset region is generated; Based on the sub-motion blur kernels of each preset region, a motion blur kernel is generated; The motion blur kernel is used to deblur the first image to obtain the target image.

2. The method according to claim 1, characterized in that, The motion information of the camera module is the motion information of the reference imaging point on the imaging surface of the image sensor in the camera module; The step of determining the motion trajectory of a preset imaging point within each preset region, based on the motion information of the camera module, includes: For each preset region, determine the positional relationship between the preset imaging point and the reference imaging point within the preset region; Based on the motion information of the reference imaging point and the positional relationship, the position of the preset imaging point is determined at preset time intervals; The motion trajectory of the preset imaging point is generated based on each location.

3. The method according to claim 1, characterized in that, The step of generating a motion blur kernel based on the motion information of the camera module includes: Based on the translation information in the X direction, the rotation information around the Y direction, and the rotation information around the Z direction of the image sensor in the camera module, a motion blur kernel component in the X direction is generated; the plane containing the X direction and the Y direction is parallel to the imaging surface of the image sensor, and the Z direction is perpendicular to the imaging surface of the image sensor; Based on the translation information in the Y direction, the rotation information around the X direction, and the rotation information around the Z direction of the image sensor in the camera module, a motion blur kernel component in the Y direction is generated. A motion blur kernel is generated based on the motion blur kernel components in the X direction and the motion blur kernel components in the Y direction.

4. The method according to any one of claims 1 to 3, characterized in that, The step of applying the motion blur kernel to the first image to obtain the target image includes: The first image is processed using a preset processing method to obtain the second image; The second image is convolved using the motion blur kernel to obtain the third image; The difference between the third image and the first image is determined. If the difference is greater than a preset threshold, the second image is adjusted to obtain a new second image. The new second image is then iteratively processed by performing convolution processing on the second image using the motion blur kernel until the difference is less than or equal to the preset threshold. The latest second image is then used as the target image.

5. A calibration method, characterized in that, Applied to electronic devices, in the process of calibrating the electronic device, a first gyroscope is provided at the target gyroscope position in the electronic device, and a second gyroscope is provided at the camera module position in the electronic device; the method includes: Acquire the first gyroscope data collected by the first gyroscope and the second gyroscope data collected by the second gyroscope when the electronic device is in motion; Based on the first gyroscope data and the second gyroscope data, a conversion relationship between the first gyroscope and the second gyroscope is obtained; the conversion relationship is used by the electronic device to convert the motion information of the target gyroscope into the motion information of the camera module during use; the imaging surface of the image sensor in the camera module is divided into at least two preset regions; the preset regions include regions of interest and regions of non-interest, and the division density of the regions of interest is higher than that of the regions of non-interest. The motion information is used to determine the motion trajectory of a preset imaging point in each preset region based on the motion information of the camera module, and to generate a sub-motion blur kernel for the preset region based on the motion trajectory of the preset imaging point. A motion blur kernel is then generated based on the sub-motion blur kernels of each preset region.

6. The method according to claim 5, characterized in that, During the calibration process of the electronic device, the rotating shaft passes through the first gyroscope.

7. An image processing apparatus, characterized in that, Applied to electronic devices, the device includes: The data acquisition module is used to acquire a first image through the camera module of the electronic device, and to acquire target gyroscope data through the target gyroscope of the electronic device during the acquisition of the first image. The motion information determination module is used to determine the motion information of the target gyroscope based on the target gyroscope data; A conversion module is used to convert the motion information of the target gyroscope into the motion information of the camera module based on the target conversion relationship between the target gyroscope and the camera module. The target conversion relationship is obtained by the electronic device during calibration based on gyroscope data from a first gyroscope and a second gyroscope. The first gyroscope is located at the target gyroscope position within the electronic device, and the second gyroscope is located at the camera module position within the electronic device. The image sensor imaging surface in the camera module is divided into at least two preset regions. Each preset region includes a region of interest (ROI) and a region of non-ROI, with the ROI having a higher division density than the non-ROI. The blur kernel generation module is used to determine the motion trajectory of a preset imaging point in each preset region based on the motion information of the camera module, and generate a sub-motion blur kernel for the preset region based on the motion trajectory of the preset imaging point; and generate a motion blur kernel based on the sub-motion blur kernels of each preset region. The deblurring module is used to deblur the first image using the motion blur kernel to obtain the target image.

8. A calibration device, characterized in that, Applied to electronic devices, during the calibration process of the electronic device, a first gyroscope is provided at the target gyroscope position in the electronic device, and a second gyroscope is provided at the camera module position in the electronic device; the device includes: The data acquisition module is used to acquire the first gyroscope data collected by the first gyroscope and the second gyroscope data collected by the second gyroscope when the electronic device is in motion. A calibration module is used to calibrate the conversion relationship between the first gyroscope and the second gyroscope based on the first gyroscope data and the second gyroscope data; the conversion relationship is used by the electronic device to convert the motion information of the target gyroscope into the motion information of the camera module during use; the imaging surface of the image sensor in the camera module is divided into at least two preset regions; the preset regions include regions of interest and regions of non-interest, and the division density of the regions of interest is higher than that of the regions of non-interest; The motion information is used to determine the motion trajectory of a preset imaging point in each preset region based on the motion information of the camera module, and to generate a sub-motion blur kernel for the preset region based on the motion trajectory of the preset imaging point. A motion blur kernel is then generated based on the sub-motion blur kernels of each preset region.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, the processor performs the steps of the image processing method as described in any one of claims 1 to 4, or the steps of the calibration method as described in any one of claims 5 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the image processing method as described in any one of claims 1 to 4, or the steps of the calibration method as described in any one of claims 5 to 6.

Citation Information

Patent Citations

  • Image data recovery method and device

    CN107395961A

  • Attitude measurement method of noninertial system vision and dual-gyroscope multirate CKF fusion

    CN107782309A

  • Image reconstruction method and device and electronic equipment

    CN114862680A