Calibration method, target calibration equipment, device, electronic device and storage medium

Abstract

The present application relates to a calibration method, target calibration device, apparatus, electronic device, storage medium, and computer program product. The method comprises: determining a first blur kernel of the camera module based on a first image; the first image is obtained by the camera module when the electronic device is calibrated by a target calibration device; determining a second blur kernel of the gyroscope based on gyroscope data corresponding to the first image; calibrating a conversion relationship between the gyroscope and the camera module based on the first blur kernel and the second blur kernel; the conversion relationship is used to convert the blur kernel of the gyroscope to the blur kernel of the camera module. This method can improve calibration accuracy.

Description

Technical Field

[0001] The present application relates to the field of imaging technology, and in particular to a calibration method, target calibration equipment, apparatus, electronic equipment, and computer-readable storage medium. Background Art

[0002] When electronic devices are shooting, especially in night scenes or low-light environments, images can be blurry due to hand shaking. To address this issue, gyroscope data is often used to compensate for blurry images and produce clearer images. However, this image processing method requires pre-modeling based on the electronic device's structural design drawings to estimate the relationship between the gyroscope and the camera module.

[0003] However, there will be certain tolerances when assembling electronic devices. The relative positions of the gyroscope and the camera module in each electronic device are different. Using the same set of design parameters for calibration will result in large errors. When calibrating the relationship between the gyroscope and the camera module, there is a problem of low accuracy. Summary of the Invention

[0004] Embodiments of the present application provide a calibration method, target calibration device, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of calibration.

[0005] In a first aspect, the present application provides a calibration method. The method comprises:

[0006] Determining a first blur kernel of the camera module based on a first image, wherein the first image is obtained by the camera module when the electronic device is calibrated by a target calibration device;

[0007] determining a second blur kernel of the gyroscope based on gyroscope data corresponding to the first image;

[0008] Based on the first blur kernel and the second blur kernel, a conversion relationship between the gyroscope and the camera module is calibrated; the conversion relationship is used to convert the blur kernel of the gyroscope into the blur kernel of the camera module.

[0009] In a second aspect, the present application further provides a target calibration device. The target calibration device comprises:

[0010] A point light source, used to emit light and project it onto a camera module in an electronic device;

[0011] A fixture, placed on the motion platform, for fixing the electronic device; and

[0012] The motion platform is used to control the movement of the fixture to adjust the shooting distance between the electronic device and the point light source.

[0013] In a third aspect, the present application further provides a calibration device. The device comprises:

[0014] A first determination module is configured to determine a first blur kernel of the camera module based on a first image, wherein the first image is obtained by the camera module when the electronic device is calibrated by a target calibration device;

[0015] a second determining module, configured to determine a second blur kernel of the gyroscope based on gyroscope data corresponding to the first image;

[0016] A calibration module is used to calibrate the conversion relationship between the gyroscope and the camera module based on the first blur kernel and the second blur kernel; the conversion relationship is used to convert the blur kernel of the gyroscope into the blur kernel of the camera module.

[0017] In a fourth aspect, the present application further provides an electronic device. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0018] Determining a first blur kernel of the camera module based on a first image, wherein the first image is obtained by the camera module when the electronic device is calibrated by a target calibration device;

[0019] determining a second blur kernel of the gyroscope based on gyroscope data corresponding to the first image;

[0020] Based on the first blur kernel and the second blur kernel, a conversion relationship between the gyroscope and the camera module is calibrated; the conversion relationship is used to convert the blur kernel of the gyroscope into the blur kernel of the camera module.

[0021] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0022] Determining a first blur kernel of the camera module based on a first image, wherein the first image is obtained by the camera module when the electronic device is calibrated by a target calibration device;

[0023] determining a second blur kernel of the gyroscope based on gyroscope data corresponding to the first image;

[0024] Based on the first blur kernel and the second blur kernel, a conversion relationship between the gyroscope and the camera module is calibrated; the conversion relationship is used to convert the blur kernel of the gyroscope into the blur kernel of the camera module.

[0025] In a sixth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0026] Determining a first blur kernel of the camera module based on a first image, wherein the first image is obtained by the camera module when the electronic device is calibrated by a target calibration device;

[0027] determining a second blur kernel of the gyroscope based on gyroscope data corresponding to the first image;

[0028] Based on the first blur kernel and the second blur kernel, a conversion relationship between the gyroscope and the camera module is calibrated; the conversion relationship is used to convert the blur kernel of the gyroscope into the blur kernel of the camera module.

[0029] The above-mentioned calibration method, target calibration device, apparatus, electronic device, computer-readable storage medium, and computer program product determine the first blur kernel of the camera module based on the first image, which is obtained by the camera module when the electronic device is calibrated by the target calibration device; determine the second blur kernel of the gyroscope based on the gyroscope data corresponding to the first image; then, based on the first blur kernel of the camera module and the second blur kernel of the gyroscope, the conversion relationship between the gyroscope and the camera module can be calibrated, avoiding the problem of large errors caused by calibration using preset design parameters, and improving the accuracy of calibrating the conversion relationship between the gyroscope and the camera module. Moreover, the conversion relationship between the gyroscope and the camera module can also be achieved when the relative position coordinates of the gyroscope and the camera module are unknown. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0031] Figure 1 is a flow chart of a calibration method in one embodiment;

[0032] Figure 2 A schematic diagram of an electronic device determining a second blur kernel of a gyroscope in one embodiment;

[0033] Figure 3 Schematic diagram of acquiring a first image of a central field of view by a camera module in one embodiment;

[0034] Figure 4 Schematic diagram of acquiring a first image of a peripheral field of view by a camera module in one embodiment;

[0035] Figure 5 A schematic diagram of an electronic device capturing a photo in a stationary state and extracting a first image from a second image in one embodiment;

[0036] Figure 6 A flowchart of deblurring an actual captured image based on gyroscope data in one embodiment;

[0037] Figure 7 is a flow chart of a calibration method in another embodiment;

[0038] Figure 8 Schematic diagram of an electronic device calibrating a blur kernel of a central field of view corresponding to a first calibration device in one embodiment;

[0039] Figure 9 Schematic diagram of an electronic device calibrating a blur kernel of a peripheral field of view corresponding to a first calibration device in one embodiment;

[0040] Figure 10 A schematic diagram of an electronic device calibrating a blur kernel of a preset calibration field of view corresponding to a second calibration device in one embodiment;

[0041] Figure 11 is a structural block diagram of a calibration device in one embodiment;

[0042] Figure 12 FIG. 1 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0044] In one embodiment, Figure 1As shown, a calibration method is provided. This embodiment uses the method applied to an electronic device including a camera module and a gyroscope as an example. The electronic device can be a terminal or a server. It can be understood that the method can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. Among them, the terminal can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, smart cars, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The service can be implemented using an independent server or a server cluster consisting of multiple servers.

[0045] In this embodiment, the method includes the following steps 102 to 106:

[0046] Step 102: determining a first blur kernel of the camera module based on the first image; the first image is obtained by the camera module when the electronic device is calibrated by a target calibration device.

[0047] A camera module is a module that contains multiple camera components. These components can include a lens, image sensor, VCM (Voice Coil Motor) / mount, filter, circuit board, etc. The image sensor converts light signals entering the electronic device into electrical signals to generate images, while the gyroscope captures gyroscope data, including angular acceleration and linear acceleration.

[0048] A blur kernel is actually a matrix. Convolving a clear image with the blur kernel causes the image to become blurred, hence the name blur kernel. A blur kernel is a type of convolution kernel, and image convolution is essentially a matrix convolution. In other words, image blur can be viewed as the process of convolving a clear image with the blur kernel to produce a blurred image. The camera module's first blur kernel can include motion blur caused by the camera module when capturing the first image and optical blur caused by the camera module's own optical aberrations.

[0049] The point spread function (PSF) describes the response of an optical system to a point impulse function during the imaging process in the spatial domain. It can be used to describe the imaging effect of the system, that is, the blur kernel of the system.

[0050] The target calibration device is used to calibrate the first blur kernel of the camera module. The target calibration device may include devices such as a point light source, a fixture, a motion platform, and an optical vibration isolation platform.

[0051] Optionally, when the electronic device is calibrated by the target calibration device, the point light source in the target calibration device is photographed by the camera module to obtain a first image, and the first image is determined as the first blur kernel of the camera module.

[0052] It can be understood that under ideal circumstances, the image obtained by the camera module when shooting a point light source is a point. However, due to the movement of the camera module during the shooting process and the optical aberration of the camera module itself, under actual circumstances, the image obtained by the camera module when shooting a point light source is not a point, that is, a blurred image. The blurred image can represent the first blur kernel of the camera module.

[0053] Step 104 : Determine a second blur kernel of the gyroscope based on the gyroscope data corresponding to the first image.

[0054] The second blur kernel of the gyroscope is motion blur generated by the movement of the gyroscope during the shooting time of the first image.

[0055] Optionally, the electronic device determines the coordinate position of the gyroscope at each preset time interval based on the gyroscope data corresponding to the first image, and integrates each coordinate position over time to obtain a second blur kernel of the gyroscope.

[0056] The electronic device can obtain multiple sets of gyroscope data within the shooting time of the first image, each set of gyroscope data includes the angular acceleration and linear acceleration of the gyroscope; the angular acceleration is integrated twice to obtain the rotation angle of the gyroscope, and the linear acceleration is integrated twice to obtain the movement amount of the gyroscope; based on the rotation angle and movement amount of the gyroscope each time, the coordinate position of the gyroscope each time can be determined.

[0057] like Figure 2 The figure shows a schematic diagram of the electronic device determining the second blur kernel of the gyroscope. The electronic device determines the coordinate position of the gyroscope at times t1, t2, t3, ..., tn, and integrates each coordinate position over time to obtain the second blur kernel of the gyroscope.

[0058] Step 106 , calibrating a conversion relationship between the gyroscope and the camera module based on the first blur kernel and the second blur kernel; the conversion relationship is used to convert the blur kernel of the gyroscope into the blur kernel of the camera module.

[0059] The conversion relationship between the gyroscope and the camera module can be a conversion matrix, a conversion formula, etc., but is not limited thereto.

[0060] Optionally, the electronic device sets a conversion variable, subtracts the second blur kernel of the gyroscope from the first blur kernel of the camera module and multiplies it by the conversion variable, and uses the least squares method to fit the minimum difference, and uses the conversion variable corresponding to the minimum difference as the conversion matrix.

[0061] The calibration method described above determines the first blur kernel of the camera module based on a first image, which is obtained by the camera module when the electronic device is calibrated by the target calibration device. The second blur kernel of the gyroscope is determined based on the gyroscope data corresponding to the first image. Based on the first blur kernel of the camera module and the second blur kernel of the gyroscope, the conversion relationship between the gyroscope and the camera module can be calibrated. This avoids the problem of large errors caused by calibration using preset design parameters and improves the accuracy of the calibration of the conversion relationship between the gyroscope and the camera module. Furthermore, the conversion relationship between the gyroscope and the camera module can be achieved even when the relative position coordinates of the gyroscope and the camera module are unknown.

[0062] In one embodiment, a method for acquiring a first image includes: determining a target calibration device from at least two candidate calibration devices based on a preset shooting distance; the shooting distance is the distance between a camera module and a point light source in the target calibration device; when the electronic device is calibrated by the target calibration device, acquiring a first image of a preset calibration field of view through the camera module.

[0063] The target calibration device includes a point light source, which can be, for example, an optical fiber or a point light source composed of a collimator and a star point plate.

[0064] The field of view represents the maximum range that the camera module can observe, usually expressed in degrees. The larger the field of view, the larger the observation range. The calibration field of view is the field of view for electronic equipment calibration. The calibration field of view can include the center field of view, edge field of view, and other fields of view. Figure 3 The figure shows a schematic diagram of obtaining the first image of the central field of view through the camera module, that is, parallel light is vertically incident on the camera module and projected onto the imaging surface of the image sensor through the lens, so that the central field of view can be calibrated. Figure 4 The figure shows a schematic diagram of obtaining the first image of the edge field of view through the camera module, that is, parallel light is incident on the camera module obliquely and vertically, and is projected onto the imaging surface of the image sensor through the lens, so that the edge field of view can be calibrated.

[0065] It is understandable that when the electronic device is calibrated by the target calibration device in different fields of view, the blur kernel of the camera module obtained is different. Therefore, it is necessary to obtain the first image for different preset calibration fields of view to calibrate the different preset calibration fields of view.

[0066] Optionally, based on a preset shooting distance, a target calibration device is determined from at least two candidate calibration devices, including: if the preset shooting distance is greater than a preset distance threshold, a first calibration device is determined from at least two candidate calibration devices; the first calibration device belongs to the target calibration device, and the point light source in the first calibration device includes a parallel light tube and a star point plate, the parallel light tube is connected to the star point plate, and emits parallel light through the hole on the star point plate to project onto the camera module.

[0067] The preset distance threshold can be set as needed. For example, the preset distance threshold can be 1 m (meter).

[0068] The collimator is used to emit parallel light. The star point plate has a hole at its center. It can be understood that the collimator emits parallel light through the hole in the star point plate, acting as a point light source. The diameter of the hole in the star point plate is smaller than a preset threshold.

[0069] It is understandable that the image of a point light source on the camera module through an ideal optical system cannot exceed 2 pixels in size. According to the formula for calculating the magnification of the image:

[0070]

[0071] Where u is the object distance, v is the image distance, l is the object height (i.e., the diameter of the hole on the star point plate), and l′ is the image height. Assuming that the size of a single pixel on the imaging surface of the camera module is p, the diameter l of the hole on the star point plate can be obtained as:

[0072]

[0073]

[0074] Among them, obtaining the first image of the preset calibration field of view through the camera module includes: obtaining the preset calibration field of view; after adjusting the camera module to the angle corresponding to the preset calibration field of view, obtaining the first image of the preset calibration field of view through the camera module.

[0075] Optionally, based on a preset shooting distance, a target calibration device is determined from at least two candidate blur core calibration devices, including: if the preset shooting distance is less than or equal to a preset distance threshold, a second calibration device is determined from at least two candidate blur core calibration devices; the second calibration device belongs to the target calibration device, and the point light source in the second calibration device includes an optical fiber, which emits light and projects it onto the camera module.

[0076] Among them, the core diameter of the optical fiber should meet the following requirements:

[0077]

[0078] u is the object distance, v is the image distance, l is the object height, i.e. the core diameter of the optical fiber, l′ is the image height, and p is the size of a single pixel on the imaging surface of the camera module.

[0079] Among them, the second calibration device also includes an optical fiber bracket for fixing the optical fibers of at least two calibration fields of view; obtaining a first image of a preset calibration field of view through a camera module, including: shooting through the camera module to obtain a second image containing at least two calibration fields of view; extracting the first image of the preset calibration field of view from the second image.

[0080] It can be understood that at least two optical fibers with calibrated fields of view can be fixed on the optical fiber bracket at the same time; by photographing at least two optical fibers with a camera module, a second image containing at least the calibrated field of view can be obtained; from the second image, the first image of the preset calibrated field of view can be extracted.

[0081] like Figure 5 The figure shows a schematic diagram of an electronic device capturing images while stationary and extracting a first image from a second image. An optical fiber holder holds eight optical fibers with calibrated fields of view, with 502 representing one of the calibrated fields of view. The electronic device captures the image through a camera module, obtaining a second image 504 containing the eight calibrated fields of view. From second image 504, a first image 506 of the preset calibrated field of view corresponding to optical fiber 502 is extracted.

[0082] Optionally, the electronic device uses the area where the preset calibration field of view is located from the second image as the first image. Optionally, the electronic device extracts an image of a point light source of the preset calibration field of view from the second image as the first image.

[0083] It is understandable that when an electronic device is calibrated by a target calibration device, it needs to be calibrated for different calibration fields of view. At different shooting distances, the calibration methods for different calibration fields of view are different. Therefore, different calibration devices need to be designed for different shooting distances. Based on the preset shooting distance, the target calibration device that matches the shooting distance can be accurately determined from at least two candidate calibration devices, so that when the electronic device is calibrated by the target calibration device, a more accurate first image of the preset calibration field of view can be obtained through the camera module.

[0084] Furthermore, if the preset shooting distance is greater than the preset distance threshold, after adjusting the camera module to the angle corresponding to the preset calibration field of view, the first image of the preset calibration field of view can be accurately captured by the camera module.

[0085] If the preset shooting distance is less than or equal to the preset distance threshold, the camera module is used to shoot a second image containing at least two calibrated fields of view, and then the first image of the preset calibrated field of view is extracted from the second image. Then, the second image can calibrate the at least two calibrated fields of view, avoiding shooting an image once for each calibrated field of view, which can improve the calibration efficiency of the first blur kernel of the camera module and save resources of the electronic device.

[0086] In one embodiment, the conversion relationship between the gyroscope and the camera module is calibrated according to the first blur kernel and the second blur kernel, including: obtaining the first blur kernel and the second blur kernel corresponding to the preset calibration field of view and the preset shooting distance; the shooting distance is the distance between the camera module and the point light source in the target calibration device, and the calibration field of view and the shooting distance constitute a pair of calibration parameters; for each pair of calibration parameters, the conversion relationship between the gyroscope and the camera module is calibrated according to the first blur kernel and the second blur kernel.

[0087] A calibration field of view and a shooting distance constitute a pair of calibration parameters. For example, the preset calibration fields of view include A1, A2, and A3, and the preset shooting distances include B1 and B2. Five pairs of calibration parameters can be constructed: (A1, B1), (A1, B2), (A2, B1), (A2, B2), (A3, B1), and (A3, B2).

[0088] In this embodiment, the electronic device can obtain the first blur kernel and the second blur kernel corresponding to each pair of calibration parameters, and calibrate each pair of calibration parameters to accurately obtain the conversion relationship between the gyroscope and the camera module corresponding to each pair of calibration parameters.

[0089] In one embodiment, the conversion relationship between the gyroscope and the camera module is calibrated based on the first blur kernel and the second blur kernel, including: obtaining a first blur kernel data set composed of multiple first blur kernels, and obtaining a second blur kernel data set composed of multiple second blur kernels; based on the first blur kernel data set and the second blur kernel data set, fitting a conversion matrix between the gyroscope and the camera module; the conversion matrix is ​​used to convert the blur kernel of the gyroscope into the blur kernel of the camera module.

[0090] The number of blur kernels of the first blur kernel in the first blur kernel data set and the number of blur kernels of the second blur kernel in the second blur kernel data set are the same, and the number of blur kernels is greater than the number of pixels of the first blur kernel or the second blur kernel.

[0091] Exemplarily, the size of the first blur kernel or the second blur kernel is m*n, that is, m pixels per row and n pixels per column. Then, for a pair of calibration parameters, the number of measurements required, that is, the number of blur kernel measurements N, is greater than m*n.

[0092] Optionally, the electronic device converts each first blur kernel into a vector, and forms a first blur kernel data set with the multiple first blur kernels; converts each second blur kernel into a vector, and forms a second blur kernel data set with the multiple second blur kernels.

[0093] Exemplarily, the sizes of the first blur kernel p and the second blur kernel p′ are both m*n matrices. The first blur kernel p and the second blur kernel p′ are rearranged and converted into vectors (1, m*n). Multiple first blur kernels p constitute a first blur kernel data set P of (N, m*n), and multiple second blur kernels p′ constitute a second blur kernel data set P′ of (N, m*n).

[0094] Optionally, the electronic device uses a least squares method to fit a conversion matrix between the gyroscope and the camera module based on the first blur kernel data set and the second blur kernel data set, and the conversion matrix satisfies:

[0095] argmin∑(PP′T) 2

[0096] Where P is the first fuzzy kernel data set, P′ is the second fuzzy kernel data set, and T is the transformation matrix.

[0097] For each pair of calibration parameters, the corresponding transformation matrix can be calibrated using the above method.

[0098] In this embodiment, a first blur kernel data set consisting of multiple first blur kernels is obtained, and a second blur kernel data set consisting of multiple second blur kernels is obtained; based on the first blur kernel data set and the second blur kernel data set, the conversion matrix between the gyroscope and the camera module can be accurately fitted.

[0099] In one embodiment, Figure 6 As shown, after calibrating the conversion relationship between the gyroscope and the camera module according to the first blur kernel and the second blur kernel, the method further includes:

[0100] Step 602: Acquire a third image of the camera module in the actual shooting scene and gyroscope data corresponding to the third image.

[0101] The actual shooting scene may be a scene actually shot by the electronic device other than the calibration scene. For example, the actual shooting scene may be a scene such as shooting scenery outdoors or shooting a user indoors.

[0102] Step 604 : Determine a third blur kernel of the gyroscope based on the gyroscope data corresponding to the third image.

[0103] The third blur kernel of the gyroscope is motion blur generated by the movement of the gyroscope during the shooting time of the third image.

[0104] Optionally, the electronic device determines the coordinate position of the gyroscope at each preset time interval based on the gyroscope data corresponding to the third image, and integrates each coordinate position over time to obtain a third blur kernel of the gyroscope.

[0105] The electronic device can obtain multiple sets of gyroscope data within the shooting time of the third image, each set of gyroscope data includes the angular acceleration and linear acceleration of the gyroscope; the angular acceleration is integrated twice to obtain the rotation angle of the gyroscope, and the linear acceleration is integrated twice to obtain the movement amount of the gyroscope; based on the rotation angle and movement amount of the gyroscope each time, the coordinate position of the gyroscope each time can be determined.

[0106] Step 606: Convert the third blur kernel of the gyroscope into the fourth blur kernel of the camera module according to the conversion relationship.

[0107] Optionally, the electronic device substitutes the third blur kernel of the gyroscope into the conversion relationship between the gyroscope and the camera module to obtain the fourth blur kernel of the camera module.

[0108] Furthermore, the conversion relationship is a conversion matrix, and the electronic device multiplies the third blur kernel of the gyroscope by the conversion matrix to obtain the fourth blur kernel of the camera module.

[0109] pr=pr′*T

[0110] Where pr is the fourth blur kernel of the camera module, pr′ is the third blur kernel of the gyroscope, and T is the transformation matrix.

[0111] Step 608 : Deblurring the third image using a fourth blur kernel to obtain a fourth image.

[0112] Optionally, the electronic device may use a patch-wise deconvolution algorithm to deconvolve the third image using a fourth blur kernel to obtain a fourth image, wherein the fourth image has a higher definition than the third image.

[0113] In this embodiment, the electronic device obtains a third image of the camera module in the actual shooting scene, and gyroscope data corresponding to the third image; based on the gyroscope data corresponding to the third image, the third blur kernel of the gyroscope is determined; then, according to the conversion relationship, the third blur kernel of the gyroscope can be converted into the fourth blur kernel of the camera module, and the third image is deblurred using the fourth blur kernel to obtain a clearer fourth image.

[0114] In one implementation, the above method also includes: when the camera module is in a stationary state in the target calibration device, obtaining the fifth blur kernel corresponding to the camera module; using the fifth blur kernel to deconvolve the first blur kernel to obtain a processed first blur kernel; calibrating the conversion relationship between the gyroscope and the camera module based on the first blur kernel and the second blur kernel, including: calibrating the conversion relationship between the gyroscope and the camera module based on the processed first blur kernel and the second blur kernel.

[0115] Optionally, the camera module determines a fifth blur kernel of the camera module based on the fifth image; the fifth image is obtained by the camera module when the electronic device is in a stationary state and is calibrated by the target calibration device.

[0116] It can be understood that when the camera module is in a stationary state in the target calibration device, the fifth blur kernel corresponding to the camera module is obtained. The fifth blur kernel includes the optical blur corresponding to the optical aberration of the camera module itself, but does not include the motion blur of the camera module.

[0117] Therefore, the electronic device uses the fifth blur kernel to perform deconvolution processing on the first blur kernel, which can eliminate the optical blur corresponding to the optical aberration of the camera module itself in the first blur kernel, and obtain a more accurately processed first blur kernel. Therefore, based on the processed first blur kernel and second blur kernel, the conversion relationship between the gyroscope and the camera module can be more accurately calibrated.

[0118] In one embodiment, Figure 7 As shown in the figure, another calibration method is provided: select a preset shooting distance, select a target calibration device based on the shooting distance, and select a calibration field of view; the calibration field of view and the shooting distance constitute a pair of calibration parameters; for each pair of calibration parameters, repeat N times to collect data samples. The process of each data sample collection is as follows:

[0119] When an electronic device is calibrated by a target calibration device, a first image is obtained through a camera module, and the six-axis translation stage is controlled to move randomly during the shooting process of the camera module; based on the first image, a first blur kernel of the camera module is determined; when the camera module is in a stationary state in the target calibration device, a fifth blur kernel corresponding to the camera module is obtained, and the fifth blur kernel is used to deconvolve the first blur kernel to remove lens blur in the first blur kernel, thereby obtaining a processed first blur kernel; based on the gyroscope data corresponding to the first image, a motion trajectory recorded by the gyroscope is obtained, and based on the motion trajectory, a second blur kernel of the gyroscope is determined.

[0120] The electronic device collects data samples N times to obtain the first blur kernels of the N camera modules and the second blur kernels of the N gyroscopes; the first blur kernels of the N camera modules constitute a first blur kernel data set, and the second blur kernels of the N gyroscopes constitute a second blur kernel data set; based on the first blur kernel data set and the second blur kernel data set, the least squares method is used to fit the conversion matrix between the gyroscope and the camera module.

[0121] In one embodiment, a calibration method is provided for an electronic device including a camera module and a gyroscope. The calibration method includes the following steps:

[0122] The electronic device executes step A1 or step A2.

[0123] Step A1: if the preset shooting distance is greater than the preset distance threshold, a first calibration device is determined from at least two candidate calibration devices, and the first calibration device belongs to the target calibration device; when the electronic device is calibrated by the target calibration device, a preset calibration field of view is obtained; after the camera module is adjusted to an angle corresponding to the preset calibration field of view, a first image of the preset calibration field of view is obtained by photographing the camera module; wherein, the shooting distance is the distance between the camera module and the point light source in the target calibration device, and the point light source in the first calibration device includes a parallel light tube and a star point plate, the parallel light tube is connected to the star point plate, and emits parallel light through the hole on the star point plate to project onto the camera module.

[0124] Step A2: If the preset shooting distance is less than or equal to the preset distance threshold, a second calibration device is determined from at least two candidate blur kernel calibration devices, and the second calibration device belongs to the target calibration device; shooting is performed through the camera module to obtain a second image containing at least two calibration fields of view; from the second image, a first image of the preset calibration field of view is extracted; wherein, the second calibration device also includes an optical fiber and an optical fiber bracket, the optical fiber emits light and projects it onto the camera module, and the optical fiber bracket is used to fix the optical fiber of at least two calibration fields of view.

[0125] The electronic device continues to execute steps A3 to A8.

[0126] Step A3: Determine a first blur kernel of the camera module based on the first image.

[0127] In step A4, when the camera module is in a stationary state in the target calibration device, a fifth blur kernel corresponding to the camera module is obtained; and the first blur kernel is deconvolved using the fifth blur kernel to obtain a processed first blur kernel.

[0128] Step A5: determining a second blur kernel of the gyroscope based on the gyroscope data corresponding to the first image.

[0129] Step A6: Obtain the preset calibration field of view and the preset shooting distance, the corresponding second blur kernel and the processed first blur kernel; the shooting distance is the distance between the camera module and the point light source in the target calibration device, and the calibration field of view and the shooting distance constitute a pair of calibration parameters.

[0130] Step A7, for each pair of calibration parameters, obtain a first blur kernel data set consisting of multiple processed first blur kernels, and obtain a second blur kernel data set consisting of multiple second blur kernels; based on the first blur kernel data set and the second blur kernel data set, fit the conversion matrix between the gyroscope and the camera module; the conversion matrix is ​​used to convert the blur kernel of the gyroscope into the blur kernel of the camera module.

[0131] Step A8, obtain a third image of the camera module in the actual shooting scene, and the gyroscope data corresponding to the third image; determine the third blur kernel of the gyroscope based on the gyroscope data corresponding to the third image; convert the third blur kernel of the gyroscope into the fourth blur kernel of the camera module according to the conversion matrix; use the fourth blur kernel to deblur the third image to obtain a fourth image.

[0132] In one embodiment, a target calibration device is characterized in that the target calibration device includes: a point light source for emitting light to project onto a camera module in an electronic device; a clamp placed on a moving platform for fixing the electronic device; and a moving platform for controlling the movement of the clamp to adjust the shooting distance between the electronic device and the point light source.

[0133] A point light source is a light source idealized as a point mass. The motion stage consists of a six-axis translation stage and guide rails. The fixture is placed on the six-axis translation stage, which is then placed on the guide rails. The six-axis translation stage controls the fixture's movement along the X, Y, and Z axes. The guide rails control the six-axis translation stage to adjust the shooting distance between the electronic device and the point light source. The six-axis translation stage can control the fixture's movement and / or rotation along the X, Y, and Z axes.

[0134] Optionally, the point light source includes a parallel light tube and a star point plate; the parallel light tube is connected to the star point plate, and emits parallel light through the hole on the star point plate to project onto the camera module.

[0135] Optionally, the point light source includes an optical fiber; the optical fiber emits light and projects it onto the camera module.

[0136] Furthermore, the target calibration device also includes an optical vibration isolation platform for receiving the motion platform. The optical vibration isolation platform is used to eliminate vibrations generated in the target calibration device.

[0137] Figure 8 Schematic diagram of an electronic device calibrating a blur kernel of a central field of view corresponding to a first calibration device in an embodiment. Figure 9 This is a schematic diagram of an electronic device calibrating the blur kernel of the edge field of view corresponding to a first calibration device in one embodiment. When the electronic device calibrates the blur kernel of the center field of view corresponding to the first calibration device, the electronic device faces the star point plate so that light enters the camera module perpendicularly. When the electronic device calibrates the blur kernel of the edge field of view corresponding to the first calibration device, the electronic device adjusts its angle so that light enters the camera module obliquely. If the shooting distance is greater than a preset distance threshold, the first calibration device is selected for calibration, where the preset distance threshold is 1 meter.

[0138] The first calibration device includes: a fixture, placed on a six-axis translation stage, used to fix the electronic device; a collimator and a star point plate, the collimator is connected to the star point plate, and emits parallel light through the hole on the star point plate to project onto the camera module of the electronic device; the six-axis translation stage is placed on a guide rail, used to control the movement of the fixture on the X-axis, Y-axis and Z-axis; the guide rail is used to control the six-axis translation stage to adjust the shooting distance between the electronic device and the point light source, so as to adjust the shooting distance between the electronic device and the point light source; an optical vibration isolation platform, which supports the guide rail and is used to eliminate the vibration generated in the first calibration device.

[0139] The first calibration device includes: a fixture, placed on a six-axis translation stage, used to fix the electronic device; a collimator and a star point plate, the collimator is connected to the star point plate, and emits parallel light through the hole on the star point plate to project onto the camera module of the electronic device; the six-axis translation stage is placed on a guide rail, used to control the movement of the fixture on the X-axis, Y-axis and Z-axis; the guide rail is used to control the six-axis translation stage to adjust the shooting distance between the electronic device and the point light source, so as to adjust the shooting distance between the electronic device and the point light source; an optical vibration isolation platform, which supports the guide rail and is used to eliminate the vibration generated in the first calibration device.

[0140] Figure 10 Figure 1 is a schematic diagram illustrating an electronic device calibrating a blur kernel for a preset calibration field of view corresponding to a second calibration device, in one embodiment. The second calibration device is used for calibration if the shooting distance is less than or equal to a preset distance threshold. The preset calibration distance is 1 meter, and a shooting distance less than or equal to the preset distance threshold indicates close focus.

[0141] The second calibration device includes: a fixture, placed on a six-axis translation stage, used to fix the electronic device; an optical fiber bracket, used to fix at least two optical fibers; an optical fiber, which emits light and projects it onto the camera module of the electronic device; the six-axis translation stage is placed on a guide rail, used to control the movement of the fixture on the X-axis, Y-axis and Z-axis; the guide rail is used to control the six-axis translation stage to adjust the shooting distance between the electronic device and the point light source to adjust the shooting distance between the electronic device and the point light source; an optical vibration isolation platform, which supports the guide rail and is used to eliminate the vibration generated in the first calibration device.

[0142] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0143] Based on the same inventive concept, the present application also provides a calibration device for implementing the aforementioned calibration method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more calibration device embodiments provided below can be found in the above-mentioned limitations of the calibration method and will not be repeated here.

[0144] In one embodiment, Figure 11 As shown, a calibration device is provided, including: a first determination module 1102, a second determination module 1104 and a calibration module 1106, wherein:

[0145] The first determination module 1102 is used to determine a first blur kernel of the camera module based on the first image; the first image is obtained by the camera module when the electronic device is calibrated by the target calibration device.

[0146] The second determining module 1104 is configured to determine a second blur kernel of the gyroscope based on the gyroscope data corresponding to the first image.

[0147] The calibration module 1106 is used to calibrate the conversion relationship between the gyroscope and the camera module based on the first blur kernel and the second blur kernel; the conversion relationship is used to convert the blur kernel of the gyroscope into the blur kernel of the camera module.

[0148] The above-mentioned calibration device determines the first blur kernel of the camera module based on the first image, and the first image is obtained through the camera module when the electronic device is calibrated by the target calibration device; based on the gyroscope data corresponding to the first image, the second blur kernel of the gyroscope is determined; then, according to the first blur kernel of the camera module and the second blur kernel of the gyroscope, the conversion relationship between the gyroscope and the camera module can be calibrated, avoiding the problem of large errors caused by calibration using preset design parameters, and can improve the accuracy of calibration of the conversion relationship between the gyroscope and the camera module.

[0149] In one embodiment, the above-mentioned device also includes an image acquisition module; the image acquisition module is used to determine a target calibration device from at least two candidate calibration devices based on a preset shooting distance; the shooting distance is the distance between the camera module and the point light source in the target calibration device; when the electronic device is calibrated by the target calibration device, the first image of the preset calibration field of view is acquired through the camera module.

[0150] In one embodiment, the above-mentioned image acquisition module is also used to determine a first calibration device from at least two candidate calibration devices if the preset shooting distance is greater than a preset distance threshold; the first calibration device belongs to the target calibration device, and the point light source in the first calibration device includes a parallel light tube and a star point plate, the parallel light tube is connected to the star point plate, and emits parallel light through the hole on the star point plate to project onto the camera module.

[0151] In one embodiment, the image acquisition module is further used to acquire a preset calibration field of view; after adjusting the camera module to an angle corresponding to the preset calibration field of view, the first image of the preset calibration field of view is captured by the camera module.

[0152] In one embodiment, the above-mentioned image acquisition module is also used to determine a second calibration device from at least two candidate blur kernel calibration devices if the preset shooting distance is less than or equal to a preset distance threshold; the second calibration device belongs to the target calibration device, and the point light source in the second calibration device includes an optical fiber, which emits light and projects it onto the camera module.

[0153] In one embodiment, the second calibration device also includes an optical fiber bracket for fixing the optical fibers of at least two calibration fields of view; the above-mentioned image acquisition module is also used to shoot through a camera module to obtain a second image containing at least two calibration fields of view; and extract the first image of the preset calibration field of view from the second image.

[0154] In one embodiment, the calibration module 1106 is also used to obtain a preset calibration field of view and a preset shooting distance, and the corresponding first blur kernel and second blur kernel; the shooting distance is the distance between the camera module and the point light source in the target calibration device, and the calibration field of view and the shooting distance constitute a pair of calibration parameters; for each pair of calibration parameters, the conversion relationship between the gyroscope and the camera module is calibrated according to the first blur kernel and the second blur kernel.

[0155] In one embodiment, the calibration module 1106 is also used to obtain a first blur kernel data set consisting of multiple first blur kernels, and to obtain a second blur kernel data set consisting of multiple second blur kernels; based on the first blur kernel data set and the second blur kernel data set, a conversion matrix between the gyroscope and the camera module is fitted; the conversion matrix is ​​used to convert the blur kernel of the gyroscope into the blur kernel of the camera module.

[0156] In one embodiment, the above-mentioned device also includes a deblurring module; the above-mentioned image acquisition module is also used to obtain a third image of the camera module in the actual shooting scene, and the gyroscope data corresponding to the third image; the above-mentioned second determination module 1104 is also used to determine the third blur kernel of the gyroscope based on the gyroscope data corresponding to the third image; the above-mentioned first determination module 1102 is also used to convert the third blur kernel of the gyroscope into a fourth blur kernel of the camera module according to the conversion relationship; the above-mentioned deblurring module is also used to use the fourth blur kernel to deblur the third image to obtain a fourth image.

[0157] In one embodiment, the above-mentioned device also includes a deblurring module; the above-mentioned first determination module 1102 is also used to obtain the fifth blur kernel corresponding to the camera module when the camera module is in a stationary state in the target calibration device; the deblurring module is used to use the fifth blur kernel to perform deconvolution processing on the first blur kernel to obtain a processed first blur kernel; the above-mentioned calibration module 1106 is also used to calibrate the conversion relationship between the gyroscope and the camera module based on the processed first blur kernel and the second blur kernel.

[0158] Each module in the calibration device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in an electronic device in hardware form, or can be stored in a memory in the electronic device in software form, so that the processor can call and execute the corresponding operations of each module.

[0159] In one embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 12As shown. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a calibration method is implemented. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device casing, or an external keyboard, touchpad or mouse.

[0160] Those skilled in the art will understand that Figure 12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0161] The present application also provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, cause the processors to perform the steps of the calibration method.

[0162] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the calibration method.

[0163] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0164] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0165] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A calibration method, characterized in that: Applied to an electronic device including a camera module and a gyroscope, the method includes: Determining a first blur kernel of the camera module based on a first image, wherein the first image is obtained by the camera module when the electronic device is calibrated by a target calibration device; determining a second blur kernel of the gyroscope based on gyroscope data corresponding to the first image; Based on the first blur kernel and the second blur kernel, a conversion relationship between the gyroscope and the camera module is calibrated; the conversion relationship is used to convert the blur kernel of the gyroscope into the blur kernel of the camera module.

2. The method according to claim 1, characterized in that The method for acquiring the first image includes: Determining a target calibration device from at least two candidate calibration devices based on a preset shooting distance; the shooting distance is the distance between the camera module and a point light source in the target calibration device; When the electronic device is calibrated by the target calibration device, a first image of a preset calibration field of view is acquired through the camera module.

3. The method according to claim 2, characterized in that The step of determining a target calibration device from at least two candidate calibration devices based on a preset shooting distance includes: If the preset shooting distance is greater than the preset distance threshold, a first calibration device is determined from at least two candidate calibration devices; the first calibration device belongs to the target calibration device, and the point light source in the first calibration device includes a parallel light tube and a star point plate, the parallel light tube is connected to the star point plate, and emits parallel light through the hole on the star point plate to project onto the camera module.

4. The method according to claim 3, characterized in that The step of acquiring a first image of a preset calibration field of view by the camera module includes: Get the preset calibration field of view; After the camera module is adjusted to an angle corresponding to the preset calibration field of view, a first image of the preset calibration field of view is obtained by photographing with the camera module.

5. The method according to claim 2, characterized in that The step of determining a target calibration device from at least two candidate blur kernel calibration devices based on a preset shooting distance includes: If the preset shooting distance is less than or equal to the preset distance threshold, a second calibration device is determined from at least two candidate blur kernel calibration devices; the second calibration device belongs to the target calibration device, and the point light source in the second calibration device includes an optical fiber, which emits light and projects it onto the camera module.

6. The method according to claim 5, characterized in that The second calibration device further includes an optical fiber holder for fixing at least two optical fibers of the calibration field of view; the first image of the preset calibration field of view is obtained by the camera module, including: Shooting with the camera module to obtain a second image including at least two calibrated fields of view; A first image of a preset calibration field of view is extracted from the second image.

7. The method according to claim 1, characterized in that The calibrating the conversion relationship between the gyroscope and the camera module according to the first blur kernel and the second blur kernel includes: Obtaining a preset calibration field of view and a preset shooting distance, and corresponding first and second blur kernels; the shooting distance is the distance between the camera module and a point light source in the target calibration device, and the calibration field of view and the shooting distance constitute a pair of calibration parameters; For each pair of calibration parameters, a conversion relationship between the gyroscope and the camera module is calibrated according to the first blur kernel and the second blur kernel.

8. The method according to any one of claims 1 to 7, characterized in that The calibrating the conversion relationship between the gyroscope and the camera module according to the first blur kernel and the second blur kernel includes: Acquire a first fuzzy kernel data set consisting of a plurality of first fuzzy kernels, and acquire a second fuzzy kernel data set consisting of a plurality of second fuzzy kernels; Based on the first blur kernel data set and the second blur kernel data set, a conversion matrix between the gyroscope and the camera module is fitted; the conversion matrix is ​​used to convert the blur kernel of the gyroscope into the blur kernel of the camera module.

9. The method according to any one of claims 1 to 7, characterized in that After calibrating the conversion relationship between the gyroscope and the camera module according to the first blur kernel and the second blur kernel, the method further includes: Acquire a third image of the camera module in the actual shooting scene, and gyroscope data corresponding to the third image; determining a third blur kernel of the gyroscope based on gyroscope data corresponding to the third image; According to the conversion relationship, the third blur kernel of the gyroscope is converted into a fourth blur kernel of the camera module; The fourth blur kernel is used to perform deblurring processing on the third image to obtain a fourth image.

10. The method according to any one of claims 1 to 7, characterized in that The method further comprises: When the camera module is in a stationary state in the target calibration device, obtaining a fifth blur kernel corresponding to the camera module; performing deconvolution processing on the first blur kernel using the fifth blur kernel to obtain a processed first blur kernel; The calibrating the conversion relationship between the gyroscope and the camera module according to the first blur kernel and the second blur kernel includes: The conversion relationship between the gyroscope and the camera module is calibrated according to the processed first blur kernel and the second blur kernel.

11. A calibration device, characterized in that: Applicable to an electronic device including a camera module and a gyroscope, the device includes: A first determination module is configured to determine a first blur kernel of the camera module based on a first image, wherein the first image is obtained by the camera module when the electronic device is calibrated by a target calibration device; a second determining module, configured to determine a second blur kernel of the gyroscope based on gyroscope data corresponding to the first image; A calibration module is used to calibrate the conversion relationship between the gyroscope and the camera module based on the first blur kernel and the second blur kernel; the conversion relationship is used to convert the blur kernel of the gyroscope into the blur kernel of the camera module.

12. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the calibration method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images