Parameter calibration method, storage medium, coprocessor chip and electronic device

Abstract

Embodiments of the present application disclose a parameter calibration method of a camera module, a storage medium, a coprocessor chip and an electronic device. In the embodiments, a first image output by a camera module of an electronic device is acquired, and a current calibration parameter of the camera module is acquired. A depth map is obtained according to the first image and the current calibration parameter. Whether the accuracy of the depth map is lower than a preset threshold is detected. When the accuracy of the depth map is lower than the preset threshold, the camera module is calibrated to obtain a new calibration parameter, and the current calibration parameter is updated based on the new calibration parameter. By using the scheme of the embodiments of the present application, the abnormality of the camera module can be detected in time, and the calibration parameter of the camera module can be calibrated when the abnormality is detected, thereby improving the accuracy of image processing of the electronic device.

Description

Technical Field

[0001] This application relates to the field of electronic equipment technology, specifically to a parameter calibration method for a camera module, a storage medium, a coprocessing chip, and an electronic device. Background Technology

[0002] With the continuous development of smart terminal technology, the use of electronic devices (such as smartphones and tablets) is becoming increasingly widespread. Most electronic devices are equipped with camera modules. Generally, before an electronic device leaves the factory, the camera module is calibrated to obtain calibration parameters, which are then stored in the device. However, during the use of electronic devices, incidents such as drops and collisions may occur, or the camera may age over time. These phenomena can lead to inaccurate calibration parameters. When the calibration parameters are inaccurate, subsequent image processing algorithms will experience a decline in image processing quality when using these parameters for image synthesis, blurring, and other processing. Summary of the Invention

[0003] This application provides a parameter calibration method, storage medium, coprocessing chip, and electronic device for a camera module, which can detect and calibrate the accuracy of calibration parameters and improve image processing performance.

[0004] In a first aspect, embodiments of this application provide a parameter calibration method for a camera module, including: Acquire the first image output by the camera module of the electronic device; A depth map is obtained based on the first image and the current calibration parameters; Detect whether the accuracy of the depth map is lower than a preset threshold; When the accuracy of the depth map is lower than a preset threshold, the camera module is calibrated to obtain new calibration parameters, and the current calibration parameters are updated based on the new calibration parameters.

[0005] Secondly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when run on a computer, causes the computer to execute the parameter calibration method for a camera module as provided in any embodiment of this application.

[0006] Thirdly, embodiments of this application also provide a coprocessor chip, including a central processing unit, which is used to execute the parameter calibration method for a camera module as provided in any embodiment of this application.

[0007] Fourthly, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores a computer program, characterized in that the processor, by calling the computer program, is used to execute a parameter calibration method for a camera module as provided in any embodiment of this application.

[0008] The technical solution provided in this application embodiment acquires a first image output by the camera module of an electronic device, obtains a depth map using the first image and the current calibration parameters of the camera module, detects the accuracy of the depth map, and confirms whether it is below a preset threshold. When the accuracy of the depth map is below the preset threshold, it can be determined that the camera module may have aging or other abnormal phenomena. The camera module is then calibrated to obtain new calibration parameters, and the current calibration parameters are updated based on these new parameters. The solution of this application embodiment can detect abnormalities in the camera module in a timely manner, and this detection operation can be performed in the background without manual triggering by the user, offering a certain degree of convenience. Furthermore, when an abnormality is detected, the calibration parameters of the camera module can be calibrated, thereby improving the image processing effect of the electronic device. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of the first process of the parameter calibration method for the camera module provided in the embodiments of this application.

[0011] Figure 2 This is a schematic diagram of a second process for a parameter calibration method for a camera module provided in an embodiment of this application.

[0012] Figure 3 A schematic diagram illustrating the process of extracting edge pixels in the parameter calibration method for the camera module provided in this application embodiment.

[0013] Figure 4 This is a schematic diagram of the structure of the coprocessor chip provided in an embodiment of this application.

[0014] Figure 5 This is a schematic diagram of a first structure of an electronic device provided in an embodiment of this application.

[0015] Figure 6 This is a schematic diagram of a second structure of an electronic device provided in an embodiment of this application.

[0016] Figure 7 This is a schematic diagram of a third structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0018] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0019] This application provides a parameter calibration method for a camera module. The subject executing this parameter calibration method can be the parameter calibration device for the camera module provided in this application, or an electronic device integrating the parameter calibration device for the camera module. The parameter calibration device for the camera module can be implemented in hardware or software. The electronic device can be a smartphone, tablet computer, PDA, laptop computer, or desktop computer, etc.

[0020] Please see Figure 1 , Figure 1 This is a schematic diagram of the first flowchart of the parameter calibration method for a camera module provided in this application embodiment. The specific flow of the parameter calibration method for a camera module provided in this application embodiment can be as follows: 101. Obtain the first image output by the camera module and obtain the current calibration parameters of the camera module.

[0021] Generally, various parameters, such as the calibration parameters of the camera module, are calibrated when electronic devices leave the factory. However, as electronic devices are used, situations such as replacing the camera module or a specific camera within the camera module, or the camera aging after prolonged use, may occur, leading to inaccuracies in the calibration parameters.

[0022] In image measurement and machine vision applications, to determine the 3D geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of camera imaging must be established. These geometric model parameters are the camera's calibration parameters. Calibration parameters are obtained through calibration calculations on the camera. These parameters include the camera module's intrinsic and extrinsic parameters, as well as distortion parameters. The process of solving for these parameters is called camera calibration. By determining the calibration parameters through camera calibration, lens distortion can be corrected during image processing, generating a corrected image. Alternatively, a 3D scene can be reconstructed from the image obtained using these parameters.

[0023] The calibration parameters vary depending on the type of camera module.

[0024] For a monocular camera, calibration parameters include intrinsic parameters, extrinsic parameters, and distortion parameters. Intrinsic parameters include characteristics specific to the camera itself, such as its focal length and principal point coordinates; these parameters are only related to the camera itself. Extrinsic parameters refer to the camera's position in space, generally its rotation vector R and translation vector T within a reference coordinate system. Distortion parameters refer to the deviation between the actual pixel position of an object point in the image and the theoretical projection point calculated based on the imaging model during image capture; this deviation is generally described by radial distortion parameters k1, k2, k3 and tangential distortion parameters p1, p2.

[0025] For multi-camera systems, there are also three types of parameters: intrinsic parameters, extrinsic parameters, and distortion parameters. The intrinsic and distortion parameters are the same as those for monocular cameras. The difference is that the extrinsic parameters include not only the spatial position of each camera but also the extrinsic parameters between different cameras. The extrinsic parameters between cameras describe the relative position and orientation of one camera with respect to another, and are generally represented by translation vectors and rotation vectors, such as Tx, Ty, Tz and Rx, Ry, Rz.

[0026] With the development of cameras, most camera modules in current electronic devices are now multi-camera systems. Electronic devices activate multiple cameras within the camera module to capture images in various modes, and then synthesize these multi-frame images to obtain the output image for the corresponding shooting mode. For example, in portrait mode, when capturing a specific subject, other objects are blurred. This requires calculating a depth map, which uses the calibration parameters of the camera module. Inaccurate calibration parameters can lead to poor blurring effects, such as false blurring. Therefore, the accuracy of the calibration parameters affects the final image quality of the captured image. In cases of camera aging, the calibration parameters need to be calibrated.

[0027] Based on this, the solution in this application embodiment detects the accuracy of the camera to determine whether the camera module has aging or other abnormal phenomena, and then determines whether the calibration parameters of the camera module need to be calibrated.

[0028] When the camera module malfunctions due to aging or collisions, the depth map calculated by the electronic device will be inaccurate. Therefore, the calibration algorithm in this application detects the accuracy of the depth map and determines whether the preset parameters need to be calibrated based on the detection results.

[0029] First, the original image output by the camera module, i.e., the first image, is acquired. In this embodiment, acquiring the first image can be achieved in several ways. For example, a calibration function can be set for the camera module. After the user enables the calibration function, the electronic device opens the camera application and enters shooting mode. After the user triggers the shooting command, the first image output by the camera module is acquired. Alternatively, in another embodiment, after entering shooting mode, the electronic device instructs the user to shoot a scene with a clear outline in a certain brightness environment to improve the accuracy of depth map detection.

[0030] Alternatively, in another embodiment, at preset time intervals, when a user is detected taking a photo using the camera module, the original image output by the camera module is stored in the background and recorded as the first image.

[0031] In some embodiments, the camera module includes multiple cameras; acquiring the first image output by the camera module and acquiring the current calibration parameters of the camera module includes: acquiring the first image output by each camera in the camera module to obtain multiple first images.

[0032] Since the accuracy of depth map detection requires the use of images captured by each camera, when the camera module includes multiple cameras, it is necessary to obtain the first image output by each camera in the camera module.

[0033] In addition, the electronic device also acquires the current calibration parameters of the camera module for use in depth map calculation.

[0034] 102. Based on the first image and the current calibration parameters, obtain the depth map.

[0035] A depth map is calculated based on the first image and the acquired current calibration parameters. The depth map contains depth information, which refers to the distance between the object being photographed and the camera in the image.

[0036] The method for calculating depth maps differs depending on the type of camera module. For a monocular camera, the camera module captures the same scene from at least two angles to obtain at least two first images. Feature point pairs between the two first images are calculated using a feature point matching algorithm. The depth map is then calculated based on the matched feature point pairs and the current calibration parameters.

[0037] For multi-camera systems, multiple cameras in the camera module are used to capture images, resulting in multiple first-frame images. Feature point pairs between every two first-frame images are calculated using a feature point matching algorithm. A depth map is then calculated based on the matched feature point pairs and calibration parameters.

[0038] 103. Check whether the accuracy of the depth map is lower than the preset threshold.

[0039] After calculating the depth map, its accuracy is evaluated. Low accuracy indicates significant errors in edge information. Based on this principle, edge pixels in the depth map can be extracted for detection.

[0040] For example, in one embodiment, the step of detecting whether the accuracy of the depth map is lower than a preset threshold may include: performing a composite processing on multiple first images to obtain a composite image; processing the composite image to obtain a grayscale image of the composite image; performing edge detection processing on the grayscale image to obtain first edge pixels, and performing edge detection processing on the depth map to obtain second edge pixels; calculating the grayscale value difference between the second edge pixels and the first edge pixels, determining the accuracy of the depth map based on the grayscale value difference, wherein the accuracy value is inversely proportional to the absolute value of the grayscale value difference; and detecting whether the accuracy is lower than a preset threshold.

[0041] In this embodiment, edge pixels in the grayscale image are used as a reference to detect whether the accuracy of the depth map is lower than a preset threshold. Since inaccurate calibration parameters caused by camera aging do not affect the grayscale information of the image itself, the acquired first image can be processed to obtain the corresponding grayscale image. For example, the first image can be converted to a YUV format image. YUV is an image format that represents chroma and luminance separately, where Y is the luminance signal and U and V are the chroma signals. YUV images contain grayscale information, and can be converted to grayscale images by setting the values ​​of the U and V channels in the YUV image. When the camera module contains multiple cameras, in one embodiment, any frame of the first image can be converted to obtain a YUV image, and then a grayscale image can be obtained from the YUV image; or, in another embodiment, multiple frames of the first image can be synthesized to obtain a composite image. Then, the composite image is converted to obtain a YUV image, and then a grayscale image is obtained from the YUV image.

[0042] After obtaining the grayscale image, edge detection is performed to obtain the first edge pixels. Edge detection is then performed on the depth map to obtain the second edge pixels. The purpose of using edge detection algorithms on the image is to identify points with significant brightness changes, i.e., edge pixels. This significantly reduces the image data volume, extracts irrelevant information, and preserves important interface attributes of the image. Next, the grayscale difference between the second and first edge pixels is calculated. Since the depth map and grayscale image correspond to the same scene, if the accuracy of the depth map is high, the edge information in the depth map is not significantly different from that in the grayscale image. Therefore, the accuracy of the depth map can be determined based on the difference between the second and first edge pixels. Since the number of first and second edge pixels may be large, the difference between each first edge pixel and the corresponding second edge pixel is calculated, resulting in multiple differences. The maximum, average, or median of these differences is then used as the difference between the second and first edge pixels.

[0043] The grayscale difference between the second edge pixel and the first edge pixel is calculated, and the accuracy of the depth map is determined based on this grayscale value. For example, in one embodiment, this grayscale difference is used as the accuracy of the depth map. In another embodiment, a preset accuracy corresponding to the calculated grayscale difference is determined based on a preset relationship between a preset grayscale difference and a preset accuracy, and this preset accuracy is used as the accuracy of the depth map. The accuracy is inversely proportional to the absolute value of the grayscale difference; that is, the larger the absolute value of the grayscale difference, the greater the difference between the edge information in the depth map and the edge information in the grayscale map, and thus the lower the accuracy of the depth map.

[0044] 104. When the accuracy of the depth map is lower than the preset threshold, the camera module is calibrated to obtain new calibration parameters, and the current calibration parameters are updated based on the new calibration parameters.

[0045] After calculating the accuracy of the depth map, it is determined whether the accuracy is lower than a preset threshold. If so, it indicates that there is a large error in the calibration parameters of the camera module, and it needs to be calibrated.

[0046] Next, the calibration parameters are calibrated. In one embodiment, when the accuracy of the depth map is lower than a preset threshold, the step of calibrating the preset parameters of the camera module may include: when the accuracy of the depth map is lower than the preset threshold, acquiring multiple frames of calibration images captured by multiple cameras; performing corner detection and corner matching processing on the multiple frames of calibration images to obtain multiple corner pairs; when the number of corner pairs is greater than a preset number, calibrating the camera module based on the multiple frames of calibration images to obtain new calibration parameters; and updating the current calibration parameters based on the new calibration parameters.

[0047] In this embodiment, when the accuracy of the depth map is detected to be lower than a preset threshold, a prompt message is displayed on the graphical user interface, prompting the user to take a calibration image. Before the phone leaves the factory, a specific calibration board is generally used to calibrate the camera module, such as a checkerboard calibration board. However, users may find it difficult to provide images taken with a specific calibration board. In this case, the user can be reminded to take images of scenes with clearly defined object outlines, sufficient lighting, or significant color differences. Alternatively, in another embodiment, an image of a checkerboard calibration board is provided, prompting the user to print the image of the calibration board according to a specific size to obtain the calibration image.

[0048] After the user performs the shooting operation, multiple frames of images captured by multiple cameras are acquired as calibration images. Next, corner detection and corner matching are performed on these calibration images to obtain multiple corner pairs. It is determined whether the number of corner pairs exceeds a preset number. If so, the camera module is calibrated according to preset calibration parameters to obtain new calibration parameters. For example, the new calibration parameters can be calculated using Zhang's calibration method. Conversely, if the detected corners do not meet the above requirements, the user is prompted to retake the calibration images.

[0049] In some embodiments, after obtaining new calibration parameters, the current calibration parameters can be updated based on the new calibration parameters, and the new calibration parameters can be written to the corresponding location in the memory.

[0050] In some embodiments, when the accuracy of the depth map is lower than a preset threshold, the camera module is calibrated to obtain new calibration parameters, and the current calibration parameters are updated based on the new calibration parameters. This includes: calculating the difference between the new calibration parameters and the current calibration parameters; and updating the current calibration parameters based on the new calibration parameters when the absolute value of the difference is greater than a preset difference. When the absolute value of the difference between the new calibration parameters and the current calibration parameters is less than the preset difference, updating the calibration parameters has little impact on the accuracy of the output image of the camera module. Considering that calibrating the camera module based on user-captured images may still have some error compared to calibrating the camera module in the laboratory, and the absolute value of the difference between the new calibration parameters and the current calibration parameters is small, this small difference may be within such an error range, in which case the current calibration parameters do not need to be updated. Conversely, when the absolute value of the difference between the new calibration parameters and the current calibration parameters is greater than the preset difference, and the absolute value of this large difference is likely to have exceeded such an error range, the current calibration parameters can be updated based on the new calibration parameters, and the new calibration parameters are written to the corresponding location in the memory.

[0051] In one embodiment, the solution of this application can be applied to an application processing chip in an electronic device. The application processing chip determines the accuracy of the depth map, calculates new calibration parameters when the accuracy is low, and updates the current calibration parameters based on the new calibration parameters.

[0052] In another embodiment, to improve the image processing capabilities of the electronic device, a coprocessor chip is added between the application processing chip and the camera module. This coprocessor chip can be used to perform some preprocessing on the image. The solution of this application embodiment can be applied to the coprocessor chip of the electronic device. The coprocessor chip determines the accuracy of the depth map, calculates new calibration parameters when the accuracy is low, and updates the current calibration parameters based on the new calibration parameters. Alternatively, after determining the accuracy of the depth map and calculating new calibration parameters when the accuracy is low, the coprocessor chip sends the new calibration parameters to the application processing chip of the electronic device, so that the application processing chip calculates the difference between the new calibration parameters and the current calibration parameters, and updates the current calibration parameters based on the new calibration parameters when the absolute value of the difference is greater than a preset difference value.

[0053] In practice, this application is not limited by the execution order of the described steps. Without causing conflicts, some steps may be performed in other orders or simultaneously.

[0054] As can be seen from the above, the parameter calibration method for a camera module provided in this application acquires a first image output by the camera module of an electronic device, obtains a depth map using the first image and the current calibration parameters of the camera module, detects the accuracy of the depth map, and confirms whether it is below a preset threshold. When the accuracy of the depth map is below the preset threshold, it can be determined that the camera module may have aging or other abnormal phenomena. Therefore, the camera module is calibrated to obtain new calibration parameters, and the current calibration parameters are updated based on the new calibration parameters. The solution of this application embodiment can detect abnormalities in the camera module in a timely manner, and this detection operation can be performed in the background without manual triggering by the user, thus offering a certain degree of convenience. Furthermore, when an abnormality is detected, the calibration parameters of the camera module can be calibrated, thereby improving the accuracy of image processing in the electronic device.

[0055] Based on the methods described in the preceding embodiments, the following examples will provide further detailed explanations.

[0056] Please see Figure 2 , Figure 2 This is a second flowchart illustrating the parameter calibration method for a camera module provided in an embodiment of the present invention. The method is applied to an electronic device. The electronic device includes an application processing chip, a coprocessor chip connected to the application processing chip, and a camera module connected to the coprocessor chip. The camera module includes a main camera and auxiliary cameras, wherein there can be one or more auxiliary cameras. The method includes: 201. Obtain the first images output by the main camera and the auxiliary camera to obtain multiple first images.

[0057] There are several ways to acquire the first image in this application embodiment. For example, a calibration function can be set for the camera module. After the user enables the calibration function, the electronic device opens the camera application and enters shooting mode. After the user triggers the shooting command, the first image output by the camera module is acquired. Alternatively, in another embodiment, after entering shooting mode, the electronic device instructs the user to shoot a scene with a clear outline in a certain brightness environment to improve the accuracy of depth map detection. Alternatively, in another embodiment, at preset time intervals, when the user is detected taking a picture with the camera module, the original images output by multiple cameras of the camera module are stored in the background and recorded as the first image.

[0058] 202. Obtain the current calibration parameters of the camera module, and calculate the depth map based on multiple first images and the current calibration parameters.

[0059] A depth map is calculated based on the first image. The depth map contains depth information, which refers to the distance between the object being photographed and the camera in the image. Feature point pairs are calculated between the first images from the main camera and the first images from the auxiliary camera using a feature point matching algorithm. The depth map is then calculated based on the matched feature point pairs and the current calibration parameters.

[0060] 203. Combine multiple first images to obtain a composite image, and perform format conversion on the composite image to obtain the corresponding grayscale image.

[0061] Multiple frames of the first image are synthesized to obtain a composite image. Then, the composite image is converted to obtain a YUV image, and a grayscale image is obtained from the YUV image.

[0062] 204. Perform edge detection processing on the grayscale image to obtain the first edge pixel, and perform edge detection processing on the depth image to obtain the second edge pixel.

[0063] After obtaining the grayscale image, edge detection processing is performed on it to obtain the first edge pixels. Edge detection processing is then performed on the depth image to obtain the second edge pixels. The purpose of using edge detection algorithms on the image is to identify points in the image with significant brightness changes, i.e., edge pixels.

[0064] In some embodiments, the steps of performing edge detection processing on the grayscale image to obtain first edge pixels and performing edge detection processing on the depth image to obtain second edge pixels may include: dividing the grayscale image into M×N first regions and the depth image into M×N second regions according to a preset division method; determining a target second region with a grayscale value greater than a preset threshold from the M×N second regions, and determining a target first region corresponding to the target second region from the M×N first regions; performing edge detection processing on the target first region to obtain first edge pixels, and performing edge detection processing on the target second region to obtain second edge pixels. Please refer to [link to relevant documentation]. Figure 3 , Figure 3 A schematic diagram illustrating the process of extracting edge pixels in the parameter calibration method for the camera module provided in this application embodiment.

[0065] In this embodiment, to improve the speed of edge detection, the acquired depth map and grayscale image are first processed by region segmentation, such as... Figure 3 As shown, the two images are divided into M×N regions in the same way. Figure 3In this example, M=N=4 is merely illustrative; in other embodiments, the values ​​of M and N can be set as needed. After dividing the region, the grayscale value of each region is detected, where the grayscale value of a region can be the average of the grayscale values ​​of all pixels within that region. From the 16 second regions, a target second region with a grayscale value greater than a preset threshold is determined. Then, a target first region corresponding to the target second region is selected, and edge detection processing is performed on the target first region to obtain first edge pixels. Similarly, edge detection processing is performed on the target second region to obtain second edge pixels.

[0066] 205. Calculate the gray value difference between the second edge pixel and the first edge pixel, and determine the accuracy of the depth map based on the gray value difference. The accuracy value is inversely proportional to the absolute value of the gray value difference.

[0067] Next, the grayscale difference between the second edge pixel and the first edge pixel is calculated, and the accuracy of the depth map is determined based on this grayscale value. Specifically, the grayscale difference between the second edge pixel and the first edge pixel is calculated, and the accuracy of the depth map is determined based on this grayscale value. For example, in one embodiment, this grayscale difference is used as the accuracy of the depth map. In another embodiment, a preset accuracy corresponding to the calculated grayscale difference is determined based on a preset relationship between a preset grayscale difference and a preset accuracy, and this preset accuracy is used as the accuracy of the depth map. The accuracy is inversely proportional to the absolute value of the grayscale difference; that is, the larger the absolute value of the grayscale difference, the greater the difference between the edge information in the depth map and the edge information in the grayscale map, and thus the lower the accuracy of the depth map.

[0068] 206. When the accuracy of the depth map is lower than the preset threshold, acquire multiple calibration images captured by multiple cameras.

[0069] After calculating the accuracy of the depth map, it is determined whether the accuracy is lower than a preset threshold. If so, it means that the calibration parameters of the camera module need to be calibrated.

[0070] 207. Based on multiple calibration images, the camera module is calibrated to obtain new calibration parameters.

[0071] Next, the camera module is calibrated. When the accuracy of the depth map detection falls below a preset threshold, a prompt message is displayed on the graphical user interface, prompting the user to take a calibration image. Before the phone leaves the factory, the camera module is typically calibrated using a specific calibration board, such as a checkerboard calibration board. However, users may find it difficult to provide images taken with a specific calibration board. In this case, the user can be reminded to take images of scenes with clearly defined object outlines, sufficient lighting, or significant color differences. Alternatively, in another embodiment, an image of a checkerboard calibration board is provided, prompting the user to print the image of the calibration board according to a specific size to obtain the calibration image.

[0072] After the user performs a shooting operation, multiple frames of images captured by multiple cameras are acquired as calibration images. Next, corner detection and corner matching are performed on these calibration images to obtain multiple corner pairs. It is then determined whether the number of corner pairs exceeds a preset number. If so, the camera module can be calibrated according to preset calibration parameters to obtain new calibration parameters. For example, the new calibration parameters can be calculated using Zhang's calibration method.

[0073] 208. When the difference between the new calibration parameter and the current calibration parameter is greater than the preset difference, update the current calibration parameter based on the new calibration parameter.

[0074] The execution entity for steps 201 to 207 above can be a coprocessor chip. The first image output by the camera module is directly sent to the coprocessor chip, which calculates the new calibration parameters.

[0075] After the coprocessor chip calculates the new calibration parameters, it sends them to the application processing chip. The application processing chip executes step 208 to calculate the difference between the new calibration parameters and the current calibration parameters. When the absolute value of the difference is greater than a preset difference, the current calibration parameters are updated based on the new calibration parameters.

[0076] As can be seen from the above, the parameter calibration method for the camera module proposed in this embodiment of the invention can detect abnormalities in the camera module in a timely manner, and the detection operation can be performed in the background without the need for manual triggering by the user, which is convenient. In addition, after detecting camera aging, the calibration parameters of the camera module can be calibrated, thereby improving the accuracy of image processing of electronic devices.

[0077] In one embodiment, a parameter calibration device for a camera module is also provided. This parameter calibration device for the camera module is applied to an electronic device and includes: The acquisition module is used to acquire the first image output by the camera module and to acquire the current calibration parameters of the camera module. The processing module is used to obtain a depth map based on the first image and the current calibration parameters; The detection module is used to detect whether the accuracy of the depth map is lower than a preset threshold. The calibration module is used to calibrate the camera module to obtain new calibration parameters when the accuracy of the depth map is lower than a preset threshold, and update the current calibration parameters based on the new calibration parameters.

[0078] It should be noted that the parameter calibration device for the camera module provided in this application embodiment and the parameter calibration method for the camera module in the above embodiment belong to the same concept. The parameter calibration device for the camera module can implement any of the methods provided in the parameter calibration method embodiment for the camera module. For details of its implementation process, please refer to the parameter calibration method embodiment for the camera module, which will not be repeated here.

[0079] As can be seen from the above, the parameter calibration device for the camera module proposed in this application acquires the first image output by the camera module of the electronic device, obtains a depth map through the first image and the current calibration parameters of the camera module, detects the accuracy of the depth map, and confirms whether it is lower than a preset threshold. When the accuracy of the depth map is lower than the preset threshold, it can be determined that the camera module may have abnormal phenomena such as aging. Then, the camera module is calibrated to obtain new calibration parameters, and the current calibration parameters are updated based on the new calibration parameters. The solution of this application embodiment can detect abnormalities of the camera module in a timely manner, and the detection operation can be performed in the background without manual triggering by the user, which has a certain degree of convenience. In addition, when an abnormality is detected, the calibration parameters of the camera module can be calibrated, thereby improving the accuracy of image processing of the electronic device.

[0080] This application also provides a coprocessor chip. Please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic diagram of the structure of a coprocessor chip provided in an embodiment of this application. The coprocessor chip 301 includes a central processing unit 3011, which is used for: Acquire the first image output by the camera module, and acquire the current calibration parameters of the camera module; A depth map is obtained based on the first image and the current calibration parameters; Detect whether the accuracy of the depth map is lower than a preset threshold; When the accuracy of the depth map is lower than a preset threshold, the camera module is calibrated to obtain new calibration parameters, and the current calibration parameters are updated based on the new calibration parameters.

[0081] In some embodiments, the central processing unit 3011 is configured to: calculate the difference between the new calibration parameter and the current calibration parameter; When the absolute value of the difference is greater than the preset difference, the current calibration parameter is updated based on the new calibration parameter.

[0082] In some embodiments, the central processing unit 3011 is configured to: send the new calibration parameters to the application processing chip of the electronic device, such that the application processing chip calculates the difference between the new calibration parameters and the current calibration parameters, and updates the current calibration parameters based on the new calibration parameters when the absolute value of the difference is greater than a preset difference.

[0083] In some embodiments, the camera module includes multiple cameras; the central processing unit 3011 is used to: acquire a first image output by each camera in the camera module to obtain multiple first images.

[0084] In some embodiments, the central processing unit 3011 is configured to: perform a composite processing on the plurality of first images to obtain a composite image; The synthesized image is processed to obtain a grayscale image of the synthesized image; Edge detection processing is performed on the grayscale image to obtain the first edge pixel, and edge detection processing is performed on the depth image to obtain the second edge pixel; Calculate the grayscale difference between the second edge pixel and the first edge pixel, and determine the accuracy of the depth map based on the grayscale difference, wherein the accuracy is inversely proportional to the absolute value of the grayscale difference; The accuracy is checked to see if it is lower than a preset threshold.

[0085] In some embodiments, the central processing unit 3011 is configured to: divide the grayscale image into M×N first regions and the depth image into M×N second regions according to a preset division method; From the M×N second regions, a target second region with a gray value greater than a preset threshold is determined, and a target first region corresponding to the target second region is determined from the M×N first regions; Edge detection processing is performed on the first region of the target to obtain a first edge pixel, and edge detection processing is performed on the second region of the target to obtain a second edge pixel.

[0086] In some embodiments, the central processing unit 3011 is used for: When the accuracy of the depth map is lower than a preset threshold, multiple calibration images captured by the multiple cameras are obtained. Corner detection and corner matching are performed on the multi-frame calibration images to obtain multiple corner pairs; When the number of corner point pairs is greater than the preset number, the camera module is calibrated based on the multi-frame calibration images to obtain new calibration parameters; The current calibration parameters are updated based on the new calibration parameters.

[0087] This application also provides an electronic device. The electronic device may be a smartphone, tablet computer, or similar device. Please refer to... Figure 5 , Figure 5 This is a schematic diagram of a first structure of an electronic device provided in an embodiment of this application. The electronic device 300 includes a coprocessor chip 301, an application processing chip 302 connected to the coprocessor chip, and a camera module 303. The coprocessor chip 301 is used to: acquire a first image output by the camera module of the electronic device; obtain a depth map based on the first image and the current calibration parameters; detect whether the accuracy of the depth map is lower than a preset threshold; and, when the accuracy of the depth map is lower than the preset threshold, calibrate the camera module to obtain new calibration parameters, and update the current calibration parameters based on the new calibration parameters.

[0088] This application also provides an electronic device. The electronic device may be a smartphone, tablet computer, or similar device. Please refer to... Figure 6 , Figure 6 This is a second structural schematic diagram of an electronic device provided in an embodiment of this application. The electronic device 400 includes a processor 401 and a memory 402. The processor 401 and the memory 402 are electrically connected.

[0089] The processor 401 is the control center of the electronic device 400. It connects various parts of the electronic device through various interfaces and lines. By running or calling computer programs stored in the memory 402 and calling data stored in the memory 402, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole.

[0090] Memory 402 can be used to store computer programs and data. The computer programs stored in memory 402 contain instructions that can be executed in the processor. The computer programs can be composed of various functional modules. The processor 401 executes various functional applications and data processing by calling the computer programs stored in memory 402.

[0091] In this embodiment, the processor 401 in the electronic device 400 loads the instructions corresponding to the processes of one or more computer programs into the memory 402 according to the following steps, and the processor 401 runs the computer programs stored in the memory 402 to realize various functions: Acquire the first image output by the camera module, and acquire the current calibration parameters of the camera module; A depth map is obtained based on the first image and the current calibration parameters; Detect whether the accuracy of the depth map is lower than a preset threshold; When the accuracy of the depth map is lower than a preset threshold, the camera module is calibrated to obtain new calibration parameters, and the current calibration parameters are updated based on the new calibration parameters.

[0092] In some embodiments, please refer to Figure 7 , Figure 7 This is a third structural diagram of the electronic device provided in the embodiments of this application. The electronic device 400 further includes: a radio frequency circuit 403, a display screen 404, a control circuit 405, an input unit 406, an audio circuit 407, a sensor 408, and a power supply 409. The processor 401 is electrically connected to the radio frequency circuit 403, the display screen 404, the control circuit 405, the input unit 406, the audio circuit 407, the sensor 408, and the power supply 409.

[0093] The radio frequency circuit 403 is used to transmit and receive radio frequency signals to communicate with network devices or other electronic devices via wireless communication.

[0094] The display screen 404 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of electronic devices, which can be composed of images, text, icons, videos, and any combination thereof.

[0095] The control circuit 405 is electrically connected to the display screen 404 and is used to control the display screen 404 to display information.

[0096] The input unit 406 can be used to receive input numeric or character information or user characteristic information (such as fingerprints), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. The input unit 406 may include a fingerprint recognition module.

[0097] The audio circuit 407 provides an audio interface between the user and the electronic device via a speaker and a microphone. The audio circuit 407 includes a microphone, which is electrically connected to the processor 401. The microphone is used to receive voice information input by the user.

[0098] Sensor 408 is used to collect information about the external environment. Sensor 408 may include one or more sensors such as an ambient light sensor, an accelerometer, and a gyroscope.

[0099] The power supply 409 is used to supply power to the various components of the electronic device 400. In some embodiments, the power supply 409 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system.

[0100] Although not shown in the figure, electronic device 400 may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0101] In this embodiment, the processor 401 in the electronic device 400 loads the instructions corresponding to the processes of one or more computer programs into the memory 402 according to the following steps, and the processor 401 runs the computer programs stored in the memory 402 to realize various functions: Acquire the first image output by the camera module, and acquire the current calibration parameters of the camera module; A depth map is obtained based on the first image and the current calibration parameters; Detect whether the accuracy of the depth map is lower than a preset threshold; When the accuracy of the depth map is lower than a preset threshold, the camera module is calibrated to obtain new calibration parameters, and the current calibration parameters are updated based on the new calibration parameters.

[0102] As described above, this application provides an electronic device that acquires a first image output by a camera module and obtains the current calibration parameters of the camera module. A depth map is obtained using the first image and the current calibration parameters. The accuracy of the depth map is detected and confirmed to be below a preset threshold. When the accuracy of the depth map is below the preset threshold, it can be determined that the camera module may be experiencing abnormal phenomena such as aging. The camera module is then calibrated to obtain new calibration parameters, and the current calibration parameters are updated based on these new parameters. The solution of this application can detect abnormalities in the camera module in a timely manner, and this detection operation can be performed in the background without manual triggering by the user, offering a certain degree of convenience. Furthermore, when an abnormality is detected, the calibration parameters of the camera module can be calibrated, thereby improving the image processing accuracy of the electronic device.

[0103] This application also provides a computer-readable storage medium storing a computer program. When the computer program is run on a computer, the computer executes the parameter calibration method for the camera module described in any of the above embodiments.

[0104] It should be noted that those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, which may include, but is not limited to, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0105] Furthermore, the terms "first," "second," and "third," etc., used in this application are used to distinguish different objects, not to describe a specific order. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, but some embodiments may also include steps or modules not listed, or some embodiments may include other steps or modules inherent to these processes, methods, products, or devices.

[0106] The parameter calibration method, storage medium, coprocessor chip, and electronic device of the camera module provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this application; at the same time, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A parameter calibration method for a camera module, characterized in that, include: Acquire the first image output by the camera module, wherein the camera module includes multiple cameras, acquire the first image output by each camera in the camera module, obtain multiple first images, and acquire the current calibration parameters of the camera module; A depth map is obtained based on the first image and the current calibration parameters; Detecting whether the accuracy of the depth map is lower than a preset threshold includes: performing a composite processing on the plurality of first images to obtain a composite image; processing the composite image to obtain a grayscale image of the composite image; performing edge detection processing on the grayscale image to obtain first edge pixels, and performing edge detection processing on the depth map to obtain second edge pixels; calculating the grayscale value difference between the second edge pixels and the first edge pixels, determining the accuracy of the depth map based on the grayscale value difference, wherein the accuracy value is inversely proportional to the absolute value of the grayscale value difference; and detecting whether the accuracy is lower than a preset threshold. When the accuracy of the depth map is lower than a preset threshold, the camera module is calibrated to obtain new calibration parameters, and the current calibration parameters are updated based on the new calibration parameters.

2. The method as described in claim 1, characterized in that, When the accuracy of the depth map is lower than a preset threshold, the camera module is calibrated to obtain new calibration parameters, and the current calibration parameters are updated based on the new calibration parameters, including: Calculate the difference between the new calibration parameters and the current calibration parameters; When the absolute value of the difference is greater than the preset difference, the current calibration parameter is updated based on the new calibration parameter.

3. The method as described in claim 1, characterized in that, When the accuracy of the depth map is lower than a preset threshold, the camera module is calibrated to obtain new calibration parameters, and the current calibration parameters are updated based on the new calibration parameters, including: The new calibration parameters are sent to the application processing chip of the electronic device, so that the application processing chip calculates the difference between the new calibration parameters and the current calibration parameters, and updates the current calibration parameters based on the new calibration parameters when the absolute value of the difference is greater than a preset difference.

4. The method as described in claim 1, characterized in that, The step of performing edge detection processing on the grayscale image to obtain first edge pixels and performing edge detection processing on the depth image to obtain second edge pixels includes: According to the preset division method, the grayscale image is divided into M×N first regions, and the depth image is divided into M×N second regions; From the M×N second regions, a target second region with a gray value greater than a preset threshold is determined, and a target first region corresponding to the target second region is determined from the M×N first regions; Edge detection processing is performed on the first region of the target to obtain a first edge pixel, and edge detection processing is performed on the second region of the target to obtain a second edge pixel.

5. The method according to any one of claims 2 to 4, characterized in that, When the accuracy of the depth map is lower than a preset threshold, the camera module is calibrated to obtain new calibration parameters, and the current calibration parameters are updated based on the new calibration parameters, including: When the accuracy of the depth map is lower than a preset threshold, multiple calibration images captured by the multiple cameras are obtained. Corner detection and corner matching are performed on the multi-frame calibration images to obtain multiple corner pairs; When the number of corner point pairs is greater than the preset number, the camera module is calibrated based on the multi-frame calibration images to obtain new calibration parameters; The current calibration parameters are updated based on the new calibration parameters.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run on a computer, it causes the computer to perform the parameter calibration method for the camera module as described in any one of claims 1 to 5.

7. A coprocessor chip, comprising a central processing unit, characterized in that, The central processing unit is used to execute the parameter calibration method for the camera module as described in any one of claims 1 to 5.

8. An electronic device comprising a processor and a memory, the memory storing a computer program, characterized in that, The processor invokes the computer program to execute the parameter calibration method for the camera module as described in any one of claims 1 to 5.

Images