Methods, devices, electronic equipment, and storage media for optimizing camera module parameters

Abstract

This disclosure provides a method, apparatus, electronic device, and storage medium for optimizing the parameters of a camera module. The sensor is disposed in the electronic device. The method includes: determining position parameters based on a color image captured by a first camera and a depth image, grayscale image, and confidence image captured by a second camera. The position parameters characterize the relative positional relationship between the first camera and the second camera. This disclosure obtains feature point sets from the depth image, confidence image, grayscale image, and color image captured by the camera module, respectively, and performs confidence filtering on the feature point sets based on a first preset condition. The feature point sets that meet the first preset condition are used to determine the position parameters of the camera module to obtain more accurate position parameters, thereby improving the autofocus and optical image stabilization functions of the camera module.

Description

Technical Field

[0001] This disclosure relates to the field of camera technology, and in particular to a method, apparatus, electronic device and storage medium for optimizing parameters of a camera module. Background Technology

[0002] Currently, 3D imaging cameras, primarily RGB-D cameras, are widely used in augmented reality, virtual reality, simultaneous localization and mapping (SLAM), and other fields. During image acquisition, cameras typically employ autofocus and optical image stabilization to reduce visual positioning errors caused by differences in focus positions between lenses, changes in camera orientation, and camera shake. The effectiveness of autofocus and optical image stabilization depends on the relative positions of the lenses within the camera, i.e., the camera's parameters.

[0003] Currently, camera parameters are mainly obtained by using feature detection and matching to obtain a homography matrix from the acquired grayscale and color images. However, due to the influence of the shooting scene, the obtained homography matrix often contains errors, which can easily lead to inaccurate relative positional relationships between lenses in the camera, affecting the effectiveness of autofocus and optical image stabilization. Summary of the Invention

[0004] In view of this, the present disclosure provides a method, apparatus, electronic device and readable storage medium for optimizing the parameters of a camera module, so as to at least solve the problem of camera parameter calculation error in the related art.

[0005] According to a first aspect of the present disclosure, a method for optimizing the parameters of a camera module is provided, the camera module including a first camera and a second camera, the method comprising:

[0006] The color image captured by the first camera, and the depth image, grayscale image and confidence image captured by the second camera are acquired, wherein there is a first correspondence between the pixels of the depth image, the pixels of the grayscale image and the pixels of the confidence image;

[0007] Obtain the first feature point set of the color image and the second feature point set of the grayscale image that have a second correspondence relationship;

[0008] Based on the first correspondence, the second correspondence, and the confidence of the pixels in the confidence image, a third feature point set of the color image with a third correspondence, a fourth feature point set of the grayscale image, and a fifth feature point set of the depth image are determined, wherein the confidence of the pixels in the confidence image corresponding to each feature point in the fourth feature point set satisfies a first preset condition.

[0009] Based on the third feature point set, the fourth feature point set, and the fifth feature point set, position parameters are determined, which are used to characterize the relative positional relationship between the first camera and the second camera.

[0010] In any embodiment of this disclosure, obtaining the first feature point set of the color image and the second feature point set of the grayscale image having a second correspondence includes:

[0011] Based on the current position parameters of the camera module, feature matching is performed on the color image and the grayscale image to obtain a first feature point set of the color image and a second feature point set of the grayscale image with a second correspondence.

[0012] After determining the position parameters, the method further includes:

[0013] Update the current position parameters using the defined position parameters.

[0014] In any embodiment of this disclosure, the feature matching of the color image and the grayscale image includes:

[0015] Adjust the color image and the grayscale image to be aligned in coplanar rows;

[0016] Feature matching is performed on the color image and the grayscale image that have been adjusted to coplanar row alignment.

[0017] In any embodiment of this disclosure, determining the third feature point set of the color image, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth image, based on the first correspondence, the second correspondence, and the confidence level of the pixels in the confidence image, includes:

[0018] Based on the first correspondence and the second feature point set, determine the sixth feature point set of the depth image that has a fourth correspondence with the second feature point set;

[0019] Based on the confidence scores of the pixels in the confidence image and the first correspondence, determine the confidence scores of each feature point in the sixth feature point set and each feature point in the second feature point set;

[0020] Based on the confidence level of each feature point in the second feature point set and the second correspondence, determine the confidence level of each feature point in the first feature point set;

[0021] Based on the confidence levels of the first feature point set, the second feature point set, the sixth feature point set, and the first preset condition, the third feature point set of the color image, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth image with a third correspondence are determined.

[0022] In conjunction with any embodiment of this disclosure, determining the third feature point set of the color image, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth image having a third correspondence includes:

[0023] Remove feature points in the first feature point set whose confidence level is lower than the confidence threshold to obtain the third feature point set;

[0024] Remove feature points in the second feature point set whose confidence level is lower than the confidence threshold to obtain the fourth feature point set;

[0025] Remove feature points in the sixth feature point set whose confidence level is lower than the confidence threshold to obtain the fifth feature point set.

[0026] In any embodiment of this disclosure, the position parameters include a baseline;

[0027] The step of determining the position parameters based on the third feature point set, the fourth feature point set, and the fifth feature point set includes:

[0028] Obtain the baseline of each feature point pair between the third feature point set and the fourth feature point set;

[0029] Cluster the baselines of each feature point pair, and determine the clustering results as the baseline of the camera module.

[0030] In conjunction with any embodiment of this disclosure, obtaining the baseline of each feature point pair between the third feature point set and the fourth feature point set includes:

[0031] Obtain the coordinate distance between each pair of feature points between the third feature point set and the fourth feature point set;

[0032] Based on the third correspondence, the depth value of each feature point in the fifth feature point set is determined as the depth value of the corresponding feature point pair between the third feature point set and the fourth feature point set;

[0033] Based on the coordinate distance and depth values ​​of each feature point pair and the focal length of the camera module, the baseline of each feature point pair between the third feature point set and the fourth feature point set is determined.

[0034] In any embodiment of this disclosure, obtaining the coordinate distance between each pair of feature points between the third feature point set and the fourth feature point set includes:

[0035] According to the second correspondence, the disparity set between the first feature point set and the second feature point set is obtained, wherein the disparity set includes the coordinate distance of each feature point pair between the first feature point set and the second feature point set;

[0036] The coordinate distance between each pair of feature points between the third feature point set and the fourth feature point set is obtained from the disparity set.

[0037] In any embodiment of this disclosure, the position parameter includes a rotation angle;

[0038] The step of determining the position parameters based on the third feature point set, the fourth feature point set, and the fifth feature point set includes:

[0039] Based on the third correspondence and the depth value of each feature point in the fifth feature point set, the seventh feature point set of the color image and the eighth feature point set of the grayscale image with the fifth correspondence are determined, wherein the depth value of the feature point in the fifth feature point set corresponding to each feature point in the eighth feature point set satisfies the second preset condition.

[0040] Obtain the rotation angle of each feature point pair between the seventh feature point set and the eighth feature point set;

[0041] Cluster the rotation angles of each feature point pair, and determine the clustering results as the rotation angles of the camera module.

[0042] In conjunction with any embodiment of this disclosure, determining the seventh feature point set of the color image and the eighth feature point set of the grayscale image having a fifth correspondence includes:

[0043] Remove feature points with depth values ​​greater than a first depth threshold from the third and fourth feature point sets respectively to obtain the seventh and eighth feature point sets; or,

[0044] Remove feature points with depth values ​​greater than the second depth threshold and feature points with depth values ​​less than the third depth threshold from the third feature point set and the fourth feature point set respectively to obtain the seventh feature point set and the eighth feature point set.

[0045] According to a second aspect of the present disclosure, a parameter optimization apparatus for a camera module is provided, the camera module including a first camera and a second camera, the apparatus comprising:

[0046] Image acquisition module: used to acquire a color image captured by the first camera, and a depth image, a grayscale image and a confidence image captured by the second camera, wherein there is a first correspondence between the pixels of the depth image, the pixels of the grayscale image and the pixels of the confidence image;

[0047] Online calibration module: used to obtain the first feature point set of the color image and the second feature point set of the grayscale image having a second correspondence;

[0048] Confidence filtering module: used to determine the third feature point set of the color image with the third correspondence, the fourth feature point set of the grayscale portrait and the fifth feature point set of the depth image based on the first correspondence, the second correspondence and the confidence of the pixel points of the confidence image, wherein the confidence of the pixel points of the confidence image corresponding to each feature point in the fourth feature point set satisfies the first preset condition.

[0049] Parameter determination module: used to determine position parameters based on the third feature point set, the fourth feature point set and the fifth feature point set, wherein the position parameters are used to characterize the relative positional relationship between the first camera and the second camera.

[0050] In any embodiment of this disclosure, the online calibration module acquires a first feature point set of the color image and a second feature point set of the grayscale image having a second correspondence, for the purpose of:

[0051] Based on the current position parameters of the camera module, feature matching is performed on the color image and the grayscale image to obtain a first feature point set of the color image and a second feature point set of the grayscale image with a second correspondence.

[0052] After the parameter determination module determines the position parameters, it also includes a parameter update module, used for:

[0053] Update the current position parameters using the determined position parameters.

[0054] In any embodiment of this disclosure, the feature matching of the color image and the grayscale image is used for:

[0055] Adjust the color image and the grayscale image to be aligned in coplanar rows;

[0056] Feature matching is performed on the color image and the grayscale image that have been adjusted to coplanar row alignment.

[0057] In conjunction with any embodiment of this disclosure, the confidence filtering module determines, based on the first correspondence, the second correspondence, and the confidence of the pixels in the confidence image, a third feature point set of the color image with a third correspondence, a fourth feature point set of the grayscale portrait, and a fifth feature point set of the depth map, for the following purposes:

[0058] Based on the first correspondence and the second feature point set, determine the sixth feature point set of the depth image that has a fourth correspondence with the second feature point set;

[0059] Based on the confidence scores of the pixels in the confidence image and the first correspondence, determine the confidence scores of each feature point in the sixth feature point set and each feature point in the second feature point set;

[0060] Based on the confidence level of each feature point in the second feature point set and the second correspondence, determine the confidence level of each feature point in the first feature point set;

[0061] Based on the confidence levels of the first feature point set, the second feature point set, the sixth feature point set, and the first preset condition, the third feature point set of the color image, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth image with a third correspondence are determined.

[0062] In conjunction with any embodiment of this disclosure, the confidence filtering module determines a third feature point set of the color image, a fourth feature point set of the grayscale portrait, and a fifth feature point set of the depth image that have a third correspondence, for the following purposes:

[0063] Remove feature points in the first feature point set whose confidence level is lower than the confidence threshold to obtain the third feature point set;

[0064] Remove feature points in the second feature point set whose confidence level is lower than the confidence threshold to obtain the fourth feature point set;

[0065] Remove feature points in the sixth feature point set whose confidence level is lower than the confidence threshold to obtain the fifth feature point set.

[0066] In any embodiment of this disclosure, the position parameters include a baseline; the parameter determination module determines the position parameters based on the third feature point set, the fourth feature point set, and the fifth feature point set, for:

[0067] Obtain the baseline of each feature point pair between the third feature point set and the fourth feature point set;

[0068] Cluster the baselines of each feature point pair, and determine the clustering results as the baseline of the camera module.

[0069] In any embodiment of this disclosure, obtaining the baseline of each feature point pair between the third feature point set and the fourth feature point set is used for:

[0070] Obtain the coordinate distance of each feature point pair between the third feature point set and the fourth feature point set, wherein the feature point pair includes two feature points that belong to the third feature point set and the fourth feature point set respectively;

[0071] Based on the third correspondence and the fifth feature point set, obtain the depth value of each feature point pair between the third feature point set and the fourth feature point set;

[0072] Based on the coordinate distance and depth values ​​of each feature point pair and the focal length of the camera module, the baseline of each feature point pair between the third feature point set and the fourth feature point set is obtained.

[0073] In any embodiment of this disclosure, obtaining the coordinate distance of each feature point pair between the third feature point set and the fourth feature point set is used for:

[0074] According to the second correspondence, the disparity set between the first feature point set and the second feature point set is obtained, wherein the disparity set includes the coordinate distance of each feature point pair between the first feature point set and the second feature point set;

[0075] The coordinate distance between each pair of feature points between the third feature point set and the fourth feature point set is obtained from the disparity set.

[0076] In any embodiment of this disclosure, the position parameter includes a rotation angle;

[0077] The parameter determination module determines position parameters based on the third feature point set, the fourth feature point set, and the fifth feature point set, for the purpose of:

[0078] Based on the third correspondence and the depth value of each feature point in the fifth feature point set, the seventh feature point set of the color image and the eighth feature point set of the grayscale image with the fifth correspondence are determined, wherein the depth value of the feature point in the fifth feature point set corresponding to each feature point in the eighth feature point set satisfies the second preset condition.

[0079] Obtain the rotation angle of each feature point pair between the seventh feature point set and the eighth feature point set;

[0080] Cluster the rotation angles of each feature point pair, and determine the clustering results as the rotation angles of the camera module.

[0081] In any embodiment of this disclosure, determining the seventh feature point set of the color image and the eighth feature point set of the grayscale image having a fifth correspondence is used for:

[0082] Remove feature points with depth values ​​greater than a first depth threshold from the third and fourth feature point sets respectively to obtain the seventh and eighth feature point sets; or,

[0083] Remove feature points with depth values ​​greater than the second depth threshold and feature points with depth values ​​less than the third depth threshold from the third feature point set and the fourth feature point set respectively to obtain the seventh feature point set and the eighth feature point set.

[0084] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0085] Memory for storing processor-executable instructions;

[0086] The processor is configured to execute executable instructions in the memory to implement the steps of the method described in any of the first aspects above.

[0087] According to a fourth aspect of the present disclosure, a computer storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the embodiments of the first aspect.

[0088] According to a fifth aspect of the present disclosure, a camera module is provided, including the above-described electronic device.

[0089] The technical solutions provided in this disclosure may have the following beneficial effects:

[0090] By acquiring feature point sets from depth images, confidence images, grayscale images, and color images captured by the camera module, and filtering the feature point sets based on a first preset condition, the feature point sets that meet the first preset condition are used to determine the position parameters of the camera module, thereby obtaining more accurate position parameters and improving the effect of the camera module's autofocus and optical image stabilization functions.

[0091] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0092] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0093] Figure 1AThis disclosure is a flowchart illustrating a parameter optimization method for a camera module according to an exemplary embodiment;

[0094] Figure 1B This disclosure is a flowchart illustrating another method for optimizing the parameters of a camera module according to an exemplary embodiment;

[0095] Figure 2A This disclosure is a flowchart illustrating a baseline determination method according to an exemplary embodiment;

[0096] Figure 2B This is a flowchart illustrating another baseline determination method according to an exemplary embodiment of the present disclosure;

[0097] Figure 2C This is a flowchart illustrating another baseline determination method according to an exemplary embodiment of the present disclosure;

[0098] Figure 3 This disclosure is a flowchart illustrating a method for determining a rotation angle line according to an exemplary embodiment;

[0099] Figure 4 This disclosure is a schematic diagram of a parameter optimization device for a camera module according to an exemplary embodiment;

[0100] Figure 5 This disclosure is a block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0101] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0102] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0103] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0104] Figure 1A This disclosure illustrates a flowchart of a parameter optimization method for a camera module according to an exemplary embodiment.

[0105] In step S101, a color image captured by the first camera and a depth image, a grayscale image, and a confidence image captured by the second camera are acquired, wherein the pixels of the depth image, the pixels of the grayscale image, and the pixels of the confidence image have a first correspondence relationship.

[0106] In this disclosure, the camera module includes at least two cameras. When the camera module is an RGB-D camera, the first camera can be a color camera, and the second camera can be a depth camera. The color image can be acquired by the color camera, and the depth image, grayscale image, and confidence image can be acquired by the depth camera. The color image displays the color (RGB) information of each pixel in the shooting scene; the depth image displays the distance information from the second camera to the shooting scene; the grayscale image displays the grayscale value of each pixel in the shooting scene; and the confidence image displays the depth confidence level of each pixel in the shooting scene. Since the depth image, grayscale image, and confidence image are all acquired by the second camera, the images have a first correspondence: they have the same resolution, the same number of pixels, and the same depth and confidence value for each pixel.

[0107] In step S102, the first feature point set of the color image and the second feature point set of the grayscale image with a second correspondence are obtained.

[0108] Since the color image and the grayscale image are captured by the first camera and the second camera respectively, the resolutions of the two images may differ, and the pixel correspondence is uncertain. A second correspondence between the color image and the grayscale image can be established through feature detection and matching, that is, the correspondence between feature points of the two images. This allows for the acquisition of a first feature point set for the color image and a second feature point set for the grayscale image that have the second correspondence, meaning the number of feature points is the same, and the depth value and confidence value of each feature point are identical. The feature detection can be implemented using algorithms such as SIFT and SURF, which will not be elaborated upon here.

[0109] In step S103, based on the first correspondence, the second correspondence, and the confidence of the pixels in the confidence image, a third feature point set of the color image with a third correspondence, a fourth feature point set of the grayscale image, and a fifth feature point set of the depth image are determined, wherein the confidence of the pixels in the confidence image corresponding to each feature point in the fourth feature point set satisfies a first preset condition.

[0110] The second feature point set of the grayscale image has a first correspondence with the depth image and the confidence image, and a second correspondence with the second feature point set of the color image. The confidence value of each feature point in the second feature point set can be determined by the confidence value of each pixel in the confidence image, and then the confidence value of each feature point in the first feature point set can be determined by the confidence value of each feature point in the second feature point set. In the shooting scenario, the confidence level of the depth values ​​of the edges of the subject acquired by the camera module is poor, which easily leads to errors in the parameter calculation of the camera module. After acquiring the confidence value of each pixel in the confidence image, confidence filtering can be performed on the feature point set of the depth image, the first feature point set, and the second feature point set based on the confidence value of the depth value. That is, the third feature point set of the color image, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth image with a third correspondence that satisfy the first preset condition can be obtained.

[0111] In step S104, position parameters are determined based on the third feature point set, the fourth feature point set, and the fifth feature point set. These position parameters are used to characterize the relative positional relationship between the first camera and the second camera.

[0112] The third, fourth, and fifth feature point sets are feature point sets composed of feature points whose confidence levels satisfy the first preset condition. Based on these feature point sets, the position parameters of the camera module can be redefined to improve the accuracy of the position parameters and reduce calculation errors. The position parameters characterize the relative positional relationship between the first and second cameras and may include the baseline and rotation angle of the camera module. When the camera module is an RGB-D camera, the position parameters may include any external parameter of the RGB-D camera.

[0113] The method described in this disclosure acquires feature point sets of depth images, confidence images, grayscale images, and color images collected by the camera module, respectively, and performs confidence filtering on the feature point sets based on a first preset condition. The feature point sets that meet the first preset condition are used to determine the position parameters of the camera module to obtain more accurate position parameters, thereby improving the effect of the camera module's autofocus and optical image stabilization functions.

[0114] In an optional embodiment, obtaining the first feature point set of the color image and the second feature point set of the grayscale image having a second correspondence includes:

[0115] Based on the current position parameters of the camera module, feature matching is performed on the color image and the grayscale image to obtain a first feature point set of the color image and a second feature point set of the grayscale image with a second correspondence.

[0116] Before determining new position parameters, feature detection and matching can be performed on the color image and the grayscale image based on the current relative position relationship between the first camera and the second camera, i.e., the current position parameters, to obtain a first feature point set of the color image and a second feature point set of the grayscale image with a first correspondence. The current position parameters may include the current baseline and rotation angle of the camera module.

[0117] After determining the position parameters, the method further includes:

[0118] Update the current position parameters using the defined position parameters.

[0119] After determining the new position parameters, these parameters can be used to update the current position parameters to improve the feature detection and matching results of the grayscale and color images. Alternatively, they can enable the first and second cameras to obtain a more accurate relative positional relationship, improving the autofocus and optical image stabilization effects of the camera module. In one example, the current position parameters can be updated in real-time or at set time intervals using the determined position parameters. In another example, the current position parameters can be updated using the determined position parameters after the camera module detects shaking.

[0120] The method described in this disclosure performs feature matching on the color image and the grayscale image to obtain a first feature point set of the color image and a second feature point set of the grayscale image with a second correspondence. Based on a first preset condition and the current position parameters, the feature point set is filtered for confidence. The feature point set that meets the first preset condition is used to determine the position parameters of the camera module to obtain more accurate position parameters.

[0121] In an optional embodiment, the feature matching of the color image and the grayscale image includes: adjusting the color image and the grayscale image to coplanar row alignment; and performing feature matching on the color image and the grayscale image adjusted to coplanar row alignment.

[0122] Before feature matching, the grayscale image captured by the first camera and the color image captured by the second camera are typically on different planes and exhibit shape distortion. The color image and the grayscale image can be adjusted to a coplanar alignment state so that they lie on the same plane and the y-axis coordinates of feature point pairs in both images are unified, reducing the computational burden during feature matching and position parameter calculation. This coplanar alignment is achieved by using a stereo correction algorithm to unify the planes of images captured by different cameras into a single plane, and by unifying the y-axis coordinates of the images captured by the different cameras to achieve the alignment effect.

[0123] The method described in this disclosure acquires feature point sets from depth images, confidence images, grayscale images, and color images captured by the camera module, adjusts the color images and grayscale images to be coplanar aligned, and performs confidence filtering on the feature point sets based on a first preset condition and current position parameters. The feature point sets that meet the first preset condition are used to determine the position parameters of the camera module to obtain more accurate position parameters, reducing the computational pressure in the feature matching and position parameter calculation process. The determined position parameters are then used to update the current position parameters, enabling the first camera and the second camera to obtain a more accurate relative position relationship, thereby improving the effect of the camera module's autofocus and optical image stabilization functions.

[0124] Figure 1B This disclosure illustrates a flowchart of another method for optimizing the parameters of a camera module according to an exemplary embodiment.

[0125] In step S1031, determining the third feature point set of the color image, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth image based on the first correspondence, the second correspondence, and the confidence level of the pixels in the confidence image, includes:

[0126] Based on the first correspondence and the second feature point set, a sixth feature point set of the depth image that has a fourth correspondence with the second feature point set is determined.

[0127] After feature detection and matching, the second feature point set of the grayscale image may be distorted compared to the grayscale image acquired by the depth camera. To establish a correspondence between the second feature point set and the depth image, a sixth feature point set of the depth image can be obtained by acquiring the feature points in the depth image that correspond to the second feature point set. In one example, if the grayscale image undergoes shape distortion during feature detection and matching, the image of the depth image can be transformed into the same coordinate system as the second feature point set by setting a relationship to obtain the sixth feature point set. The sixth feature point set and the second feature point set have a fourth correspondence, that is, the number of feature points is the same, and the depth value and confidence value of each feature point are the same.

[0128] In step 1032, based on the confidence of the pixels in the confidence image and the first correspondence, the confidence of each feature point in the sixth feature point set and the confidence of each feature point in the second feature point set are determined.

[0129] The depth image and the confidence image have a first correspondence, and the confidence level of each pixel in the confidence image can characterize the confidence level of each feature point in the sixth feature point set. Similarly, the second feature point set of the grayscale image and the sixth feature point set have a fourth correspondence, and the confidence level of each feature point in the sixth feature point set can characterize the confidence level of each point in the second feature point set.

[0130] In step S1033, the confidence level of the first feature point set is determined based on the confidence level of the second feature point set and the second correspondence.

[0131] The second feature point set of the grayscale image and the first feature point set of the color image have the second correspondence relationship, and the confidence level of each point in the second feature point set of the grayscale image can characterize the confidence level of each point in the first feature point set of the color image.

[0132] In step S1034, based on the confidence levels of the first feature point set, the second feature point set, the sixth feature point set, and the first preset condition, the third feature point set of the color image, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth image with a third correspondence are determined.

[0133] After obtaining the confidence level of each feature point in the first, second, and sixth feature point sets, the images can be filtered based on the confidence level of the depth values. This involves obtaining the third feature point set of the color image, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth image, all of which have a third correspondence and whose confidence levels satisfy a first preset condition. The first preset condition may include a set confidence level range that can improve the confidence level of the feature point sets.

[0134] The method described in this disclosure filters the feature point sets based on a first preset condition by using the confidence levels of the first feature point set, the second feature point set, and the sixth feature point set. The feature point sets that meet the first preset condition are then used to determine the position parameters of the camera module to obtain more accurate position parameters, thereby improving the autofocus and optical image stabilization functions of the camera module.

[0135] In an optional embodiment, determining the third feature point set of the color image, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth image having a third correspondence includes: removing feature points in the first feature point set whose confidence level is lower than a confidence threshold to obtain the third feature point set;

[0136] Remove feature points in the second feature point set whose confidence level is lower than the confidence threshold to obtain the fourth feature point set;

[0137] Remove feature points in the sixth feature point set whose confidence level is lower than the confidence threshold to obtain the fifth feature point set.

[0138] The confidence threshold is a preset confidence level threshold for feature points. In this disclosure, image feature points in the first, second, and sixth feature point sets with a confidence level lower than the confidence threshold are identified as feature points with low depth confidence and are removed. Feature points in the third, fourth, and sixth feature point sets with a confidence level higher than the confidence threshold are retained.

[0139] The method described in this disclosure removes feature points in the feature point set whose confidence level is lower than a confidence threshold, and uses the feature point set that meets the first preset condition to determine the position parameters of the camera module to obtain more accurate position parameters, thereby improving the effect of the camera module's autofocus and optical image stabilization functions.

[0140] Figure 2A This disclosure is a flowchart illustrating a baseline determination method according to an exemplary embodiment;

[0141] In step S201, the position parameters include a baseline. Determining the position parameters based on the third feature point set, the fourth feature point set, and the fifth feature point set includes: obtaining the baseline of each feature point pair between the third feature point set and the fourth feature point set.

[0142] The length of the baseline is used to characterize the positional relationship between the cameras and is a component of the camera module's positional parameters, positively correlated with the measurement range of the camera module. By acquiring the baseline of each feature point pair from the third feature point set of the color image and the fourth feature point set of the grayscale image, multiple baselines can be obtained to characterize the distance between the feature point pairs. Each feature point pair includes two feature points belonging to the third and fourth feature point sets, respectively.

[0143] In step S202, the baseline of each feature point pair is clustered, and the clustering result is determined as the baseline of the camera module.

[0144] Clustering the baselines of each feature point pair in the third feature point set of the color image and the fourth feature point set of the grayscale image, the obtained baseline length represents the distance between the optical centers of the cameras. In one example, after determining a new baseline, the baseline can be used to update the current baseline to optimize the feature detection and matching results of the grayscale image and the color image. Alternatively, it can enable the first camera and the second camera to obtain a more accurate relative positional relationship, improving the effect of the camera module's autofocus and optical image stabilization functions.

[0145] The method described in this disclosure uses a set of feature points that meet a first preset condition to determine the baseline of the camera module to obtain more accurate position parameters and updates the current baseline, so that the first camera and the second camera can obtain a more accurate relative position relationship, thereby improving the effect of the camera module's autofocus and optical image stabilization functions.

[0146] Figure 2B This is a flowchart illustrating another baseline determination method according to an exemplary embodiment of the present disclosure;

[0147] In step S2011, obtaining the baseline of each feature point pair between the third feature point set and the fourth feature point set includes: obtaining the coordinate distance of each feature point pair between the third feature point set and the fourth feature point set.

[0148] The coordinate distance between each feature point pair in the third and fourth feature point sets can be used to calculate the baseline of each feature point pair. When the color image and the grayscale image are stereo-corrected and adjusted to a line-aligned state, the y-axis coordinate values ​​of each feature point pair in the third and fourth feature point sets are the same. Therefore, the coordinate distance can include only the difference in the x-axis coordinate values, i.e., the coordinate distance in the x-axis direction of each feature point pair, to reduce the computational burden when acquiring the baseline.

[0149] In step S2012, according to the third correspondence, the depth value of each feature point in the fifth feature point set is determined as the depth value of the corresponding feature point pair between the third feature point set and the fourth feature point set.

[0150] The depth value of each feature point pair between the third and fourth feature point sets can be used to calculate the baseline of each feature point pair. The third, fourth, and fifth feature point sets that satisfy the first preset condition have the third correspondence relationship. Based on this third correspondence relationship, the depth value of each feature point in the third and fifth feature point sets can be obtained through the fifth feature point set of the depth image. The depth value represents the distance from the feature point to the camera module. When the color image and the grayscale image are stereo-corrected and adjusted to a coplanar state, in the feature point pairs, each corresponding feature point belonging to the third and fourth feature point sets has the same depth value, and this depth value can be determined as the depth value of each feature point pair.

[0151] In step S2013, the baseline of each feature point pair between the third feature point set and the fourth feature point set is determined based on the coordinate distance and depth value of each feature point pair and the focal length of the camera module.

[0152] The baseline of each feature point pair between the third feature point set and the fourth feature point set can be obtained by formula (1).

[0153]

[0154] Where B is the baseline of each feature point pair between the third feature point set and the fourth feature point set, z is the depth value of each feature point pair between the third feature point set and the fourth feature point set, and x... right-x left is the coordinate distance between each pair of feature points between the third and fourth feature point sets, and f is the focal length of the camera module. In the camera module, the focal lengths of the first and second cameras may differ. The focal length of any camera in the camera module, or any intermediate value between the two focal lengths of the cameras, can be determined through equivalent transformation to obtain the baseline of each pair of feature points. The focal length of the image can be obtained through offline calibration, which will not be elaborated further here.

[0155] The method described in this disclosure uses a set of feature points that meet a first preset condition to determine the baseline of the camera module to obtain more accurate position parameters and updates the current baseline, so that the first camera and the second camera can obtain a more accurate relative position relationship, thereby improving the effect of the camera module's autofocus and optical image stabilization functions.

[0156] Figure 2C This is a flowchart illustrating another baseline determination method according to an exemplary embodiment of the present disclosure;

[0157] In step S2011-1, according to the second correspondence, the disparity set between the first feature point set and the second feature point set is obtained, wherein the disparity set includes the coordinate distance of each feature point pair between the first feature point set and the second feature point set.

[0158] The second correspondence between the second feature point set of the grayscale image and the first feature point set of the color image includes the translation mapping relationship between each feature point in the grayscale image and the color image in the world coordinate system and the pixel coordinate system, as well as the translation mapping relationship between each pair of feature points. A transformation matrix (homography matrix) between the third and fourth feature point sets can be obtained based on the mapping relationship. Based on the distance between each pair of feature points in the homography matrix, a first disparity set between the first and second feature point sets can be obtained, which characterizes the coordinate distance between each pair of feature points in the first and second feature point sets.

[0159] In step S2011-2, the coordinate distance between each pair of feature points between the third feature point set and the fourth feature point set is obtained from the disparity set.

[0160] In the shooting scenario, the depth values ​​of the object's edge acquired by the second camera have a poor confidence level, which can easily lead to errors in the parameter calculation of the camera module. In one example, after acquiring the feature point set of the confidence image, the disparity set can be filtered based on the confidence level of the depth values ​​to obtain a second disparity set whose confidence level meets a first preset condition. The first preset condition may include a set confidence range that can improve the confidence level of the feature point set. In one example, feature points in the first disparity set below the confidence threshold can be identified as feature points with low depth confidence levels and removed. Feature points in the disparity set above the confidence threshold are retained. Each feature point in the second disparity set is used to represent the coordinate distance between each pair of feature points in the third and fourth feature point sets. Compared to obtaining the coordinate distance of each pair of feature points based on the third and fourth feature point sets, obtaining the coordinate distance of the pair of feature points from the second disparity set further reduces computational burden.

[0161] The method described in this disclosure filters the feature point set based on the disparity set of the color image and grayscale image with confidence based on the first preset condition and the current position parameter. The feature point set that meets the first preset condition is used to determine the baseline of the camera module to obtain more accurate position parameters and update the current baseline, so that the first camera and the second camera can obtain a more accurate relative position relationship, thereby improving the effect of the camera module's autofocus and optical image stabilization functions.

[0162] Figure 3 This disclosure is a flowchart illustrating a method for determining a rotation angle line according to an exemplary embodiment;

[0163] In step S301, the position parameter includes a rotation angle. Determining the position parameter based on the third feature point set, the fourth feature point set, and the fifth feature point set includes:

[0164] Based on the third correspondence and the depth value of each feature point in the fifth feature point set, the seventh feature point set of the color image and the eighth feature point set of the grayscale image with the fifth correspondence are determined, wherein the depth value of the feature point in the fifth feature point set corresponding to each feature point in the eighth feature point set satisfies the second preset condition.

[0165] The rotation angle characterizes the angular difference between the camera mirrors and is a component of the camera module's position parameters. The third feature point set of the color image, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth image have the third correspondence. The depth value of each feature point in the third and fourth feature point sets can be determined using the depth value of each feature point in the fifth feature point set. In a shooting scenario, as the distance from the subject to the camera module increases, errors in the parameter calculation of the camera module are likely to occur. After obtaining the fifth feature point set of the depth map, depth value filtering can be performed on the third and fourth feature point sets based on the depth value (i.e., the distance from the subject to the camera module), resulting in the seventh feature point set of the color image and the eighth feature point set of the grayscale image having a fifth correspondence and whose depth values ​​satisfy a second preset condition. Based on the fifth correspondence, the depth value of each feature point in the seventh and eighth feature point sets can be obtained using the fifth feature point set.

[0166] In step S302, the rotation angle of each feature point pair between the seventh feature point set and the eighth feature point set is obtained.

[0167] The fourth correspondence between the seventh and eighth feature point sets includes a rotation mapping relationship between each feature point pair. The homography matrix of each feature point pair between the seventh and eighth feature point sets can be obtained based on this mapping relationship. By decomposing the homography matrix, the rotation and translation matrices of each feature point pair can be obtained. Based on the rotation matrix, the rotation angle of each feature point pair between the seventh and eighth feature point sets can be obtained.

[0168] In step S303, the rotation angle of each feature point pair is clustered, and the clustering result is determined as the rotation angle of the camera module.

[0169] Clustering the rotation angles of each feature point pair in the seventh and eighth feature point sets, the obtained rotation angles represent the mirror angle difference between the first and second cameras. In one example, after determining a new rotation angle, the rotation angle can be used to update the current rotation angle to optimize the feature detection and matching results of the grayscale and color images. Alternatively, it can enable the first and second cameras to obtain a more accurate relative positional relationship, improving the effectiveness of the camera module's autofocus and optical image stabilization functions.

[0170] The method described in this disclosure acquires feature point sets from depth images, confidence images, grayscale images, and color images captured by the camera module, respectively. Based on a first preset condition and current position parameters, the feature point sets are subjected to confidence filtering. The feature point sets that meet the first preset condition are used to determine the rotation angle of the camera module to obtain more accurate position parameters, and the current rotation angle is updated. This enables the first camera and the second camera to obtain a more accurate relative position relationship, thereby improving the effect of the camera module's autofocus and optical image stabilization functions.

[0171] In an optional embodiment, determining the seventh feature point set of the color image and the eighth feature point set of the grayscale image, which have a fifth correspondence, based on the fourth correspondence, includes:

[0172] Remove feature points with depth values ​​greater than the first depth threshold from the third feature point set and the fourth feature point set respectively to obtain the seventh feature point set and the eighth feature point set; or, remove feature points with depth values ​​greater than the second depth threshold and depth values ​​less than the third depth threshold from the third feature point set and the fourth feature point set respectively to obtain the seventh feature point set and the eighth feature point set.

[0173] In one example, feature points in the seventh and eighth feature point sets with depth values ​​greater than a depth threshold can be identified as feature points outside the detection range of the camera module and removed. Feature points in the seventh and eighth feature point sets with depth values ​​less than the depth threshold are retained. The depth threshold can be determined based on the current baseline length of the camera module. In one example, the depth threshold can be set to 25 times the current baseline. If the current baseline length is 20mm, the depth threshold is 50cm, meaning feature points outside 50cm are filtered out.

[0174] In another example, for the process of obtaining the rotation angle, obtaining the homography matrix of a subset of feature point pairs can be used instead of obtaining the homography matrix of all feature point pairs with depth values ​​less than the depth threshold, thereby reducing the computational burden of obtaining the homography matrix. In one example, the depth threshold is 50cm, meaning that after filtering feature points outside 50cm, feature points within 40cm can also be removed, retaining feature points with a depth range of 40cm to 50cm, or the previous depth threshold can be omitted, retaining feature points within any sub-range of 50cm. The rotation angle of each feature point pair between the seventh and eighth feature point sets whose depth values ​​satisfy the preset depth range is then obtained.

[0175] The method described in this disclosure acquires feature point sets from depth images, confidence images, grayscale images, and color images captured by the camera module, respectively. Based on a first preset condition and current position parameters, the feature point sets are subjected to confidence filtering. The feature point sets that meet the first preset condition are used to determine the rotation angle of the camera module to obtain more accurate position parameters, and the current rotation angle is updated. This enables the first camera and the second camera to obtain a more accurate relative position relationship, thereby improving the effect of the camera module's autofocus and optical image stabilization functions.

[0176] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should know that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps may be performed in other orders or simultaneously.

[0177] Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by this disclosure.

[0178] Corresponding to the aforementioned application function implementation method embodiments, this disclosure also provides embodiments of application function implementation apparatus and corresponding terminals.

[0179] A block diagram of a device for optimizing the parameters of a camera module, as illustrated in an exemplary embodiment of this disclosure, is shown below. Figure 4 As shown, the camera module includes a first camera and a second camera, and the device includes:

[0180] Image acquisition module 401: used to acquire a color image captured by the first camera, and a depth image, a grayscale image and a confidence image captured by the second camera, wherein there is a first correspondence between the pixels of the depth image, the pixels of the grayscale image and the pixels of the confidence image;

[0181] Online calibration module 402: used to obtain the first feature point set of the color image and the second feature point set of the grayscale image having a second correspondence relationship;

[0182] Confidence filtering module 403: is used to determine the third feature point set of the color image with the third correspondence, the fourth feature point set of the grayscale portrait and the fifth feature point set of the depth image based on the first correspondence, the second correspondence and the confidence of the pixel points of the confidence image, wherein the confidence of the pixel points of the confidence image corresponding to each feature point in the fourth feature point set satisfies the first preset condition;

[0183] Parameter determination module 404: used to determine position parameters based on the third feature point set, the fourth feature point set and the fifth feature point set, wherein the position parameters are used to characterize the relative positional relationship between the first camera and the second camera.

[0184] In any embodiment of this disclosure, the online calibration module acquires a first feature point set of the color image and a second feature point set of the grayscale image having a second correspondence, for the purpose of:

[0185] Based on the current position parameters of the camera module, feature matching is performed on the color image and the grayscale image to obtain a first feature point set of the color image and a second feature point set of the grayscale image with a second correspondence.

[0186] After the parameter determination module determines the position parameters, it also includes a parameter update module, used for:

[0187] Update the current position parameters using the determined position parameters.

[0188] In any embodiment of this disclosure, the feature matching of the color image and the grayscale image is used for:

[0189] Adjust the color image and the grayscale image to be aligned in coplanar rows;

[0190] Feature matching is performed on the color image and the grayscale image that have been adjusted to coplanar row alignment.

[0191] In conjunction with any embodiment of this disclosure, the confidence filtering module determines, based on the first correspondence, the second correspondence, and the confidence of the pixels in the confidence image, a third feature point set of the color image with a third correspondence, a fourth feature point set of the grayscale portrait, and a fifth feature point set of the depth map, for the following purposes:

[0192] Based on the first correspondence and the second feature point set, determine the sixth feature point set of the depth image that has a fourth correspondence with the second feature point set;

[0193] Based on the confidence scores of the pixels in the confidence image and the first correspondence, determine the confidence scores of each feature point in the sixth feature point set and each feature point in the second feature point set;

[0194] Based on the confidence level of each feature point in the second feature point set and the second correspondence, determine the confidence level of each feature point in the first feature point set;

[0195] Based on the confidence levels of the first feature point set, the second feature point set, the sixth feature point set, and the first preset condition, the third feature point set of the color image, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth image with a third correspondence are determined.

[0196] In conjunction with any embodiment of this disclosure, the confidence filtering module determines a third feature point set of the color image, a fourth feature point set of the grayscale portrait, and a fifth feature point set of the depth image that have a third correspondence, for the following purposes:

[0197] Remove feature points in the first feature point set whose confidence level is lower than the confidence threshold to obtain the third feature point set;

[0198] Remove feature points in the second feature point set whose confidence level is lower than the confidence threshold to obtain the fourth feature point set;

[0199] Remove feature points in the sixth feature point set whose confidence level is lower than the confidence threshold to obtain the fifth feature point set.

[0200] In any embodiment of this disclosure, the position parameters include a baseline; the parameter determination module determines the position parameters based on the third feature point set, the fourth feature point set, and the fifth feature point set, for:

[0201] Obtain the baseline of each feature point pair between the third feature point set and the fourth feature point set;

[0202] Cluster the baselines of each feature point pair, and determine the clustering results as the baseline of the camera module.

[0203] In any embodiment of this disclosure, obtaining the baseline of each feature point pair between the third feature point set and the fourth feature point set is used for:

[0204] Obtain the coordinate distance of each feature point pair between the third feature point set and the fourth feature point set, wherein the feature point pair includes two feature points that belong to the third feature point set and the fourth feature point set respectively;

[0205] Based on the third correspondence and the fifth feature point set, obtain the depth value of each feature point pair between the third feature point set and the fourth feature point set;

[0206] Based on the coordinate distance and depth values ​​of each feature point pair and the focal length of the camera module, the baseline of each feature point pair between the third feature point set and the fourth feature point set is obtained.

[0207] In any embodiment of this disclosure, obtaining the coordinate distance of each feature point pair between the third feature point set and the fourth feature point set is used for:

[0208] According to the second correspondence, the disparity set between the first feature point set and the second feature point set is obtained, wherein the disparity set includes the coordinate distance of each feature point pair between the first feature point set and the second feature point set;

[0209] The coordinate distance between each pair of feature points between the third feature point set and the fourth feature point set is obtained from the disparity set.

[0210] In any embodiment of this disclosure, the position parameter includes a rotation angle;

[0211] The parameter determination module determines position parameters based on the third feature point set, the fourth feature point set, and the fifth feature point set, for the purpose of:

[0212] Based on the third correspondence and the depth value of each feature point in the fifth feature point set, the seventh feature point set of the color image and the eighth feature point set of the grayscale image with the fifth correspondence are determined, wherein the depth value of the feature point in the fifth feature point set corresponding to each feature point in the eighth feature point set satisfies the second preset condition.

[0213] Obtain the rotation angle of each feature point pair between the seventh feature point set and the eighth feature point set;

[0214] Cluster the rotation angles of each feature point pair, and determine the clustering results as the rotation angles of the camera module.

[0215] In any embodiment of this disclosure, determining the seventh feature point set of the color image and the eighth feature point set of the grayscale image having a fifth correspondence is used for:

[0216] Remove feature points with depth values ​​greater than a first depth threshold from the third and fourth feature point sets respectively to obtain the seventh and eighth feature point sets; or,

[0217] Remove feature points with depth values ​​greater than the second depth threshold and feature points with depth values ​​less than the third depth threshold from the third feature point set and the fourth feature point set respectively to obtain the seventh feature point set and the eighth feature point set.

[0218] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0219] Figure 5 A block diagram of an electronic device is shown according to an exemplary embodiment of the present disclosure.

[0220] Please refer to the appendix. Figure 5 The diagram illustrates, for example, a block diagram of an electronic device. For instance, device 500 could be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0221] Reference Figure 5 The device 500 may include one or more of the following components: a processing component 502, a memory 504, a power supply component 506, a multimedia component 508, an audio component 510, an input / output (I / O) interface 512, a sensor component 514, and a communication component 516.

[0222] Processing component 502 typically controls the overall operation of device 500, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 502 may include one or more modules to facilitate interaction between processing component 502 and other components. For example, processing component 502 may include a multimedia module to facilitate interaction between multimedia component 508 and processing component 502.

[0223] Memory 504 is configured to store various types of data to support the operation of device 500. Examples of this data include instructions for any application or method operating on device 500, contact data, phonebook data, messages, pictures, videos, etc. Memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0224] The power supply component 506 provides power to the various components of the device 500. The power supply component 506 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 500.

[0225] Multimedia component 508 includes a screen that provides an output interface between the device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When the device 500 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0226] Audio component 510 is configured to output and / or input audio signals. For example, audio component 510 includes a microphone (MIC) configured to receive external audio signals when device 500 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 504 or transmitted via communication component 516. In some embodiments, audio component 510 includes a speaker for outputting audio signals.

[0227] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0228] Sensor assembly 514 includes one or more sensors for providing state assessments of various aspects of device 500. For example, sensor assembly 514 may detect the on / off state of device 500, the relative positioning of components such as the display and keypad of device 500, changes in the position of device 500 or a component of device 500, the presence or absence of user contact with device 500, the orientation or acceleration / deceleration of device 500, and temperature changes of device 500. Sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 514 may include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 514 may include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0229] Communication component 516 is configured to facilitate wired or wireless communication between device 500 and other devices. Device 500 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G or 5G, or combinations thereof. In one exemplary embodiment, communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 516 includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0230] In an exemplary embodiment, the device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the power supply method of the aforementioned electronic device.

[0231] In an exemplary embodiment, this disclosure provides a non-transitory computer storage medium including instructions, such as a memory 504 including instructions, which can be executed by a processor 520 of a device 500 to complete the power supply method of the aforementioned electronic device. For example, the non-transitory computer storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0232] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0233] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for optimizing the parameters of a camera module, characterized in that, The camera module includes a first camera and a second camera, and the method includes: The color image captured by the first camera, and the depth image, grayscale image and confidence image captured by the second camera are acquired, wherein there is a first correspondence between the pixels of the depth image, the pixels of the grayscale image and the pixels of the confidence image; Obtain a first feature point set of the color image and a second feature point set of the grayscale image that have a second correspondence relationship; the first feature point set and the second feature point set that have a second correspondence relationship have the same number of feature points, and the depth value and confidence value of each pair of feature points are the same; Based on the first correspondence, the second correspondence, and the confidence of the pixels in the confidence image, a third feature point set of the color image with a third correspondence, a fourth feature point set of the grayscale image, and a fifth feature point set of the depth image are determined, wherein the confidence of the pixels in the confidence image corresponding to each feature point in the fourth feature point set satisfies a first preset condition. Based on the third feature point set, the fourth feature point set, and the fifth feature point set, position parameters are determined, which are used to characterize the relative positional relationship between the first camera and the second camera. The step of determining the third feature point set of the color image, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth image based on the first correspondence, the second correspondence, and the confidence level of the pixels in the confidence image, includes: Based on the first correspondence and the second feature point set, determine the sixth feature point set of the depth image that has a fourth correspondence with the second feature point set; Based on the confidence scores of the pixels in the confidence image and the first correspondence, determine the confidence scores of each feature point in the sixth feature point set and each feature point in the second feature point set; Based on the confidence level of each feature point in the second feature point set and the second correspondence, determine the confidence level of each feature point in the first feature point set; Based on the confidence levels of the first feature point set, the second feature point set, the sixth feature point set, and the first preset condition, the third feature point set of the color image, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth image with a third correspondence are determined.

2. The method according to claim 1, characterized in that, The step of obtaining the first feature point set of the color image and the second feature point set of the grayscale image having a second correspondence includes: Based on the current position parameters of the camera module, feature matching is performed on the color image and the grayscale image to obtain a first feature point set of the color image and a second feature point set of the grayscale image with a second correspondence. After determining the position parameters, the method further includes: Update the current position parameters using the defined position parameters.

3. The method according to claim 2, characterized in that, The feature matching of the color image and the grayscale image includes: Adjust the color image and the grayscale image to be aligned in coplanar rows; Feature matching is performed on the color image and the grayscale image that have been adjusted to coplanar row alignment.

4. The method according to claim 1, characterized in that, The step of determining the third feature point set of the color image, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth image that have a third correspondence includes: Remove feature points in the first feature point set whose confidence level is lower than the confidence threshold to obtain the third feature point set; Remove feature points in the second feature point set whose confidence level is lower than the confidence threshold to obtain the fourth feature point set; Remove feature points in the sixth feature point set whose confidence level is lower than the confidence threshold to obtain the fifth feature point set.

5. The method according to claim 1, characterized in that, The location parameters include a baseline; The step of determining the position parameters based on the third feature point set, the fourth feature point set, and the fifth feature point set includes: Obtain the baseline of each feature point pair between the third feature point set and the fourth feature point set; Cluster the baselines of each feature point pair, and determine the clustering results as the baseline of the camera module.

6. The method according to claim 5, characterized in that, The step of obtaining the baseline for each feature point pair between the third feature point set and the fourth feature point set includes: Obtain the coordinate distance between each pair of feature points between the third feature point set and the fourth feature point set; Based on the third correspondence, the depth value of each feature point in the fifth feature point set is determined as the depth value of the corresponding feature point pair between the third feature point set and the fourth feature point set; Based on the coordinate distance and depth values ​​of each feature point pair and the focal length of the camera module, the baseline of each feature point pair between the third feature point set and the fourth feature point set is determined.

7. The method according to claim 6, characterized in that, The step of obtaining the coordinate distance of each feature point pair between the third feature point set and the fourth feature point set includes: According to the second correspondence, the disparity set between the first feature point set and the second feature point set is obtained, wherein the disparity set includes the coordinate distance of each feature point pair between the first feature point set and the second feature point set; The coordinate distance between each pair of feature points between the third feature point set and the fourth feature point set is obtained from the disparity set.

8. The method according to claim 1, characterized in that, The position parameters include the rotation angle; The step of determining the position parameters based on the third feature point set, the fourth feature point set, and the fifth feature point set includes: Based on the third correspondence and the depth value of each feature point in the fifth feature point set, the seventh feature point set of the color image and the eighth feature point set of the grayscale image with the fifth correspondence are determined, wherein the depth value of the feature point in the fifth feature point set corresponding to each feature point in the eighth feature point set satisfies the second preset condition. Obtain the rotation angle of each feature point pair between the seventh feature point set and the eighth feature point set; Cluster the rotation angles of each feature point pair, and determine the clustering results as the rotation angles of the camera module.

9. The method according to claim 8, characterized in that, The determination of the seventh feature point set of the color image and the eighth feature point set of the grayscale image having a fifth correspondence includes: Remove feature points with depth values ​​greater than a first depth threshold from the third and fourth feature point sets respectively to obtain the seventh and eighth feature point sets; or, Remove feature points with depth values ​​greater than the second depth threshold and feature points with depth values ​​less than the third depth threshold from the third feature point set and the fourth feature point set respectively to obtain the seventh feature point set and the eighth feature point set.

10. A parameter optimization device for a camera module, characterized in that, The camera module includes a first camera and a second camera, and the device includes: Image acquisition module: used to acquire a color image captured by the first camera, and a depth image, a grayscale image and a confidence image captured by the second camera, wherein there is a first correspondence between the pixels of the depth image, the pixels of the grayscale image and the pixels of the confidence image; Online calibration module: used to obtain a first feature point set of the color image and a second feature point set of the grayscale image with a second correspondence; the first feature point set and the second feature point set with the second correspondence have the same number of feature points, and the depth value and confidence value of each pair of feature points are the same; Confidence filtering module: used to determine the third feature point set of the color image with the third correspondence, the fourth feature point set of the grayscale image and the fifth feature point set of the depth image based on the first correspondence, the second correspondence and the confidence of the pixel points of the confidence image, wherein the confidence of the pixel points of the confidence image corresponding to each feature point in the fourth feature point set satisfies the first preset condition. Parameter determination module: used to determine position parameters based on the third feature point set, the fourth feature point set and the fifth feature point set, wherein the position parameters are used to characterize the relative positional relationship between the first camera and the second camera; The confidence filtering module determines, based on the first correspondence, the second correspondence, and the confidence of the pixels in the confidence image, the third feature point set of the color image with the third correspondence, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth map, for the following purposes: Based on the first correspondence and the second feature point set, determine the sixth feature point set of the depth image that has a fourth correspondence with the second feature point set; Based on the confidence scores of the pixels in the confidence image and the first correspondence, determine the confidence scores of each feature point in the sixth feature point set and each feature point in the second feature point set; Based on the confidence level of each feature point in the second feature point set and the second correspondence, determine the confidence level of each feature point in the first feature point set; Based on the confidence levels of the first feature point set, the second feature point set, the sixth feature point set, and the first preset condition, the third feature point set of the color image, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth image with a third correspondence are determined.

11. The apparatus according to claim 10, characterized in that, The online calibration module acquires the first feature point set of the color image and the second feature point set of the grayscale image, which have a second correspondence, for the purpose of: Based on the current position parameters of the camera module, feature matching is performed on the color image and the grayscale image to obtain a first feature point set of the color image and a second feature point set of the grayscale image with a second correspondence. After the parameter determination module determines the position parameters, it also includes a parameter update module, used for: Update the current position parameters using the determined position parameters.

12. The apparatus according to claim 11, characterized in that, The feature matching of the color image and the grayscale image is used for: Adjust the color image and the grayscale image to be aligned in coplanar rows; Feature matching is performed on the color image and the grayscale image that have been adjusted to coplanar row alignment.

13. The apparatus according to claim 10, characterized in that, The confidence filtering module determines the third feature point set of the color image, the fourth feature point set of the grayscale image, and the fifth feature point set of the depth image that have a third correspondence, for the following purposes: Remove feature points in the first feature point set whose confidence level is lower than the confidence threshold to obtain the third feature point set; Remove feature points in the second feature point set whose confidence level is lower than the confidence threshold to obtain the fourth feature point set; Remove feature points in the sixth feature point set whose confidence level is lower than the confidence threshold to obtain the fifth feature point set.

14. The apparatus according to claim 10, characterized in that, The location parameters include a baseline; the parameter determination module determines the location parameters based on the third feature point set, the fourth feature point set, and the fifth feature point set, for the purpose of: Obtain the baseline of each feature point pair between the third feature point set and the fourth feature point set; Cluster the baselines of each feature point pair, and determine the clustering results as the baseline of the camera module.

15. The apparatus according to claim 14, characterized in that, The step of obtaining the baseline of each feature point pair between the third feature point set and the fourth feature point set is used for: Obtain the coordinate distance of each feature point pair between the third feature point set and the fourth feature point set, wherein the feature point pair includes two feature points that belong to the third feature point set and the fourth feature point set respectively; Based on the third correspondence and the fifth feature point set, obtain the depth value of each feature point pair between the third feature point set and the fourth feature point set; Based on the coordinate distance and depth values ​​of each feature point pair and the focal length of the camera module, the baseline of each feature point pair between the third feature point set and the fourth feature point set is obtained.

16. The apparatus according to claim 15, characterized in that, The step of obtaining the coordinate distance between each pair of feature points between the third feature point set and the fourth feature point set is used for: According to the second correspondence, the disparity set between the first feature point set and the second feature point set is obtained, wherein the disparity set includes the coordinate distance of each feature point pair between the first feature point set and the second feature point set; The coordinate distance between each pair of feature points between the third feature point set and the fourth feature point set is obtained from the disparity set.

17. The apparatus according to claim 10, characterized in that, The position parameters include the rotation angle; The parameter determination module determines position parameters based on the third feature point set, the fourth feature point set, and the fifth feature point set, for the purpose of: Based on the third correspondence and the depth value of each feature point in the fifth feature point set, the seventh feature point set of the color image and the eighth feature point set of the grayscale image with the fifth correspondence are determined, wherein the depth value of the feature point in the fifth feature point set corresponding to each feature point in the eighth feature point set satisfies the second preset condition. Obtain the rotation angle of each feature point pair between the seventh feature point set and the eighth feature point set; Cluster the rotation angles of each feature point pair, and determine the clustering results as the rotation angles of the camera module.

18. The apparatus according to claim 17, characterized in that, The determination of the seventh feature point set of the color image and the eighth feature point set of the grayscale image, which have a fifth correspondence, is used for: Remove feature points with depth values ​​greater than the first depth threshold from the third feature point set and the fourth feature point set respectively to obtain the seventh feature point set and the eighth feature point set; or, Remove feature points with depth values ​​greater than the second depth threshold and feature points with depth values ​​less than the third depth threshold from the third feature point set and the fourth feature point set respectively to obtain the seventh feature point set and the eighth feature point set.

19. An electronic device, characterized in that, The electronic device includes: Memory is used to store processor-executable instructions; A processor is configured to execute executable instructions in the memory to implement the steps of the method according to any one of claims 1 to 9.

20. A computer storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method described in any one of claims 1 to 9.

21. A camera module, characterized in that, Includes the electronic device as described in claim 19.

Images