Camera registration method, device, communication device and storage medium

Abstract

The present disclosure relates to a camera registration method, apparatus, device and medium, which are applied to an electronic device including at least one RGB camera and a depth camera, and include: correcting an RGB image collected by the RGB camera and an energy map collected by the depth camera according to the homography matrix between the RGB camera and the depth camera to obtain an RGB corrected image and an energy corrected image; determining a target feature point pair from the matching feature point pairs on the RGB corrected image and the energy corrected image, and the difference between the first parallax of the first feature point and the second parallax of the second feature point in the target feature point pair is greater than a preset threshold; based on the difference, adjusting the homography matrix according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio; and re-correcting the RGB image and the energy map according to the adjusted homography matrix. The registration accuracy between the RGB camera and the depth camera can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of camera technologies, and in particular, to a camera registration method, apparatus, communication device, and storage medium. Background Art

[0002] In a scenario where an RGB camera and a depth camera are used in combination, it is necessary to register the relative positional relationship between the RGB camera and the depth camera to improve the accuracy of image acquisition. In related scenarios, registration is achieved by respectively performing distortion elimination and row alignment on the left and right views corresponding to the two cameras, so that the imaging origin coordinates of the left and right views are the same, the optical axes of the two cameras are parallel, the left and right imaging planes are coplanar, and the epipolar lines are row-aligned. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present disclosure provides a camera registration method, apparatus, communication device, and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided a camera registration method applied to an electronic device including at least one RGB camera and at least one depth camera. The method includes:

[0005] Correcting an RGB image collected by the RGB camera and an energy map collected by the depth camera according to a homography matrix between the RGB camera and the depth camera to obtain an RGB corrected image and an energy corrected image;

[0006] Determining a target feature point pair from the feature point pairs that match on the RGB corrected image and the energy corrected image. The feature point pair includes a first feature point on the RGB corrected image and a second feature point on the energy corrected image, and the difference between the first parallax of the first feature point in the target feature point pair and the second parallax of the second feature point in the target feature point pair is greater than a preset threshold;

[0007] Based on the difference, adjusting the homography matrix according to a baseline distance between the RGB camera and the depth camera and a preset adjustment ratio;

[0008] Re-correcting the RGB image and the energy map according to the adjusted homography matrix.

[0009] Optionally, the determining a target feature point pair from the feature point pairs that match on the RGB corrected image and the energy corrected image includes:

[0010] For any of the feature point pairs, when the difference between the first disparity of the first feature point and the second disparity of the second feature point in the feature point pair is greater than the preset threshold, the feature point pair is used as the target feature point pair; or,

[0011] Determine candidate feature point pairs with a confidence greater than a preset confidence threshold from each of the feature point pairs, and for any of the candidate feature point pairs, when the difference between the first disparity of the first feature point and the second disparity of the second feature point in the candidate feature point pair is greater than the preset threshold, the candidate feature point pair is used as the target feature point pair.

[0012] Optionally, the adjusting the homography matrix based on the difference, according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio, includes:

[0013] Determine the magnitude of the parameter adjustment amount according to the baseline distance and the preset adjustment ratio;

[0014] When the difference between the first disparity and the second disparity is greater than the preset threshold and the first disparity is greater than the second disparity, increase the homography matrix by the parameter adjustment amount; or,

[0015] When the difference between the first disparity and the second disparity is less than the preset threshold and the first disparity is less than the second disparity, decrease the homography matrix by the parameter adjustment amount.

[0016] Optionally, the adjusting the homography matrix based on the difference, according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio, includes:

[0017] Determine the target parameters in the homography matrix that characterize the translational and rotational relationships between the RGB camera and the depth camera;

[0018] Based on the difference, according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio, adjust the target parameters in the homography matrix.

[0019] Optionally, the method includes:

[0020] After re-correcting the RGB image and the energy map according to the adjusted homography matrix, return to execute the step of determining the target feature point pair from the feature point pairs that match on the RGB corrected map and the energy corrected map until there are no target feature point pairs among the feature point pairs on the new RGB corrected map and the new energy corrected map.

[0021] Optionally, the determination of the first parallax of the first feature point includes:

[0022] Determine the RGB coordinates of the first feature point in the RGB rectified image and the energy coordinates in the energy rectified image;

[0023] Take the difference between the RGB coordinates and the energy coordinates as the first parallax.

[0024] Optionally, the determination of the second parallax of the second feature point includes:

[0025] Determine the first imaging coordinates of the second feature point in the depth camera and the second imaging coordinates in the RGB camera;

[0026] Take the difference between the first imaging coordinates and the second imaging coordinates as the second parallax.

[0027] Optionally, the determination of the first imaging coordinates of the second feature point in the depth camera includes:

[0028] Obtain the depth value of the second feature point and determine the first distance from the second feature point to the optical axis of the depth camera in the world coordinate system;

[0029] Determine the first imaging coordinates of the second feature point in the depth camera according to the focal length of the depth camera, the depth value, and the first distance.

[0030] Optionally, the determination of the second imaging coordinates of the second feature point in the RGB camera includes:

[0031] Determine the second distance from the second feature point to the optical axis of the RGB camera in the world coordinate system according to the center distance between the RGB camera and the depth camera;

[0032] Determine the second imaging coordinates of the second feature point in the RGB camera according to the focal length of the RGB camera, the depth value, and the second distance.

[0033] According to the second aspect of the embodiments of the present disclosure, there is provided a camera registration device applied to an electronic device including at least one RGB camera and at least one depth camera. The device includes:

[0034] A first rectification module configured to rectify the RGB image collected by the RGB camera and the energy map collected by the depth camera according to the homography matrix between the RGB camera and the depth camera to obtain an RGB rectified image and an energy rectified image;

[0035] A determination module, configured to determine a target feature point pair from the feature point pairs that match on the RGB correction map and the energy correction map, where the feature point pair includes a first feature point on the RGB correction map and a second feature point on the energy correction map, and the difference between the first parallax of the first feature point in the target feature point pair and the second parallax of the second feature point in the target feature point pair is greater than a preset threshold;

[0036] An adjustment module, configured to adjust the homography matrix based on the difference, according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio;

[0037] A second correction module, configured to perform secondary correction on the RGB image and the energy map according to the adjusted homography matrix.

[0038] Optionally, the determination module is configured to: for any one of the feature point pairs, when the difference between the first parallax of the first feature point in the feature point pair and the second parallax of the second feature point is greater than the preset threshold, use the feature point pair as the target feature point pair; or,

[0039] Determine candidate feature point pairs with a confidence greater than a preset confidence threshold from each of the feature point pairs, and for any one of the candidate feature point pairs, when the difference between the first parallax of the first feature point in the candidate feature point pair and the second parallax of the second feature point is greater than the preset threshold, use the candidate feature point pair as the target feature point pair.

[0040] Optionally, the adjustment module is configured to determine the magnitude of the parameter adjustment amount according to the baseline distance and the preset adjustment ratio;

[0041] When the difference between the first parallax and the second parallax is greater than the preset threshold and the first parallax is greater than the second parallax, increase the homography matrix by the parameter adjustment amount; or,

[0042] When the difference between the first parallax and the second parallax is less than the preset threshold and the first parallax is less than the second parallax, decrease the homography matrix by the parameter adjustment amount.

[0043] Optionally, the adjustment module is configured to determine the target parameters in the homography matrix that represent the translation and rotation relationship between the RGB camera and the depth camera;

[0044] Based on the difference, adjust the target parameter in the homography matrix according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio.

[0045] Optionally, the determining module is further configured to, after re-correcting the RGB image and the energy map according to the adjusted homography matrix, return to execute the step of determining the target feature point pair from the matching feature point pairs on the RGB corrected map and the energy corrected map until there is no target feature point pair among the feature point pairs on the new RGB corrected map and the new energy corrected map.

[0046] Optionally, the determining module is configured to determine the RGB coordinates of the first feature point in the RGB corrected map and the energy coordinates in the energy corrected map;

[0047] Take the difference between the RGB coordinates and the energy coordinates as the first parallax.

[0048] Optionally, the determining module is configured to determine the first imaging coordinates of the second feature point in the depth camera and the second imaging coordinates in the RGB camera;

[0049] Take the difference between the first imaging coordinates and the second imaging coordinates as the second parallax.

[0050] Optionally, the determining module is configured to obtain the depth value of the second feature point and determine the first distance from the second feature point to the optical axis of the depth camera in the world coordinate system;

[0051] Determine the first imaging coordinates of the second feature point in the depth camera according to the focal length of the depth camera, the depth value, and the first distance.

[0052] Optionally, the determining module is configured to determine the second distance from the second feature point to the optical axis of the RGB camera in the world coordinate system according to the center distance between the RGB camera and the depth camera;

[0053] Determine the second imaging coordinates of the second feature point in the RGB camera according to the focal length of the RGB camera, the depth value, and the second distance.

[0054] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0055] A processor;

[0056] A memory for storing instructions executable by the processor;

[0057] Wherein, the processor is configured to:

[0058] According to the homography matrix between the RGB camera and the depth camera, correct the RGB image collected by the RGB camera and the energy map collected by the depth camera to obtain an RGB corrected image and an energy corrected image;

[0059] Determine target feature point pairs from the feature point pairs that match on the RGB corrected image and the energy corrected image, the feature point pairs include a first feature point on the RGB corrected image and a second feature point on the energy corrected image, and the difference between the first parallax of the first feature point in the target feature point pair and the second parallax of the second feature point in the target feature point pair is greater than a preset threshold;

[0060] Based on the difference, according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio, adjust the homography matrix;

[0061] Re-correct the RGB image and the energy map according to the adjusted homography matrix.

[0062] According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium having computer program instructions stored thereon, and when the program instructions are executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0063] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

[0064] By correcting the RGB image and the energy map through the homography matrix, and determining target feature point pairs from the feature point pairs that match on the obtained RGB corrected image and energy corrected image, and further when the difference between the first parallax of the first feature point in the target feature point pair and the second parallax of the second feature point in the target feature point pair is greater than a preset threshold, based on this difference, according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio, adjusting the homography matrix can improve the registration accuracy between the RGB camera and the depth camera, and further can improve the accuracy of related applications in a scenario where the RGB camera and the depth camera are both configured.

[0065] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0067] Figure 1 is a flowchart of a camera registration method shown according to an exemplary embodiment.

[0068] Figure 2 is a schematic diagram of obtaining a depth map from an energy map shown according to an exemplary embodiment.

[0069] Figure 3 is a schematic diagram of camera calibration shown according to an exemplary embodiment.

[0070] Figure 4 is a flowchart of determining a first disparity of a first feature point shown according to an exemplary embodiment.

[0071] Figure 5 is a flowchart of determining a second disparity of a second feature point shown according to an exemplary embodiment.

[0072] Figure 6 is a schematic diagram of pixel point imaging shown according to an exemplary embodiment.

[0073] Figure 7 is a schematic diagram of implementing Figure 1 step S13 in

[0074] Figure 8 is a flowchart of another camera registration method shown according to an exemplary embodiment.

[0075] Figure 9 is a block diagram of a camera registration device shown according to an exemplary embodiment.

[0076] Figure 10 is a block diagram of a device for camera registration shown according to an exemplary embodiment. Detailed Description of the Invention

[0077] Here, exemplary embodiments will be described in detail, and examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0078] Moreover, it is worth noting that for the method embodiments provided in this disclosure, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that this disclosure is not limited by the described action sequences. For example, terms such as "S131" and "S132" are used to distinguish method steps and do not necessarily need to be understood as describing a specific execution order. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential to this disclosure.

[0079] The inventor found that by performing distortion elimination and row alignment on the left and right views for registration, limited by the number and distribution of feature points in the detected feature point set, the registration accuracy between the RGB camera and the depth camera is relatively low, thus resulting in relatively low application accuracy in related application scenarios.

[0080] Therefore, this disclosure provides a camera registration method, aiming to improve the registration accuracy between the RGB camera and the depth camera. Furthermore, in a scenario where both the RGB camera and the depth camera are configured, the accuracy of related applications can be improved. For example, in a gesture recognition scenario, the coincidence degree between the RGB image and the depth map at the edge or the best feature point of the target is higher, which can improve the gesture recognition accuracy, or the distance determination accuracy can be improved in a distance determination scenario.

[0081] Figure 1 is a flowchart of a camera registration method shown according to an exemplary embodiment, which is applied to an electronic device including at least one RGB camera and at least one depth camera. Among them, this method is used to determine the relative position relationship between the RGB camera and the depth camera. Exemplarily, if the electronic device is configured with two RGB cameras and one depth camera, this method can respectively determine the relative position relationships between the two RGB cameras and the depth camera. As Figure 1 shown, it includes the following steps.

[0082] In step S11, according to the homography matrix between the RGB camera and the depth camera, the RGB image collected by the RGB camera and the energy map collected by the depth camera are corrected to obtain an RGB corrected image and an energy corrected image.

[0083] Among them, an RGB image is collected by an RGB camera, and an energy map is collected by a depth camera. Depth values and confidence values for each pixel are obtained based on the energy map. Then, a depth map is obtained based on the depth values of each pixel, and a confidence map is obtained based on the confidence values of each pixel. The energy map is used to reflect the energy magnitude of each pixel collected by the depth camera. The depth map is used to reflect the depth value of each pixel collected by the depth camera. The confidence map is used to reflect the reliability of the depth value prediction of each pixel collected by the depth camera.

[0084] Among them, referring to Figure 2 As shown, the energy map can be converted into a grayscale map, and based on the mapping function corresponding to the depth camera, according to the grayscale values of each pixel in the grayscale map, the depth values of each pixel in the collected energy map are calculated, and then a depth map is obtained based on the depth values of each pixel.

[0085] It can be understood that, referring to Figure 3 As shown, the homography matrix is a matrix used to describe the position mapping relationship between pixel points in the world coordinate system and the pixel coordinate system, which is obtained by online calibration of the parameters between the RGB camera and the depth camera according to the internal parameters and external parameters of the cameras calibrated in the factory and combined with the actual application scenario.

[0086] Among them, the internal parameters of the camera include the focal lengths of the RGB camera and the depth camera, the axis tilt parameters, the principal point coordinates in the imaging plane, and the principal point coordinates relative to the principal point in the imaging plane; the external parameters of the camera include the rotation matrix describing the rotation transformation from the world coordinate system to the camera coordinate system, and the translation matrix describing the translation transformation from the world coordinate system to the camera coordinate system.

[0087] In specific implementation, by multiplying the homography matrix with the RGB image collected by the RGB camera, the RGB coordinates of the pixel points in the RGB image are corrected to obtain an RGB corrected image; similarly, by multiplying the homography matrix with the energy map collected by the depth camera, the energy coordinates of the pixel points in the energy map are corrected to obtain an energy corrected image.

[0088] In step S12, target feature point pairs are determined from the matching feature point pairs on the RGB corrected image and the energy corrected image.

[0089] Among them, the feature point pairs include the first feature points on the RGB corrected image and the second feature points on the energy corrected image, and the difference between the first parallax of the first feature points in the target feature point pairs and the second parallax of the second feature points in the target feature point pairs is greater than a preset threshold.

[0090] Among them, the difference between the first disparity of the first feature point in the target feature point pair and the second disparity of the second feature point in the target feature point pair is greater than a preset threshold, including: the difference between the first disparity minus the second disparity is greater than the preset threshold, or the difference between the second disparity minus the first disparity is greater than the preset threshold. That is, the absolute value of the difference between the first disparity and the second disparity is greater than the preset threshold.

[0091] During implementation, feature point detection is performed on the RGB corrected image to obtain the first feature point, and feature point detection is performed on the energy corrected image to obtain the second feature point. The first feature point in the RGB corrected image is matched with the second feature point in the energy corrected image to obtain a matching feature point pair. Among them, the matching feature point pair is used to represent the same pixel point in the world coordinate system, the RGB coordinates projected in the RGB corrected image, and the energy coordinates projected in the energy corrected image.

[0092] On this basis, Figure 4 is a flowchart showing a method for determining the first disparity of the first feature point according to an exemplary embodiment. Refer to Figure 4 as shown, and includes the following steps:

[0093] In step S41, determine the RGB coordinates of the first feature point in the RGB corrected image and the energy coordinates in the energy corrected image.

[0094] Determine the RGB coordinates of the first feature point after feature matching in the RGB corrected image and the energy coordinates in the energy corrected image.

[0095] In step S42, take the difference between the RGB coordinates and the energy coordinates as the first disparity.

[0096] During implementation, the first disparity D1 is equal to the difference between the RGB coordinates and the energy coordinates.

[0097] On the basis of the above embodiment, Figure 5 is a flowchart showing a method for determining the second disparity of the second feature point according to an exemplary embodiment. Refer to Figure 5 as shown, and includes the following steps:

[0098] In step S51, determine the first imaging coordinates of the second feature point in the depth camera and the second imaging coordinates in the RGB camera.

[0099] In this step, determining the first imaging coordinates of the second feature point in the depth camera includes:

[0100] Obtain the depth value of the second feature point, and determine the first distance from the second feature point in the world coordinate system to the optical axis of the depth camera. Among them, obtain the depth value of the second feature point from the depth map.

[0101] Determine the first imaging coordinates of the second feature point in the depth camera based on the focal length, depth value, and first distance of the depth camera.

[0102] Specifically, refer to Figure 6 As shown, in this embodiment, the right-eye camera is the depth camera, and the left-eye camera is the RGB camera. Calculate the first imaging coordinates of the second feature point in the depth camera through the following formula:

[0103]

[0104] where f r is the focal length of the depth camera, Z is the depth value of the second feature point p, X is the first distance from the second feature point p to the optical axis of the depth camera in the world coordinate system, and x r is the first imaging coordinates of the imaging point p of the second feature point in the depth camera r .

[0105] In this step, determining the second imaging coordinates of the second feature point in the RGB camera includes:

[0106] Determine the second distance from the second feature point to the optical axis of the RGB camera in the world coordinate system according to the center distance between the RGB camera and the depth camera.

[0107] Refer to Figure 6 As shown, the second distance is equal to the first distance minus the center distance, where the center distance is the baseline distance. Taking the second feature point p as a feature point outside the depth camera and the RGB camera as an example for calculation, the second distance is equal to the first distance minus the center distance.

[0108] Based on the above embodiment, if the second feature point is a feature point between the depth camera and the RGB camera, the second distance is the center distance minus the first distance.

[0109] Determine the second imaging coordinates of the second feature point in the RGB camera according to the focal length, depth value, and second distance of the RGB camera.

[0110] Specifically, refer to Figure 6 As shown, calculate the second imaging coordinates of the second feature point in the RGB camera through the following formula:

[0111]

[0112] where f l is the focal length of the RGB camera. In this embodiment, the focal lengths of the depth camera and the RGB camera are equal, b is the center distance, and x l is the imaging point p of the second feature point in the RGB cameral The second imaging coordinates.

[0113] In step S52, the difference between the first imaging coordinates and the second imaging coordinates is taken as the second parallax.

[0114] During implementation, the second parallax D2 is the difference between the first imaging coordinate and the second imaging coordinate.

[0115] Specifically, the second disparity D2 is calculated by the following formula:

[0116]

[0117] Then D2=丨x r -x l When the focal lengths of the depth camera and the RGB camera are equal, D2 is equal to the ratio of the center distance to the depth value.

[0118] On the basis of the above embodiment, in step S12, determining the target feature point pair from the feature point pairs matching on the RGB correction image and the energy correction image includes:

[0119] For any feature point pair, when the difference between the first disparity of the first feature point in the feature point pair and the second disparity of the second feature point in the feature point pair is greater than a preset threshold, the feature point pair is taken as a target feature point pair.

[0120] Specifically, any one feature point pair is selected, and the difference between the first disparity of the first feature point in the feature point pair and the second disparity of the second feature point in the feature point pair is calculated. When the difference is less than or equal to a preset threshold, a new feature point pair is reselected from the unselected feature point pairs until the difference corresponding to the feature point pair is greater than the preset threshold. The feature point pair is taken as the target feature point pair, and the traversal calculation process ends.

[0121] In one embodiment, candidate feature point pairs whose confidence is greater than a preset confidence threshold are determined from each feature point pair, and for any candidate feature point pair, if the difference between the first disparity of the first feature point and the second disparity of the second feature point in the candidate feature point pair is greater than a preset threshold, the candidate feature point pair is used as a target feature point pair.

[0122] In one embodiment, the preset confidence threshold corresponds to the application environment type one by one, for example, when the application environment type is an indoor environment, it corresponds to the first preset confidence threshold, and when the application environment type is an outdoor environment, it corresponds to the second preset confidence threshold. Through the relationship between the confidence and the preset confidence threshold, candidate feature points can be screened, thereby improving the accuracy of feature point pairs, and thereby improving the accuracy of registration.

[0123] Specifically, for any pair of feature points, the confidence level of the second feature point in the pair of feature points is determined according to the confidence map. When the confidence level of the feature point is less than or equal to a preset confidence threshold, the pair of feature points is determined as a non-candidate pair of feature points. When the confidence level of the feature point is greater than the preset confidence threshold, the pair of feature points is determined as a candidate pair of feature points. The method for determining the target pair of feature points from the candidate pairs of feature points is the same as the method for determining the target pair of feature points from the pairs of feature points described above, and will not be elaborated here.

[0124] In step S13, based on this difference, the homography matrix is adjusted according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio.

[0125] In a possible implementation manner, the preset threshold may be 0, that is, no adjustment is made when the first parallax is equal to the second parallax, and when the first parallax is not equal to the second parallax, the homography matrix needs to be adjusted.

[0126] Based on the above embodiments, Figure 7 is a flowchart of step S13 shown according to an exemplary embodiment. Refer to Figure 1 as shown. In step S13, the adjusting the homography matrix based on this difference according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio includes: Figure 7 In step S131, according to the baseline distance and the preset adjustment ratio, the magnitude of the parameter adjustment amount is determined.

[0127] Specifically, the product of the baseline distance and the preset adjustment ratio is used as the parameter adjustment amount. For example, when the baseline distance is 100 mm and the preset adjustment ratio is one percent, the magnitude of the parameter adjustment amount is determined to be 1 mm.

[0128] In step S132, when the difference between the first parallax and the second parallax is greater than the preset threshold and the first parallax is greater than the second parallax, the homography matrix is increased by the parameter adjustment amount.

[0129] In step S133, when the difference between the first parallax and the second parallax is less than the preset threshold and the first parallax is less than the second parallax, the homography matrix is decreased by the parameter adjustment amount.

[0130] When the difference between the first parallax and the second parallax is greater than the preset threshold and the first parallax is greater than the second parallax, it indicates that the collected depth value is smaller than the real value. Therefore, it is necessary to increase the external parameters between the RGB camera and the depth camera. Thus, the homography matrix is increased according to the magnitude of the parameter adjustment amount. Continuing with the above embodiment for illustration, the homography matrix is increased by 1 mm.

[0131] In step S133, when the difference between the first parallax and the second parallax is less than the preset threshold and the first parallax is less than the second parallax, the homography matrix is decreased by the parameter adjustment amount.

[0132] Similarly, when the difference between the first parallax and the second parallax is less than the preset threshold and the first parallax is less than the second parallax, it indicates that the collected depth value is larger than the true value. Therefore, it is necessary to reduce the extrinsic parameters between the RGB camera and the depth camera. Thus, the homography matrix is reduced according to the amount of parameter adjustment. Continuing with the above embodiment for illustration, the homography matrix is reduced by 1 mm.

[0133] Based on the above embodiment, in step S13, adjusting the homography matrix according to the difference, the baseline distance between the RGB camera and the depth camera, and the preset adjustment ratio includes:

[0134] Determine the target parameters in the homography matrix that represent the translational and rotational relationships between the RGB camera and the depth camera.

[0135] Specifically, determine that the translation matrix in the homography matrix is the target parameter and / or determine that the rotation matrix in the homography matrix is the target parameter.

[0136] Based on the difference, adjust the target parameters in the homography matrix according to the baseline distance between the RGB camera and the depth camera and the preset adjustment ratio.

[0137] Specifically, when the difference between the first parallax and the second parallax is greater than the preset threshold and the first parallax is greater than the second parallax, it indicates that the collected depth value is smaller than the true value. Therefore, it is necessary to increase the extrinsic parameters between the RGB camera and the depth camera. Thus, the target parameters in the homography matrix are increased according to the amount of parameter adjustment.

[0138] When the difference between the first parallax and the second parallax is less than the preset threshold and the first parallax is less than the second parallax, it indicates that the collected depth value is larger than the true value. Therefore, it is necessary to reduce the extrinsic parameters between the RGB camera and the depth camera. Thus, the target parameters in the homography matrix are reduced according to the amount of parameter adjustment.

[0139] In step S14, re - correct the RGB image and the energy map according to the adjusted homography matrix.

[0140] Specifically, multiply the adjusted homography matrix by the RGB image to obtain a new RGB image, and multiply the adjusted homography matrix by the energy map to obtain a new energy map.

[0141] In the above technical solution, the RGB image and the energy map are corrected through the homography matrix, and the target feature point pair is determined from the matching feature point pairs on the obtained RGB corrected image and energy corrected image. Furthermore, when the difference between the first disparity of the first feature point in the target feature point pair and the second disparity of the second feature point in the target feature point pair is greater than a preset threshold, based on this difference, according to the baseline distance between the RGB camera and the depth camera and the preset adjustment ratio, the homography matrix is adjusted to obtain a more accurate relative position relationship between the RGB camera and the depth camera, which can improve the registration accuracy between the RGB camera and the depth camera. Furthermore, in a scenario where the RGB camera and the depth camera are both configured, the accuracy of related applications can be improved.

[0142] Based on the above embodiment, the method includes:

[0143] After re-correcting the RGB image and the energy map according to the adjusted homography matrix, return to execute the step of determining the target feature point pair from the matching feature point pairs on the RGB corrected image and the energy corrected image until there are no target feature point pairs among the feature point pairs on the new RGB corrected image and the new energy corrected image.

[0144] Specifically, according to the confidence map of the depth camera, determine the confidence of the new second feature point. When the confidence of the new second feature point is less than or equal to the preset confidence threshold, determine the corresponding feature point pair as a non-candidate target feature point pair. If the confidence of the second feature point of all new feature point pairs is less than or equal to the preset confidence threshold, it is determined that there are no target feature point pairs among the feature point pairs on the new RGB corrected image and the new energy corrected image.

[0145] When the confidence of the new second feature point is greater than the preset confidence threshold, determine the corresponding feature point pair as a candidate target feature point pair. For the candidate target feature point pair, determine the new RGB coordinates of the first feature point of the candidate target feature point pair in the new RGB corrected image and the new energy coordinates in the new energy corrected image, and then calculate the difference between the new RGB coordinates and the new energy coordinates as the first disparity.

[0146] Furthermore, for the candidate target feature point pair, calculate the second disparity of the new second feature point in the same way as in the above embodiment. And calculate the difference between the first disparity and the second disparity. When the difference corresponding to any candidate target feature point pair is less than the preset threshold, it is determined that there are no target feature point pairs among the feature point pairs on the new RGB corrected image and the new energy corrected image.

[0147] The technical solution of the present disclosure will be described below through a specific embodiment. See Figure 8As shown, the method includes the following steps:

[0148] Before the device leaves the factory, calibrate the internal parameters of the RGB camera and the depth camera through factory calibration, and then complete the binocular calibration result based on the internal parameters of the RGB camera and the depth camera, that is, the external parameters between the RGB camera and the depth camera, to obtain the homography matrix.

[0149] Collect the RGB image through the RGB camera and collect the corresponding energy map, depth map and confidence map through the depth camera. For the RGB image and the energy map, perform stereo correction of pixel points through the homography matrix to obtain the RGB corrected map and the energy corrected map.

[0150] Perform feature point detection on the RGB corrected map and the energy corrected map, respectively detect the first feature point and the second feature point from the pixel points, and perform feature matching on the first feature point and the second feature point to obtain the feature point pair.

[0151] For the second feature point in the feature point pair, detect the second feature point according to the depth map and the confidence map, and determine again whether the feature point pair is matched. When the feature point pair is matched, determine the confidence of the second feature point according to the confidence map, and judge whether the confidence of the second feature point is greater than the preset confidence threshold. When the confidence of the second feature point is less than or equal to the preset confidence threshold, reselect a new feature point pair until the confidence of the second feature point of all feature point pairs is less than or equal to the preset confidence threshold, that is, when there is no target feature point pair, end the camera registration.

[0152] When the confidence of the second feature point is greater than the preset confidence threshold, determine that the feature point pair is a candidate feature point pair. And calculate the second parallax according to the basic information of the depth camera and the RGB camera, where the basic information is the internal parameters of the depth camera and the RGB camera.

[0153] At the same time, for the first feature point in the feature point pair, determine the RGB coordinates according to the RGB corrected map and determine the energy coordinates according to the energy corrected map, and calculate the first parallax according to the RGB coordinates and the energy coordinates. Then, when the first parallax is equal to the second parallax, reselect the feature point pair until the first parallax and the second parallax of all feature point pairs are equal, and determine that the camera does not need to be registered. When the first parallax and the second parallax are not equal, determine the size of the parameter adjustment amount for adjusting the homography matrix according to the center distance and the preset adjustment ratio, and based on the size relationship between the first parallax and the second parallax, adjust the homography matrix according to the parameter adjustment amount, and perform stereo correction on the RGB image and the energy map again according to the adjusted homography matrix until there is no target feature point pair to end the camera registration.

[0154] Based on the same inventive concept, the present disclosure also provides a camera registration device, which is applied to an electronic device including at least one RGB camera and at least one depth camera. The device can implement all or part of the steps of the camera registration method in a software, hardware, or a combination of both manners. Figure 9 FIG. is a block diagram of a camera registration device 100 shown according to an exemplary embodiment, as Figure 9 shown, the device 100 includes: a first correction module 110, a determination module 120, an adjustment module 130, and a second correction module 140.

[0155] Among them, the first correction module 110 is configured to correct the RGB image collected by the RGB camera and the energy map collected by the depth camera according to the homography matrix between the RGB camera and the depth camera, so as to obtain an RGB corrected image and an energy corrected image;

[0156] The determination module 120 is configured to determine a target feature point pair from the matching feature point pairs on the RGB corrected image and the energy corrected image. The feature point pair includes a first feature point on the RGB corrected image and a second feature point on the energy corrected image. The difference between the first parallax of the first feature point in the target feature point pair and the second parallax of the second feature point in the target feature point pair is greater than a preset threshold;

[0157] The adjustment module 130 is configured to adjust the homography matrix based on the difference, according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio;

[0158] The second correction module 140 is configured to re-correct the RGB image and the energy map according to the adjusted homography matrix.

[0159] The above device can improve the registration accuracy between the RGB camera and the depth camera, and further improve the accuracy of related applications in the scenario where the RGB camera and the depth camera are simultaneously configured.

[0160] Optionally, the determination module 120 is configured to: for any one of the feature point pairs, when the difference between the first parallax of the first feature point in the feature point pair and the second parallax of the second feature point is greater than the preset threshold, use the feature point pair as the target feature point pair; or,

[0161] Determine candidate feature point pairs with a confidence greater than a preset confidence threshold from each of the said feature point pairs, and for any one of the said candidate feature point pairs, when the difference between the first disparity of the first feature point and the second disparity of the second feature point in the said candidate feature point pair is greater than the preset threshold, take the said candidate feature point pair as the target feature point pair.

[0162] Optionally, the adjustment module 130 is configured to determine the magnitude of the parameter adjustment amount according to the baseline distance and the preset adjustment ratio;

[0163] When the difference between the first disparity and the second disparity is greater than the preset threshold and the first disparity is greater than the second disparity, increase the homography matrix by the parameter adjustment amount; or,

[0164] When the difference between the first disparity and the second disparity is less than the preset threshold and the first disparity is less than the second disparity, decrease the homography matrix by the parameter adjustment amount.

[0165] Optionally, the adjustment module 130 is configured to determine the target parameters in the homography matrix that characterize the translational and rotational relationships between the RGB camera and the depth camera;

[0166] Based on the difference, adjust the target parameters in the homography matrix according to the baseline distance between the RGB camera and the depth camera and the preset adjustment ratio.

[0167] Optionally, the determination module 120 is further configured to, after re - correcting the RGB image and the energy map according to the adjusted homography matrix, return to execute the step of determining the target feature point pair from the feature point pairs that match on the RGB corrected map and the energy corrected map until there is no such target feature point pair among the feature point pairs on the new RGB corrected map and the new energy corrected map.

[0168] Optionally, the determination module 120 is configured to determine the RGB coordinates of the first feature point in the RGB corrected map and the energy coordinates in the energy corrected map;

[0169] Take the difference between the RGB coordinates and the energy coordinates as the first disparity.

[0170] Optionally, the determination module 120 is configured to determine the first imaging coordinates of the second feature point in the depth camera and the second imaging coordinates in the RGB camera;

[0171] Take the difference between the first imaging coordinate and the second imaging coordinate as the second parallax.

[0172] Optionally, the determining module 120 is configured to obtain the depth value of the second feature point and determine a first distance from the second feature point to the optical axis of the depth camera in the world coordinate system;

[0173] Determine a first imaging coordinate of the second feature point in the depth camera according to the focal length of the depth camera, the depth value, and the first distance.

[0174] Optionally, the determining module 120 is configured to determine a second distance from the second feature point to the optical axis of the RGB camera in the world coordinate system according to the center distance between the RGB camera and the depth camera;

[0175] Determine a second imaging coordinate of the second feature point in the RGB camera according to the focal length of the RGB camera, the depth value, and the second distance.

[0176] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0177] In addition, it is worth noting that the modules in the above embodiments may be independent devices or the same device in specific implementations. For example, the first correction module 110 and the second correction module 140 may be the same module or two modules. The present disclosure does not limit this.

[0178] The present disclosure also provides an electronic device, including:

[0179] A processor;

[0180] A memory for storing instructions executable by the processor;

[0181] Wherein, the processor is configured to:

[0182] According to the homography matrix between the RGB camera and the depth camera, correct the RGB image collected by the RGB camera and the energy map collected by the depth camera to obtain an RGB corrected image and an energy corrected image;

[0183] Determine target feature point pairs from the feature point pairs that match on the RGB correction map and the energy correction map, where the feature point pairs include a first feature point on the RGB correction map and a second feature point on the energy correction map, and the difference between the first parallax of the first feature point in the target feature point pair and the second parallax of the second feature point in the target feature point pair is greater than a preset threshold;

[0184] Based on the difference, adjust the homography matrix according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio;

[0185] Perform secondary correction on the RGB image and the energy map according to the adjusted homography matrix.

[0186] The present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored, and when the program instructions are executed by a processor, the steps of the camera registration method provided by the present disclosure are implemented.

[0187] Figure 10 It is a block diagram of a device 800 for camera registration shown according to an exemplary embodiment. For example, the device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0188] Refer to Figure 10 , the device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0189] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, telephone calls, data communication, camera photographing operations, and camera registration operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above camera registration method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0190] The memory 804 is configured to store various types of data to support the operation of the device 800. Examples of such data include instructions for any application or method operating on the device 800. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0191] The power component 806 provides power to the various components of the device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0192] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundaries of the touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. Among them, the front camera and / or the rear camera are configured with at least one RGB camera and at least one depth camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0193] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when the device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 includes a speaker for outputting audio signals.

[0194] The I / O interface 812 provides an interface between the processing component 802 and a peripheral interface module, and the peripheral interface module can be a keyboard, a click wheel, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

[0195] The sensor assembly 814 includes one or more sensors for providing an assessment of the status of the device 800 in various aspects. For example, the sensor assembly 814 can detect the on / off state of the device 800, the relative positioning of components, such as the display and keypad of the device 800. The sensor assembly 814 can also detect a change in the position of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0196] The communication component 816 is configured to facilitate communication between the device 800 and other devices in a wired or wireless manner. The device 800 can access a wireless network based on communication standards, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0197] In an exemplary embodiment, the device 800 can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above-described camera registration method.

[0198] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the above instructions can be executed by a processor 820 of the device 800 to complete the above-described camera registration method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0199] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for performing the above-described camera registration method when executed by the programmable device.

[0200] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and examples are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0201] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A camera registration method, characterized in that, Applied to an electronic device including at least one RGB camera and at least one depth camera, the method includes: According to the homography matrix between the RGB camera and the depth camera, correct the RGB image collected by the RGB camera and the energy map collected by the depth camera to obtain an RGB corrected image and an energy corrected image; Determine target feature point pairs from the matching feature point pairs on the RGB corrected image and the energy corrected image, where the feature point pairs include a first feature point on the RGB corrected image and a second feature point on the energy corrected image, and the difference between the first parallax of the first feature point in the target feature point pair and the second parallax of the second feature point in the target feature point pair is greater than a preset threshold; Based on the difference, adjust the homography matrix according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio; Perform re-correction on the RGB image and the energy map according to the adjusted homography matrix.

2. The method according to claim 1, wherein The determining target feature point pairs from the matching feature point pairs on the RGB corrected image and the energy corrected image includes: For any of the feature point pairs, when the difference between the first parallax of the first feature point in the feature point pair and the second parallax of the second feature point is greater than the preset threshold, use the feature point pair as the target feature point pair; or, Determine candidate feature point pairs with a confidence greater than a preset confidence threshold from each of the feature point pairs, and for any of the candidate feature point pairs, when the difference between the first parallax of the first feature point in the candidate feature point pair and the second parallax of the second feature point is greater than the preset threshold, use the candidate feature point pair as the target feature point pair.

3. The method according to claim 2, wherein The adjusting the homography matrix based on the difference, according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio, includes: Determine the magnitude of the parameter adjustment amount according to the baseline distance and the preset adjustment ratio; When the difference between the first parallax and the second parallax is greater than the preset threshold and the first parallax is greater than the second parallax, increase the homography matrix by the parameter adjustment amount; or, When the difference between the first parallax and the second parallax is less than the preset threshold and the first parallax is less than the second parallax, decrease the homography matrix by the parameter adjustment amount.

4. The method according to claim 1, characterized in that, The adjusting the homography matrix based on the difference, according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio, includes: Determine the target parameters in the homography matrix that represent the translation and rotation relationship between the RGB camera and the depth camera; Based on the difference, adjust the target parameters in the homography matrix according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio.

5. The method according to claim 2, characterized in that, The method includes: After re - correcting the RGB image and the energy map according to the adjusted homography matrix, return to execute the step of determining the target feature point pair from the matching feature point pairs on the RGB corrected map and the energy corrected map until there is no such target feature point pair among the feature point pairs on the new RGB corrected map and the new energy corrected map.

6. The method according to any one of claims 2-5, characterized in that The determination of the first disparity of the first feature point includes: Determine the RGB coordinates of the first feature point in the RGB corrected map and the energy coordinates in the energy corrected map; Take the difference between the RGB coordinates and the energy coordinates as the first disparity.

7. The method according to any one of claims 2-5, characterized in that, The determination of the second disparity of the second feature point includes: Determine the first imaging coordinates of the second feature point in the depth camera and the second imaging coordinates in the RGB camera; Take the difference between the first imaging coordinates and the second imaging coordinates as the second disparity.

8. The method according to claim 7, characterized in that The determination of the first imaging coordinates of the second feature point in the depth camera includes: Obtain the depth value of the second feature point and determine the first distance from the second feature point to the optical axis of the depth camera in the world coordinate system; According to the focal length of the depth camera, the depth value, and the first distance, determine the first imaging coordinates of the second feature point in the depth camera.

9. The method according to claim 8, wherein The determination of the second imaging coordinates of the second feature point in the RGB camera includes: According to the center distance between the RGB camera and the depth camera, determine the second distance from the second feature point to the optical axis of the RGB camera in the world coordinate system; According to the focal length of the RGB camera, the depth value, and the second distance, determine the second imaging coordinates of the second feature point in the RGB camera.

10. A camera registration device, characterized in that, Applied to an electronic device including at least one RGB camera and at least one depth camera, the device includes: A first correction module configured to correct the RGB image collected by the RGB camera and the energy map collected by the depth camera according to the homography matrix between the RGB camera and the depth camera to obtain an RGB corrected map and an energy corrected map; A determination module configured to determine a target feature point pair from the matching feature point pairs on the RGB corrected map and the energy corrected map, the feature point pair including a first feature point on the RGB corrected map and a second feature point on the energy corrected map, and the difference between the first disparity of the first feature point in the target feature point pair and the second disparity of the second feature point in the target feature point pair is greater than a preset threshold; An adjustment module configured to adjust the homography matrix based on the difference, according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio; A second correction module configured to re - correct the RGB image and the energy map according to the adjusted homography matrix.

11. An electronic device, characterized in that, Includes: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to: According to the homography matrix between the RGB camera and the depth camera, correct the RGB image collected by the RGB camera and the energy map collected by the depth camera to obtain an RGB corrected image and an energy corrected image; Determine a target feature point pair from the matching feature point pairs on the RGB corrected image and the energy corrected image, the feature point pair includes a first feature point on the RGB corrected image and a second feature point on the energy corrected image, and the difference between the first disparity of the first feature point in the target feature point pair and the second disparity of the second feature point in the target feature point pair is greater than a preset threshold; Based on the difference, adjust the homography matrix according to the baseline distance between the RGB camera and the depth camera and a preset adjustment ratio; Recorrect the RGB image and the energy map according to the adjusted homography matrix.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the program instruction is executed by the processor, it implements the steps of the method described in any one of claims 1-9.

Images