Depth information calculation method, electronic device, and storage medium

Abstract

A depth information calculation method, an electronic device and a storage medium, the depth information calculation method comprising: obtaining depth information of at least one key point determined based on a depth information algorithm; determining that the depth information corresponding to at least one of the at least one key point satisfies a first preset condition, and taking the key point corresponding to the depth information satisfying the first preset condition as a to-be-processed key point, the first preset condition comprising: the depth value is a negative number, or the depth value is a non-negative number and the sum of the re-projection errors is greater than or equal to a first threshold; processing the to-be-processed key point to correct the to-be-processed key point and the corresponding depth information, and updating the depth information. The depth information calculation method can improve the accuracy of the depth information, make the depth information effective and accurate, make the depth information of the key point both meet the expectation and have a small deviation from the true value, and can improve the overall calculation efficiency and the effect of the subsequent processing link.

Description

Technical Field

[0001] Embodiments of this disclosure relate to a depth information calculation method, an electronic device, and a storage medium. Background Technology

[0002] Augmented Reality (AR) technology integrates virtual information with the real world, involving various technologies such as multimedia, 3D modeling, real-time tracking, intelligent interaction, and sensing. AR technology simulates virtual information (such as computer-generated text, images, 3D models, music, and videos) and applies it to the real world, making the virtual and real-world information complementary, thus "enhancing" the real world. AR technology requires the 3D reconstruction of the surrounding environment, which necessitates the use of Simultaneous Localization and Mapping (SLAM) algorithms. SLAM algorithms can achieve localization during movement and build maps based on this localization, thereby enabling complex localization and mapping functions. Summary of the Invention

[0003] At least one embodiment of this disclosure provides a depth information calculation method, comprising: acquiring depth information of at least one key point determined based on a depth information algorithm; determining that the depth information corresponding to at least one of the at least one key point satisfies a first preset condition, and taking the key point corresponding to the depth information satisfying the first preset condition as a key point to be processed, wherein the first preset condition includes: the depth value is negative, or the depth value is non-negative and the sum of reprojection errors is greater than or equal to a first threshold; processing the key point to be processed to correct the key point to be processed and the corresponding depth information, and updating the depth information.

[0004] For example, in a method provided in one embodiment of this disclosure, the first preset condition includes that the depth value is non-negative and the sum of the reprojection errors is greater than or equal to the first threshold. The method processes the key point to be processed to correct the key point and its corresponding depth information, and updates the depth information. This includes: calculating the Hamming distance between the descriptors of the key point to be processed in two frames of images; confirming that the Hamming distance satisfies a second preset condition, wherein the second preset condition includes that the Hamming distance is less than a second threshold; and recalculating the depth information of the key point to be processed using an optimization algorithm to update the depth information.

[0005] For example, in a method provided in one embodiment of this disclosure, the sum of the reprojection errors is the sum of the reprojection errors of the key point to be processed in each of the two frames of images.

[0006] For example, in a method provided in one embodiment of this disclosure, the key points to be processed are processed to correct the key points and their corresponding depth information, and the depth information is updated. The method further includes: confirming that the Hamming distance does not meet the second preset condition; updating the association relationship of the key points to be processed in the two frames of images, and using the depth information algorithm to determine the depth information of the updated key points, and updating the depth information.

[0007] For example, in a method provided in one embodiment of this disclosure, updating the association relationship of the key point to be processed in the two frames of images, and using the depth information algorithm to determine the depth information of the updated key point, updating the depth information includes: traversing within a preset range of the projection points of the key point to be processed in the later frame of the two frames of images and determining a new projection point, replacing the projection point of the key point to be processed with the new projection point; using the depth information algorithm to calculate the corresponding depth information based on the updated association relationship, and updating the depth information.

[0008] For example, in a method provided in one embodiment of this disclosure, traversing within a preset range of the projection points of the key point to be processed in the later frame of the two-frame images and determining the new projection point, and replacing the projection point of the key point to be processed with the new projection point, includes: determining pixels within a preset range of the location of the projection point of the key point to be processed in the later frame of the two-frame images; selecting at least one pixel as a candidate point based on a preset rule; calculating a binary-coded feature descriptor for each candidate point; selecting, from the calculated feature descriptors, a candidate point corresponding to a feature descriptor whose Hamming distance to the feature descriptor of the projection point in the earlier frame of the two-frame images satisfies the second preset condition; and using the selected candidate point as the new projection point to replace the projection point of the key point to be processed.

[0009] For example, in one embodiment of the method provided in this disclosure, the preset range includes a 5*5 pixel range.

[0010] For example, in a method provided in one embodiment of this disclosure, the preset rule includes at least one of the following rules: all pixels within the preset range except for the projection point of the key point to be processed are used as candidate points; among the pixels within the preset range except for the projection point of the key point to be processed, a portion of the pixels are selected according to a preset step size; and among the pixels within the preset range except for the projection point of the key point to be processed, a preset number of pixels are randomly selected.

[0011] For example, in a method provided in one embodiment of this disclosure, the first preset condition includes the depth value being negative, processing the key point to be processed to correct the key point to be processed and the corresponding depth information, and updating the depth information, including: discarding the key point to be processed with the depth value being negative.

[0012] For example, in a method provided in one embodiment of this disclosure, the depth information algorithm includes a triangulation algorithm. The depth information is obtained through the three-dimensional coordinate information of the key points in the world coordinate system. The three-dimensional coordinate information of the key points in the world coordinate system is obtained through the two-dimensional coordinate information of the projection points in each frame of the image and the real-time pose of the camera. The triangulation algorithm includes: calculating the three-dimensional coordinate information of the projection points in the camera coordinate system in the two frames of the image used in the triangulation algorithm, wherein the three-dimensional coordinate information in the camera coordinate system is normalized coordinate information; and calculating the three-dimensional coordinate information of the key points in the world coordinate system based on the poses of the key points in the two frames of the image respectively.

[0013] For example, in one embodiment of this disclosure, the method is used for an augmented reality device.

[0014] At least one embodiment of this disclosure also provides an electronic device, including: a processor; a memory including one or more computer program modules; wherein the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules are used to implement the depth information calculation method provided in any embodiment of this disclosure.

[0015] At least one embodiment of this disclosure also provides a non-transitory computer-readable storage medium storing non-transitory computer-readable instructions, which, when executed by a computer, implement the depth information calculation method provided in any embodiment of this disclosure. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0017] Figure 1 This is a schematic diagram illustrating an application scenario of a depth information calculation method provided in some embodiments of this disclosure;

[0018] Figure 2 A flowchart illustrating a depth information calculation method provided for at least one embodiment of this disclosure;

[0019] Figure 3An exemplary flowchart of a triangulation algorithm provided in this disclosure embodiment;

[0020] Figure 4 for Figure 2 An exemplary flowchart of step S30;

[0021] Figure 5 for Figure 4 An exemplary flowchart of step S35;

[0022] Figure 6 for Figure 5 An exemplary flowchart of step S351;

[0023] Figure 7 A schematic diagram illustrating the workflow of a depth information calculation method provided in some embodiments of this disclosure;

[0024] Figure 8A This is a schematic diagram of a three-dimensional point cloud map;

[0025] Figure 8B This is a schematic diagram of a three-dimensional point cloud map generated using the depth information obtained by the depth information calculation method provided in the embodiments of this disclosure;

[0026] Figure 9 A schematic block diagram of an electronic device provided for some embodiments of this disclosure;

[0027] Figure 10 A schematic block diagram of another electronic device provided for some embodiments of this disclosure;

[0028] Figure 11 This is a schematic diagram of a storage medium provided for some embodiments of this disclosure. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0030] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “including,” “comprising,” or “containing,” and similar terms mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. The terms “connected,” “linked,” or similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “upper,” “lower,” “left,” and “right,” etc., are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0031] In AR technology, AR glasses are a common application device. AR glasses are head-mounted devices that combine real-world scenes with virtual ones. Their main functions include real-time estimation of the system's position and angle, perception of the surrounding environment, and 3D reconstruction. Due to the limited computing resources of AR glasses, deep learning methods requiring significant computational power cannot be used to reconstruct the environment. Typically, AR glasses employ SLAM (Single-Loop AM) technology for reconstruction. In monocular camera SLAM reconstruction schemes, estimating the depth information of the environment is crucial. Depth information can affect the robustness and accuracy of the localization, mapping, and loop closure optimization modules within SLAM.

[0032] Currently, depth estimation can be categorized into two methods based on camera type: one for monocular cameras and one for stereo cameras. For stereo cameras, the depth of keypoints can be calculated based on the disparity information of two images. For monocular cameras, the pose of two or more frames and associated keypoint information are used, along with a triangulation algorithm to calculate depth information. However, the depth values ​​calculated by the triangulation algorithm can be abnormal; for example, the depth value may be negative or have a very large error. This can negatively impact the accuracy of the SLAM algorithm's calculations.

[0033] At least one embodiment of this disclosure provides a depth information calculation method, an electronic device, and a storage medium. This depth information calculation method can improve the accuracy of depth information, making it effective and precise. It ensures that the depth information of key points both meets expectations and has minimal deviation from the actual value, thereby improving overall calculation efficiency and the effectiveness of subsequent processing steps.

[0034] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the same reference numerals will be used to refer to the same elements described in different drawings.

[0035] At least one embodiment of this disclosure provides a depth information calculation method. The depth information calculation method includes: acquiring depth information of at least one key point determined based on a depth information algorithm; determining that the depth information corresponding to at least one of the at least one key point satisfies a first preset condition, and using the key point corresponding to the depth information satisfying the first preset condition as a key point to be processed, the first preset condition including: a negative depth value, or a non-negative depth value and a sum of reprojection errors greater than or equal to a first threshold; processing the key point to be processed to correct the key point to be processed and its corresponding depth information, and updating the depth information.

[0036] The algorithms for AR devices (such as AR glasses) mainly include modules for camera pose estimation, planar estimation, and illumination sensing. Camera pose estimation is implemented using the SLAM algorithm. When performing pose estimation, the SLAM algorithm primarily performs two functions: localization and mapping. These two functions can be implemented through corresponding functional modules, such as a localization module and a mapping module. Taking monocular SLAM as an example, the localization module estimates the system's position and pose in real time. The pose estimation process requires 3D points (i.e., spatial points with 3D coordinates) generated by the mapping module as input. The mapping module estimates the depth information of the key points based on the image pose estimated by the localization module and the pixel plane keypoint information, thereby obtaining the 3D coordinates of the key points. The localization module and the mapping module in the SLAM algorithm are interconnected and mutually influential; the coordinates of the 3D points (the depth information of the key points) play a crucial role.

[0037] Figure 1 This diagram illustrates an application scenario of a depth information calculation method provided in some embodiments of this disclosure. Figure 1 As shown, in the SLAM operation process, the depth of key points is first estimated using a triangulation algorithm. After obtaining the depth information, the depth information calculation method provided in this embodiment needs to be used to process any abnormal depth information, making the processed depth information more accurate. Then, the obtained depth information can be used in a loop closure optimization process to correct accumulated errors. The obtained depth information can also be used in a Visual Inertial Odometry (VIO) process to obtain the optimal solution for pose coordinates by calculating and maximizing the posterior probability of the object's position and attitude coordinates under visual and inertial measurement data, thereby tracking the target position in real time. Figure 1In the illustrated computational flow, the depth information obtained using the depth information calculation method provided in this embodiment is processed and features high precision and accuracy, thereby improving the processing performance of the loop closure optimization process and the VIO process. For detailed descriptions of the triangulation estimation keypoint depth process, the loop closure optimization process, and the VIO process, please refer to conventional designs; they will not be repeated here.

[0038] Figure 2 This is a flowchart illustrating a depth information calculation method provided for at least one embodiment of this disclosure. Figure 2 As shown, in some embodiments, the depth information calculation method includes the following operations.

[0039] Step S10: Obtain the depth information of at least one key point determined by the depth information algorithm;

[0040] Step S20: Determine that the depth information corresponding to at least one of the key points satisfies the first preset condition, and take the key point corresponding to the depth information that satisfies the first preset condition as the key point to be processed. The first preset condition includes: the depth value is negative, or the depth value is non-negative and the sum of reprojection errors is greater than or equal to the first threshold.

[0041] Step S30: Process the key points to be processed to correct the key points and their corresponding depth information, and update the depth information.

[0042] This depth information calculation method is used, for example, in Simultaneous Localization and Mapping (SLAM) algorithms based on monocular cameras, and further, for example, in SLAM algorithms in augmented reality devices (e.g., AR glasses).

[0043] For example, in step S10, depth information of one or more keypoints is acquired, which is determined based on a depth information algorithm. The depth information algorithm may include a triangulation algorithm, and the depth information can be obtained through the three-dimensional coordinates of the keypoints in the world coordinate system. The three-dimensional coordinates of the keypoints in the world coordinate system are obtained through the two-dimensional coordinates of the projection points in each frame of the image and the real-time pose of the camera. That is, the triangulation algorithm can be used when the SLAM algorithm estimates the depth of keypoints. For example, the depth of a keypoint refers to the distance from the keypoint in the image to the optical center of the camera. For detailed explanations of keypoints and depth, please refer to the conventional design; further details are omitted here.

[0044] like Figure 3 As shown, in some examples, the triangulation algorithm includes the following operations.

[0045] Step S101: Calculate the three-dimensional coordinate information of the projection points in the two frames of images used in the triangulation algorithm in the camera coordinate system, wherein the three-dimensional coordinate information in the camera coordinate system is the coordinate information after normalization.

[0046] Step S102: Calculate the three-dimensional coordinate information of the key points in the world coordinate system based on the poses of the key points in the two frames of images.

[0047] For example, as described in steps S101 and S102 above, the depth of the keypoints can be calculated by triangulation based on the pixel coordinates of the projection points between two frames. The input to the triangulation algorithm is the pose of the two images and the projection pixel coordinates (i.e., the coordinates of the projection points in the images) of the points in physical space (i.e., the keypoints). As shown in formula (1), x a The normalized coordinates of the projection points in the camera coordinate system of the first frame image are represented by x. b The normalized coordinates of the projection points in the camera coordinate system of the second frame image are represented by P1, P2, and X. Here, normalized coordinates mean dividing the coordinates of all points in 3D space by the Z-axis coordinate; that is, all point coordinates are transformed to a unit distance in front of the camera. The plane at a unit distance in front of the camera is also called the normalized plane.

[0048]

[0049] For example, pose can be represented by a camera pose matrix, which reflects the camera's position in the world coordinate system and represents the viewing angle (viewing position and viewing angle), that is, the viewing angle selected when capturing the image. For example, the camera pose matrix can be a 4×4 matrix, which contains rotation matrices (e.g., a 3×3 matrix) and translation vectors (e.g., a 3×1 vector). For a detailed explanation of the camera pose matrix, please refer to the standard design documentation; it will not be elaborated upon here.

[0050] The meaning of formula (1) above is that by adjusting the coordinates X of the world coordinate system, the normalized coordinates of the camera's optical center and projection point are ultimately collinear with the spatial point (i.e., the key point). Regarding the solution of formula (1), x in formula (1) can be... a Transform the cross product P1X into the form x a The antisymmetric matrix form of x in formula (1) b Transform the cross product P2X into the form x b The antisymmetric matrix form, that is, respectively (x a )^ and (x bAs shown in formula (2), the left side of formula (2) is decomposed by Singular Value Decomposition (SVD), and the singular vector corresponding to the smallest singular value is the solution for X. Thus, the 3D coordinates X of the keypoint in the world coordinate system can be obtained, and the depth information is obtained accordingly. For a detailed explanation of the process of solving for keypoints using the triangulation algorithm, as well as the 3D coordinates and depth information, please refer to conventional design; it will not be elaborated here.

[0051]

[0052] For example, in some examples, before step S10, the following operation can be performed: tracking at least one keypoint in multiple frames of images using optical flow. The projection points of the keypoint in the two images can be associated using optical flow. Optical flow uses the grayscale consistency assumption to calculate the velocity of the keypoint in the pixel plane and then estimate the coordinates of the keypoint in the new image. The coordinates of the keypoint in the new image are the coordinates of the projection point in the current image. In the embodiments of this disclosure, the projection points (FAST corner points) of the image can be tracked using optical flow, and then triangulation is performed based on the coordinates of the projection points in the two frames to calculate the depth of the keypoint. Optical flow has advantages such as high speed and low computational cost in feature association, but it may produce incorrect associations. Incorrect associations will lead to incorrect depth estimates in subsequent steps, so appropriate processing is required in subsequent steps.

[0053] For example, in step S20, it is determined that the depth information corresponding to at least one of the key points satisfies a first preset condition, and the key point corresponding to the depth information that satisfies the first preset condition is taken as a key point to be processed. The first preset condition includes: the depth value is negative, or the depth value is non-negative and the sum of reprojection errors is greater than or equal to a first threshold.

[0054] For example, in this step, it is necessary to determine whether the depth information of each key point meets the first preset condition, and select the key points whose depth information meets the first preset condition as key points to be processed. There can be one or more key points to be processed, and the embodiments of this disclosure do not limit this. For example, the depth of the key points obtained by triangulation calculation may have problems such as negative depth values ​​or poor depth accuracy. This is due to inaccurate association of key points (e.g., inaccurate association of projection points in the image) and / or low accuracy of the triangulation algorithm. Therefore, the first preset condition includes the case of negative depth values, as well as the case of inaccurate depth calculated by triangulation. For the case of inaccurate depth calculated by triangulation, specifically, it means that the depth value is non-negative and the sum of reprojection errors is greater than or equal to the first threshold.

[0055] For example, the sum of reprojection errors refers to the sum of the reprojection errors of the keypoints to be processed in each of the two image frames. In the aforementioned triangulation calculation, two image frames were used. The keypoints were reprojected onto the first image frame and the reprojection error was calculated. The keypoints were then reprojected onto the second image frame and the reprojection error was calculated again. These two reprojection errors were then added together to obtain the sum of the reprojection errors. The reprojection error refers to the deviation between the projected point of the keypoint onto the image plane and the observation point (the observation point is the pixel on the image plane corresponding to the projected point).

[0056] For example, the sum of reprojection errors can be calculated using formula (3). In formula (3), B represents the sum of reprojection errors, u i Let S represent the coordinates of the observation point in the i-th frame, K represent the camera intrinsic parameters, and S represent the coordinates of the observation point in the i-th frame. i Represents the normalization coefficient. Let P represent the pose of the i-th frame in the Lie group space, and let P represent the coordinates of the spatial point (i.e., the three-dimensional spatial coordinates of the key point).

[0057]

[0058] For example, by determining the relationship between the sum of reprojection errors and a first threshold, the accuracy of the keypoint's depth can be judged. If the sum of reprojection errors is less than the first threshold, it indicates high depth accuracy, and the corresponding keypoint can be used for subsequent operations, such as subsequent depth information estimation and updating. If the sum of reprojection errors is greater than or equal to the first threshold, it indicates inaccurate depth, requiring correction and processing of the keypoint and its depth information. For example, the value range of the first threshold can be 1–3, or 1.5–2, 1–3.5, 1–2, etc., thereby selecting keypoints with insufficient accuracy while minimizing computational load, thus improving the overall processing effect. It should be noted that the specific value of the first threshold can be determined according to actual needs and is not limited to the values ​​listed above; the embodiments of this disclosure do not impose such limitations.

[0059] For example, in step S30, the key points to be processed are processed to correct the key points and their corresponding depth information, and the depth information is updated. That is, in this step, key points with negative depth values ​​are processed, and / or key points with non-negative depth values ​​and whose sum of reprojection errors is greater than or equal to a first threshold are processed, thereby correcting these erroneous key points and depth information, thereby updating the depth information and obtaining more accurate and precise key points and depth information.

[0060] For example, in some examples, the first preset condition includes a non-negative depth value and a sum of reprojection errors greater than or equal to a first threshold. That is, in this case, the depth value of the key point to be processed selected based on the first preset condition is greater than or equal to zero, and the sum of its reprojection errors is greater than or equal to the first threshold. In this case, step S30 may further include the following operations, such as... Figure 4 As shown.

[0061] Step S31: Calculate the Hamming distance between the descriptors of the key points to be processed in the two frames of images;

[0062] Step S32: Confirm that the Hamming distance meets the second preset condition, wherein the second preset condition includes the Hamming distance being less than the second threshold;

[0063] Step S33: Use an optimization algorithm to recalculate the depth information of the key points to be processed, and obtain new depth information;

[0064] Step S34: Confirm that the Hamming distance does not meet the second preset condition;

[0065] Step S35: Update the association between the key points to be processed in the two frames of images, and use the depth information algorithm to determine the depth information of the updated key points.

[0066] For example, in step S31, the Hamming distance between the descriptors of the keypoint to be processed in two frames of images is calculated. These two frames are the images used in the depth information algorithm used in step S10, for example, the two frames used in the triangulation calculation. For example, the descriptor of the keypoint to be processed in each of the two frames can be calculated separately. That is, the descriptor of the keypoint to be processed in the first frame image is calculated, and the descriptor of the keypoint to be processed in the second frame image is calculated. For example, the descriptor includes binary-coded feature descriptors, such as BRIEF (Binary Robust Independent Elementary Features) descriptors. After obtaining the BRIEF descriptors of the keypoint to be processed in the two frames of images, the Hamming distance between these two BRIEF descriptors can be calculated.

[0067] It should be noted that in the embodiments of this disclosure, the descriptors are not limited to the BRIEF descriptor, and the distance between descriptors is not limited to the Hamming distance. Other types of descriptors and distances can also be used, which can be determined according to actual needs. The embodiments of this disclosure do not limit this.

[0068] Next, it is necessary to determine whether the Hamming distance between the descriptors of the key point to be processed in the two frames of images satisfies a second preset condition. For example, the second preset condition is that the Hamming distance is less than a second threshold. It should be noted that the second threshold can be any applicable specific value, which can be determined according to actual needs, and the embodiments of this disclosure do not limit it.

[0069] For example, in step S32, if the Hamming distance is less than the second threshold, it is confirmed that the Hamming distance meets the second preset condition. In this step, if it is confirmed that the Hamming distance meets the second preset condition, that is, if it is confirmed that the Hamming distance is less than the second threshold, it means that the projection point associated by the optical flow method is accurate.

[0070] For example, in step S33, an optimization algorithm is used to recalculate the depth information of the key points to be processed, resulting in new depth information. That is, since it is confirmed that the Hamming distance is less than the second threshold, meaning the key point descriptor matching is successful and the projection points associated by the optical flow method are accurate, it indicates that the inaccurate depth value is due to the low precision of the depth information algorithm (e.g., the triangulation algorithm). Therefore, in this step, an optimization algorithm is used to recalculate the depth information of the key points to be processed, thereby obtaining new depth information. For example, the optimization algorithm used is different from the depth information algorithm, that is, different from the algorithm used to calculate the key points in step S10. This optimization algorithm minimizes the sum of reprojection errors corresponding to the new depth information, and can obtain key points with higher precision.

[0071] For example, an optimization algorithm can refer to any method used for optimization calculations. For instance, in some examples, an optimization method can be used to solve the problem by projecting the three-dimensional coordinates P of the keypoint to be processed in space onto all frames that observe that point, as shown in Equation (4). The depth of the keypoint is adjusted so that the deviation between its projection on all observation frames and the pixel coordinates of the observation point (e.g., the coordinates of the pixel point corresponding to the projection point on the observation frame) is very small. In Equation (4), u i Let S represent the coordinates of the observation point in the i-th frame, K represent the camera intrinsic parameters, and S represent the coordinates of the observation point in the i-th frame. i Represents the normalization coefficient. Let P represent the pose of the i-th frame in the Lie group space, and let P represent the spatial point coordinates (i.e., the 3D coordinates of the keypoint). By solving formula (4), the optimized new 3D points (i.e., new keypoints with 3D coordinates) can be obtained, and the new 3D points and their corresponding new depth information can be used for subsequent calculations. It should be noted that any applicable optimization algorithm can be used, not limited to the algorithm shown in formula (4), which can be determined according to actual needs. The embodiments of this disclosure do not impose any restrictions on this.

[0072]

[0073] For example, in step S34, if the Hamming distance is greater than or equal to the second threshold, it is confirmed that the Hamming distance does not meet the second preset condition. In this step, if it is confirmed that the Hamming distance does not meet the second preset condition, that is, if it is confirmed that the Hamming distance is greater than or equal to the second threshold, it means that the projection points associated by the optical flow method are inaccurate, and the association relationship needs to be adjusted, and then the triangulation calculation needs to be re-performed.

[0074] For example, in step S35, the association between the key points to be processed in the two frames of images is updated, and the depth information of the updated key points is determined using a depth information algorithm. That is, since it is confirmed that the Hamming distance is greater than or equal to the second threshold, it is confirmed that the projection points associated by the optical flow method are inaccurate. Therefore, it is explained that the inaccurate depth values ​​are caused by the inaccurate projection points of the key points in the two frames of images. Thus, in this step, the association between the key points to be processed in the two frames of images is updated, and the depth information of the updated key points is determined using a depth information algorithm.

[0075] Figure 5 for Figure 4 An exemplary flowchart of step S35. Figure 5 As shown, in some examples, step S35 may further include the following operations.

[0076] Step S351: Traverse within a preset range of the projection points of the key points to be processed in the next frame of the two frames and determine new projection points, then replace the projection points of the key points to be processed with the new projection points.

[0077] Step S352: Calculate the corresponding depth information based on the updated association relationship using the depth information algorithm.

[0078] For example, in step S351, since it has been confirmed that the Hamming distance is greater than or equal to the second threshold, that is, it has been confirmed that the descriptor matching of the key point has failed, the projection point position associated by the optical flow method is inaccurate. Therefore, it is explained that the inaccurate depth value is caused by the inaccurate association of the projection points. Therefore, in this step, the preset range of the projection points of the key point to be processed in the second frame of the two frames is traversed and a new projection point is determined. The new projection point replaces the projection point of the key point to be processed (that is, replaces the original projection point).

[0079] For example, in step S352, since a new projection point has been determined in the second frame image, a new projection point association can be established between the first frame image and the second frame image. Therefore, in this step, the corresponding depth information is recalculated based on the updated association using a depth information algorithm (e.g., triangulation algorithm).

[0080] Figure 6 for Figure 5An exemplary flowchart of step S351. Figure 6 As shown, in some examples, step S351 may further include the following operations.

[0081] Step S3511: Determine the pixels within a preset range of the projection point location of the key point to be processed in the second frame of the two-frame image.

[0082] Step S3512: Select at least one pixel as a candidate pixel based on preset rules;

[0083] Step S3513: Calculate the binary-coded feature descriptor for each candidate point;

[0084] Step S3514: Select candidate points from the calculated feature descriptors whose Hamming distance to the feature descriptors of the projection points in the previous frame of the two frames satisfies the second preset condition, and use the selected candidate points as new projection points to replace the projection points of the key points to be processed.

[0085] For example, in step S3511, pixels within a preset range of the projection point location of the key point to be processed can be determined in the second frame of the image (e.g., the second frame). For example, the preset range may include a 5*5 pixel range. Of course, the embodiments of this disclosure are not limited to this; the preset range can be any size pixel range, such as 6*6, 10*10, 3*3, etc., and is not limited to a square; it can be any shape such as a rectangle or a circle, depending on actual needs. The embodiments of this disclosure do not impose any restrictions on this.

[0086] For example, in step S3512, at least one pixel is selected as a candidate pixel based on a preset rule. For example, the preset rule may include at least one of the following rules: all pixels within a preset range except for the projection point of the key point to be processed are selected as candidate pixels; among the pixels within the preset range except for the projection point of the key point to be processed, a subset of pixels are selected according to a preset step size; among the pixels within the preset range except for the projection point of the key point to be processed, a preset number of pixels are randomly selected. For example, the preset rule may be one or more of the above rules.

[0087] For example, in some examples, assuming the preset range is a 5x5 pixel range, all pixels within this range except for the projection point can be used as candidate points; that is, all 24 pixels within this range except for the projection point can be used as candidate points. For example, in other examples, still assuming the preset range is a 5x5 pixel range, some pixels can be selected from these 24 pixels according to a preset step size, for example, selecting one pixel every other pixel (i.e., a step size of 2), thus selecting 12 pixels as candidate points. For example, in other examples, still assuming the preset range is a 5x5 pixel range, a preset number of pixels can be randomly selected from these 24 pixels as candidate points, such as randomly selecting 1, 5, 8, 10, 15, or any other number. It should be noted that in the embodiments of this disclosure, the preset rules are not limited to the specific rules described above, and can also be any other applicable rules, which can be determined according to actual needs. The embodiments of this disclosure do not limit this. The number of candidate points can be one or more, and the embodiments of this disclosure do not limit this.

[0088] For example, in step S3513, a binary-coded feature descriptor is calculated for each candidate point, such as the BRIEF descriptor for each candidate point.

[0089] For example, in step S3514, candidate points are selected from the calculated feature descriptors whose Hamming distance to the feature descriptors of the projection points in the previous frame of the two frames satisfies a second preset condition. These selected candidate points are then used as new projection points to replace the projection points of the key points to be processed (i.e., to replace the original projection points). In other words, the previous step calculated the BRIEF descriptors of pixels near the projection points in the second frame. Therefore, in this current step, candidate points are selected from these BRIEF descriptors whose Hamming distance to the BRIEF descriptors of the projection points of the key points to be processed in the first frame satisfies the second preset condition, and these selected points are used as new projection points in the second frame, thus replacing the original projection points in the second frame. For example, the second preset condition is that the Hamming distance is less than a second threshold. For example, in some examples, further, the candidate point corresponding to the feature descriptor with the smallest Hamming distance to the feature descriptor of the projection point in the previous frame of the two frames can be selected from the calculated feature descriptors. The selected candidate point is used as the new projection point to replace the original projection point. In this way, the most accurate key point can be selected.

[0090] By using the above method, the current pixel plane projection point C can be adjusted to a more precise position C', making the new projection point C' more accurate. This disclosure provides a projection point association algorithm that traverses all pixels within a 5x5 image neighborhood of the current projection point C, calculates the Hamming distance between the descriptor of each pixel and the descriptor of the projection point in the first frame image, and finally selects the pixel C' whose Hamming distance satisfies a second preset condition (further, for example, minimum) to replace C as the new projection point. Then, C' is used to recalculate the triangulation of the keypoints. This can improve the accuracy of the keypoints and compensate for the shortcomings of the optical flow method.

[0091] In the embodiments of this disclosure, optical flow and descriptor matching algorithms are combined. The descriptor matching algorithm offers advantages such as high projection point matching accuracy, scale invariance, and rotation invariance, while the optical flow method offers advantages such as high speed and low computational cost. This results in high accuracy of projection point association with low overall computational cost and high speed. This combination of optical flow and descriptor matching avoids the problems of low accuracy and erroneous associations that occur when using only optical flow, and avoids the problems of high computational cost and long processing time that occur when using only descriptor matching. The combined approach can quickly associate projection points between frames while ensuring the accuracy of projection point association, thereby effectively improving overall processing efficiency and accuracy.

[0092] For example, in some examples, the first preset condition includes a negative depth value; that is, in this case, the depth value of the key point to be processed selected based on the first preset condition is less than zero. In this case, step S30 may further include the following operation.

[0093] Step S36: Discard key points with negative depth values.

[0094] For example, in step S36, keypoints with negative depth values ​​need to be filtered out and discarded. This is because the depth information estimated by the triangulation algorithm may contain anomalies, such as negative depths. These are keypoints with inaccurate depth values ​​and therefore cannot be used for subsequent processing, so they need to be discarded.

[0095] Figure 7 This is a schematic diagram illustrating the workflow of a depth information calculation method provided in some embodiments of this disclosure. The following is in conjunction with... Figure 7 The workflow of the depth information calculation method provided in the embodiments of this disclosure will be briefly described.

[0096] like Figure 7As shown, in the SLAM algorithm, when implementing localization and mapping functions, the depth information of key points is first estimated using a triangulation algorithm. Then, it is determined whether the depth value of each key point is less than zero. If the depth value is less than zero, it indicates an incorrect key point, and therefore negative depth points are removed, for example, step S36 above. If the depth value is greater than or equal to zero, the sum of the reprojection errors of the key points is calculated. The sum of the reprojection errors is the sum of the reprojection errors of the key points in the two frames of images respectively.

[0097] Then, it is determined whether the sum of reprojection errors is less than a first threshold (e.g., threshold S). If the sum of reprojection errors is less than the first threshold, it indicates that the accuracy of the key point is high, and therefore the depth estimation is successful. This key point can be used for subsequent loop closure optimization and VIO.

[0098] If the sum of reprojection errors is greater than or equal to the first threshold, it indicates that the depth estimation of the keypoint is inaccurate. In this case, further judgment is made. For example, the descriptors of the keypoint in the two frames are extracted, and it is determined whether the Hamming distance between the descriptors is less than the second threshold, that is, whether the descriptors are successfully matched.

[0099] If the Hamming distance between descriptors is less than the second threshold, it indicates that the descriptor matching is successful, meaning that the projection points tracked by the optical flow method are correct. The reason for the inaccurate depth estimation is due to the accuracy of the triangulation algorithm. Therefore, an optimized algorithm is used to recalculate the depth information of the key points.

[0100] If the Hamming distance between descriptors is greater than or equal to the second threshold, it indicates that the descriptor matching has failed, meaning that the projection points tracked by the optical flow method are inaccurate. The inaccurate depth estimation is due to the inaccuracy of the optical flow tracking, so it is necessary to redetermine the association relationship of the projection points. For example, one can traverse around the projection point to find the corresponding pixel, find the pixel whose Hamming distance to the corresponding keypoint in another frame satisfies the second preset condition (further, for example, minimum), replace the original projection point with the new pixel, and then recalculate the triangulation.

[0101] The above method can handle depth anomalies in AR glasses and process keypoint triangulation anomalies. For example, this method can be modularly encapsulated (e.g., into a depth anomaly processing module) and embedded into the SLAM mapping module. This module improves the ability of the monocular AR glasses' SLAM algorithm to perceive surrounding environmental information, estimates the depth information of keypoints with high accuracy, and eliminates erroneous depth information. This ensures that the depth information of keypoints used in the VIO optimization process is valid and accurate, and also ensures that the depth information of keypoints used by the relocalization module in the loop closure optimization module is valid and accurate. Applying the depth information calculation method provided in this embodiment to the mapping module facilitates the reconstruction of the system's surrounding environment into a 3D point cloud map, improving the accuracy of the 3D point cloud map. This depth information calculation method is suitable for monocular AR glasses and can be used to estimate the depth information of the environment using triangulation algorithms.

[0102] For example, a depth anomaly processing module can be added after the triangulation algorithm to optimize the depth information based on the depth information calculation method provided in this embodiment. If the depth deviation of a keypoint is large and the association accuracy is high (corresponding to the cases described above where the depth value is greater than or equal to zero, the sum of reprojection errors is greater than or equal to the first threshold, and the Hamming distance is less than the second threshold), then the optimized algorithm is used to re-estimate the depth. If the depth deviation of a keypoint is large and the association is poor (corresponding to the cases described above where the depth value is greater than or equal to zero, the sum of reprojection errors is greater than or equal to the first threshold, and the Hamming distance is greater than or equal to the second threshold), then the position of the projection point of the keypoint on the pixel plane is adjusted until the association is correct before re-triangulation. If the depth value of a keypoint is negative, then the keypoint is directly discarded. Therefore, by using the above method, key points with depth errors (such as key points with negative depth values) are eliminated, and the depth information of key points with large depth error deviations is recalculated by optimizing the calculation. By adjusting the position of the projection point of the key point on the pixel plane and recalculating the depth information, the depth value of each key point is made effective and highly accurate. This ensures that the depth information of the key points in the mapping module is both in line with expectations and has a small deviation from the true value, thereby improving the performance of the subsequent loop closure optimization module and VIO module.

[0103] Figure 8A This is a schematic diagram of a three-dimensional point cloud map. Figure 8B This is a schematic diagram of a 3D point cloud map generated using the depth information obtained by the depth information calculation method provided in this embodiment of the disclosure. Figure 8A and Figure 8B In the diagram, rectangles represent cameras, lines represent camera trajectories, and dots represent point clouds.

[0104] like Figure 8AAs shown, due to the lack of depth anomaly processing, many 3D points with negative depth (i.e., keypoints with 3D coordinates) appear behind the camera. These negative depth keypoints will affect the accuracy of subsequent VIO pose estimation and loop closure optimization. Figure 8B As shown, by using the depth information calculation method provided in this embodiment, negative depth keypoints behind the camera are removed, and inaccurate keypoints are corrected. The keypoints after anomaly processing can be used for pose estimation in the VIO system and participate in subsequent loop closure optimization. In typical schemes without depth anomaly processing, both negative depth keypoints and inaccurate depth keypoints participate in the calculation of the perspective-n-point (PNP) for loop closure optimization, leading to poor final loop closure performance. In contrast, the depth information calculation method provided in this embodiment can obtain more accurate pose constraints for the current frame and the loop closure frame.

[0105] It should be noted that the execution order of the various steps in the depth information calculation method is not limited in the embodiments of this disclosure. Although the execution process of each step has been described in a specific order above, this does not constitute a limitation on the embodiments of this disclosure. The various steps in this depth information calculation method can be executed serially or in parallel, depending on actual needs. The depth information calculation method may also include more or fewer steps, depending on actual needs.

[0106] At least one embodiment of this disclosure also provides an electronic device. This electronic device can improve the accuracy of depth information, making the depth information effective and precise, ensuring that the depth information of key points both meets expectations and has minimal deviation from the true value, thereby improving overall computational efficiency and the effectiveness of subsequent processing steps.

[0107] Figure 9 This is a schematic block diagram of an electronic device provided for some embodiments of this disclosure. For example... Figure 9 As shown, the electronic device 300 includes a processor 310 and a memory 320. The memory 320 stores non-transitory computer-readable instructions (e.g., one or more computer program modules). The processor 310 executes the non-transitory computer-readable instructions, which, when executed by the processor 310, can perform one or more steps in the depth information calculation method described above. The memory 320 and the processor 310 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0108] For example, processor 310 may be a central processing unit (CPU), a digital signal processor (DSP), or other processing unit with data processing and / or program execution capabilities, such as a field-programmable gate array (FPGA); for example, the central processing unit (CPU) may be an x86 or ARM architecture. Processor 310 may be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 300 to perform desired functions.

[0109] For example, memory 320 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and processor 310 may run one or more computer program modules to implement various functions of electronic device 300. Various application programs and various data, as well as various data used and / or generated by the application programs, may also be stored in the computer-readable storage medium.

[0110] It should be noted that, in the embodiments of this disclosure, the specific functions and technical effects of the electronic device 300 can be referred to the description of the depth information calculation method above, and will not be repeated here.

[0111] Figure 10 This is a schematic block diagram of another electronic device provided in some embodiments of this disclosure. The electronic device 400 is, for example, suitable for implementing the depth information calculation method provided in embodiments of this disclosure. The electronic device 400 can be an AR device, such as AR glasses. It should be noted that... Figure 10 The illustrated electronic device 400 is merely an example and does not impose any limitation on the functionality and scope of use of the embodiments of this disclosure.

[0112] like Figure 10As shown, electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 410, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 420 or a program loaded from storage device 480 into random access memory (RAM) 430. The RAM 430 also stores various programs and data required for the operation of electronic device 400. The processing device 410, ROM 420, and RAM 430 are interconnected via bus 440. Input / output (I / O) interface 450 is also connected to bus 440.

[0113] Typically, the following devices can be connected to I / O interface 450: input devices 460 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 470 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 480 including, for example, magnetic tapes, hard disks, etc.; and communication devices 490. Communication device 490 allows electronic device 400 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 10 An electronic device 400 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 400 may alternatively implement or have more or fewer devices.

[0114] For example, according to embodiments of this disclosure, Figure 2 The depth information calculation method shown can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program including program code for performing the aforementioned depth information calculation method. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 490, or installed from a storage device 480, or installed from a ROM 420. When the computer program is executed by the processing device 410, it can perform the functions defined in the depth information calculation method provided in the embodiments of this disclosure.

[0115] At least one embodiment of this disclosure also provides a storage medium storing non-transitory computer-readable instructions. When these instructions are executed by a computer, they can implement the depth information calculation method provided in any embodiment of this disclosure. Using this storage medium, the accuracy of depth information can be improved, making the depth information effective and precise. This ensures that the depth information of key points both meets expectations and has minimal deviation from the actual value, thereby improving overall computational efficiency and the effectiveness of subsequent processing steps.

[0116] Figure 11 This is a schematic diagram of a storage medium provided for some embodiments of this disclosure. For example... Figure 11As shown, the storage medium 500 stores non-transitory computer-readable instructions 510. For example, when the non-transitory computer-readable instructions 510 are executed by a computer, one or more steps in the depth information calculation method described above can be performed.

[0117] For example, the storage medium 500 can be used in the aforementioned electronic device 300. For example, the storage medium 500 can be a memory in the electronic device 400. For example, for relevant descriptions of the storage medium 500, please refer to the corresponding description of the memory of the electronic device 400 above, which will not be repeated here.

[0118] The following points need to be explained:

[0119] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0120] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0121] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure should be determined by the scope of protection of the claims.

Claims

1. A method for calculating depth information, comprising: Obtain depth information of at least one key point determined by a depth information algorithm; Determine that the depth information corresponding to at least one of the at least one key point satisfies a first preset condition, and take the key point corresponding to the depth information that satisfies the first preset condition as a key point to be processed. The first preset condition includes: the depth value is negative, or the depth value is non-negative and the sum of reprojection errors is greater than or equal to a first threshold. The key points to be processed are processed to correct the key points and their corresponding depth information, and the depth information is updated. Wherein, in response to the first preset condition including the depth value being non-negative and the sum of the reprojection errors being greater than or equal to the first threshold, the step of processing the key point to be processed to correct the key point to be processed and the corresponding depth information, and updating the depth information, includes: Calculate the Hamming distance between the descriptors of the key points to be processed in the two frames of images; Confirm that the Hamming distance meets a second preset condition, wherein the second preset condition includes that the Hamming distance is less than a second threshold. The depth information of the key points to be processed is recalculated using an optimized algorithm, and the depth information is updated.

2. The method according to claim 1, wherein, The sum of the reprojection errors is the sum of the reprojection errors of the key point to be processed in each of the two frames of images.

3. The method according to claim 1, wherein, Processing the key points to be processed to correct the key points and their corresponding depth information, and updating the depth information, further includes: It is confirmed that the Hamming distance does not meet the second preset condition; The association between the key points to be processed in the two frames of images is updated, and the depth information of the updated key points is determined using the depth information algorithm, and the depth information is updated.

4. The method according to claim 3, wherein, The association between the key points to be processed in the two frames of images is updated, and the depth information of the updated key points is determined using the depth information algorithm. Updating the depth information includes: Traverse within a preset range of the projection points of the key points to be processed in the next frame of the two frames and determine new projection points, then replace the projection points of the key points to be processed with the new projection points. The depth information is calculated based on the updated association relationship using the depth information algorithm, and the depth information is updated.

5. The method according to claim 4, wherein, Traversing within a preset range of the projection points of the key points to be processed in the latter frame of the two images and determining the new projection points, then replacing the projection points of the key points to be processed with the new projection points, includes: Determine the pixels within a preset range of the projection point of the key point to be processed in the next frame of the two-frame images. At least one pixel is selected as a candidate pixel based on preset rules. Calculate the binary-encoded feature descriptor for each candidate point; In the calculated feature descriptors, select the candidate points corresponding to the feature descriptors whose Hamming distance to the feature descriptors of the projection points in the previous frame of the two frames satisfies the second preset condition, and use the selected candidate points as the new projection points to replace the projection points of the key points to be processed.

6. The method according to claim 5, wherein, The preset range includes a 5*5 pixel range.

7. The method according to claim 5, wherein, The preset rules include at least one of the following rules: All pixels within the preset range, except for the projection points of the key points to be processed, are used as candidate points. Within the preset range, a portion of the pixels other than the projection points of the key points to be processed are selected according to a preset step size. A preset number of pixels are randomly selected from the pixels within the preset range, excluding the projection points of the key points to be processed.

8. The method according to claim 1, wherein, The first preset condition includes the depth value being negative. The key points to be processed are processed to correct the key points and their corresponding depth information, and the depth information is updated, including: Discard the key points to be processed that have a negative depth value.

9. The method according to claim 1, wherein, The depth information algorithm includes a triangulation algorithm. The depth information is obtained through the three-dimensional coordinate information of the key points in the world coordinate system. The three-dimensional coordinate information of the key points in the world coordinate system is obtained through the two-dimensional coordinate information of the projection points in each frame of the image and the real-time pose of the camera. The triangulation algorithm includes: Calculate the three-dimensional coordinate information of the projection points in the camera coordinate system in the two frames of images used in the triangulation algorithm, wherein the three-dimensional coordinate information in the camera coordinate system is normalized coordinate information; Based on the poses of the key points in the two frames of images, the three-dimensional coordinate information of the key points in the world coordinate system is calculated.

10. The method according to claim 1, wherein, The method is used for augmented reality devices.

11. An electronic device, comprising: processor; Memory, including one or more computer program modules; The one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules are used to implement the depth information calculation method of claim 1.

12. A non-transitory computer-readable storage medium storing non-transitory computer-readable instructions that, when executed by a computer, implement the depth information calculation method of claim 1.

Images