Method and device for determining pose transformation relationship, terminal and storage medium

Abstract

The present disclosure relates to a method and device for determining a pose transformation relationship, a terminal and a storage medium, wherein the terminal comprises an RGB-D camera, and the method comprises: determining a plurality of first feature points according to a first color image output by a color camera component of the RGB-D camera when the RGB-D camera is at a first position; determining a plurality of second feature points according to a second color image output by the color camera component when the RGB-D camera is at a second position; determining a plurality of first matching feature point pairs matched in RGB according to the first image information and the second image information from the plurality of first feature points and the plurality of second feature points; and determining a first homography matrix according to the plurality of first matching feature point pairs, the first homography matrix being used to represent a pose transformation relationship of the RGB-D camera from the first position to the second position. The method can greatly reduce the false matching, so that the first homography matrix can more accurately determine the pose transformation relationship of the RGB-D camera from the first position to the second position.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of terminals, and in particular, to a method and apparatus for determining a pose transformation relationship, a terminal, and a storage medium. BACKGROUND

[0002] An RGB-D camera generally includes a color (RGB) camera component and a depth (D, full name: depth) camera component, and the RGB-D camera has a wide range of uses in the field of visual positioning.

[0003] However, the pose transformation relationship of the RGB-D camera generally includes two types. The first type is determined only according to an image captured by the color camera component; and the second type is determined only according to an image captured by the depth camera component. The accuracy of the pose transformation relationship determined by the above two methods is poor. SUMMARY

[0004] To overcome the problems in the related art, the present disclosure provides a method and apparatus for determining a pose transformation relationship, a terminal, and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for determining a pose transformation relationship is provided, which is applied to a terminal including an RGB-D camera, and the method includes:

[0006] determining a plurality of first feature points according to a first color image output by a color camera component of the RGB-D camera when the RGB-D camera is at a first position;

[0007] determining a plurality of second feature points according to a second color image output by the color camera component when the RGB-D camera is at a second position;

[0008] determining a plurality of first matching feature point pairs that are matched in RGB according to first image information and second image information, from the plurality of first feature points and the plurality of second feature points, wherein the first image information is image information output by a depth camera component of the RGB-D camera when the RGB-D camera is at the first position; and the second image information is image information output by the depth camera component when the RGB-D camera is at the second position;

[0009] determining a first homography matrix according to the plurality of first matching feature point pairs, the first homography matrix being used to represent a pose transformation relationship of the RGB-D camera from the first position to the second position.

[0010] Optionally, the first image information includes a first depth image and a first confidence image, and the second image information includes a second depth image and a second confidence image.

[0011] The method comprises the following steps:

[0012] According to the calibration information of the RGB-D camera, and the first depth image and the first confidence image, depth information and confidence information corresponding to the first feature points are determined;

[0013] The first feature points whose depth information and confidence information satisfy a first set condition are determined as third feature points;

[0014] According to the calibration information, and the second depth image and the second confidence image, depth information and confidence information corresponding to the second feature points are determined;

[0015] The second feature points whose depth information and confidence information satisfy a second set condition are determined as fourth feature points;

[0016] The third feature points and the fourth feature points that are matched in RGB are determined as the first matching feature point pairs.

[0017] Optionally, the first set condition comprises:

[0018] The confidence information is greater than or equal to a first set confidence; and / or,

[0019] The depth information is greater than or equal to a first set depth and less than or equal to a second set depth.

[0020] Optionally, the second set condition comprises:

[0021] The confidence information is greater than or equal to a second set confidence; and / or,

[0022] The depth information is greater than or equal to a third set depth and less than or equal to a fourth set depth.

[0023] Optionally, the method comprises the following steps:

[0024] If the similarity between the depth information of the third feature point and the depth information of the fourth feature point in the first matching feature point pair is less than or equal to a first set similarity, the first matching feature point pair is determined as a second matching feature point pair;

[0025] The first homography matrix is determined according to the second matching feature point pairs.

[0026] Optionally, the method comprises the following steps:

[0027] determine, according to the first depth image and the second depth image, extrinsic parameter estimation information of the depth camera assembly;

[0028] determine, according to a plurality of the second matching feature point pairs, a second homography matrix;

[0029] determine, according to the second homography matrix, the first color image and the second color image, disparity information;

[0030] determine, according to the extrinsic parameter estimation information, the disparity information and intrinsic parameters of the color camera assembly, depth estimation information corresponding to the second matching feature point pairs;

[0031] if a similarity between the depth information of at least one feature point in the second matching feature point pair and the corresponding depth estimation information is greater than or equal to a second set similarity, determine the second matching feature point pair as a third matching feature point pair;

[0032] determine, according to a plurality of the third matching feature point pairs, the first homography matrix.

[0033] According to a second aspect of the embodiments of the present disclosure, a device for determining a pose transformation relationship is provided, which is applied to a terminal including an RGB-D camera, and the device includes a determination module, which is configured to:

[0034] determine a plurality of first feature points according to a first color image output by a color camera assembly of the RGB-D camera when the RGB-D camera is at a first position;

[0035] determine a plurality of second feature points according to a second color image output by the color camera assembly when the RGB-D camera is at a second position;

[0036] determine, according to first image information and second image information, a plurality of first matching feature point pairs from the plurality of first feature points and the plurality of second feature points, wherein the first image information is image information output by a depth camera assembly of the RGB-D camera when the RGB-D camera is at the first position, and the second image information is image information output by the depth camera assembly when the RGB-D camera is at the second position;

[0037] determine, according to a plurality of the first matching feature point pairs, a first homography matrix, which is used to represent a pose transformation relationship of the RGB-D camera from the first position to the second position.

[0038] Optionally, the first image information includes a first depth image and a first confidence image, and the second image information includes a second depth image and a second confidence image.

[0039] The determination module is configured to:

[0040] determine, according to calibration information of the RGB-D camera and the first depth image and the first confidence image, depth information and confidence information corresponding to the first feature points;

[0041] determine, as third feature points, the first feature points whose depth information and confidence information satisfy a first set condition;

[0042] determine, according to the calibration information and the second depth image and the second confidence image, depth information and confidence information corresponding to the second feature points;

[0043] determine, as fourth feature points, the second feature points whose depth information and confidence information satisfy a second set condition;

[0044] determine, as the first matched feature point pairs, the third feature points and the fourth feature points matched in RGB.

[0045] Optionally, the first set condition comprises:

[0046] the confidence information is greater than or equal to a first set confidence; and / or,

[0047] the depth information is greater than or equal to a first set depth and less than or equal to a second set depth.

[0048] Optionally, the second set condition comprises:

[0049] the confidence information is greater than or equal to a second set confidence; and / or,

[0050] the depth information is greater than or equal to a third set depth and less than or equal to a fourth set depth.

[0051] Optionally, the determination module is configured to:

[0052] if a similarity between the depth information of the third feature point and the depth information of the fourth feature point in the first matched feature point pair is less than or equal to a first set similarity, determine the first matched feature point pair as a second matched feature point pair;

[0053] determine the first homography matrix according to a plurality of the second matched feature point pairs.

[0054] Optionally, the determination module is configured to:

[0055] determine extrinsic estimation information of the depth camera assembly according to the first depth image and the second depth image;

[0056] determine a second homography matrix according to a plurality of the second matched feature point pairs.

[0057] determining disparity information according to the second homography matrix, the first color image and the second color image;

[0058] determining depth estimation information corresponding to the second matched feature point pair according to the extrinsic parameter estimation information, the disparity information and intrinsic parameters of the color camera assembly;

[0059] if a similarity between the depth information of at least one feature point in the second matched feature point pair and the corresponding depth estimation information is greater than or equal to a second set similarity, determining the second matched feature point pair as a third matched feature point pair;

[0060] determining the first homography matrix according to a plurality of the third matched feature point pairs.

[0061] According to a third aspect of the embodiments of the present disclosure, a terminal is provided, and the terminal comprises:

[0062] a processor;

[0063] a memory for storing instructions executable by the processor;

[0064] The processor is configured to perform the method according to any one of the first aspect.

[0065] According to a fourth aspect of the embodiments of the present disclosure, a non-transitory computer readable storage medium is provided, and when instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to perform the method according to any one of the first aspect.

[0066] The technical solution provided by the embodiments of the present disclosure can have the following beneficial effects: in the method, the feature points are determined according to the color image captured by the RGB-D camera, the determined feature points are screened according to the depth image and the confidence image captured by the RGB-D camera, then the first matched feature point pairs are determined from the feature points corresponding to the first position and the feature points corresponding to the second position, which can greatly reduce the mismatched feature point pairs, and finally the first homography matrix is determined based on the more reliable first matched feature point pairs, which can more accurately determine the pose transformation relationship of the RGB-D camera from the first position to the second position.

[0067] The method is based on the physical characteristics of color images, depth images and confidence images, better avoids the mismatch of feature points, optimizes the determination method of the pose transformation relationship, and makes the finally determined first homography matrix more accurately represent the pose transformation relationship of the RGB-D camera from the first position to the second position, thereby improving the use experience of technologies such as simultaneous localization and mapping (SLAM), augmented reality (AR), virtual reality (VR), assisted driving and autonomous driving.

[0068] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0069] The accompanying drawings incorporated in and forming a part of the specification illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure, together with the description.

[0070] Figure 1 is a flowchart of a determination method of a pose transformation relationship according to an exemplary embodiment.

[0071] Figure 2 is a flowchart of a determination method of a pose transformation relationship according to an exemplary embodiment.

[0072] Figure 3 is a flowchart of a determination method of a pose transformation relationship according to an exemplary embodiment.

[0073] Figure 4 is a flowchart of a determination method of a pose transformation relationship according to an exemplary embodiment.

[0074] Figure 5 is a schematic diagram of a determination method of a pose transformation relationship according to an exemplary embodiment.

[0075] Figure 6 is a block diagram of a pose estimation device according to an exemplary embodiment.

[0076] Figure 7 is a block diagram of a terminal according to an exemplary embodiment. DETAILED DESCRIPTION

[0077] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The description below refers to the accompanying drawings, which show, by way of example, specific embodiments with which this disclosure can be practiced. The following description, however, is not intended to limit the scope of this disclosure to one or more particular embodiments described. Rather, this disclosure describes certain example embodiments, but it is not intended to exclude other embodiments that are not described but that are within the scope of the present disclosure as defined by the appended claims.

[0078] The present disclosure provides a pose change relationship determination method, applied to a terminal. In the method, feature points are determined according to a color image captured by an RGB-D camera, and the determined feature points are filtered according to a depth image and a confidence image captured by the RGB-D camera. Then, a plurality of first matching feature point pairs of RGB features are determined from the filtered feature points corresponding to a first position and the filtered feature points corresponding to a second position. This can greatly reduce the number of mis-matched feature point pairs. Finally, a first homography matrix is determined based on the more reliable plurality of first matching feature point pairs, which can more accurately determine the pose transformation relationship of the RGB-D camera from the first position to the second position.

[0079] The method is based on the physical characteristics of the color image, the depth image, and the confidence image, which can better avoid the mis-matching of feature points and optimize the determination method of the pose transformation relationship. The finally determined first homography matrix can more accurately represent the pose transformation relationship of the RGB-D camera from the first position to the second position, thereby improving the user experience of technologies such as simultaneous localization and mapping (SLAM), augmented reality (AR), virtual reality (VR), assisted driving, and autonomous driving.

[0080] In one exemplary embodiment, a pose transformation relationship determination method is provided, applied to a terminal including an RGB-D camera. The RGB-D camera generally includes a color (RGB) camera component and a depth (D, full name depth) camera component. Referring to Figure 1 The method includes:

[0081] S110, determining a plurality of first feature points according to a first color image output by the color camera component of the RGB-D camera when the RGB-D camera is at a first position;

[0082] S120, determining a plurality of second feature points according to a second color image output by the color camera component when the RGB-D camera is at a second position;

[0083] S130, determining, according to the first image information and the second image information, a plurality of first matching feature point pairs of the RGB feature from the plurality of first feature points and the plurality of second feature points;

[0084] S140, determining a first homography matrix according to the plurality of first matching feature point pairs, the first homography matrix being used to represent a pose transformation relationship of the RGB-D camera from the first position to the second position.

[0085] In step S110 and step S120, after the RGB-D camera captures the first color image at the first position, the terminal can perform feature detection based on the first color image, so as to determine a plurality of first feature points. The plurality of first feature points can constitute a feature point set, which can be denoted as a first feature point set.

[0086] Wherein, the i-th first feature point in the first feature point set can be denoted as , wherein, represents the x-axis pixel coordinate of the i-th first feature point, represents the y-axis pixel coordinate of the first feature point, represents the red component of the first feature point, represents the green component of the first feature point, represents the blue component of the first feature point.

[0087] The determination manner of the second feature point can refer to the first feature point. Wherein, after the RGB-D camera captures the first color image at the first position, the terminal can perform feature detection based on the second color image, so as to determine a plurality of second feature points. The plurality of second feature points can constitute a feature point set, which can be denoted as a second feature point set.

[0088] Wherein, the feature detection includes Sift (Scale Invariant Feature Transform) feature, surf (Speeded Up Robust Feature) feature, optical flow feature, etc., which are not limited herein.

[0089] In step S130, the first image information is the image information output by the depth camera assembly of the RGB-D camera when the RGB-D camera is at the first position. The second image information is the image information output by the depth camera assembly when the RGB-D camera is at the second position.

[0090] The first image information can include, but is not limited to, a first depth image and / or a first confidence image output by the depth camera assembly when the RGB-D camera is at the second position.

[0091] The first feature point can correspond to depth information in the first depth image based on the calibration information of the RGB-D camera. The depth information in the first depth image corresponds to confidence information in the first confidence image. Therefore, the first feature point can correspond to confidence information in the first confidence image. The reliability of the first feature point can be determined based on the depth information and the confidence information of the first feature point. The first feature points with poor reliability can be removed. That is, the first feature points with better reliability can be selected from the first feature points based on the depth information and the confidence information.

[0092] It should be noted that the first feature points can be selected based on the depth information or the confidence information, and the selection method is not limited herein.

[0093] For example, the first image information includes the first depth image. The depth information can be a depth value. If the depth value corresponding to the first feature point in the first depth image is not within a set depth value range, that is, the depth value is too large or too small, the reliability of the first feature point is considered to be poor, and the first matching feature point pair can be removed when determining the first matching feature point pair.

[0094] For example, the first image information can include the first confidence image. The confidence information can be a confidence value. If the confidence value corresponding to the first feature point in the first confidence image is less than a set confidence value, the reliability of the first feature point is considered to be poor, and the first matching feature point pair can be removed when determining the first matching feature point pair.

[0095] The second feature points with poor reliability can be removed by referring to the first feature points. Thus, the matching feature point pair with better reliability of the RGB feature matching can be determined.

[0096] In addition, the matching feature point pairs with poor depth information matching degree in the matching feature point pairs of the RGB feature matching can be selected and removed, and the matching feature point pairs with high depth information matching degree can be reserved to further avoid false matching and improve the reliability of the matching feature point pairs. The reserved matching feature point pairs are determined as the final first matching feature point pairs, so that the first matching feature point pairs are more reliable.

[0097] For example, the depth information can be a depth value. Two feature points in a matching feature point pair of the RGB feature matching can be denoted as feature point A1 and feature point A2 respectively, the depth value corresponding to the feature point A1 can be denoted as depth value , the depth value corresponding to the feature point A2 can be denoted as depth value , if the difference between the depth value and the depth value is greater than a set depth value, it is considered that the feature point A1 and the feature point A2 can belong to a false matching, and in determining the first matching feature point pair, this matching feature point pair can be removed to improve the reliability of the first matching feature point pair.

[0098] It should be noted that the RGB feature matching refers to determining whether the red (R) component, the green (G) component and the blue (B) component of two feature points are matched respectively. In the matching feature point pair of the RGB feature matching, the red (R) component, the green (G) component and the blue (B) component of the two feature points are matched respectively. The matching can be the same or similar, which can be determined according to actual requirements, and is not limited herein.

[0099] In step S140, homography estimation can be performed according to the plurality of first matching feature point pairs, so as to determine a homography matrix, which is denoted as a first homography matrix. The first homography matrix is used to represent the pose transformation relationship of the RGB-D camera from the first position to the second position. Since the first matching feature point pairs are obtained after step S130, the false matching feature point pairs can be greatly reduced, so that the first homography matrix can more accurately represent the pose transformation relationship of the RGB-D camera from the first position to the second position.

[0100] The method is based on the physical characteristics of the color image, the depth image and the confidence image, better avoids the false matching of the feature points, optimizes the determination method of the pose transformation relationship, so that the finally determined first homography matrix can more accurately represent the pose transformation relationship of the RGB-D camera from the first position to the second position, thereby improving the use experience of the technologies such as simultaneous localization and mapping (SLAM), augmented reality (AR), virtual reality (VR), assisted driving and automatic driving.

[0101] In one example embodiment, a determination method of a pose transformation relationship is provided, which is applied to a terminal including an RGB-D camera. In the method, the first image information includes a first depth image and a first confidence image, and the second image information includes a second depth image and a second confidence image.

[0102] wherein the reference Figure 2 According to the first image information and the second image information, the determining of the plurality of first matching feature point pairs matched with the RGB feature from the plurality of first feature points and the plurality of second feature points can include:

[0103] S210, determining, according to the calibration information of the RGB-D camera and the first depth image and the first confidence image, the depth information and the confidence information corresponding to the first feature points;

[0104] S220, determining the first feature points whose depth information and confidence information satisfy the first set condition as third feature points;

[0105] S230, determining, according to the calibration information and the second depth image and the second confidence image, the depth information and the confidence information corresponding to the second feature points;

[0106] S240, determining the second feature points whose depth information and confidence information satisfy the second set condition as fourth feature points;

[0107] S250, determining the third feature points and the fourth feature points matched with the RGB feature as the first matching feature point pairs.

[0108] In step S210, the depth information corresponding to the first feature points in the first depth image can be determined based on the calibration information of the RGB-D camera, and the depth information in the first depth image and the confidence information in the first confidence image have a corresponding relationship, so the confidence information corresponding to the first feature points in the first confidence image can be determined. The calibration information refers to the calibration information between the color camera component and the depth camera component in the RGB-D camera, which can be determined based on Zhang Zhengyou calibration method.

[0109] In step S220, the first set condition can include at least one of the following:

[0110] Condition 1: the confidence information is greater than or equal to the first set confidence;

[0111] Condition 2: the depth information is greater than or equal to the first set depth and less than or equal to the second set depth.

[0112] If the confidence information corresponding to the first feature points is greater than or equal to the first set confidence, it can be determined that the first feature points satisfy condition 1. If the depth information corresponding to the first feature points is greater than or equal to the first set depth and less than or equal to the second set depth, it can be determined that the first feature points satisfy condition 2.

[0113] Example 1,

[0114] The first setting condition comprises condition 1 and condition 2, wherein the confidence information can be a confidence value, the first setting confidence is denoted as The depth information can be a depth value, the first setting depth is denoted as The second setting depth is denoted as .

[0115] In this example, if the confidence value corresponding to the first feature point is greater than the first setting confidence , and the depth value corresponding to the first feature point is greater than or equal to the first setting depth , and the depth value corresponding to the first feature point is less than or equal to the second setting depth , it is considered that the depth value and the confidence value of the first feature point satisfy the first setting condition, and the first feature point can be determined as the third feature point.

[0116] The determined third feature point can constitute a feature point set, and the feature point set is denoted as a third feature point set.

[0117] In step S230, the depth information corresponding to the second feature point in the second depth image can be determined based on the calibration information of the RGB-D camera, and the depth information in the second depth image has a corresponding relationship with the confidence information in the second confidence image, so the confidence information corresponding to the second feature point in the second confidence image can be determined. The calibration information refers to the calibration information between the color camera component and the depth camera component in the RGB-D camera, which can be determined based on the Zhang Zhengyou calibration method.

[0118] In step S240, the second setting condition can be set with reference to the first setting condition, and the second setting condition can comprise at least one of the following:

[0119] Condition 3: the confidence information is greater than or equal to the second setting confidence;

[0120] Condition 4: the depth information is greater than or equal to the third setting depth and less than or equal to the fourth setting depth.

[0121] If the confidence information corresponding to the second feature point is greater than or equal to the second setting confidence, it is determined that the second feature point satisfies condition 3. If the depth information corresponding to the second feature point is greater than or equal to the third setting depth and less than or equal to the fourth setting depth, it is determined that the second feature point satisfies condition 4.

[0122] The condition 3 can be the same as the condition 1, that is, the second set confidence can be the same as the first set confidence. The condition 4 can be the same as the condition 2, that is, the third set depth can be the same as the first set depth, and the fourth set depth can be the same as the second set depth. The second set condition can be the same as the first set condition, for example, the first set condition includes the condition 1 and the condition 2, the second set condition includes the condition 3 and the condition 4, and the condition 1 is the same as the condition 2, and the condition 3 is the same as the condition 4.

[0123] The fourth feature points determined in this step can form a feature point set, which can be denoted as a fourth feature point set.

[0124] In step S250, the first matching feature point pair can be determined based on the plurality of third feature points determined in step S220 and the plurality of fourth feature points determined in step S240. It should be noted that the third feature points used in this step can be all the third feature points determined in step S220, or can be part of the third feature points determined in step S220, which is not limited herein. Similarly, the fourth feature points used in this step can be all the fourth feature points determined in step S240, or can be part of the fourth feature points determined in step S240, which is not limited herein.

[0125] The third feature points belong to the first color image, and the fourth feature points belong to the second color image. The third feature points and the fourth feature points matched in the RGB feature matching manner can be determined from the plurality of third feature points and the plurality of fourth feature points. Then, each group of matched third feature points and fourth feature points determined is determined as a first matching feature point pair.

[0126] In the first matching feature point pair, the red component of the third feature point matches the red component of the fourth feature point, the green component of the third feature point matches the green component of the fourth feature point, and the blue component of the third feature point matches the blue component of the fourth feature point. Wherein, matching can mean the same.

[0127] It should be noted that since the third feature points are the first feature points satisfying the first set condition, and the fourth feature points are the second feature points satisfying the second set condition, the reliability of the third feature points and the fourth feature points is higher, so that the reliability of the finally determined first matching feature point pair is higher, that is, the matching of the feature points in the first matching feature point pair is better, and the first homography matrix determined based on the first matching feature point pair can better represent the pose transformation relationship of the RGB-D camera from the first position to the second position.

[0128] In the method, the third feature points are selected from the first feature points based on a first set condition, the fourth feature points are selected from the second feature points based on a second set condition, and the first matching feature point pairs are determined based on the third feature points and the fourth feature points, so that the matching of the feature points in the first matching feature point pairs is better, and thus the first homography matrix can better represent the pose transformation relationship of the RGB-D camera between the first position and the second position.

[0129] In one example embodiment, a method for determining a pose transformation relationship is provided, which is applied to a terminal including an RGB-D camera. Referring to Figure 3 In the method, the first homography matrix can be determined according to the plurality of second matching feature point pairs.

[0130] In step S310, if the similarity of the depth information of the third feature point and the depth information of the fourth feature point in the first matching feature point pair is less than or equal to the first set similarity, it is considered that the depth information of the third feature point and the fourth feature point in the first matching feature point pair is matched, it is considered that the matching degree of the third feature point and the fourth feature point is better, and the first matching feature point pair is determined as the second matching feature point pair. Therefore, the matching of the feature points in the second matching feature point pair is better than that in the first matching feature point pair.

[0131] In step S320, the first homography matrix is determined according to the plurality of second matching feature point pairs.

[0132] In step S310, if the similarity of the depth information of the third feature point and the depth information of the fourth feature point in the first matching feature point pair is less than or equal to the first set similarity, it is considered that the depth information of the third feature point and the fourth feature point in the first matching feature point pair is matched, it is considered that the matching degree of the third feature point and the fourth feature point is better, and the first matching feature point pair is determined as the second matching feature point pair. Therefore, the matching of the feature points in the second matching feature point pair is better than that in the first matching feature point pair.

[0133] The first set similarity can be set before the terminal is shipped, or can be set after the terminal is shipped. In addition, the first set similarity can be modified after being set to better meet the needs of users. The specific value of the first set similarity can be determined according to actual conditions, which is not limited herein.

[0134] Example 1,

[0135] The depth information can be a depth value, and the first set similarity can be 95%.

[0136] In the first matching feature point pair, the depth information corresponding to the third feature point can be denoted as a depth value , and the depth information corresponding to the fourth feature point can be denoted as a depth value The depth value and the depth value the smaller value and the larger value, and then divides the smaller value by the larger value to obtain a percentage, which is recorded as the similarity of the depth values with the depth values .

[0137] In this example, if the determined similarity is greater than or equal to 95%, the corresponding first matching feature point pair can be determined as the second matching feature point pair.

[0138] It should be noted that, in addition to the above manner, the similarity of the depth information of the third feature point and the depth information of the fourth feature point can also be determined by other manners, which are not limited herein.

[0139] In step S320, homography estimation can be performed on the plurality of second matching feature point pairs, thereby obtaining a first homography matrix. Since the matching of the feature points in the second matching feature point pairs is better than that of the first matching feature point pairs, the first homography matrix determined based on the second matching feature point pairs can more accurately represent the pose transformation relationship of the RGB-D camera between the first position and the second position.

[0140] In this method, the plurality of first matching feature point pairs are screened according to the first similarity, thereby selecting a plurality of second matching feature point pairs with better matching, and then the first homography matrix is determined based on the second matching feature point pairs, so that the first homography matrix can more accurately represent the pose transformation relationship of the RGB-D camera between the first position and the second position.

[0141] In one example embodiment, a method for determining a pose transformation relationship is provided, which is applied to a terminal including an RGB-D camera. As shown in Figure 4 the method, the first homography matrix is determined according to the plurality of second matching feature point pairs, which can include:

[0142] S410, determining pose change information according to the first depth image and the second depth image;

[0143] S420, determining a second homography matrix according to the plurality of second matching feature point pairs;

[0144] S430, determining disparity information according to the second homography matrix, the first color image and the second color image;

[0145] S440, determining depth estimation information corresponding to the second matching feature point pair according to the pose change information, the disparity information and the intrinsic parameter of the color camera assembly;

[0146] S450, if the similarity of the depth information of at least one feature point in the second matching feature point pair and the corresponding depth estimation information is greater than or equal to a second set similarity, the second matching feature point pair is determined as a third matching feature point pair.

[0147] S460, determining the first homography matrix according to the plurality of third matching feature point pairs.

[0148] In step S410, the camera coordinate system corresponding to the target when the RGB-D camera is at the first position can be determined based on the monocular target positioning principle and the first depth image, and the camera coordinate system can be recorded as the first camera coordinate system. The camera coordinate system corresponding to the target when the RGB-D camera is at the second position can be determined based on the monocular target positioning principle and the second depth image, and the camera coordinate system can be recorded as the second camera coordinate system.

[0149] Then, based on the estimation principle of the binocular position relationship and the first camera coordinate system and the second camera coordinate system, the extrinsic parameter estimation information of the depth camera assembly in the RGB-D camera is completed.

[0150] In step S420, the second homography matrix can be determined by homography estimation according to the plurality of second matching feature point pairs.

[0151] In step S430, the image registration processing can be performed on the first color image and the second color image according to the second homography matrix, so as to determine the disparity information. It should be noted that in addition to the above method, the disparity information can also be determined by other methods, which are not limited herein.

[0152] In step S440, the depth estimation information corresponding to the second matching feature point pair can be determined by using the triangulation principle according to the extrinsic parameter estimation information, the disparity information and the intrinsic parameter of the color camera assembly. It should be noted that in addition to the triangulation principle, the above depth estimation information can also be determined based on other principles.

[0153] In step S450, after the depth estimation information corresponding to the second matching feature point pair is determined, it can be judged whether the depth information of the feature points in the corresponding second matching feature point pair is reliable based on the depth estimation information.

[0154] If the similarity between the depth information of at least one feature point in the second matching feature point pair and the corresponding depth estimation information is greater than or equal to the second set similarity, it is considered that the depth information of the feature points in the second matching feature point pair has higher reliability, and the second matching feature point pair can be determined as the third matching feature point pair.

[0155] It should be noted that the setting method of the second set similarity can refer to the first set similarity, which is not described herein.

[0156] Example 1,

[0157] ​The second set similarity is 95%, the depth information can be a depth value, and the depth estimation information can be a depth estimation value. If the similarity between the depth value of the third feature point in the second matching feature point pair and the depth estimation value corresponding to the second matching feature point pair is greater than or equal to 95%, the second matching feature point pair is determined as the third matching feature point pair.

[0158] Example 2,

[0159] The second set similarity is 95%, the depth information can be a depth value, and the depth estimation information can be a depth estimation value. If the similarity between the depth value of the third feature point in the second matching feature point pair and the depth estimation value corresponding to the second matching feature point pair is greater than or equal to 95%, the second matching feature point pair is determined as the third matching feature point pair.

[0160] Example 3,

[0161] The second set similarity is 95%, the depth information can be a depth value, and the depth estimation information can be a depth estimation value. If the similarity between the depth value of the third feature point in the second matching feature point pair and the depth estimation value corresponding to the second matching feature point pair is greater than or equal to 95%, and the similarity between the depth value of the fourth feature point in the second matching feature point pair and the depth estimation value corresponding to the second matching feature point pair is greater than or equal to 95%, the second matching feature point pair is determined as the third matching feature point pair.

[0162] In step S460, homography estimation can be performed on the plurality of third matching feature point pairs to obtain a first homography matrix. Since the reliability of the feature points in the third matching feature point pairs is better than that of the second matching feature point pairs, the first homography matrix determined based on the third matching feature point pairs can more accurately represent the pose transformation relationship of the RGB-D camera from the first position to the second position.

[0163] In the method, the plurality of second matching feature point pairs are screened according to the second similarity, so as to select a plurality of third matching feature point pairs with better reliability, and then the first homography matrix is determined based on the third matching feature point pairs, so that the first homography matrix can more accurately represent the pose transformation relationship of the RGB-D camera from the first position to the second position.

[0164] In one example embodiment, a method for determining a pose transformation relationship is provided, which is applied to a terminal, and the terminal includes an RGB-D camera.

[0165] Reference Figure 5As shown, in the method, a first color image captured by the color camera assembly when the RGB-D camera is at the first position, a first depth image and a first confidence image captured by the depth camera module when the RGB-D camera is at the first position can be acquired first, and then a second color image captured by the color camera assembly when the RGB-D camera is at the second position, a second depth image and a second confidence image captured by the depth camera module when the RGB-D camera is at the second position can be acquired.

[0166] Feature extraction is performed on the first color image to determine a plurality of first feature points in the first color image, and the plurality of first feature points form a first feature point set. Feature extraction is performed on the second color image to determine a plurality of second feature points in the second color image, and the plurality of second feature points form a second feature point set.

[0167] According to the calibration information of the color camera assembly and the depth camera module in the RGB-D camera, and the first depth image and the first confidence image, the depth value and the confidence value corresponding to the first feature points in the first feature point set are determined. And according to the above-mentioned calibration information, and the second depth image and the second confidence image, the depth value and the confidence value corresponding to the second feature points in the second feature point set are determined.

[0168] The first feature points satisfying the first set condition are determined as third feature points. Wherein, it can be judged whether the confidence value corresponding to the first feature point is greater than the first set confidence, and it can be judged whether the depth value corresponding to the first feature point is greater than or equal to the first set depth and less than or equal to the second set depth. If the judgment results are all yes, the first feature point can be retained and determined as a third feature point. Otherwise, the first feature point can be discarded. According to the above-mentioned manner, a plurality of third feature points can be determined, and the plurality of third feature points form a third feature point set.

[0169] The second feature points satisfying the second set condition are determined as fourth feature points. Wherein, it can be judged whether the confidence value corresponding to the second feature point is greater than the second set confidence, and it can be judged whether the depth value corresponding to the second feature point is greater than or equal to the third set depth and less than or equal to the fourth set depth. If the judgment results are all yes, the second feature point can be retained and determined as a fourth feature point. Otherwise, the second feature point can be discarded. According to the above-mentioned manner, a plurality of fourth feature points can be determined, and the plurality of fourth feature points form a fourth feature point set.

[0170] RGB feature matching processing is performed on the third feature point set and the fourth feature point set, and the third feature points and the fourth feature points matched by RGB feature matching are determined as a first matched feature point pair. In the first matched feature point pair, the red component, the green component and the blue component of the third feature point are matched with the red component, the green component and the blue component of the fourth feature point, respectively.

[0171] According to the above manner, a plurality of first matching feature point pairs can be determined.

[0172] Then, it is judged whether the similarity of the depth value of the third feature point and the depth value of the fourth feature point in the first matching feature point pair is less than or equal to the first set similarity. If the judgment result is yes, the first matching feature point pair can be retained and determined as a second matching feature point pair. Otherwise, the first matching feature point pair is discarded.

[0173] According to the above manner, a plurality of second matching feature point pairs can be determined. The matching degree of the feature points in the second matching feature point pair is better than that of the feature points in the first matching feature point pair.

[0174] Based on the plurality of second matching feature point pairs, homography estimation is performed to determine a second homography matrix. Then, according to the second homography matrix, the first color image and the second color image are registered to determine the parallax information.

[0175] Based on the single target estimation principle and the double target estimation principle, the extrinsic parameter estimation information of the depth camera assembly is determined according to the first depth image and the second depth image.

[0176] Then, according to the extrinsic parameter estimation information, the parallax information and the intrinsic parameter of the color camera assembly, the depth estimation value corresponding to the second matching feature point pair is determined by triangulation.

[0177] Then, the similarity of the depth value of the fourth feature point in the second matching feature point pair and the depth estimation value corresponding to the second matching feature point pair is determined, and it is judged whether the similarity is greater than or equal to the second set similarity. If the judgment result is yes, the second matching feature point pair is retained and determined as a third matching feature point pair. Otherwise, the second matching feature point pair is discarded.

[0178] According to the above manner, a plurality of third matching feature point pairs can be determined. The reliability of the third matching feature point pair is better than that of the first matching feature point pair.

[0179] Then, according to the plurality of third matching feature point pairs, the first homography matrix can be determined by homography estimation.

[0180] In the method, the depth information is multi-layered and multi-mode integrated into the determination process of the pose change relationship of the RGB-D camera, and the matching feature point pairs are screened in combination with the disparity information, the physical characteristics of the color image, the depth image and the confidence image are fully utilized, the determination process of the pose transformation relationship is optimized, the first homography matrix obtained finally can better represent the pose transformation relationship of the RGB-D camera from the first position to the second position, so that the use experience of the technologies such as simultaneous localization and mapping (SLAM), augmented reality (AR), virtual reality (VR), assisted driving and automatic driving can be improved.

[0181] In an example embodiment, a determination apparatus of a pose transformation relationship is provided, applied to a terminal including an RGB-D camera, and the apparatus is used to implement the method described above. For example, referring to FIG. 1, the apparatus includes a determination module 101, which is configured to: Figure 6

[0182] determine a plurality of first feature points according to a first color image output by a color camera component of the RGB-D camera when the RGB-D camera is at the first position;

[0183] determine a plurality of second feature points according to a second color image output by the color camera component when the RGB-D camera is at the second position;

[0184] determine a plurality of first matching feature point pairs from the plurality of first feature points and the plurality of second feature points according to first image information and second image information, wherein the first image information is image information output by a depth camera component of the RGB-D camera when the RGB-D camera is at the first position, and the second image information is image information output by the depth camera component when the RGB-D camera is at the second position;

[0185] determine a first homography matrix according to the plurality of first matching feature point pairs, and the first homography matrix is used to represent the pose transformation relationship of the RGB-D camera from the first position to the second position.

[0186] In an example embodiment, a determination apparatus of a pose transformation relationship is provided, applied to a terminal including an RGB-D camera, and in the apparatus, the first image information includes a first depth image and a first confidence image, and the second image information includes a second depth image and a second confidence image. For example, referring to FIG. 1, the determination module 101 is configured to: Figure 6

[0187] ​​According to the calibration information of the RGB-D camera, and the first depth image and the first confidence image, depth information and confidence information corresponding to the first feature points are determined;

[0188] The first feature points whose depth information and confidence information satisfy the first set condition are determined as third feature points;

[0189] According to the calibration information, and the second depth image and the second confidence image, depth information and confidence information corresponding to the second feature points are determined;

[0190] The second feature points whose depth information and confidence information satisfy the second set condition are determined as fourth feature points;

[0191] The third feature points and the fourth feature points matched by the RGB features are determined as the first matched feature point pairs.

[0192] In an example embodiment, a device for determining a pose transformation relationship is provided, and is applied to a terminal including an RGB-D camera. In the device, the first set condition includes:

[0193] The confidence information is greater than or equal to a first set confidence; and / or,

[0194] The depth information is greater than or equal to a first set depth and less than or equal to a second set depth.

[0195] In an example embodiment, a device for determining a pose transformation relationship is provided, and is applied to a terminal including an RGB-D camera. In the device, the second set condition includes:

[0196] The confidence information is greater than or equal to a second set confidence; and / or,

[0197] The depth information is greater than or equal to a third set depth and less than or equal to a fourth set depth.

[0198] In an example embodiment, a device for determining a pose transformation relationship is provided, and is applied to a terminal including an RGB-D camera. As shown in FIG. 1, the device includes a determining module 101. Figure 6 The determining module 101 is configured to:

[0199] If the similarity between the depth information of the third feature point and the depth information of the fourth feature point in the first matched feature point pair is less than or equal to a first set similarity, the first matched feature point pair is determined as a second matched feature point pair.

[0200] A first homography matrix is determined according to the second matched feature point pairs.

[0201] In an example embodiment, there is provided a device for determining a pose transformation relationship, applied to a terminal, the terminal comprising an RGB-D camera, a reference Figure 6 As shown in FIG. 1, in the device, a determining module 101 is configured to:

[0202] determine, according to the first depth image and the second depth image, the extrinsic estimation information of the depth camera assembly;

[0203] determine, according to the plurality of second matched feature point pairs, a second homography matrix;

[0204] determine, according to the second homography matrix, the first color image and the second color image, the disparity information;

[0205] determine, according to the extrinsic estimation information, the disparity information and the intrinsic of the color camera assembly, the depth estimation information corresponding to the second matched feature point pairs;

[0206] if the similarity between the depth information of at least one feature point in the second matched feature point pair and the corresponding depth estimation information is greater than or equal to a second set similarity, determine the second matched feature point pair as a third matched feature point pair;

[0207] determine, according to the plurality of third matched feature point pairs, a first homography matrix.

[0208] In an example embodiment, there is provided a terminal, such as a mobile phone, a notebook computer, a tablet computer, a wearable device, etc.

[0209] As shown in FIG. 4, the terminal 400 can include one or more of the following components: a processing component 402, a memory 404, a power supply component 406, a multimedia component 408, an audio component 410, an input / output (I / O) interface 412, a sensor component 414 and a communication component 416. Figure 7

[0210] The processing component 402 usually controls overall operations of the terminal 400, such as operations associated with display, phone call, information communication, camera operation and recording operation. The processing component 402 can include one or more processors 420 to execute instructions to complete all or part of steps of the methods described above. Further, the processing component 402 can include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 can include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.

[0211] ​The memory 404 is configured to store various types of information to support the operation of the terminal 400. Examples of such information include instructions for any application programs or methods operating on the terminal 400, contact information, phonebook information, messages, pictures, videos, and the like. The memory 404 can be implemented by any type of volatile or nonvolatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic storage, flash memory, magnetic or optical disk.

[0212] The power supply component 406 supplies power for the various components of the terminal 400. The power supply component 406 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the terminal 400.

[0213] The multimedia component 408 includes a screen providing an output interface between the terminal 400 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swiping, and gestures on the touch panel. The touch sensor can not only sense a boundary of a touching or swiping action, but also detect duration and pressure related to the touching or swiping action. In some embodiments, the multimedia component 408 includes a front camera module and / or a rear camera module. The front camera module and / or the rear camera module can receive external multimedia information when the terminal 400 is in an operating mode, such as a photographing mode or a video mode. Each of the front camera module and the rear camera module can be a fixed optical lens system or have a focal length and optical zoom capability.

[0214] The audio component 410 is configured to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC) configured to receive external audio signals when the terminal 400 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 404 or transmitted via the communication component 416. In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

[0215] The I / O interface 412 provides an interface between the processing component 402 and peripheral interface modules, which can be a keyboard, a click wheel, a button, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0216] The sensor component 414 includes one or more sensors to provide the terminal 400 with state evaluation of various aspects. For example, the sensor component 414 can detect an open / closed state of the terminal 400, relative positioning of components, such as a display and keypad of the terminal 400, a change in position of the terminal 400 or a component of the terminal 400, presence or absence of user contact with the terminal 400, orientation or acceleration / deceleration / rotation of the terminal 400, and temperature changes of the terminal 400. The sensor component 414 can include proximity sensor(s) configured to detect presence of nearby objects without any physical touch. The sensor component 414 can further include a light sensor(s), such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 414 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0217] The communication component 416 is configured to facilitate wired or wireless communication between the terminal 400 and another terminal. The terminal 400 can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, or a combination thereof. In an example embodiment, the communication component 416 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 416 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra-WideBand (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0218] In an example embodiment, the terminal 400 can be implemented with one or more Application-Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, or other electronic elements to perform the above-described methods.

[0219] In an example embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 404 including instructions, is also provided, which can be executed by the processor 420 of the terminal 400 to implement the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage terminal, etc. When the instructions in the storage medium are executed by the processor of the terminal, the terminal is enabled to perform the methods shown in the above-described embodiments.

[0220] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

[0221] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

[0222] It is to be understood that the disclosure is not limited to the precise construction described above and shown in the attached drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the disclosure is limited only by the claims that follow.

Claims

1. A method for determining pose transformation relationships, applied to a terminal, the terminal including an RGB-D camera, characterized in that, The method includes: Based on the first color image output by the color imaging component of the RGB-D camera at the first position, a plurality of first feature points are determined; Based on the second color image output by the color imaging component of the RGB-D camera at the second position, a plurality of second feature points are determined; Based on the first image information and the second image information, multiple first matching feature point pairs for RGB feature matching are determined from multiple first feature points and multiple second feature points. The first image information is the image information output by the depth camera component of the RGB-D camera when the RGB-D camera is at the first position; the second image information is the image information output by the depth camera component when the RGB-D camera is at the second position. Based on multiple first matching feature point pairs, a first homography matrix is ​​determined. The first homography matrix is ​​used to characterize the pose transformation relationship of the RGB-D camera from the first position to the second position. The first image information includes a first depth image and a first confidence image, and the second image information includes a second depth image and a second confidence image; The step of determining multiple first matching feature point pairs for RGB feature matching from multiple first feature points and multiple second feature points based on first image information and second image information includes: Based on the calibration information of the RGB-D camera, as well as the first depth image and the first confidence image, the depth information and confidence information corresponding to the first feature point are determined; The first feature point whose depth information and confidence information satisfy the first set condition is determined as the third feature point; Based on the calibration information, the second depth image, and the second confidence image, the depth information and confidence information corresponding to the second feature point are determined; The second feature point whose depth information and confidence information satisfy the second set condition is determined as the fourth feature point; The third and fourth feature points that match the RGB features are determined as the first matching feature point pair.

2. The method according to claim 1, characterized in that, The first set conditions include: The confidence level information is greater than or equal to the first set confidence level; and / or, The depth information is greater than or equal to the first set depth and less than or equal to the second set depth.

3. The method according to claim 1, characterized in that, The second setting conditions include: The confidence level information is greater than or equal to the second set confidence level; and / or, The depth information is greater than or equal to the third set depth, and less than or equal to the fourth set depth.

4. The method according to claim 1, characterized in that, The step of determining the first homography matrix based on multiple first matching feature point pairs includes: If the similarity between the depth information of the third feature point and the depth information of the fourth feature point in the first matching feature point pair is less than or equal to the first set similarity, then this first matching feature point pair is determined as the second matching feature point pair. The first homography matrix is ​​determined based on multiple pairs of second matching feature points.

5. The method according to claim 4, characterized in that, Determining the first homography matrix based on multiple pairs of second matching feature points includes: Based on the first depth image and the second depth image, determine the extrinsic parameter estimation information of the depth camera component; The second homography matrix is ​​determined based on multiple pairs of the second matching feature points; Based on the second homography matrix, the first color image, and the second color image, determine the disparity information; Based on the extrinsic parameter estimation information, the disparity information, and the intrinsic parameters of the color camera component, the depth estimation information corresponding to the second matching feature point is determined; If the similarity between the depth information of at least one feature point in the second matching feature point pair and the corresponding depth estimation information is greater than or equal to the second set similarity, then this second matching feature point pair is determined as the third matching feature point pair. The first homography matrix is ​​determined based on multiple pairs of third matching feature points.

6. A device for determining pose transformation relationships, applied to a terminal, the terminal including an RGB-D camera, characterized in that, The device includes a determining module, the determining module being configured to: Based on the first color image output by the color imaging component of the RGB-D camera at the first position, a plurality of first feature points are determined; Based on the second color image output by the color imaging component of the RGB-D camera at the second position, a plurality of second feature points are determined; Based on the first image information and the second image information, multiple first matching feature point pairs for RGB feature matching are determined from multiple first feature points and multiple second feature points. The first image information is the image information output by the depth camera component of the RGB-D camera when the RGB-D camera is at the first position; the second image information is the image information output by the depth camera component when the RGB-D camera is at the second position. Based on multiple first matching feature point pairs, a first homography matrix is ​​determined. The first homography matrix is ​​used to characterize the pose transformation relationship of the RGB-D camera from the first position to the second position. The first image information includes a first depth image and a first confidence image, and the second image information includes a second depth image and a second confidence image; The determining module is used for: Based on the calibration information of the RGB-D camera, as well as the first depth image and the first confidence image, the depth information and confidence information corresponding to the first feature point are determined; The first feature point whose depth information and confidence information satisfy the first set condition is determined as the third feature point; Based on the calibration information, the second depth image, and the second confidence image, the depth information and confidence information corresponding to the second feature point are determined; The second feature point whose depth information and confidence information satisfy the second set condition is determined as the fourth feature point; The third and fourth feature points that match the RGB features are determined as the first matching feature point pair.

7. The apparatus according to claim 6, characterized in that, The first set conditions include: The confidence level information is greater than or equal to the first set confidence level; and / or, The depth information is greater than or equal to the first set depth and less than or equal to the second set depth.

8. The apparatus according to claim 6, characterized in that, The second setting conditions include: The confidence level information is greater than or equal to the second set confidence level; and / or, The depth information is greater than or equal to the third set depth, and less than or equal to the fourth set depth.

9. The apparatus according to claim 6, characterized in that, The determining module is used for: If the similarity between the depth information of the third feature point and the depth information of the fourth feature point in the first matching feature point pair is less than or equal to the first set similarity, then this first matching feature point pair is determined as the second matching feature point pair. The first homography matrix is ​​determined based on multiple pairs of second matching feature points.

10. The apparatus according to claim 9, characterized in that, The determining module is used for: Based on the first depth image and the second depth image, determine the extrinsic parameter estimation information of the depth camera component; The second homography matrix is ​​determined based on multiple pairs of the second matching feature points; Based on the second homography matrix, the first color image, and the second color image, determine the disparity information; Based on the extrinsic parameter estimation information, the disparity information, and the intrinsic parameters of the color camera component, the depth estimation information corresponding to the second matching feature point is determined; If the similarity between the depth information of at least one feature point in the second matching feature point pair and the corresponding depth estimation information is greater than or equal to the second set similarity, then this second matching feature point pair is determined as the third matching feature point pair. The first homography matrix is ​​determined based on multiple pairs of third matching feature points.

11. A terminal, characterized in that, The terminal includes: processor; Memory used to store the processor's executable instructions; The processor is configured to perform the method as described in any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the terminal, the terminal is able to perform the method as described in any one of claims 1 to 5.

Images