Pose determination method and apparatus, computer-readable storage medium, and electronic device

Abstract

The present disclosure provides a pose determination method, a pose determination device, a computer readable storage medium and an electronic device, and relates to the technical field of computers. The pose determination method comprises: obtaining feature point information of a first image frame and feature point information of a second image frame, the first image frame and the second image frame being adjacent image frames; determining the number of matched feature point pairs in the first image frame and the second image frame by using the feature point information of the first image frame and the feature point information of the second image frame; if the number of matched feature point pairs is greater than a confidence threshold, determining the pose of a camera in combination with the feature point information of the first image frame and the feature point information of the second image frame; and if the number of matched feature point pairs is less than or equal to the confidence threshold, keeping the pose of the camera unchanged. The present disclosure can improve the stability of image tracking.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a pose determination method, a pose determination device, a computer-readable storage medium, and an electronic device. Background Technology

[0002] Augmented Reality (AR) is a technology that integrates virtual elements into the real world, enabling interaction between the virtual and real worlds. With the increasing processing power of terminal devices, AR technology can be widely applied in various fields such as gaming, education, healthcare, and transportation.

[0003] Image tracking, as an important component of augmented reality technology, has made great progress in recent years and has been widely applied in real-world augmented reality applications.

[0004] However, due to possible interference in the scene, the determined camera pose may be inaccurate, which in turn leads to poor stability of the image tracking algorithm. Summary of the Invention

[0005] This disclosure provides a pose determination method, a pose determination device, a computer-readable storage medium, and an electronic device, thereby overcoming, to at least a certain extent, the problem of poor algorithm stability in image tracking caused by inaccurate pose.

[0006] According to a first aspect of this disclosure, a pose determination method is provided, comprising: acquiring feature point information of a first image frame and feature point information of a second image frame, wherein the first image frame and the second image frame are adjacent image frames; using the feature point information of the first image frame and the feature point information of the second image frame to determine the number of matching feature point pairs in the first image frame and the second image frame; if the number of matching feature point pairs is greater than a confidence threshold, then combining the feature point information of the first image frame and the feature point information of the second image frame to determine the pose of a camera.

[0007] According to a second aspect of this disclosure, a pose determination apparatus is provided, comprising: an information acquisition module, configured to acquire feature point information of a first image frame and a second image frame, wherein the first image frame and the second image frame are adjacent image frames; a quantity determination module, configured to determine the number of matching feature point pairs in the first image frame and the second image frame using the feature point information of the first image frame and the second image frame; and a result determination module, configured to determine the pose of a camera by combining the feature point information of the first image frame and the feature point information of the second image frame if the number of matching feature point pairs is greater than a confidence threshold.

[0008] According to a third aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the pose determination method described above.

[0009] According to a fourth aspect of this disclosure, an electronic device is provided, including a processor; and a memory for storing one or more programs, which, when executed by the processor, cause the processor to implement the pose determination method described above.

[0010] In some embodiments of this disclosure, the first image frame and the second image frame are adjacent image frames. The number of matching feature point pairs between the first and second image frames is determined using feature point information from both frames. If the number of matching feature point pairs is greater than a confidence threshold, the camera pose is determined by combining the feature point information from both frames. If the number of matching feature point pairs is less than or equal to the confidence threshold, the camera pose remains unchanged. This disclosure constrains the pose determination process by using the number of matching feature point pairs between adjacent frames, avoiding the influence of environmental interference such as lighting or occlusion on pose determination, thereby improving the stability of image tracking.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0013] Figure 1 A schematic diagram illustrating an application scenario of the pose determination scheme according to an embodiment of the present disclosure is provided.

[0014] Figure 2 An exemplary schematic diagram illustrates the execution phase of the pose determination scheme according to an embodiment of the present disclosure;

[0015] Figure 3 A flowchart illustrating a pose determination method according to an exemplary embodiment of the present disclosure is shown schematically;

[0016] Figure 4 The illustration shows schematic diagrams of the various stages of the augmented reality algorithm involved in the exemplary scheme of this disclosure;

[0017] Figure 5A flowchart illustrating the entire method process of the augmented reality scheme according to an embodiment of the present disclosure is shown schematically;

[0018] Figure 6 The diagram illustrates a comparison of the tracking accuracy of the disclosed solution with that of some other technologies.

[0019] Figure 7 The diagram illustrates a comparison between the present disclosure and other techniques in terms of frame rate tracking.

[0020] Figure 8 The diagram illustrates a comparison of the present disclosure's solution with some other technologies in terms of recognition time.

[0021] Figure 9 A block diagram of a pose determination apparatus according to an exemplary embodiment of the present disclosure is shown schematically;

[0022] Figure 10 A block diagram of a pose determination apparatus according to another exemplary embodiment of the present disclosure is shown schematically;

[0023] Figure 11 A block diagram of a pose determination apparatus according to yet another exemplary embodiment of the present disclosure is shown schematically.

[0024] Figure 12 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown schematically. Detailed Implementation

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0026] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0027] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances. Furthermore, all the terms "first" and "second" used below are for distinction purposes only and should not be construed as limiting the scope of this disclosure.

[0028] In augmented reality technology, image tracking may suffer from issues such as tracking loss and latency due to various reasons, resulting in poor real-time performance, stability, and robustness. These reasons include external factors such as occlusion and changes in lighting, as well as combined camera motion factors such as scaling, tilting, movement, translation, and rotation.

[0029] The effectiveness of image tracking largely depends on the accuracy of camera pose. This disclosure provides a new pose determination scheme to improve the accuracy of pose determination and eliminate or at least reduce the impact of the above factors on image tracking.

[0030] It should be noted that the pose determination scheme of this disclosure can be implemented by augmented reality glasses. That is, the pose determination method of this disclosure can be executed by augmented reality glasses, and the pose determination device of this disclosure can be configured in augmented reality glasses.

[0031] However, as a lightweight technical solution, the pose determination scheme disclosed herein can be implemented by other terminal devices besides augmented reality glasses. Other terminal devices include smartphones, tablets, etc., and this disclosure does not limit them.

[0032] Figure 1 A schematic diagram illustrating an application scenario of the pose determination scheme according to an embodiment of this disclosure is provided. (See reference...) Figure 1 Users can turn on the augmented reality glasses 100 to scan the scene around them and obtain continuous image frames.

[0033] The process by which augmented reality glasses 100 determines its current pose may include: first, the augmented reality glasses 100 can acquire the current image frame, extract the feature point information of the current image frame using a feature extraction algorithm, and save the feature point information of the current image frame.

[0034] Next, the augmented reality glasses 100 can acquire the next image frame of the current image frame, extract the feature point information of the next image frame, and use the feature point information of the current image frame and the feature point information of the next image frame to determine the number of matching feature point pairs in the current image frame and the next image frame, which is used as the feature point pose confidence. In the embodiments of this disclosure, the feature point pose confidence can also be referred to as confidence or pose confidence.

[0035] Then, the augmented reality glasses 100 can compare the pose confidence of the feature point with a confidence threshold.

[0036] If the confidence score of the feature point pose is less than or equal to the confidence score threshold, it means that the accuracy of the current tracking algorithm output is insufficient. In this case, the camera pose can be kept unchanged, that is, the current pose of the augmented reality glasses 100 is determined to be the same as the pose determined last time.

[0037] If the confidence score of the feature point pose is greater than the confidence score threshold, it indicates that the accuracy of the current tracking algorithm's output is normal. In this case, the augmented reality glasses 100 can estimate the camera pose by combining the extracted feature point information. Specifically, the camera pose can be estimated by solving the homography matrix.

[0038] It should be understood that in the embodiments of this disclosure, when the number of feature point pairs matched in adjacent frames is less than or equal to the confidence threshold, the camera pose remains unchanged. In other words, the pose calculation process is not performed in this case, which can reduce the amount of computation and optimize system performance.

[0039] In the process of estimating camera pose using homography matrix, if the position of the feature point transformed by homography matrix in the current image frame differs significantly from the position of the feature point in the next image frame, then on the one hand, the camera pose remains unchanged and the next frame is processed; on the other hand, the confidence threshold can be adaptively adjusted according to the difference, and the updated confidence threshold can be applied to the processing of the next frame.

[0040] It should be noted that the process by which the augmented reality glasses 100 determines the current camera pose utilizes the current image frame and the next image frame. In this process, the data of the current image frame needs to be stored first, while waiting for the data of the next image frame.

[0041] In other embodiments of this disclosure, the augmented reality glasses 100 can also estimate the camera's pose when capturing the current image frame based on the previous image frame. This processing method does not require storing the data of the current image frame and waiting for the data of the next image frame; it determines the camera's pose in real time when capturing the current image frame.

[0042] Specifically, firstly, the augmented reality glasses 100 can use the feature point information of the current image frame and the feature point information of the previous image frame to determine the number of matching feature point pairs in the current image frame and the previous image frame, which serves as the feature point pose confidence score. Next, the augmented reality glasses 100 can compare this feature point pose confidence score with a confidence threshold. If the feature point pose threshold is less than or equal to the confidence threshold, the camera pose can remain unchanged. If the feature point pose threshold is greater than the confidence threshold, the camera pose can be determined by combining the feature point information of the current image frame and the feature point information of the previous image frame. This process may also include the adaptive adjustment of the confidence threshold described above, which will not be elaborated further.

[0043] After determining the camera pose, the augmented reality glasses 100 can also render virtual objects based on that camera pose. Thus, the user can see virtual objects in the real world through the lenses of the augmented reality glasses 100. For example... Figure 1 As shown, the ball can be, for example, a virtual object rendered by augmented reality glasses 100, which can only be seen by the user's eyes through augmented reality glasses 100.

[0044] During the rendering of virtual objects, the augmented reality glasses 100 can obtain information about the virtual objects from the server and then render and display them based on that information. Furthermore, the augmented reality glasses 100 can locally store information about the virtual objects, meaning that it can still render and display virtual objects even when the glasses are offline.

[0045] Figure 2 A schematic diagram illustrating the execution phase of the pose determination scheme according to an embodiment of this disclosure is provided. (See reference...) Figure 2 The first image frame and the second image frame are adjacent frames. The pose determination scheme of this embodiment is implemented based on the image data of the first image frame and the second image frame. That is, after the first image frame and the second image frame are obtained, the camera pose can be obtained based on the pose determination process of this embodiment.

[0046] Once the camera pose is obtained, other aspects of augmented reality technology can be further realized, such as image tracking, virtual object rendering, and virtual-real interaction.

[0047] The pose determination method of this embodiment will be described below using augmented reality glasses as an example.

[0048] Figure 3 A flowchart illustrating an exemplary embodiment of the pose determination method of this disclosure is shown schematically. (Reference) Figure 3 The pose determination method may include the following steps:

[0049] S32. Obtain feature point information of the first image frame and feature point information of the second image frame, wherein the first image frame and the second image frame are adjacent image frames.

[0050] In the embodiments of this disclosure, the first image frame and the second image frame can be any adjacent image frames in a video, which can be a video obtained by scanning the scene environment with augmented reality glasses. For example, the first image frame can be the nth frame in the video, and the second image frame can be the (n+1)th frame in the video. Another example is that the first image frame can be the (n-1)th frame in the video, and the second image frame can be the nth frame in the video. Furthermore, the first image frame and the second image frame can also be two consecutive image frames acquired through an algorithm or manual operation.

[0051] The first image frame can also be the current image frame captured by the augmented reality glasses. In this case, the second image frame is the next image frame or the previous image frame of the current image frame.

[0052] After acquiring a first image frame, augmented reality glasses can extract feature point information from the first image frame, and after acquiring a second image frame, they can extract feature point information from the second image frame. This disclosure does not restrict the order in which the first and second image frames are acquired, nor the order in which feature point information is extracted from the first and second image frames. For example, the process of extracting feature point information can be performed after acquiring the first and second image frames. Or, for another example, the feature point information of the first image frame can be extracted after acquiring the first image frame, and then the second image frame can be acquired.

[0053] The algorithm for extracting feature point information according to the embodiments of this disclosure will be described below.

[0054] Feature point extraction and description algorithms can be divided into absolute value-based algorithms and comparison-based algorithms. Absolute value-based algorithms typically construct histograms by quantizing image grayscale and gradients, while comparison-based algorithms generally use binary descriptions, generating binary sequences from point-to-point comparison results to construct descriptors.

[0055] The ORB (Oriented Fast and Rotated BRIEF) algorithm is a comparison-based algorithm with fast computation and matching speed, making it suitable for terminal devices such as augmented reality glasses. This disclosure embodiment can use the ORB algorithm to extract feature point information from image frames.

[0056] However, it should be understood that algorithms such as SIFT (Scale-Invariant Feature Transform), SURF (Speed ​​Up Robust Features), BRISK (Binary Robust Invariant Scalable Keypoints), and AKAZE (Accelerated-KAZE) can also be used to extract feature point information of image frames, and this disclosure does not limit this.

[0057] In an exemplary embodiment of this disclosure, the algorithm model used to extract feature point information of the first image frame is the same as the algorithm model used to extract feature point information of the second image frame.

[0058] In addition, some other embodiments of this disclosure can also identify image frames and determine whether to perform feature point extraction operations on two image frames and subsequent processing based on the identification results.

[0059] Taking the first image frame as an example, firstly, the augmented reality glasses can acquire the first image frame and extract its feature point information. Next, the augmented reality glasses can compare the feature point information of the first image frame with the feature point information of a sample image. If the feature point information of the first image frame matches the feature point information of the sample image, the feature point extraction operation of the first image frame and subsequent processing can be performed. If the feature point information of the first image frame does not match the feature point information of the sample image, no processing is performed on the first image frame; in this case, the camera pose remains unchanged.

[0060] The sample image is an image of the same scene as the first image frame. The algorithm model for extracting feature point information from the sample image is the same as that for extracting feature point information from the first image frame; for example, both use the ORB algorithm model to extract feature point information. In the target tracking scenario, if a predetermined object (e.g., a ball, animal, face, gesture, etc.) is to be tracked, the feature point information of the first image frame matches the feature point information of the sample image. That is, the feature point information can determine that both the first image frame and the sample image contain the predetermined object.

[0061] In addition, the sample images can be one or more pre-determined images. For example, the parameters in the feature extraction algorithm model can be trained using the sample images to obtain the final algorithm model for extracting feature point information of the first image frame and the second image frame. This disclosure does not limit this process.

[0062] S34. Using the feature point information of the first image frame and the feature point information of the second image frame, determine the number of matching feature point pairs in the first image frame and the second image frame.

[0063] By acquiring feature point information in step S32, the augmented reality glasses can obtain the feature point information in the first image frame and the second image frame. If feature point A in the first image frame can match feature point B in the second image frame, then feature point A and feature point B constitute a matched feature point pair. It can be understood that a matched feature point pair includes two feature points, one located in the first image frame and the other located in the second image frame.

[0064] First, augmented reality glasses can use the feature point information of the first image frame and the feature point information of the second image frame to obtain the similarity between the feature point information of the feature points in the first image frame and the feature points in the second image frame, and determine the candidate feature point pair set based on the similarity.

[0065] Similarity can be calculated by measuring distance, such as Hamming distance, Euclidean distance, or Mahalanobis distance.

[0066] Taking Hamming distance as an example, let Xi be the descriptor of a feature point in the first image frame and Yj be the descriptor of a feature point in the second image frame. Then, the Hamming distance between them can be expressed as follows:

[0067] dij=Xi⊕Yj

[0068] The smaller the Hamming distance (dij) value, the greater the similarity between descriptors; the larger the Hamming distance (dij) value, the smaller the similarity between descriptors.

[0069] Specifically, augmented reality glasses can obtain the similarity of feature point information between target feature points in the first image frame and each feature point in the second image frame. The target feature point can be any feature point in the first image frame. In other words, for each feature point in the first image frame, the similarity of feature point information between it and the feature points in the second image frame is calculated.

[0070] After calculating the similarity between the target feature point and each feature point in the second image frame, the augmented reality glasses can identify the feature point with the highest similarity (lowest distance) among the feature points in the second image frame, and form a candidate feature point pair with the target feature point.

[0071] For example, this process can use the kd-Tree (K-dimensional Tree, high-dimensional indexed tree data structure) algorithm to retrieve feature points and find the feature point pair with the highest similarity as candidate feature point pairs.

[0072] Then, the augmented reality glasses can filter out the matching feature point pairs in the first image frame and the second image frame from the set of candidate feature point pairs, and thus obtain the number of matching feature point pairs.

[0073] Specifically, augmented reality glasses can compare the similarity between candidate feature points in a set of candidate feature point pairs with a similarity threshold, and select feature point pairs with a similarity greater than the threshold from the set of candidate feature point pairs as the matching feature point pairs in the first image frame and the second image frame. This disclosure does not limit the specific value of the similarity threshold.

[0074] For example, a candidate feature point pair consists of feature point a and feature point b. If, after comparison, the similarity between feature point a and feature point b is less than a similarity threshold, then feature point a and feature point b cannot be considered a matching feature point pair between the first and second image frames. Another candidate feature point pair consists of feature point c and feature point d. If, after comparison, the similarity between feature point c and feature point d is greater than a similarity threshold, then feature point c and feature point d are determined to be a matching feature point pair between the first and second image frames.

[0075] After each matching feature point pair is determined, its quantity can be determined. In the exemplary embodiments of this disclosure, this quantity is the determined feature point pose confidence, also known as confidence or pose confidence.

[0076] After determining the number of matching feature point pairs, augmented reality glasses can compare that number with a confidence threshold.

[0077] S36. If the number of matched feature point pairs is greater than the confidence threshold, the camera pose is determined by combining the feature point information of the first image frame and the feature point information of the second image frame.

[0078] First, augmented reality glasses can determine the linear mapping relationship between the first image frame and the second image frame based on the feature point information of the first image frame and the feature point information of the second image frame, and obtain the homography matrix between the first image frame and the second image frame by combining the linear mapping relationship between the first image frame and the second image frame.

[0079] Specifically, the linear mapping relationship between the first and second image frames can be obtained based on the feature point matching relationship between the first and second image frames. Then, the rotation and translation matrices between the camera and the 2D mark can be solved using the homography matrix relationship of the plane and the initial estimate. Subsequently, the accurate homography matrix is ​​calculated using a nonlinear iterative optimization method with the reprojection error as the objective function.

[0080] Next, the augmented reality glasses can use this homography matrix to determine the camera's pose.

[0081] Specifically, the positional information of the feature points in the first image frame can be converted into intermediate positional information using the solved homography matrix. If the position of the feature points in the first image frame is denoted as {P0}, the intermediate positional information as {P0T}, and the homography matrix as Hn, then the intermediate positional information can be expressed as follows:

[0082] P0T=P0*Hn

[0083] After determining the intermediate position information, the average distance between feature points in the first image frame and feature points in the second image frame can be calculated based on the intermediate position information and the position information of feature points in the second image frame. Here, feature point {P1} in the second image frame is a feature point that matches feature point {P0} in the first image frame. That is, for each matching feature point pair, the distance between the intermediate position information and the corresponding feature point in the second image frame is calculated, and then the average of all distances is calculated.

[0084] In addition, for ease of subsequent comparison, the final average distance can be the normalized average distance.

[0085] After obtaining the average distance mentioned above, the camera pose can be determined by combining this average distance.

[0086] Augmented reality glasses can compare the average distance with a distance threshold. If the average distance is less than or equal to the distance threshold, the camera pose is determined using the homography matrix. Specifically, the homography matrix can be decomposed into rotation and translation matrices to determine the camera pose. The camera pose can be understood as the pose of the augmented reality glasses, which is the pose when the first or second image frame is captured.

[0087] If the average distance is greater than the distance threshold, the camera pose remains unchanged and is not updated; that is, the determined camera pose is the same as the previously determined pose.

[0088] Furthermore, if the average distance is greater than the distance threshold, the confidence threshold is considered low. In this case, the confidence threshold can be adjusted, specifically by increasing it. In other words, the confidence threshold disclosed herein can be a variable threshold.

[0089] For example, the confidence threshold can be adjusted based on the difference between the average distance and the distance threshold. The adjustment is made by increasing the confidence threshold, and the extent of the increase is related to the difference between the average distance and the distance threshold. Specifically, the increase can be positively correlated with this difference; that is, the larger the difference between the average distance and the distance threshold, the greater the increase in the confidence threshold; the smaller the difference, the smaller the increase in the confidence threshold.

[0090] S38. If the number of matched feature point pairs is less than or equal to the confidence threshold, then keep the camera pose unchanged.

[0091] If the number of matching feature point pairs in the first image frame and the second image frame determined in step S34 is less than or equal to the confidence threshold, it indicates that the current pose determination accuracy may be insufficient due to external factors or the camera's own condition. In this case, the augmented reality glasses keep the camera pose unchanged and then process the next frame.

[0092] Figure 4 A schematic diagram illustrating the stages involved in the augmented reality processing according to embodiments of this disclosure is shown. (Reference) Figure 4 The augmented reality processing can include an image recognition stage 41, a confidence determination stage 43, a pose determination stage 45, and a virtual object rendering stage 47.

[0093] In image recognition stage 41, the augmented reality glasses can determine whether the current image frame matches the sample image; that is, whether the current image frame meets the requirements specified by the sample image. This matching is performed using feature point information extracted using the same feature extraction algorithm model. If the current image frame matches the sample image, the process proceeds to the next stage. If the current image frame does not match the sample image, the process continues with the next frame.

[0094] In the confidence determination stage 43, the augmented reality glasses can determine the number of feature point pairs that match the current image frame and its neighboring frames, which is the confidence of the feature point pose.

[0095] In the pose determination stage 45, the augmented reality glasses can combine the comparison results of the feature point pose confidence and a confidence threshold to determine the pose of the camera when capturing the current image frame.

[0096] In the virtual object rendering stage 47, the augmented reality glasses can render the virtual object in the lenses based on the pose of the current image frame captured by the camera and the information of the virtual object, thereby realizing the presentation of the virtual object in the real world. This disclosure does not limit the type of virtual object; for example, virtual cartoon characters, virtual animals, virtual objects, virtual radiance, etc.

[0097] The following will refer to Figure 5 The entire process of implementing the augmented reality processing method based on augmented reality glasses according to the embodiments of this disclosure will be described.

[0098] In step S502, the current image frame is obtained.

[0099] In step S504, feature point information of the current image frame is extracted.

[0100] In step S506, based on the feature point information of the current image frame and the feature point information of the sample image, it is determined whether the current image frame matches the sample image. If they match, step S508 is executed; otherwise, the process returns to step S502, and an image frame is acquired at the next moment.

[0101] In step S508, the next image frame is acquired.

[0102] In step S510, feature point information of the next image frame is extracted.

[0103] In step S512, the number of matching feature point pairs in the current image frame and the next image frame is calculated using the feature point information of the current image frame and the feature point information of the next image frame. This number is used as the feature point pose confidence, or simply confidence.

[0104] In step S514, it is determined whether the confidence level is greater than the confidence threshold. If it is greater, then step S518 is executed; otherwise, step S516 is executed.

[0105] In step S516, the camera pose remains unchanged, and step S508 is executed again.

[0106] In step S518, the homography matrix is ​​solved by combining the feature point information of the current image frame and the feature point information of the next image frame.

[0107] In step S520, the intermediate position information is obtained using the homography matrix and the feature point information of the current image frame.

[0108] In step S522, the average distance to the corresponding feature point in the next image frame is calculated using the intermediate position information.

[0109] In step S524, it is determined whether the average distance is greater than a distance threshold. If it is greater, then step S516 is executed; otherwise, step S526 is executed.

[0110] In step S526, the camera pose can be updated using the homography matrix.

[0111] In step S528, the determined latest pose can be used to render virtual objects.

[0112] Additionally, step S530 can be executed if the average distance is greater than the distance threshold.

[0113] In step S530, the confidence threshold used in step S514 can be adaptively updated so that it can be applied to the processing of the next frame.

[0114] The following section describes the effectiveness of the image tracking process using the implementation scheme of this disclosure, based on test data.

[0115] The effectiveness of this disclosed solution can be evaluated using three metrics: recognition time, tracking accuracy, and tracking frame rate. Recognition time refers to the time required for a single image to be effectively recognized; tracking accuracy is the ratio of successfully tracked frames to the number of effective frames, i.e., the percentage of frames with tracking errors less than a threshold. This threshold is generally defined as 1% of the largest image pixel size or 20px, and the mean error is the ratio of the sum of reprojection errors of all successfully tracked frames to the number of successfully tracked frames; tracking frame rate refers to the number of effective frames tracked within 1 second.

[0116] Preparation of the test set:

[0117] The data acquisition tool was used to obtain YUV data of the same resolution as the RGB camera output of the augmented reality glasses and processed by the algorithm. The dataset included five recording scenarios: image angle rotation, image distance changes, image motion blur, image occlusion, and environmental brightness changes.

[0118] For obtaining algorithm prediction results and ground truth data based on the test set:

[0119] The ground truth data consists of the position of the target image in the image frame of all images in the manually labeled test set, that is, the pixel positions of the four corner points in the image plane coordinate system.

[0120] For the algorithm's prediction results: The test set simulates the real frame rate and is sent to the image tracking algorithm module (mokc method) to obtain the tracking state and rotation-translation (RT) matrix of each frame; the four corner points of the target image; then the four corner points of the target image are reprojected onto the video frame through the RT matrix; the positions (pixels) of the four corner points predicted by the algorithm in the image frame are obtained.

[0121] During this process, the recognition time, the time when the image is first sent to the algorithm, and the time difference between the output of the effective tracking status are statistically analyzed. The tracking frame rate is also determined: the number of effective tracking frames per second during tracking and recognition.

[0122] For algorithm result evaluation:

[0123] The sum of the distance errors between the estimated positions of the four corner points in each frame of the target map corresponding to the effective tracking state output by the algorithm and their ground truth values ​​in the dataset is used as the reprojection error.

[0124] The algorithm outputs the effective tracking status and the reprojection error, marking it as successful tracking if they meet the threshold; otherwise, tracking fails. Here, tracking rate = number of successfully tracked frames / number of effective frames, and mean error = sum of reprojection errors of all successfully tracked frames / number of successfully tracked frames.

[0125] For algorithm evaluation scenarios and test set design, the following dimensions are included:

[0126] Robustness comparison of image angle changes: Under the same and suitable conditions, simulating a scenario where a person wears AR glasses, the same image is captured by handheld AR glasses and rotated. The rotation covers the x, y, and z axes, forming a set of test images with different rotation angles;

[0127] Robustness comparison of image distance transformation: Under the same conditions and in a suitable scenario, a scenario of a person wearing AR glasses is simulated. The AR device is used to collect images and move them back and forth to form a test set of image distance transformations.

[0128] Image motion blur robustness comparison: Under the same conditions and in a suitable scenario, different Gaussian blurs are applied to the same image to form a set of images with different degrees of blur, simulating a set of motion blur test sets;

[0129] Image occlusion robustness comparison: Under the same conditions and in a suitable scenario, a scenario of a person wearing AR glasses is simulated. The AR glasses device is used to collect images of the same image with increasing occlusion degree to form a set of image test sets with different occlusion degrees.

[0130] Robustness comparison of ambient light and darkness transformation: Under the same and suitable conditions, a scenario is simulated where a person wears AR glasses. The AR glasses device is used to collect the same image, and the ambient light is increased to a certain extent to form a set of image test sets with different levels of light and darkness.

[0131] Figure 6 , Figure 7 and Figure 8Test results comparing tracking accuracy, tracking frame rate, and recognition time are shown. These results demonstrate that the image tracking process applying the exemplary scheme of this disclosure can effectively improve image tracking accuracy and enhance the robustness and stability of the algorithm without reducing the recognition frame rate or slightly increasing the recognition time.

[0132] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0133] Furthermore, this example embodiment also provides a pose determination device.

[0134] Figure 9 A block diagram of a pose determination apparatus according to an exemplary embodiment of the present disclosure is shown schematically. (Reference) Figure 9 The pose determination device 9 according to an exemplary embodiment of the present disclosure may include an information acquisition module 91, a quantity determination module 93 and a result determination module 95.

[0135] Specifically, the information acquisition module 91 can be used to acquire feature point information of the first image frame and the second image frame, wherein the first image frame and the second image frame are adjacent image frames; the quantity determination module 93 can be used to determine the number of matching feature point pairs in the first image frame and the second image frame using the feature point information of the first image frame and the second image frame; the result determination module 95 can be used to determine the camera pose by combining the feature point information of the first image frame and the feature point information of the second image frame if the number of matching feature point pairs is greater than the confidence threshold, and to keep the camera pose unchanged if the number of matching feature point pairs is less than or equal to the confidence threshold.

[0136] According to an exemplary embodiment of the present disclosure, the quantity determination module 93 can be configured to perform: using feature point information of the first image frame and feature point information of the second image frame, obtaining the similarity of feature point information between feature points in the first image frame and feature points in the second image frame, and determining a set of candidate feature point pairs based on the similarity; filtering out matching feature point pairs in the first image frame and the second image frame from the set of candidate feature point pairs, so as to determine the quantity of matching feature point pairs.

[0137] According to an exemplary embodiment of the present disclosure, the quantity determination module 93 can be configured to perform: obtaining the similarity of feature point information between the target feature point in the first image frame and each feature point in the second image frame; wherein the target feature point is any feature point in the first image frame; and determining the feature point with the largest similarity result among the feature points in the second image frame, so as to form a candidate feature point pair with the target feature point.

[0138] According to an exemplary embodiment of the present disclosure, the quantity determination module 93 may be configured to perform: comparing the similarity between each candidate feature point in the candidate feature point pair set with a similarity threshold; and selecting feature point pairs with a similarity greater than the similarity threshold from the candidate feature point pair set as the matching feature point pairs in the first image frame and the second image frame.

[0139] According to exemplary embodiments of this disclosure, reference is made to Figure 10 Compared to the pose determination device 9, the pose determination device 10 may also include an image recognition module 101.

[0140] Specifically, the image recognition module 101 can be configured to perform: acquiring a first image frame and extracting feature point information from the first image frame; comparing the feature point information of the first image frame with the feature point information of a sample image; and, if the feature point information of the first image frame matches the feature point information of the sample image, performing a process of determining the number of matching feature point pairs in the first image frame and the second image frame; wherein the model for extracting feature point information from the first image frame is the same as the model for extracting feature point information from the sample image.

[0141] According to an exemplary embodiment of this disclosure, the process by which the result determination module 95 determines the camera pose by combining the feature point information of the first image frame and the feature point information of the second image frame can be configured to perform: determining a linear mapping relationship between the first image frame and the second image frame based on the feature point information of the first image frame and the feature point information of the second image frame; obtaining a homography matrix between the first image frame and the second image frame by combining the linear mapping relationship between the first image frame and the second image frame; and determining the camera pose based on the homography matrix.

[0142] According to an exemplary embodiment of the present disclosure, the process by which the result determination module 95 determines the camera pose based on the homography matrix can be configured to perform: using the homography matrix, converting the position information of feature points in the first image frame into intermediate position information; obtaining the average distance between feature points in the first image frame and feature points in the second image frame based on the intermediate position information and the position information of feature points in the second image frame; and determining the camera pose by combining the average distance.

[0143] According to an exemplary embodiment of the present disclosure, the process by which the result determination module 95 determines the camera pose in combination with the average distance can be configured to perform: comparing the average distance with a distance threshold; if the average distance is less than or equal to the distance threshold, determining the camera pose using a homography matrix; if the average distance is greater than the distance threshold, keeping the camera pose unchanged.

[0144] According to an exemplary embodiment of the present disclosure, the process by which the result determination module 95 determines the camera pose using the homography matrix can be configured to perform: decomposing the homography matrix to obtain a rotation matrix and a translation matrix in order to determine the camera pose.

[0145] According to exemplary embodiments of this disclosure, reference is made to Figure 11 Compared to the pose determination device 9, the pose determination device 11 may also include a threshold update module 111.

[0146] Specifically, the threshold update module 111 can be configured to increase the confidence threshold when the average distance is greater than the distance threshold. Specifically, the threshold update module 111 can adjust the confidence threshold based on the difference between the average distance and the distance threshold.

[0147] Since the various functional modules of the pose determination device in this embodiment are the same as those in the above-described method embodiment, they will not be described again here.

[0148] Figure 12 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown.

[0149] The electronic device (such as augmented reality glasses) of the exemplary embodiments of this disclosure can be configured as follows: Figure 12 In the form of. It should be noted that, Figure 12 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0150] The electronic device disclosed herein includes at least a processor and a memory, the memory being used to store one or more programs, which, when executed by the processor, enable the processor to implement the pose determination method of the exemplary embodiments of this disclosure.

[0151] Specifically, such as Figure 12 As shown, the electronic device 12 may include a processor 121, a memory 122, an input-output unit 123, and a communication unit 124. The input-output unit 123 may include at least one display component 1231, an optical component 1232, a sensor 1233, and an audio component 1234.

[0152] It is understood that the structures illustrated in the embodiments of this disclosure do not constitute a specific limitation on the electronic device 12. In other embodiments of this disclosure, the electronic device 12 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0153] Processor 121 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. Additionally, processor 121 may include memory for storing instructions and data.

[0154] The input-output unit 123 can be used to acquire data from or send data to the outside.

[0155] At least one display component 1231 can be used to display images, typically referring to a displayable lens.

[0156] Optical component 1232 can be used to form an optical system for presenting an image to a user, and may include static components such as waveguides, static optical couplers, and fixed lenses. Additionally, it may include adjustable optical components, such as adjustable resonators, tunable lenses, dynamically adjustable couplers, adjustable liquid crystal holographic couplers, etc. The device structures included in optical component 1232 can be arranged in series, and this disclosure does not limit their combination.

[0157] In addition to the depth sensor described in this disclosure, sensor 1233 may also include, for example, an infrared sensor, a temperature sensor, an acceleration sensor, etc. Furthermore, this disclosure does not limit the placement of the sensor; for example, the depth sensor can be placed on the frame of eyeglasses.

[0158] The audio component 1234 may include a microphone for acquiring voice commands and other audio inputs, and a speaker for providing audio outputs.

[0159] The communication unit 124 may include a wired and / or wireless communication module for transmitting signals with external devices.

[0160] This disclosure also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device.

[0161] Computer-readable storage media can be, for example—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0162] A computer-readable storage medium can be sent, propagated, or transmitted for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0163] A computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to perform the methods described in the embodiments of this disclosure.

[0164] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0165] The units described in the embodiments of this disclosure can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the unit itself.

[0166] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0167] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0168] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0169] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0170] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A pose determination method, characterized in that, include: Obtain feature point information of the first image frame and feature point information of the second image frame, wherein the first image frame and the second image frame are adjacent image frames; Using the feature point information of the first image frame and the feature point information of the second image frame, determine the number of matching feature point pairs in the first image frame and the second image frame; If the number of matched feature point pairs is greater than the confidence threshold, then the linear mapping relationship between the first image frame and the second image frame is determined based on the feature point information of the first image frame and the feature point information of the second image frame. By combining the linear mapping relationship between the first image frame and the second image frame, the homography matrix between the first image frame and the second image frame is obtained; Using the homography matrix, the position information of the feature points in the first image frame is converted into intermediate position information; Based on the intermediate position information and the position information of the feature points in the second image frame, the average distance between the feature points in the first image frame and the feature points in the second image frame is obtained; combined with the average distance, the pose of the camera is determined.

2. The pose determination method according to claim 1, characterized in that, Using the feature point information of the first image frame and the feature point information of the second image frame, determining the number of matching feature point pairs in the first image frame and the second image frame includes: Using the feature point information of the first image frame and the feature point information of the second image frame, the similarity between the feature point information of the feature points in the first image frame and the feature points in the second image frame is obtained, and a set of candidate feature point pairs is determined based on the similarity. The number of matching feature point pairs is determined by filtering the candidate feature point pair set to select matching feature point pairs between the first image frame and the second image frame.

3. The pose determination method according to claim 2, characterized in that, Obtaining the similarity of feature point information between feature points in the first image frame and feature points in the second image frame, and determining a set of candidate feature point pairs based on the similarity, including: The similarity of feature point information between the target feature point in the first image frame and each feature point in the second image frame is obtained respectively; wherein, the target feature point is any feature point in the first image frame; The feature point with the highest similarity among all feature points in the second image frame is determined to form a candidate feature point pair with the target feature point.

4. The pose determination method according to claim 2, characterized in that, Selecting matching feature point pairs between the first image frame and the second image frame from the candidate feature point pair set includes: The similarity between each candidate feature point in the candidate feature point pair set is compared with a similarity threshold. Feature point pairs with a similarity greater than the similarity threshold are selected from the candidate feature point pair set and used as the matching feature point pairs between the first image frame and the second image frame.

5. The pose determination method according to claim 1, characterized in that, The pose determination method further includes: Acquire the first image frame and extract the feature point information of the first image frame; The feature point information of the first image frame is compared with the feature point information of the sample image; If the feature point information of the first image frame matches the feature point information of the sample image, the process of determining the number of matching feature point pairs in the first image frame and the second image frame is executed. The model for extracting feature point information from the first image frame is the same as the model for extracting feature point information from the sample image.

6. The pose determination method according to claim 1, characterized in that, Determining the camera pose based on the average distance includes: The average distance is compared with a distance threshold. If the average distance is less than or equal to the distance threshold, the pose of the camera is determined using the homography matrix; If the average distance is greater than the distance threshold, the camera pose remains unchanged.

7. The pose determination method according to claim 6, characterized in that, Determining the camera pose using the homography matrix includes: The homography matrix is ​​decomposed to obtain the rotation matrix and translation matrix, thereby determining the pose of the camera.

8. The pose determination method according to claim 6, characterized in that, The pose determination method further includes: If the average distance is greater than the distance threshold, the confidence threshold is increased.

9. The pose determination method according to claim 1, characterized in that, The pose determination method further includes: If the number of matched feature point pairs is less than or equal to the confidence threshold, the camera pose remains unchanged.

10. A pose determination device, characterized in that, include: The information acquisition module is used to acquire feature point information of a first image frame and feature point information of a second image frame, wherein the first image frame and the second image frame are adjacent image frames; The quantity determination module is used to determine the number of matching feature point pairs in the first image frame and the second image frame using the feature point information of the first image frame and the feature point information of the second image frame. The result determination module is used to determine the linear mapping relationship between the first image frame and the second image frame based on the feature point information of the first image frame and the feature point information of the second image frame if the number of matched feature point pairs is greater than the confidence threshold. By combining the linear mapping relationship between the first image frame and the second image frame, the homography matrix between the first image frame and the second image frame is obtained; Using the homography matrix, the position information of the feature points in the first image frame is converted into intermediate position information; Based on the intermediate position information and the position information of the feature points in the second image frame, the average distance between the feature points in the first image frame and the feature points in the second image frame is obtained; combined with the average distance, the pose of the camera is determined.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the pose determination method as described in any one of claims 1 to 9.

12. An electronic device, characterized in that, include: processor; A memory for storing one or more programs, which, when executed by the processor, cause the processor to implement the pose determination method as described in any one of claims 1 to 9.

Images