Trajectory point position information sequence generation method and augmented reality display device

Abstract

Embodiments of the present disclosure disclose a trajectory point pose information sequence generation method and an augmented reality display device. A specific implementation of the method comprises: collecting an image frame sequence and an inertial measurement data sequence corresponding to a preset scene containing three-dimensional feature points; generating an initial trajectory point pose information sequence based on the image frame sequence and the inertial measurement data sequence; performing a sliding window loop optimization process on the image frame sequence to generate at least one optimized trajectory point pose information corresponding to at least one image frame in the image frame sequence; and performing an update process on the initial trajectory point pose information sequence based on the at least one optimized trajectory point pose information to obtain a trajectory point pose information sequence. The implementation reduces the trajectory drift degree of the device trajectory displayed based on the trajectory point pose sequence.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, specifically to a method for generating trajectory point pose information sequences and an augmented reality display device. Background Technology

[0002] Trajectory point pose information sequence generation is a technique for generating trajectory point pose sequences of augmented reality display devices (e.g., AR devices) during movement. Currently, the common approach to generating trajectory point pose sequences of augmented reality display devices during movement is to use the VIO algorithm to estimate the pose sequence of the augmented reality display device during movement using environmental and motion data.

[0003] However, the inventors have discovered that when generating the trajectory point pose sequence of an augmented reality display device during movement using the above method, the following technical problems often arise: VIO relies on data correlation between adjacent frames for pose estimation, but sensor noise, visual feature matching errors, and IMU (Inertial Measurement Unit) drift accumulate over time, causing the trajectory deviation to gradually increase. This results in poor accuracy of the trajectory point pose sequence generated by the augmented reality display device during movement, leading to a high degree of trajectory drift in the device trajectory displayed based on the trajectory point pose sequence.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure propose a method for generating trajectory point pose information sequences and an augmented reality display device to solve one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a method for generating a trajectory point pose information sequence. The method includes: for a preset scene containing three-dimensional feature points, acquiring an image frame sequence and an inertial measurement data sequence corresponding to the preset scene; generating an initial trajectory point pose information sequence based on the image frame sequence and the inertial measurement data sequence; performing sliding window loop closure optimization processing on the image frame sequence to generate at least one optimized trajectory point pose information corresponding to at least one image frame in the image frame sequence; and updating the initial trajectory point pose information sequence based on the at least one optimized trajectory point pose information to obtain the trajectory point pose information sequence.

[0008] Secondly, some embodiments of this disclosure provide an augmented reality display device, which is an augmented reality head-mounted display device or an augmented reality handheld display device, and the augmented reality display device includes: one or more processors; a camera device for acquiring image frames corresponding to the preset scene at a preset acquisition frequency, and in response to determining that the acquired image frames meet preset labeling conditions, marking the acquired image frames as loop closure candidate image frames to obtain an image frame sequence containing at least one loop closure candidate image frame; an inertial measurement component for acquiring an inertial measurement data sequence; a storage device for storing one or more programs; a display module for imaging to display the trajectory corresponding to the trajectory point pose information sequence; when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect.

[0009] The above embodiments of this disclosure have the following beneficial effects: the trajectory point pose information sequence generation method of some embodiments of this disclosure improves the accuracy of the generated trajectory point pose sequence and reduces the trajectory drift of the device trajectory displayed based on the trajectory point pose sequence. Specifically, the reason for the poor accuracy of the generated trajectory point pose sequence and the high degree of trajectory drift of the device trajectory displayed based on the trajectory point pose sequence is that VIO relies on the data association between adjacent frames for pose estimation, but sensor noise, visual feature matching error, and IMU (Inertial Measurement Unit) drift accumulate over time, causing the trajectory deviation to gradually increase, which in turn leads to poor accuracy of the trajectory point pose sequence generated by the augmented reality display device during movement, resulting in a high degree of trajectory drift of the device trajectory displayed based on the trajectory point pose sequence. Based on this, the trajectory point pose information sequence generation method of some embodiments of this disclosure first collects the image frame sequence and inertial measurement data sequence corresponding to the preset scene containing three-dimensional feature points. Thus, the image frame sequence and inertial measurement data sequence collected by the augmented reality display device during movement can be obtained. Next, based on the aforementioned image frame sequence and inertial measurement data sequence, an initial trajectory point pose information sequence is generated. This generates an initial trajectory point pose information sequence representing the trajectory of the augmented reality display device during movement. Then, a sliding window loop closure optimization process is performed on the aforementioned image frame sequence to generate at least one optimized trajectory point pose information corresponding to at least one image frame in the sequence. This allows the use of sliding window loop closure optimization to generate at least one optimized trajectory point pose information for updating the initial trajectory point pose information sequence. Finally, based on the at least one optimized trajectory point pose information, the initial trajectory point pose information sequence is updated to obtain the trajectory point pose information sequence. Thus, the initial trajectory point pose information sequence can be optimized using at least one optimized trajectory point pose information, reducing accumulated errors in the initial trajectory point pose information sequence, reducing the difference between the trajectory represented by the generated trajectory point pose information sequence and the actual trajectory of the augmented reality display device during movement, and reducing the degree of trajectory drift of the device trajectory displayed based on the trajectory point pose sequence. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0011] Figure 1 This is a flowchart of some embodiments of the trajectory point pose information sequence generation method applying some embodiments of this disclosure; Figure 2 These are schematic diagrams of structures from other embodiments of the trajectory point pose information sequence generation method according to this disclosure; Figure 3 This is an internal test diagram based on the trajectory point pose information sequence generation method disclosed herein; Figure 4 This is a schematic diagram of the hardware structure of an augmented reality display device according to this disclosure. Detailed Implementation

[0012] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0013] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0014] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0015] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0016] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0017] Figure 1 A flowchart 100 is shown, illustrating some embodiments of the trajectory point pose information sequence generation method according to the present disclosure. The trajectory point pose information sequence generation method includes the following steps: This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] Step 101: For a preset scene containing three-dimensional feature points, collect the image frame sequence and inertial measurement data sequence corresponding to the preset scene.

[0019] In some embodiments, the execution subject (e.g., an AR device) of the trajectory point pose information sequence generation method can acquire an image frame sequence and an inertial measurement data sequence corresponding to a preset scene containing three-dimensional feature points. The execution subject can be an augmented reality display device (e.g., an AR device). The augmented reality display device can be an augmented reality head-mounted display device (e.g., a user-worn AR device) or an augmented reality handheld display device (e.g., a handheld AR device). The image frame sequence can be an image sequence acquired by the augmented reality display device during movement within the preset scene. Each inertial measurement data point in the inertial measurement data sequence corresponds to a specific image frame in the image frame sequence. The execution subject can acquire its own inertial measurement data while acquiring the image frames. The inertial measurement data includes acceleration and angular velocity.

[0020] In some optional implementations of certain embodiments, the aforementioned execution entity may acquire image frame sequences and inertial measurement data sequences corresponding to the aforementioned preset scenario through the following steps: The first step involves acquiring image frames corresponding to the aforementioned preset scene using a camera device at a preset acquisition frequency. In response to determining that the acquired image frames meet preset annotation conditions, the acquired image frames are marked as loop closure candidate image frames, resulting in an image frame sequence containing at least one loop closure candidate image frame. The preset annotation conditions may include, but are not limited to, one of the following: the image frame is observed more than a preset number of times (e.g., the number of times the image frame successfully matches a historical frame is greater than 3); the similarity of environmental features (such as planes, lines, key points) between the image frame and historical frames is greater than a preset value (e.g., the similarity between wall features in the image frame and wall features in a preset historical frame is greater than a preset value); or there is corresponding user interaction information (e.g., user interaction information where the image frame becomes an anchor point). The aforementioned camera device can be a camera on an augmented reality display device.

[0021] The second step involves acquiring an inertial measurement data sequence using the aforementioned inertial measurement unit. Each image frame in the image frame sequence corresponds to one inertial measurement data point in the inertial measurement data sequence. The image frames and corresponding inertial measurement data points are acquired at the same time. The inertial measurement unit includes an accelerometer and a gyroscope.

[0022] Step 102: Generate an initial trajectory point pose information sequence based on the image frame sequence and the inertial measurement data sequence.

[0023] In some embodiments, the execution entity can generate an initial trajectory point pose information sequence based on the image frame sequence and the inertial measurement data sequence. In practice, the execution entity can generate the initial trajectory point pose information sequence based on the image frame sequence and the inertial measurement data sequence using visual inertial odometry (VIO) technology. The initial trajectory point pose information sequence can characterize the motion trajectory and attitude of the augmented reality display device in three-dimensional space. Each initial trajectory point pose information in the initial trajectory point pose information sequence corresponds to one image frame in the image frame sequence. The initial trajectory point pose information can represent the position and attitude of the augmented reality display device in three-dimensional space; for example, the initial trajectory point pose information can be "position (m): (0, 0, 0), attitude (quaternion): (0.0, 0.0, 0.0, 1.0)". Optionally, in practice, the execution entity can also generate the initial trajectory point pose information sequence based on the image frame sequence and the inertial measurement data sequence using tight coupling and loose coupling methods.

[0024] Step 103: Perform sliding window loop closure optimization on the image frame sequence to generate at least one optimized trajectory point pose information corresponding to at least one image frame in the image frame sequence.

[0025] In some embodiments, the execution entity can perform sliding window loop closure optimization processing on the image frame sequence to generate at least one optimized trajectory point pose information corresponding to at least one image frame in the image frame sequence. In practice, the execution entity can use a preset image as the loop closure image. Then, the execution entity can input at least one image frame from the image frame sequence into the sliding window. Next, for each image frame in the sliding window, the execution entity can calculate the relative pose between the loop closure image and the image frame as the loop closure constraint through feature matching and geometric verification. Simultaneously, to minimize the loop closure constraint, visual reprojection error, and IMU pre-integration error, the execution entity uses a nonlinear optimization algorithm (such as Ceres Solver or g2o) to solve for the optimized pose information corresponding to the image frame as the optimized trajectory point pose information. The optimized trajectory point pose information can represent the position and orientation of the augmented reality display device acquiring the image frames in three-dimensional space. The preset image can be a pre-selected scene image containing the aforementioned three-dimensional feature points.

[0026] In some optional implementations of certain embodiments, the execution entity may perform sliding window loop closure optimization processing on the image frame sequence through the following steps to generate at least one optimized trajectory point pose information corresponding to at least one image frame in the image frame sequence: The first step is to divide the aforementioned image frame sequence using a sliding window method to obtain an initial image frame queue sequence. The first initial image frame queue in this sequence includes at least one loopback candidate image frame. In practice, the execution entity can divide the initial image frame queue sequence using a sliding window with a first preset size and a second preset step size, and input each initial image frame grouped into the queue to obtain the initial image frame queue sequence. For example, the image frame sequence could be "{image frame 1, image frame 2, image frame 3, image frame 4, image frame 5, image frame 6, image frame 7, image frame 8}", with a sliding window size of 3 and a sliding step size of 1. The initial image frame queue sequence can be "{{image frame 1, image frame 2, image frame 3}, {image frame 2, image frame 3, image frame 4}, {image frame 3, image frame 4, image frame 5}, {image frame 4, image frame 5, image frame 6}, {image frame 5, image frame 6, image frame 7}, {image frame 6, image frame 7, image frame 8}}.

[0027] The second step is to perform the following steps for each initial image frame queue in the above initial image frame queue sequence: The first sub-step is to determine the last initial image frame in the initial image frame queue as the reference image frame.

[0028] The second sub-step involves determining at least one loopback candidate image frame in the above image frame sequence that is located before the above reference image frame as a loopback candidate image frame sequence.

[0029] The third sub-step is to generate optimized trajectory point pose information corresponding to the last initial image frame in the initial image frame queue based on the initial image frame queue, in response to determining that there are loop closure candidate image frames in the initial image frame queue.

[0030] Therefore, by using a sliding window and a sequence of candidate image frames with loop closures, the pose drift of the reference frame can be corrected by loop closure constraints within the local window, generating pose information of at least one optimized trajectory point corresponding to at least one image frame in the image frame sequence.

[0031] In the process of adopting technical solutions to address the problems mentioned in the background section, the following issues often arise: In the process of generating the optimized trajectory point pose information corresponding to the last initial image frame in the initial image frame queue, based on the loop closure candidate image frames in the initial image frame queue and the initial image frame queue, the system will still attempt feature matching (such as SIFT, ORB) and geometric verification (such as RANSAC) when the loop closure candidate image frames in the initial image frame queue do not co-exist with the last initial image frame. These operations consume CPU and GPU resources, resulting in a waste of computing power. Simultaneously, when there are multiple loop closure candidate image frames in the initial image frame queue, a random loop closure candidate image is selected to generate the optimized trajectory point pose information. The randomly selected loop closure image frame may only have a small co-existing region with the last initial image frame, or even not overlap at all. During the generation of the optimized trajectory point pose information, the optimization algorithm requires more iterations to barely converge, resulting in a waste of computing power.

[0032] Faced with the above-mentioned technical problems, the inventors decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity may generate optimized trajectory point pose information corresponding to the last initial image frame in the initial image frame queue based on the initial image frame queue through the following steps: The first step involves determining that the number of loop closure candidate image frames included in the initial image frame queue is one and that the included loop closure candidate image frames co-occur with the last initial image frame, thus identifying the loop closure candidate image frames included in the initial image frame queue as reference loop closure image frames. In practice, the executing entity can use a feature point extraction algorithm (e.g., SIFT algorithm) to perform feature point extraction processing on the loop closure candidate image frames and the last initial image frame respectively, obtaining a first feature point extraction information set corresponding to the loop closure candidate image frames and a second feature point extraction information set corresponding to the last initial image frame. Then, the executing entity can input the first feature point extraction information set and the second feature point extraction information set into a pre-trained feature point matching model (e.g., SuperGlue model, LoFTR model) to obtain each feature point extraction information matching pair. In response to determining that the number of feature point extraction information matching pairs in each of the above feature point extraction information matching pairs is greater than a preset value, the executing entity can determine that the loop closure candidate image frames co-occur with the last initial image frame. Specifically, each first feature point extraction information in the aforementioned first feature point extraction information set can represent a feature point in the loop closure candidate image frame. This first feature point extraction information can be represented using a feature descriptor. Similarly, each second feature point extraction information in the aforementioned second feature point extraction information set can represent a feature point in the last initial image frame. This second feature point extraction information can also be represented using a feature descriptor.

[0033] The second step is to determine each loopback candidate image frame included in the initial image frame queue as a loopback candidate image frame to be screened in response to the determination that the number of loopback candidate image frames included in the initial image frame queue is greater than one.

[0034] The third step is to determine the last initial image frame as the target image frame.

[0035] The fourth step involves selecting one candidate image frame from the pool of candidate loop closure images that meets the preset maximum common viewing area selection criteria, and using it as the reference loop closure image frame. The preset maximum common viewing area selection criteria can be defined as having the largest common viewing area with the target image frame. The size of the common viewing area with the target image frame can be represented by the number of feature point pairs matching between the candidate loop closure image frame and the target image frame. Optionally, the size of the common viewing area with the target image frame can also be represented by the number of pixels within the overlapping area of ​​the candidate loop closure image frame and the target image frame. The executing entity can find the overlapping area between the candidate loop closure image frame and the target image frame through image registration (such as homography transformation or fundamental matrix).

[0036] The fifth step involves arranging the inertial measurement data corresponding to each initial image frame in the inertial measurement data sequence to obtain the target inertial measurement data sequence. In practice, firstly, the executing entity can identify each inertial measurement data corresponding to each initial image frame in the inertial measurement data sequence as a target inertial measurement data. Then, the executing entity can arrange the target inertial measurement data according to their sequential order within the inertial measurement data sequence to obtain the target inertial measurement data sequence.

[0037] Step 6: Based on the target inertial measurement data sequence, the initial image frame queue, the reference loopback image frame, and the target image frame, generate optimization objective function information. This optimization objective function information can be a mathematical expression that integrates IMU constraint information, visual constraint information (reprojection error information), and loopback error terms. This expression describes the combined influence of all constraints on the trajectory point pose (position and attitude). In practice, the execution entity can generate IMU constraint information based on the target inertial measurement data sequence and preset inertial measurement unit noise parameters. Then, based on the initial image frame queue, the 3D coordinates of the 3D feature points, and the camera parameter information of the camera device, reprojection error information is generated as visual constraint information. Next, based on the reference loopback image frame and the target image frame, a loopback error term is generated. Finally, based on the IMU constraint information, the visual constraint information, and the loopback error term, the optimization objective function information is generated.

[0038] The seventh step involves iteratively optimizing the objective function corresponding to the aforementioned objective function information to generate optimized trajectory point pose information corresponding to the target image frame. In practice, the executing entity can use nonlinear optimization algorithms such as the Gauss-Newton method or the Levenberg-Marquardt method to iteratively optimize the objective function to obtain optimized trajectory point pose information corresponding to the target image frame. This optimized trajectory point pose information can represent the position and orientation of the augmented reality display device acquiring the target image frame in three-dimensional space.

[0039] The above-described technical solution and its related content, as an inventive point of this disclosure, solve the technical problem of "waste of computer computing resources." Factors leading to this waste often include: During the process of generating optimized trajectory point pose information corresponding to the last initial image frame in the initial image frame queue, based on the loop closure candidate image frames in the initial image frame queue and the initial image frame queue itself, when the loop closure candidate image frames in the initial image frame queue do not co-exist with the last initial image frame, the system still attempts to perform feature matching (such as SIFT, ORB) and geometric verification (such as RANSAC). These operations consume CPU and GPU resources, resulting in a waste of computer computing resources. Furthermore, when there are multiple loop closure candidate image frames in the initial image frame queue, randomly selecting one to generate optimized trajectory point pose information may result in the randomly selected loop closure image frame having only a small co-existing area with the last initial image frame, or even no overlap at all. This requires the optimization algorithm to undergo more iterations to barely converge during the generation of optimized trajectory point pose information, further wasting computer computing resources. Solving these factors can reduce the waste of computer computing resources. To achieve this effect, firstly, in response to determining that the number of loop closure candidate image frames included in the initial image frame queue is one and that the included loop closure candidate image frames co-view with the last initial image frame, the loop closure candidate image frames included in the initial image frame queue are determined as reference loop closure image frames. Thus, even if the initial image frame queue includes a loop closure candidate image frame that co-views with the initial image frame, the loop closure candidate image frame can be determined as a reference loop closure image frame for generating optimized objective function information and optimized trajectory point pose information. Next, in response to determining that the number of loop closure candidate image frames included in the initial image frame queue is greater than one, each loop closure candidate image frame included in the initial image frame queue is determined as a loop closure candidate image frame to be filtered. Thus, each loop closure candidate image frame to be filtered included in the initial image frame queue can be determined. Then, the last initial image frame is determined as the target image frame. Finally, one loop closure candidate image frame among the various loop closure candidate image frames to be filtered that meets the preset maximum co-view area filtering condition is determined as the maximum co-view image and used as the reference loop closure image frame. Therefore, one candidate loop closure image with the largest shared viewing area with the target image frame can be selected from each candidate loop closure image frame to be screened as a reference loop closure image frame for generating the optimized objective function information and the optimized trajectory point pose information. Then, the inertial measurement data corresponding to each initial image frame included in the initial image frame queue in the above inertial measurement data sequence are arranged to obtain the target inertial measurement data sequence. Thus, the target inertial measurement data sequence used to generate the optimized objective function information can be obtained.Next, based on the target inertial measurement data sequence, the initial image frame queue, the reference loop closure image frame, and the target image frame, optimized objective function information is generated. Then, the objective function corresponding to the above optimized objective function information is iteratively optimized to generate optimized trajectory point pose information corresponding to the target image frame. Thus, iterative optimization of the objective function corresponding to the objective function information can generate optimized trajectory point pose information. Also, when generating the optimized trajectory point pose information corresponding to the target image frame based on the initial image frame queue, if the initial image frame queue contains only one loop closure candidate image frame that shares a common view with the last initial image frame, it is determined as the reference loop closure image frame and used to generate the optimized objective function information and optimized trajectory point pose information. Alternatively, if the initial image frame queue contains multiple loop closure candidate image frames, the last initial image frame is first determined as the target image frame, and then one loop closure candidate image frame with the largest common view area with the target image frame is selected from all the loop closure candidate image frames to be screened as the reference loop closure image frame, ensuring that the selected reference loop closure image frame has a large common view area with the target image frame, rather than being randomly selected. Furthermore, when generating optimized trajectory point pose information based on the selected reference loop closure image frame, unnecessary generation operations are avoided when there is no common view. At the same time, when there are multiple loop closure candidate image frames, the image frame with the largest common view area is selected as the reference loop closure image frame, which ensures that the objective function can converge faster when generating optimized trajectory point pose information, reducing the waste of computer computing resources.

[0040] In some optional implementations of certain embodiments, after determining at least one loopback candidate image frame preceding the reference image frame in the above image frame sequence as a loopback candidate image frame sequence, the execution entity may further perform the following steps: First, in response to determining that no loop closure candidate image frame exists in the above-mentioned initial image frame queue, the following steps are performed: The first sub-step involves replacing the loopback candidate image frames in the aforementioned loopback candidate image frame sequence that meet the preset filtering criteria with the first initial image frame in the aforementioned initial image frame queue, thereby updating the aforementioned initial image frame queue. The preset filtering criteria can be that the shared viewing area with the last initial image frame in the initial image frame queue is the largest.

[0041] The second sub-step involves generating the optimized trajectory point pose information corresponding to the last initial image frame in the updated initial image frame queue, based on the updated initial image frame queue.

[0042] Therefore, in cases where there are no loop closure candidate image frames in the initial image frame queue, the initial image frame queue can be updated by replacing the loop closure candidate image frames, and the optimized trajectory point pose information can be generated based on the updated queue, thereby enhancing the adaptability and robustness of the system in the absence of direct loop closure matching.

[0043] In some optional implementations of certain embodiments, the execution entity may generate optimized trajectory point pose information corresponding to the last initial image frame in the updated initial image frame queue by the following steps: The first step is to determine the last initial image frame in the updated initial image frame queue as the target image frame.

[0044] The second step is to determine the loop closure candidate image frames in the updated initial image frame queue as reference loop closure candidate image frames.

[0045] The third step involves performing co-view detection processing on the target image frame and the reference loop closure candidate image frame to obtain co-view detection information. In practice, the execution entity can use a feature point extraction algorithm (e.g., SIFT algorithm) to perform feature point extraction processing on the target image frame and the reference loop closure candidate image frame respectively, obtaining a target image frame feature point extraction information set corresponding to the target image frame and a reference loop closure candidate image frame feature point extraction information set corresponding to the reference loop closure candidate image frame. Then, the execution entity can input the target image frame feature point extraction information set and the reference loop closure candidate image frame feature point extraction information set into a pre-trained feature point matching model (e.g., SuperGlue model, LoFTR model) to obtain matching pairs of feature point extraction information. In response to the determination that the number of matching pairs of feature point extraction information in each matching pair is greater than a preset value, the execution entity can determine the information representing the co-view between the target image frame and the reference loop closure candidate image frame as co-view detection information. In response to the determination that the number of feature point extraction information matching pairs in each feature point extraction information matching pair is less than or equal to a preset value, the aforementioned execution entity can determine the information indicating that the target image frame and the reference loop closure candidate image frame co-exist as co-existence detection information. This co-existence detection information can be a Boolean value; for example, True indicates that the target image frame and the reference loop closure candidate image frame co-exist, and False indicates that the target image frame and the reference loop closure candidate image frame do not co-exist.

[0046] Fourth step: Based on the above common-view detection information and the updated initial image frame queue, generate optimized trajectory point pose information corresponding to the above target image frame.

[0047] Therefore, common-view detection can be performed using the target image frame and the reference loop closure candidate image frame in the updated initial image frame queue. Based on the detection results, the pose information of the trajectory points corresponding to the target image frame can be optimized to correct the accumulated error in the trajectory and improve the consistency between the trajectory corresponding to the optimized trajectory point pose information and the real trajectory.

[0048] In some optional implementations of certain embodiments, the execution entity may generate optimized trajectory point pose information corresponding to the target image frame based on the aforementioned common-view detection information and the updated initial image frame queue through the following steps: In the first step, in response to determining that the common-view detection information indicates that the target image frame and the reference loop closure candidate image frame have common view, the inertial measurement data corresponding to each initial image frame included in the initial image frame queue in the inertial measurement data sequence are arranged to obtain the target inertial measurement data sequence, and the reference loop closure candidate image frame is determined as the reference loop closure image frame.

[0049] The second step involves generating optimized objective function information based on the aforementioned target inertial measurement data sequence, the aforementioned initial image frame queue, the aforementioned reference loopback image frame, and the aforementioned target image frame.

[0050] The third step involves iteratively optimizing the objective function corresponding to the aforementioned objective function information to generate optimized trajectory point pose information corresponding to the target image frame. In practice, the executing entity can use nonlinear optimization algorithms such as the Gauss-Newton method or the Levenberg-Marquardt method to iteratively optimize the objective function corresponding to the objective function information to obtain optimized trajectory point pose information.

[0051] Therefore, an objective function can be constructed based on inertial measurement data, an initial image frame queue, a reference loop closure image frame, and a target image frame. The objective function can then be iteratively optimized to generate more accurate optimized trajectory point pose information corresponding to the target image frame.

[0052] In some optional implementations of certain embodiments, the execution entity may generate optimized objective function information based on the target inertial measurement data sequence, the initial image frame queue, the reference loopback image frame, and the target image frame through the following steps: The first step involves generating IMU constraint information based on the target inertial measurement data sequence and preset inertial measurement unit (IMU) noise parameters. These preset IMU noise parameters can be a noise covariance matrix, representing the random errors introduced by the IMU when measuring angular velocity and acceleration. In practice, the executing entity can generate IMU constraint information using an inertial navigation algorithm based on the target IMU data sequence and preset IMU noise parameters. This IMU constraint information can represent IMU constraints.

[0053] The second step involves generating reprojection error information as visual constraint information based on the initial image frame queue, the 3D coordinates of the 3D feature points, and the camera parameter information of the camera device. In practice, for each initial image frame in the initial image frame queue, the execution entity can identify the pixel position of the 3D feature points in the initial image frame using the SIFT algorithm. Then, the execution entity can use the camera pose, camera extrinsic parameters, and camera intrinsic parameters to convert the 3D coordinates of the 3D feature points into projected coordinates. Afterward, the execution entity can determine the Euclidean distance between the pixel position and the projected coordinates as the reprojection error corresponding to the initial image frame. Next, the execution entity can use Bundle Adjustment (BA) in visual SLAM to construct visual constraint information in the form of residual terms (or factor graph edges). This visual constraint information can represent visual constraints.

[0054] The third step involves generating a loop closure error term based on the aforementioned reference loop closure image frame and the target image frame. In practice, the executing agent can detect feature point pairs between the target image frame and the reference loop closure image frame using feature matching or a deep learning model. Then, for each feature point pair, the executing agent can determine its projection error between the two frames. Afterward, the executing agent can call a general nonlinear optimization library (such as Ceres Solver, g2o, GTSAM, etc.) to generate the loop closure error term based on each feature point pair. This loop closure error term can represent the loop closure constraint.

[0055] The fourth step involves generating an optimization objective function based on the aforementioned IMU constraint information, visual constraint information, and loop closure error term. This objective function can be a weighted sum of IMU constraints, visual constraints, and loop closure constraints. In practice, the executing entity can determine the optimization objective function as the weighted sum of these three factors.

[0056] Step 104: Based on at least one optimized trajectory point pose information, update the initial trajectory point pose information sequence to obtain the trajectory point pose information sequence.

[0057] In some embodiments, the execution entity may update the initial trajectory point pose information sequence based on at least one optimized trajectory point pose information to obtain a trajectory point pose information sequence.

[0058] In some optional implementations of certain embodiments, the execution entity may update the initial trajectory point pose information sequence based on at least one optimized trajectory point pose information to obtain a trajectory point pose information sequence: First, for each optimized trajectory point pose information in at least one of the above optimized trajectory point pose information, perform the following steps: The second step is to determine the image frames in the above image frame sequence that correspond to the pose information of the optimized trajectory points as reference image frames.

[0059] The second step is to replace the optimized trajectory point pose information with the initial trajectory point pose information corresponding to the reference image frame in the initial trajectory point pose information sequence, so as to update the initial trajectory point pose information sequence.

[0060] The fourth step is to determine the updated initial trajectory point pose information sequence as the trajectory point pose information sequence.

[0061] In some optional implementations of certain embodiments, after updating the initial trajectory point pose information sequence based on at least one optimized trajectory point pose information to obtain the trajectory point pose information sequence, the execution entity may further perform the following steps: The system displays the trajectory corresponding to the above-mentioned trajectory point pose information sequence. In practice, the aforementioned execution entity can use a display module to perform imaging to display the trajectory corresponding to the above-mentioned trajectory point pose information sequence.

[0062] Therefore, the trajectory corresponding to the sequence of trajectory point pose information can be displayed.

[0063] The above embodiments of this disclosure have the following beneficial effects: the trajectory point pose information sequence generation method of some embodiments of this disclosure improves the accuracy of the generated trajectory point pose sequence and reduces the trajectory drift of the device trajectory displayed based on the trajectory point pose sequence. Specifically, the reason for the poor accuracy of the generated trajectory point pose sequence and the high degree of trajectory drift of the device trajectory displayed based on the trajectory point pose sequence is that VIO relies on the data association between adjacent frames for pose estimation, but sensor noise, visual feature matching error, and IMU (Inertial Measurement Unit) drift accumulate over time, causing the trajectory deviation to gradually increase, which in turn leads to poor accuracy of the trajectory point pose sequence generated by the augmented reality display device during movement, resulting in a high degree of trajectory drift of the device trajectory displayed based on the trajectory point pose sequence. Based on this, the trajectory point pose information sequence generation method of some embodiments of this disclosure first collects the image frame sequence and inertial measurement data sequence corresponding to the preset scene containing three-dimensional feature points. Thus, the image frame sequence and inertial measurement data sequence collected by the augmented reality display device during movement can be obtained. Next, based on the aforementioned image frame sequence and inertial measurement data sequence, an initial trajectory point pose information sequence is generated. This generates an initial trajectory point pose information sequence representing the trajectory of the augmented reality display device during movement. Then, a sliding window loop closure optimization process is performed on the aforementioned image frame sequence to generate at least one optimized trajectory point pose information corresponding to at least one image frame in the sequence. This allows the use of sliding window loop closure optimization to generate at least one optimized trajectory point pose information for updating the initial trajectory point pose information sequence. Finally, based on the at least one optimized trajectory point pose information, the initial trajectory point pose information sequence is updated to obtain the trajectory point pose information sequence. Thus, the initial trajectory point pose information sequence can be optimized using at least one optimized trajectory point pose information, reducing accumulated errors in the initial trajectory point pose information sequence, reducing the difference between the trajectory represented by the generated trajectory point pose information sequence and the actual trajectory of the augmented reality display device during movement, and reducing the degree of trajectory drift of the device trajectory displayed based on the trajectory point pose sequence.

[0064] Further reference Figure 2 This illustrates a flowchart 200 of another embodiment of the trajectory point pose information sequence generation method. The flowchart 200 of this trajectory point pose information sequence generation method includes the following steps: Step 201: For a preset scene containing three-dimensional feature points, an image frame corresponding to the preset scene is acquired by a camera device at a preset acquisition frequency, and in response to determining that the acquired image frame meets the preset labeling conditions, the acquired image frame is marked as a loop closure candidate image frame, thus obtaining an image frame sequence containing at least one loop closure candidate image frame.

[0065] In some embodiments, the execution entity may, for a preset scene containing three-dimensional feature points, acquire image frames corresponding to the preset scene using a camera device at a preset acquisition frequency, and in response to determining that the acquired image frames meet preset labeling conditions, mark the acquired image frames as loop closure candidate image frames to obtain an image frame sequence containing at least one loop closure candidate image frame.

[0066] Step 202: Acquire inertial measurement data sequence through inertial measurement unit.

[0067] In some embodiments, the execution entity can acquire an inertial measurement data sequence using the inertial measurement component. Each image frame in the image frame sequence corresponds to one inertial measurement data point in the inertial measurement data sequence.

[0068] Step 203: Generate an initial trajectory point pose information sequence based on the image frame sequence and the inertial measurement data sequence.

[0069] In some embodiments, the execution entity may generate an initial trajectory point pose information sequence based on the image frame sequence and the inertial measurement data sequence.

[0070] Step 204: Perform sliding window loop closure optimization on the image frame sequence to generate at least one optimized trajectory point pose information corresponding to at least one image frame in the image frame sequence.

[0071] In some embodiments, the execution entity may perform sliding window loop closure optimization processing on the image frame sequence to generate at least one optimized trajectory point pose information corresponding to at least one image frame in the image frame sequence.

[0072] Step 205: Based on at least one optimized trajectory point pose information, update the initial trajectory point pose information sequence to obtain the trajectory point pose information sequence.

[0073] In some embodiments, the execution entity may update the initial trajectory point pose information sequence based on at least one optimized trajectory point pose information to obtain a trajectory point pose information sequence.

[0074] From Figure 2 It can be seen from this that, with Figure 1 Compared to the description of some corresponding embodiments, Figure 2The flow 200 of the trajectory point pose information sequence generation method in some corresponding embodiments embodies the following: acquiring image frames corresponding to the aforementioned preset scene using a camera device at a preset acquisition frequency; and, in response to determining that the acquired image frames meet preset annotation conditions, marking the acquired image frames as loop closure candidate image frames to obtain an image frame sequence containing at least one loop closure candidate image frame. Inertial measurement data sequences are acquired through the aforementioned inertial measurement component, wherein each image frame in the aforementioned image frame sequence corresponds to one inertial measurement data in the aforementioned inertial measurement data sequence. Because the acquisition of image frames is automated, marking the acquired image frames as loop closure candidate image frames in real time when they meet the preset annotation conditions, and because both image frame acquisition and loop closure candidate image frame marking are performed automatically, the system can quickly process large amounts of image data and rapidly generate an image frame sequence containing loop closure candidate image frames. This provides timely input for the generation of the trajectory point pose information sequence, reduces manual intervention in image annotation, and improves the system's response speed when generating the trajectory point pose information sequence.

[0075] In some embodiments, the specific implementation of steps 203-205 and the resulting technical effects can be found in [reference]. Figure 1 Steps 102-104 in the corresponding embodiments will not be repeated here.

[0076] like Figure 3 As shown, Figure 3 In (a), the true trajectory can be the motion trajectory of the augmented reality display device in three-dimensional space, represented by an orange trajectory line. In (b), the orange trajectory line can represent the true trajectory, and the blue trajectory line can be the trajectory corresponding to the initial trajectory point pose information sequence. In (c), the orange trajectory line can represent the true trajectory, and the green trajectory line can represent the trajectory corresponding to the corrected trajectory point pose information sequence.

[0077] The following is for reference. Figure 4 It shows a schematic diagram of the hardware structure of an augmented reality display device 400 with display capabilities.

[0078] like Figure 4As shown, the augmented reality display device 400 includes a processing unit (CPU) 401, a memory (ROM) 402, an input unit 403, and an output unit 404, wherein the processing unit 401, memory 402, input unit 403, and output unit 404 are interconnected via a bus 405. Here, the methods according to some embodiments of this disclosure can be implemented as a computer program and stored in the memory 402. The processing unit 401 in the augmented reality display device 400 implements the trajectory point pose information sequence generation function defined in some embodiments of this disclosure by calling the aforementioned computer program stored in the memory 402. In some implementations, the input unit 403 may include devices such as a camera, microphone, gyroscope, accelerometer, and magnetometer, and the output unit 404 may be a display module or other device used to display the trajectory corresponding to the aforementioned trajectory point pose information sequence. The display module may include an optomechanical system and optical elements. The optical elements may include prisms, freeform surfaces, BirdBaths, optical waveguides, and other optical components. Therefore, when the processing device 401 calls the computer program to execute the trajectory point pose information sequence generation function, it can control the input unit 403 to collect the image frame sequence and inertial measurement data sequence corresponding to the preset scene, and control the output unit 404 to display the trajectory corresponding to the trajectory point pose information sequence.

[0079] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0080] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0081] The aforementioned computer-readable medium may be included in the aforementioned augmented reality display device; or it may exist independently and not assembled into the augmented reality display device. The aforementioned computer-readable medium carries one or more programs that, when executed by the augmented reality display device, cause the augmented reality display device to: acquire an image frame sequence and an inertial measurement data sequence corresponding to a preset scene containing three-dimensional feature points; generate an initial trajectory point pose information sequence based on the image frame sequence and the inertial measurement data sequence; perform sliding window loop closure optimization processing on the image frame sequence to generate at least one optimized trajectory point pose information corresponding to at least one image frame in the image frame sequence; and update the initial trajectory point pose information sequence based on the at least one optimized trajectory point pose information to obtain a trajectory point pose information sequence.

[0082] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0084] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0085] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for generating a sequence of trajectory point pose information, comprising: For a preset scene containing three-dimensional feature points, an image frame sequence and an inertial measurement data sequence corresponding to the preset scene are collected; Based on the image frame sequence and the inertial measurement data sequence, an initial trajectory point pose information sequence is generated; Perform sliding window loop closure optimization on the image frame sequence to generate at least one optimized trajectory point pose information corresponding to at least one image frame in the image frame sequence, including: The image frame sequence is divided into an initial image frame queue sequence by means of a sliding window, wherein the first initial image frame queue in the initial image frame queue sequence includes at least one loopback candidate image frame. For each initial image frame queue in the initial image frame queue sequence, perform the following steps: The last initial image frame in the initial image frame queue is determined as the reference image frame; At least one loopback candidate image frame in the image frame sequence that is located before the reference image frame is determined as the loopback candidate image frame sequence; In response to determining that no loop closure candidate image frame exists in the initial image frame queue, the following steps are performed: Replace the first initial image frame in the initial image frame queue with a loop closure candidate image frame that meets the preset filtering conditions in the loop closure candidate image frame sequence to update the initial image frame queue; The last initial image frame in the updated initial image frame queue is determined as the target image frame; The loop closure candidate image frames in the updated initial image frame queue are determined as reference loop closure candidate image frames; Perform common-view detection processing on the target image frame and the reference loop closure candidate image frame to obtain common-view detection information; In response to determining that the common-view detection information indicates that the target image frame and the reference loop closure candidate image frame have common-view, the inertial measurement data corresponding to each initial image frame included in the initial image frame queue in the inertial measurement data sequence are arranged to obtain the target inertial measurement data sequence, and the reference loop closure candidate image frame is determined as the reference loop closure image frame. Based on the target inertial measurement data sequence, the initial image frame queue, the reference loopback image frame, and the target image frame, optimize the objective function information. The objective function corresponding to the optimized objective function information is iteratively optimized to generate optimized trajectory point pose information corresponding to the target image frame; Based on the at least one optimized trajectory point pose information, the initial trajectory point pose information sequence is updated to obtain the trajectory point pose information sequence.

2. The method according to claim 1, wherein, After updating the initial trajectory point pose information sequence based on the at least one optimized trajectory point pose information to obtain the trajectory point pose information sequence, the method further includes: Display the trajectory corresponding to the sequence of pose information of the trajectory points.

3. The method according to claim 1, wherein, The acquisition of image frame sequences and inertial measurement data sequences corresponding to the preset scene includes: The camera device acquires image frames corresponding to the preset scene at a preset acquisition frequency, and in response to determining that the acquired image frames meet the preset labeling conditions, the acquired image frames are labeled as loop closure candidate image frames, thus obtaining an image frame sequence containing at least one loop closure candidate image frame. An inertial measurement data sequence is acquired through an inertial measurement unit, wherein each image frame in the image frame sequence corresponds to one inertial measurement data in the inertial measurement data sequence.

4. The method according to claim 1, wherein, Each optimized trajectory point pose information in the at least one optimized trajectory point pose information corresponds to an image frame in the image frame sequence, each initial trajectory point pose information in the initial trajectory point pose information sequence corresponds to an image frame in the image frame sequence, and the step of updating the initial trajectory point pose information sequence based on the at least one optimized trajectory point pose information to obtain a trajectory point pose information sequence includes: For each optimized trajectory point pose information in the at least one optimized trajectory point pose information, perform the following steps: The image frame in the image frame sequence that corresponds to the pose information of the optimized trajectory point is determined as the reference image frame; The optimized trajectory point pose information replaces the initial trajectory point pose information corresponding to the reference image frame in the initial trajectory point pose information sequence, thereby updating the initial trajectory point pose information sequence. The updated initial trajectory point pose information sequence is determined as the trajectory point pose information sequence.

5. The method according to claim 1, wherein, After determining at least one loopback candidate image frame in the image frame sequence that precedes the reference image frame as a loopback candidate image frame sequence, the method further includes: In response to determining that there are loop closure candidate image frames in the loop closure candidate image frame sequence in the initial image frame queue, the optimized trajectory point pose information corresponding to the last initial image frame in the initial image frame queue is generated based on the initial image frame queue.

6. The method according to claim 1, wherein, The process of generating optimized objective function information based on the target inertial measurement data sequence, the initial image frame queue, the reference loopback image frame, and the target image frame includes: Based on the target inertial measurement data sequence and preset inertial measurement component noise parameters, IMU constraint information is generated; Based on the initial image frame queue, the three-dimensional coordinates of the three-dimensional feature points, and the camera parameter information of the camera device, reprojection error information is generated as visual constraint information; Based on the reference loopback image frame and the target image frame, a loopback error term is generated; Based on the IMU constraint information, the visual constraint information, and the loop closure error term, optimization objective function information is generated.

7. An augmented reality display device, which is an augmented reality head-mounted display device or an augmented reality handheld display device, and said augmented reality display device comprises: One or more processors; A camera device is used to acquire image frames corresponding to the preset scene at a preset acquisition frequency, and in response to determining that the acquired image frames meet preset labeling conditions, to label the acquired image frames as loop closure candidate image frames, thereby obtaining an image frame sequence containing at least one loop closure candidate image frame. Inertial measurement unit (IMU) is used to acquire inertial measurement data sequences. Storage device for storing one or more programs; A display module is used in imaging to display the trajectory corresponding to the sequence of trajectory point pose information. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

Images