Lightweight pose measurement method fusing inertial measurement unit scale information

Abstract

The application provides a lightweight pose measurement method fusing inertial measurement unit scale information, comprising the following steps: determining the initial pose and working state of an AR device in a real world at a first time according to a first time image and attitude information; determining the relative pose between the AR device at the second time and the AR device at the first time according to a second time image and attitude information and the working state; and performing different preset output strategies on the relative pose to obtain the real-time pose of the AR device in the real world at the second time; the calculation and processing procedure of the real-time pose of the AR device is greatly optimized, the real-time pose result can be efficiently and quickly output, the complicated processing and operation procedure is avoided, and the demand of a scene with low accuracy requirement of position and attitude information in special industries such as AR navigation is met.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of AR navigation, and particularly relates to a lightweight pose measurement method fusing scale information of an inertial measurement unit. BACKGROUND

[0002] In the AR field, a device estimates its position and attitude in the real space by using various sensors, so as to combine virtual content with real content. However, in some special business scenarios (such as AR navigation), the pose estimation algorithm does not need to provide high-precision position and attitude information, nor does it need strong time continuity, but only needs to ensure that the position and yaw angle are accurate within a certain range.

[0003] In the existing technology, most of the pose estimation is performed by using a visual inertial odometer, which can achieve high-precision output results, but needs to be matched with a corresponding high-end device for use, thereby resulting in high cost and excessive computation. Although the visual inertial odometer has low requirements for the data quality output by a camera and an inertial measurement unit, it has certain requirements for sensor timestamps, external parameters and stability. Moreover, for some low-end devices, the sensor timestamps cannot be aligned, and even are disordered, so that the low-end devices cannot be applied to the visual inertial odometer. Moreover, the calculation complexity of the visual inertial odometer is higher than that of the visual odometer, so the AR device memory and computing power are also required, and the device also needs to allocate the memory and computing power to other modules, so that the computing resources of the entire system become extremely nervous. SUMMARY

[0004] In order to overcome the deficiencies of the prior art, the application provides a lightweight pose measurement method fusing scale information of an inertial measurement unit, so as to solve the problems of high cost and inapplicability of low-end devices in the prior art.

[0005] One embodiment of the application provides a lightweight pose measurement method fusing scale information of an inertial measurement unit, comprising the following steps:

[0006] According to the image and attitude information at the first time, the initial pose and working state of the AR device in the real world at the first time are determined;

[0007] According to the image and attitude information at the second time and the working state, the relative pose between the AR device at the second time and the AR device at the first time is determined;

[0008] Different preset output strategies are performed for the relative pose, so as to obtain the real-time pose of the AR device in the real world at the second time.

[0009] In one of the embodiments, the determining of the initial pose and the working state of the AR device in the real world at the first time according to the image at the first time and the pose information comprises:

[0010] obtaining the pose information and the preset height position of the AR device at the first time, obtaining the relative pose of the AR device with respect to the real world based on the pose information through a preset algorithm, and determining the initial pose of the AR device in the real world at the first time according to the preset height position and the relative pose;

[0011] obtaining the image at the first time based on the initial pose;

[0012] determining the working state according to the map point of the image at the first time.

[0013] In one of the embodiments, the obtaining of the pose information and the preset height position of the AR device at the first time, the obtaining of the relative pose of the AR device with respect to the real world based on the pose information through a preset algorithm, and the determining of the initial pose of the AR device in the real world at the first time according to the preset height position and the relative pose comprises:

[0014] constructing a first coordinate system, wherein the Z-axis of the first coordinate system is arranged in the opposite direction of the gravity direction of the real world;

[0015] obtaining the pose information and the preset height position of the AR device at the first time based on the first coordinate system;

[0016] generating a second coordinate system according to the preset height position and the relative pose;

[0017] wherein the first coordinate system is the reference pose of the real world;

[0018] the second coordinate system is the initial pose of the AR device at the first time.

[0019] In one of the embodiments, the determining of the working state according to the map point of the image at the first time comprises:

[0020] obtaining the feature point of the image at the first time to generate the map point of the image at the first time;

[0021] judging the map point of the image at the first time according to a preset depth threshold;

[0022] determining the working state according to the judgment result.

[0023] In one of the embodiments, the determining of the relative pose between the AR device at the second time and the AR device at the first time according to the image at the second time, the pose information and the working state comprises:

[0024] acquire the second-time image and pose information and the working state; wherein the working state comprises an initialization state and a tracking state;

[0025] if the working state is the initialization state, re-determine the working state according to the second-time image;

[0026] if the working state is the tracking state, construct a target function and output the relative pose.

[0027] In one of the embodiments, the constructing a target function and outputting the relative pose comprises:

[0028] generating an initial relative pose according to the second-time pose information and the first-time pose information;

[0029] re-projecting the first-time map points into the second-time image according to the initial relative pose and a camera model to obtain feature point positions of the first-time map points in the second-time image;

[0030] performing re-projection matching on the feature point positions and feature points of the second-time image;

[0031] constructing a target function according to the re-projection matching result and outputting an optimal result;

[0032] wherein the optimal result is the relative pose.

[0033] In one of the embodiments, the performing different preset output strategies on the relative pose to obtain the real-time pose of the AR device in the real world comprises:

[0034] performing a first output strategy on a matching result of the relative pose and a preset error threshold to obtain the real-time pose of the AR device in the real world;

[0035] performing a second output strategy on a scale-degraded relative pose to obtain the real-time pose of the AR device in the real world.

[0036] In one of the embodiments, the performing a first output strategy on a matching result of the relative pose and a preset error threshold to obtain the real-time pose of the AR device in the real world comprises:

[0037] if the relative pose exceeds the preset error threshold, performing a giving strategy;

[0038] if the relative pose does not exceed the preset error threshold, performing a combination strategy.

[0039] In one of the embodiments, the assigning strategy is configured to assign the initial pose to the second-time image to obtain the real-time pose of the AR device in the real world at the second time;

[0040] And / or, the combining strategy is configured to combine the relative pose with the initial pose to obtain the real-time pose of the AR device in the real world at the second time.

[0041] In one of the embodiments, the second output strategy comprises:

[0042] According to the user walking speed at the second time and the time difference between the second-time picture and the first-time picture, the distance of the user walking is obtained;

[0043] The position information in the relative pose is obtained to form a position vector, and the length of the position vector is obtained;

[0044] The ratio of the distance of the user walking and the length of the position vector is obtained to obtain a speed-up factor;

[0045] According to the speed-up factor and the position vector, the real-time position information of the AR device in the real world at the second time is obtained;

[0046] The real-time position information and the second-time pose information are spliced to obtain the real-time pose of the AR device in the real world at the second time.

[0047] The lightweight pose measurement method provided by the above embodiments has the following beneficial effects:

[0048] By optimizing the first-time picture and the pose information, the initial pose and the working state of the AR device are efficiently obtained, and the relative pose of the AR device at the second time relative to the first time is determined based on the first-time picture and the pose information. At the same time, different preset output strategies are performed on the relative pose to obtain the real-time pose of the AR device in the real world at the second time. The calculation and processing process of the real-time pose of the AR device is greatly optimized, the real-time pose result can be efficiently and quickly output, the tedious processing and operation process is avoided, the demand of the scene where the accuracy requirement of the position and pose information is not high in the special industry such as AR navigation is met, and the strong time continuity is not required, which ensures that the low-end device can be used. Not only the working cost is reduced, but also the industry demand is met. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only show some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from the structures shown in these drawings without any creative effort.

[0050] Figure 1 The working method schematic diagram of the fusion inertial measurement unit scale information lightweight pose measurement method provided by the embodiments of the present application;

[0051] Figure 2 The working flow schematic diagram of the fusion inertial measurement unit scale information lightweight pose measurement method in the embodiments of the present application; Figure 1

[0052] Figure 3 The working flow schematic diagram of the fusion inertial measurement unit scale information lightweight pose measurement method in the embodiments of the present application; Figure 2

[0053] Figure 4 The working flow schematic diagram of the fusion inertial measurement unit scale information lightweight pose measurement method in the embodiments of the present application; Figure 3

[0054] Figure 5 The working flow schematic diagram of the fusion inertial measurement unit scale information lightweight pose measurement method in the embodiments of the present application; Figure 1

[0055] Figure 6 The working flow schematic diagram of the fusion inertial measurement unit scale information lightweight pose measurement method in the embodiments of the present application; Figure 1

[0056] Figure 7 The working flow schematic diagram of the fusion inertial measurement unit scale information lightweight pose measurement method in the embodiments of the present application; Figure 1

[0057] Figure 8 The working principle schematic diagram of the fusion inertial measurement unit scale information lightweight pose measurement method in the embodiments of the present application; Figure 1

[0058] Figure 9 The working principle schematic diagram of the fusion inertial measurement unit scale information lightweight pose measurement method in the embodiments of the present application; Figure 1

[0059] Figure 10 The working principle schematic diagram of the fusion inertial measurement unit scale information lightweight pose measurement method in the embodiments of the present application;​​​​​​​​Figure 1 Coordinate system schematic diagram of a lightweight pose measurement method fusing inertial measurement unit scale information in the embodiment of the present application;

[0060] Figure 11 For Figure 1 Map point projection schematic diagram of a lightweight pose measurement method fusing inertial measurement unit scale information in the embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0062] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, motion condition, etc. between components in a certain specific posture, and if the specific posture changes, the directional indications also change accordingly.

[0063] In addition, if the embodiments of the present application involve descriptions such as “first”, “second”, etc., the descriptions of “first”, “second”, etc. are only for description purposes, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by “first”, “second” can explicitly or implicitly include at least one of the features. In addition, “and / or” or “and / or” appearing throughout the text means that the three parallel schemes are included, for example, “A and / or B” includes A scheme, or B scheme, or A and B are satisfied at the same time. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the fact that a person of ordinary skill in the art can realize it, and when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, and is not within the protection scope claimed by the present application.

[0064] In the field of AR, devices will estimate their own position and posture in the real space by using various sensors, so as to combine virtual content with real content. Therefore, the pose estimation accuracy determines the upper limit of the effect of AR content, and is one of the core algorithms of the entire system.

[0065] Generally, the sensors commonly used in civilian AR devices include global positioning system (GPS), Bluetooth, inertial measurement unit (IMU), camera, and laser sensor, etc. The Bluetooth and GPS can only provide position information, with poor accuracy and low frequency. However, the AR device has a requirement for the frequency of pose estimation, and thus the two sensors are not suitable for being the main sensors for pose estimation of the AR device.

[0066] The laser sensor has high accuracy and low algorithm complexity, but it has high requirements for the power consumption and cost of the device, and thus most AR device manufacturers do not equip the sensor.

[0067] The camera can provide images containing a large amount of physical world information. The visual odometry is to generate a high-precision and high-frequency pose estimation result by using time-continuous images. However, the visual odometry will introduce large noise due to image blurring under fast motion, which will seriously reduce the estimation accuracy. In addition, the visual odometry corresponding to a single camera does not have scale information (real-world physical size).

[0068] The inertial measurement unit (IMU) is a sensor that can accurately estimate short-time fast motion. In addition, the accelerometer of the IMU can provide scale information, which can complement the camera to some extent and extend the visual inertial odometry. The visual inertial odometry is a mainstream choice for pose estimation of the AR device.

[0069] The visual inertial odometry is to add the data of the inertial measurement unit to the optimization process of the visual odometry. The visual calculation result can eliminate the noise and error of the inertial measurement unit. In addition, the inertial measurement unit provides scale and prior information for the vision under fast motion, so as to make the estimation more accurate and robust.

[0070] Referring to Figures 1-11 One embodiment of the present application provides a lightweight pose measurement method fusing scale information of an inertial measurement unit, comprising the following steps:

[0071] S100, determining an initial pose and a working state of an AR device in a real world at a first time according to a first-time image and pose information;

[0072] In the embodiment, the AR device is provided with an inertial measurement unit and a camera. The first-time image is acquired by the camera, and the pose information of the AR device corresponding to the first-time image is acquired by the inertial measurement unit. The pose information includes angular velocity and acceleration. The initial pose of the AR device in the real world at the first time is determined according to the pose information, and the working state is determined according to the first-time image. Specifically, the steps include:

[0073] In one of the embodiments, the determining the initial pose and working state of the AR device in the real world at the first time according to the image at the first time and the pose information comprises:

[0074] S110, acquiring the pose information of the AR device at the first time and a preset height position, and obtaining the relative pose of the AR device relative to the real world based on the pose information through a preset algorithm, and determining the initial pose of the AR device in the real world at the first time according to the preset height position and the relative pose;

[0075] In the embodiment, the pose information of the AR device relative to the real world at the first time is obtained by constructing the coordinate relationship between the AR device and the real world, and the initial pose of the AR device in the real world at the first time is obtained by combining the pose information with the preset height position. The preset height position can be obtained according to a big data prediction model or can be self-defined according to the use height. The initial height of the device is set to obtain the initial pose. Specifically, the steps include:

[0076] In one of the embodiments, the acquiring the pose information of the AR device at the first time and a preset height position, and obtaining the relative pose of the AR device relative to the real world based on the pose information through a preset algorithm, and determining the initial pose of the AR device in the real world at the first time according to the preset height position and the relative pose comprises:

[0077] S111, constructing a first coordinate system, wherein the Z-axis of the first coordinate system is set in the opposite direction of the gravity direction of the real world;

[0078] In the embodiment, the reference pose of the real world is constructed to obtain the coordinate relationship between the AR device and the real world and obtain the initial pose.

[0079] S112, acquiring the pose information of the AR device at the first time and the preset height position based on the first coordinate system;

[0080] In the embodiment, the pose information output by the inertial measurement unit is acquired, and the relative pose of the AR device relative to the first coordinate system or relative to the gravity direction is obtained through a preset algorithm, so that the acquisition process of the initial pose is optimized.

[0081] S113, generating a second coordinate system according to the preset height position and the relative pose;

[0082] The first coordinate system is the reference pose of the real world.

[0083] The second coordinate system is the initial pose of the AR device at the first time.

[0084] Specifically, by defining the AR device in the coordinate system relationship of the real world and the variable definition, it is assumed that there is a first coordinate system C in the real world, which is opposite to the direction of gravity global , and a second coordinate system (AR device itself coordinate system) is C local , then the initial pose of the AR device at time t0 (initial time of running) in space is T local0_to_global , the initial pose includes initial position information and initial attitude information (or first time position information and first time attitude information), the relationship is as shown in Figure 10 .

[0085] It is assumed that the AR device is located at a height H init , then the initial position of the AR device can be set as (0, 0, H init ), and the received inertial measurement unit (IMU) angular velocity and acceleration results are [w x , w y , w z , a x , a y , a z ], then the relative attitude R local0_to_global of the AR device relative to the direction of gravity can be obtained through a preset algorithm, the preset algorithm is used to obtain the relative attitude R local0_to_global of the AR device relative to the direction of gravity, the preset algorithm can be a complementary filtering algorithm (not limited to this algorithm, other algorithms can be used instead), and the initial pose of the AR device is T local0_to_global = [R local0_to_global , (0, 0, H init )].

[0086] Among them, the attitude information is the measurement data of the inertial measurement unit, including angular velocity and acceleration.

[0087] S120, obtaining the first time image based on the initial pose;

[0088] S130, determining the working state according to the map point of the first time image.

[0089] In one of the embodiments, the working state is determined according to the map point of the first time image, including:

[0090] S131, obtaining the feature point of the first time image, and generating the map point of the first time image;

[0091] S132, judging the map point of the first time image according to a preset depth threshold;

[0092] S133, determining the working state according to the judgment result.

[0093] In the embodiment, the working state is determined by the information of the first time image, when the first time image arrives, the feature points in the first time image are extracted, and the map points of the first time image are generated by the camera model and the feature point position information, wherein the feature points are two-dimensional data or two-dimensional information, and the map points are three-dimensional data or three-dimensional models. The three-dimensional model is generated according to the two-dimensional data in the image, and can be projected in reality. Since the greater the depth of the map point (the greater the distance between the three-dimensional model and the camera), the greater the error of the subsequent optimization result, a depth threshold of the map point is needed to screen the generated map points. If the number of the screened map points is less than the preset depth threshold, the initialization fails, and the working state of the system is still the initialization state. If the initialization is successful, the working state of the system is the tracking state.

[0094] The feature points of the first time image after screening and the corresponding map points of the first time image are saved for the second time image registration to calculate the relative pose, and the attitude information output by the inertial measurement unit corresponding to the first time image is also saved.

[0095] S200, determining the relative pose between the second time AR device and the first time AR device according to the second time image, the attitude information and the working state;

[0096] In the embodiment,

[0097] In one of the embodiments, the determining the relative pose between the second time AR device and the first time AR device according to the second time image, the attitude information and the working state comprises:

[0098] S210, obtaining the second time image, the attitude information and the working state; wherein the working state comprises an initialization state and a tracking state;

[0099] S220, if the working state is the initialization state, re-determining the working state according to the second time image;

[0100] S230, if the working state is the tracking state, constructing a target function and outputting the relative pose.

[0101] In one of the embodiments, the constructing a target function and outputting the relative pose comprises:

[0102] S231, generating an initial relative pose according to the second time attitude information and the first time attitude information;

[0103] S232, reproject the first time map point into the second time image according to the initial relative pose and camera model, to obtain a feature point position of the first time map point in the second time image;

[0104] S233, reproject matching is performed on the feature point position and the feature points of the second time image;

[0105] S234, a target function is constructed according to the result of the reproject matching, and an optimal result is output;

[0106] The optimal result is the relative pose.

[0107] Specifically, the new image (second time image) and the data output by the inertial measurement unit are obtained, and it is first judged whether the working state is the initialization state. If it is still maintained in the initialization state, then the initialization is reperformed through the second time image.

[0108] If the working state is the tracking state, then the feature points pi in the first time image (first time image) and the corresponding map points P world are taken. The initial relative pose T local1_to_local0_init between the two frames is calculated according to the angular velocity and acceleration output by the inertial measurement unit corresponding to the current image (second time image) and the first time image.

[0109] The map point in the first time image is reprojected into the current image to obtain the feature point position P p = π(T local1_to_loca10_init *P world ), wherein π represents the camera model (3D points under the camera coordinates are converted to pixel position points under the second time image), so that the image matching pixel points of the same map points corresponding to the two time images can be obtained, as shown in Figure 11 In a short time camera small motion range, the pixel gray values of the feature points corresponding to the same map points projected into different images are the same, so the matching pixel point pixel gray value can be constructed as an error to construct a target function, as follows:

[0110] T local1_to_local0 = arg minΣ[I(π(T local1_to_local0_init *p world ))-I(pi)]

[0111] The above target function can be solved by Gauss-Newton method, and finally the relative pose of the adjacent two frames with the minimum difference of the matching pixel point pixel gray value is the optimal result (relative pose);

[0112] S300, execute different preset output strategies according to the relative pose, so that the AR device at the second time has a real-time pose in the real world.

[0113] In this embodiment, by judging the relative pose, the corresponding preset output strategy is executed according to the judgment result, so as to improve the accuracy of the real-time pose and improve the work efficiency, and greatly optimize the output process.

[0114] In one embodiment, the different preset output strategies are executed according to the relative pose, so that the AR device at the second time has a real-time pose in the real world, comprising:

[0115] S310, execute a first output strategy according to the matching result of the relative pose and a preset error threshold, to obtain a real-time pose of the AR device in the real world at the second time;

[0116] In one embodiment, the first output strategy is executed according to the matching result of the relative pose and a preset error threshold, to obtain a real-time pose of the AR device in the real world at the second time, comprising:

[0117] S311, if the relative pose exceeds the preset error threshold, execute a giving strategy;

[0118] S312, if the relative pose does not exceed the preset error threshold, execute a combination strategy.

[0119] In one embodiment, the giving strategy is configured to assign the initial pose to the second time image to obtain a real-time pose of the AR device in the real world at the second time;

[0120] And / or, the combination strategy is configured to combine the relative pose and the initial pose to obtain a real-time pose of the AR device in the real world at the second time.

[0121] In this embodiment, the first output strategy is used to efficiently generate and output the real-time pose, greatly improving the work efficiency, optimizing the processing flow, and meeting the use of low-end devices, having great adaptability and practicality.

[0122] Specifically, it is judged whether the re-projection error of the map point re-projection optimization exceeds a preset error threshold. If the re-projection error exceeds the preset error threshold, it means that the feature point matching fails, and the relative pose of the first time image in space is assigned to the current image, T local1_togobal = T local1_to_loca10 ;

[0123] If the optimized re-projection error does not exceed the set threshold, the relative pose Tlocal1_to_local0 a relative pose T of the first time image in space local10_to_gobal In combination, a relative pose T of the current image frame in space can be finally obtained local1_to_gobal = T local1_to_local0 *T local10_to_gobal ;

[0124] S320, performing a second output strategy for the scale-degraded relative pose to obtain a real-time pose of the AR device in the real world at the second time.

[0125] In one of the embodiments, the second output strategy comprises:

[0126] S321, obtaining a distance of user walking according to a user walking speed at the second time and a time difference between the picture at the second time and the picture at the first time;

[0127] S322, obtaining position information in the relative pose to form a position vector and obtaining a length of the position vector;

[0128] S323, obtaining a speed-up ratio of the distance of user walking to the length of the position vector;

[0129] S324, obtaining real-time position information of the AR device in the real world at the second time according to the speed-up ratio and the position vector;

[0130] S325, splicing the real-time position information and the second time attitude information to obtain a real-time pose of the AR device in the real world at the second time.

[0131] In the embodiment, the scale-degraded relative pose is compensated by the second output strategy, so as to improve the accuracy of the output real-time pose, and the corresponding processing can be performed for various relative poses, so as to ensure the accuracy, precision and efficiency of the output result.

[0132] According to the result of the re-projection matching of the relative pose and the relative pose, it is determined whether the scale of the relative pose is degraded. Specifically, the number of re-projection matching points (a small part of points will be excluded because they do not meet the re-projection error) and the re-projection error are combined to determine whether the scale corresponding to the relative pose results of the optimized adjacent two frames is degraded (the scale is degraded: the position information almost does not change, only the attitude information changes). In order to correct the scale degradation, a pedometer algorithm based on the output acceleration of the inertial measurement unit is introduced, and the pedometer can obtain the current walking speed v u of the user, so that the distance of the user advancing between adjacent images can be calculated to compensate the position change result.

[0133] If the judgment condition is determined to be degraded, the distance user_dis = v u *(t1-t0) is calculated according to the time difference between the first time image and the current image, and then the position information is extracted from the relative pose of the adjacent images to form a position vector:

[0134] v local1_to_local0 =[x local1_to_local0 ,y local1_to_local0， ,z local1_to_local0 ]

[0135] The position vector is used as a direction vector, and then the ratio of the distance user_dis and the length of the position vector V local1_to_ local0 .norm is used as a multiplier to act on the direction vector to obtain the position result of the relative pose of the adjacent images after the pedometer scale correction, and the compensation formula is as follows:

[0136] Based on the above pose result, the feature points in the second time image are extracted, and the map points of the second time image are generated through the camera model and the feature point position information, and the preset depth threshold of the map points of the second time image is set to screen the generated map points of the second time image. The screened feature points and the corresponding map points are saved for the next time image registration to calculate the relative pose, and the angular velocity and acceleration information output by the inertial measurement unit corresponding to the second time image are also saved.

[0137] The position information in the relative pose obtained by reprojecting the map points of the second time image is extracted and spliced with the second attitude information output by the inertial measurement unit of the second time image, and finally the pose result of the current AR device relative to the actual space is obtained. Wherein, the first time is the last time, and the second time is the current time.

[0138] By optimizing the first time image and the attitude information, the initial pose and working state of the AR device are efficiently obtained, and the relative pose of the second time AR device relative to the first time is determined based on the first time image and the attitude information. At the same time, by executing different preset output strategies for the relative pose, the real-time pose of the second time AR device in the real world is obtained; greatly optimizing the calculation and processing process of the real-time pose of the AR device, efficiently and quickly outputting the real-time pose result, avoiding the cumbersome processing and operation process, meeting the needs of the scene where the accuracy of position and attitude information is not high in special industries such as AR navigation, and without strong time continuity, ensuring that low-end devices can be used, not only reducing the working cost, but also meeting the industry demand.

[0139] In the embodiment, the initial pose of the AR device is obtained through the pose information output in real time by the inertial measurement unit and the information of the images generated at the last time and the current time, and the relative pose between two adjacent frames is calculated through a target function, and the real-time pose of the AR device relative to the actual space (real world) at the current time is obtained according to the result of the relative pose, which not only meets the use of low-end devices with weak time continuity, but also meets the use demand of special scenes such as AR navigation and the like which have low accuracy requirements for position information and attitude information, and greatly reduces the working cost.

[0140] The visual inertial odometer still needs to follow the standard process of the visual odometer, that is, initialization, generation of map points through mutual correlation of continuous multiple frames of feature points in a sliding window, registration and optimization of the map points relative to the current image frame to estimate the pose, which can ensure smooth trajectory, accurate scale and high precision. Although the visual inertial odometer has low requirements for the data quality output by the camera and the inertial measurement unit, it has certain requirements for the sensor timestamp, external parameter and stability. For some low-end devices, the sensor timestamp cannot be aligned, and even is chaotic, so the visual inertial odometer cannot be used. The calculation complexity of the visual inertial odometer is higher than that of the visual odometer, so the memory and computing power of the AR device are also required, and the memory and computing power of the device also need to be allocated to other modules, so that the computing resources of the whole system become extremely nervous. In some special business scenarios (such as AR navigation), the pose estimation algorithm does not actually need to provide high-precision position and attitude information, nor does it need strong time continuity, but only needs to ensure that the position and yaw angle are accurate within a certain range. For low-end AR devices and scenes with degraded pose requirements (only accurate position and yaw angle are required), the scheme of the present application is used to replace the current mainstream visual inertial odometer.

[0141] One of the embodiments of the present application further provides a measurement system, comprising:

[0142] An acquisition module is configured to acquire images and attitude information.

[0143] A processing module is configured to process the images and the attitude information to generate the initial pose, the relative pose and the real-time pose.

[0144] A storage module is configured to store the images, the attitude information, the initial pose, the relative pose and the real-time pose.

[0145] One of the embodiments of the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is executed by the processor to implement the steps of the head-mounted device virtual-real fusion rendering method based on timing control according to any one of the above.

[0146] One of the embodiments of the present application also provides a storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the head-mounted device virtual-real fusion rendering method based on timing control according to any one of the above.

[0147] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, device, system or computer program product. Therefore, the embodiments of the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can adopt a computer program product implemented on one or more computer usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program codes.

[0148] The embodiments of the present application are described with reference to flowcharts and / or block diagrams of the method, terminal device (apparatus), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to produce a machine, so that the instructions executed by the computer or other programmable data processing terminal devices produce a device implemented in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one or more flows and / or blocks Figure 1 The device that implements the functions specified in one or more flows and / or blocks

[0149] These computer program instructions can also be stored in a computer readable memory capable of guiding the computer or other programmable data processing terminal devices to work in a specific way, so that the instructions stored in the computer readable memory produce a manufactured product including instruction devices, which implement the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one or more flows and / or blocks Figure 1 The device that implements the functions specified in one or more flows and / or blocks These computer program instructions can also be loaded into the computer or other programmable data processing terminal devices, so that a series of operation steps are performed on the computer or other programmable terminal devices to produce a computer implemented process, so that the instructions executed on the computer or other programmable terminal devices provide a device for implementing the flowcharts and / or block diagrams.Figure 1 one or more processes and / or functions specified in one or more blocks Figure 1 one or more processes and / or functions specified in one or more blocks

[0150] The above merely provides the preferred embodiments of the application, and is not intended to limit the patent scope of the application. Any equivalent structure variations made according to the application concept, or direct / indirect application in other related technical fields, are included in the patent protection scope of the application.

Claims

1. A lightweight pose measurement method fusing inertial measurement unit scale information, characterized in that, The method comprises the following steps: According to the image and attitude information at the first time, the initial pose and working state of the AR device in the real world at the first time are determined; According to the image and attitude information at the second time and the working state, the relative pose between the AR device at the second time and the AR device at the first time is determined; Different preset output strategies are executed for the relative pose to obtain the real-time pose of the AR device in the real world at the second time; The different preset output strategies are executed for the relative pose to obtain the real-time pose of the AR device in the real world at the second time, comprising: A first output strategy is executed for the matching result of the relative pose and a preset error threshold to obtain the real-time pose of the AR device in the real world at the second time; A second output strategy is executed for the scale-degraded relative pose to obtain the real-time pose of the AR device in the real world at the second time; The second output strategy comprises: The distance walked by the user is obtained according to the user walking speed at the second time and the time difference between the picture at the second time and the picture at the first time; The position information in the relative pose is obtained to form a position vector, and the length of the position vector is obtained; The speed-up is obtained by taking the ratio of the distance walked by the user and the length of the position vector; The real-time position information of the AR device in the real world at the second time is obtained by combining the speed-up with the position vector; The real-time pose of the AR device in the real world at the second time is obtained by splicing the real-time position information and the attitude information at the second time.

2. The lightweight pose measurement method fusing inertial measurement unit scale information according to claim 1, wherein, The initial pose and working state of the AR device in the real world at the first time are determined according to the image and attitude information at the first time, comprising: The attitude information of the AR device at the first time and a preset height position are obtained, the relative attitude of the AR device with respect to the real world is obtained by a preset algorithm based on the attitude information, and the initial pose of the AR device in the real world at the first time is determined according to the preset height position and the relative attitude; The image at the first time is obtained based on the initial pose; The working state is determined according to the map points of the image at the first time.

3. The lightweight pose measurement method fusing inertial measurement unit scale information according to claim 2, wherein, The attitude information of the AR device at the first time and a preset height position are obtained, the relative attitude of the AR device with respect to the real world is obtained by a preset algorithm based on the attitude information, and the initial pose of the AR device in the real world at the first time is determined according to the preset height position and the relative attitude, comprising: A first coordinate system is constructed, wherein the Z-axis of the first coordinate system is arranged in the opposite direction of the gravity direction of the real world; The attitude information of the AR device at the first time and the preset height position are obtained based on the first coordinate system; A second coordinate system is generated according to the preset height position and the relative attitude; The first coordinate system is the reference pose of the real world; The second coordinate system is the initial pose of the AR device at the first time.

4. The lightweight pose measurement method fusing inertial measurement unit scale information according to claim 2, wherein, The working state is determined according to the map points of the image at the first time, comprising: The feature points of the image at the first time are obtained to generate the map points of the image at the first time; According to a preset depth threshold, judging the map point of the first time image; According to the judgment result, determining the working state.

5. The lightweight pose measurement method fusing inertial measurement unit scale information according to claim 1, wherein, According to the second time image, the attitude information and the working state, determining the relative pose between the second time AR device and the first time AR device, comprising: Obtaining the second time image, the attitude information and the working state; wherein the working state comprises an initialization state and a tracking state; If the working state is the initialization state, re-determining the working state according to the second time image; If the working state is the tracking state, constructing a target function and outputting the relative pose.

6. The lightweight pose measurement method fusing inertial measurement unit scale information according to claim 5, wherein, The construction of the target function and the output of the relative pose, comprising: According to the second time attitude information and the first time attitude information, generating an initial relative pose; According to the initial relative pose and the camera model, re-projecting the first time map point into the second time image to obtain the feature point position of the first time map point in the second time image; According to the feature point position and the feature point of the second time image, performing re-projection matching; According to the result of the re-projection matching, constructing a target function and outputting an optimal result; Wherein, the optimal result is the relative pose.

7. The lightweight pose measurement method fusing inertial measurement unit scale information according to claim 1, wherein, The first output strategy is executed according to the matching result of the relative pose and the preset error threshold, and the real-time pose of the second time AR device in the real world is obtained, comprising: If the relative pose exceeds the preset error threshold, the assignment strategy is executed, and the assignment strategy is to assign an initial pose to the second time image to obtain the real-time pose; If the relative pose does not exceed the preset error threshold, the combination strategy is executed, and the combination strategy is to combine the relative pose with the initial pose to obtain the real-time pose.

Images