Pose estimation method, virtual display image generation method, and related device thereof

Abstract

The present application discloses a pose estimation method, comprising: positioning feature points by means of a first thread to obtain position information; storing inertial data of an inertial sensor into a cache module; by means of the first thread, updating a first pose on the basis of the inertial data and the position information to obtain a second pose; and by means of a second thread, performing prediction on the basis of target inertial data and the second pose to obtain a target pose. The present application can reduce the power consumption of pose prediction and reduce the resource consumption.

Description

Pose estimation method, virtual display image generation method and related devices thereof TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a pose estimation method, a virtual display image generation method and related devices thereof. BACKGROUND

[0002] With the development of virtual reality (VR), augmented reality (AR) and mixed reality (MR) technologies, there is an increasing demand for accurate estimation of user head pose. The accuracy of head pose estimation will directly affect the naturalness and immersion of user experience.

[0003] Traditional head pose estimation methods rely on image data captured by a camera or sensor data collected based on an inertial measurement unit (IMU). However, estimation methods using a single data source often have limitations. For example, visual methods based on image data may perform poorly in environments with changing lighting or unobvious features, while pose estimation methods based on IMU may drift after long-time integration of sensor data, resulting in inaccurate real-time pose estimation. In head-mounted display devices that include spatial calculations (such as AR glasses, VR headsets, etc.), low-latency and high-precision head pose estimation is a core requirement and basic function of such devices.

[0004] Furthermore, due to the size limitations of head-mounted display devices, strict requirements are placed on the use of computing resources. TECHNICAL PROBLEM

[0005] Current technologies often require excessive computing resources to ensure the accuracy of pose prediction, but this limits the use of such head pose estimation techniques on head-mounted display devices, making them unable to be deployed on low-cost and low-power chips. TECHNICAL SOLUTION

[0006] The embodiments of the present application provide a pose estimation method, a virtual display image generation method and related devices thereof, which can solve the technical problem that current pose prediction requires a large amount of computing resources and cannot be deployed on low-cost and low-power chips. In a first aspect, the embodiments of the present application provide a pose estimation method applied to a head-mounted display device, the head-mounted display device comprising a visual sensor, an inertial sensor and a processor, the processor being configured with a first thread, a second thread and a buffer module, the frame rate of the inertial sensor being greater than the frame rate of the visual sensor; the method comprising:

[0007] acquire a current frame image collected by a visual sensor;

[0008] position a feature point in the current frame image through a first thread to obtain position information of the feature point;

[0009] store inertia data detected by the inertia sensor between the current frame image and a historical frame image to a buffer module;

[0010] update, through the first thread, a first pose of the head-mounted display device at a time corresponding to the historical frame image according to the position information and the inertia data to obtain a second pose;

[0011] predict, through a second thread, a pose of the head-mounted display device at a time corresponding to the current frame image as a target pose according to target inertia data of the inertia sensor at a current time and the second pose.

[0012] Optionally, the pose estimation method can comprise outputting the target pose according to a frame rate of the inertia sensor.

[0013] Optionally, the updating, through the first thread, of the first pose of the head-mounted display device at the time corresponding to the historical frame image according to the position information and the inertia data to obtain the second pose can comprise:

[0014] preprocessing, through the first thread, the inertia data to obtain preprocessing data, the preprocessing data being used to represent a degree of change in the pose of the head-mounted display device between the time corresponding to the current frame image and the time corresponding to the historical frame image;

[0015] updating, through the first thread, the first pose according to the preprocessing data and the position information by calling a preset target optimization model to obtain the second pose.

[0016] The above scheme can realize fusion of the position of the feature point and the inertia data, so that the data after fusion can be used to accurately deduce the pose. Furthermore, the first pose can be updated through the target optimization model, so that the efficiency of updating the pose can be further improved.

[0017] Optionally, the updating, through the first thread, of the first pose according to the preprocessing data and the position information by calling the preset target optimization model to obtain the second pose can comprise:

[0018] determining, through the first thread, a first prediction error according to the preprocessing data and the first pose;

[0019] determining, through the first thread, a second prediction error according to the position information and a predicted position of the feature point, the position information being true position information of the feature point, and the predicted position being information obtained by predicting the position of the feature point;

[0020] The first thread is used to call a target optimization model, and the first pose is updated according to the first prediction error and the second prediction error, so as to obtain a second pose.

[0021] The first prediction error and the second prediction error can be used to accurately optimize the first pose, so that the second pose obtained through optimization can obtain a smooth, robust and high-precision pose estimation result.

[0022] Optionally, the first thread is used to determine the first prediction error according to the preprocessed data and the first pose, including: the first thread is used to predict the pose of the head-mounted display device at the moment corresponding to the current frame image according to the preprocessed data and the first pose, so as to obtain a third pose.

[0023] The first thread is used to determine the first prediction error according to the preprocessed data, the first pose and the third pose.

[0024] The preprocessed data, the first pose and the third pose can be used to determine the first prediction error corresponding to the inertial data, so that the pose can be constrained through the first prediction error, and the accuracy of the pose prediction can be improved.

[0025] Optionally, the first thread is used to determine the first prediction error according to the preprocessed data, the first pose and the third pose, including:

[0026] The first thread is used to determine a rotation prediction error according to a rotation increment in the preprocessed data, a historical rotation state in the first pose and a rotation state in the third pose.

[0027] The first thread is used to determine a speed prediction error according to a speed increment in the preprocessed data, a historical speed state in the first pose and a speed state in the third pose.

[0028] The first thread is used to determine a translation prediction error according to a translation increment in the preprocessed data, a historical translation state in the first pose and a translation state in the third pose.

[0029] The first thread is used to determine the first prediction error according to the rotation prediction error, the speed prediction error and the translation prediction error. Optionally, the position information includes first pixel coordinates and spatial coordinates corresponding to the feature points, the predicted position includes second pixel coordinates corresponding to the feature points, the first thread is used to determine the second prediction error according to the position information and the predicted position of the feature points, including:

[0030] The first thread is used to project the spatial coordinates according to the parameters of the visual sensor, so as to obtain the second pixel coordinates of the feature points; and the second prediction error is determined according to the relative distance between the first pixel coordinates and the second pixel coordinates.

[0031] The second prediction error is determined by the relative distance between the pixel coordinates, which can further constrain the pose, and further improve the accuracy of the pose prediction.

[0032] Optionally, after storing the inertial data detected by the inertial sensor between the current frame image and the historical frame image into the buffer module, the method further includes:

[0033] When the current frame image is detected, the first thread is woken up.

[0034] The first thread reads the inertial data in the buffer module when the buffer module is not occupied.

[0035] The above scheme can realize exclusive access to the inertial data in the buffer module, avoid resource waste caused by thread competition for resources, and further improve the attitude derivation efficiency and ensure the safety of thread access.

[0036] Optionally, the target pose of the head-mounted display device at the time corresponding to the current frame image is predicted by the second thread according to the target inertial data of the inertial sensor at the current time and the second pose, including:

[0037] The second thread obtains the updated bias parameter of the inertial sensor from the first thread.

[0038] The second thread predicts the pose of the head-mounted display device at the time corresponding to the current frame image according to the second pose, the target inertial data of the inertial sensor at the current time, and the updated bias parameter, to obtain the target pose.

[0039] The above scheme predicts the pose of the head-mounted display device at the time corresponding to the current frame image in combination with the updated bias parameter, which can reduce the bias error and further improve the accuracy of the predicted pose. Moreover, by inlining the calculation process of the target pose into the second thread instead of concentrating it in the first thread, the computing power can be dispersed, and the consumption of computing power resources can be reduced.

[0040] Optionally, the first thread is used to locate the feature points in the current frame image to obtain the position information of the feature points, including:

[0041] The first thread is used to locate the feature points in the current frame image to obtain the first pixel coordinates of the feature points in the current frame image.

[0042] The first thread is used to map the feature points according to the first pixel coordinates to obtain the spatial coordinates of the feature points in the world coordinate system.

[0043] The first thread is used to determine the position information according to the first pixel coordinates and the spatial coordinates.

[0044] The above scheme can accurately represent the real position of the feature point through the first pixel coordinate and the spatial coordinate, further enable the real position of the feature point to accurately determine the second prediction error, and thus improve the accuracy of the pose prediction. Optionally, the above mapping, by the first thread, the feature point according to the first pixel coordinate, to obtain the spatial coordinate of the feature point in the world coordinate system, includes:

[0045] The mapping relationship between the first pixel coordinate and the spatial coordinate is obtained by the first thread;

[0046] The spatial coordinate of the feature point in the world coordinate system is obtained by mapping the first pixel coordinate according to the mapping relationship by the first thread.

[0047] The above scheme can achieve accurate mapping of the spatial coordinate corresponding to the feature point, and improve the accuracy of the position of the feature point.

[0048] Optionally, the above mapping, by the first thread, the feature point in the current frame image, to obtain the first pixel coordinate of the feature point in the current frame image, includes:

[0049] The current frame image is preprocessed by the first thread to obtain a preprocessed image;

[0050] The first pixel coordinate is obtained by positioning the feature point in the preprocessed image according to the historical frame image in the head-mounted display device by the first thread.

[0051] The above scheme can achieve fast positioning of the feature point by positioning the feature point in the preprocessed image through the historical frame image, and improve the efficiency of obtaining the first pixel coordinate.

[0052] Optionally, the frame rate of the first thread is the same as the frame rate of the visual sensor, and the frame rate of the second thread is the same as the frame rate of the inertial sensor.

[0053] In a second aspect, the embodiments of the present application provide a method for generating a virtual display image, applied to a head-mounted display device, the head-mounted display device comprising a visual sensor, an inertial sensor and a processor, the processor being configured with a first thread, a second thread and a buffer module, the frame rate of the inertial sensor being greater than the frame rate of the visual sensor; the method comprising:

[0054] Obtaining a current frame image collected by the visual sensor;

[0055] The position information of the feature point is obtained by positioning the feature point in the current frame image by the first thread;

[0056] The inertial data detected by the inertial sensor between the current frame image and the historical frame image is stored in the buffer module;

[0057] update, according to the position information and the inertial data, a first pose of the head-mounted display device at a time corresponding to the historical frame image to obtain a second pose;

[0058] update, according to the position information and the inertial data, a first pose of the head-mounted display device at a time corresponding to the historical frame image to obtain a second pose;

[0059] update, according to the position information and the inertial data, a first pose of the head-mounted display device at a time corresponding to the historical frame image to obtain a second pose;

[0060] update, according to the position information and the inertial data, a first pose of the head-mounted display device at a time corresponding to the historical frame image to obtain a second pose;

[0061] obtain a current frame image collected by the vision sensor;

[0062] update, according to the position information and the inertial data, a first pose of the head-mounted display device at a time corresponding to the historical frame image to obtain a second pose;

[0063] update, according to the position information and the inertial data, a first pose of the head-mounted display device at a time corresponding to the historical frame image to obtain a second pose;

[0064] update, according to the position information and the inertial data, a first pose of the head-mounted display device at a time corresponding to the historical frame image to obtain a second pose;

[0065] update, according to the position information and the inertial data, a first pose of the head-mounted display device at a time corresponding to the historical frame image to obtain a second pose.

[0066] update, according to the position information and the inertial data, a first pose of the head-mounted display device at a time corresponding to the historical frame image to obtain a second pose;

[0067] obtain a current frame image collected by the vision sensor;

[0068] update, according to the position information and the inertial data, a first pose of the head-mounted display device at a time corresponding to the historical frame image to obtain a second pose;

[0069] update, according to the position information and the inertial data, a first pose of the head-mounted display device at a time corresponding to the historical frame image to obtain a second pose;

[0070] The first pose of the head-mounted display device at the time corresponding to the historical frame image is updated according to the position information and the inertial data by the first thread, and a second pose is obtained;

[0071] The pose of the head-mounted display device at the time corresponding to the current frame image is predicted as a target pose according to the target inertial data of the inertial sensor at the current time and the second pose by the second thread.

[0072] A virtual display image of a scene in which the head-mounted display device is located is rendered according to the target pose.

[0073] In a fifth aspect, the embodiments of the present application further provide a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is adapted to be loaded by a processor to execute the pose estimation method in the first aspect.

[0074] In a sixth aspect, the embodiments of the present application further provide a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is adapted to be loaded by a processor to execute the virtual display image generation method in any of the second aspect.

[0075] In a seventh aspect, the embodiments of the present application further provide a computer program product, including a computer program or instructions, and the computer program or instructions are executed by a processor to execute the pose estimation method in any of the first aspect.

[0076] In an eighth aspect, the embodiments of the present application further provide a computer program product, including a computer program or instructions, and the computer program or instructions are executed by a processor to execute the virtual display image generation method in any of the second aspect. Advantages

[0077] In the embodiment of the present application, the first thread is used to track and locate the feature points in the current frame image obtained by the visual sensor in the head-mounted display device, and the position information of the feature points is obtained. In this way, the image data can be processed by the first thread alone without occupying the resources of the original thread of the visual sensor. The obtained image data can be shared in real time with other threads or algorithm modules in the head-mounted display device, and the fast acquisition of the position of the feature points can be ensured. Therefore, the head-mounted display device can complete the feature tracking task on different types of processors with extremely low CPU occupancy and in a short time. Meanwhile, since the frame rate of the inertial sensor in the head-mounted display device is higher than that of the visual sensor, a large amount of inertial data can be collected by the inertial sensor between the corresponding time points of the current frame image and the historical frame image. Therefore, in order to reduce the occupancy of the computing resources and the power consumption of the processor, the inertial data detected by the inertial sensor between the current frame image and the historical frame image is stored in the buffer module, so as to reduce the occupancy of the computing resources of the processor. According to the position information of the feature points and the inertial data obtained from the buffer module, the first pose corresponding to the historical frame image is updated by the first thread to obtain the second pose. In this way, the data fusion of multiple sensors can be realized by the first thread, the accuracy of the pose prediction can be improved, and the estimation of the multi-degree-of-freedom attitude can be completed in real time at the frame rate of the visual sensor while the computing power remains almost unchanged. Moreover, since the frame rate of the visual sensor is lower than that of the inertial sensor, when the first thread fuses the data of multiple sensors at a lower frame rate than another thread, the waiting time of the processor can be increased, and the power consumption of the processor can be reduced. The second thread is the thread corresponding to the inertial sensor. According to the target inertial data of the inertial sensor at the current time and the second pose, the pose of the head-mounted display device at the time corresponding to the current frame image is predicted as the target pose by the second thread. In this way, part of the calculation process of the pose can be integrated into the second thread, instead of concentrating all the pose calculation processes in one thread, so that the computing power can be dispersed. In this way, the accurate prediction and real-time output of the pose can be realized by occupying a small amount of computing resources, so that the chip with low cost and low power consumption can be deployed. BRIEF DESCRIPTION OF DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0079] FIG. 1 is a flowchart of a pose estimation method provided by the present application;

[0080] FIG. 2 is a flow diagram of a camera-based pose estimation provided by the present application;

[0081] FIG. 3 is a flow diagram of a method for generating a virtual display image provided by the present application;

[0082] FIG. 4 is a structural diagram of a head-mounted display device provided by an embodiment of the present application;

[0083] FIG. 5 is a structural diagram of a pose estimation apparatus provided by an embodiment of the present application;

[0084] FIG. 6 is a structural diagram of an electronic device provided by an embodiment of the present application. Embodiments of the present application

[0085] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0086] The embodiments of the present application provide a pose estimation method, a method for generating a virtual display image, and related devices. The related devices can be integrated in an electronic device, which can be a server or a terminal or the like.

[0087] The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0088] The terminal can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, or the like, but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application.

[0089] In addition, "multiple" in the embodiments of the present application refers to two or more. "First" and "second" and the like in the embodiments of the present application are used to distinguish the description, and cannot be understood as implying relative importance.

[0090] The following will be described in detail. It should be noted that the order of the following embodiments is not limited as the preferred order of the embodiments.

[0091] Traditional visual-based head pose estimation often relies on image data captured by a camera. However, a single visual sensor is susceptible to changes in environmental lighting, motion blur, and other factors, resulting in inaccurate head pose estimation results. Head pose estimation based on inertial sensors, on the other hand, can accumulate errors, making it difficult to effectively estimate object movement and accurately predict head pose. Therefore, head pose estimation based on a single sensor is often inaccurate. Embodiments of the present application propose to combine multiple sensors to estimate head pose, fuse the data characteristics of multiple sensors, and improve the accuracy of head pose estimation.

[0092] Furthermore, in head-mounted display devices that include spatial computing, low-latency and high-precision head pose estimation is a core requirement and basic function of such devices. To meet the requirements of low-latency and high-precision head pose estimation, related technologies apply a simultaneous localization and mapping (SLAM) algorithm architecture to head-mounted display devices. The algorithm architecture includes sensor data processing, visual odometry processing, backend optimization, mapping, and loop detection. Its workflow can be divided into at least three threads: feature tracking, sensor data processing, and mapping optimization. The operation of at least three threads requires a large amount of computing resources. That is, the current SLAM algorithm architecture applied to head-mounted display devices has very high requirements for device processors and chips, consumes a large amount of computing resources, and high-performance chips mean high costs. Therefore, embodiments of the present application propose a lightweight algorithm architecture that occupies fewer resources and consumes less power. This architecture not only meets the high-precision and real-time update requirements of head-mounted display devices for head pose estimation, but also meets the low-latency requirements for image display rendering. Moreover, it can be deployed in low-power chips, further reducing costs and power consumption.

[0093] Specifically, the embodiment of the present application provides a pose estimation method, which is applied to a head-mounted display device. The head-mounted display device comprises a visual sensor, an inertial sensor and a processor. The processor is configured with a first thread, a second thread and a buffer module. The first thread is used to track and locate feature points in a current frame image obtained by the visual sensor in the head-mounted display device, so as to obtain position information of the feature points. In this way, the image data can be processed by the first thread alone without occupying the resources of the original thread of the visual sensor. The image data obtained can be shared in real time with other threads or algorithm modules in the head-mounted display device. In addition, the position information of the feature points can be obtained quickly. Therefore, the head-mounted display device can complete the feature tracking task on different types of processors with extremely low CPU single-core occupancy and in a short time. Meanwhile, since the frame rate of the inertial sensor in the head-mounted display device is higher than that of the visual sensor, a large amount of inertial data can be collected by the inertial sensor between the corresponding time points of the current frame image and a historical frame image captured by the visual sensor. Therefore, in order to reduce the occupancy of the computing resources and the power consumption of the processor, the buffer module is used to store the inertial data detected by the inertial sensor between the current frame image and the historical frame image. In this way, the occupancy of the computing resources of the processor can be reduced. According to the position information of the feature points and the inertial data obtained from the buffer module, the first pose of the head-mounted display device at the time point corresponding to the historical frame image is updated by the first thread, and a second pose is obtained. In this way, the data fusion of multiple sensors can be realized by the first thread, the accuracy of the pose prediction can be improved, and the estimation of the multi-degree-of-freedom attitude can be completed in real time at the frame rate of the visual sensor while the computing power remains almost unchanged. In addition, since the frame rate of the visual sensor is lower than that of the inertial sensor, when the first thread fuses the data of multiple sensors at a lower frame rate than another thread, the waiting time of the processor can be increased, and the power consumption of the processor can be reduced. The second thread is a thread corresponding to the inertial sensor. According to the target inertial data of the inertial sensor at the current time point and the second pose obtained from the first thread, the pose of the head-mounted display device at the time point corresponding to the current frame image is predicted as a target pose by the second thread. In this way, part of the calculation process of the pose can be integrated into the second thread, instead of concentrating all the pose calculation processes in one thread, so that the computing power can be dispersed. The pose estimation method in the embodiment of the present application can realize accurate prediction and real-time output of the pose by occupying a small amount of computing resources, so that the pose estimation method can be deployed on a chip with low cost and low power consumption.

[0094] Referring to FIG. 1, FIG. 1 is a flowchart of a pose estimation method according to an embodiment of the present application. The pose estimation method is applied to a head-mounted display device. The head-mounted display device comprises a visual sensor, an inertial sensor and a processor. The processor is configured with a first thread, a second thread and a buffer module. The frame rate of the inertial sensor is higher than that of the visual sensor. The pose estimation method comprises the following steps.

[0095] S101, acquire a current frame image collected by a vision sensor.

[0096] In the embodiment, the pose estimation method can be applied to a head-mounted display device, and can also be applied to other devices with a simulated reality scene function. The head-mounted display device includes but is not limited to a vision sensor, an inertial sensor, a processor, and the like. The vision sensor is used to collect images, such as a camera or a laser. The current frame image is an image collected by the vision sensor, such as a left eye image and a right eye image captured by a camera. The inertial sensor (IMU) is used to detect and measure the acceleration and rotational motion of the device, and the principle is to realize it through the law of inertia.

[0097] In some embodiments, the motion state of the device, such as acceleration and rotation, can also be collected by other sensors, such as an acceleration sensor, a speed sensor, and the like.

[0098] The vision sensor and the inertial sensor in the head-mounted display device are synchronized using the same clock source, and the frame rate of the inertial sensor is higher than that of the vision sensor. The frame rate of the vision sensor can be between 20 hz and 60 hz, or can be 20 hz or 60 hz. The frame rate of the inertial sensor or other sensors used to obtain the motion state of the device can be between 100 hz and 1000 hz, for example, can be 100 hz, 500 hz, or 1000 hz.

[0099] S102, through a first thread, locate a feature point in the current frame image to obtain position information of the feature point.

[0100] In the embodiment, the first thread (such as a tracking thread) is a thread in the head-mounted display device for processing images, data fusion, and pose calculation. The frame rate of the first thread corresponds to the frame rate of the vision sensor, such as a positive proportional relationship between the frame rate of the first thread and the frame rate of the vision sensor. In some embodiments, the frame rate of the first thread is equal to the frame rate of the vision sensor. The first thread is responsible for most of the data operations in the head-mounted display device.

[0101] Through the first thread corresponding to the vision sensor in the head-mounted display device, the feature point in the current frame image is tracked, and the position information of the feature point in the current frame image can be obtained. The feature point is a pixel point with a significant feature in the image, and the pixel points in the current frame image are screened through a preset pixel point screening strategy to obtain the feature point. The position information includes the pixel coordinates of the feature point in the current frame image and the spatial coordinates of the feature point in the three-dimensional space. The feature point can be located through the position information.

[0102] Specifically, in some embodiments, the acquired current frame image is subjected to pyramid image processing by the first thread to obtain a pyramid image of the current frame; then, sub-pixel matching is performed between a historical feature point in a historical frame image and a feature point in the current frame image to determine the position of the feature point in the pyramid image of the current frame. It is determined whether the current frame image is a key frame image, and if the current frame image is a key frame image, the feature point of the key frame image is extracted to perform depth calculation based on the feature point. In some embodiments, if the current frame image includes a left-eye image and a right-eye image, one image (such as the left-eye image) in the current frame image can be subjected to feature point extraction and then matching. For example, the feature point is extracted by a fast corner detection algorithm (Features from accelerated segment test, FAST) or a Harris corner detection algorithm (Harris-Corner), so that a low-end processor can also perform efficient feature point extraction. At the same time, in order to improve the efficiency and accuracy of pose derivation, interval blocking can be used to make the extracted feature points uniformly distributed in the image. Then, according to the relative position of the binocular camera after calibration, the epipolar line between the left-eye image and the right-eye image is determined; matching calculation is performed on the epipolar line to obtain binocular matching points. In this way, all feature points in the current frame image are obtained. The matching calculation on the epipolar line can be realized by normalized cross correlation (Normalized cross correlation, NCC), single shot multi box detector (Single Shot MultiBox Detector, SSD), sum of absolute differences (Sum of Absolute Differences, SAD), or the like.

[0103] S103, store the inertia data detected by the inertia sensor between the current frame image and the historical frame image to the buffer module. In this embodiment, the inertia data refers to the data collected by the inertia sensor in the head-mounted display device between the time corresponding to the current frame image and the time corresponding to the historical frame image. The historical frame image can be a frame image adjacent to the current frame image. The inertia data specifically includes the zero bias corresponding to the gyroscope and the accelerometer in the inertia sensor, and the readings on the inertia sensor body. Since the frame rate of the inertia sensor is higher than that of the vision sensor, the inertia sensor will collect a large amount of data between the time corresponding to the current frame image and the time corresponding to the historical frame image. By the buffer module, the inertia data detected by the inertia sensor between the current frame image and the historical frame image is stored, which can realize the caching of the inertia data.

[0104] The buffer module can be a memory on the processor, and when the processor needs to obtain data in the buffer module, the data can be quickly read directly from the buffer module. In some embodiments, the buffer module can also be a separate module in the head-mounted display device, such as a memory, which can provide larger storage space for data. In other embodiments, the buffer module can also be a module embedded in the second thread, which is used to more quickly receive inertial data.

[0105] S104, updating, by the first thread, the first pose of the head-mounted display device at the time corresponding to the historical frame image according to the position information and the inertial data, to obtain a second pose.

[0106] In this embodiment, the first pose is the pose of the head-mounted display device at the time corresponding to the historical frame image. The first pose at least includes any one of the rotation state of the head-mounted display device at the historical frame, the velocity state of the head-mounted display device at the historical frame in the world coordinate system, and the translation state of the head-mounted display device at the historical frame in the world coordinate system. By the first thread, the first pose of the head-mounted display device is updated according to the position information of the feature points and the inertial data obtained from the buffer module, that is, the second pose of the head-mounted display device is obtained. The second pose can be obtained by updating the first pose of the head-mounted display device according to the obtained position information and inertial data through a preset target optimization model or a Kalman filter.

[0107] The second thread is a thread corresponding to the inertial sensor in the head-mounted display device, which is used to process the data of the inertial sensor in the head-mounted display device. The frame rate of the second thread corresponds to the frame rate of the inertial sensor, such as a positive proportional relationship between the frame rate of the second thread and the frame rate of the inertial sensor. In some embodiments, the frame rate of the second thread is equal to the frame rate of the inertial sensor.

[0108] It can be understood that if the frame rate of the inertial sensor is greater than the frame rate of the visual sensor, the frame rate of the second thread corresponding to the inertial sensor is greater than the frame rate of the first thread corresponding to the visual sensor.

[0109] In some embodiments, when the first thread obtains the second pose, the second thread waiting in the head-mounted display device can be awakened. For example, the waiting function of the condition variable corresponding to the second pose is called by the first thread, and according to the waiting function, the second thread is awakened and the mutual exclusion lock is passed to the second thread, so that the second thread can timely perform pose prediction according to the obtained mutual exclusion lock, and avoid wasting too much time and competing with other threads for computing resources.

[0110] S105, predicting, by the second thread, a pose of the head-mounted display device at the time corresponding to the current frame image as a target pose according to the target inertial data of the inertial sensor at the current time and the second pose.

[0111] The target inertial data refers to the inertial data of the inertial sensor at the current time.

[0112] The second thread is usually used to drive the inertial sensor and read the data of the inertial sensor. In the embodiment, the pose prediction of the head-mounted display device at the time corresponding to the current frame image is integrated into the second thread, so that the computing power is dispersed, and the second thread can efficiently and accurately predict the pose of the head-mounted display device at the time corresponding to the current frame image according to the obtained second pose and the target inertial data. Specifically, the prediction model or the pre-integration function is called by the second thread, and the pose of the head-mounted display device at the time corresponding to the current frame image is predicted according to the obtained second pose and the target inertial data of the inertial sensor at the current time, to obtain a target pose. The target pose includes a multi-degree-of-freedom attitude, such as a six-degree-of-freedom attitude of the head-mounted display device at the time corresponding to the current frame image, which specifically includes three rotational degrees of freedom (pitch angle, roll angle, yaw angle) and three translational degrees of freedom (displacement of X, Y, and Z axes), and velocity (X, Y, and Z axes), and the multi-degree-of-freedom attitude can be represented by a rotational state, a velocity state, and a translational state. The rotational state can be represented by a rotation matrix R wb , which is used to represent the mapping and rotation relationship between the world coordinate system and the body coordinate system of the sensor in a three-dimensional space; the translational state can be represented by p w , which is used to represent the translational vector of the head-mounted display device in the world coordinate system; and the velocity state can be represented by v w , which is used to represent the velocity of the head-mounted display device in the world coordinate system.

[0113] Further, since real-time rendering usually requires more than 60 Hz, the frame rate of the inertial sensor in the embodiment can be set to more than 100 Hz, and the frame rate of the visual sensor can be set to more than 20 Hz. Therefore, the pose of the head-mounted display device at the time corresponding to the current frame image is predicted by the second thread and then output, so that real-time rendering of the scene in the head-mounted device can be realized, and the delay is reduced. In addition, in the central processing unit (CPU) scheduling of the head-mounted display device, the priority of the second thread can be set to the highest priority, so as to avoid the second thread competing with other threads for the resources of the CPU, and ensure that the pose output by the second thread has a stable frequency.

[0114] The embodiment tracks and positions the feature points in the current frame image obtained by the visual sensor in the head-mounted display device through the first thread, obtains the position information of the feature points, so that the image data is processed by the first thread alone without occupying the resources of the original thread of the visual sensor. Not only can the obtained image data be shared in real time to other threads or algorithm modules in the head-mounted display device, but also the fast acquisition of the position of the feature points can be ensured, so that the head-mounted display device can complete the feature tracking task on different types of processors with extremely low CPU occupancy and less time. At the same time, since the frame rate of the inertial sensor in the head-mounted display device is greater than that of the visual sensor, between the corresponding moments when the visual sensor captures the current frame image and the historical frame image, the inertial sensor will collect a large amount of inertial data. Therefore, in order to reduce the occupation of computing resources and reduce the power consumption of the processor, the inertial data detected by the inertial sensor between the current frame image and the historical frame image is stored through the buffer module, which can reduce the occupation of the processor computing resources. Through the first thread, the first pose corresponding to the historical frame image of the head-mounted display device is updated according to the position information of the feature points and the inertial data obtained from the buffer module to obtain the second pose, so that the data fusion of multiple sensors can be realized through the first thread, the accuracy of pose prediction is improved, and the estimation of multi-degree-of-freedom attitude can be completed in real time at the frame rate of the visual sensor while the computing power remains almost unchanged. Moreover, since the frame rate of the visual sensor is smaller than that of the inertial sensor, when the first thread fuses the data of multiple sensors at a lower frame rate than another thread, the waiting time of the processor can be increased and the power consumption of the processor can be reduced. The second thread is the thread corresponding to the inertial sensor. Through the second thread, the pose of the head-mounted display device at the moment corresponding to the current frame image is predicted as the target pose according to the target inertial data of the inertial sensor at the current moment and the second pose, which can realize the dispersion of computing power by inlining part of the pose calculation process into the second thread instead of concentrating all the pose calculation processes in one thread. Therefore, the accurate prediction and real-time output of the pose can be realized by occupying a small amount of computing resources, so that it can be deployed on a low-cost and low-power chip.

[0115] In some embodiments of the present application, in order to realize the real-time output of the pose of the head-mounted display device and the subsequent real-time display rendering, the target pose is output at the frame rate of the inertial sensor.

[0116] The frame rate of the inertial sensor in the head-mounted display device is taken as the output frame rate of the pose in the embodiments of the present application, because the frame rate of the inertial sensor is greater than the frame rate of the visual sensor, so that the output of the predicted target pose can be output at a higher frame rate of the inertial sensor, the output frame rate of the pose is improved, the delay of the attitude calculation is minimized, the image output delay of the head-mounted display device is further reduced, and subsequent real-time image display rendering of the head-mounted display device is realized.

[0117] In some embodiments of the present application, the above updating, by the first thread, of the first pose of the head-mounted display device at the time corresponding to the historical frame image according to the position information and the inertial data to obtain the second pose comprises:

[0118] The first thread is used to pre-process the inertial data to obtain pre-processed data, and the pre-processed data is used to represent the degree of change in the pose of the head-mounted display device between the time corresponding to the current frame image and the time corresponding to the historical frame image.

[0119] The first thread is used to call a preset target optimization model to update the first pose according to the pre-processed data and the position information to obtain the second pose.

[0120] In the embodiments, the inertial data includes data of the inertial sensor between the time corresponding to the adjacent two frame images, and the pre-processed data of the head-mounted display device can be obtained by pre-integrating the inertial data. The pre-processed data includes but is not limited to rotation increment, velocity increment and translation increment, and the pre-processed data is used to represent the degree of change in the pose of the head-mounted display device between the time corresponding to the current frame image and the time corresponding to the historical frame image.

[0121] In some embodiments, the first thread can be used to calculate the rotation increment, the velocity increment and the translation increment between the adjacent two frames according to the inertial data through the following formulas.

[0122] wherein, ΔR ij is the rotation increment, Δv ij is the velocity increment, Δp ij is the translation increment, b g is the zero bias of the gyroscope in the inertial sensor in the inertial data (usually a 3x1 vector), b a is the zero bias of the accelerometer in the inertial sensor (usually a 3x1 vector), w b (t) is the angular velocity reading on the inertial sensor body at time t (usually a 3x1 vector), a b (t) is the acceleration reading on the inertial sensor body at time t (usually a 3x1 vector), i, j, k are time, t is a time point (usually in seconds), and Δt is a time interval.

[0123] The target optimization model can be an optimization model constructed based on an optimization problem, or an artificial intelligence model constructed. The target optimization model is called by the first thread to update the first pose of the head-mounted display device according to the preprocessed data and the position information, and a second pose of the head-mounted display device is obtained. Specifically, the first pose includes a rotation state of the head-mounted display device at a historical frame, a velocity state of the head-mounted display device at the historical frame in a world coordinate system, and a translation state of the head-mounted display device at the historical frame in the world coordinate system. Therefore, the second pose includes an updated rotation state of the rotation state in the first pose, an updated velocity state of the velocity state in the first pose, and an updated translation state of the translation state in the first pose.

[0124] In some embodiments, the bias of the inertial sensor also changes over time, which also causes random walk changes. Therefore, the second pose can also include updated zero biases of the gyroscope and the accelerometer in the inertial sensor, respectively. The target optimization model is called by the first thread to update the first pose of the head-mounted display device according to the preprocessed data and the position information, and all the second poses can be obtained.

[0125] Further, in some embodiments, in order to keep the amount of calculation within a controllable range, the dimension size of the second pose to be estimated can be controlled by a time sliding window. Specifically, the length of the time sliding window can be L, and the set of second poses to be estimated in the target optimization model is the second poses within the length range of the time sliding window. The second poses within the length range of the time sliding window can be represented as follows: X i ={R wb (i),v w (i),p w (i),b g (i),b a (i)} T

[0126] wherein L is the length of the time sliding window, X i is the second pose at time i, the subscript w represents the world coordinate system, the subscript b represents the body coordinate system of the inertial sensor, R wb (i) is the rotation state at time i (usually a 3x3-dimensional rotation matrix), p w (i) is the translation state at time i in the world coordinate system (usually a 3x1-dimensional vector), v w (i) is the velocity state at time i in the world coordinate system (usually a 3x1-dimensional vector), b g (i) is the zero bias of the gyroscope in the inertial sensor at time i, and b a(i) is the zero offset of the accelerometer in the inertial sensor at time i.

[0127] The embodiment updates the first pose of the head-mounted display device according to the preprocessed data and the position information through the target optimization model in the first thread, obtains the second pose of the head-mounted display device, realizes the fusion of the feature point position and the inertial data, and improves the accuracy of the pose deduction. Moreover, when the first thread fuses the feature point position and the inertial data, the fusion can be performed at the frame rate of the visual sensor. Since the frame rate of the visual sensor is less than the frame rate of the inertial sensor, the second thread can complete the processing of one frame at a faster speed. When the first thread fuses at the frame rate of the visual sensor, the processing speed of the first thread is slower than that of the first thread. Therefore, by fusing the feature point position and the inertial data at the frame rate of the visual sensor through the first thread, the waiting time of the processor can be increased, and the power consumption of the processor can be reduced. In some embodiments of the present application, the above-mentioned updating the first pose to obtain the second pose through the first thread and calling the preset target optimization model according to the preprocessed data and the position information comprises:

[0128] determining, through the first thread, the first prediction error according to the preprocessed data and the first pose;

[0129] determining, through the first thread, the second prediction error according to the position information and the predicted position of the feature point, the position information being the real position information of the feature point, and the predicted position being the information obtained by predicting the position of the feature point;

[0130] updating, through the first thread and according to the first prediction error and the second prediction error, the first pose to obtain the second pose.

[0131] In the embodiment, the first prediction error refers to a prediction error generated based on the inertial sensor. Through the first thread, the pose of the head-mounted display device at the time corresponding to the current frame image can be predicted according to the preprocessed data and the first pose, to obtain a third pose of the head-mounted display device at the time corresponding to the current frame image. The third pose can be specifically calculated according to the preprocessed data and the first pose through a preset prediction model, such as a trained state prediction model, or pre-integration. Further, through the first thread, the first prediction error can be determined according to the preprocessed data, the first pose and the third pose. For example, through the first thread, the first prediction error is obtained by performing residual calculation according to the preprocessed data, the first pose and the third pose. The second prediction error refers to an error of a predicted position of a feature point. Through the first thread, the second prediction error can be determined according to the position information and the predicted position of the feature point. The position information is the real position information of the feature point, and the predicted position is the information obtained by predicting the position of the feature point. Specifically, in some embodiments, the predicted position of the feature point can be a pixel coordinate of the feature point predicted in the image, or a spatial coordinate of the feature point predicted in the world coordinate system. The second prediction error can be determined by calculating the predicted pixel coordinate and the pixel coordinate in the position information corresponding to the feature point. Alternatively, the second prediction error can also be determined by calculating the predicted spatial coordinate and the spatial coordinate in the position information corresponding to the feature point.

[0132] The first pose of the head-mounted display device is updated according to the first prediction error and the second prediction error through the target optimization model in the first thread, to obtain an updated first pose, i.e., a second pose of the head-mounted display device. Thus, by adjusting the pose through the error, the obtained second pose can avoid interference of various noises, and further make the prediction of the pose of the head-mounted display device at the time corresponding to the current frame image more accurate.

[0133] In some embodiments, the target optimization model can be represented by the following objective function: P i ={x i ,y i ,z i} T

[0134] wherein, is the first prediction error, is the second prediction error, t, i, j are time, P i is the spatial coordinate of the i-th point in the three-dimensional space.

[0135] The first pose of the head-mounted display device is updated according to the obtained first prediction error and second prediction error through the target optimization model in the first thread, to obtain a second pose of the head-mounted display device.

[0136] The embodiment updates the first pose of the head-mounted display device according to the first prediction error and the second prediction error through the target optimization model in the first thread, and obtains the second pose of the head-mounted display device, thereby realizing accurate optimization of the first pose of the head-mounted display device, and further enabling the second pose obtained through optimization to obtain a smooth, robust and high-precision pose estimation result.

[0137] In some embodiments of the present application, the determination of the first prediction error through the first thread according to the preprocessed data and the first pose includes:

[0138] The third pose of the head-mounted display device at the time corresponding to the current frame image is predicted through the first thread according to the preprocessed data and the first pose.

[0139] The first prediction error is determined through the first thread according to the preprocessed data, the first pose and the third pose.

[0140] In the embodiment, the third pose also includes the rotation state, the velocity state and the translation state of the head-mounted display device predicted in the current frame. With respect to the target pose, the third pose is the pose of the head-mounted display device at the time corresponding to the current frame image, which is predicted based on the first pose before optimization. Specifically, the preprocessed data includes the rotation increment, the velocity increment and the translation increment, and the first pose includes the historical rotation state, the historical velocity state and the historical translation state. Therefore, the rotation state in the third pose can be predicted by calculating the rotation increment in the preprocessed data and the historical rotation state in the first pose through the first thread according to the integral pose estimation. The velocity state in the third pose can be predicted by calculating the velocity increment in the preprocessed data and the historical velocity state in the first pose through the first thread according to the integral pose estimation. The translation state in the third pose can be predicted by calculating the translation increment in the preprocessed data and the historical translation state in the first pose through the first thread according to the integral pose estimation. The rotation state, the velocity state and the translation state in the third pose can be calculated by the following formulas: ΔR ij wb (i) T R wb (j) Δv ij wb (i) T (v w (j)-v w (t)-g w Δt ij )

[0141] wherein, ΔR ij is the rotation increment between i time and j time, and Δv​​ij is the velocity increment between i time and j time, Δp ij is the translation increment between i time and j time, R wb (i) is the rotation state at i time, R wb (j) is the rotation state at j time, v w (i) is the velocity state at i time, v w (j) is the velocity state at j time, g w is the gravity acceleration in the world coordinate system, Δt ij is the time interval between i time and j time, p w (j) is the translation state at j time, p w (i) is the translation state at i time.

[0142] Through the first thread, based on the rotation state, the velocity state and the translation state in the third pose obtained, respectively, residual calculation is performed with the rotation increment, the velocity increment and the translation increment in the preprocessed data corresponding to the rotation state, the velocity state and the translation state, and the historical rotation state, the historical velocity state and the historical translation state in the first pose, so as to obtain the rotation prediction error, the velocity prediction error and the translation prediction error of the head-mounted display device. Through the rotation prediction error, the velocity prediction error and the translation prediction error, the first prediction error of the head-mounted display device can be determined.

[0143] In some embodiments, the rotation prediction error can be calculated by the following formula: r ΔR = Log(ΔR ij R wb (j)TR wb (i))

[0144] wherein, ΔR ij is the rotation increment, R wb (j) is the rotation state at j time, which is the rotation state in the third pose in the embodiment, R wb (i) is the rotation state at i time, which is the historical rotation state in the embodiment.

[0145] In some embodiments, the velocity prediction error can be calculated by the following formula: r Δv = R wb (i) T (v w (j)-v w (i)-g w Δt ij )-Δv ij

[0146] wherein, R wb (i) is the historical rotation state in the embodiment, Δv ijis the velocity increment, v w (j) is the velocity state in the third pose, v w (i) is the historical velocity state, g w is the gravity acceleration in the world coordinate system, Δt ij is the time interval between the i-th moment and the j-th moment.

[0147] In some embodiments, the translation prediction error can be calculated by the following formula:

[0148] wherein, in the present embodiment, R wb (i) is the historical rotation state, Δp ij is the translation increment, p w (j) is the translation state in the world coordinate system at the j-th moment, which is the translation state in the third pose in the present embodiment; p w (i) is the translation state in the world coordinate system at the i-th moment, which is the historical translation state in the present embodiment; v w (i) is the velocity state in the world coordinate system at the i-th moment, which is the historical velocity state in the present embodiment; g w is the gravity acceleration in the world coordinate system, Δt ij is the time interval between the i-th moment and the j-th moment.

[0149] The first prediction error can be calculated by the following formula:

[0150] wherein, r ΔR is the rotation prediction error, r Δv is the velocity prediction error, r Δp is the translation prediction error.

[0151] The third pose, the preprocessed data and the first pose are used to determine the first prediction error of the inertial sensor of the head-mounted display device between the adjacent two frames of images in the present embodiment, so that the accuracy of the pose prediction can be further improved by constraining the pose by the first prediction error.

[0152] In some embodiments of the present application, the position information includes the first pixel coordinates and the spatial coordinates of the feature points, the predicted position includes the second pixel coordinates of the feature points, and the second prediction error is determined according to the position information and the predicted position of the feature points by the first thread, including:

[0153] The second pixel coordinates of the feature points are obtained by projecting the spatial coordinates according to the parameters of the visual sensor by the first thread, and the second prediction error is determined according to the relative distance between the first pixel coordinates and the second pixel coordinates.

[0154] In the embodiment, the visual sensor can be a camera, which corresponds to internal parameters and external parameters that can be calibrated for the camera. Therefore, the calibration parameters of the visual sensor include the internal parameters and the external parameters. The internal parameters include, but are not limited to, the focal length and the principal point of the camera, and the external parameters include, but are not limited to, the rotation matrix and the translation vector. According to the calibration parameters of the visual sensor, the second pixel coordinates of the feature points can be obtained by projecting the spatial coordinates of the feature points onto the image plane through the first thread. It can be understood that the second pixel coordinates refer to the pixel coordinates predicted for the positions of the feature points in the current frame image. The second prediction error can be obtained by calculating the error between the second pixel coordinates and the first pixel coordinates. The error can be calculated by the Euclidean distance or the normalized distance between the pixel coordinates.

[0155] The embodiment determines the second prediction error through the first pixel coordinates and the second pixel coordinates, so that the pose is further constrained through the second prediction error, and the accuracy of the pose prediction can be further improved.

[0156] In some embodiments of the present application, after the inertial data detected by the inertial sensor between the current frame image and the historical frame image is stored in the buffer module, the method further includes:

[0157] When the current frame image is detected, the first thread is woken up.

[0158] Through the first thread, the inertial data in the buffer module is read when the buffer module is not occupied.

[0159] In the embodiment, the historical frame image can be a frame image adjacent to the current frame image, such as the image of the previous frame of the current frame in the frame sequence, or the image of the key frame adjacent to the current frame in the frame sequence. Through the second thread corresponding to the inertial sensor in the head-mounted display device, the inertial data of the inertial sensor between the time corresponding to the current frame image and the time corresponding to the historical frame image is collected before the current frame image arrives, and the inertial data is stored in the buffer module of the head-mounted display device. Since the frame rate of the inertial sensor is much higher than the frame rate of the visual sensor, all the inertial data between the adjacent two frame images can be collected through the buffer module before the current frame image arrives.

[0160] When the current frame image is detected to arrive, i.e., when the current frame image is detected, the collection of the inertial data by the second thread is stopped, and the first thread waiting in the head-mounted display device is woken up. The first thread acquires a mutual exclusion lock, wherein the mutual exclusion lock is used to ensure that only one thread can access the inertial data of the buffer module at a time. If the first thread can acquire the mutual exclusion lock, it indicates that the buffer module is not occupied, and the first thread reads the inertial data from the buffer module and empties the buffer module.

[0161] The mutual exclusion lock can prevent multiple threads from simultaneously accessing the inertial data of the buffer module, realize exclusive access to the inertial data in the buffer module, avoid resource waste caused by thread competition for resources, further improve the attitude deduction efficiency, and ensure the safety of thread access.

[0162] In some embodiments of the present application, the above-mentioned prediction of the pose of the head-mounted display device at the time corresponding to the current frame image as the target pose by the second thread according to the target inertial data of the inertial sensor at the current time and the second pose includes:

[0163] The second thread acquires the updated bias parameter of the inertial sensor from the first thread.

[0164] The second thread predicts the pose of the head-mounted display device at the time corresponding to the current frame image according to the second pose, the target inertial data of the inertial sensor at the current time, and the updated bias parameter, to obtain the target pose.

[0165] In the present embodiment, since the bias of the inertial sensor also changes randomly with time, the prediction of the pose of the head-mounted display device at the time corresponding to the current frame image has an error. Therefore, the second thread sends the zero biases of the gyroscope and the accelerometer in the inertial sensor to the first thread. The target optimization model in the first thread updates the bias parameter of the inertial sensor based on the acquired zero biases of the gyroscope and the accelerometer in the inertial sensor, to obtain the updated bias parameter. The updated bias parameter at least includes the updated zero bias b g of the gyroscope and the updated zero bias b aWhen the first thread calculates the updated bias parameters, the waiting second thread is woken up, and the updated bias parameters are obtained by the second thread from the first thread. Then, the target pose of the head-mounted display device at the time corresponding to the current frame image is predicted by the second thread according to the updated bias parameters, the second pose and the target inertial data of the inertial sensor at the current time, to obtain the target pose of the head-mounted display device. The target pose can also be obtained by integrating the rotation, velocity and displacement respectively according to the updated bias parameters, the second pose and the target inertial data. The integral attitude estimation is as follows: R wb (t+△t)=R wb (t)Exp((ω b (t)-b g )△t) v w (t+△t)=v w (t)+g w Δt+R wb (t)(a b (t)--b a )△t

[0166] R wb (t+△t) is a rotation matrix for mutual conversion between the world coordinate system and the body coordinate system corresponding to the current frame, v w (t+△t) is the speed of the head-mounted display device at the time corresponding to the current frame in the world coordinate system, p w (t+△t) is the translation vector of the head-mounted display device at the time corresponding to the current frame in the world coordinate system, R wb (t) is the rotation state of the head-mounted display device in the second pose in the world coordinate system, v w (t) is the speed state of the head-mounted display device in the second pose in the world coordinate system, p w (t) is the translation state of the head-mounted display device in the second pose in the world coordinate system, b g is the zero offset of the gyroscope in the second pose, i.e., the updated zero offset of the gyroscope, b a is the zero offset of the accelerometer in the second pose, i.e., the updated zero offset of the accelerometer, ω b (t) is the angular velocity reading on the body of the inertial sensor at the current time, a b (t) is the acceleration reading on the body of the inertial sensor at the current time, g w is the gravitational acceleration in the world coordinate system.

[0167] The embodiment updates the bias parameter, the second pose and the inertial data, predicts the pose of the head-mounted display device at the time corresponding to the current frame image, and obtains the target pose of the head-mounted display device, thereby realizing accurate prediction of the target pose of the head-mounted display device and improving the accuracy of the target pose. Moreover, since the calculation process of the target pose requires less computing power, the calculation process of the target pose is inlaid in the second thread instead of being concentrated in the first thread, so that the computing power is dispersed and the consumption of computing power resources is reduced. Moreover, after the second thread obtains the target pose, the target pose can be output at the frame rate of the inertial sensor, thereby further realizing real-time updating and rendering of the pose.

[0168] In some embodiments of the present application, the above-mentioned positioning of the feature point in the current frame image by the first thread to obtain the position information of the feature point includes:

[0169] The first pixel coordinate of the feature point in the current frame image is obtained by the first thread positioning the feature point in the current frame image.

[0170] The spatial coordinate of the feature point in the world coordinate system is obtained by the first thread mapping the feature point according to the first pixel coordinate.

[0171] The position information is determined by the first thread according to the first pixel coordinate and the spatial coordinate.

[0172] In the embodiment, the first pixel coordinate refers to the two-dimensional coordinate of the feature point in the current frame image, such as (u, v). The first pixel coordinate of the feature point in the current frame image can be determined by the first thread positioning the feature point in the current frame image. The feature point is extracted from the first pixel coordinate, and the spatial coordinate of the feature point in the world coordinate system, such as the three-dimensional coordinate of the feature point in the world coordinate system, can be obtained by mapping the feature point according to the first pixel coordinate.

[0173] By determining the first pixel coordinate and the spatial coordinate, the real position of the feature point can be accurately represented, and the real position of the feature point can be used to accurately determine the second prediction error, thereby improving the accuracy of pose prediction.

[0174] In some embodiments of the present application, the above-mentioned mapping of the feature point according to the first pixel coordinate by the first thread to obtain the spatial coordinate of the feature point in the world coordinate system includes:

[0175] The mapping relationship between the first pixel coordinate and the spatial coordinate is obtained by the first thread.

[0176] The spatial coordinate of the feature point in the world coordinate system is obtained by the first thread mapping the first pixel coordinate according to the mapping relationship.

[0177] In the embodiment, the mapping relationship between the first pixel coordinates and the spatial coordinates can be characterized by target mapping parameters, and the spatial coordinates of the feature point in the world coordinate system can be obtained by mapping the first pixel coordinates through the target mapping parameters. The target mapping parameters can be parameters as shown below:

[0178] Wherein (u, v) is the first pixel coordinates of the feature point, and according to the mapping of the first pixel coordinates through the target mapping parameters, the spatial coordinates as shown below can be obtained:

[0179] Wherein (x, y, z) is the spatial coordinates, and η is the target mapping parameter.

[0180] Further, in some embodiments, according to the spatial coordinates and the inverse depth, the point cloud corresponding to the feature point can be parameterized to obtain the parameterized point cloud data. Specifically, in the parameterization of the traditional point cloud, the inverse depth representation method is often used to resist the degeneration of the feature point distance being too far and the parallax angle being too small, etc. However, this parameterization method is very dependent on the accuracy of the feature point in the first extraction. Due to the existence of image noise, the coordinate position of the feature point in the extraction often has certain error. In order to overcome this error, the spatial coordinates and the inverse depth ρ generated according to the first pixel coordinates are used to parameterize the point cloud, and then the point cloud q can be represented as q i (u, v, d) = (x, y, z, ρ) T Wherein (u, v) is the first pixel coordinates of the feature point, d is the depth, (x, y, z) is the spatial coordinates, and ρ is the inverse depth. Since the parameterized point cloud data has observation values, the parameters in it can be optimized to further optimize and adjust the coordinates of the feature point, thereby reducing the error of extracting the coordinates of the feature point.

[0181] In the embodiment, the mapping relationship between the first pixel coordinates and the spatial coordinates is used to map the first pixel coordinates to obtain the spatial coordinates of the feature point in the world coordinate system, which realizes the accurate extraction of the spatial coordinates corresponding to the feature point and reduces the error of coordinate mapping.

[0182] In some embodiments of the present application, the above-mentioned positioning of the feature point in the current frame image through the first thread to obtain the first pixel coordinates of the feature point in the current frame image includes:

[0183] The first thread is used to pre-process the current frame image to obtain a pre-processed image;

[0184] The first thread is used to position the feature point in the pre-processed image according to the historical frame image in the head-mounted display device to obtain the first pixel coordinates.

[0185] In the embodiment, the matching efficiency of the feature points can be improved by pre-processing the current frame image. The pre-processing includes but is not limited to pyramid image processing, filtering, histogram equalization, edge detection and other processing procedures. Taking pyramid image processing as an example, the first thread is used to perform pyramid image processing on the current frame image to obtain the pyramid image of the current frame, i.e., the pre-processed image. For example, a preset pyramid level is obtained, and the first thread is used to perform pyramid image processing on the current frame image according to the pyramid level to obtain the pyramid image of the current frame. The feature points in the current frame image are identified based on the pyramid image. Due to the movement of the head-mounted display device, the positions of the feature points in the pyramid image of the current frame can be determined according to the positions of the historical feature points in the historical frame image. For example, the historical feature points in the previous frame image can be matched with the current frame image by optical flow estimation such as Lucas-Kanade optical flow method to determine the positions of the feature points in the pyramid image of the current frame. The feature points at the positions are identified to obtain the first pixel coordinates of the feature points in the current frame image.

[0186] In some embodiments, in order to speed up the positioning of the feature points in the current frame image and shorten the image pre-processing time, the affine transformation of the local image region can also be performed on the current frame image to reduce the iteration number of the pyramid image processing and the iteration number of the optical flow estimation in the pre-processing operation.

[0187] In some embodiments, in order to quickly determine the positions of the feature points in the current frame image, the depth information of the feature points (which can be called from the back-end calculation result of the head-mounted display device, for example) and the trajectory data deduced based on the collection data of the inertial sensor can be obtained to predict the position of the current frame image before the posture calculation of the head-mounted display device; and the positions of the feature points in the current frame image and the pyramid information can be predicted according to the camera projection relationship, so that the optical flow operation does not need to be iterated by multiple pyramid layers, which can greatly save the calculation time. In order to enable the optimization of the optical flow estimation to quickly converge, the historical frame image can be rotated and distorted in combination with the rotation difference between the current frame image and the historical frame image, which can make the image texture closer, thereby reducing the optical flow iteration, further improving the posture deduction efficiency and enabling the deployment on a low-cost chip.

[0188] In the embodiment, the first thread is used to pre-process the current frame image, which can improve the accuracy of feature point extraction. Then, the first pixel coordinates of the feature points in the current frame image are determined according to the positions of the historical feature points in the historical frame image by the first thread, which realizes the efficient and accurate acquisition of the first pixel coordinates and improves the posture deduction efficiency.

[0189] The steps of the pose estimation method in the present application are described below in combination with a specific implementation scenario.

[0190] As shown in FIG. 2, FIG. 2 is a flowchart of a camera-based pose estimation method according to the present application. Specifically, a current frame image is acquired by a camera on a pair of glasses in a head-mounted display device, and then a pyramid image is obtained by performing pyramid calculation on the current frame image by a first thread corresponding to the camera. Sub-pixel matching is performed on the pyramid image and a historical frame image by the first thread, and it is determined whether the current frame image is a key frame image. If the current frame image is a key frame image, feature point extraction is performed on the current frame image by the first thread, a feature point pair for binocular matching is determined, and binocular depth calculation is performed according to the binocular feature point pair.

[0191] Before the current frame image arrives, the second thread collects inertial data between the current frame image and the historical frame image and stores the inertial data in a buffer module. Since the frame rate of the inertial sensor is greater than the frame rate of the camera, when the current frame image arrives, the second thread has collected all the inertial data between the adjacent two frames in the buffer module. The first thread extracts the inertial data from the buffer module, and fuses the inertial data and the position information of the feature points by the first thread to determine a second pose of the head-mounted display device. The first thread wakes up the second thread waiting in the head-mounted display device according to the second pose, and the second thread acquires the second pose. According to the acquired second pose and the inertial data, the second thread predicts the pose of the head-mounted display device at the time corresponding to the current frame image to obtain a target pose. The second thread outputs the target pose to a rendering layer application to enable the head-mounted display device to update the pose and the rendered scene.

[0192] Thus, based on the sensors in the glasses of the head-mounted display device, data can be collected in real time as the user moves without relying on any sensors outside the glasses. The mutual correction between the sensors on the glasses is utilized to improve the accuracy of pose estimation, and efficient interaction between threads can reduce the consumption of computing resources, so that the pose estimation method can be deployed on a low-cost and low-power chip.

[0193] To better implement the pose estimation method provided in the embodiments of the present application, the embodiments of the present application further provide a virtual display image generation method applied to a head-mounted display device. The head-mounted display device includes a visual sensor, an inertial sensor and a processor, the processor is configured with a first thread, a second thread and a buffer module, and the frame rate of the inertial sensor is greater than the frame rate of the visual sensor.

[0194] Referring to FIG. 3, FIG. 3 is a flowchart of a virtual display image generation method according to an embodiment of the present application. The virtual display image generation method includes:

[0195] S301, acquire a current frame image collected by a vision sensor;

[0196] S302, locate a feature point in the current frame image by a first thread to obtain position information of the feature point;

[0197] S303, store inertia data detected by the inertia sensor between the current frame image and a historical frame image to a buffer module;

[0198] S304, update a first pose of the head-mounted display device at a time corresponding to the historical frame image according to the position information and the inertia data by the first thread to obtain a second pose;

[0199] S305, predict a pose of the head-mounted display device at a time corresponding to the current frame image as a target pose according to target inertia data of the inertia sensor at a current time and the second pose by a second thread.

[0200] S306, render a virtual display image of a scene where the head-mounted display device is located according to the target pose.

[0201] After S305 is performed, the target pose can be output according to a frame rate of the inertia sensor.

[0202] In this embodiment, in order to realize real-time output of the pose of the head-mounted display device and subsequent real-time display rendering, the frame rate of the inertia sensor in the head-mounted display device can be used as the output frame rate of the target pose. Because the frame rate of the inertia sensor is greater than the frame rate of the vision sensor, when the predicted target pose is output, the target pose can be output at a higher frame rate of the inertia sensor, the output frame rate of the pose is improved, the delay of the attitude calculation is minimized, the image output delay of the head-mounted display device is further reduced, and subsequent real-time image display rendering of the head-mounted display device is realized.

[0203] In this embodiment, after the target pose is obtained, the head-mounted display device can render a virtual display image of a scene where the head-mounted display device is located according to the target pose, and realize 3D rendering of the scene.

[0204] The meanings of the terms are the same as those in the above-mentioned pose estimation method, and specific implementation details can be referred to the description in the method embodiment. In order to better implement the pose estimation method provided in the embodiments of the present application, the embodiments of the present application also provide a head-mounted display device based on the above-mentioned pose estimation method. The meanings of the terms are the same as those in the above-mentioned pose estimation method, and specific implementation details can be referred to the description in the method embodiment.

[0205] For example, as shown in FIG. 4, the head-mounted display device can include a vision sensor, an inertia sensor, and a processor, and the processor is configured to execute the above-mentioned pose estimation method or the above-mentioned virtual display image generation method.

[0206] To facilitate better implementation of the pose estimation method provided by the embodiments of the present application, the embodiments of the present application further provide a pose estimation device based on the above-mentioned pose estimation method, applied to a head-mounted display device. The meanings of the terms are the same as in the above-mentioned pose estimation method, and the specific implementation details can be referred to the description in the method embodiments.

[0207] For example, as shown in FIG. 5, the pose estimation device can include an acquisition module 501, a data processing module 502, a buffering module 503, a pose solving module 504, and a pose prediction module 505. Among them,

[0208] The acquisition module 501 is configured to acquire a current frame image collected by a vision sensor.

[0209] The data processing module 502 is configured to, through a first thread, locate a feature point in the current frame image to obtain position information of the feature point.

[0210] The buffering module 503 is configured to store inertial data detected by an inertial sensor between the current frame image and a historical frame image.

[0211] The pose solving module 504 is configured to, through the first thread, update a first pose of the head-mounted display device at a time corresponding to the historical frame image according to the position information and the inertial data to obtain a second pose.

[0212] The pose prediction module 505 is configured to, through a second thread, predict a pose of the head-mounted display device at a time corresponding to the current frame image as a target pose according to target inertial data of the inertial sensor at the current time and the second pose.

[0213] In an embodiment of the present application, the pose estimation device further includes:

[0214] The target pose output module is configured to output the target pose according to a frame rate of the inertial sensor.

[0215] In an embodiment of the present application, the above-mentioned pose solving module 504 includes:

[0216] The first preprocessing submodule is configured to, through the first thread, pre-process the inertial data to obtain pre-processed data, the pre-processed data being used to represent a degree of change in the pose of the head-mounted display device between the time corresponding to the current frame image and the time corresponding to the historical frame image.

[0217] The updating submodule is configured to, through the first thread, call a preset target optimization model, update the first pose according to the pre-processed data and the position information to obtain the second pose.

[0218] In an embodiment of the present application, the above-mentioned updating submodule includes:

[0219] a first prediction unit, configured to determine, by using the first thread, a first prediction error according to the preprocessed data and the first pose;

[0220] a second prediction unit, configured to determine, by using the first thread, a second prediction error according to position information and a predicted position of the feature point, the position information being real position information of the feature point, and the predicted position being information obtained by predicting the position of the feature point; and an updating unit, configured to call, by using the first thread, the target optimization model, and update the first pose according to the first prediction error and the second prediction error to obtain a second pose.

[0221] In an embodiment of the present application, the first prediction unit comprises:

[0222] a first prediction sub-unit, configured to predict, by using the first thread, a pose of the head-mounted display device at a time corresponding to a current frame image according to the preprocessed data and the first pose to obtain a third pose;

[0223] a first confirmation sub-unit, configured to determine, by using the first thread, the first prediction error according to the preprocessed data, the first pose and the third pose.

[0224] In an embodiment of the present application, the first confirmation sub-unit is specifically configured to:

[0225] determine, by using the first thread, a rotation prediction error according to a rotation increment in the preprocessed data, a historical rotation state in the first pose and a rotation state in the third pose;

[0226] determine, by using the first thread, a speed prediction error according to a speed increment in the preprocessed data, a historical speed state in the first pose and a speed state in the third pose;

[0227] determine, by using the first thread, a translation prediction error according to a translation increment in the preprocessed data, a historical translation state in the first pose and a translation state in the third pose;

[0228] determine, by using the first thread, the first prediction error according to the rotation prediction error, the speed prediction error and the translation prediction error. In an embodiment of the present application, the position information comprises first pixel coordinates and spatial coordinates corresponding to the feature point, and the predicted position comprises second pixel coordinates corresponding to the feature point, and the second prediction unit comprises:

[0229] a projection sub-unit, configured to project, by using the first thread, the spatial coordinates according to parameters of the visual sensor to obtain the second pixel coordinates of the feature point;

[0230] a second confirmation sub-unit, configured to determine the second prediction error according to a relative distance between the first pixel coordinates and the second pixel coordinates.

[0231] In an embodiment of the present application, the pose estimation apparatus further comprises:

[0232] The wake-up module is configured to wake up the first thread when the current frame image is detected.

[0233] The reading module is configured to read the inertial data in the buffer module by the first thread when the buffer module is not occupied.

[0234] In an embodiment of the present application, the pose prediction module 505 comprises:

[0235] The bias parameter acquisition submodule is configured to acquire the updated bias parameter of the inertial sensor from the first thread by the second thread; and the target pose determination submodule is configured to predict the pose of the head-mounted display device at the time corresponding to the current frame image according to the second pose, the target inertial data of the inertial sensor at the current time and the updated bias parameter by the second thread, to obtain the target pose.

[0236] In an embodiment of the present application, the data processing module 502 comprises:

[0237] The positioning submodule is configured to position the feature points in the current frame image by the first thread, to obtain the first pixel coordinates of the feature points in the current frame image.

[0238] The mapping submodule is configured to map the feature points according to the first pixel coordinates by the first thread, to obtain the spatial coordinates of the feature points in the world coordinate system.

[0239] The position information determination submodule is configured to determine the position information according to the first pixel coordinates and the spatial coordinates by the first thread. In an embodiment of the present application, the mapping submodule comprises:

[0240] The mapping unit is configured to acquire the mapping relationship between the first pixel coordinates and the spatial coordinates by the first thread.

[0241] The spatial coordinate determination unit is configured to map the first pixel coordinates according to the mapping relationship by the first thread, to obtain the spatial coordinates of the feature points in the world coordinate system.

[0242] In an embodiment of the present application, the positioning submodule comprises:

[0243] The image preprocessing unit is configured to pre-process the current frame image by the first thread, to obtain the pre-processed image; and the pixel coordinate determination unit is configured to position the feature points in the pre-processed image according to the historical frame images in the head-mounted display device by the first thread, to obtain the first pixel coordinates.

[0244] In an embodiment of the present application, the frame rate of the first thread is the same as the frame rate of the visual sensor, and the frame rate of the second thread is the same as the frame rate of the inertial sensor.

[0245] In practice, the above various modules can be implemented as independent entities, or can be combined as the same or several entities, and the specific implementation manners of the above various modules and the corresponding beneficial effects can be referred to the method embodiments, which will not be described here.

[0246] The embodiments of the present application also provide an electronic device, which can be a terminal with virtual reality (VR), or augmented reality (AR), or mixed reality (MR) and the like virtual reality simulation functions, such as the head-mounted display device described above.

[0247] As shown in FIG. 6, it shows a structural schematic diagram of an electronic device related to the embodiments of the present application, in particular:

[0248] The electronic device can include a processor 601 with one or more processing cores, a memory 602 with one or more computer readable storage media, a power supply 603, an input unit 604, and the like. Those skilled in the art can understand that the structure of the electronic device shown in FIG. 6 does not constitute a limitation on the electronic device, and can include more or fewer components than those shown, or combine certain components, or different component arrangements. Among them:

[0249] The processor 601 is the control center of the electronic device, which connects various parts of the entire electronic device through various interfaces and lines, and performs various functions and processes data of the electronic device by running or executing computer programs and / or modules stored in the memory 602, and calling data stored in the memory 602. Optionally, the processor 601 can include one or more processing cores; preferably, the processor 601 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 601.

[0250] The memory 602 can be used to store computer programs and modules, and the processor 601 executes various functions and data processing by running the computer programs and modules stored in the memory 602. The memory 602 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, computer programs required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 602 can include a high-speed random access memory, and can also include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 602 can also include a memory controller to provide the processor 601 with access to the memory 602.

[0251] The electronic device also includes a power supply 603 for powering the various components. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 603 can also include one or more than one direct current or alternating current power supply, a recharging system, a power supply fault detection circuit, a power supply converter or inverter, a power supply state indicator, and the like.

[0252] The electronic device can also include an input unit 604, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control. Although not shown, the electronic device can also include a display unit, etc., which will not be described here. In particular, in the present embodiment, the processor 601 in the electronic device will load one or more executable files corresponding to the processes of the computer programs into the memory 602 according to the following instructions, and the processor 601 will run the computer programs stored in the memory 602, so as to realize various functions, such as:

[0253] Obtaining a current frame image collected by a visual sensor;

[0254] Positioning a feature point in the current frame image through a first thread to obtain position information of the feature point;

[0255] Storing inertia data detected by the inertia sensor between the current frame image and a historical frame image to a buffer module;

[0256] Updating, through the first thread, a first pose of the head-mounted display device at a time corresponding to the historical frame image according to the position information and the inertia data to obtain a second pose;

[0257] The second thread predicts, according to the target inertial data of the inertial sensor at the current time and the second pose, a pose of the head-mounted display device at a time corresponding to the current frame image as a target pose.

[0258] For example, in some embodiments, the processor 601 in the electronic device can load an executable file corresponding to a process of one or more computer programs into the memory 602 according to the following instructions, and run the computer program stored in the memory 602 by the processor 601, so as to implement various functions, such as:

[0259] Obtaining a current frame image collected by a visual sensor;

[0260] Locating feature points in the current frame image by the first thread to obtain position information of the feature points;

[0261] Storing inertial data detected by the inertial sensor between the current frame image and a historical frame image to a buffer module;

[0262] Updating, by the first thread, a first pose of the head-mounted display device at a time corresponding to the historical frame image according to the position information and the inertial data to obtain a second pose;

[0263] Predicting, by the second thread, a pose of the head-mounted display device at a time corresponding to the current frame image as a target pose according to the target inertial data of the inertial sensor at the current time and the second pose;

[0264] Rendering a virtual display image of a scene in which the head-mounted display device is located according to the target pose.

[0265] The specific implementation manners of the above operations and the corresponding beneficial effects can be referred to the detailed description of the corresponding embodiments of the pose estimation method and the virtual display image generation method, and will not be described here.

[0266] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by a computer program, or by a computer program controlling related hardware, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0267] Therefore, an embodiment of the present application provides a computer readable storage medium, which stores a computer program capable of being loaded by a processor to execute steps in any one of the pose estimation methods provided by the embodiments of the present application and / or steps in any one of the virtual display image generation methods provided by the embodiments of the present application. For example, the computer program can execute the following steps:

[0268] Obtaining a current frame image collected by a visual sensor;

[0269] locating, by the first thread, the feature points in the current frame image to obtain position information of the feature points;

[0270] storing, to the buffer module, inertia data detected by the inertia sensor between the current frame image and the historical frame image;

[0271] updating, by the first thread, a first pose of the head-mounted display device at a time corresponding to the historical frame image according to the position information and the inertia data to obtain a second pose;

[0272] predicting, by the second thread, a pose of the head-mounted display device at a time corresponding to the current frame image as a target pose according to target inertia data of the inertia sensor at the current time and the second pose.

[0273] For example, in some embodiments, the computer program can perform the following steps:

[0274] obtaining a current frame image collected by a vision sensor;

[0275] locating, by the first thread, the feature points in the current frame image to obtain position information of the feature points;

[0276] storing, to the buffer module, inertia data detected by the inertia sensor between the current frame image and the historical frame image;

[0277] updating, by the first thread, a first pose of the head-mounted display device at a time corresponding to the historical frame image according to the position information and the inertia data to obtain a second pose;

[0278] predicting, by the second thread, a pose of the head-mounted display device at a time corresponding to the current frame image as a target pose according to target inertia data of the inertia sensor at the current time and the second pose.

[0279] rendering a virtual display image of a scene in which the head-mounted display device is located according to the target pose.

[0280] The specific implementation manners and corresponding beneficial effects of the above operations can be referred to the foregoing embodiments, and will not be described herein. The computer readable storage medium can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0281] Due to the computer program stored in the computer readable storage medium, the steps in any of the pose estimation method and / or the virtual display image generation method provided by the embodiments of the present application can be executed, thus the beneficial effects of any of the pose estimation method and / or the virtual display image generation method provided by the embodiments of the present application can be achieved, which will be described in detail in the foregoing embodiments, and will not be repeated here.

[0282] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned pose estimation method and / or the above-mentioned virtual display image generation method.

[0283] The above describes in detail a pose estimation method, a virtual display image generation method and related devices provided by the embodiments of the present application. The principles and implementation manners of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A pose estimation method applied to a head-mounted display device, wherein, The head-mounted display device comprises a visual sensor, an inertial sensor and a processor, the processor is configured with a first thread, a second thread and a buffer module, the frame rate of the inertial sensor is greater than that of the visual sensor; the method comprises: acquiring a current frame image collected by the visual sensor; locating a feature point in the current frame image through the first thread to obtain position information of the feature point; storing inertial data detected by the inertial sensor between the current frame image and a historical frame image to the buffer module; updating a first pose of the head-mounted display device at a time corresponding to the historical frame image according to the position information and the inertial data through the first thread to obtain a second pose; predicting a pose of the head-mounted display device at a time corresponding to the current frame image as a target pose according to target inertial data of the inertial sensor at a current time and the second pose through the second thread.

2. The pose estimation method of claim 1, a frame rate of the inertial sensor is greater than a frame rate of the vision sensor, wherein, Further comprising: outputting the target pose at the frame rate of the inertial sensor.

3. The pose estimation method of claim 1, wherein, The updating of the first pose of the head-mounted display device at the time corresponding to the historical frame image according to the position information and the inertial data through the first thread to obtain the second pose comprises: preprocessing the inertial data through the first thread to obtain preprocessing data, the preprocessing data being used to represent a degree of change in the pose of the head-mounted display device between the time corresponding to the current frame image and the time corresponding to the historical frame image; calling a preset target optimization model through the first thread, updating the first pose according to the preprocessing data and the position information to obtain the second pose.

4. The pose estimation method of claim 3, wherein, The updating of the first pose according to the preprocessing data and the position information through the first thread to obtain the second pose by calling a preset target optimization model comprises: determining a first prediction error according to the preprocessing data and the first pose through the first thread; determining a second prediction error according to the position information and a predicted position of the feature point through the first thread, the position information being true position information of the feature point, and the predicted position being information obtained by predicting the position of the feature point; calling the target optimization model through the first thread, updating the first pose according to the first prediction error and the second prediction error to obtain the second pose.

5. The pose estimation method of claim 4, wherein, The determination of the first prediction error according to the preprocessing data and the first pose through the first thread comprises: predicting the pose of the head-mounted display device at the time corresponding to the current frame image according to the preprocessing data and the first pose through the first thread to obtain a third pose; determining the first prediction error according to the preprocessing data, the first pose and the third pose through the first thread. The determination of the first prediction error according to the preprocessing data, the first pose and the third pose through the first thread comprises:

6. The pose estimation method of claim 5, wherein, ​ determining, by the first thread, a rotation prediction error according to a rotation increment in the preprocessed data, a historical rotation state in the first pose, and a rotation state in the third pose; determining, by the first thread, a velocity prediction error according to a velocity increment in the preprocessed data, a historical velocity state in the first pose, and a velocity state in the third pose; determining, by the first thread, a translation prediction error according to a translation increment in the preprocessed data, a historical translation state in the first pose, and a translation state in the third pose; determining, by the first thread, the first prediction error according to the rotation prediction error, the velocity prediction error, and the translation prediction error.

7. The pose estimation method of claim 4, wherein, The position information includes first pixel coordinates and spatial coordinates corresponding to the feature points, the predicted position includes second pixel coordinates corresponding to the feature points, and the first thread determines a second prediction error according to the position information and the predicted position of the feature points, including: projecting, by the first thread, the spatial coordinates according to parameters of the visual sensor to obtain the second pixel coordinates of the feature points; determining the second prediction error according to a relative distance between the first pixel coordinates and the second pixel coordinates. After storing the inertial data detected by the inertial sensor between the current frame image and the historical frame image to the buffer module, the method further includes:

8. The pose estimation method of claim 1, wherein, when the current frame image is detected, waking up the first thread; reading, by the first thread, the inertial data in the buffer module when the buffer module is not occupied. The second thread predicts a target pose of the head-mounted display device at a time corresponding to the current frame image as a target pose according to target inertial data of the inertial sensor at the current time and the second pose, including:

9. The pose estimation method of claim 1, wherein, acquiring, by the second thread, the updated bias parameters of the inertial sensor from the first thread; predicting, by the second thread, the pose of the head-mounted display device at a time corresponding to the current frame image according to the second pose, target inertial data of the inertial sensor at the current time, and the updated bias parameters, to obtain the target pose. The first thread locates feature points in the current frame image to obtain position information of the feature points, including:

10. The pose estimation method of claim 1, wherein, locating, by the first thread, feature points in the current frame image to obtain first pixel coordinates of the feature points in the current frame image; mapping, by the first thread, the feature points according to the first pixel coordinates, obtaining spatial coordinates of the feature points in a world coordinate system; determining, by the first thread, the position information according to the first pixel coordinates and the spatial coordinates. The first thread maps the feature points according to the first pixel coordinates to obtain spatial coordinates of the feature points in a world coordinate system, including:

11. The pose estimation method of claim 10, wherein, acquiring, by the first thread, a mapping relationship between the first pixel coordinates and the spatial coordinates; ​ The first thread is used to map the first pixel coordinate according to the mapping relationship, so as to obtain a spatial coordinate of the feature point in the world coordinate system.

12. The pose estimation method of claim 10, wherein, The first thread is used to map the first pixel coordinate according to the mapping relationship, so as to obtain a spatial coordinate of the feature point in the world coordinate system. The first thread is used to map the first pixel coordinate according to the mapping relationship, so as to obtain a spatial coordinate of the feature point in the world coordinate system. The first thread is used to map the first pixel coordinate according to the mapping relationship, so as to obtain a spatial coordinate of the feature point in the world coordinate system.

13. The pose estimation method of claim 1, wherein, The first thread is used to map the first pixel coordinate according to the mapping relationship, so as to obtain a spatial coordinate of the feature point in the world coordinate system.

14. A method for generating a virtual display image, applied to a head-mounted display device, wherein, The head-mounted display device comprises a visual sensor, an inertial sensor and a processor, the processor is configured with a first thread, a second thread and a buffer module, the frame rate of the inertial sensor is greater than the frame rate of the visual sensor; the method comprises: acquiring a current frame image collected by the visual sensor; locating a feature point in the current frame image by using the first thread to obtain position information of the feature point; storing inertial data detected by the inertial sensor between the current frame image and a historical frame image into the buffer module; updating a first pose of the head-mounted display device at a time corresponding to the historical frame image according to the position information and the inertial data by using the first thread to obtain a second pose; predicting a pose of the head-mounted display device at a time corresponding to the current frame image as a target pose according to target inertial data of the inertial sensor at a current time and the second pose by using the second thread; generating a virtual display image of a scene in which the head-mounted display device is located according to the target pose.

15. The method of claim 14, wherein a frame rate of the inertial sensor is greater than a frame rate of the vision sensor. The target pose is output at the frame rate of the inertial sensor. The head-mounted display device comprises a visual sensor, an inertial sensor and a processor, the processor is configured with a first thread, a second thread and a buffer module, the processor is configured to:

16. A head-mounted display device, wherein, acquire a current frame image collected by the visual sensor; locate a feature point in the current frame image by using the first thread to obtain position information of the feature point; store inertial data detected by the inertial sensor between the current frame image and a historical frame image into the buffer module; update a first pose of the head-mounted display device at a time corresponding to the historical frame image according to the position information and the inertial data by using the first thread to obtain a second pose; predict a pose of the head-mounted display device at a time corresponding to the current frame image as a target pose according to target inertial data of the inertial sensor at a current time and the second pose by using the second thread. The processor is further configured to:

17. The head-mounted display device of claim 16, wherein, output the target pose at the frame rate of the inertial sensor. ​ 18. A head-mounted display device, wherein, The head-mounted display device comprises a vision sensor, an inertial sensor and a processor, the processor is configured with a first thread, a second thread and a buffer module, the frame rate of the inertial sensor is greater than that of the vision sensor, and the processor is configured to: acquire a current frame image collected by the vision sensor; position a feature point in the current frame image through the first thread to obtain position information of the feature point; store inertial data detected by the inertial sensor between the current frame image and a historical frame image to the buffer module; update a first pose of the head-mounted display device at a time corresponding to the historical frame image according to the position information and the inertial data through the first thread to obtain a second pose; predict a pose of the head-mounted display device at a time corresponding to the current frame image as a target pose according to target inertial data of the inertial sensor at a current time and the second pose through the second thread; render a virtual display image of a scene where the head-mounted display device is located according to the target pose.

19. The head-mounted display device of claim 18, wherein, The processor is further configured to: output the target pose at the frame rate of the inertial sensor.

20. A computer readable storage medium, wherein, The computer readable storage medium stores a computer program, and the computer program is adapted to be loaded by the processor to execute the pose estimation method in any one of claims 1 to 13 or the generation method of the virtual display image in claim 14.

Images