Method and device for predicting positioning of head-mounted display device, medium and program product

Abstract

The invention provides a method and equipment for predicting positioning of head-mounted display equipment, a medium and a program product. The method can comprise the steps of obtaining vehicle driving data of a vehicle at a current moment, motion data of a head-mounted display device in the vehicle at the current moment, and estimated positioning data of the head-mounted display device at a previous moment of the current moment, the pre-estimated positioning data at any moment comprises pre-estimated position data and pre-estimated attitude data of the head-mounted display equipment at the moment; and performing data fusion processing on the motion data at the current moment, the vehicle driving data and the estimated positioning data at the previous moment by using a data fusion technology to obtain estimated positioning data of the head-mounted display equipment at the next moment of the current moment. According to the technical scheme, the accuracy and robustness of the positioning result of the head-mounted display device can be remarkably improved, and it is ensured that a user obtains stable immersion experience in the vehicle driving process.

Description

Technical Field

[0001] One or more embodiments of the present application relate to the technical field of head-mounted display devices, and in particular, to a method, device, medium, and program product for predicting the positioning of a head-mounted display device. Background Art

[0002] Currently, integrating head-mounted display devices such as AR (Augmented Reality) devices and VR (Virtual Reality) devices with vehicles has become an important technological trend. Using a head-mounted display device in a vehicle can provide rich information display for the driver and enhance the driver's driving experience. However, the prerequisite for realizing the foregoing functions is to accurately position the head-mounted display device.

[0003] In the related art, the positioning prediction mainly relies on the inertial measurement unit (IMU) and camera inside the head-mounted display device. However, during the driving process of the vehicle, bumps and vibrations are generated, which easily cause large errors in the data collected by the IMU and camera, and further result in inaccurate positioning of the head-mounted display device, affecting the user experience. Summary of the Invention

[0004] The present application provides a method, device, medium, and program product for predicting the positioning of a head-mounted display device to solve the deficiencies in the related art.

[0005] According to a first aspect of one or more embodiments of the present application, a method for predicting the positioning of a head-mounted display device is provided, including:

[0006] Obtain the vehicle driving data of the vehicle at the current moment, the motion data of the head-mounted display device in the vehicle at the current moment, and the estimated positioning data of the head-mounted display device at the previous moment of the current moment, where the estimated positioning data at any moment includes the estimated position data and estimated attitude data of the head-mounted display device at that moment;

[0007] Use data fusion technology to perform data fusion processing on the motion data at the current moment, the vehicle driving data, and the estimated positioning data at the previous moment to obtain the estimated positioning data of the head-mounted display device at the next moment of the current moment.

[0008] Optionally, the data fusion technology is used to perform data fusion processing on the motion data at the current moment, the vehicle driving data, and the estimated positioning data at the previous moment to obtain the estimated positioning data of the head-mounted display device at the next moment of the current moment, including: predicting based on the vehicle driving data at the current moment and the estimated positioning data at the previous moment to obtain the initial estimated positioning data of the head-mounted display device at the next moment; using the motion data and the vehicle driving data at the current moment to correct the initial estimated positioning data to obtain the estimated positioning data at the next moment.

[0009] Optionally, it further includes: obtaining the visual data of the head-mounted display device at the current moment, and generating the reference map corresponding to the current moment and the relative estimated positioning data of the head-mounted display device in the reference map according to the visual data at the current moment and the motion data at the current moment; the data fusion technology is used to perform data fusion processing on the motion data at the current moment, the vehicle driving data, and the estimated positioning data at the previous moment to obtain the estimated positioning data of the head-mounted display device at the next moment of the current moment, including: performing data fusion processing on the relative estimated positioning data, the motion data at the current moment, the vehicle driving data, and the estimated positioning data at the previous moment to obtain the estimated positioning data of the head-mounted display device at the next moment.

[0010] Optionally, it further includes: determining the driving state of the vehicle according to the vehicle driving data, and determining the first fusion weight corresponding to the vehicle driving data, the second fusion weight corresponding to the relative estimated positioning data, and the third fusion weight corresponding to the motion data according to the determined driving state; the performing data fusion processing on the relative estimated positioning data, the motion data at the current moment, the vehicle driving data, and the estimated positioning data at the previous moment includes: performing data fusion processing on the relative estimated positioning data, the motion data at the current moment, the vehicle driving data, and the estimated positioning data at the previous moment based on the first fusion weight, the second fusion weight, and the third fusion weight.

[0011] Optionally, it further includes: when the vehicle driving data includes vehicle position data, correcting the estimated positioning data of the head-mounted display device at the next moment according to the vehicle position data; and / or, when the estimated positioning data at the next moment indicates that the position or posture of the head-mounted display device changes abnormally at the next moment, obtaining the historical driving data of the vehicle before the current moment, and correcting the estimated positioning data at the next moment according to the historical driving data.

[0012] Optionally, it further includes: obtaining the actual positioning data of the head-mounted display device at any moment; optimizing the parameters involved in the data fusion technology according to the actual positioning data at the any moment and the estimated positioning data at that moment.

[0013] According to a second aspect of one or more embodiments of the present application, there is provided a device for predicting the positioning of a head-mounted display device, including:

[0014] A data acquisition unit, configured to acquire the vehicle driving data of the vehicle at the current moment, the motion data of the head-mounted display device in the vehicle at the current moment, and the estimated positioning data of the head-mounted display device at the previous moment of the current moment, wherein the estimated positioning data at any moment includes the estimated position data and estimated attitude data of the head-mounted display device at that moment;

[0015] A prediction unit, configured to perform data fusion processing on the motion data at the current moment, the vehicle driving data, and the estimated positioning data at the previous moment by using a data fusion technology to obtain the estimated positioning data of the head-mounted display device at the next moment of the current moment.

[0016] According to a third aspect of one or more embodiments of the present application, there is provided an electronic device, including:

[0017] A processor;

[0018] A memory for storing instructions executable by the processor;

[0019] Wherein, the processor realizes the method described in any one of the embodiments in the first aspect above by running the executable instructions.

[0020] According to a fourth aspect of one or more embodiments of the present application, there is provided a computer-readable storage medium, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in any one of the embodiments in the first aspect above are realized.

[0021] According to a fifth aspect of one or more embodiments of the present application, there is provided a computer program product, including a computer program and / or instructions, and when the computer program and / or instructions are executed by a processor, the steps of the method described in any one of the embodiments in the first aspect above are realized.

[0022] As can be seen from the above technical solutions, in one or more embodiments of the present application, by obtaining the vehicle driving data of the vehicle at the current moment, the motion data of the head-mounted display device at the current moment, and the estimated positioning data of the head-mounted display device at the previous moment, and then fusing these three to predict the estimated positioning data of the head-mounted display device at the next moment. Since the vehicle driving data of the vehicle at the current moment can reflect the real motion state of the vehicle in real time, predicting the positioning of the head-mounted display device in combination with the vehicle driving data can effectively compensate for the errors caused by vehicle dynamic characteristics such as vehicle bumps and vibrations, thereby significantly improving the accuracy and robustness of the estimated positioning data of the head-mounted display device, ensuring that users obtain a smooth, stable and high-quality visual experience during vehicle driving, and avoiding visual discomfort caused by inaccurate positioning of the head-mounted display device.

[0023] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present application and used together with the specification to explain the principles of the present application.

[0025] Figure 1 is a schematic structural diagram of a positioning prediction system for a head-mounted display device provided by an exemplary embodiment.

[0026] Figure 2 is a schematic flowchart of a method for predicting the positioning of a head-mounted display device provided by an exemplary embodiment.

[0027] Figure 3 is an interactive flowchart of a method for predicting the positioning of a head-mounted display device provided by an exemplary embodiment.

[0028] Figure 4 is a schematic structural diagram of an electronic device provided by an exemplary embodiment.

[0029] Figure 5 is a block diagram of a device for predicting the positioning of a head-mounted display device provided by an exemplary embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] Next, one or more embodiments of the present application will be described in detail.

[0031] Figure 1 For a schematic structural diagram of a positioning prediction system for a head-mounted display device provided by an exemplary embodiment. As Figure 1 shown, the positioning prediction system may include a head-mounted display device 11, a network 12, and a vehicle 13.

[0032] The head-mounted display device 11 refers to a display device mounted on the head, which is usually used for virtual reality, augmented reality or mixed reality. In this application, the head-mounted display device 11 may include, but is not limited to: VR devices, AR devices, and MR (Mixed Reality) devices. During the operation of the positioning prediction system, the head-mounted display device 11 can collect data through the internal IMU and camera, and display virtual images to the user wearing the head-mounted display device 11.

[0033] The vehicle 13 can be any vehicle. The head-mounted display device 11 is a device worn by the user inside the vehicle 13. The vehicle 13 and the head-mounted display device 11 can be directly or indirectly connected through wired communication or wireless communication, and this application does not make special restrictions.

[0034] Based on Figure 1 For the positioning prediction system of the head-mounted display device shown, when the user in the vehicle 13 wears the head-mounted display device 11, it is necessary to perform positioning prediction on the head-mounted display device 11. Exemplarily, the vehicle 13 obtains the vehicle driving data at the current moment through its own configured sensors, while the head-mounted display device 11 can obtain the motion data at the current moment through the internal IMU, and obtain the estimated positioning data of itself at the previous moment of the current moment from the locally stored data. Then, data fusion processing is performed on the estimated positioning data of the head-mounted display device 11 at the previous moment, the motion data at the current moment, and the vehicle driving data of the vehicle 13 at the current moment to obtain the estimated positioning data of the head-mounted display device 11 at the next moment. It should be noted that the above data fusion processing operation can be executed by the head-mounted display device 11. In this case, the vehicle 13 needs to send the vehicle driving data to the head-mounted display device 11. In addition, the above data fusion processing operation can also be executed by the vehicle 13. In this case, the head-mounted display device 11 needs to send its estimated positioning data at the previous moment and the motion data at the current moment to the vehicle 13, and receive the estimated positioning data at the next moment returned by the vehicle 13.

[0035] Next, taking the head-mounted display device 11 in the Figure 1 system architecture described above as an example to execute the method for predicting the positioning of the head-mounted display device provided in the embodiments of this application, an exemplary description will be given. Please refer to Figure 2 , Figure 2 which is a schematic flowchart of a method for predicting the positioning of a head-mounted display device provided in an exemplary embodiment. It can be understood that during the vehicle driving process, the positioning of the head-mounted display device will change continuously, so the positioning of the head-mounted display device at each moment can be predicted. The positioning prediction method may include the following steps:

[0036] S201. Obtain the vehicle driving data of the vehicle at the current moment, the motion data of the head-mounted display device in the vehicle at the current moment, and the estimated positioning data of the head-mounted display device at the previous moment of the current moment. Among them, the estimated positioning data at any moment includes the estimated position data and the estimated attitude data of the head-mounted display device at that moment.

[0037] The current moment can be any moment during the vehicle driving process. The head-mounted display device can obtain the vehicle driving data of the vehicle at the current moment from the vehicle. The vehicle can be any vehicle, and the vehicle driving data can include but are not limited to vehicle speed, vehicle acceleration, angular velocity, vehicle position, etc. A variety of sensors are pre-deployed on the vehicle, such as accelerometers, gyroscopes, vehicle speed sensors, and GPS, etc. The vehicle can collect its own vehicle driving data in real time through these sensors and send the collected vehicle driving data to the head-mounted display device. Exemplarily, the accelerometers and gyroscopes deployed on the vehicle can be used to capture the linear acceleration and angular velocity information of the vehicle, including the acceleration in the front-back, left-right, and up-down directions of the vehicle, and the rotational angular velocity of the vehicle. The sampling accuracy of the accelerometers and gyroscopes can be set to ensure the real-time capture of the dynamic changes of the vehicle. For example, the sampling frequency can be set to 100 Hz or above, so as to ensure the capture of real-time and accurate vehicle linear acceleration and angular velocity information through this high sampling accuracy. The vehicle speed sensor can obtain the driving speed information of the vehicle by detecting the wheel speed. The GPS sensor can provide the position information of the vehicle.

[0038] The motion data of the head-mounted display device can include but are not limited to: the acceleration, angular velocity, etc. of the head-mounted display device. The head-mounted display device captures its own motion data at each moment, and the "motion data" described in this application are all actual values. Generally, the acceleration and angular velocity changes of the head-mounted display device are captured through the IMU (including accelerometers and gyroscopes) of the head-mounted display device. The IMU can be set to sample at a high frequency (such as 200 Hz), so as to ensure more delicate motion tracking of the head-mounted display device.

[0039] In an embodiment, after the vehicle collects the vehicle driving data at the current moment, it can analyze the collected vehicle driving data and determine the driving state of the vehicle (such as a uniform driving state, an accelerating driving state, a decelerating driving state, etc.) according to the analysis result. The vehicle can send the collected vehicle driving data and the associated driving state obtained by the analysis to the head-mounted display device. Furthermore, the head-mounted display device can use a data fusion technology that matches the vehicle driving state in the subsequent data fusion and prediction processes to process, so as to further improve the accuracy of the predicted positioning data.

[0040] In one embodiment, before performing data fusion, the vehicle driving data can be subjected to filtering processing and / or denoising processing. The filtering processing is to remove the high-frequency noise and transient interference contained in the vehicle driving data, so as to ensure that the vehicle driving data after filtering is smoother and more stable, which helps to improve the stability and reliability of data fusion. The denoising processing is to reduce the random noise in the vehicle driving data. Generally, when performing filtering processing and denoising processing simultaneously, the vehicle driving data can be first subjected to filtering processing, and then the filtered vehicle driving data can be subjected to denoising processing. It should be noted that the specific implementation manners of the filtering processing and the denoising processing in this application are not particularly limited. Exemplarily, the filtering processing can be implemented by filtering techniques, such as low-pass filtering, moving average filtering, etc. The denoising processing can be implemented by denoising techniques, such as median filtering, adaptive filtering, etc. In addition, it should be emphasized that the execution party of the filtering processing and the denoising processing in this application is not particularly limited. For example, the vehicle can perform filtering processing and / or denoising processing on the vehicle driving data, and then send the processed vehicle driving data to the head-mounted display device. This method helps to reduce the data processing pressure of the head-mounted display device. Or, the vehicle directly sends the collected vehicle driving data to the head-mounted display device, and then the head-mounted display device performs filtering processing and / or denoising processing on the vehicle driving data. This method can reduce the waiting time for the head-mounted display device to obtain the vehicle driving data.

[0041] In this embodiment, by performing filtering processing and / or denoising processing on the vehicle driving data, invalid data such as noise in the vehicle driving data can be effectively removed, and the interference of the invalid data on the positioning prediction can be reduced. In this way, the quality of the vehicle driving data used for data fusion can be improved, and further the reliability and accuracy of the positioning prediction can be improved.

[0042] S202. Use data fusion technology to perform data fusion processing on the motion data at the current moment, the vehicle driving data, and the estimated positioning data at the previous moment to obtain the estimated positioning data of the head-mounted display device at the next moment of the current moment.

[0043] Data fusion technology refers to the comprehensive processing of data from multiple sources to obtain more accurate and comprehensive information than a single source. The "data fusion technology" described in this embodiment includes, but is not limited to, weighted average method, Bayesian network, and multi-sensor data fusion technology. Exemplarily, multi-sensor data fusion technology can be used to perform data fusion processing on the vehicle driving data of the vehicle at the current moment, the motion data of the head-mounted display device at the current moment, and its estimated positioning data at the previous moment, to obtain the estimated positioning data of the head-mounted display device at the next moment. The multi-sensor data fusion technology aims to integrate data from multiple sensors to overcome the limitations of a single sensor, thereby outputting higher-quality information after integration and processing. In this embodiment, the information output by the multi-sensor data fusion technology is the estimated positioning data of the head-mounted display device at the next moment. There are many multi-sensor data fusion technologies, such as Kalman filter fusion, particle filter fusion, etc., and this application does not limit them.

[0044] In the above embodiment, by obtaining the vehicle driving data of the vehicle at the current moment, the motion data of the head-mounted display device at the current moment, and the estimated positioning data of the head-mounted display device at the previous moment, and then fusing these three to predict the estimated positioning data of the head-mounted display device at the next moment. Since the vehicle driving data of the vehicle at the current moment can reflect the real motion state of the vehicle in real time, combining the vehicle driving data to predict the positioning of the head-mounted display device can effectively compensate for the errors caused by vehicle dynamic characteristics such as vehicle bumps and vibrations, thereby significantly improving the accuracy and robustness of the estimated positioning data of the head-mounted display device, ensuring that users obtain a smooth, stable and high-quality visual experience during vehicle driving, and avoiding visual discomfort caused by inaccurate positioning of the head-mounted display device. And, the above method of using a recursive structure for positioning prediction means that only the vehicle driving data at the current moment, the motion data of the head-mounted display device, and the estimated positioning data of the head-mounted display device at the previous moment are required for prediction, without storing and processing all historical data, which helps to reduce the consumption of system storage resources.

[0045] In one embodiment, the data fusion processing process may include: predicting based on the vehicle driving data of the vehicle at the current moment and the estimated positioning data of the head-mounted display device at the previous moment to obtain the initial estimated positioning data of the head-mounted display device at the next moment. Furthermore, using the motion data of the head-mounted display device at the current moment and the vehicle driving data of the vehicle at the current moment to correct the initial estimated positioning data to obtain the estimated positioning data of the head-mounted display device at the next moment.

[0046] Exemplarily, taking the Kalman filter fusion technology as an example, the above data fusion processing process is described as follows: The state vector of the head-mounted display device can be defined, and this state vector includes the position, speed, acceleration, and attitude of the head-mounted display device. It should be noted that the state vector of the head-mounted display device will be iteratively updated during the Kalman filter fusion process, and the vehicle driving data (such as acceleration, angular velocity) and the motion data collected by the head-mounted display device through the IMU jointly constitute the observed values of the state vector. Specifically, obtain the state vector of the head-mounted display device at the previous moment, and this state vector includes the motion data of the head-mounted display device at the previous moment and the estimated positioning data at the previous moment. Then, input the state vector at the previous moment and the vehicle driving data at the current moment into a motion model (such as a uniform motion model or a uniformly accelerated motion model) to predict the initial estimated positioning data of the head-mounted display device at the next moment by the motion model. Furthermore, use the observed values at the current moment (that is, the vehicle driving data of the vehicle at the current moment and the motion data of the head-mounted display device at the current moment) to correct the initial estimated positioning data to obtain the estimated positioning data of the head-mounted display device at the next moment.

[0047] This embodiment not only relies on the vehicle driving data at the current moment for preliminary prediction, but also further combines the vehicle driving data at the current moment and the motion data of the head-mounted display device to correct the result of the preliminary prediction, thereby enhancing the influence degree of the vehicle driving data at the current moment on the finally obtained estimated positioning data and helping to obtain relatively accurate estimated positioning data.

[0048] In one embodiment, the driving state of the vehicle is not constant. There are differences in the data fusion technologies adapted to different driving states. Exemplarily, the driving state of the vehicle can be divided into a linear state and a non-linear state. The linear state can be understood as that the dynamic model of the vehicle can be described by a linear equation, and the non-linear state can be understood as that the dynamic model of the vehicle cannot be simply described by a linear equation.

[0049] The driving state of a vehicle can be determined based on the vehicle driving data. For example, when the vehicle driving data indicates that the vehicle is in any one of uniform linear motion, uniformly accelerated linear motion, and uniformly decelerated linear motion, it can be determined that the vehicle is in a linear state. In such a case, the Kalman filter fusion technology can be used for data fusion processing. The Kalman filter fusion technology is applicable to linear dynamic systems, and it updates the state estimation of the head-mounted display device through continuous iteration. When the vehicle driving data indicates that the vehicle is in motion such as turning or sudden acceleration, it can be determined that the vehicle is in a non-linear state. In such a case, the extended Kalman filter fusion (EKF) technology can be used for data fusion processing. The extended Kalman filter fusion is a non-linear version of the Kalman filter fusion. It converts the non-linear problem into a linear problem by linearizing the non-linear function (such as Taylor series expansion), and then uses the idea of the Kalman filter fusion technology for state estimation. In this way, even when the vehicle is in a non-linear state, accurate state estimation can be achieved.

[0050] In this embodiment, by judging the driving state of the vehicle, and then selecting a data fusion processing technology adapted to the driving state of the vehicle for data fusion processing, it is ensured that the positioning data of the head-mounted display device can be accurately predicted in any driving state, improving the accuracy and reliability of the positioning prediction, and at the same time effectively expanding the application scope of the positioning prediction scheme of this application.

[0051] In one embodiment, since the data collected by sensors often has noise, and the core of the multi-sensor data fusion technology lies in effectively combining the noisy sensor data and the prediction based on the dynamic model to generate the optimal state estimation of the head-mounted display device. Therefore, in the process of using the multi-sensor data fusion technology for data fusion, a noise matrix can be predefined, such as the covariance matrix of the system noise and the measurement noise. Among them, the system noise reflects the degree of internal uncertainty of the positioning prediction system, such as inaccurate models, random perturbations in the prediction process, etc. The measurement noise reflects the deviation between the sensor measurement value and the true state of the positioning prediction system, and this deviation may be caused by limitations of the sensor itself, environmental interference, electromagnetic interference, etc.

[0052] During the positioning prediction process, first, the predefined noise matrix is used to denoise the vehicle driving data at the current moment, the motion data of the head-mounted display device at the current moment, and the estimated positioning data at the previous moment. Then, data fusion processing is performed on the denoised data. This method can determine which data has a higher degree of uncertainty and which data is more reliable during the data fusion process by reasonably defining the covariance matrices of system noise and measurement noise. This helps the multi-sensor data fusion technology better balance the relationship between model prediction and actual measurement, thereby improving the accuracy of the finally predicted positioning data. In addition, in this embodiment, the covariance matrices of system noise and measurement noise can be designed according to the characteristics of each sensor deployed on the head-mounted display device and the vehicle, as well as the driving environment, so as to ensure that the positioning prediction system of this solution can maintain good positioning prediction performance under different driving environments / driving states.

[0053] In one embodiment, the visual data of the head-mounted display device at the current moment can be obtained. Exemplarily, the visual data at the current moment can be captured by the camera of the head-mounted display device. The visual data is used to reflect the surrounding environment information of the head-mounted display device and can include depth data and image data. At the same time, the motion data of the head-mounted display device at the current moment (i.e., the data obtained through the IMU) is obtained. Then, a reference map corresponding to the current moment and the relative estimated positioning data of the head-mounted display device in the reference map are generated according to the visual data and motion data of the head-mounted display device at the current moment. The reference map can be a point cloud map, a feature point map, a grid map, etc. Exemplarily, map construction and the generation of relative estimated positioning data can be achieved through visual tracking algorithms. Visual tracking algorithms can include but are not limited to the SLAM (Simultaneous Localization and Mapping) algorithm, etc. Specifically, the visual data and motion data of the head-mounted display device at the current moment are used as the input of the visual tracking algorithm, and the visual tracking algorithm will output the reference map corresponding to the current moment and the relative estimated positioning data of the head-mounted display device in the reference map. The relative estimated positioning data reflects the position and attitude of the head-mounted display device at the current moment relative to the reference coordinate system (i.e., the reference map).

[0054] Then, the relative estimated positioning data, the estimated positioning data of the head-mounted display device at the previous moment, the motion data at the current moment, and the vehicle driving data of the vehicle at the current moment are subjected to data fusion processing to predict the estimated positioning data of the head-mounted display device at the next moment. Taking the Kalman filter fusion technology as an example, this data fusion processing process includes: predicting based on the vehicle driving data of the vehicle at the current moment and the estimated positioning data of the head-mounted display device at the previous moment to obtain the initial estimated positioning data of the head-mounted display device at the next moment. Furthermore, the initial estimated positioning data is corrected by using the motion data of the head-mounted display device at the current moment, the vehicle driving data of the vehicle at the current moment, and the relative estimated positioning data to obtain the estimated positioning data of the head-mounted display device at the next moment. In this way, by adding the observation data of "relative estimated positioning data", the accuracy and reliability of the finally obtained estimated positioning data can be further improved.

[0055] In this embodiment, by introducing the visual data of the head-mounted display device, the head-mounted display device can better understand its own environment, and thus predict the positioning data of the head-mounted display device in combination with the key environmental feature points reflected in the visual data, which can significantly improve the accuracy of the prediction result.

[0056] In one embodiment, when the vehicle is in a non-uniform driving state such as accelerating, decelerating, or turning, there will be relative motion between the vehicle and the head-mounted display device. In the related art, this relative motion will cause changes in the external environment of the head-mounted display device, and further cause errors in the visual data obtained by the head-mounted display device, affecting the accuracy of the predicted positioning data. In view of this, the embodiment of the present application proposes: according to the driving state of the vehicle, the first fusion weight corresponding to the vehicle driving data, the second fusion weight corresponding to the relative estimated positioning data, and the third fusion weight corresponding to the motion data at the current moment are determined in real time. Exemplarily, the first fusion weight, the second fusion weight, and the third fusion weight can be preset according to actual business requirements, and then during the vehicle driving process, these three fusion weights are dynamically adjusted according to the driving state of the vehicle. For example, when the driving state of the vehicle indicates that the vehicle is in a uniform driving state at the current moment, it means that there is no relative motion between the vehicle and the head-mounted display device, so there is no need to adjust the first fusion weight, the second fusion weight, and the third fusion weight. When the driving state of the vehicle indicates that the vehicle is in a non-uniform driving state such as accelerating, decelerating, or turning at the current moment, the first fusion weight can be increased, and the second fusion weight and the third fusion weight can be decreased.

[0057] Then, data fusion processing is performed according to the determined first fusion weight, second fusion weight, and third fusion weight. Exemplarily, the data fusion processing process includes: predicting based on the vehicle driving data of the vehicle at the current moment and the estimated positioning data of the head-mounted display device at the previous moment to obtain the initial estimated positioning data of the head-mounted display device at the next moment. Furthermore, the initial estimated positioning data is corrected according to the vehicle driving data, relative estimated positioning data, motion data, and the fusion weights corresponding to these three data respectively to obtain the estimated positioning data of the head-mounted display device at the next moment.

[0058] Since the relative estimated positioning data is generated based on visual data, therefore, by adjusting the fusion weights to dynamically adjust the influence degrees of the vehicle driving data, relative estimated positioning data, and motion data in the data fusion process, the visual errors caused by external environment changes can be significantly reduced, and the accuracy and reliability of positioning prediction can be improved.

[0059] In one embodiment, during the process of data fusion using multi-sensor data fusion technology, some errors may be introduced. In addition, the positioning prediction for the head-mounted display device during the vehicle driving process is a long-term process. Therefore, the introduced errors will gradually accumulate during long-term operation, resulting in the final predicted positioning data deviating from the true positioning data. Based on this, the embodiment of the present application proposes to correct the estimated positioning data of the head-mounted display device at the next moment using the vehicle position data at the current moment.

[0060] Exemplarily, the vehicle position data at the current moment can be collected by the GPS deployed on the vehicle, and this vehicle position data is the absolute position data of the vehicle. The GPS can provide the absolute geographical location coordinates (longitude, latitude, and altitude) of the vehicle as a position reference. Taking the Kalman filter fusion technology as an example, during the data fusion process, the vehicle position data can be used as the observation input of the Kalman filter fusion. In the update step of the Kalman filter fusion, the vehicle position data is used to calculate the Kalman gain and accordingly correct the initial estimated positioning data. In addition, in order to save system overhead, it is not necessary to correct using the vehicle position data at each moment, but the vehicle position data can be used for correction periodically, so as to suppress the growth of cumulative errors in the Kalman filter fusion process and maintain the accuracy of the positioning prediction result.

[0061] In addition, if the predicted positioning data at the next moment indicates an abnormal change in the position or attitude of the head-mounted display device at the next moment, the historical driving data of the vehicle before the current moment can be obtained, and the predicted positioning data at the next moment can be corrected according to the historical driving data. For example, if the predicted positioning data at the next moment indicates that the head-mounted display device will suddenly deviate or the deviation distance is too long, it can be determined that the position or attitude of the head-mounted display device will change abnormally at the next moment. At this time, the historical driving data of the vehicle is analyzed. If the historical driving data indicates that the vehicle is driving smoothly, it can be determined that the foregoing abnormal change is caused by sensor error, and then the predicted positioning data at the next moment can be corrected. If the historical driving data indicates that the vehicle vibrates or jolts, then the foregoing abnormal change may be caused by the vehicle vibration / jolt. In this case, it is not necessary to correct the predicted positioning data at the next moment. It should be noted that the historical driving data of the vehicle in this application is not particularly limited. For example, the vehicle driving data within 10 seconds before the current moment can be used as the historical driving data.

[0062] In this embodiment, correcting the predicted positioning data of the head-mounted display device at the next moment through the vehicle position data and / or the historical driving data of the vehicle helps to prevent various errors in the prediction process from interfering with the positioning prediction result, thereby improving the accuracy of the positioning prediction result.

[0063] In one embodiment, a feedback mechanism can be set in the positioning prediction system of the head-mounted display device. Exemplarily, the actual positioning data of the positioning prediction system of the head-mounted display device at any moment and the predicted positioning data at that moment can be obtained. Then, according to the difference between the actual positioning data and the predicted positioning data at any moment (this difference is equivalent to feedback), the parameters involved in the data fusion technology are optimized.

[0064] Since the optimized parameters are generated according to the difference between the predicted positioning data and the actual positioning data at the same moment, using the optimized parameters for subsequent positioning prediction can improve the accuracy and stability of the subsequent positioning prediction result.

[0065] In this embodiment, by dynamically optimizing the parameters involved in the data fusion technology through the real-time feedback of the positioning prediction system of the head-mounted display device, the high precision and high stability of the positioning prediction system of the head-mounted display device can be effectively ensured, and the accuracy of the positioning prediction can be improved.

[0066] In one embodiment, when a head-mounted display device displays virtual content to a user, there can be multiple display modes, such as a monocular display mode (this mode has only one display), a binocular display mode (this mode has two displays, corresponding to the user's left and right eyes respectively, and can provide stereoscopic vision), a transparent display mode (this mode allows the user to still see the real environment when displaying virtual content), a closed display mode (this mode completely blocks external light and allows the user to only see the displayed content), and so on. Generally, during vehicle driving, the head-mounted display device will display virtual content to the user through a travel mode. In the travel mode, the user can view virtual content and the real environment outside the vehicle at the same time. In the related art, it is necessary for the user to manually switch the travel mode, which increases the complexity of user operation. In view of this, the embodiments of the present application propose to automatically switch the travel mode according to the driving state of the vehicle. Exemplarily, the driving state of the vehicle is determined according to the vehicle driving data at the current moment. If the driving state of the vehicle meets a predefined condition, the display mode of the head-mounted display device can be automatically switched to the travel mode, so that the user can view the surrounding environment outside the vehicle in the travel mode. Among them, the predefined condition may include but is not limited to: the vehicle speed is less than a preset vehicle speed threshold, and the vehicle acceleration is less than a preset acceleration threshold. When the driving state of the vehicle meets the foregoing predefined condition, it means that the vehicle speed is relatively small or the vehicle acceleration is relatively small. Such a situation often indicates that the user wants to view the environment outside the vehicle. Therefore, when the driving state of the vehicle meets the predefined condition, automatically switching the travel mode can not only improve the simplicity of user operation, but also meet the actual needs of the user. It should be noted that the foregoing predefined condition is only an example, and those skilled in the art can set the predefined condition according to actual needs, and the present application does not limit this.

[0067] Figure 3 FIG. 4 is an interaction flowchart of a method for predicting the positioning of a head-mounted display device shown in an exemplary embodiment. The interaction process involves a vehicle and a head-mounted display device inside the vehicle. The interaction process may include the following steps:

[0068] S301: The vehicle obtains the vehicle driving data at the current moment through sensors configured by itself.

[0069] S302: The vehicle filters and denoises the obtained vehicle driving data, and determines the driving state of the vehicle according to the processed vehicle driving data.

[0070] S303: The vehicle associates and sends the vehicle driving data at the current moment and the determined driving state to the head-mounted display device.

[0071] S304: The head-mounted display device obtains the motion data at the current moment, and at the same time obtains the estimated positioning data of itself at the previous moment from the data stored locally.

[0072] S305: The head-mounted display device performs data fusion processing on the estimated positioning data at the previous moment, the motion data at the current moment, and the vehicle driving data at the current moment to obtain the estimated positioning data at the next moment.

[0073] S306: The head-mounted display device determines whether the received driving state meets a predefined condition.

[0074] If the judgment result indicates that the vehicle driving state meets the predefined condition, the display mode is switched to the travel mode.

[0075] If the judgment result indicates that the vehicle driving state does not meet the predefined condition, the existing display mode is maintained.

[0076] It should be noted that S305 and S306 can be executed in parallel or serially, and this application does not limit the order of these two steps.

[0077] Figure 4 is a schematic structural diagram of an electronic device shown according to an exemplary embodiment of the present application. Refer to Figure 4 , at the hardware level, the electronic device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410. Of course, it may also include other hardware required for other services. The processor 402 reads the corresponding computer program from the non-volatile memory 410 into the memory 408 and then runs it. Of course, in addition to the software implementation, this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, and can also be hardware or a logic device.

[0078] Corresponding to the above method embodiment, the present application also provides an embodiment of a device for predicting the positioning of a head-mounted display device.

[0079] Figure 5 is a block diagram of a device for predicting the positioning of a head-mounted display device shown according to an exemplary embodiment of the present application. Refer to Figure 5 , the device includes a data acquisition unit 501 and a prediction unit 502, where:

[0080] The data acquisition unit 501 is configured to acquire the vehicle driving data of the vehicle at the current moment, the motion data of the head-mounted display device in the vehicle at the current moment, and the estimated positioning data of the head-mounted display device at the previous moment of the current moment. Wherein, the estimated positioning data at any moment includes the estimated position data and the estimated attitude data of the head-mounted display device at that moment;

[0081] The prediction unit 502 is configured to perform data fusion processing on the motion data at the current moment, the vehicle driving data, and the estimated positioning data at the previous moment by using a data fusion technology, so as to obtain the estimated positioning data of the head-mounted display device at the next moment of the current moment.

[0082] Optionally, the prediction unit 502 is specifically configured to: perform prediction based on the vehicle driving data at the current moment and the estimated positioning data at the previous moment to obtain the initial estimated positioning data of the head-mounted display device at the next moment; use the motion data and the vehicle driving data at the current moment to correct the initial estimated positioning data to obtain the estimated positioning data at the next moment.

[0083] Optionally, the device further includes: a map generation unit 503, configured to obtain the visual data of the head-mounted display device at the current moment, and generate a reference map corresponding to the current moment and the relative estimated positioning data of the head-mounted display device in the reference map according to the visual data at the current moment and the motion data at the current moment.

[0084] The prediction unit 502 is specifically configured to: perform data fusion processing on the relative estimated positioning data, the motion data at the current moment, the vehicle driving data, and the estimated positioning data at the previous moment to obtain the estimated positioning data of the head-mounted display device at the next moment.

[0085] Optionally, the device further includes: a fusion weight determination unit 504, configured to determine the driving state of the vehicle according to the vehicle driving data, and determine a first fusion weight corresponding to the vehicle driving data, a second fusion weight corresponding to the relative estimated positioning data, and a third fusion weight corresponding to the motion data according to the determined driving state;

[0086] The prediction unit 502 is specifically configured to: perform data fusion processing on the relative estimated positioning data, the motion data at the current moment, the vehicle driving data, and the estimated positioning data at the previous moment based on the first fusion weight, the second fusion weight, and the third fusion weight.

[0087] Optionally, the device further includes:

[0088] A correction unit 505, configured to correct the estimated positioning data of the head-mounted display device at the next moment according to the vehicle position data when the vehicle driving data includes vehicle position data; and / or, when the estimated positioning data at the next moment indicates that the position or posture of the head-mounted display device changes abnormally at the next moment, obtain the historical driving data of the vehicle before the current moment, and correct the estimated positioning data at the next moment according to the historical driving data.

[0089] Optionally, the device further includes:

[0090] An optimization unit 506, configured to obtain the actual positioning data of the head-mounted display device at any moment; optimize the parameters involved in the data fusion technology according to the actual positioning data and the estimated positioning data at that moment.

[0091] For the specific implementation process of the functions and roles of each module in the above device, please refer to the implementation process of the corresponding steps in the above method for details, which will not be elaborated here.

[0092] The device or module illustrated in the above embodiments can be specifically implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver device, a game console, a tablet computer, a wearable device, or a combination of any several of these devices.

[0093] In a typical configuration, a computer includes one or more processors, including a central processing unit (CPU) and a graphics processing unit (GPU), an input / output interface, a network interface, and a memory. Among them, the central processing unit is used for computing simulation, and the graphics processing unit is used for outputting high-quality three-dimensional images.

[0094] The memory may include non-permanent memory in the computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of, for example, read-only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

[0095] A computer-readable medium includes permanent and non-permanent, removable and non-removable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transitory media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0096] Corresponding to the embodiments of the foregoing method, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of any of the embodiments of the foregoing method are implemented.

[0097] Corresponding to the embodiments of the foregoing method, the present application further provides a computer program product, including a computer program and / or instructions, and when the computer program and / or instructions are executed by a processor, the steps of any of the embodiments of the foregoing method are implemented.

[0098] The foregoing is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for predicting the location of a head mounted display device, characterized in that: include: Acquire vehicle driving data of the vehicle at a current moment, motion data of a head mounted display device in the vehicle at the current moment, and estimated positioning data of the head mounted display device at a moment before the current moment, wherein the estimated positioning data at any moment includes estimated position data and estimated posture data of the head mounted display device at the moment; The motion data at the current moment, the vehicle driving data and the estimated positioning data at the previous moment are fused by using data fusion technology to obtain the estimated positioning data of the head mounted display device at the next moment after the current moment.

2. The method according to claim 1, characterized in that The method of using the data fusion technology to perform data fusion processing on the motion data at the current moment, the vehicle driving data, and the estimated positioning data at the previous moment to obtain the estimated positioning data of the head mounted display device at the next moment after the current moment includes: Predicting based on the vehicle driving data at the current moment and the estimated positioning data at the previous moment to obtain initial estimated positioning data of the head mounted display device at the next moment; The initial estimated positioning data is corrected using the motion data at the current moment and the vehicle driving data to obtain the estimated positioning data at the next moment.

3. The method according to claim 1, characterized in that Also includes: Acquire visual data of the head mounted display device at the current moment, and generate a reference map corresponding to the current moment and relative estimated positioning data of the head mounted display device in the reference map according to the visual data at the current moment and the motion data at the current moment; The method of using data fusion technology to perform data fusion processing on the motion data at the current moment, the vehicle driving data and the estimated positioning data at the previous moment to obtain the estimated positioning data of the head-mounted display device at the next moment after the current moment includes: performing data fusion processing on the relative estimated positioning data, the motion data at the current moment, the vehicle driving data and the estimated positioning data at the previous moment to obtain the estimated positioning data of the head-mounted display device at the next moment.

4. The method according to claim 3, characterized in that The method further includes: determining a driving state of the vehicle according to the vehicle driving data, and determining a first fusion weight corresponding to the vehicle driving data, a second fusion weight corresponding to the relative estimated positioning data, and a third fusion weight corresponding to the motion data according to the determined driving state; The data fusion processing of the relative estimated positioning data, the motion data at the current moment, the vehicle driving data and the estimated positioning data at the previous moment includes: data fusion processing of the relative estimated positioning data, the motion data at the current moment, the vehicle driving data and the estimated positioning data at the previous moment based on the first fusion weight, the second fusion weight and the third fusion weight.

5. The method according to claim 1, characterized in that Also includes: In a case where the vehicle driving data includes vehicle position data, correcting the estimated positioning data of the head mounted display device at the next moment according to the vehicle position data; and / or, When the estimated positioning data at the next moment indicates that the position or posture of the head mounted display device at the next moment has changed abnormally, the historical driving data of the vehicle before the current moment is obtained, and the estimated positioning data at the next moment is corrected according to the historical driving data.

6. The method according to claim 1, characterized in that Also includes: Obtaining actual positioning data of the head mounted display device at any time; Optimize the parameters involved in the data fusion technology according to the actual positioning data at any moment and the estimated positioning data at that moment.

7. A device for predicting the location of a head mounted display device, characterized in that: include: a data acquisition unit, configured to acquire vehicle driving data of the vehicle at a current moment, motion data of a head mounted display device in the vehicle at the current moment, and estimated positioning data of the head mounted display device at a moment before the current moment, wherein the estimated positioning data at any moment includes estimated position data and estimated posture data of the head mounted display device at the moment; The prediction unit is used to use data fusion technology to perform data fusion processing on the motion data at the current moment, the vehicle driving data and the estimated positioning data at the previous moment, so as to obtain the estimated positioning data of the head-mounted display device at the next moment after the current moment.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 6 by running the executable instructions.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The method comprises a computer program and / or instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images