State prediction methods and head-mounted display devices

Abstract

This application provides a state prediction method and a head-mounted display device, relating to the field of head-mounted display device technology. The method, applied to a head-mounted display device, includes: after detecting a future state prediction command for the wearer of the head-mounted display device, acquiring historical state time-series data of the wearer within a preset sliding time window; determining target anchor point state data and a feature scaling factor for the historical state time-series data; standardizing the historical state time-series data based on the target anchor point state data and the feature scaling factor to obtain standardized historical state time-series data; inputting the standardized historical state time-series data into a future state prediction model to obtain standardized future state time-series data; and performing the inverse operation of standardizing the standardized future state time-series data based on the target anchor point state data and the feature scaling factor to obtain the wearer's future state time-series data. This application can improve the accuracy of the head-mounted display device in predicting the wearer's future state.

Description

Technical Field

[0001] This application relates to the field of head-mounted display technology, and more particularly to a state prediction method and a head-mounted display device. Background Technology

[0002] Currently, in immersive interactive systems such as virtual reality, augmented reality, and mixed reality, head-mounted displays (HMDs) serve as core terminals, and their user experience is closely related to the real-time performance and accuracy of system responses. To achieve functions such as low-latency rendering, eye-tracking compensation, motion sickness suppression, and intelligent interactive decision-making, there is an urgent need for high-precision prediction of the wearer's future state based on key behavioral characteristics such as head posture, eye movement trajectory, and physiological state. This predictive capability has become a crucial technological support for improving the intelligence level of HMDs and optimizing human-machine collaboration efficiency.

[0003] However, the wearer status data collected by head-mounted displays is typically a high-dimensional, multimodal, and non-stationary time-series signal. Its dynamic characteristics are complex and easily affected by individual differences, environmental interference, and sudden movements (such as rapid head turning, blinking, and head shaking), resulting in significant time-varying and heterogeneous data distribution. Furthermore, data collected by different sensor channels (such as gyroscopes, accelerometers, and eye-tracking sensors) vary considerably in dimensions, amplitude range, and time response characteristics. Directly inputting the raw time-series data into the model can easily lead to numerical instability, gradient explosion, or slow model convergence, severely impacting the accuracy and robustness of the prediction results.

[0004] Therefore, effectively preprocessing historical time-series data before model input to eliminate dimensional differences and stabilize data distribution has become an indispensable prerequisite for improving the accuracy of head-mounted display devices in predicting the wearer's future state. Summary of the Invention

[0005] The main objective of this application is to provide a state prediction method and a head-mounted display device, which aims to improve the accuracy of the head-mounted display device in predicting the future state of the wearer.

[0006] To achieve the above objectives, this application proposes a state prediction method, which is applied to a head-mounted display device and includes:

[0007] After detecting a future state prediction command for the wearer of the head-mounted display device, the historical state time series data of the wearer within a preset sliding time window is obtained;

[0008] Determine the target anchor point state data of the historical state time series data, and determine the feature scaling factor of the historical state time series data;

[0009] Based on the target anchor point state data and the feature scaling factor, the historical state time series data is standardized to obtain standardized historical state time series data.

[0010] The standardized historical state time series data is input into the pre-trained future state prediction model to obtain the standardized future state time series data predicted by the future state prediction model.

[0011] Based on the target anchor point state data and the feature scaling factor, the inverse operation of the standardization process is performed on the standardized future state time series data to obtain the wearer's future state time series data.

[0012] In addition, to achieve the above objectives, this application also proposes a head-mounted display device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the state prediction method as described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the state prediction method described above.

[0014] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the state prediction method described above.

[0015] This application provides a state prediction method and a head-mounted display device, relating to the field of head-mounted display device technology. The state prediction method is applied to a head-mounted display device and includes: after detecting a future state prediction command for a wearer of the head-mounted display device, acquiring historical state time-series data of the wearer within a preset sliding time window; determining target anchor point state data of the historical state time-series data and determining a feature scaling factor for the historical state time-series data; standardizing the historical state time-series data based on the target anchor point state data and the feature scaling factor to obtain standardized historical state time-series data; inputting the standardized historical state time-series data into a pre-trained future state prediction model to obtain standardized future state time-series data predicted by the future state prediction model; and performing the inverse operation of the standardization process on the standardized future state time-series data based on the target anchor point state data and the feature scaling factor to obtain the wearer's future state time-series data. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the first embodiment of the state prediction method of this application;

[0019] Figure 2 This is a schematic diagram of a scene generated by the feature scaling factor in an example of this application;

[0020] Figure 3 This is a schematic diagram of a future state prediction scenario in a specific embodiment of this application;

[0021] Figure 4 This is a normalized scenario diagram in a specific embodiment of this application;

[0022] Figure 5 This is a schematic diagram of an anti-normalization scenario in a specific embodiment of this application;

[0023] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the state prediction method in the embodiments of this application.

[0024] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0025] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0026] When preprocessing time-series data to eliminate dimensional differences and improve the stability of model input, Z-score normalization based on sliding time windows can be used in related technologies. However, traditional Z-score normalization methods have significant drawbacks when processing time-series signals such as head state time-series data and eye state time-series data collected by head-mounted displays.

[0027] Because such time-series data exhibits significant time-varying characteristics and dynamic complexity, it is easily affected by individual differences, environmental interference, and sudden actions (such as rapid head turning, blinking, and head shaking). Its statistical distribution displays strong non-stationary characteristics. The Z-score normalization method relies on the arithmetic mean within a window for centering, implicitly assuming that the data is approximately stationary within that window. However, in real-world scenarios, the wearer's state may change drastically within a short period. In such cases, historical means may fail to reflect the current true state in a timely manner, instead smoothing or diluting key local features of state abrupt changes at the end of the time series, leading to distorted normalization results and consequently affecting the accuracy of subsequent future state predictions.

[0028] To address this, the solution of this application embodiment is a state prediction method applied to a head-mounted display device, comprising: after detecting a future state prediction instruction for the wearer of the head-mounted display device, acquiring historical state time-series data of the wearer within a preset sliding time window; determining target anchor state data of the historical state time-series data, and determining a feature scaling factor of the historical state time-series data; standardizing the historical state time-series data based on the target anchor state data and the feature scaling factor to obtain standardized historical state time-series data; inputting the standardized historical state time-series data into a pre-trained future state prediction model to obtain standardized future state time-series data predicted by the future state prediction model; and performing the inverse operation of the standardization process on the standardized future state time-series data based on the target anchor state data and the feature scaling factor to obtain the future state time-series data of the wearer.

[0029] It should be noted that, in the embodiments of this application, the target anchor point state data is the historical state data with a later time sequence in the historical state time sequence data, or it is obtained by weighted fusion of historical state data with different time sequences in the historical state time sequence data. In the process of weighted fusion, the later the historical state data is, the greater its corresponding weight.

[0030] This application abandons the traditional Z-score standardization method that uses the arithmetic mean as the center for feature scaling. Instead, it uses time-sensitive target anchor point state data as the central benchmark, constructing a new standardization framework adapted to the non-stationary, highly dynamic temporal data characteristics of head-mounted displays. This ensures that the central benchmark closely reflects the wearer's current real state and accurately reflects the wearer's latest state change trend. It effectively avoids the lag and smoothing effect brought about by mean-centered standardization when facing sudden actions or state transitions. Thus, in the standardization process, feature scaling is performed centered on the target anchor point state data that best represents the wearer's current real state, thereby prioritizing responses to recent state changes. This allows abrupt features to be preserved and enhanced during the standardization process, providing more discriminative input for the prediction model. Ultimately, without increasing model complexity, it significantly improves the accuracy and real-time performance of state prediction.

[0031] The innovation and technical contribution of this application lies in breaking through the reliance of traditional Z-score standardization on the "stationarity assumption" and proposing a dynamic centralization mechanism centered on time-weighted anchor points. This method, by shifting the standardization benchmark from "historical average" to "recent representative," achieves precise adaptation to non-stationary, highly dynamic physiological signals, significantly improving the preprocessing stage's responsiveness to state mutations. Without increasing the complexity of the prediction model, it optimizes feature quality from the data input source, allowing mutation features to be preserved or even enhanced during the standardization process, thereby providing the model with more timely and discriminative input representations.

[0032] Please refer to the description of the eye protection method of the first embodiment above (especially the writing style and structure) to complete the description of the state prediction method of the first embodiment of this application below (focusing on the explanation of terminology, the effect analysis of technical features, and the analysis of the role of each operation step in the technical solution):

[0033] The head-mounted display device in this application embodiment may include, but is not limited to, devices such as MR (Mixed Reality) devices—MR helmets or MR glasses, AR (Augmented Reality) devices—AR helmets or AR glasses, VR (Virtual Reality) devices—VR helmets or VR glasses, XR (Extended Reality) devices—XR helmets or XR glasses, or some combination thereof. In the following embodiments of this application, for ease of description, the head-mounted display device is used as the execution subject for illustration.

[0034] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0035] This application proposes a state prediction method according to a first embodiment.

[0036] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the state prediction method of this application.

[0037] In this embodiment, the state prediction method is applied to a head-mounted display device, and the state prediction method may include steps S100~S500:

[0038] Step S100: After detecting the future state prediction instruction for the wearer of the head-mounted display device, obtain the historical state time sequence data of the wearer within a preset sliding time window;

[0039] It should be noted that, in this embodiment, the future state prediction command refers to a control signal triggered by the control system of the head-mounted display device or an external application to initiate the prediction of the wearer's state at a certain point in the future or within a certain time period. This future state prediction command can be automatically triggered by the system periodically (e.g., once every 10 milliseconds), or it can be triggered in response to specific interactive events (e.g., when the user is about to enter a highly dynamic virtual scene, or when eye-tracking control functions are activated).

[0040] In this embodiment, the wearer's state can be further categorized into various types, such as head state, eye state, mouth state, and nose state. Correspondingly, the future state prediction command can be further categorized into future head state prediction command, future eye state prediction command, future nose state prediction command, and future mouth state prediction command.

[0041] It should also be noted that, in this embodiment, time-series data refers to a series of observations arranged in chronological order. State data refers to the quantitative representation of the wearer's physiological, behavioral, or physical characteristics at a certain moment or within a certain period, reflecting their specific state at that point in time or within that period. Specifically, it can be a snapshot value at a single point in time, or an average value or other statistical measure over a period. State time-series data refers to a series of state data of the wearer arranged in chronological order. Historical state time-series data refers to the state time-series data of the wearer within a preset sliding time window. The preset sliding time window is a pre-set sliding time window, the length of which can be flexibly set according to the needs of the state prediction task (e.g., 50ms, 100ms, or 200ms), and generally ends at the current moment.

[0042] It is easy to understand that in this embodiment, the state data can be further subdivided into head state data, eye state data, mouth state data, etc. Head state data refers to data related to the wearer's head state at a certain moment or over a certain period of time, specifically including head three-dimensional position, head Euler angles, head angular velocity, etc. Eye state data refers to data related to the wearer's eye state at a certain moment or over a certain period of time, specifically including eyeball position, fixation point coordinates, eyelid opening and closing degree, eye movement velocity, blinking frequency, etc.

[0043] This embodiment obtains the wearer's historical state time series data within a preset sliding time window in real time by responding to the future state prediction command, providing the original data foundation for subsequent standardization processing and future state prediction.

[0044] Step S200: Determine the target anchor point state data of the historical state time series data, and determine the feature scaling factor of the historical state time series data.

[0045] It should be noted that, in this embodiment, the target anchor point state data is a reference benchmark used for centralized processing of historical state time series data, and its selection method directly determines the responsiveness of the standardization process to the wearer's current real state.

[0046] Traditional Z-score normalization methods typically use the arithmetic mean within a sliding time window as a reference benchmark for centering, implicitly assuming that the data is approximately stationary within the sliding time window. However, in real-world use cases of head-mounted displays, the wearer's physical state (such as head posture and eye movement) often exhibits significant non-stationarity and abrupt changes due to sudden actions such as rapid head turning and blinking. This causes the mean within the sliding time window to lag significantly behind the current real-world state, failing to accurately reflect the wearer's latest dynamic trends.

[0047] Therefore, this embodiment abandons the traditional mean-centered strategy and proposes a new standardization paradigm centered on "target anchor point state data". This target anchor point state data is not the average of all historical state data within a preset sliding time window, but rather selected from later historical state time series data, or obtained by weighted fusion of multiple historical state data at different time series positions (including at least one historical state data point ranked among the last three in the historical state time series data). The closer the historical state data is to the current moment (i.e., the later the historical state data), the higher the weight is assigned (e.g., using exponential decay weighting). This design allows the target anchor point state data to closely match the wearer's current real state, possessing strong time sensitivity and state foresight. Especially when sudden actions occur, it can quickly capture state transitions, avoiding the smoothing distortion caused by the "tail effect" of traditional means.

[0048] Understandably, the feature scaling factor is used to control the scaling transformation during the standardization process, eliminating dimensional differences between different channels or dimensions in historical state data. In this embodiment, the feature scaling factor can be calculated based on the standard deviation, range, or dynamic fluctuation amplitude of historical state time series data, ensuring that data from each channel is modeled at a uniform scale, thereby improving numerical stability and model convergence efficiency.

[0049] It is worth mentioning that in this embodiment, each channel can be assigned a corresponding feature scaling factor, or multiple channels can share the same feature scaling factor. This embodiment does not impose specific limitations on this and can be set according to actual needs.

[0050] This embodiment constructs a dynamic standardization mechanism for non-stationary time-series signals by introducing target anchor point state data and feature scaling factors. Its core function is to transform the standardized centralized benchmark from "historical average" to "recent representative", thereby prioritizing the response to the wearer's latest state changes, effectively preserving and strengthening abrupt change features, and providing more discriminative and timely input representations for subsequent prediction models.

[0051] Step S300: Based on the target anchor point state data and feature scaling factor, the historical state time series data is standardized to obtain standardized historical state time series data.

[0052] It should be noted that standardization is a crucial preliminary step for improving the quality of model input and enhancing model robustness. In this embodiment, the standardization process includes two sub-processes: First, using the target anchor point state data as a benchmark, the historical state time series data is centered, that is, the target anchor point state data is subtracted point by point to obtain the centered time series; second, using the feature scaling factor as the denominator, the centered data is feature-scaled (normalized) to finally generate standardized historical state time series data.

[0053] Compared to traditional Z-score normalization methods that rely on window mean, the anchor-centered mechanism in this embodiment significantly improves the model's response speed to state abrupt changes. For example, at the moment when the wearer suddenly turns their head, the window mean is still at a low level because it contains a large amount of previous low-speed data, causing the current high-speed state to be "compressed" or "diluted." However, the target anchor state data in this embodiment focuses on recent data and can quickly reflect the high-speed state, so that the abrupt change features still maintain high amplitude and high recognizability after normalization, avoiding information decay.

[0054] Therefore, the technical role of this step is to optimize the quality of input features from the source of data preprocessing without increasing the complexity of the model, so that the future state prediction model can more accurately capture short-term dynamic changes, especially in challenging scenarios such as fast action and high acceleration, and significantly improve the accuracy and real-time performance of future state prediction.

[0055] Step S400: Input the standardized historical state time series data into the pre-trained future state prediction model to obtain the standardized future state time series data predicted by the future state prediction model.

[0056] It should be noted that, in this embodiment, the future state prediction model is a deep learning model pre-trained based on a large amount of historical state data. It can learn long-term dependencies and dynamic evolution patterns from standardized historical state time series data and output prediction results for several future time steps.

[0057] For example, this future state prediction model can be pre-trained using supervised learning. The training process can be based on a large amount of multimodal historical state time-series data collected from wearers in real-world usage scenarios, along with their corresponding future state time-series data, as sample pairs. Specifically, the sample features of each training sample are historical state time-series data within a certain sliding time window (such as head angular velocity, acceleration, eye movement coordinates, etc.), which are preprocessed using the same target anchor standardization method as in this embodiment to form a standardized input sequence. The corresponding sample labels are the actual observed state sequences (i.e., the actual future state time-series data) for several consecutive time steps after the historical window, also encoded using the same standardization parameters to ensure that the model learns the mapping relationship within the standardized space. The training data covers various usage scenarios (such as stationary, walking, rapid head turning, reading, interactive operations, etc.), fully covering typical behavioral patterns such as state abrupt changes and non-stationary dynamics, to improve the model's generalization ability. By minimizing the temporal loss function (such as mean squared error or dynamic time warping loss) between the predicted output and the true label, the model learns the nonlinear mapping law from standardized historical state data to standardized future state data, ultimately obtaining the ability to accurately infer the wearer's future state. This training mechanism ensures that the model is highly consistent with the standardized method proposed in this application in terms of data distribution and processing logic, thereby achieving optimal prediction performance in actual deployment.

[0058] Since the input data has been standardized through the target anchoring mechanism, the model receives high-quality features that have undergone dynamic alignment and scale unification, which helps to accelerate training convergence and improve generalization ability. The standardized future state time series data output by the model are in the same standardized space as the input, which facilitates subsequent unified processing.

[0059] This step realizes the core function of inferring future trends from historical observations and serves as the intelligent decision-making center of the entire state prediction process. The accuracy of its prediction results highly depends on the effectiveness of the standardization process in the preceding steps, and the anchor-centralization mechanism proposed in this embodiment is the key technical support for ensuring this accuracy.

[0060] Step S500 involves performing the inverse operation of standardizing the standardized future state time series data based on the target anchor point state data and the feature scaling factor to obtain the wearer's future state time series data.

[0061] It should be noted that standardized future state time series data itself does not have physical meaning and cannot be directly used for system control (such as rendering viewpoint shifts and eye-tracking compensation). Therefore, it is necessary to restore it to future state time series data in real physical space through inverse normalization (i.e., the inverse operation of normalization).

[0062] This embodiment uses the same target anchor point state data and feature scaling factor as in step S300 to perform the inverse operation of standardization processing on the standardized future state time series data. This ensures the consistency of parameters between the forward standardization and reverse restoration processes, avoiding prediction deviations caused by changes in the target anchor point state data or feature scaling factor. Especially in multi-frame continuous prediction or rolling prediction scenarios, using the same anchor point reference can ensure the smoothness and continuity of the prediction trajectory.

[0063] This step, as the output terminal of the entire prediction process, completes the mapping and restoration from "standardized space" to "physical space," making the prediction results interpretable and executable, and providing high-precision, low-latency state input for downstream tasks such as real-time rendering and interactive response of the head-mounted display device.

[0064] In summary, this embodiment proposes a novel state prediction method for head-mounted display devices. Its core innovation lies in abandoning the static processing method centered on the arithmetic mean in traditional Z-score standardization, and instead using the "target anchor point state data" generated by time-weighted fusion as a dynamic centralization benchmark to construct an adaptive standardization framework that adapts to non-stationary and highly dynamic physiological time-series data.

[0065] This embodiment achieves three major technological breakthroughs:

[0066] 1. Overcoming the limitations of the "stationarity assumption": Traditional mean-centeredness relies on data stationarity, while this embodiment breaks free from this assumption through the anchoring mechanism, making it suitable for real-world scenarios such as rapid head turns, blinks, and sudden actions;

[0067] 2. Enhanced mutation response capability: The target anchor point focuses on the recent state, has strong time sensitivity, can quickly respond to state transitions, and avoids the loss of key features caused by mean smoothing;

[0068] 3. Enhance the quality of prediction model input: Without changing the model structure, optimize the preprocessing stage to significantly improve the discriminativeness and timeliness of input features, thereby improving short-term prediction accuracy and system response speed.

[0069] Based on the first embodiment described above, a state prediction method according to a second embodiment of this application is proposed.

[0070] In this embodiment, the same or similar content as in the above embodiments can be referred to the above description, and will not be repeated hereafter.

[0071] In this embodiment, the future state prediction instruction includes the future head state prediction instruction;

[0072] When the future state prediction instruction is the future head state prediction instruction, the historical state time series data is the historical head state time series data, the target anchor point state data is the target anchor point head state data, the feature scaling factor is the first feature scaling factor, the standardized historical state time series data is the standardized historical head state time series data, the future state prediction model is the future head state prediction model, the standardized future state time series data is the standardized future head state time series data, and the future state time series data is the future head state time series data.

[0073] It should be noted that, in this embodiment, the future head state prediction command is a specific type of future state prediction command, specifically referring to the predictive control signal triggered by the system to achieve head posture-related functions (such as low-latency image rendering, virtual view prediction, motion sickness suppression, head gesture recognition, etc.). The execution goal of this future head state prediction command is to predict the state of the wearer's head at a certain time or within a certain period of time in the future, thereby providing forward-looking input for the graphics engine or interactive system.

[0074] The future head state prediction model is a deep learning model specifically designed for predicting head states. During the training phase, it learns a nonlinear mapping relationship from standardized historical head state time series data to standardized future head state time series data, and can output head pose prediction results for multiple future time steps during the inference phase.

[0075] This embodiment concretizes the general state prediction framework of the first embodiment and applies it to the head state prediction scenario, achieving high-precision, low-latency prediction of the most critical and frequent state variable in head-mounted displays—head posture. Its technical effects are mainly reflected in the following three aspects:

[0076] 1. Precisely Adapting to the Non-Stationary Characteristics of Head Movements: Head movements are typically sudden and highly dynamic, such as rapid scanning, evasive actions, or sudden head turns, causing drastic changes in head state data within a very short time. Traditional mean-centered standardization methods often suffer from severe lag due to the "tailing" effect of historical data when faced with such abrupt changes, leading to distorted input features. However, the target anchor point head state data used in this embodiment, because it focuses on recent high-weight data, closely reflects the true state of current head movements and maintains a high response speed even when abrupt changes occur. This ensures that the standardized input sequence truly reflects the movement transitions, significantly improving the model's prediction accuracy for rapid head movements.

[0077] 2. Improve the real-time performance and stability of graphics rendering and interaction systems: In virtual reality or augmented reality systems, the accuracy of head pose prediction directly determines the latency compensation effect of image rendering. Large prediction deviations can lead to screen tearing, jitter, or increased motion sickness. This embodiment optimizes the preprocessing stage, enabling the future head state prediction model to receive more discriminative input features, thereby obtaining more reliable future head state time-series data. This provides the graphics engine with more accurate viewpoint shift predictions, effectively reducing end-to-end system latency and enhancing immersion and user experience.

[0078] 3. Support for Multi-channel Collaborative Modeling and Independent Parameter Management: This embodiment introduces a proprietary parameter name, "First Feature Scaling Factor," reflecting the design philosophy of differentiated processing for different state modes. Head states involve multiple sensor channels (such as triaxial angular velocity and triaxial acceleration), each with varying dynamic ranges and noise characteristics. Using an independently adjustable scaling factor facilitates refined normalization, preventing any single channel from dominating the overall scale. Simultaneously, this design provides a modular foundation for subsequent expansion to multi-task joint prediction (such as head + eye joint prediction). Each subsystem can share a unified prediction framework but possesses independent parameter configuration space, enhancing system flexibility and maintainability.

[0079] The unique technical contribution of this embodiment in the overall technical solution is that it successfully implements the generalized "target anchor point standardization" mechanism into the core head posture prediction task of head-mounted display devices, completing the key transformation from abstract methodology to specific application scenarios.

[0080] In summary, this embodiment improves the accuracy of head-mounted display devices in predicting the wearer's future head condition.

[0081] In one feasible implementation, the step of determining the target anchor point state data of the historical state time series data in step S200 above may include steps S210 to S230:

[0082] Step S210: Select multiple historical head state data as candidate anchor head state data from the historical head state time series data, and determine the time series position of each candidate anchor head state data in the historical head state time series data.

[0083] It should be noted that, in this embodiment, candidate anchor head state data refers to a set of representative historical observation points selected from historical head state time series data, which are used to participate in subsequent weighted fusion to generate the final target anchor head state data. These candidate data points can be screened using various strategies, such as selecting data from the last N consecutive frames within the sliding time window (e.g., N=3 or 5), or using a hybrid strategy of equal-interval sampling combined with dense sampling at the end.

[0084] Preferably, to ensure high responsiveness to the latest head state, the candidate anchor head state data should include at least one historical head state data that ranks among the last three in terms of temporal position in the historical head state time series data.

[0085] In this embodiment, the temporal position is used to characterize the relative temporal order of a certain historical header state data within the historical header state temporal data. It is typically represented by the time offset or index number between the historical header state data and the current time. For example, if the historical header state temporal data contains 10 time points (t-9 to t), the historical header state data at time t is the latest in temporal position, with an index number of 9 (or a relative position of 0), while the data point at time t-9 is the earliest in temporal position, with an index number of 0 (or a relative position of 9). This temporal position information is a crucial basis for subsequent calculation of weighting coefficients.

[0086] This implementation extracts a set of key sample points with temporal representativeness from the original historical data, providing a data foundation for constructing a dynamic and forward-looking centralized benchmark. Compared to directly using the arithmetic mean of the entire sliding time window as the centralized benchmark, this implementation focuses on recent data that has the greatest impact on the current state, enhancing the system's sensitivity to sudden actions.

[0087] Step S220: Calculate the weight coefficient of each candidate anchor head state data based on its temporal position in the historical head state time series data. The larger the temporal position of the candidate anchor head state data in the historical head state time series data, the larger the weight coefficient of the candidate anchor head state data.

[0088] It should be noted that, in this embodiment, the weighting coefficient is a numerical value used to measure the importance of each candidate anchor point's head state data in the weighted fusion process. Its allocation principle follows the time decay law: the closer a candidate data point is to the current moment, the higher its weighting coefficient, and vice versa. This design is based on a core physiological and behavioral assumption—the wearer's current head state is mainly determined by their recent behavior, and the influence of earlier historical states rapidly weakens over time.

[0089] The weighting coefficients can be calculated using various mathematical functions, such as:

[0090] Exponential decay function: wi = α n-1-ki , α∈(0,1), where wi is the weight coefficient of the i-th candidate anchor head state data, α is the attenuation coefficient, n is the number of historical head state data in the historical head state time series data, and ki is the time series position of the i-th candidate anchor head state data in the historical head state time series data.

[0091] Linear decay function: wi=1-β*(n-1-ki), where wi is the weight coefficient of the i-th candidate anchor head state data, β is the linear decay rate, n is the number of historical head state data in the historical head state time series data, and ki is the time series position of the i-th candidate anchor head state data in the historical head state time series data.

[0092] Fixed high weight strategy: Only the 1-2 candidate anchor points with the largest time sequence position are given high weight (e.g., 0.8, 0.7), and the rest are given lower weight (e.g., 0.3, 0.2).

[0093] Preferably, the weighting coefficients can be normalized after calculation so that their sum is 1, which facilitates subsequent weighted fusion operations.

[0094] This implementation method establishes a time priority mechanism, enabling the system to dynamically differentiate the influence of different historical state data. By assigning higher weights to recent states, this implementation method effectively improves the response speed of the target anchor point to sudden state changes, avoiding the "smoothing delay" problem caused by the traditional mean due to equal weighting. For example, at the moment when the user suddenly turns their head, only the high-speed states of the last 1-2 frames are given dominant weights, thereby causing the generated target anchor point head state data to quickly jump to the high-speed range, truly reflecting the current motion trend.

[0095] Step S230: Based on the weight coefficients of the header state data of each candidate anchor point, perform weighted fusion on the header state data of each candidate anchor point to obtain the target anchor point header state data of the historical header state time series data.

[0096] It should be noted that weighted fusion refers to multiplying the head state data of each candidate anchor point by its corresponding weight coefficient and then summing the results to obtain a comprehensive scalar or vector value, which serves as the final target anchor point head state data. For multidimensional state data (such as three-dimensional angular velocity), this fusion process can be performed independently in each dimension.

[0097] The final fusion result is no longer a "lagging mean" that reflects the overall trend within the window, but a "forward-looking reference point" that focuses on recent dynamics. It can more accurately represent the wearer's current real state, and shows stronger robustness and responsiveness, especially in non-steady motion scenarios.

[0098] This step, as the core of target anchor point generation, completes the information condensation process from "multi-point screening" to "single-point representation," and its output will be directly used as the centralized benchmark in subsequent standardization processing. Through this implementation method, the system can significantly improve the quality and timeliness of input features without increasing model complexity.

[0099] This implementation significantly enhances the dynamic adaptability and engineering feasibility of the standardized preprocessing stage by refining the generation process of target anchor point head state data. Its core technical effect lies in: explicitly selecting key data points at the end of historical head state time-series data as candidate anchor points, and assigning higher weight coefficients based on their temporal position, followed by weighted fusion. This ensures that the generated target anchor point head state data closely matches the wearer's current actual head state, enabling rapid response to sudden movements (such as rapid head turning or instantaneous shaking). This effectively overcomes the lag problem caused by the "tail effect" in traditional mean-centered methods, significantly reducing prediction latency. Simultaneously, using this forward-looking benchmark (i.e., target anchor point head state data) during the centering process avoids "pulling down" the current high-speed state to the historical average level, effectively preserving the amplitude information of abrupt changes, and improving the signal-to-noise ratio and feature discriminativeness of the standardized input data in both the time and frequency domains. This facilitates future head state prediction models to more accurately capture key motion patterns and improve short-term prediction accuracy.

[0100] In addition, the candidate point selection strategy, weight calculation function (such as exponential decay and linear decay) and its parameters (such as decay coefficient α) all support flexible configuration and can be personalized according to different application scenarios (such as games and reading) or individual user behavior characteristics, further enhancing the system's adaptability, universality and user experience.

[0101] In one feasible implementation, step S300 above may include steps S310 to S320:

[0102] Step S310: Based on the target anchor point head state data, the historical head state time series data is centralized to obtain centralized historical head state time series data.

[0103] Step S320: Perform feature scaling on the centralized historical head state time series data based on the first feature scaling factor to obtain standardized historical head state time series data.

[0104] It should be noted that in this embodiment, the standardization process is explicitly decomposed into two sequentially executed sub-processes: centralization processing and feature scaling processing, in order to achieve refined modeling and numerical normalization of historical head state time series data.

[0105] In this embodiment, the centralized processing refers to subtracting the target anchor head state data from the historical head state data corresponding to each time point in the historical head state time series data, using the target anchor head state data generated in step S230 as a reference.

[0106] Through this centralization process, this implementation shifts the "center" of the data distribution to a forward-looking anchor point that reflects the current real state, thereby preserving the dynamic characteristics of the mutation signal and preventing key features from being weakened or smoothed out by the centralization process.

[0107] In this embodiment, feature scaling refers to using the first feature scaling factor as the feature scaling scale to perform amplitude compression or expansion on the centralized data (i.e., centralized historical head state time series data).

[0108] By using this feature scaling process, this implementation effectively eliminates the dimensional differences and amplitude imbalances between different feature channels, thereby improving the numerical stability and convergence speed of the model input.

[0109] The step of determining the feature scaling factor of historical state time series data in step S200 above may include steps S240 to S250:

[0110] Step S240: Calculate the standard deviation of the historical head state time series data, and use the standard deviation of the historical head state time series data as the first feature scaling factor of the historical head state time series data.

[0111] Step S250, or, calculate the standard deviation of the centralized historical head state time series data, and use the standard deviation of the centralized historical head state time series data as the first feature scaling factor of the historical head state time series data.

[0112] This implementation provides two paths for generating the first feature scaling factor. The first method directly calculates the standard deviation of the original historical head state time series data and uses it as the first feature scaling factor (corresponding to step S240). The second method first performs centering processing, then calculates the standard deviation of the centered historical head state time series data and uses it as the first feature scaling factor (corresponding to step S250).

[0113] In contrast, the second approach focuses more on the dispersion of data around the target anchor point, and can dynamically respond to the local fluctuation characteristics of the current state. Especially when there are violent movements such as rapid head turns, it can more accurately capture the instantaneous change amplitude, thereby generating more adaptive scaling parameters.

[0114] The two strategies described above can be flexibly selected or combined according to actual needs, and this implementation method does not impose specific limitations on them.

[0115] This implementation significantly improves the adaptability and modeling accuracy of the standardization preprocessing stage to non-stationary head movements by decoupling the standardization process into two ordered steps: centering and feature scaling. Combined with a dynamic generation mechanism of the target anchor point head state data, this approach enhances the standardization preprocessing stage's adaptability to non-stationary head movements. Specifically, firstly, historical head state time-series data is centered based on the target anchor point head state data. This transforms the data distribution reference from a lagging window mean to a forward-looking anchor point focusing on the current real state, effectively preserving the abrupt changes in characteristics under sudden actions such as rapid head turns and avoiding the attenuation of key information caused by the "tail effect" in traditional methods. Subsequently, a first feature scaling factor is used to normalize the centered data, eliminating dimensional differences and amplitude imbalances between different sensor channels and improving the numerical stability of the input signal.

[0116] Crucially, this implementation proposes two paths for generating feature scaling factors: either directly calculating the standard deviation based on the original historical data (S240), or calculating the standard deviation based on residual fluctuations after centering (S250). The latter can more accurately reflect the local dynamic range around the current state, achieving adaptive adjustment of scale normalization. This two-stage mechanism of "anchor point centering first, then dynamic scaling" not only ensures dual optimization of input features in terms of temporal sensitivity and numerical consistency, but also supports flexible configuration of parameter strategies according to application scenarios. Thus, without increasing model complexity, it significantly enhances the ability of the future head state prediction model to capture highly dynamic and non-stationary behaviors and improves its prediction robustness, providing high-quality pre-processing data for head-mounted displays to achieve low-latency rendering and high-precision interaction.

[0117] In one feasible implementation, the future head state prediction model includes a learnable first feature scaling parameter, and the step of determining the feature scaling factor of historical state time series data in step S200 above may further include step S260:

[0118] Step S260: The learnable first feature scaling parameter is used as the first feature scaling factor of the historical head state time series data.

[0119] It should be noted that, in this embodiment, the learnable first feature scaling parameter refers to a set of parameters embedded within the future head state prediction model and automatically optimized during model training via backpropagation. This parameter replaces or assists in using the traditionally manually calculated standard deviation as a feature scaling factor. This learnable first feature scaling parameter can be a scalar (suitable for all channels sharing a scaling factor) or a vector (each channel corresponding to an independent learnable scaling parameter). Its initial value can be set to the average standard deviation obtained from the training data statistics, and it is continuously adjusted during training based on the gradient of the loss function to find the optimal normalization scale. This learnable first feature scaling parameter, along with other weights in the future head state prediction model, participates in end-to-end training, making the normalization process no longer an independent preprocessing step but an integral part of the overall prediction modeling, adaptively matching the model's requirements for the input scale.

[0120] This implementation method, for the first time, transforms the feature scaling factor in the standardization process from a statistical quantity into a learnable parameter of the model, achieving a paradigm shift from "static preprocessing" to "dynamic joint optimization." Traditional methods rely on the standard deviation of the input data for feature scaling, and their feature scaling factor is independent of the model training process, making it difficult to guarantee optimal matching with the model's internal representation space. This implementation method, however, introduces a learnable first feature scaling parameter, enabling the future head state prediction model to autonomously adjust the input scale during training, optimizing the first feature scaling parameter in reverse to minimize prediction error. This achieves the synergistic evolution of input standardization and predictive modeling. This mechanism not only enhances the system's adaptability to different wearers and usage habits but also strengthens the model's robustness to uncertainties such as sensor noise and individual physiological differences. It fundamentally optimizes the end-to-end information flow efficiency from raw data to future state prediction, providing a new technical path for building a highly integrated and intelligent state prediction system.

[0121] In one feasible implementation, the future head state prediction model includes learnable head state gating parameters, a preset first feature scaling maximum value, and a preset first feature scaling minimum value. The step S200 above, which determines the feature scaling factor for historical state time-series data, may further include steps S270-S280:

[0122] Step S270: Input the learnable head state gating parameters into the preset activation function to obtain the first function activation value corresponding to the head state gating parameters;

[0123] Step S280: Calculate the first feature scaling factor of the historical head state time series data based on the first feature scaling maximum value, the first feature scaling minimum value, and the first function activation value.

[0124] It should be noted that, in this embodiment, the learnable head state gating parameter is one or a set of trainable parameters within the model, used to sense and respond to the wearer's current head movement state (such as stillness, slow movement, violent rotation, etc.), and generate control signals through nonlinear transformation. The first function activation value is the output value obtained after inputting the head state gating parameter into a bounded activation function such as Sigmoid or Tanh, and its range is usually limited to [0,1] or [-1,1], used as interpolation weights. The first feature scaling maximum value and the first feature scaling minimum value are pre-set boundary parameters, representing the maximum and minimum scaling scales allowed under extreme motion states (such as violent shaking or complete stillness), respectively.

[0125] In step S280, the final first feature scaling factor can be calculated using linear interpolation, for example: scale = min_scale + (max_scale - min_scale) * g, where scale is the first feature scaling factor, min_scale is the minimum value of the first feature scaling, max_scale is the maximum value of the first feature scaling, and g is the activation value of the first function. This mechanism allows the first feature scaling factor to be dynamically adjusted based on the future head state prediction model's judgment of the current head state, achieving refined adaptive normalization.

[0126] This implementation introduces a collaborative mechanism between learnable head state gating parameters and preset boundary values, enabling the scaling factor to adjust in real time based on the model's understanding of the wearer's current behavioral patterns. It automatically increases the scaling range to preserve high-frequency details when vigorous movement is detected, and decreases the range to suppress noise interference in stable conditions. This state-aware capability not only enhances the intelligence and contextual adaptability of the standardization process but also embodies the advanced design concept of integrating prior knowledge (such as the relationship between motion intensity and signal fluctuations) into deep learning models. This significantly enhances the system's robustness and prediction accuracy in complex and dynamic real-world scenarios, laying a crucial technological foundation for context-aware, low-latency interaction in head-mounted displays.

[0127] In one example, such as Figure 2The original gating parameter gate (i.e., the head state gating parameter) can be activated by the sigmoid function: g = sigmoid(gate) to obtain the activation value g (i.e. the first function activation value). Then, the activation value g is mapped to the final scaling factor scale (i.e. the first feature scaling factor) through a linear mapping.

[0128] Based on the first embodiment described above, a state prediction method according to a third embodiment of this application is proposed.

[0129] In this embodiment, the same or similar content as in the above embodiments can be referred to the above description, and will not be repeated hereafter.

[0130] In this embodiment, the future state prediction instruction includes a future eye state prediction instruction;

[0131] When the future state prediction instruction is the future eye state prediction instruction, the historical state time series data is the historical eye state time series data, the target anchor point state data is the target anchor point eye state data, the feature scaling factor is the second feature scaling factor, the standardized historical state time series data is the standardized historical eye state time series data, the future state prediction model is the future eye state prediction model, the standardized future state time series data is the standardized future eye state time series data, and the future state time series data is the future eye state time series data.

[0132] It should be noted that, in this embodiment, the future eye state prediction command is a specific type of future state prediction command, specifically referring to the predictive control signal triggered by the system to achieve eye-related functions (such as low-latency image rendering, virtual perspective prediction, motion sickness suppression, etc.). The execution goal of this future eye state prediction command is to predict the state of the wearer's eyes at a certain time or within a certain period of time in the future, thereby providing forward-looking input for the graphics engine or interactive system.

[0133] The future eye state prediction model is a deep learning model specifically designed to predict eye state. During the training phase, it learns a nonlinear mapping relationship from standardized historical eye state time series data to standardized future eye state time series data, and can output eye pose prediction results for multiple future time steps during the inference phase.

[0134] This embodiment concretizes the general state prediction framework of the second embodiment and applies it to the eye state prediction scenario, achieving high-precision, low-latency prediction of the most critical and frequent state variable in head-mounted displays—eye posture. Its technical effects are mainly reflected in the following three aspects:

[0135] 1. Precisely Adapting to the Non-Stationary Characteristics of Eye Movements: Eye movements are typically sudden and highly dynamic, with eye state data often changing drastically within a very short time. Traditional mean-centered standardization methods often lag significantly due to the "tailing" effect of historical data when faced with such sudden changes, leading to distorted input features. However, the target anchor point eye state data used in this embodiment, because it focuses on recent high-weight data, closely reflects the true state of current eye movements and maintains a high response speed even when sudden changes occur. This ensures that the standardized input sequence accurately reflects movement transitions, significantly improving the model's prediction accuracy for rapid eye movements.

[0136] 2. Improve the real-time performance and stability of graphics rendering and interaction systems: In virtual reality or augmented reality systems, the accuracy of eye pose prediction directly determines the latency compensation effect of image rendering. Large prediction deviations can lead to screen tearing, jitter, or increased motion sickness. This embodiment optimizes the preprocessing stage, enabling the future eye state prediction model to receive more discriminative input features, thereby obtaining more reliable future eye state time-series data. This provides the graphics engine with more accurate viewpoint shift predictions, effectively reducing end-to-end system latency and enhancing immersion and user experience.

[0137] 3. Support for Multi-channel Collaborative Modeling and Independent Parameter Management: This embodiment introduces a proprietary parameter name, "Second Feature Scaling Factor," reflecting the design philosophy of differentiated processing for different state modalities. Eye states involve multiple sensor channels (such as eye movement velocity, eyelid opening / closing, and gaze coordinates), each with varying dynamic ranges and noise characteristics. Using an independently adjustable scaling factor facilitates refined normalization, preventing any single channel from dominating the overall scale. Simultaneously, this design provides a modular foundation for subsequent expansion to multi-task joint prediction (such as eye + head joint prediction). Each subsystem can share a unified prediction framework but possesses independent parameter configuration space, enhancing system flexibility and maintainability.

[0138] The unique technical contribution of this embodiment in the overall technical solution is that it successfully implements the generalized "target anchor point standardization" mechanism into the core eye posture prediction task of head-mounted display devices, completing the key transformation from abstract methodology to specific application scenarios.

[0139] In summary, this embodiment improves the accuracy of the head-mounted display device in predicting the wearer's future eye condition.

[0140] In one feasible implementation, the step of determining the target anchor point state data of the historical state time series data in step S200 above may include steps A10 to A30:

[0141] Step A10: Select multiple historical eye state data as candidate anchor point eye state data from the historical eye state time series data, and determine the temporal position of each candidate anchor point eye state data in the historical eye state time series data.

[0142] It should be noted that, in this embodiment, candidate anchor point eye state data refers to a set of representative historical observation points selected from historical eye state time series data, which are used to participate in subsequent weighted fusion to generate the final target anchor point eye state data.

[0143] Preferably, to ensure high responsiveness to the latest eye condition, the candidate anchor eye condition data should include at least one historical eye condition data point that ranks among the last three in the historical eye condition time series data.

[0144] Step A20: Calculate the weight coefficient of each candidate anchor point eye state data based on its temporal position in the historical eye state time series data. The larger the temporal position of the candidate anchor point eye state data in the historical eye state time series data, the larger the weight coefficient of the candidate anchor point eye state data.

[0145] Step A30: Based on the weight coefficients of the eye state data of each candidate anchor point, the eye state data of each candidate anchor point is weighted and fused to obtain the target anchor point eye state data of the historical eye state time series data.

[0146] In one feasible implementation, step S300 above may include steps S330~S340:

[0147] Step S330: Based on the target anchor point eye state data, the historical eye state time series data is centralized to obtain centralized historical eye state time series data.

[0148] Step S340: Perform feature scaling on the centralized historical eye state time series data based on the second feature scaling factor to obtain standardized historical eye state time series data.

[0149] In this embodiment, the centralization process refers to subtracting the target anchor point eye state data from the historical eye state data corresponding to each time point in the historical eye state time series data, using the target anchor point eye state data generated in step A30 as a reference.

[0150] In this embodiment, feature scaling refers to using the second feature scaling factor as the feature scaling scale to perform amplitude compression or expansion on the centered data (i.e., centered historical eye state time series data).

[0151] The step of determining the feature scaling factor of historical state time series data in step S200 above may include steps A40 to A50:

[0152] Step A40: Calculate the standard deviation of the historical eye state time series data, and use the standard deviation of the historical eye state time series data as the second feature scaling factor of the historical eye state time series data.

[0153] Step A50, or, calculate the standard deviation of the centralized historical eye state time series data, and use the standard deviation of the centralized historical eye state time series data as the second feature scaling factor of the historical eye state time series data.

[0154] In one feasible implementation, the future eye state prediction model includes a learnable second feature scaling parameter, and the step of determining the feature scaling factor of the historical state time series data in step S200 above may further include step A60:

[0155] Step A60: The learnable second feature scaling parameter is used as the second feature scaling factor for the historical eye state time series data.

[0156] It should be noted that, in this embodiment, the learnable second feature scaling parameter refers to a set of parameters embedded within the future eye state prediction model and automatically optimized during model training via backpropagation. This parameter replaces or assists in using the traditionally manually calculated standard deviation as the feature scaling factor. This learnable second feature scaling parameter can be a scalar (suitable for all channels sharing a scaling factor) or a vector (each channel corresponding to an independent learnable scaling parameter). Its initial value can be set to the average standard deviation obtained from the training data statistics, and it is continuously adjusted during training based on the gradient of the loss function to find the optimal normalization scale. This learnable second feature scaling parameter, along with other weights in the future eye state prediction model, participates in end-to-end training, making the normalization process no longer an independent preprocessing step but an integral part of the overall prediction modeling, adaptively matching the model's requirements for the input scale.

[0157] In one feasible implementation, the future eye state prediction model includes learnable eye state gating parameters, a preset maximum value for second feature scaling, and a preset minimum value for second feature scaling. The step S200 above, which determines the feature scaling factor for historical state time-series data, may further include steps A70-A80:

[0158] Step A70: Input the learnable eye state gating parameters into the preset activation function to obtain the second function activation value corresponding to the eye state gating parameters;

[0159] Step A80: Calculate the second feature scaling factor of the historical eye state time series data based on the maximum value of the second feature scaling, the minimum value of the second feature scaling, and the activation value of the second function.

[0160] It should be noted that, in this embodiment, the learnable eye state gating parameter is one or a set of trainable parameters within the model, used to sense and respond to the wearer's current eye movement state (such as stillness, slow movement, violent rotation, etc.), and generate control signals through nonlinear transformation. The second function activation value is the output value obtained after inputting the eye state gating parameter into a bounded activation function such as Sigmoid or Tanh, and its range is usually limited to [0,1] or [-1,1], used as interpolation weights. The maximum and minimum values ​​of the second feature scaling are pre-set boundary parameters, representing the maximum and minimum scaling scales allowed under extreme motion states, respectively.

[0161] To facilitate understanding of the state prediction method in the above embodiments of this application, a specific embodiment is provided:

[0162] like Figure 3 As shown, in this specific embodiment, an AAR (Adaptive Anchor Residual Normalization Layer) module can be constructed, and it can have two model modes. The first mode is the normalization mode, i.e., mode=norm, which is used to execute step S200 to determine the target anchor state data of the historical state time series data and determine the feature scaling factor of the historical state time series data, and step S300 to standardize the historical state time series data based on the target anchor state data and the feature scaling factor to obtain standardized historical state time series data, and to standardize the model input. The second mode is the denormalization mode, i.e., mode=denorm, which is used to execute the inverse operation of step S500 to standardize the standardized future state time series data based on the target anchor state data and the feature scaling factor to obtain the wearer's future state time series data, and to standardize the model output.

[0163] like Figure 4As shown, in the normalization mode of mode=norm, the input historical state time series data is processed by the gating module in the AAR module to generate a feature scaling factor and by the dynamic anchor calculation module to determine the target anchor state data. Then, it is first centered by subtracting the target anchor state data from the original historical state time series data to obtain centered historical state time series data. Then, it is multiplied by the feature scaling factor to complete the feature scaling process. Finally, the output is the standardized historical state time series data norm_data, the target anchor state data anchor, and the feature scaling factor scale. The standardized historical state time series data norm_data is provided to the future state prediction model, and the target anchor state data anchor and the feature scaling factor scale are used by the AAR module in the inverse normalization mode.

[0164] like Figure 5 As shown, in the inverse normalization mode of mode=denorm, the standardized future state time series data norm_data2 output by the future state prediction model is first divided by the feature scaling factor scale, and then the target anchor state data anchor is added back to complete the inverse operation of the normalization process, and finally outputs the future state time series data denorm_data in the real physical space.

[0165] It should be noted that the above embodiments are only used to assist in understanding this application and do not constitute a limitation on the state prediction method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0166] In addition, please refer to Figure 6 , Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the state prediction method in this application embodiment.

[0167] This application also provides a head-mounted display device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the state prediction method in the above embodiments.

[0168] The head-mounted display devices in the embodiments of this application may include, but are not limited to, head-mounted display devices such as Mixed Reality (MR) glasses and MR helmets, Augmented Reality (AR) glasses and AR helmets, Virtual Reality (VR) glasses and VR helmets, Extended Reality (XR) glasses and XR helmets, or some combination thereof.

[0169] The following is for reference. Figure 6 It shows a structural schematic diagram of a head-mounted display device suitable for implementing the embodiments of this application. Figure 6 The head-mounted display device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0170] like Figure 6 As shown, the head-mounted display device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the head-mounted display device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the head-mounted display to communicate wirelessly or wiredly with other devices to exchange data. While head-mounted display devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0171] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0172] The head-mounted display device provided in this application, employing the state prediction method described in the above embodiments, can improve the accuracy of the head-mounted display device in predicting the wearer's future state. Compared with the prior art, the beneficial effects of the head-mounted display device provided in this application are the same as those of the state prediction method provided in the above embodiments, and other technical features of the head-mounted display device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0173] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0174] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the above claims.

[0175] In addition, this application also provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to perform the steps of the state prediction method in the above embodiments.

[0176] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0177] The aforementioned computer-readable storage medium may be included in the head-mounted display device; or it may exist independently and not assembled into the head-mounted display device.

[0178] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the head-mounted display device, the head-mounted display device: after detecting a future state prediction instruction for the wearer of the head-mounted display device, acquires historical state time-series data of the wearer within a preset sliding time window; determines target anchor state data of the historical state time-series data and determines a feature scaling factor for the historical state time-series data; performs standardization processing on the historical state time-series data based on the target anchor state data and the feature scaling factor to obtain standardized historical state time-series data; inputs the standardized historical state time-series data into a pre-trained future state prediction model to obtain standardized future state time-series data predicted by the future state prediction model; and performs the inverse operation of standardization processing on the standardized future state time-series data based on the target anchor state data and the feature scaling factor to obtain the wearer's future state time-series data.

[0179] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0180] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0181] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0182] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for performing the steps of the above-described state prediction method, which can improve the accuracy of the head-mounted display device in predicting the wearer's future state. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the state prediction method provided in the above embodiments, and will not be repeated here.

[0183] Furthermore, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the state prediction method as described in the above embodiments.

[0184] The computer program product provided in this application can improve the accuracy of head-mounted display devices in predicting the wearer's future state. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the state prediction method provided in the above embodiments, and will not be repeated here.

[0185] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A state prediction method characterized by, The state prediction method is applied to a head-mounted device, and the method comprises: After detecting a future state prediction instruction for a wearer of the head-mounted device, obtaining historical state time series data of the wearer in a preset sliding time window; determining target anchor point state data of the historical state time series data, and determining a feature scaling factor of the historical state time series data; based on the target anchor point state data and the feature scaling factor, the historical state time series data is standardized to obtain standardized historical state time series data; inputting the standardized historical state time series data into a pre-trained future state prediction model to obtain standardized future state time series data predicted by the future state prediction model; based on the target anchor point state data and the feature scaling factor, the inverse operation of the standardization processing is performed on the standardized future state time series data to obtain the future state time series data of the wearer; wherein the step of determining the target anchor point state data of the historical state time series data comprises: filtering a plurality of historical state data from the historical state time series data as candidate anchor point state data, and determining the time sequence position of each candidate anchor point state data in the historical state time series data; according to the time sequence position of each candidate anchor point state data in the historical state time series data, the weight coefficient of each candidate anchor point state data is calculated, wherein the greater the time sequence position of the candidate anchor point state data in the historical state time series data, the greater the weight coefficient of the candidate anchor point state data; according to the weight coefficient of each candidate anchor point state data, each candidate anchor point state data is weighted and fused to obtain the target anchor point state data of the historical state time series data.

2. The state prediction method according to claim 1, characterized by, The future state prediction instruction comprises a future head state prediction instruction; in the case that the future state prediction instruction is the future head state prediction instruction, the historical state time series data is historical head state time series data, the target anchor point state data is target anchor point head state data, the feature scaling factor is a first feature scaling factor, the standardized historical state time series data is standardized historical head state time series data, the future state prediction model is a future head state prediction model, the standardized future state time series data is standardized future head state time series data, the future state time series data is future head state time series data, the historical state data is historical head state data, and the candidate anchor point state data is candidate anchor point head state data.

3. The state prediction method of claim 2, wherein, The step of standardizing the historical state time series data based on the target anchor point state data and the feature scaling factor to obtain standardized historical state time series data comprises: based on the target anchor point head state data, the historical head state time series data is centralized to obtain centralized historical head state time series data; based on the first feature scaling factor, the centralized historical head state time series data is feature-scaled to obtain standardized historical head state time series data; The step of determining the feature scaling factor of the historical state time series data comprises: calculating the standard deviation of the historical head state time series data, and taking the standard deviation of the historical head state time series data as the first feature scaling factor of the historical head state time series data; or calculating the standard deviation of the centralized historical head state time series data, and taking the standard deviation of the centralized historical head state time series data as the first feature scaling factor of the historical head state time series data.

4. The state prediction method according to claim 2, characterized by, The future head state prediction model comprises a learnable first feature scaling parameter, and the step of determining the feature scaling factor of the historical state time series data comprises: taking the learnable first feature scaling parameter as the first feature scaling factor of the historical head state time series data.

5. The state prediction method of claim 2, wherein, The future head state prediction model comprises a learnable head state gating parameter, a preset first feature scaling maximum value and a preset first feature scaling minimum value, and the step of determining the feature scaling factor of the historical state time series data comprises: inputting the learnable head state gating parameter into a preset activation function to obtain a first function activation value corresponding to the head state gating parameter; calculating the first feature scaling factor of the historical head state time series data according to the first feature scaling maximum value, the first feature scaling minimum value and the first function activation value.

6. The state prediction method of claim 1, wherein The future state prediction instruction comprises a future eye state prediction instruction. In a case where the future state prediction instruction is the future eye state prediction instruction, the historical state time series data is historical eye state time series data, the target anchor state data is target anchor eye state data, the feature scaling factor is a second feature scaling factor, the standardized historical state time series data is standardized historical eye state time series data, the future state prediction model is a future eye state prediction model, the standardized future state time series data is standardized future eye state time series data, the future state time series data is future eye state time series data, the historical state data is historical eye state data, and the candidate anchor state data is candidate anchor eye state data.

7. The state prediction method according to claim 6, characterized by, The step of performing the standardization processing on the historical state time series data based on the target anchor state data and the feature scaling factor to obtain standardized historical state time series data comprises: performing the centralization processing on the historical eye state time series data based on the target anchor eye state data to obtain centralized historical eye state time series data; performing the feature scaling processing on the centralized historical eye state time series data based on the second feature scaling factor to obtain standardized historical eye state time series data; The step of determining the feature scaling factor of the historical state time series data comprises: calculating the standard deviation of the historical eye state time series data, and taking the standard deviation of the historical eye state time series data as the second feature scaling factor of the historical eye state time series data; or calculating the standard deviation of the historical eye state time series data, and taking the standard deviation of the historical eye state time series data as the second feature scaling factor of the historical eye state time series data; or A standard deviation of the centered historical ocular state time series data is computed and used as a second feature scaling factor for the historical ocular state time series data.

8. A head-mounted device, comprising: The device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the state prediction method according to any one of claims 1 to 7.

Images