A multi-sensor spatio-temporal fusion nystagmus-posture correlation analysis method and system

Abstract

This invention discloses a multi-sensor spatiotemporal fusion method and system for oculomotor attitude correlation analysis, belonging to the field of biosignal processing and motion navigation technology. The method simultaneously acquires eye images and head inertial data, merging them into a fused data stream through spatiotemporal alignment and coordinate unification; calculates the spatial angle between eye movement and head movement vectors and extracts dynamic correlation features; based on these features, outputs a motion pattern identifier by combining a kinematically constrained state inference model; generates a three-dimensional head motion navigation command sequence based on this identifier, and performs closed-loop adjustments based on real-time state feedback during execution. This invention achieves quantitative analysis of the correlation between specific eye movement patterns and head movements, and adaptive motion guidance based on state feedback.

Description

Technical Field

[0001] This invention relates to the fields of biosignal processing and motion navigation technology, and more specifically, to a method and system for multi-sensor spatiotemporal fusion oculomotor attitude correlation analysis. Background Technology

[0002] The clinical identification and management of benign paroxysmal positional vertigo (BPPV) relies on the observation and interpretation of characteristic nystagmus induced by specific head position changes. Currently, this process is highly dependent on the operator's experience, and suffers from strong subjectivity, difficulty in quantification and standardization. With the development of sensing technology, solutions using video nystagmus maps or inertial sensors for assisted assessment have emerged. However, existing sensor-based assisted technologies either focus only on single-modal signals (such as eye movements or head movements only), or, while mentioning multiple sensors, do not fully consider the dynamic vector relationship between eye and head movements in three-dimensional space and their coupling with physiological structures. This makes it difficult to directly infer the motion state from the data, and therefore cannot provide precise guidance for the operation.

[0003] Existing technologies include solutions that utilize multi-sensor data fusion to improve the accuracy of environmental perception or map building. For example, Chinese patent CN114359408A discloses a spatiotemporal fusion optimization method for multi-source sensors, which improves the accuracy of subsequent data fusion by performing spatiotemporal alignment and deviation correction on data of the same target object perceived by different sensors. However, this method focuses on the calibration drift problem after long-term sensor operation and does not address how to model the deep spatiotemporal correlation between human physiological signals and motion signals based on anatomy and physiology. Another Chinese patent CN120510315A discloses a dynamic spatiotemporal synchronous mapping method based on multi-sensor data fusion, which improves the spatiotemporal synchronization of map building through modules such as point cloud distortion correction and time alignment. This solution focuses on the synchronization and fusion of environmental geometric information, and its processing of the correlation analysis between objects and targets and physiological motion signals, which requires inference of microscopic motion states, is fundamentally different. Therefore, a multi-sensor spatiotemporal fusion method and system for oculomotor attitude correlation analysis is proposed to address the above problems. Summary of the Invention

[0004] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a multi-sensor spatiotemporal fusion oculomotor attitude correlation analysis method and system, addressing the problems in the prior art that make it difficult to perform high-precision spatiotemporal fusion and deep correlation analysis of eye movement and head motion signals, as well as the lack of closed-loop motion guidance capabilities based on real-time state feedback.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multi-sensor spatiotemporal fusion method for oculomotor attitude correlation analysis, comprising the following steps: S1, through hardware synchronization trigger signal, synchronously acquires eye image sequence and head inertial measurement data sequence of target object; this hardware synchronization method is used to ensure that multi-source heterogeneous sensor data have a unified time reference and eliminate time asynchronous error caused by independent clock drift.

[0006] S2, process the eye image sequence to generate a three-dimensional eye motion vector sequence based on the head coordinate system; process the head inertial measurement data sequence to generate a head rotation axis vector sequence; convert the raw sensor data into a vector form that uniformly represents the direction and speed of motion, providing basic data for subsequent spatiotemporal correlation analysis.

[0007] S3, using the timestamp of the head inertial measurement data sequence as a reference, performs time interpolation alignment on the eyeball three-dimensional motion vector sequence, and unifies the eyeball three-dimensional motion vector to the head coordinate system through a pre-calibrated coordinate transformation matrix to form a spatiotemporally synchronized fused data stream; through time interpolation and spatial coordinate transformation, achieves accurate correspondence between eyeball movement and head movement in physical space, and constructs a data structure that can be used for joint analysis.

[0008] S4, calculate the dynamic spatial angle between the eye movement vector and the head rotation axis vector in the fused data stream, and extract the feature parameter set of the dynamic spatial angle changing over time; the spatial angle quantifies the instantaneous geometric relationship between the nystagmus plane and the head rotation plane, and its dynamic change characteristics reflect the specific pattern of the vestibular-ocular reflex.

[0009] S5, input the feature parameter set and the current head movement pattern into a preset correlation analysis model. The model is constructed based on preset kinematic constraints to obtain and output the corresponding movement pattern identifier. The correlation analysis model maps continuous eye-movement-head movement correlation data into discrete movement state categories by matching observation features with preset constraints.

[0010] S6. Based on the motion pattern identifier, a preset three-dimensional head target posture sequence is mapped, and based on the difference between the real-time head posture and the target posture, a three-dimensional spatial navigation command is generated and output. The analyzed motion state is converted into specific spatial action guidance, realizing a closed loop from state recognition to operation guidance.

[0011] Furthermore, step S2, which generates a three-dimensional eye movement vector sequence, specifically includes: The pupil center displacement is extracted from the eye image sequence, and the horizontal and vertical rotation components of the eyeball are calculated. The rotation of the eyeball in the image plane is estimated based on the two-dimensional translation of the pupil center.

[0012] The torsional rotation component of the eyeball around the visual axis is calculated by matching iris texture features or using polar coordinate transformation phase correlation method; high-precision detection of the degree of freedom of eyeball rotation around the optical axis is achieved by utilizing the unique texture of the iris or phase information in polar coordinates of the image.

[0013] By combining the horizontal, vertical, and torsional rotation components, a three-dimensional motion vector characterizing the instantaneous rotational angular velocity of the eyeball is constructed. The rotational information from the three orthogonal directions is then synthesized into a complete three-dimensional description of the eyeball's rotational state.

[0014] Furthermore, the feature parameter set extracted in step S4 specifically includes: The maximum value of the dynamic spatial angle within one characteristic motion cycle; this maximum value represents the peak value of the deviation between the nystagmus plane and the head rotation plane.

[0015] The time point at which the maximum value is reached; this time point reflects the location where the key phase occurs in the characteristic motion pattern.

[0016] And, the average rate of change before and after reaching the maximum value. The rate of change describes the trend of the spatial angle rising and falling near the peak, used to distinguish different dynamic patterns.

[0017] Furthermore, the correlation analysis model in step S5 is a probabilistic model based on discrete state space and Bayesian update, and its operation process includes: Based on the current head movement pattern and movement state, a predefined state transition probability matrix is ​​queried to predict the state probability distribution of the previous moment. This prediction step relies on prior kinematic knowledge to infer the possible evolution of the movement state under the current head movement.

[0018] Based on the currently extracted set of feature parameters, the observation likelihood of each discrete state is calculated; this calculation step evaluates the degree of matching between the currently observed feature data and each hypothetical state.

[0019] Based on the predicted probability distribution and the observation likelihood, the posterior probability distribution of the current state is obtained through Bayesian update, and the state with the highest posterior probability is output as the motion pattern identifier. By fusing prior predictions and current observations through the Bayesian framework, the optimal probability estimate of the current motion state is obtained.

[0020] Furthermore, the construction of the state transition probability matrix incorporates a correction factor related to the head rotation angular acceleration: When the absolute value of the head rotation angular acceleration exceeds a first preset threshold, the state transition probability in the same direction as the head rotation is reduced, and the reduced probability value is allocated to the transition probability of maintaining the original state. This correction factor simulates the physical effect of inertia or fluid resistance causing lag in the response of a moving body during violent acceleration, making the state prediction more consistent with physical laws.

[0021] Furthermore, it also includes signal interference immunity steps: Real-time monitoring of head rotation angular velocity; When the head rotation angular velocity exceeds a second preset threshold, it is determined that the current eye movement is dominated by the physiological vestibular reflex. The fused data stream for the corresponding time period is marked as invalid, and the extraction of the feature parameter set and the updating of the correlation analysis model are paused. This step is used to filter out physiological eye movement interference caused by the user's active and rapid head turning, ensuring that the analysis object is the target feature movement pattern and improving the robustness of the system.

[0022] Furthermore, the generation of three-dimensional space navigation instructions in step S6 specifically includes: The motion pattern identifier is matched with a pre-stored standard motion trajectory library to obtain an ordered three-dimensional head target posture sequence; based on the currently identified state, a predefined optimal action sequence designed for that state is invoked.

[0023] Based on real-time acquired head posture data, the rotation axis direction and rotation angle between the current posture and the next target posture are calculated; the abstract target posture is transformed into specific, immediately executable rotational motion parameters.

[0024] The rotation axis direction and rotation angle are converted into visual, auditory, or tactile guidance commands and output. Through a multimodal human-computer interaction interface, intuitive and clear operation instructions are provided to the user.

[0025] Furthermore, this includes the closed-loop adjustment steps for navigation commands: During the execution of navigation instructions, steps S4 to S5 are repeated to update the motion mode identifier in real time, thereby achieving real-time tracking of otolith position; The updated motion pattern identifier is compared with the expected trajectory. If the deviation exceeds the tolerance, the subsequent target attitude sequence and navigation commands are regenerated based on the latest motion pattern identifier. By comparing real-time status feedback with the expected path, the guidance strategy is dynamically adjusted, enabling the system to adapt to individual differences and uncertainties in the operation process, thus achieving adaptive closed-loop navigation.

[0026] Furthermore, it also includes a task completion assessment step: During navigation, the sum of posterior probabilities of states belonging to a preset subset of task completion states output by the correlation analysis model is continuously monitored; this sum of probabilities comprehensively reflects the system's confidence that the moving body has reached the target area.

[0027] When the sum of the probabilities continuously exceeds a third preset threshold for a third preset duration, and simultaneously the set of characteristic parameters is detected to be below a resting threshold, a task phase completion prompt signal is generated. Combining the high-confidence state with the disappearance of characteristic motion signals provides an objective and quantitative basis for terminating the operation process.

[0028] An oculomotor attitude correlation analysis system for implementing the above method using multi-sensor spatiotemporal fusion includes: The data acquisition module is used to synchronously acquire eye image sequences and head inertial measurement data sequences through hardware synchronous trigger signals; this module is responsible for acquiring raw, synchronous multimodal physiological and motion signals.

[0029] The data processing and fusion module, connected to the data acquisition module, is used to execute steps S2 and S3 to generate a spatiotemporally synchronized fused data stream. This module is responsible for preprocessing, coordinate transformation, and time-space alignment of the raw data to form fused data that can be used for advanced analysis.

[0030] The feature extraction and analysis module, connected to the data processing and fusion module, is used to execute step S4 and extract the dynamic spatial angle feature parameter set; this module calculates and quantifies the core motion-related features from the fused data.

[0031] The association analysis module, connected to the feature extraction and analysis module, has a built-in association analysis model for executing step S5 and outputting motion pattern identifiers; this module runs the core algorithm model to map feature data into meaningful motion state classification results.

[0032] The navigation instruction generation and output module, connected to the correlation analysis module, is used to execute step S6, generating and outputting three-dimensional spatial navigation instructions. This module generates specific spatial guidance instructions based on the state analysis results and outputs them through a human-machine interface.

[0033] The technical effects and advantages of this invention are as follows: This invention acquires eye images and head inertial data through hardware synchronous triggering, and performs high-precision alignment and fusion using temporal interpolation and a pre-calibrated spatial transformation matrix, unifying eye movement information into the head coordinate system. This process constructs a temporally synchronized and spatially unified eye-head motion joint data stream, providing a precise data foundation for subsequent quantitative correlation analysis. By calculating the dynamic spatial angle between the eye movement vector and the head motion rotation axis vector in this data stream, and extracting its peak value, temporal sequence, and trend characteristics, the observed nystagmus phenomenon is quantitatively correlated with the head motion that triggers it in three-dimensional space, thereby transforming subjective visual observation into a calculable and comparable set of objective features.

[0034] Based on the extracted dynamic spatial correlation features, this invention employs a correlation analysis model that integrates kinematic constraints and probabilistic reasoning. This model discretizes the possible states of the target moving object, performs prior predictions based on head movements and predefined state transition probabilities including physical correction factors, and calculates state likelihood by combining real-time observation features. It then continuously updates the most probable motion pattern identifier using Bayesian methods. This method enables dynamic probabilistic tracking of intrinsic motion states that cannot be directly observed. The angular acceleration correction factor in the model simulates motion lag effects, which helps improve the physical plausibility of state predictions.

[0035] Based on real-time inferred motion pattern identifiers, the system can match pre-stored standard motion trajectories and generate corresponding 3D head target posture sequences. By calculating the difference between the real-time head posture and the target posture, and converting it into specific rotation axis and angle commands, the system guides the user's operation in a multimodal manner. During the guidance process, the system continuously tracks the status and updates the identifiers, comparing them with the expected trajectory. When a path deviation is detected, subsequent navigation commands can be dynamically adjusted to form closed-loop control. This helps adapt the guidance process to individual differences and uncertainties in operation.

[0036] The system incorporates anti-interference logic based on head angular velocity thresholds. When rapid head rotation is detected, the system determines that the current eye movement is primarily caused by the physiological vestibular reflex and pauses the analysis of data for that period. This mechanism helps distinguish between target characteristic movement patterns and interfering physiological responses, reducing misjudgments and thus improving the system's analysis accuracy in real-world scenarios. Furthermore, by continuously monitoring the probability of task completion states and the disappearance of characteristic motion signals, the system can provide a relatively objective, data- and model-based basis for determining the completion of each stage of the operation. Attached Figure Description

[0037] Fig. 1 This is a flowchart illustrating the overall workflow of the method of the present invention; Fig. 2This is a flowchart of the spatiotemporal fusion and feature extraction process of the present invention; Fig. 3 This is a branch structure diagram of the state reasoning and decision-making logic of the present invention; Fig. 4 This is a flowchart of the closed-loop navigation guidance and system adjustment process of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] Example 1 As attached Figs. 1 to 4 The biological motion signal processing and navigation system based on multi-sensor spatiotemporal fusion, as shown, is implemented as follows: I. System Hardware Implementation Foundation The operation of this system relies on a set of high-precision, time-synchronized multimodal motion data acquisition and processing hardware to achieve synchronous measurement and analysis of head spatial motion and fine eye movements. In one embodiment, the hardware configuration and key parameters are as follows: 1. First sensor (eye motion acquisition unit): using a sampling frequency of... High-speed near-infrared video acquisition device. The value range is from 60Hz to 200Hz, and in this embodiment... The refresh rate was set to 100Hz. This device is used to acquire high-resolution sequences of eye images containing details of the pupil and iris texture. The device is coupled to a head fixation device via a rigid connection bracket to ensure that the spatial pose of its optical center relative to the head remains constant throughout the measurement process.

[0040] 2. Second sensor (head motion acquisition unit): using a sampling frequency of... Microelectromechanical systems inertial measurement unit, The value range is from 100Hz to 500Hz, and in this embodiment... The frequency is set to 200Hz. This unit is used to synchronously acquire three-axis angular velocity and three-axis acceleration data of the head. This unit and the aforementioned eye acquisition device are fixed on the same rigid bracket to form a unified head-mounted sensing module.

[0041] 3. Synchronization and Processing Unit: Employs an embedded computer system. This system is equipped with a synchronization clock generation circuit, which simultaneously sends synchronization pulses to the first and second sensors via a hardware trigger signal line. This ensures that all acquired data packets are appended with a precise timestamp originating from the same physical clock, achieving microsecond-level time synchronization accuracy.

[0042] 4. Output Unit: An optical see-through head-mounted display or an external display screen is used to visualize 3D navigation commands, real-time analysis data, and system status.

[0043] II. Detailed Implementation Process of the Method and Steps S1: Simultaneously acquire eye image sequences and head inertial measurement data sequences. The processing unit sends hardware synchronization trigger pulses to the first and second sensors. The first sensor, at a frequency... Output image frame sequence, with each frame appended with a timestamp based on the pulse synchronization. The second sensor uses frequency Output angular velocity data sequence Each set of data is appended with a synchronization timestamp. The system establishes two data buffers to store the above data in chronological order, forming an initial eye image sequence and a head inertial measurement data stream.

[0044] S2: Convert and generate a 3D eye movement vector sequence and a head rotation axis vector sequence. Generate a three-dimensional motion vector sequence of the eyeball For each valid image frame in the eye image sequence, perform the following operations: The input is a grayscale image. The pupil center localization algorithm, based on threshold segmentation and ellipse fitting, is used for processing. The output of this algorithm is the pixel coordinates of the pupil center in the image coordinate system. And the fitting parameters of the ellipse; if the fitting confidence level is higher than a preset threshold... (For example If the pupil detection is successful, then the pupil detection is considered valid; otherwise, the eye movement vector corresponding to that frame is set to zero.

[0045] For two consecutive valid images and Calculate the pixel displacement of the pupil center Based on the pre-calibrated internal parameters (focal length) of the first sensor. pixel size This converts pixel displacement into an approximate rotation angle of the eyeball in the camera coordinate system. ,in , .

[0046] Extraction of ocular torsion component: To obtain the rotational component of the eyeball around the visual axis The system employs an iris texture feature matching algorithm. Scale-invariant feature transform descriptors are extracted from the segmented iris region of the image. Feature point matching is performed between consecutive frames, and a random sampling consensus algorithm is used to remove mismatches. Finally, based on motion estimation of the matched point pairs, the ocular torsional angular velocity components are calculated. As an alternative implementation, the iris region image can be transformed into polar coordinates, and then the rotation angle can be calculated using the phase correlation method.

[0047] Comprehensive level components Vertical component and torsional components Construct a three-dimensional vector The direction of this vector is perpendicular to that of the vector. The modulus of the composite plane of revolution (determined by the right-hand rule) is... , is used to characterize the instantaneous rotational angular velocity vector of the eyeball.

[0048] Perform the above calculations on each frame to obtain the sequence. This refers to the three-dimensional motion vector sequence of the eyeball.

[0049] Generate head rotation axis vector sequence For each set of data in the head inertial measurement data stream Calculate the instantaneous resultant angular velocity scalar .

[0050] Set a static judgment threshold ,like Then calculate the instantaneous rotation axis unit vector of the head at that moment. .like If so, the head is determined to be in a quasi-static state. .

[0051] Calculations are performed at all times to obtain the sequence. That is, the head rotation axis vector sequence.

[0052] S3: Generate a spatiotemporally synchronized fused data stream To achieve eye movement vector sequence With head rotation axis vector sequence To achieve high-precision time synchronization and spatial coordinate unification, the following steps are performed.

[0053] Time alignment: due to Select a higher sampling rate The timestamp of the sequence as the base time series .for Each point in time In the sequence Find two adjacent points in the timestamp and (satisfy ).

[0054] set up and The eye movement vectors corresponding to each time point are respectively and To maintain smooth motion, a cubic spline interpolation algorithm is used to calculate... Eye movement vectors after time alignment Its interpolation function is obtained by fitting multiple adjacent data points, and its accuracy is higher than that of linear interpolation.

[0055] For all Calculations were performed to obtain a time-precisely aligned sequence of eye movement vectors. .

[0056] Spatial coordinate unification: Through a pre-executed hand-eye calibration process, a fixed rotation transformation matrix is ​​obtained from the first sensor (camera) coordinate system to the second sensor (IMU, i.e., head) coordinate system. (one (The orthogonal matrix). This calibration process requires the user to perform a set of known, non-coplanar head translation movements, and then use the least squares method or singular value decomposition algorithm to solve for the optimal rotation matrix to eliminate coordinate system deviations caused by differences in the physical installation position of the sensor. The calibration residual is typically less than [value missing]. .

[0057] Each eye movement vector in E' Transform to head coordinate system: , That is, the eye movement vector unified in the head coordinate system.

[0058] Obtain the fused data stream Ultimately, merge the data streams. It is based on time A sorted list where each element is a triple: ,in From , This is the corresponding transformed eye movement vector. for The corresponding head rotation axis vector.

[0059] S4: Calculate and extract dynamic spatial correlation features Within a complete characteristic motion pattern analysis cycle (e.g., a standardized sequence of head posture changes lasting several seconds), a quantifiable correlation exists between specific head spatial movements and specific eye movement responses. The spatial angle between the eye movement vector and the head rotation axis vector... and its dynamic evolution characteristics These constitute the core observation variables for inferring the current motion pattern.

[0060] Calculate the spatial angle sequence For fused data streams Each data point in the data is only valid if its corresponding data point is .... Length of the module Greater than the minimum effective value (For example )and Subsequent calculations are only performed when the vector is not zero, in order to eliminate the influence of noise and static periods.

[0061] For data points that meet the conditions, calculate the vector. and The spatial angle between The calculation formula is: ; in, The unit is degrees, and its value range is [0, 180].

[0062] Calculate all for one complete characteristic motion cycle The spatial angle sequence is obtained. .

[0063] Extract feature parameters: In sequence Find the maximum value in the middle, denoted as .

[0064] Record The corresponding specific time is denoted as .

[0065] Extracting trend parameters: Let the start time of this characteristic motion cycle be... The end time is .

[0066] In the time interval Inside, to All The data points are linearly fitted using the least squares method to obtain the slope of the fitted line, denoted as . .

[0067] In the time interval Inside, to All The data points are linearly fitted using the least squares method to obtain the slope of the fitted line, denoted as . .

[0068] As an alternative implementation, the average rate of change can also be obtained by calculating the ratio of the difference in function values ​​at the corresponding endpoints of the fitted line to the time interval.

[0069] Thus far, the extracted dynamic spatial correlation features have been obtained. That is, a set of specific numerical values: .

[0070] S5: Determine motion pattern identifiers through correlation analysis models Before starting the analysis or guiding process, initialize the system state probability distribution. If there is no prior information, it can be set to a uniform distribution. If the side to be analyzed is known, the corresponding initial probability can be appropriately increased.

[0071] 5.1 Model Framework Define the set of discrete states To perform state analysis on a typical circular spatial motion constraint model (e.g., a three-dimensional pipe model approximating a quarter-circle arc), its idealized centerline is uniformly divided into 5 continuous segments from one end (denoted as end A) to the other end (denoted as end B).

[0072] The state set is defined as follows: ,in This represents the segment closest to end A. This represents the segment closest to end B, and each state corresponds to a possible position segment of the virtual moving body in this spatial constraint model.

[0073] Establish the basic state transition probability matrix : It is The matrix, whose elements are set according to the geometric topology of the spatial constraint model and the motion law of the virtual moving body under the action of simulated physical forces (such as viscosity and gravity). For example, a head motion pattern is defined as follows: (Corresponding to rotation about a specific spatial axis). In In this mode, based on the geometric orientation of the model and the direction of the external force field, the virtual moving body moves from... Duan Xiang The probability of moving to segment B is relatively high, the probability of staying in the original segment is next, and the probability of moving in the opposite direction is extremely low.

[0074] Therefore, it can be set , , For each predefined typical head movement pattern Create a corresponding one for each matrix.

[0075] As a way to obtain the basic state transition probability matrix With expected value of features This implementation method allows for the establishment of a computer simulation environment that includes the geometry of the three-dimensional spatial constraint model and the dynamics model of the virtual moving body's mass. In this simulation, the initial position of the virtual moving body is set to a certain state. Simulate standard motion patterns The head angular velocity sequence was used to simulate the motion trajectory of a virtual moving body through computational fluid dynamics, and a large number of simulations were conducted to statistically analyze its state transitions. Through action Later transferred to The frequency, after normalization, can be used as... The basis for estimating probability values.

[0076] Meanwhile, the virtual sensor data generated by the simulation is processed through the S1-S4 steps of this method to obtain the expected features corresponding to that state. .

[0077] 5.2 Dynamic Application of Correction Factors Based on Motion Dynamics To more realistically simulate the inertial delay effect of a virtual moving body in a viscous medium, a dynamic correction related to the head rotation angular acceleration is introduced into the state transition probability.

[0078] Calculate the modulus of head rotation angular acceleration in real time ,in This is the angular velocity vector from the second sensor. Set the angular acceleration threshold. and attenuation coefficient .

[0079] when At that time, the state transition probability matrix currently in use ,in This is the pattern determined based on the current head movement.

[0080] when At that time, All transition probability elements in the array that represent "moving forward one position segment along the current head rotation direction" (e.g.) ), multiplied by a coefficient Attenuate the decay. Simultaneously, the decrease in probability value of each decayed element is averaged and added to the "remain in its original state" probability element in its row (e.g., ...). The matrix obtained after this correction is used as the actual state transition probability matrix P' at the current moment.

[0081] 5.3 Model Execution Process set up Indicates in At any given moment, the virtual moving body is in a state. to The probability distribution.

[0082] 1. Prediction (calculating prior distribution): In At any given moment, based on the current head movement pattern and real-time calculated angular acceleration Obtain matrix P' as described in Section 5.2. Calculate the prior probability distribution. (Matrix multiplication).

[0083] 2. Calculate the observation likelihood. For each state Preset a set of ideal feature expectations obtained through simulation or experiment. For the actual features extracted at the current moment. Calculate its relationship with each Weighted Euclidean distance: ,in Let be the weighting coefficients of each feature component, then the observation likelihood is... The likelihood vector is obtained. .

[0084] 3. Bayesian update (compile posterior distribution): Combine the prior distribution with the observation likelihood and normalize it to obtain the posterior state probability distribution G[t] at the current time.

[0085] For each state Its posterior probability The calculation formula is: ; in yes Corresponding state The prior probability. .

[0086] 4. Output motion pattern identifier: Find the state with the highest probability value in the posterior distribution G[t]. .Will As the "sports mode indicator" for the current moment. Output.

[0087] Head movement patterns The determination logic: the head movement pattern The determination is based on the current and recent head rotation axis vector. The dominant direction in the head coordinate system is matched with the predefined feature directions of each pattern. For example, if If the direction of the feature continuously approaches the preset "Mode A" in the spatial model, it is determined to be... Modes. Various modes The correspondence with a standardized sequence of spatial motions is predefined.

[0088] S6: Generate and output a sequence of three-dimensional space navigation commands. Mapping target attitude: The system pre-stores a standard motion trajectory library. This library defines identifiers for different initial motion patterns. Starting from a predetermined sequence of 3D head target poses to achieve specific guidance or analysis objectives, this mapping relationship exists in the system as a database or configuration file.

[0089] For a given input identifier The output of querying this library is an ordered list, where each element is a target head pose description. This description uses unit quaternions. This is represented, for example, for input identifiers. The output standard trajectory may contain four target poses. Each posture Using a unit quaternion This indicates that the design of these trajectories is based on the principles of spatial kinematics and aims to guide virtual moving bodies along a constrained model in a predetermined direction.

[0090] Real-time motion command calculation: The system acquires the current head posture quaternion in real time through the second sensor. Let the target pose to be guided to be... Its corresponding quaternion is Calculation from arrive Relative rotation quaternions .Will Converting to axis-angle representation yields the rotation axis direction (a three-dimensional unit vector). and rotation angle (Scalar, unit: degree).

[0091] Output guidance information: Draw a 3D arrow graphic on the screen of the output unit, whose direction is... Overlap, length or color can be coded The size. Simultaneously, a voice prompt is generated: "Please turn your head in the direction of the arrow, at an angle of approximately..." "Degree". Guiding information can also be presented in tactile form (such as vibration).

[0092] Closed-loop adjustment of navigation commands: During the boot process, the system outputs and executes a navigation instruction in the foreground thread, while simultaneously using a background thread (based on the sampling frequency of the second sensor). (e.g., 200Hz) S4 and S5 are executed continuously in parallel to achieve motion mode identification. Real-time tracking and updates.

[0093] The system internally maintains a value based on the initial identifier. The expected trajectory of motion. For example, from Initially, the expected trajectory is as follows: If the system detects that the actual update is identified as... The next sign is expected to be... If so, then a path deviation is determined to have occurred.

[0094] At this point, the system immediately interrupts the current boot sequence and uses the latest identifier. Starting from this point, the standard motion trajectory library is queried again to generate a new set of target posture sequences (e.g., the adjustment path after entering the B-end region of the model), and the subsequent navigation commands are dynamically adjusted accordingly to achieve closed-loop control based on real-time feedback.

[0095] Task completion assessment: In the later stages of the boot process, the system initiates the task completion evaluation logic.

[0096] 1. Define the task completion status set ,in This represents the B-end region of the model where the virtual moving object enters. For a set of states An additionally defined virtual state is used to characterize the condition that the motion signal has returned to rest and the system has a very high degree of certainty.

[0097] virtual state probability The system directly sets the parameters based on the disappearance of characteristic motion signals: if the characteristic eye-tracking parameters extracted in S4 (such as...) ) remains below the resting threshold (e.g. ),but ;otherwise .

[0098] 2. The posterior probability distribution G[t] of the continuously monitored system state belongs to The probability and ,Right now .

[0099] 3. Set probability threshold and duration Second.

[0100] 4. When The value in continuous All within the duration greater than If the system determines that the characteristic motion signal has disappeared, then the system determines that the target phase of the current guided task has been achieved, and gives a "task phase completed" prompt signal through the interface and voice of the output unit.

[0101] Signal interference suppression implementation: To prevent interference from physiological vestibular-ocular reflexes caused by users actively and rapidly turning their heads, the system incorporates anti-interference logic. This involves real-time monitoring of the head rotation angular velocity scalar. Set angular velocity threshold (about ).when At this point, the system determines that the eye movements during this time period are primarily caused by the physiological vestibular reflex, rather than the characteristic movement pattern to be analyzed. The system automatically merges the data stream for this time period. Marked as "invalid region", feature extraction in S4 and model update in S5 are paused until head movement returns to normal. This mechanism improves the system's robustness and analytical accuracy in real-world usage scenarios.

[0102] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-sensor spatiotemporal fusion method for oculomotor attitude correlation analysis, characterized in that, Includes the following steps: S1, through hardware synchronous trigger signal, synchronously acquires eye image sequence and head inertial measurement data sequence of target object; S2, process the eye image sequence to generate a three-dimensional eye movement vector sequence based on the head coordinate system; process the head inertial measurement data sequence to generate a head rotation axis vector sequence; S3, using the timestamp of the head inertial measurement data sequence as a reference, perform time interpolation and alignment on the eyeball three-dimensional motion vector sequence, and unify the eyeball three-dimensional motion vector to the head coordinate system through a pre-calibrated coordinate transformation matrix to form a spatiotemporally synchronized fused data stream; S4, calculate the dynamic spatial angle between the eye movement vector and the head rotation axis vector in the fused data stream, and extract the feature parameter set of the dynamic spatial angle changing over time; S5, input the feature parameter set and the current head movement pattern into a preset correlation analysis model. The model is constructed based on preset kinematic constraints to obtain and output the corresponding movement pattern identifier. S6. Based on the motion mode identifier, a preset three-dimensional head target posture sequence is mapped, and based on the difference between the real-time head posture and the target posture, a three-dimensional spatial navigation command is generated and output.

2. The multi-sensor spatiotemporal fusion oculomotor attitude correlation analysis method according to claim 1, characterized in that, The step S2, which generates a three-dimensional eye movement vector sequence, specifically includes: Extract the pupil center displacement from the eye image sequence and calculate the horizontal and vertical rotation components of the eyeball; The torsional rotation component of the eyeball around the visual axis is calculated by iris texture feature matching or polar coordinate transformation phase correlation method. By combining the horizontal, vertical, and torsional rotation components, a three-dimensional motion vector representing the instantaneous rotational angular velocity of the eyeball is constructed.

3. The multi-sensor spatiotemporal fusion oculomotor attitude correlation analysis method according to claim 1, characterized in that, The feature parameter set extracted in step S4 specifically includes: The maximum value of the dynamic spatial angle within one characteristic motion cycle; The time point at which the maximum value is reached; And the average rate of change before and after reaching the maximum value.

4. The multi-sensor spatiotemporal fusion oculomotor attitude correlation analysis method according to claim 1, characterized in that, The correlation analysis model in step S5 is a probabilistic model based on discrete state space and Bayesian update, and its operation process includes: Based on the current head movement pattern and movement state, query the predefined state transition probability matrix to predict the state probability distribution of the previous moment; Based on the currently extracted set of feature parameters, calculate the observation likelihood of each discrete state; Based on the predicted probability distribution and the observed likelihood, the posterior probability distribution of the current state is obtained through Bayesian update, and the state with the highest posterior probability is output as the motion mode identifier.

5. The multi-sensor spatiotemporal fusion oculomotor attitude correlation analysis method according to claim 4, characterized in that, The construction of the state transition probability matrix incorporates a correction factor related to the head rotation angular acceleration: When the absolute value of the head rotation angular acceleration exceeds the first preset threshold, the state transition probability in the same direction as the head rotation is reduced, and the reduced probability value is allocated to the transition probability of maintaining the original state.

6. The multi-sensor spatiotemporal fusion oculomotor attitude correlation analysis method according to claim 1, characterized in that, It also includes signal anti-interference steps: Real-time monitoring of head rotation angular velocity; When the head rotation angular velocity exceeds the second preset threshold, it is determined that the current eye movement is dominated by the physiological vestibular reflex, the fusion data stream of the corresponding time period is marked as invalid, and the extraction of the feature parameter set and the updating of the correlation analysis model are paused.

7. The multi-sensor spatiotemporal fusion oculomotor attitude correlation analysis method according to claim 1, characterized in that, The step S6, which generates three-dimensional space navigation instructions, specifically includes: The motion pattern identifier is matched with a pre-stored standard motion trajectory library to obtain an ordered three-dimensional head target posture sequence; Based on the real-time acquired head pose data, the rotation axis direction and rotation angle between the current pose and the next target pose are calculated. The rotation axis direction and rotation angle are converted into visual, auditory, or tactile guidance commands for output.

8. The multi-sensor spatiotemporal fusion oculomotor attitude correlation analysis method according to claim 1 or 7, characterized in that, It also includes the closed-loop adjustment steps for navigation commands: During the execution of navigation instructions, steps S4 to S5 are repeated to update the motion mode identifier in real time; The updated motion pattern identifier is compared with the expected motion trajectory. If the deviation exceeds the tolerance, the subsequent target attitude sequence and navigation instructions are regenerated based on the latest motion pattern identifier.

9. The multi-sensor spatiotemporal fusion oculomotor attitude correlation analysis method according to claim 4, characterized in that, It also includes a task completion assessment step: During navigation, the sum of the posterior probabilities of states belonging to a preset subset of task completion states output by the correlation analysis model is continuously monitored; When the probability sum continuously exceeds a third preset threshold for a third preset duration, and the feature parameter set is simultaneously detected to be below a resting threshold, a task phase completion prompt signal is generated.

10. A multi-sensor spatiotemporal fusion oculomotor attitude correlation analysis system for implementing the method of any one of claims 1-9, characterized in that, include: The data acquisition module is used to synchronously acquire eye image sequences and head inertial measurement data sequences via hardware synchronous trigger signals; The data processing and fusion module is connected to the data acquisition module and is used to execute steps S2 and S3 to generate a spatiotemporally synchronized fused data stream; The feature extraction and analysis module, connected to the data processing and fusion module, is used to execute step S4 and extract the dynamic spatial angle feature parameter set; The correlation analysis module is connected to the feature extraction and analysis module, and has a built-in correlation analysis model. It is used to execute step S5 and output motion pattern identifiers. The navigation instruction generation and output module is connected to the correlation analysis module and is used to execute step S6 to generate and output three-dimensional spatial navigation instructions.

Images