Optical system calibration method and system and electronic equipment

Abstract

The invention belongs to the technical field of photoelectric control, and particularly discloses an optical system calibration method and system and electronic equipment, and the method comprises the steps: obtaining an initial eye posture of a user in a static watching state, and building a light spot initial posture matrix; measuring the angular velocity and the angular acceleration of the head of the user in real time, and calculating a current head posture estimated value of lag compensation based on the total delay factor of the sensor; performing high-order angular motion prediction based on the lag compensation estimated value of the current head posture to generate a feed-forward compensation control quantity; according to the current eye pose output by the eye movement tracking module in real time, the deviation angle of the current eye sight line is calculated through a sight line mapping kernel function; jointly inputting the feedforward compensation control quantity and the deviation angle into a proportional differential closed-loop controller, and executing dynamic optical axis deflection through an optical actuator; the method has the following advantages: high-precision, low-delay, high-individual-adaptability and light-weight projection light spot alignment control is realized in a dynamic environment, and the stability of an optical system is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of photoelectric control technology, and in particular to an optical system calibration method, system and electronic equipment. Background Art

[0002] Modern wearable micro-projection devices are gradually being used in scenarios such as visual function training and immersive display. Existing devices usually align the projection optical axis with the user's retinal fovea through a one-time static calibration when the user first wears it, and rely on a fixed mechanical structure to maintain the alignment. However, during actual use, the user's head frequently rotates rapidly and shakes slightly, and the direction of the projection optical axis relative to the eyeball will shift instantaneously. At the same time, the eyeball itself has autonomous micro-movements, further amplifying the probability of the light spot leaving the fovea. To reduce the impact of the offset, existing solutions mostly adopt two solutions:

[0003] One approach is to add rigid constraints or a larger image stabilization frame to maintain a high mechanical coupling strength between the headset and the head. This approach increases the overall weight, reduces wearing comfort, and is subject to the micro-deformation of the human head and the loosening of the wearer over time, making it difficult to avoid slight deviations.

[0004] The second approach uses single-path eye movement feedback, detecting that the projected light spot has left the fovea before making corrections. However, due to the lack of feedforward prediction for high-speed head movements, this method can produce significant lag in scenarios involving large or high-acceleration head turns, and users will still experience image drift or blur.

[0005] In summary, when faced with rapid head movements and autonomous eye movements, there is a lack of a closed-loop control mechanism that can simultaneously integrate head feedforward predictions and real-time eye movement feedback. This makes it difficult to continuously and accurately keep the projected light spot stably aligned with the direction of the fovea under full dynamic conditions, resulting in a significant decline in training effects and display experience. Summary of the Invention

[0006] The present invention aims to provide an optical system calibration method, system and device to solve or improve the above-mentioned technical problem that existing wearable micro-projection devices lack high-speed closed-loop calibration that integrates head movement prediction and eye movement feedback, making it difficult for the light spot to continuously align with the fovea during rapid head movement.

[0007] In view of this, a first aspect of the present invention is to provide an optical system calibration method.

[0008] A third aspect of the present invention is to provide a system.

[0009] A third aspect of the present invention is to provide an electronic device.

[0010] The first aspect of the present invention provides an optical system calibration method, comprising: obtaining the initial eye posture of the user in a static gaze state, and establishing an initial posture matrix of the light spot, and jointly serving as a reference for the dynamic control of the optical actuator; measuring the angular velocity and angular acceleration of the user's head in real time, and calculating the current head posture estimation value of lag compensation based on the total delay factor of the sensor; performing high-order angular motion prediction based on the lag compensation estimation value of the current head posture to generate a feedforward compensation control amount based on the lag compensation estimation value of the current head posture; calculating the offset angle of the current eye line of sight through the line of sight mapping kernel function according to the current eye posture output in real time by the eye movement tracking module; inputting the feedforward compensation control amount and the offset angle of the current eye line of sight into a proportional differential closed-loop controller to generate a correction amount for feedback control, and driving the optical actuator to perform dynamic optical axis deflection through the correction amount to ensure in real time that the projected image is correctly aligned with the retinal fovea direction defined by the initial eye posture.

[0011] In any of the above technical solutions, the initial eye posture is determined by calculating the position of the pupil center relative to the cornea reflection when the user is looking at a preset calibration point; the position of the pupil center relative to the cornea reflection is determined by capturing the eyeball image and then using an image feature extraction algorithm to determine the spatial orientation of the user's initial gaze.

[0012] In any of the above technical solutions, the initial posture matrix of the light spot is represented by a homogeneous transformation matrix, and includes a three-dimensional rotation matrix and a position vector of the optical system in the initial state.

[0013] In any of the above technical solutions, the total sensor delay factor includes the sensor signal acquisition delay, filtering processing delay and data transmission delay of the inertial measurement unit.

[0014] In any of the above technical solutions, the high-order angular motion prediction uses the second-order Taylor series expansion of the current head angular velocity and angular acceleration to predict the head posture change trend at future moments.

[0015] In any of the above technical solutions, the gaze mapping kernel function is a radial basis function, and the parameters of the gaze mapping kernel function are obtained when the user initially calibrates eye tracking.

[0016] In any of the above technical solutions, the proportional differential closed-loop controller is a proportional differential controller, and the proportional gain and differential gain are dynamically adjusted according to the lag compensation estimate of the current head posture and the offset angle of the current eye line of sight.

[0017] In any of the above technical solutions, the optical actuator is a MEMS micromirror, and the dynamic optical axis deflection is achieved by adjusting the deflection angle distribution of the MEMS micromirror.

[0018] The second aspect of the present invention provides a system, comprising: a posture measurement module, which is used to measure the angular velocity and angular acceleration of the user's head in real time, and calculate the current head posture estimate of lag compensation based on the total delay factor of the sensor; an eye tracking module, which is used to collect the initial eye posture of the user in a static gaze state, and output the user's current eye posture in real time; a control calculation module, which is connected to the posture measurement module and the eye tracking module respectively, and is used to perform high-order angular motion prediction based on the lag compensation estimate of the current head posture to generate a feedforward compensation control amount based on the lag compensation estimate of the current head posture; the offset angle of the current eye line of sight is obtained by calculating the line of sight mapping kernel function according to the current eye posture output in real time by the eye tracking module; and the feedforward compensation control amount and the offset angle of the current eye line of sight are jointly input into a proportional differential closed-loop controller to generate a feedback control correction amount; an optical actuation module, which is connected to the control calculation module, and is used to perform a dynamic optical axis deflection operation according to the feedback control correction amount to ensure in real time that the projected image is correctly aligned with the retinal fovea direction defined by the initial eye posture.

[0019] A third aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned optical system calibration method when executing the computer program.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] By integrating high-order angular motion feedforward prediction of head posture and real-time eye movement feedback, a high-frequency closed-loop control mechanism is formed, which improves the image stabilization accuracy of the projected light spot to the micro-arc level, ensuring that the light spot always accurately locks on the fovea under dynamic conditions.

[0022] The introduction of a feedforward prediction mechanism enables the system to preemptively offset projection offsets caused by rapid head rotation and jitter, effectively reducing system response lag from traditional tens of milliseconds to sub-milliseconds, eliminating visual drift and blur in scenarios with significant head movement.

[0023] By introducing a personalized gaze mapping kernel function during initial calibration and establishing a unified benchmark for the initial eye pose and spot pose matrix, frequent recalibration is no longer necessary during long-term use. This greatly enhances the system's adaptability to individual user differences and significantly improves the calibration stability of the device over long-term wear.

[0024] Eliminating the need for a bulky mechanical image stabilization frame, the lightweight MEMS micromirror actuator and small IMU sensor achieve precise image stabilization, significantly reducing overall weight and power consumption, and significantly improving long-term wearing comfort.

[0025] It further expands the applicability of the wearable micro-projection system in diverse scenarios such as low vision training, precise visual function detection, outdoor AR navigation, and immersive interactive games, effectively enhancing the stability and quality of visual training effects and visual experience.

[0026] Additional aspects and advantages of embodiments according to the present invention will become apparent in the following description or may be learned through practice of embodiments according to the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0028] Figure 1 is a flow chart of the method of the present invention;

[0029] Figure 2 is a basic calibration and control flow chart of the present invention;

[0030] Figure 3 Schematic diagram of sensor delay compensation in the delay compensation and prediction process of the present invention;

[0031] Figure 4 Schematic diagram of high-order angular motion prediction in the delay compensation and prediction process of the present invention;

[0032] Figure 5 Schematic diagram of feedforward mapping in the delay compensation and prediction process of the present invention;

[0033] Figure 6 This is a closed-loop control dynamic adjustment flow chart of the present invention;

[0034] Figure 7 This is a flow chart of multimodal data fusion of the present invention;

[0035] Figure 8 is a flowchart of the fault recovery and recalibration of the present invention;

[0036] Figure 9 It is a system logic block diagram of the present invention;

[0037] Figure 10 Schematic diagram of data transmission between modules in the system of the present invention;

[0038] Figure 11 The figure is a schematic structural diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0039] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0040] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0041] See also Figures 1-11 , an optical system calibration method, system and device according to some embodiments of the present invention are described below.

[0042] The embodiment of the first aspect of the present invention provides a method for calibrating an optical system. In some embodiments of the present invention, such as Figures 1-8 As shown, the method includes the following steps:

[0043] S101, obtaining the initial eye posture of the user in a static gaze state, and establishing an initial light spot posture matrix, which serve together as a reference for dynamic control of the optical actuator.

[0044] Specifically, during system startup, initial alignment of the light spot is performed. The controller places optical components, such as mirrors or optical prisms, in reference positions and acquires initial head and eye poses as a reference. Calibration is performed to measure the position of the initial light spot relative to the user's line of sight or the device coordinate system, and an initial state transformation matrix is ​​established to represent the reference alignment state.

[0045] As can be seen above, during the initialization alignment process, the currently measured head posture is set as the zero bias reference, and the light spot is adjusted to the target initial position (e.g., the center of the field of view). At the same time, a data structure is established to store calibration parameters and status, including the initial eye posture, the initial light spot position, and the internal bias of the device. The inertial measurement unit (IMU) and eye tracker are reset and calibrated so that their readings output zero error in the initial state. Ensure that all subsequent steps start with a common initial coordinate reference, and that the light spot and the user's line of sight are accurately aligned at the beginning.

[0046] Specifically, the initial posture matrix of the light spot is as follows:

[0047]

[0048] Where R0 is the three-dimensional rotation matrix of the optical axis of the headset relative to the world coordinates at the moment of startup; p0 is the three-dimensional position vector of the light source reference point in the world coordinates; and T0 is the complete initial pose mapping in homogeneous coordinate form.

[0049] Here, during the system startup phase, the light spot and the line of sight are precisely aligned, and all measuring devices are reset to zero bias. The controller thus establishes a unified coordinate reference. Each subsequent head posture update and eye movement offset calculation is directly based on this reference for differential calculation, which can completely eliminate accumulated drift, ensure accurate mapping of feedforward prediction and feedback correction, and continuously maintain the stable landing of the projected light spot on the fovea, thereby significantly improving the overall accuracy and reliability of visual training and display interaction.

[0050] S102 , measuring the angular velocity and angular acceleration of the user's head in real time, and calculating a lag-compensated current head posture estimation value based on a total sensor delay factor.

[0051] Specifically, the user's head movement is continuously monitored to obtain real-time posture change information. A common method is to read data from the head's IMU sensor, such as gyroscope angular velocity and acceleration, and combine it with possible optical position tracking to infer the head's instantaneous posture. Due to the time delay and noise in the sensor output, time synchronization and filtering are required.

[0052] As shown above, a data structure stores the most recent head pose and angular velocity for recursive calculations. A high-speed loop is used to read IMU data, and digital filtering, such as low-pass filtering or Kalman filtering, is applied to the raw data to reduce noise. The head pose estimate is updated during each control cycle. Interactively, the IMU transmits data to the processor via interrupts or a high-speed bus, ensuring low latency and high accuracy in head motion detection. The head pose estimate output from this step is used for subsequent feedforward compensation calculations.

[0053] Specifically, the lag-compensated attitude prediction is as follows:

[0054]

[0055] Where q h is the head posture quaternion; τ s is the total delay between sensor measurement and processing; ω h is the head angular velocity vector [rad s -1 ]; Ω(·) is the 4×4 antisymmetric matrix generation operator that embeds the angular velocity into the quaternion exponential mapping; is quaternion multiplication; is the current head posture estimate after lag compensation; t is the target time processed by the current system, that is, the time point at which the head posture is expected to be known.

[0056] Here, the head posture estimation is obtained in real time and delayed after compensation, so that the system always knows the real spatial direction of the user's head in the current control cycle; with the help of high-speed sampling, filtering denoising and time synchronization, high-precision, low-jitter posture vectors can be provided within milliseconds, providing reliable input for the subsequent feedforward compensation algorithm, thereby offsetting the impact of rapid head rotation on the spot position in advance, avoiding control lag and image drift, and ensuring the response speed and stability of the stabilization closed loop.

[0057] S103, performing high-order angular motion prediction based on the lag compensation estimate of the current head posture to generate a feedforward compensation control amount based on the lag compensation estimate of the current head posture; according to the current eye posture output in real time by the eye tracking module, the offset angle of the current eye line of sight is calculated by the line of sight mapping kernel function.

[0058] Specifically, based on detected head movement, the system performs feedforward compensation calculations to proactively adjust the light spot position in the event of rapid head movement. Feedforward compensation uses the rate of change of head posture to predict the head posture in the near future and calculates the required pre-adjustment of the light spot accordingly. Specifically, a head rotation angle is predicted at a given time based on the current estimated head angular velocity and system control delay. The head posture change is then converted into angular compensation commands for the optical actuator via an adjustable mapping tensor.

[0059] As can be seen from the above, nonlinear compensation strategies can be incorporated into the calculation process. Specifically, when the rotation angle is large, a nonlinear function is used to compress the compensation amount to avoid overshoot. The feedforward module can also adjust the gain based on the motion dynamics, providing weighted compensation for motion in different frequency bands. The results of the above feedforward calculations are stored in the control instruction buffer, pending superposition with feedback corrections before being sent to the actuator. In hardware, this process is primarily performed within the processor and does not directly generate action, but it provides predictive adjustments for subsequent steps to reduce overall system lag.

[0060] Specifically, the feedback compensation control amount is calculated using the following formula:

[0061] High-order angular motion prediction:

[0062]

[0063] Where K ff is a programmable feedforward gain scalar or a 3×3 diagonal matrix; α h is the head angular acceleration vector, dω h / dt numerical derivative; τ c is the control pipeline delay; Δθ ff To predict the cumulative head rotation angle within the time window for pre-deflection of the optical actuator.

[0064] Angular motion is mapped to the drive space:

[0065] u ff (t) = M map Δθ ff (t)

[0066] Where M map is a 3×N mapping matrix that projects the three-axis angle compensation to N actuation degrees of freedom; u ff It is the actuator feedforward driving vector, including the MEMS mirror XY deflection angle.

[0067] Specifically, the eye tracking module obtains the user's current eye position to accurately assess the alignment of the light spot relative to the line of sight. A typical approach uses an infrared eye-tracking camera to capture the position of the pupil and corneal reflection, and then uses an image processing algorithm to calculate the eye rotation angle or line of sight direction vector. Due to the nonlinear relationship between eye image features and line of sight angle, nonlinear kernel functions are often used within the algorithm for mapping correction. Specifically, the radial basis kernel function can be used to estimate the eye rotation angle based on the offset of the pupil center relative to the eye center.

[0068] As can be seen above, the acquired eye pose is typically represented as a rotation matrix or quaternion relative to the head coordinate system. The controller reads this data structure and determines the difference between the current eye gaze direction and the initial reference. In terms of hardware interaction, the eye tracking camera captures images at a high frame rate and performs real-time processing on the onboard DSP (digital signal processor) or FPGA (field programmable gate array), transmitting the results to the main controller via a high-speed interface. In each loop, the system updates the eye pose and compares it with the initial eye pose, providing basic data for error calculation.

[0069] Specifically, the eye pose is obtained through the following formula:

[0070] Gaze mapping kernel function:

[0071]

[0072] Where Δx and Δy are the pixel displacements of the pupil center relative to the corneal reflection point; r iris is the pixel scale of the iris radius; κ ij are the polynomial kernel coefficients, obtained offline through user calibration; Δφ, Δψ are the horizontal and vertical sight angles [rad]; P, Q are the polynomial orders of the kernel function.

[0073] Current gaze quaternion:

[0074]

[0075] Where, E x , E yis the pure imaginary quaternion of the X and Y axis units in three-dimensional space; q e is the eyeball posture quaternion.

[0076] Here, by performing high-order angular motion prediction on the lag-compensated head posture, the system calculates the required pre-deflection of the light spot before any control delay occurs. Simultaneously, the current eye gaze offset angle is acquired and mapped in real time, enabling the controller to simultaneously grasp two key pieces of information: "head movement trend" and "eye movement error" during each cycle. The former provides fast, proactive motion compensation, while the latter provides precise line of sight alignment correction. Superimposed as a unified control command, these two elements simultaneously offset the effects of high-speed head rotation and micro-eye movements on light spot positioning within milliseconds, significantly reducing image stabilization lag and overshoot, ensuring the projected light spot remains locked to the fovea, and improving the stability and accuracy of visual training and immersive displays.

[0077] S104, the feedforward compensation control amount and the offset angle of the current eye line of sight are input into the proportional differential closed-loop controller to generate a correction amount for feedback control, and the correction amount is used to drive the optical actuator to perform dynamic optical axis deflection to ensure in real time that the projected image is correctly aligned with the direction of the retinal fovea defined by the initial eye posture.

[0078] Specifically, the feedback error of the spot alignment is calculated based on the latest head and eye information. This error reflects the deviation of the current spot position from the ideal position and is used to drive feedback correction control. First, the pose residual is calculated using eye pose data. This is the rotational deviation of the user's eye line of sight relative to the initial reference, represented by a unit pose residual rotation quaternion. The residual quaternion represents the rotation required to rotate from the initial line of sight to the current line of sight, assuming that the initial reference position corresponds to the ideal spot alignment. The residual quaternion is converted to an equivalent rotation vector; specifically, by taking the logarithmic mapping of the quaternion to obtain a 3D small-angle vector, the magnitude and direction of the eye line of sight deviation are obtained. Next, the pose error is combined with any linear color deviation (such as the deviation of the visual axis translation caused by head translation) to form a comprehensive error vector. A weighted dynamic error aggregation method is used to assign weights to errors from different sources to account for both instantaneous errors and accumulated deviations.

[0079] Specifically, the feedback error is calculated using the following formula:

[0080] Calculate the attitude residual quaternion:

[0081]

[0082] Where W φ , W p is the weight matrix of rotation error and translation error; p spot is the three-dimensional unit vector of the current light spot projection point; p foveais the three-dimensional unit vector in the ideal direction of the fovea; δq x , δq y , δq z is the imaginary component of the residual quaternion, corresponding to the small angle of the three axes; e(t) is the comprehensive attitude-position error vector.

[0083] Foveal direction error vector:

[0084] e(t)=W φ 2vec(δq)+W p (p spot -p fovea )

[0085] As can be seen above, the calculated error is stored in the feedback control data structure, including, for example, the angular errors of the three rotational axes and possible translational error components. This error vector is used in the next step of feedback correction control to drive the light spot back to the ideal alignment position. The entire error calculation process is repeated every control cycle, providing real-time deviation information for closed-loop control.

[0086] Specifically, based on the feedback error, feedback correction control is performed to calculate the correction commands that need to be applied to the optical actuators to gradually eliminate the residual error. The controller can employ strategies such as classic PID control or state feedback control to convert the error into actuator adjustments. Feedback control utilizes the posture residuals obtained in the previous step and converts them into the required angle or displacement commands for the specific actuators. To this end, the mapping tensor or its inverse is again used to map the error vector to the control variables of each actuator.

[0087] As can be seen from the above, the feedforward instructions and feedback correction instructions are accumulated in the controller to obtain the total actuator command. The command vector corresponds to the rotation angle and voltage of the motor that drives the lens or reflector. The controller sends the actuator command to the relevant execution hardware, including outputting it to the motor driver through the DAC (digital-to-analog converter) to trigger physical movement to adjust the light spot. In hardware interaction, in order to ensure fast response, the actuator usually adopts a high-speed servo motor or piezoelectric actuator, and sets the limit and acceleration control in the drive loop to prevent overshoot or oscillation. At the same time, the position sensor (such as encoder) integrated on the actuator will feedback the actual movement for the next monitoring and verification. Through feedback correction control, the system can make fine adjustments to the error of each control cycle and gradually approach the ideal alignment state.

[0088] Specifically, the feedback correction control is achieved through the following formula:

[0089]

[0090] Where K P , K Dis the proportional gain and differential gain matrix; Δt c To control the cycle period; u fb Feedback drive vector to the actuator.

[0091] Specifically, after the actuator performs calibration, the position of the light spot is updated. The system monitors the new light spot position in real time to evaluate the alignment effect and provide data support for the next cycle. The monitoring process includes reading the actuator's feedback sensor data, including the motor encoder position used to reflect the current angle of the lens, and using the eye movement sensor to measure the relative position of the user's line of sight and the light spot again. To filter out measurement noise and obtain a smooth error trend, the system applies time recursive filtering to the monitored light spot position.

[0092] As can be seen above, in the monitoring data structure, the system will update the current spot position, recent movement trends, and residual error size. If the error is not completely eliminated after correction, the controller can use the remaining error as the new initial value and continue correction in the next cycle. At the same time, the monitoring module can check whether the error is within the allowable threshold. If the spot alignment error is large (exceeding the threshold), it may trigger an alarm or restart the calibration process. In terms of hardware, monitoring mainly involves quickly reading encoder, sensor, and camera data, and ensuring that the main control process is not blocked through methods such as DMA. The result of this step is an accurate assessment of the current state of the system, providing a basis for the continuous operation of closed-loop control.

[0093] Specifically, the spot position recursive filtering calculation is performed using the following formula:

[0094]

[0095] Where μ is the recursive filter smoothing coefficient; is the estimated unit vector of the filtered spot.

[0096] Here, the feedforward compensation and real-time eye gaze offset are fed together into a proportional-differential closed-loop controller, enabling it to proactively offset head movement trends and finely correct for eye movement errors within each control cycle. The weighted fusion of these corrections directly drives a high-speed optical actuator, and a dual feedback chain consisting of an encoder and an eye movement sensor monitors the results in real time, forming a high-frequency closed-loop "prediction-correction-verification" system. This continuously compresses residual errors within milliseconds, preventing overshoot oscillations and ensuring that the light spot remains locked to the fovea without drifting with head or eye movement. This ensures that the training image and line of sight remain perfectly aligned over time, significantly improving the accuracy of dynamic image stabilization and the stability of visual interaction.

[0097] Furthermore, a cyclic closed-loop control system is established, where the above steps are executed cyclically according to a fixed control period, forming a complete closed-loop control system. In each cycle, head movement is detected, feedforward compensation is calculated, eye position is acquired, feedback error is calculated, feedback correction is applied, and the spot position is monitored and updated, thereby continuously correcting the spot alignment. The loop typically runs at high speed, for example, hundreds to thousands of times per second, to ensure that the spot can be stably aligned even with rapid head movements, and that the system advances according to the iterative relationship.

[0098] As can be seen above, repeating the control calculation process allows the error to gradually converge with increasing iterations. Closed-loop control achieves dynamic image stabilization through the tight integration of feedforward and feedback: the feedforward channel provides fast motion compensation prediction, and the feedback channel provides precise error correction, with the two complementing each other. Thanks to the well-structured control logic and real-time correction, the light spot can be kept stably aligned with the target position throughout operation. All steps together form a closed-loop control chain, from sensor data input to actuator output and then to sensor verification, forming a self-correcting control loop that ensures that the dynamic image stabilization effect is automatically maintained without human intervention. After each cycle ends, the next cycle immediately begins again, starting with head motion detection. This continuous operation ensures that the stability and accuracy of the light spot alignment are constantly controlled.

[0099] Specifically, the controller is operated at a fixed period Δt c cycle:

[0100] t n+1 =t n +Δt c , n=0,1,2,...

[0101] Among them, at each t n Steps S101-S104 are executed sequentially at all times to cyclically update all state variables and command vectors, thereby realizing continuous dynamic image stabilization control coupled with feedforward prediction and feedback correction.

[0102] Here, head movement detection, feedforward prediction, eye movement feedback, error correction and result monitoring are strung together into a self-closed control chain through high-frequency loops, which can continuously iterate error convergence within milliseconds; the feedforward channel first offsets large movements, and the feedback channel then makes fine corrections. The two are seamlessly connected in each cycle, allowing the system to keep the projected light spot firmly locked on the fovea for a long time without human intervention, and no drift or overshoot will occur even under conditions of drastic head movement or changes in line of sight, thereby providing users with a stable and clear visual experience at all times.

[0103] The present invention provides an optical system calibration method. A prediction path pre-offsets the displacement caused by high-speed head movement, and a feedback path refines the residual caused by micro-movement of the eyeball. The two iteratively converge within a millisecond-level closed loop, so that the projected light spot always locks on the fovea under full field of view dynamics, and the steady-state deviation can be controlled to the micro-arc level.

[0104] By using high-order angular motion prediction to compensate for system delays, compensation instructions can still be given in advance under the scenario of maximum head acceleration; combined with PD dynamic gain, the system phase lag is compressed to a single cycle (sub-10ms), and users are almost unaware of image drift or blur.

[0105] During initialization, the gaze mapping kernel is trained based on the user's unique pupil-corneal geometric features, and subsequent algorithms continue to operate on the same coordinate reference. It can automatically compensate for wear micro-displacement and temperature drift, and the alignment accuracy remains stable during long-term use without the need for frequent recalibration.

[0106] The dual improvements in image stabilization accuracy and dynamic response make the device suitable for scenarios with stringent requirements on visual stability, such as low vision rehabilitation, precision visual function testing, outdoor AR navigation, and high-action games, significantly improving training effects and immersive experience.

[0107] Specifically, by implementing the optical system calibration method, the system performance achieved a qualitative leap before and after calibration: before calibration, the spot center error exceeded 2° and the compensation delay was as high as 15ms, resulting in significant image drift and response lag; after calibration, through dynamic delay compensation and high-order angular motion prediction, the spot center error was compressed to within 0.1°, and the compensation delay was reduced to 2ms, allowing the spot to accurately lock onto the fovea in real time. At the same time, the optimized control algorithm stabilizes the drift rate of the MEMS micromirror from ±1.5° to ±0.05° within 10 minutes of continuous operation, significantly improving the long-term stability of the optical system and meeting the stringent requirements for dynamic image stabilization in virtual reality training and augmented reality interaction.

[0108] In any of the above embodiments, the initial eye posture is determined by calculating the position of the pupil center relative to the corneal reflection when the user is looking at a preset calibration point; the position of the pupil center relative to the corneal reflection is determined by capturing the eyeball image and then using an image feature extraction algorithm to determine the spatial orientation of the user's initial gaze.

[0109] In this embodiment, by collecting the relative positions of the pupil center and corneal reflection when the user is looking at the calibration point at startup, and inferring the initial gaze direction through an image feature algorithm, the system can establish an accurate and personalized foveal direction reference for each user; this reference eliminates the initial deviation caused by differences in eyeball structure and wearing and assembly errors, and provides a unified reference for all subsequent head movement prediction, eye movement feedback and spot correction, thereby ensuring that image stabilization control always operates around the user's actual line of sight, significantly improving calibration accuracy and wearing adaptability.

[0110] In any of the above embodiments, the initial posture matrix of the light spot is represented by a homogeneous transformation matrix, and includes a three-dimensional rotation matrix and a position vector of the optical system in the initial state.

[0111] In this embodiment, the initial posture of the light spot is represented by a homogeneous transformation matrix containing a three-dimensional rotation matrix and a position vector, so that the optical system, the head coordinate system and the eye line of sight can be matrix multiplied, superimposed and quickly inversely calculated in the same four-dimensional homogeneous coordinate framework; this unified mathematical expression not only compresses the amount of posture update calculation, but also facilitates the seamless mapping of the head movement and eye movement compensation amounts to the actual spatial position of the light spot through simple matrix left or right multiplication in subsequent control cycles, avoiding frequent coordinate system conversion errors and significantly improving the real-time and numerical stability of dynamic calibration.

[0112] In any of the above embodiments, the total sensor delay factor includes a sensor signal acquisition delay, a filtering processing delay, and a data transmission delay of the inertial measurement unit.

[0113] In this embodiment, the sensor signal acquisition delay, filtering processing delay and data transmission delay are unified into a total sensor delay factor, so that the control algorithm can use a clear and calibrated time amount to perform uniform lag compensation on the IMU data. This not only eliminates the phase error caused by multiple asynchronous delays in attitude estimation, but also simplifies the time parameters of the prediction model, ensuring that the feedforward angular motion prediction is always based on attitude input synchronized with physical reality, thereby significantly reducing control lag and improving the dynamic consistency and numerical accuracy of the stabilization closed loop.

[0114] In any of the above embodiments, the high-order angular motion prediction uses the second-order Taylor series expansion of the current head angular velocity and angular acceleration to predict the head posture change trend at future moments.

[0115] In this embodiment, the second-order Taylor series expansion of angular velocity and angular acceleration is used to perform high-order angular motion prediction of the head posture, which can provide a more accurate estimate of the future turning angle in advance within the control delay window; compared with the first-order linear prediction using only angular velocity, this method completely retains the influence of angular acceleration on the curvature of the rotation curve, so that the feedforward compensation amount fits the actual motion trajectory, significantly reducing the estimation error and light spot overshoot in the scenario of rapid acceleration and deceleration of the head turning, thereby improving the response speed and pre-adjustment accuracy of the image stabilization control.

[0116] In any of the above embodiments, the gaze mapping kernel function is a radial basis function, and the parameters of the gaze mapping kernel function are obtained when the user initially calibrates the eye tracking.

[0117] In this embodiment, a radial basis function is used as the gaze mapping kernel, and its parameters are individually trained during the user's first calibration, so that the nonlinear relationship between each user's pupil-corneal reflection geometric characteristics and the actual gaze angle can be accurately fitted; this adaptive mapping not only compensates for differences in eyeball structure, but also improves the resolution of extremely small angle changes, allowing the system to output high-precision gaze offset angles in real time during operation, providing a reliable feedback basis for closed-loop calibration, thereby continuously ensuring the accuracy and stability of light spot alignment.

[0118] In any of the above embodiments, the proportional differential closed-loop controller is a proportional differential controller, and the proportional gain and the differential gain are dynamically adjusted according to the lag compensation estimate of the current head posture and the offset angle of the current eye line of sight.

[0119] In this embodiment, the proportional P gain and differential D gain are designed to be adjusted in real time based on the current head posture lag compensation estimate and the current eye gaze offset angle. This allows the closed-loop controller to automatically switch to the optimal response under both stable gaze and drastic head turns. When the head movement is small and the offset angle is minimal, the gain is reduced to suppress high-frequency noise and avoid amplifying errors; while when the head acceleration is large or the offset angle increases dramatically, the gain is increased to accelerate error convergence. This dynamic gain strategy balances steady-state accuracy with transient speed, significantly improving the stability margin and tracking performance of the PD closed-loop across the entire motion domain, preventing overshoot oscillation while ensuring that the light spot remains firmly locked to the fovea.

[0120] In any of the above embodiments, the optical actuator is a MEMS micromirror, and the dynamic optical axis deflection is achieved by adjusting the deflection angle distribution of the MEMS micromirror.

[0121] In this embodiment, a MEMS micromirror with extremely low inertia is used as an optical actuator, and dynamic optical axis deflection is achieved by precisely controlling its dual-axis deflection angle, so that the image stabilization system can complete rapid scanning angles at the micro-radian level within a kilohertz bandwidth; compared with traditional motors or liquid crystal devices, MEMS micromirrors are small in size, low in power consumption, and highly linear, and can maintain excellent dynamic response and repeatability in a lightweight head-mounted platform, so that the correction amount generated in each control cycle can be instantly converted into light spot direction adjustment, ensuring that the light spot can still stably lock the fovea without perceptual lag in high-speed head movement scenarios.

[0122] Furthermore, the following adaptations are made to the uncontrollable state of the human body, such as the problem of a large head shake or sudden braking:

[0123] Taylor expansion of the small angle rotation of the posture to the third order term:

[0124]

[0125] Where ω(t) is the angular velocity vector; is the angular acceleration vector; is the angular jump rate; τ c The total delay of the control closed loop, including sampling + calculation + actuation; Δθ 3rd is the cumulative turning angle within the prediction window.

[0126] Estimate the angular rate of change:

[0127] j(t)≈[α(t)-α(t-Δt)] / Δt

[0128] Delay adaptive window, angle jerk prediction accuracy and τ c Establishment interval related:

[0129] If the system calculation + actuation delay measured in this round increases, the jerk term weight can be increased proportionally:

[0130]

[0131] Where κ is an empirical coefficient ranging from 0.3 to 0.5; τ0 is the calibration delay.

[0132] If the speed is low and stable, the jerk weights are automatically shrunk to suppress overfitting and noise amplification.

[0133] As can be seen from the above, in a high-dynamic environment with sudden start and stop of the head, the third-order angular motion feedforward can significantly reduce the spot overshoot and steady-state error; combined with dynamic weighting, filtering and mechanical bandwidth matching, it can maintain foveal lock without the user's perception. Simulation results show that at a peak angular velocity of 400° / s and a peak angular acceleration of 8×10 3 ° / s 2 The peak value of artificial jerk reached 1.2×10 5 ° / s 3 In high-dynamic scenarios, the maximum spot offset is 4.6° without compensation, which can be reduced to 0.71° after using second-order angular motion compensation, and further compressed to 0.22° after introducing third-order acceleration prediction, which is about 70% lower than the peak error of the second-order model.

[0134] The embodiment of the second aspect of the present invention provides a system such as Figure 9 and Figure 10 As shown, the system includes:

[0135] The posture measurement module is used to measure the angular velocity and angular acceleration of the user's head in real time, and calculate the lag-compensated current head posture estimate based on the total sensor delay factor.

[0136] The eye tracking module is used to collect the user's initial eye posture in a static gaze state and output the user's current eye posture in real time.

[0137] The control calculation module is connected to the posture measurement module and the eye tracking module respectively, and is used to perform high-order angular motion prediction based on the lag compensation estimate of the current head posture to generate a feedforward compensation control amount based on the lag compensation estimate of the current head posture; the offset angle of the current eye line of sight is calculated through the line of sight mapping kernel function according to the current eye posture output in real time by the eye tracking module; and the feedforward compensation control amount and the offset angle of the current eye line of sight are input together into the proportional differential closed-loop controller to generate the correction amount of feedback control.

[0138] The optical actuation module is connected to the control calculation module and is used to perform dynamic optical axis deflection operations according to the correction amount of feedback control to ensure in real time that the projected image is correctly aligned with the direction of the retinal fovea defined by the initial eye posture.

[0139] The present invention provides a system in which the posture measurement module and the eye tracking module each provide millisecond-level high-precision data. The control calculation module fuses the two pieces of information, which not only offsets the light spot displacement caused by high-speed head rotation in advance, but also corrects the slight deviation of the eye line in real time. The projected light spot is always locked on the fovea, and visual training and immersive display images no longer produce drift or smear. The high-order angular motion prediction and dynamic gain proportional differential control in the control calculation module form a dual mechanism of advance compensation and fine closed loop, which significantly shortens the system phase lag and maintains steady-state convergence under conditions of rapid and large head movements. The user does not feel calibration overshoot. Based on the personalized line of sight mapping kernel obtained by initialization calibration, the calibration benchmark is completely consistent with the eye structure of each user, eliminating the systematic deviation caused by traditional public mapping and improving the alignment accuracy of long-term wear. At the same time, no bulky mechanical constraints are required, maintaining light weight and comfort. The optical actuation module uses high-speed, low-inertia MEMS micromirrors with low power consumption and small size, making it suitable for integration into head-mounted platforms. It cooperates with position feedback sensors to form an inner loop self-check, and can maintain rotation repeatability and temperature drift stability during long-term operation, providing reliable, high-dynamic range image stabilization core capabilities for applications such as augmented reality and low vision training.

[0140] Embodiments of the third aspect of the present invention provide electronic devices. In some embodiments of the present invention, such as Figure 11 As shown, an electronic device is provided, which includes: electronic devices such as desktop computers, notebooks, handheld computers and cloud servers. The electronic device 3 may include but is not limited to a processor 301 and a memory 302. Those skilled in the art will understand that Figure 11 This is merely an example of the electronic device 3 and does not limit the electronic device 3 . The electronic device 3 may include more or fewer components than shown in the figure, or different components.

[0141] The processor 301 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0142] The memory 302 can be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. The memory 302 can also include both an internal storage unit of the electronic device 3 and an external storage device. The memory 302 is used to store computer programs and other programs and data required by the electronic device.

[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0144] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0145] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0146] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0147] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure, and should all be included in the scope of protection of the present disclosure.

Claims

1. A method for calibrating an optical system, characterized in that: The steps include: Obtain the user's initial eye position in a static gaze state and establish the initial light spot pose matrix, which serve as the reference for the dynamic control of the optical actuator. Measure the angular velocity and angular acceleration of the user's head in real time, and calculate the lag-compensated current head pose estimate based on the total sensor delay factor; Performing high-order angular motion prediction based on the lag compensation estimate of the current head posture to generate a feedforward compensation control amount based on the lag compensation estimate of the current head posture; and calculating the offset angle of the current eye gaze through a gaze mapping kernel function based on the current eye posture output in real time by the eye tracking module; The feedforward compensation control amount and the offset angle of the current eye line of sight are input together into a proportional differential closed-loop controller to generate a correction amount for feedback control, and the correction amount is used to drive the optical actuator to perform dynamic optical axis deflection to ensure in real time that the projected image is correctly aligned with the direction of the retinal fovea defined by the initial eye posture.

2. The optical system calibration method according to claim 1, wherein: The initial eye posture is determined by calculating the position of the pupil center relative to the cornea reflection when the user is looking at a preset calibration point; the position of the pupil center relative to the cornea reflection is determined by capturing an eyeball image and then using an image feature extraction algorithm to determine the spatial orientation of the user's initial gaze.

3. The optical system calibration method according to claim 1, wherein: The light spot initial posture matrix is ​​represented by a homogeneous transformation matrix and includes a three-dimensional rotation matrix and a position vector of the optical system in the initial state.

4. The optical system calibration method according to claim 1, wherein: The total sensor delay factor includes sensor signal acquisition delay, filtering processing delay and data transmission delay of the inertial measurement unit.

5. The optical system calibration method according to claim 1, wherein: The high-order angular motion prediction uses the second-order Taylor series expansion of the current head angular velocity and angular acceleration to predict the head posture change trend at future moments.

6. The optical system calibration method according to claim 1, wherein: The gaze mapping kernel function is a radial basis function, and the parameters of the gaze mapping kernel function are obtained when the user initially calibrates the eye tracking.

7. The optical system calibration method according to claim 1, characterized in that: The proportional differential closed-loop controller is a proportional differential controller, and the proportional gain and the differential gain are dynamically adjusted according to the lag compensation estimation value of the current head posture and the offset angle of the current eye line of sight.

8. The optical system calibration method according to claim 1, wherein: The optical actuator is a MEMS micromirror, and the dynamic optical axis deflection is achieved by adjusting the deflection angle distribution of the MEMS micromirror.

9. A system for implementing the optical system calibration method according to any one of claims 1 to 8, characterized in that: include: The posture measurement module is used to measure the angular velocity and angular acceleration of the user's head in real time and calculate the hysteresis-compensated current head posture estimate based on the total sensor delay factor; The eye tracking module is used to collect the user's initial eye position in a static gaze state and output the user's current eye position in real time; a control calculation module, connected to the posture measurement module and the eye tracking module, respectively, for performing high-order angular motion prediction based on the hysteresis compensation estimate of the current head posture to generate a feedforward compensation control amount based on the hysteresis compensation estimate of the current head posture; calculating the offset angle of the current eye line of sight through a line of sight mapping kernel function according to the current eye posture output in real time by the eye tracking module; and inputting the feedforward compensation control amount and the offset angle of the current eye line of sight into a proportional differential closed-loop controller to generate a correction amount for feedback control; An optical actuation module is connected to the control calculation module and is used to perform a dynamic optical axis deflection operation according to the correction amount of the feedback control to ensure in real time that the projected image is correctly aligned with the retinal fovea defined by the initial eye posture.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of the optical system calibration method according to any one of claims 1 to 8 are implemented.

Images