An adaptive anti-shake handheld gimbal holographic display control system

Abstract

The application relates to the technical field of holographic display control, and discloses a self-adaptive anti-shake handheld gimbal holographic display control system which comprises a space-time calibration module, a viewpoint reconstruction module, a data filtering module and a servo closed loop module. The space-time calibration module is used for aligning the time stamp of communication hardware and establishing a space rigid body kinematics reference coordinate system; the viewpoint reconstruction module is used for performing space inverse transformation to solve the human eye relative space prediction coordinates based on an instantaneous deflection vector; the data filtering module is used for extracting a time delay parameter and calculating a posterior human eye space coordinate through a Kalman filter; and the servo closed loop module outputs a target attitude offset vector to a handheld gimbal controller when the relative space angle exceeds a physical boundary threshold. The application eliminates the data operation time difference and cumulative drift deviation, and realizes self-adaptive anti-shake closed loop control under the premise of maintaining the holographic optical effective view area covering the human eye range.

Description

Technical Field

[0001] This invention relates to the field of holographic display control technology, specifically to an adaptive anti-shake handheld gimbal holographic display control system. Background Technology

[0002] The handheld gimbal holographic display control system is a hardware platform that combines a mechanical anti-shake mechanism with a three-dimensional spatial imaging component. In practical application scenarios, the holographic display panel is mounted on the end of a three-axis motor, and the image acquisition module integrated inside the device is responsible for capturing the physical observation coordinates of the user's eyes in real time.

[0003] To present a three-dimensional image with a sense of depth, the control system needs to acquire the position parameters of the human eye in absolute space and update the view rendering matrix in real time by combining the physical diffraction constant of the holographic grating. When the person is walking or shaking their hand, the handheld gimbal controller reads the internal sensor data to calculate the posture deviation and drives the three-axis motor to output the compensation torque in the opposite direction to maintain the posture of the holographic display panel in absolute space and ensure that the perspective relationship of the image is aligned with the observation viewpoint.

[0004] Current control schemes, when coordinating mechanical compensation actions and visual tracking, lack an underlying clock alignment mechanism between heterogeneous hardware. This leads to misalignment between image exposure acquisition timestamps and motor encoder reading timestamps, and the failure to uniformly construct a spatial rigid body kinematic reference coordinate system results in low spatiotemporal matching accuracy between visual tracking data and mechanical anti-shake actions. Image acquisition equipment experiences data processing delays spanning multiple frames, and the calculation module performs feedforward operations, generating cumulative drift deviations in coordinate data during control cycle loops, compromising the accuracy of the rendering viewpoint during anti-shake operation. Simultaneously, the control system only locks the absolute spatial attitude, lacking closed-loop monitoring of the relative angle between the local normal of the holographic display panel and the line-of-sight vector. When the observer's position deviates from the panel's frontal view and exceeds the physical boundary threshold of the diffraction viewing area, the device cannot output a target attitude offset vector to drive the motor to perform attitude following actions, causing the observer to leave the holographic optical coverage area and lose the 3D image.

[0005] Therefore, the purpose of this invention is to provide an adaptive anti-shake handheld gimbal holographic display control system to overcome the shortcomings of the prior art. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an adaptive anti-shake handheld gimbal holographic display control system. This system solves the problems of low spatiotemporal matching accuracy between visual tracking and mechanical anti-shake actions, inaccurate rendering viewpoints due to cumulative drift deviations in coordinate data, and loss of 3D images due to the lack of closed-loop monitoring, which causes the observer to leave the holographic optical coverage area.

[0007] To achieve the above objectives, the present invention provides the following technical solution: An adaptive image stabilization handheld gimbal holographic display control system includes: The spatiotemporal calibration module is used to align the timestamps of the holographic display panel, handheld gimbal controller and image acquisition device connected in the communication connection, and to establish a spatial rigid body kinematic reference coordinate system in combination with physical parameters; The viewpoint reconstruction module is used to calculate the instantaneous deflection vector in the spatial rigid body kinematic reference coordinate system, and to perform an inverse spatial transformation based on the instantaneous deflection vector to calculate the human eye's relative spatial predicted coordinates. The data filtering module is used to extract time delay parameters based on the timestamp and calculate the posterior human eye spatial coordinates using a discrete Kalman filter. The servo closed-loop module is used to calculate the relative spatial angle based on the posterior human eye spatial coordinates, and output the target attitude offset vector to the handheld gimbal controller when the relative spatial angle is greater than or equal to the set physical boundary threshold of the holographic effective diffraction field of view.

[0008] Preferably, the spatiotemporal calibration module aligns the timestamps of the holographic display panel, the handheld gimbal controller, and the image acquisition device connected in the communication link, including: The spatiotemporal calibration module activates a hardware interrupt triggering mechanism based on the vertical synchronization signal emitted by the underlying display controller of the holographic display panel connected in communication. The spatiotemporal calibration module sends a first hardware interrupt command to the handheld gimbal controller, and uses the first global frame sequence number marker, which is assigned by the handheld gimbal controller to read and latch the absolute angle value of the three-axis motor encoder at the current command time, as the timestamp for reading the three-axis motor encoder data. The spatiotemporal calibration module sends a second hardware interrupt command to the image acquisition device, uses the second global frame sequence number of the image acquisition device as the timestamp for the image exposure acquisition, and binds the first global frame sequence number to the second global frame sequence number.

[0009] The spatiotemporal calibration module sends a second hardware interrupt command to the image acquisition device, using the second global frame sequence number marker of the image acquisition device as the timestamp for the image exposure acquisition, and numerically binding the first global frame sequence number marker with the second global frame sequence number marker.

[0010] The spatiotemporal calibration module establishes a time domain reference network with the underlying display controller as the main clock, detects changes in the level of the vertical synchronization signal to activate external event triggering logic, and the image acquisition device responds to the second hardware interrupt instruction to start the exposure control sequence. The spatiotemporal calibration module establishes a timing mapping relationship by binding the first global frame sequence number marker and the second global frame sequence number marker in the system memory stack, and completes the timestamp alignment operation between heterogeneous hardware.

[0011] Preferably, the spatiotemporal calibration module establishes a spatial rigid body kinematic reference coordinate system in conjunction with physical parameters, including: The spatiotemporal calibration module extracts the spatial translation offset vectors between the pitch axis, roll axis and yaw axis of the handheld gimbal to form the physical parameters of the gimbal's mechanical rotation center, and obtains the installation offset, three-dimensional rotation deviation angle and optical center normal parameters of the holographic display panel connected to the communication. The spatiotemporal calibration module adds the physical parameters of the gimbal's mechanical rotation center to the installation offset to generate a static translation homogeneous matrix, converts the three-dimensional rotation deviation angle into a static rotation homogeneous matrix, and multiplies the static translation homogeneous matrix with the static rotation homogeneous matrix to calculate the transfer matrix. The spatiotemporal calibration module uses the transfer matrix and the optical center normal parameters of the holographic display panel to construct a static rigid body transformation equation and perform multiplication to establish the spatial rigid body kinematic reference coordinate system.

[0012] The spatiotemporal calibration module establishes a multi-level physical reference coordinate system within the handheld gimbal. It extracts the spatial translation offset vector by reading pre-programmed mechanical structure drawing parameters from the non-volatile memory within the handheld gimbal controller. The calculation logic for the static rigid body transformation equation involves matrix-wise addition of the spatial translation offset vector of the three-axis motor and the installation offset to generate a static translation homogeneous matrix. This is combined with the static rotation homogeneous matrix derived from the three-dimensional rotation deviation angle. The transfer matrix is ​​then calculated by multiplying the static translation homogeneous matrix with the static rotation homogeneous matrix. This transfer matrix remains constant in system memory during system operation.

[0013] Preferably, the viewpoint reconstruction module calculates the instantaneous deflection vector in the spatial rigid body kinematic reference coordinate system, including: The viewpoint reconstruction module retrieves the absolute angle value of the three-axis motor encoder latched under the current timestamp and performs differential calculation with the absolute angle value latched under the previous frame timestamp to generate transient rotation angle integral incremental data. The viewpoint reconstruction module substitutes the transient rotation angle integral increment data into the three-dimensional space rotation equation to generate a global rotation transformation matrix. The viewpoint reconstruction module uses the global rotation transformation matrix, the transfer matrix, and the initial global normal vector to perform matrix multiplication to complete the forward kinematics calculation, thereby solving for the instantaneous deflection vector of the holographic display panel in the global absolute space.

[0014] Within the microsecond-level control cycle of the three-axis motor performing anti-shake compensation, the viewpoint reconstruction module polls the encoder hardware registers inside the three-axis motor at high frequency via a high-speed serial peripheral interface bus. The specific execution logic of the forward kinematics calculation involves sequentially multiplying the global rotation transformation matrix, the transfer matrix, and the initial global normal vector calibrated at the factory of the holographic display panel. The calculated instantaneous deflection vector represents the actual pointing vector of the physical optical center of the holographic display panel in space at the instant the anti-shake mechanical action occurs.

[0015] Preferably, the viewpoint reconstruction module performs an inverse spatial transformation based on the instantaneous deflection vector to calculate the relative spatial predicted coordinates of the human eye, including: The viewpoint reconstruction module reads the actual human eye observation coordinates acquired by the image acquisition device from the system memory buffer. These coordinates are marked with the latest timestamp and have been aligned with the spatial rigid body kinematic reference coordinate system. The viewpoint reconstruction module performs a matrix transpose operation on the global rotation transformation matrix to generate an inverse relative rotation matrix; The viewpoint reconstruction module performs matrix multiplication between the inverse relative rotation matrix and the actual observation coordinates of the human eye, performs inverse spatial transformation of the physical coordinate system to calculate the predicted relative spatial coordinates of the human eye, and transmits the predicted relative spatial coordinates of the human eye to the graphics processing unit.

[0016] The viewpoint reconstruction module sets the user's head in a relatively stationary state within the current microsecond-level anti-shake control cycle in the mathematical model. Since the global rotation transformation matrix is ​​an orthogonal matrix, the viewpoint reconstruction module completes the inverse operation through matrix transpose to generate the inverse relative rotation matrix. The spatial inverse transformation calculation logic is to multiply the inverse relative rotation matrix with the actual observation coordinates of the human eye to calculate the three-dimensional relative spatial position of the human eye relative to the holographic display panel that has undergone physical deflection during the delay blind zone time.

[0017] Preferably, the graphics processing unit performs offset superposition calculation on the holographic view matrix of the current rendering cycle using the human eye's relative spatial predicted coordinates, including: The graphics processing unit enables a late-binding direct-write channel in the underlying display driver, and asynchronously retrieves the human eye's relative spatial prediction coordinates from the independent video memory address space through the late-binding direct-write channel; The graphics processing unit extracts the numerical components of the human eye's relative spatial predicted coordinates on the three spatial rectangular coordinate axes to construct a three-dimensional translation transformation matrix. The graphics processing unit performs multiplication and superposition calculations on the three-dimensional translation transformation matrix and the reference holographic view matrix to generate an updated holographic view matrix for the current rendering cycle.

[0018] The late-binding direct-write channel bypasses the application layer graphics interface module by modifying the instruction buffer submission mechanism of the graphics driver layer, directly connecting the shader operation unit inside the graphics processing unit to an independent video memory address space. At the critical moment when the 3D vertex coordinate calculation task is submitted for rasterization, the graphics processing unit reads the reference holographic view matrix and sequentially writes the numerical components of the human eye's relative spatial predicted coordinates into a fourth-order homogeneous matrix with diagonal elements of value one to construct a 3D translation transformation matrix. The system uses matrix multiplication and superposition calculations to physically align the perspective relationship of the output image with the physically displaced optical devices.

[0019] Preferably, the data filtering module extracts time delay parameters based on the timestamp, including: The data filtering module asynchronously reads the actual human eye observation coordinates from the system data communication queue. These coordinates are time-stamped and generated into a data structure by the image acquisition device and output by the human eye. The data filtering module reads the first global frame sequence number marker, subtracts the second global frame sequence number marker carried in the actual human eye observation coordinate data structure from the first global frame sequence number marker, and calculates the delay frame span caused by hardware processing. The data filtering module multiplies the delay frame span with the physical refresh cycle time constant to calculate the time delay parameter in the absolute time dimension, and writes the time delay parameter as a timing label into the memory address append bit of the actual observation coordinates of the human eye.

[0020] The image acquisition device uses an internally built-in facial keypoint classifier to extract the two-dimensional pixel coordinates of the pupil center. Combined with a reference pupillary distance constant and camera physical parameters, this is inversely mapped to three-dimensional depth coordinates using a perspective N-point pose estimation algorithm. The image acquisition device then uses a transfer matrix to uniformly transform the three-dimensional depth coordinates in the local camera coordinate system to a spatial rigid body kinematic reference coordinate system, generating the actual observation coordinates for the human eye. These coordinates, along with a global frame number marker, are then memory-aligned and packaged.

[0021] Preferably, the data filtering module calculates the posterior human eye spatial coordinates using a discrete Kalman filter, including: The data filtering module substitutes the time delay parameter into the kinematic differential equation of the discrete Kalman filter to construct the state transition matrix; The data filtering module uses the actual observation coordinates of the human eye with the time series label as the system observation input to construct the observation vector of the current time node, retrieves the relative spatial prediction coordinates of the human eye at a historical moment that is strictly aligned with the time series label, inputs them into the state transition matrix to perform the prior mathematical recursion of the state equation, and generates a prior human eye spatial state estimation vector. The data filtering module calculates the Kalman gain matrix by combining the preset measurement noise covariance matrix and the error covariance matrix of the prior human eye spatial state estimation vector. The Kalman gain matrix is ​​then used to filter and fuse the coordinate residual values ​​between the observation vector and the prior human eye spatial state estimation vector to calculate the posterior human eye spatial state estimation vector. The updated three-dimensional position components are extracted from the posterior human eye spatial state estimation vector, and the three-dimensional position components are converted into the posterior human eye spatial coordinates.

[0022] The data filtering module instantiates a discrete Kalman filter in system memory, substituting the time delay parameter as a physical variable of the time step into the kinematic differential equation. The state transition matrix represents the uniform motion model of the user's eye within the absolute time span. The computational logic for Kalman filter state update involves extracting the coordinate residual between the observed vector and the prior estimated vector of the user's eye space, multiplying it by the Kalman gain matrix, and then adding it to the prior estimated vector of the user's eye space. The data filtering module replaces the stationary reference point in the computation buffer with the posterior coordinates of the user's eye space, clearing the current cumulative drift bias value of the system.

[0023] Preferably, the servo closed-loop module calculates the relative spatial angle based on the posterior human eye spatial coordinates, including: The servo closed-loop module extracts the absolute three-dimensional coordinates of the optical center of the holographic display panel in the spatial rigid body kinematic reference coordinate system and the local normal of the holographic display panel, and subtracts the absolute three-dimensional coordinates of the optical center of the holographic display panel from the posterior human eye spatial coordinates to generate a line-of-sight vector. The servo closed-loop module performs a three-dimensional vector dot product operation between the gaze vector and the local normal of the holographic display panel, divides the result of the three-dimensional vector dot product operation by the product of the magnitude of the gaze vector and the magnitude of the local normal of the holographic display panel, and calculates the relative spatial angle using the inverse cosine function.

[0024] The absolute three-dimensional coordinates of the optical center of the holographic display panel are analytically determined based on the origin offset component of the transfer matrix during the system initialization phase. The gaze vector represents the straight line direction of the user's actual gaze when viewing the holographic display panel in three-dimensional physical space. The calculation logic for the relative spatial angle is to solve for the dot product of the gaze vector and the local normal of the holographic display panel, divided by the product of the magnitude of the gaze vector and the magnitude of the local normal of the holographic display panel, and then use the inverse cosine function to calculate the physical angle by which the human eye deviates from the direction of the normal to the holographic display panel.

[0025] Preferably, when the relative spatial angle is greater than or equal to a set physical boundary threshold of the holographic effective diffraction field of view, the servo closed-loop module outputs a target attitude offset vector to the handheld gimbal controller, including: When the value of the relative spatial angle is greater than or equal to the physical boundary threshold of the holographic effective diffraction field of view, the servo closed-loop module extracts the difference between the relative spatial angle and the physical boundary threshold of the holographic effective diffraction field of view and defines it as the out-of-bounds angle error. The servo closed-loop module projects the gaze vector onto the local three-dimensional rectangular coordinate system of the holographic display panel, calculates the physical projection angle of the gaze vector on the horizontal projection plane and the vertical projection plane, calculates the ratio of the physical projection angle on the horizontal projection plane and the physical projection angle on the vertical projection plane to the relative spatial angle, and extracts the spatial geometric weight coefficients in the horizontal direction and the spatial geometric weight coefficients in the vertical direction respectively. The servo closed-loop module multiplies the out-of-bounds angle error by the spatial geometric weight coefficient in the horizontal direction and the spatial geometric weight coefficient in the vertical direction, respectively, to calculate the target attitude offset vector composed of the pitch axis compensation angle and the yaw axis compensation angle. The target attitude offset vector, the preset smoothing attenuation coefficient, and the current absolute space target setting value are then vector-superimposed to generate an updated absolute space target setting value, which is then written to the handheld gimbal controller.

[0026] The servo closed-loop module reads the physical boundary threshold of the holographic effective diffraction field of view to determine whether the human eye has left the effective imaging space. The calculation logic for overwriting and updating the absolute spatial target setting value is to add the smooth attenuation coefficient and the target attitude offset vector to the current absolute spatial target setting value. The smooth attenuation coefficient limits the maximum angular acceleration of a single anti-shake action. The handheld gimbal controller outputs a pulse width modulation signal based on the control deviation to drive the three-axis motor to perform attitude following action, causing the relative spatial angle to shrink and fall back to within the physical boundary threshold range of the holographic effective diffraction field of view.

[0027] This invention provides an adaptive image stabilization handheld gimbal holographic display control system. It has the following beneficial effects: 1. This invention, by setting up a spatiotemporal calibration module, utilizes the vertical synchronization signal emitted by the bottom-level display controller of the holographic display panel to activate the hardware interrupt triggering mechanism, aligning the timestamp of the handheld gimbal controller reading the three-axis motor encoder data with the timestamp of the image acquisition device performing image exposure acquisition. This can eliminate the time difference in data processing between heterogeneous hardware, unify the spatial rigid body kinematic reference coordinate system, and improve the spatiotemporal matching accuracy of visual tracking and mechanical anti-shake actions in the handheld gimbal holographic display control system.

[0028] 2. This invention extracts time delay parameters by setting up a data filtering module to calculate the number of frames of delay caused by hardware processing. It substitutes the actual observation coordinates of the human eye with time sequence labels into the Kalman filter to calculate the posterior human eye spatial coordinates. The calculated posterior human eye spatial coordinates are used to replace the stationary reference point in the calculation buffer, which can clear the cumulative drift deviation value generated by the system in the control cycle, and ensure the accuracy of the rendering viewpoint of the handheld gimbal holographic display control system in high-frequency anti-shake operation.

[0029] 3. This invention calculates the relative spatial angle between the line-of-sight vector and the local normal of the holographic display panel by setting a servo closed-loop module. When the relative spatial angle exceeds the physical boundary threshold of the effective holographic diffraction field of view, the target attitude offset vector is calculated and written into the handheld gimbal controller. By overwriting the absolute spatial target setting value, the three-axis motor is driven to perform attitude following action, actively forcing the relative spatial angle to shrink and fall back. Under the premise of maintaining the effective holographic optical field of view covering the range of human eyes, the adaptive anti-shake closed-loop control of the handheld gimbal is realized. Attached Figure Description

[0030] Figure 1 This is a system architecture diagram of the present invention; Figure 2 This is a flowchart of the method of the present invention; Figure 3 This is a flowchart of the hardware interrupt triggering mechanism and timestamp alignment method of the present invention; Figure 4 This is a flowchart of the servo closed-loop anti-shake control and attitude tracking method of the present invention; Figure 5 This is a comparison chart of the cumulative drift deviation before and after the Kalman filter correction of this invention; Figure 6 This is a comparison diagram of the relative spatial angle change and holographic view area boundary crossing under the image stabilization compensation control of the present invention. Detailed Implementation

[0031] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] See attached document Figure 1 This invention provides an adaptive anti-shake handheld gimbal holographic display control system, which includes a spatiotemporal calibration module, a viewpoint reconstruction module, a data filtering module, and a servo closed-loop module.

[0033] The spatiotemporal calibration module is connected to the handheld gimbal controller, the holographic display panel, and the image acquisition device. The viewpoint reconstruction module is connected to the spatiotemporal calibration module. The data filtering module is connected to the image acquisition device. The servo closed-loop module receives the data output by the data filtering module and is connected to the handheld gimbal controller.

[0034] See attached document Figure 2 This invention relates to an adaptive image stabilization handheld gimbal holographic display control system, including an adaptive image stabilization handheld gimbal holographic display control method, specifically comprising the following steps: The spatiotemporal calibration module uses the vertical synchronization signal emitted by the bottom-level display controller of the holographic display panel to construct a hardware interrupt triggering mechanism, aligning the timestamp of the handheld gimbal controller reading the three-axis motor encoder data with the timestamp of the image acquisition device performing image exposure acquisition; The spatiotemporal calibration module extracts the physical parameters of the gimbal's mechanical rotation center, combines them with the optical center normal parameters of the holographic display panel to calculate the transfer matrix, and establishes a spatial rigid body kinematics reference coordinate system. During the control cycle of the three-axis motor performing anti-shake compensation, the viewpoint reconstruction module extracts the transient rotation angle integral increment data output by the encoder, and performs forward calculation in combination with the transfer matrix to obtain the instantaneous deflection vector of the holographic display panel in the global absolute space. The viewpoint reconstruction module performs inverse spatial transformation based on the instantaneous deflection vector, calculates the relative spatial predicted coordinates of the human eye under the current physical posture, and transmits the relative spatial predicted coordinates of the human eye to the graphics processing unit. The graphics processing unit retrieves the relative spatial predicted coordinates of the human eye during the vertex shader processing stage of the graphics rendering pipeline and performs offset superposition calculation on the holographic view matrix of the current rendering cycle. The data filtering module obtains the actual observation coordinates of the human eye output by the image acquisition device, extracts the time delay parameter based on the timestamp, inputs the actual observation coordinates of the human eye with time sequence label into the Kalman filter to calculate the posterior human eye spatial coordinates, and corrects the cumulative drift deviation in the output data of the viewpoint reconstruction module. The servo closed-loop module generates a gaze vector based on the posterior human eye spatial coordinates and calculates the relative spatial angle between the gaze vector and the local normal of the holographic display panel. When the relative spatial angle exceeds the physical boundary threshold of the holographic effective diffraction field of view, the servo closed-loop module calculates the target attitude offset vector and sends it to the instruction input bus of the handheld gimbal controller, overwriting the absolute spatial target setting value of the control loop inside the handheld gimbal controller, and driving the three-axis motor to perform attitude following action.

[0035] Please see the appendix Figure 3 The process of constructing a hardware interrupt triggering mechanism and performing timestamp alignment using the vertical synchronization signal emitted by the underlying display controller of the holographic display panel is as follows: The spatiotemporal calibration module establishes a time domain reference network with the underlying display controller of the holographic display panel as the main clock. The spatiotemporal calibration module establishes a physical connection with the underlying display controller through physical general-purpose input / output pins or tear-resistant hardware synchronization lines. The internal hardware of the handheld gimbal controller includes a microcontroller or field-programmable gate array, and the image acquisition device includes a front-facing color camera or an infrared camera. The spatiotemporal calibration module unifies the internal clock sources of the underlying display controller, the handheld gimbal controller, and the image acquisition device to the time domain reference network.

[0036] The underlying display controller outputs a vertical synchronization signal according to the preset physical refresh cycle of the holographic display panel. The spatiotemporal calibration module detects the level changes of the vertical synchronization signal in real time. When the spatiotemporal calibration module detects the rising or falling edge of the vertical synchronization signal, it activates the hardware interrupt triggering mechanism. Based on the external event triggering logic, the spatiotemporal calibration module sends a first hardware interrupt command to the handheld gimbal controller and a second hardware interrupt command to the image acquisition device.

[0037] The handheld gimbal controller responds to the first hardware interrupt instruction, suspends non-core tasks, reads the angle register value of the three-axis motor encoder at the current instruction time, and writes the angle register value to the specified memory address for latching. The handheld gimbal controller assigns an auto-incrementing first global frame sequence number marker to the latched absolute angle data and uses the first global frame sequence number marker as the timestamp for reading the three-axis motor encoder data.

[0038] The image acquisition device responds to the second hardware interrupt instruction, driving the internal image sensor to start the exposure control sequence for image acquisition. The image acquisition device assigns a corresponding second global frame sequence number to the image data frame output by the exposure, and uses the second global frame sequence number as the timestamp for performing image exposure acquisition. The spatiotemporal calibration module binds the first global frame sequence number and the second global frame sequence number in the system memory stack to establish a timing mapping relationship and complete the timestamp alignment operation between heterogeneous hardware. For the low-level exposure driving logic and the conventional control operation of register read and write timing of the internal image sensor of the image acquisition device, those skilled in the art can write a driver program based on the datasheet of the specific sensor model. The low-level exposure driving logic and the conventional control operation of register read and write timing are well-known technologies in the field and will not be described in detail here.

[0039] The spatiotemporal calibration module extracts the physical parameters of the gimbal's mechanical rotation center, combines them with the optical center normal parameters of the holographic display panel to calculate the transfer matrix, and establishes a spatial rigid body kinematics reference coordinate system. The specific process is as follows: The spatiotemporal calibration module establishes a multi-level physical reference coordinate system inside the handheld gimbal. The multi-level physical reference coordinate system includes the gimbal base coordinate system with the bottom of the handheld gimbal grip as the origin, the three-axis motor end effector coordinate system located at the end of the three-axis motor, and the holographic panel local coordinate system with the physical geometric center of the holographic display panel as the origin. The spatiotemporal calibration module extracts the spatial translation offset vectors between the pitch axis, roll axis, and yaw axis by reading the mechanical structure drawing parameters pre-burned in the non-volatile memory inside the handheld gimbal controller. The spatial translation offset vectors constitute the physical parameters of the gimbal mechanical rotation center. The physical parameters of the gimbal mechanical rotation center are characterized as the three-dimensional coordinate offset matrix of each rotation axis relative to the origin of the gimbal base coordinate system.

[0040] The spatiotemporal calibration module reads the factory calibration file of the holographic display panel to obtain the installation offset and three-dimensional rotational deviation angle when the holographic display panel is assembled to the end effector of the three-axis motor. The value range of the installation offset is determined by the machining and assembly tolerances, and is usually within the millimeter range. The three-dimensional rotational deviation angle is the assembly error angle, and its value range is usually between -2° and +2°. Based on the installation offset and the three-dimensional rotational deviation angle, the spatiotemporal calibration module determines the spatial coordinate position of the optical center of the holographic display panel in the coordinate system of the end effector of the three-axis motor. The spatiotemporal calibration module extracts the unit vector of the holographic display panel perpendicular to the screen's light-emitting plane and defines the unit vector as the normal parameter of the optical center of the holographic display panel.

[0041] The spatiotemporal calibration module constructs a static rigid body transformation equation using the physical parameters of the gimbal's mechanical rotation center and the optical center normal parameters of the holographic display panel. It then adds the spatial translation offset vector of the three-axis motor to the installation offset matrix to generate a static translation homogeneous matrix. The module converts the three-dimensional rotation deviation angle into a static rotation homogeneous matrix. Finally, it multiplies the static translation homogeneous matrix with the static rotation homogeneous matrix to calculate the transfer matrix. The formula for calculating the transfer matrix is ​​as follows: ; in, Represents the transfer matrix; Represents a static translation homogeneous matrix; The physical parameters of the gimbal's mechanical rotation center are determined by reading the non-volatile memory inside the handheld gimbal controller; The installation offset is determined by the physical dimensions and assembly tolerances of the holographic display panel assembly process. Represents a static rotation homogeneous matrix; , , These represent the angular offset values ​​of the three-dimensional rotational deviation angles on the three spatial orthogonal axes of roll, pitch, and yaw, respectively, obtained based on factory optical calibration tests.

[0042] The spatiotemporal calibration module multiplies the optical center normal parameters of the holographic display panel with the transfer matrix to establish an initial spatial rigid body kinematic reference coordinate system for the holographic display panel in the coordinate system of the gimbal base. The initial spatial rigid body kinematic reference coordinate system serves as a static reference for calculating the instantaneous deflection vector within a microsecond-level control cycle. The transfer matrix remains constant in the system memory during system operation. For homogeneous matrix multiplication operations between multi-level physical reference coordinate systems and standard mathematical derivations of rigid body translation and rotation, those skilled in the art can use the Diane-Hartenberg parameter method for calculation.

[0043] The viewpoint reconstruction module extracts the transient rotation angle integral increment data output by the encoder, combines it with the transfer matrix to perform forward solving, and obtains the instantaneous deflection vector of the holographic display panel in the global absolute space. The specific process is as follows: Within the microsecond-level control cycle of the three-axis motor performing anti-shake compensation, the viewpoint reconstruction module polls the encoder hardware register inside the three-axis motor at high frequency through the high-speed serial peripheral interface bus inside the handheld gimbal controller. The viewpoint reconstruction module retrieves the absolute angle value latched at the current timestamp and performs differential calculation between the absolute angle value latched at the current timestamp and the absolute angle value latched at the adjacent previous frame timestamp to generate transient rotation angle integral increment data.

[0044] The transient rotation angle integral increment data is specifically represented as the angle change in a three-dimensional coordinate system, including pitch axis angle increment, roll axis angle increment, and yaw axis angle increment. The value range of pitch axis angle increment, roll axis angle increment, and yaw axis angle increment is determined by the maximum single image stabilization compensation amplitude of the handheld gimbal controller, and the value range is usually between -5° and +5°.

[0045] The viewpoint reconstruction module substitutes the pitch axis angle increment, roll axis angle increment, and yaw axis angle increment into the three-dimensional space rotation equation to generate a global rotation transformation matrix. The viewpoint reconstruction module then calls the transfer matrix calculated and established by the spatiotemporal calibration module from the system memory.

[0046] The viewpoint reconstruction module extracts the initial global normal vector of the holographic display panel in a static state. The module then performs matrix multiplication using the global rotation transformation matrix, transfer matrix, and initial global normal vector to complete the forward kinematics calculation. Through this calculation, the viewpoint reconstruction module determines the instantaneous deflection vector of the holographic display panel in global absolute space. This instantaneous deflection vector represents the actual pointing vector of the physical optical center of the holographic display panel in space at the instant the anti-shake mechanical action occurs. The forward kinematics calculation formula for the instantaneous deflection vector is as follows: ; in, Indicates the instantaneous deflection vector; Represents the global rotation transformation matrix; This represents the pitch axis rotation increment, determined by extracting encoder values ​​and performing differential calculations. This represents the roll axis angle increment, determined by extracting encoder values ​​and performing differential calculations. The increment of the yaw axis angle is determined by extracting encoder values ​​and performing differential calculations. This represents the transfer matrix, which is calculated by the spatiotemporal calibration module during the establishment of the spatial rigid body kinematics reference coordinate system. The initial global normal vector is determined by the factory optical parameter calibration of the holographic display panel. For the calculation rules of the three-dimensional spatial rotation equation based on the trigonometric function expansion of Euler angles, those skilled in the art can directly execute it by calling the standard underlying mathematical function library.

[0047] The viewpoint reconstruction module performs an inverse spatial transformation based on the instantaneous deflection vector, calculates the relative spatial predicted coordinates of the human eye under the current physical posture, and transmits the relative spatial predicted coordinates of the human eye to the graphics processing unit. The specific process is as follows: After acquiring the instantaneous deflection vector and global rotation transformation matrix of the holographic display panel in global absolute space, the viewpoint reconstruction module reads the last valid actual human eye observation coordinates from the system memory buffer. Valid actual human eye observation coordinates are specifically represented as the human eye's three-dimensional spatial coordinate data in the system memory buffer, bearing the latest timestamp and aligned with the spatial rigid body kinematic reference coordinate system.

[0048] The actual observation coordinates of the human eye are obtained by the image acquisition device within the historical effective sampling period and are marked as valid in the time domain reference network constructed by the spatiotemporal calibration module. Since the mechanical image stabilization compensation period is in the microsecond range and the exposure and image processing period of the image acquisition device is in the millisecond range, the viewpoint reconstruction module sets the user's head in a relatively static state in the global absolute space within the current microsecond-level image stabilization control period in the mathematical model.

[0049] The viewpoint reconstruction module performs a matrix inversion operation on the global rotation transformation matrix to generate an inverse relative rotation matrix. Since the global rotation transformation matrix is ​​physically orthogonal, the matrix inversion operation performed by the viewpoint reconstruction module is specifically implemented by transposing the global rotation transformation matrix. The inverse relative rotation matrix represents the reverse displacement mapping relationship between the spatial stationary reference point and the local coordinate system of the holographic display panel when the holographic display panel undergoes physical anti-shake rotation.

[0050] The viewpoint reconstruction module performs matrix multiplication between the inverse relative rotation matrix and the actual observation coordinates of the human eye, executes the inverse spatial transformation of the physical coordinate system, and calculates the predicted spatial coordinates of the human eye relative to the physical coordinate system. These predicted spatial coordinates represent the three-dimensional relative spatial position of the user's eye relative to the holographic display panel, which is currently undergoing a slight physical deflection, during the delay period before the image acquisition device outputs a new frame of visual tracking data. The formula for calculating the inverse spatial transformation is as follows: ; in, Represents the relative spatial predicted coordinates of the human eye; This represents the inverse relative rotation matrix, which is obtained by inverting or transposing the global rotation transformation matrix. This represents the actual coordinates observed by the human eye, output by the image acquisition device and read from the system memory buffer.

[0051] After calculating the relative spatial predicted coordinates of the human eye, the viewpoint reconstruction module directly writes the relative spatial predicted coordinates of the human eye into the independent video memory address space where the graphics processing unit is located through the high-speed peripheral component interconnect data bus inside the system. The graphics processing unit receives the relative spatial predicted coordinates of the human eye and stores them in a register, which serves as the core physical calculation parameter for the underlying graphics rendering pipeline to perform viewpoint updates. For the underlying mathematical operation logic of matrix inversion and the bus-based data transmission protocol between processor memory, those skilled in the art can directly implement it using standard linear algebra calculation library and conventional bus communication instructions.

[0052] The specific process by which the graphics processing unit retrieves the relative spatial predicted coordinates of the human eye during the vertex shader processing stage of the graphics rendering pipeline and performs offset superposition calculation on the holographic view matrix of the current rendering cycle is as follows: The graphics processing unit enables a late-binding direct-write channel in the underlying display driver. The specific implementation of the late-binding direct-write channel is that the graphics processing unit modifies the instruction buffer submission mechanism of the graphics driver layer, opens access permissions to the memory-mapped input / output registers, and the late-binding direct-write channel bypasses the application layer graphics interface module, directly connecting the shader operation unit inside the graphics processing unit to the independent video memory address space that stores the relative spatial prediction coordinates of the human eye.

[0053] At the critical moment when the 3D vertex coordinate calculation task of the current rendering cycle is about to be submitted to rasterization processing, specifically one machine clock cycle before the vertex shader is about to execute the model-view projection matrix multiplication instruction, the graphics processing unit asynchronously retrieves the latest written human eye relative spatial prediction coordinates from the independent video memory address space through the late-bound direct write channel.

[0054] The graphics processing unit reads the reference holographic view matrix allocated by the rendering engine during the current frame initialization phase. The reference holographic view matrix represents the default rendering space perspective relationship of the system in the absence of mechanical anti-shake offset. The graphics processing unit extracts the numerical components of the human eye's relative spatial predicted coordinates on the three spatial rectangular coordinate axes and constructs a three-dimensional translation transformation matrix based on the numerical components on the three spatial rectangular coordinate axes.

[0055] The 3D translation transformation matrix is ​​specifically represented as a fourth-order homogeneous matrix with all diagonal elements equal to 1. The graphics processing unit (GPU) sequentially writes the three numerical components of the human eye's predicted spatial coordinates into the first three rows of the fourth column of the fourth-order homogeneous matrix. The GPU then performs a multiplication and superposition calculation on the 3D translation transformation matrix and the reference holographic view matrix to generate the updated holographic view matrix for the current rendering cycle. The specific formula for calculating the offset superposition of the holographic view matrix is ​​as follows: ; in, Represents the holographic view matrix for the current rendering cycle; This represents a three-dimensional translation transformation matrix constructed based on the human eye's predicted coordinates relative to space. Represents the relative spatial predicted coordinates of the human eye; The reference holographic view matrix is ​​set by the system's internal graphics rendering engine.

[0056] The graphics processing unit continues to execute subsequent fragment shader calculation programs based on the holographic view matrix of the current rendering cycle, completing the rasterization projection of the three-dimensional holographic scene space model onto the two-dimensional physical pixel matrix of the holographic display panel. Since the holographic view matrix of the current rendering cycle directly superimposes anti-shake transient parameters at the bottom layer, the holographic image sub-pixel interlacing perspective relationship output by the graphics processing unit can be physically perspective aligned with the optical devices that undergo mechanical displacement on the holographic display panel, eliminating the holographic rendering viewpoint misalignment phenomenon. For the data flow logic of the vertex shader's underlying rendering pipeline and the matrix operation process of the three-dimensional model rasterization projection, those skilled in the art can write shader scripts based on the underlying shading language of the standard graphics application programming interface.

[0057] The specific process by which the data filtering module obtains the actual observation coordinates of the human eye output by the image acquisition device and extracts the time delay parameter is as follows: The image acquisition device integrates an image signal processor that receives raw image frames generated by the image sensor. The processor performs grayscale conversion and edge detection pixel operations on these frames, runs an internally configured facial keypoint classifier to identify facial regions, and extracts the two-dimensional pixel coordinates of the pupil centers of the user's eyes. The device's internal non-volatile memory stores standard human facial reference pupil distance physical constants. These physical constants are extracted and, combined with the factory-calibrated camera intrinsic parameter matrix and lens distortion physical parameters, a perspective N-point pose estimation algorithm is used to inversely map the two-dimensional pixel coordinates of the pupil centers to three-dimensional depth coordinates in the local camera coordinate system. The device then uses the transfer matrix calculated by the spatiotemporal calibration module to uniformly transform the three-dimensional depth coordinates from the local camera coordinate system to a spatial rigid body kinematic reference coordinate system, generating the actual observation coordinates of the human eye.

[0058] After completing the coordinate calculation, the image acquisition device reads the second global frame sequence number marker written to the hardware register by the spatiotemporal calibration module during the original exposure stage of the image. The image acquisition device aligns the actual observation coordinates of the human eye with the second global frame sequence number marker in memory, packages them to generate a data structure and pushes it into the system data communication queue. The data filtering module asynchronously reads the actual observation coordinates of the human eye with the time sequence marker from the system data communication queue.

[0059] The data filtering module accesses the system's underlying clock thread and reads the latest first global frame sequence number marker of the rendering pipeline currently being executed by the graphics processing unit. The data filtering module subtracts the second global frame sequence number marker carried in the human eye's actual observation coordinate data structure from the first global frame sequence number marker to calculate the delay frame span caused by hardware processing. The data filtering module extracts the physical refresh cycle time constant preset by the underlying display controller of the holographic display panel. The data filtering module multiplies the delay frame span with the physical refresh cycle time constant to calculate the time delay parameter in the absolute time dimension. The data filtering module writes the time delay parameter as a timing tag into the memory address append bits of the human eye's actual observation coordinates, thus constructing the human eye's actual observation coordinates with timing tags. For the edge detection algorithm at the bottom layer of the image signal processor and the coordinate reprojection mapping mathematical calculation process based on the perspective N-point pose estimation algorithm, those skilled in the art can directly implement it by calling standard open-source computer vision computing libraries.

[0060] The data filtering module inputs the actual observed coordinates of the human eye with time-series labels into the Kalman filter to calculate the posterior spatial coordinates of the human eye. The specific process of correcting the cumulative drift bias in the output data of the viewpoint reconstruction module is as follows: The data filtering module instantiates a discrete Kalman filter in system memory. The data filtering module configures the state vector of the discrete Kalman filter to include three-dimensional position components and three-dimensional velocity components. The data filtering module parses the time delay parameter from the memory address append bits of the actual observation coordinates of the human eye with time sequence labels. The data filtering module directly substitutes the time delay parameter as a physical variable of time step into the kinematic differential equation of the discrete Kalman filter to construct the underlying state transition matrix.

[0061] The state transition matrix is ​​specifically represented as a sixth-order square matrix. In the internal algebraic operations of the state transition matrix, the three-dimensional position component at the next moment is equal to the three-dimensional position component at the current moment plus the product of the three-dimensional velocity component at the current moment and the time delay parameter. The state transition matrix represents the uniform physical motion model of the user's eye within the absolute time span corresponding to the time delay parameter.

[0062] The data filtering module uses the actual observation coordinates of the human eye with time-series labels as the system observation input to construct the observation vector of the current time node. The data filtering module accesses the calculation records stored in the system history buffer queue by the viewpoint reconstruction module, retrieves the human eye relative spatial prediction coordinates of the historical moment that are strictly aligned with the time-series labels, and inputs the retrieved human eye relative spatial prediction coordinates into the state transition matrix to perform the prior mathematical recursion of the state equation, generating the prior human eye spatial state estimation vector.

[0063] The data filtering module reads the measurement noise covariance matrix preset within the system. The initial values ​​of the diagonal elements of the measurement noise covariance matrix are determined by the physical noise constant of the image sensor pixels calibrated at the factory of the image acquisition device. The data filtering module combines the error covariance matrix of the prior human eye spatial state estimation vector to calculate and obtain the Kalman gain matrix.

[0064] The data filtering module extracts the coordinate residual values ​​between the observed vector and the prior human eye spatial state estimation vector. It then uses the Kalman gain matrix to filter and fuse these residual values, calculating the posterior human eye spatial state estimation vector. The data filtering module extracts the updated 3D position components from this posterior human eye spatial state estimation vector and converts them into posterior human eye spatial coordinates. The formula for calculating the Kalman filter state update is as follows: ; in, This represents the posterior human eye spatial state estimation vector; This represents the prior spatial state estimation vector of the human eye, which is generated by recursively using the state equation after retrieving the relative spatial prediction coordinates of the human eye at historical moments from the data filtering module. The Kalman gain matrix is ​​calculated based on the error covariance matrix and the measurement noise covariance matrix. The observation vector is constructed from the actual human eye observation coordinates with time-series labels. The observation model matrix is ​​used to map the state vector to the data observation space. It is specifically composed of an identity matrix and a zero matrix, which are used to extract the three-dimensional position components in the state vector and compare them with the observation vector.

[0065] Because the viewpoint reconstruction module lacks macroscopic human eye movement data during the high-frequency feedforward calculation stage, the system will generate a cumulative drift deviation in the register after multiple control cycles, deviating from the actual physical position. The data filtering module uses the calculated posterior human eye space coordinates and replaces the stationary reference point in the viewpoint reconstruction module's calculation buffer with the posterior human eye space coordinates through a system memory overwrite instruction. The data filtering module uses the replacement operation to clear the current cumulative drift deviation value of the system, ensuring that the human eye relative space prediction coordinates generated by the viewpoint reconstruction module in subsequent control cycles have the true physical reference coordinates. For the specific initialization value setting of the internal error covariance matrix of the discrete Kalman filter and the iterative logic of the update cycle, those skilled in the art can derive and write corresponding digital signal processing programs based on the standard linear optimal estimation control theory.

[0066] Please see the appendix Figure 4 The servo closed-loop module generates a gaze vector based on the posterior spatial coordinates of the human eye, and calculates the specific process of the relative spatial angle between the gaze vector and the local normal of the holographic display panel as follows: The servo closed-loop module receives the posterior human eye spatial coordinates output by the data filtering module. The servo closed-loop module extracts the absolute three-dimensional coordinates of the optical center of the holographic display panel in the spatial rigid body kinematic reference coordinate system. The absolute three-dimensional coordinates are analytically determined by the spatiotemporal calibration module based on the origin offset component of the transfer matrix during the system initialization phase. The servo closed-loop module subtracts the absolute three-dimensional coordinates of the optical center of the holographic display panel from the posterior human eye spatial coordinates to generate a gaze vector. The gaze vector represents the straight line direction of the user's actual gaze when viewing the holographic display panel in three-dimensional physical space.

[0067] The servo closed-loop module extracts the local normal of the holographic display panel from the system memory. The local normal of the holographic display panel is a unit vector perpendicular to the light-emitting plane of the holographic display panel. The local normal of the holographic display panel is determined by the spatiotemporal calibration module and stored in the system memory when the system initializes the spatial rigid body kinematic reference coordinate system.

[0068] The servo closed-loop module performs a three-dimensional vector dot product operation between the gaze vector and the local normal of the holographic display panel. The module then divides the result of this three-dimensional vector dot product by the product of the magnitude of the gaze vector and the magnitude of the local normal of the holographic display panel. The relative spatial angle is calculated using the inverse cosine function. This relative spatial angle represents the physical angle by which the human eye deviates from the direction of the holographic display panel's normal. The formula for calculating the relative spatial angle is: ; in, Indicates the relative spatial angle; The gaze vector is calculated by subtracting the posterior spatial coordinates of the human eye from the absolute three-dimensional coordinates of the optical center of the holographic display panel. This represents a local normal to the holographic display panel, perpendicular to the light-emitting plane of the holographic display panel; Represents the magnitude of the line-of-sight vector; This indicates the magnitude of the local normal of the holographic display panel; This represents the inverse cosine function, used to solve for the corresponding physical angle obtained by dividing the dot product of the line-of-sight vector and the local normal of the holographic display panel by the product of their magnitudes.

[0069] The servo closed-loop module reads the physical boundary threshold of the holographic effective diffraction field of view set in the system's internal non-volatile memory. The physical boundary threshold of the holographic effective diffraction field of view is set according to the holographic grating diffraction equation. The specific value is determined by taking the arcsine of the ratio of the pixel pitch constant of the spatial light modulator at the bottom layer of the holographic display panel to the physical parameter of the working wavelength at the center of the backlight module. The value range of the physical boundary threshold of the holographic effective diffraction field of view is usually between 10° and 20°. The servo closed-loop module inputs the relative spatial angle and the physical boundary threshold of the holographic effective diffraction field of view into the bottom logic comparator for numerical comparison calculation.

[0070] If the relative spatial angle is less than the physical boundary threshold of the holographic effective diffraction field of view, the servo closed-loop module determines that the human eye is currently within the effective holographic imaging space, and the handheld gimbal controller maintains the anti-shake control logic of locking the global absolute space.

[0071] If the relative spatial angle is greater than or equal to the physical boundary threshold of the effective holographic diffraction field of view, it indicates that the human eye is about to leave the optical area where a complete holographic three-dimensional image can be observed. The servo closed-loop module determines that the human eye is about to leave the effective holographic imaging space, and then triggers the servo closed-loop module to send an intervention control command to the handheld gimbal controller. For the register addressing comparison operation of the underlying logic comparator and the mathematical calculation steps of the square root of the three-dimensional vector magnitude, those skilled in the art can directly implement the hardware assembly instructions by calling the standard arithmetic logic unit inside the microprocessor.

[0072] When the relative spatial angle exceeds the physical boundary threshold of the holographic effective diffraction field of view, the servo closed-loop module calculates the target attitude offset vector and sends it to the command input bus of the handheld gimbal controller, overwriting the absolute spatial target setting value of the internal control loop of the handheld gimbal controller. The specific process of driving the three-axis motor to perform attitude tracking is as follows: The servo closed-loop module extracts the difference data between the relative spatial angle and the physical boundary threshold of the holographic effective diffraction field of view, defining the difference data as the out-of-bounds angle error. The servo closed-loop module projects the gaze vector onto the local three-dimensional Cartesian coordinate system of the holographic display panel, calculates the physical projection angle of the gaze vector on the horizontal and vertical projection planes, and calculates the ratio of the physical projection angles on the horizontal and vertical projection planes to the relative spatial angle, extracting spatial geometric weight coefficients. The servo closed-loop module multiplies the out-of-bounds angle error by the spatial geometric weight coefficients in the horizontal and vertical directions, respectively, to calculate the pitch axis compensation angle and yaw axis compensation angle required for the three-axis motor to restore the holographic field of view alignment. The servo closed-loop module defines the two-dimensional vector formed by the pitch axis compensation angle and yaw axis compensation angle as the target attitude offset vector.

[0073] The smoothing attenuation coefficient is a physical damping weight constant between 0 and 1. The smoothing attenuation coefficient is used to limit the maximum angular acceleration of a single mechanical stabilization action, preventing low-frequency mechanical tracking actions from physically interfering with high-frequency mechanical stabilization compensation.

[0074] The handheld gimbal controller operates at a cascaded proportional-integral-derivative (PID) control algorithm. When no holographic viewpoint out-of-bounds event occurs, the absolute spatial target setpoint of the outer position loop in the cascaded PID control algorithm remains statically constant to maintain the holographic display panel's fixed posture in physical absolute space. The servo closed-loop module obtains low-level hardware physical permissions to read and write internal control registers via the handheld gimbal controller's command input bus. The servo closed-loop module reads the current absolute spatial target setpoint for the current control cycle.

[0075] The servo closed-loop module performs vector superposition calculations on the target attitude offset vector, smoothing attenuation coefficient, and the current absolute space target setting value to generate an updated absolute space target setting value. The specific calculation formula for overwriting and updating the absolute space target setting value is as follows: ; in, This indicates the updated absolute space target setting. This indicates the current absolute space target setting value, which is obtained by the servo closed-loop module from the position outer loop register of the cascade proportional-integral-derivative control algorithm via the instruction input bus; This represents the smoothing attenuation coefficient, which is determined by the physical parameter of the maximum permissible angular acceleration of the handheld gimbal controller, and its value is set between 0 and 1. The target attitude offset vector is calculated from the projection components of the out-of-bounds angle error in the local three-dimensional Cartesian coordinate system.

[0076] The servo closed-loop module writes the updated absolute space target setting value back into the position outer loop register of the handheld gimbal controller, completing the dynamic overwrite operation of the absolute space target setting value. Based on the updated absolute space target setting value, the handheld gimbal controller calculates the control deviation between the actual mechanical measurement posture and the updated absolute space target setting value.

[0077] The handheld gimbal controller outputs pulse-width modulation signals based on control deviations to drive a three-axis motor to perform attitude tracking actions. The three-axis motor drives the holographic display panel to physically deflect towards the actual spatial position of the human eye, actively forcing the relative spatial angle to shrink and fall back within the physical boundary threshold of the holographic effective diffraction field of view. By performing dynamic overwrite operations and mechanical physical deflection actions, the handheld gimbal control system establishes an adaptive anti-shake feedback closed loop with the priority constraint of maintaining the holographic optical effective field of view covering the range of the human eye.

[0078] For the nested error calculation logic of the position outer loop and the speed inner loop in the cascade proportional-integral-derivative control algorithm, and the underlying hardware driving method for adjusting the output torque of the brushless motor by adjusting the duty cycle of the pulse width modulation signal, those skilled in the art can directly configure it using the general computing architecture of the standard brushless DC motor control chip. The underlying motor closed-loop control logic is a well-known technology in this field and will not be described in detail here.

[0079] Specific application examples: To verify the effectiveness of the adaptive anti-shake handheld gimbal holographic display control system proposed in this invention in solving the problems of holographic display viewpoint misalignment, cumulative drift deviation, and 3D image tearing and loss caused by the human eye leaving the effective diffraction field of view in sizzling motion scenarios, this embodiment is based on an outdoor handheld tracking shot and real-time preview of holographic projection application scenario, combined with the attached... Figure 5 and attached Figure 6 The data shown will be explained in detail.

[0080] Appendix Figure 5 and attached Figure 6 The data in this document are all internal calculation data recorded in real time by this system under multiple high-frequency jitter conditions, and are compared with traditional fixed-attitude anti-shake control algorithms.

[0081] In the application scenario of this embodiment, the tester walks quickly while holding a gimbal equipped with a holographic display panel. The preset physical refresh cycle time constant of the holographic display panel is set to 16.6ms (corresponding to a 60Hz refresh rate), and the control cycle of microsecond-level anti-shake compensation is set to 500μs. The factory optical calibration parameters indicate that the physical boundary threshold of the holographic effective diffraction field of view of the underlying spatial light modulator of the holographic display panel is set to 15°. The system has formulated a joint control scheme of viewpoint reconstruction feedforward and servo closed-loop feedback to address the risks of viewpoint drift and viewpoint boundary crossing under high-frequency jitter.

[0082] During the system initialization phase, the spatiotemporal calibration module extracts the spatial translation offset vector from the handheld gimbal base to the end of the three-axis motor. The mounting offset of the holographic panel Combined with the factory-calibrated three-dimensional rotational deviation angle , , .

[0083] According to the formula for calculating the transfer matrix The system establishes a spatial rigid body kinematic reference coordinate system and establishes a static reference benchmark with a microsecond-level control cycle.

[0084] During the test, the tester experienced a transient displacement, and the system performed anti-shake compensation.

[0085] Within a certain microsecond-level control cycle, the encoder differentially calculates the triaxial rotation angle increment. , , .

[0086] The viewpoint reconstruction module retrieves the last valid actual human eye observation coordinates from the system memory buffer. The system performs a transpose inverse operation on the global rotation transformation matrix, using the inverse spatial transformation formula. The relative spatial coordinates of the human eye under the current physical posture are calculated. .

[0087] Subsequently, the graphics processing unit retrieves the predicted spatial coordinates of the human eye during the shader processing stage. Based on the formula By superimposing translation transformation matrices, a holographic view matrix for the current rendering cycle is generated, directly eliminating viewpoint deviations caused by mechanical displacement in the underlying rendering pipeline.

[0088] During continuous microsecond-level feedforward calculations, the system asynchronously acquires macroscopic real-world data through image acquisition devices to correct cumulative calculus errors.

[0089] The image acquisition device recognizes the user's face and completes perspective N-point pose estimation, outputs the actual observation coordinates of the human eye with time-series markers, and pushes them into the system communication queue.

[0090] Combined with appendix Figure 5 The data shows that Figure 5 The horizontal axis represents the control cycle (times) of the Chinese labels, and the vertical axis represents the coordinate drift deviation (mm) of the Chinese labels. In the traditional viewpoint reconstruction algorithm without feedback calibration (the light gray dashed broken line in the figure), due to the lack of macroscopic benchmark correction, the drift deviation of the absolute spatial coordinates rapidly climbs to more than 5.0mm after the 200th control cycle, causing slight jitter and spatial misalignment of the holographic image. In this embodiment (the dark gray solid broken line in the figure), the data filtering module extracts the time delay parameter generated by the hardware processing (the delay frame span is 2 frames, and the absolute delay corresponds to 33.2ms), and uses this to construct the time step parameter of the state transition matrix.

[0091] The data filtering module retrieves historically aligned relative spatial prediction coordinates of the human eye, performs mathematical recursion, and generates a priori estimation vector of the human eye's spatial state. When the actual human eye observation coordinates with a 33.2ms time series label enter the system, an observation vector is constructed. Then, the system updates the formula according to the Kalman filter. The Kalman gain matrix is ​​calculated by combining the current measured noise covariance. The coordinate residuals are filtered and fused. (From the attached...) Figure 5As can be seen from the dark gray broken line trajectory, under the Kalman correction mechanism of this system, each time image acquisition data is obtained (approximately every 66 microseconds of control cycles), the coordinate drift deviation is instantly reset and converged, and is maintained within the low tolerance threshold of 1.0mm throughout the process. By continuously overwriting the reference point, the accumulated drift deviation is cleared.

[0092] The effectiveness of this system in preventing the image from disappearing when the gimbal panel rotates violently due to the anti-shake motion was verified.

[0093] Because the tester moved the gimbal significantly laterally while holding it, the system extracted the post-hoc human eye spatial coordinates to generate a line-of-sight vector. The local normal of the current holographic display panel is denoted as... .

[0094] From the appendix Figure 6 The data shows that Figure 6 The horizontal axis represents the running time (s) of the Chinese labels, and the vertical axis represents the relative spatial angle (°) of the Chinese labels. Figure 6 When using a traditional anti-shake scheme that locks the absolute spatial position (light gray broken line in the figure), the gimbal mechanical structure simply maintains the static posture of the panel in three-dimensional space. As a result, at 2.5s, the relative spatial angle quickly exceeds the 15° threshold and reaches a maximum of 22°. At this time, the tester's eye is completely out of the effective light output area, and the observed holographic image structure is discontinuous and disappears. In this embodiment (dark gray broken line in the figure), the servo closed-loop module monitors the spatial relative angle at high frequency.

[0095] At 2.4s, the system uses the inverse cosine function formula. The current relative spatial angle is calculated to be 15.2°. The underlying logic comparator determines that the relative spatial angle (15.2°) is greater than the preset effective diffraction field of view physical boundary threshold (15.0°). The servo closed-loop module immediately extracts the out-of-bounds angle error (0.2°) and calculates the two-dimensional target attitude offset vector required for the three-axis motor to restore the holographic field of view alignment. At this point, the system extracts the preset smooth attenuation coefficient. The formula is overridden based on the absolute space target setpoint. The updated absolute space target setting value Write the reverse value into the position outer loop register of the handheld gimbal controller.

[0096] As attached Figure 6 As shown, in the interval from 2.5s to 3.0s, the relative spatial angle in this embodiment is forcibly reduced, the three-axis motor of the gimbal is actively driven and physically deflected towards the actual spatial position of the human eye, the broken line value quickly drops back and stabilizes in the safe range of 12° to 14°, actively keeping the human eye within the coverage of the holographic effective viewing area.

[0097] Summary of application examples: This embodiment demonstrates the engineering practicality of the adaptive anti-shake handheld gimbal holographic display control system and method. On one hand, the system pre-superimposes the bias matrix in the underlying rendering pipeline of the graphics processing unit by using the forward solution and spatial inverse transformation of the instantaneous deflection vector, and uses a Kalman filter with time-stamped labels to eliminate the cumulative drift deviation caused by the microsecond-level calculation process, so that the three-dimensional coordinates are continuously kept within 1.0mm error. On the other hand, by using real-time comparison of the spatial angle between the line-of-sight vector and the local normal, the inherent defect of traditional gimbal anti-shake, which simply locks the absolute attitude and causes the loss of the holographic view area, is overcome. When the system determines that the relative spatial angle exceeds the physical boundary of diffraction, it directly overwrites the target set value of the underlying cascade control loop and drives the mechanical structure to perform active tracking, ensuring the continuity and observation stability of the holographic display in high-frequency mechanical jitter scenarios.

Claims

1. An adaptive image stabilization handheld gimbal holographic display control system, characterized in that, include: The spatiotemporal calibration module is used to align the timestamps of the holographic display panel, handheld gimbal controller and image acquisition device connected in the communication connection, and to establish a spatial rigid body kinematic reference coordinate system in combination with physical parameters; The viewpoint reconstruction module is used to calculate the instantaneous deflection vector in the spatial rigid body kinematic reference coordinate system, and to perform an inverse spatial transformation based on the instantaneous deflection vector to calculate the human eye's relative spatial predicted coordinates. The data filtering module is used to extract time delay parameters based on the timestamp and calculate the posterior human eye spatial coordinates using a discrete Kalman filter. The servo closed-loop module is used to calculate the relative spatial angle based on the posterior human eye spatial coordinates, and output the target attitude offset vector to the handheld gimbal controller when the relative spatial angle is greater than or equal to the set physical boundary threshold of the holographic effective diffraction field of view.

2. The adaptive image stabilization handheld gimbal holographic display control system according to claim 1, characterized in that, The spatiotemporal calibration module aligns the timestamps of the holographic display panel, handheld gimbal controller, and image acquisition device connected in the communication link, including: The spatiotemporal calibration module activates a hardware interrupt triggering mechanism based on the vertical synchronization signal emitted by the underlying display controller of the holographic display panel connected in communication. The spatiotemporal calibration module sends a first hardware interrupt command to the handheld gimbal controller, and uses the first global frame sequence number marker, which is assigned by the handheld gimbal controller to read and latch the absolute angle value of the three-axis motor encoder at the current command time, as the timestamp for reading the three-axis motor encoder data. The spatiotemporal calibration module sends a second hardware interrupt command to the image acquisition device, uses the second global frame sequence number of the image acquisition device as the timestamp for the image exposure acquisition, and binds the first global frame sequence number to the second global frame sequence number.

3. The adaptive image stabilization handheld gimbal holographic display control system according to claim 2, characterized in that, The spatiotemporal calibration module establishes a spatial rigid body kinematics reference coordinate system based on physical parameters, including: The spatiotemporal calibration module extracts the spatial translation offset vectors between the pitch axis, roll axis and yaw axis of the handheld gimbal to form the physical parameters of the gimbal's mechanical rotation center, and obtains the installation offset, three-dimensional rotation deviation angle and optical center normal parameters of the holographic display panel connected to the communication. The spatiotemporal calibration module adds the physical parameters of the gimbal's mechanical rotation center to the installation offset to generate a static translation homogeneous matrix, converts the three-dimensional rotation deviation angle into a static rotation homogeneous matrix, and multiplies the static translation homogeneous matrix with the static rotation homogeneous matrix to calculate the transfer matrix. The spatiotemporal calibration module uses the transfer matrix and the optical center normal parameters of the holographic display panel to construct a static rigid body transformation equation and perform multiplication to establish the spatial rigid body kinematic reference coordinate system.

4. The adaptive image stabilization handheld gimbal holographic display control system according to claim 3, characterized in that, The viewpoint reconstruction module calculates the instantaneous deflection vector in the spatial rigid body kinematic reference coordinate system, including: The viewpoint reconstruction module retrieves the absolute angle value of the three-axis motor encoder latched under the current timestamp and performs differential calculation with the absolute angle value latched under the previous frame timestamp to generate transient rotation angle integral incremental data. The viewpoint reconstruction module substitutes the transient rotation angle integral increment data into the three-dimensional space rotation equation to generate a global rotation transformation matrix. The viewpoint reconstruction module uses the global rotation transformation matrix, the transfer matrix, and the initial global normal vector to perform matrix multiplication to complete the forward kinematics calculation, thereby solving for the instantaneous deflection vector of the holographic display panel in the global absolute space.

5. The adaptive image stabilization handheld gimbal holographic display control system according to claim 4, characterized in that, The viewpoint reconstruction module performs an inverse spatial transformation based on the instantaneous deflection vector to calculate the relative spatial predicted coordinates of the human eye, including: The viewpoint reconstruction module reads the actual human eye observation coordinates acquired by the image acquisition device from the system memory buffer. These coordinates are marked with the latest timestamp and have been aligned with the spatial rigid body kinematic reference coordinate system. The viewpoint reconstruction module performs a matrix transpose operation on the global rotation transformation matrix to generate an inverse relative rotation matrix; The viewpoint reconstruction module performs matrix multiplication between the inverse relative rotation matrix and the actual observation coordinates of the human eye, performs inverse spatial transformation of the physical coordinate system to calculate the predicted relative spatial coordinates of the human eye, and transmits the predicted relative spatial coordinates of the human eye to the graphics processing unit.

6. The adaptive image stabilization handheld gimbal holographic display control system according to claim 5, characterized in that, The graphics processing unit uses the relative spatial predicted coordinates of the human eye to perform offset superposition calculations on the holographic view matrix of the current rendering cycle, including: The graphics processing unit enables a late-binding direct-write channel in the underlying display driver, and asynchronously retrieves the human eye's relative spatial prediction coordinates from the independent video memory address space through the late-binding direct-write channel; The graphics processing unit extracts the numerical components of the human eye's relative spatial predicted coordinates on the three spatial rectangular coordinate axes to construct a three-dimensional translation transformation matrix. The graphics processing unit performs multiplication and superposition calculations on the three-dimensional translation transformation matrix and the reference holographic view matrix to generate an updated holographic view matrix for the current rendering cycle.

7. The adaptive image stabilization handheld gimbal holographic display control system according to claim 2, characterized in that, The data filtering module extracts time delay parameters based on the timestamp, including: The data filtering module asynchronously reads the actual human eye observation coordinates from the system data communication queue. These coordinates are time-stamped and generated into a data structure by the image acquisition device and output by the human eye. The data filtering module reads the first global frame sequence number marker, subtracts the second global frame sequence number marker carried in the actual human eye observation coordinate data structure from the first global frame sequence number marker, and calculates the delay frame span caused by hardware processing. The data filtering module multiplies the delay frame span with the physical refresh cycle time constant to calculate the time delay parameter in the absolute time dimension, and writes the time delay parameter as a timing label into the memory address append bit of the actual observation coordinates of the human eye.

8. The adaptive image stabilization handheld gimbal holographic display control system according to claim 7, characterized in that, The data filtering module calculates the posterior human eye spatial coordinates using a discrete Kalman filter, including: The data filtering module substitutes the time delay parameter into the kinematic differential equation of the discrete Kalman filter to construct the state transition matrix; The data filtering module uses the actual observation coordinates of the human eye with the time series label as the system observation input to construct the observation vector of the current time node, retrieves the relative spatial prediction coordinates of the human eye at a historical moment that is strictly aligned with the time series label, inputs them into the state transition matrix to perform the prior mathematical recursion of the state equation, and generates a prior human eye spatial state estimation vector. The data filtering module calculates the Kalman gain matrix by combining the preset measurement noise covariance matrix and the error covariance matrix of the prior human eye spatial state estimation vector. The Kalman gain matrix is ​​then used to filter and fuse the coordinate residual values ​​between the observation vector and the prior human eye spatial state estimation vector to calculate the posterior human eye spatial state estimation vector. The updated three-dimensional position components are extracted from the posterior human eye spatial state estimation vector, and the three-dimensional position components are converted into the posterior human eye spatial coordinates.

9. The adaptive image stabilization handheld gimbal holographic display control system according to claim 1, characterized in that, The servo closed-loop module calculates the relative spatial angle based on the posterior human eye spatial coordinates, including: The servo closed-loop module extracts the absolute three-dimensional coordinates of the optical center of the holographic display panel in the spatial rigid body kinematic reference coordinate system and the local normal of the holographic display panel, and subtracts the absolute three-dimensional coordinates of the optical center of the holographic display panel from the posterior human eye spatial coordinates to generate a line-of-sight vector. The servo closed-loop module performs a three-dimensional vector dot product operation between the gaze vector and the local normal of the holographic display panel, divides the result of the three-dimensional vector dot product operation by the product of the magnitude of the gaze vector and the magnitude of the local normal of the holographic display panel, and calculates the relative spatial angle using the inverse cosine function.

10. The adaptive image stabilization handheld gimbal holographic display control system according to claim 9, characterized in that, When the relative spatial angle is greater than or equal to the set physical boundary threshold of the holographic effective diffraction field of view, the servo closed-loop module outputs the target attitude offset vector to the handheld gimbal controller, including: When the value of the relative spatial angle is greater than or equal to the physical boundary threshold of the holographic effective diffraction field of view, the servo closed-loop module extracts the difference between the relative spatial angle and the physical boundary threshold of the holographic effective diffraction field of view and defines it as the out-of-bounds angle error. The servo closed-loop module projects the gaze vector onto the local three-dimensional rectangular coordinate system of the holographic display panel, calculates the physical projection angle of the gaze vector on the horizontal projection plane and the vertical projection plane, calculates the ratio of the physical projection angle on the horizontal projection plane and the physical projection angle on the vertical projection plane to the relative spatial angle, and extracts the spatial geometric weight coefficients in the horizontal direction and the spatial geometric weight coefficients in the vertical direction respectively. The servo closed-loop module multiplies the out-of-bounds angle error by the spatial geometric weight coefficient in the horizontal direction and the spatial geometric weight coefficient in the vertical direction, respectively, to calculate the target attitude offset vector composed of the pitch axis compensation angle and the yaw axis compensation angle. The target attitude offset vector, the preset smoothing attenuation coefficient, and the current absolute space target setting value are then vector-superimposed to generate an updated absolute space target setting value, which is then written to the handheld gimbal controller.

Images