An ar hearing-aid auxiliary interaction system and method based on eye movement triggering

Abstract

The application relates to the field of AR augmented reality technology, in particular to an AR hearing-impaired auxiliary interaction system and method based on eye movement triggering, which comprises a control circuit and an eye movement capturing module, a camera module and a microphone module connected with the control circuit; the control circuit is configured to keep only the eye movement capturing module working after the system is powered on, to determine a target according to fluctuation variance and observation time length, to wake up the camera through GPIO and to perform lip image interception on only a local area corresponding to a target direction, and to write beamforming coefficients into the microphone module to lock a sound pickup main lobe and to close other direction channels. Through linkage of various modules, a hardware control mechanism is adopted, high power consumption and signal confusion caused by long-time work of a user are avoided, and low-power-consumption, high-anti-interference directional audio and video data collection is realized.

Description

Technical Field

[0001] This invention relates to the field of AR (Augmented Reality) technology, specifically to an AR assistive interaction system and method for the hearing impaired based on eye-tracking triggering. Background Technology

[0002] With the increasing maturity of Augmented Reality (AR) technology, its application in assistive communication for people with hearing and speech impairments has become an important research direction. Smart wearable devices use AR display modules to convert external language into visual text subtitles in real time, or to convert users' sign language or lip movements into speech, thereby building a two-way, efficient communication bridge between hearing or speech impediments and those with hearing.

[0003] Currently, various head-mounted assistive communication devices have been publicly disclosed in the industry. For example, the smart glasses system disclosed in Chinese patents CN202511814174.6 and CN202421772927.2 picks up external speech through a built-in microphone and converts it into visual text. At the same time, it captures sign language movements through a visual acquisition module and broadcasts them outward in the form of speech, thus constructing a basic two-way interaction chain. Similarly, the solutions disclosed in Chinese patents CN202510597349.6, CN202510464428.X, and CN202510389510.0, on smart glasses equipped with an optical waveguide display module, utilize a front-facing image acquisition device to capture hand movements or lip shapes, and combine this with a large 3D CNN or Transformer architecture model, either locally or in the cloud, to perform temporal modeling and contextual integration. For example, Chinese patents CN202410751054.5 and CN202410466657.0 disclose an auxiliary system based on mixed reality devices, microphone arrays, and multi-sound source tracking units. This system uses specific algorithms to calculate the locations of multiple sound sources and generates spatial indicator arrows pointing to the sound's location within the AR / MR display screen, thus achieving environmental sound classification and visual sound source localization. Chinese patent CN202323067036.5 discloses a wearable device for deaf and mute individuals based on sign language and speech two-way translation. This device features an additional physical heat dissipation mechanism on the top of the AR glasses to address localized heat generation during the two-way translation process.

[0004] However, the following shortcomings still exist in the existing technology: (1) The existing sound-light conversion and sign language and lip-reading comparison schemes usually adopt a rough architecture in which each perception module receives all time and all scene. This not only leads to a significant increase in the overall computing power consumption, but also, in complex social scenarios with multiple speakers overlapping dialogues, due to the widely accepted control mechanism of the system, relying solely on image acquisition or conventional microphone pickup, it is very easy to input the lateral non-target environment voice and action features into the processing core, resulting in confusion in the recognition of the machine and false alarms in transmission; (2) In the current technology, when the external mobile terminal or cloud server performs image reconstruction, the image is reconstructed by the external mobile terminal or cloud server. The data wireless transmission and graphics rendering have high latency and the image cannot be fixed. When the user suddenly changes the angle, it will cause serious dynamic ghosting, tearing or character misalignment in the user's field of view, which can easily cause visual dizziness and fatigue. (3) The existing technology uses traditional parallel power supply. The battery is usually concentrated on one side of the temple. The deployment of the battery on one side will generate heat continuously during long-term operation, which may cause serious local high heat and burn the user's face. On the other hand, the deployment on one side will also cause obvious left and right mechanical center of gravity offset, which will aggravate the mechanical pressure on the bridge of the nose and one side of the ear canal and reduce the user's comfort. Summary of the Invention

[0005] The purpose of this invention is to provide an AR hearing-impaired assistive interaction system based on eye-tracking triggering. This method is used to solve the problems in the prior art, such as excessive power consumption caused by the continuous concurrent operation of multimodal perception modules, signal confusion and recognition delay in complex scenes, dynamic ghosting and visual dizziness caused by AR subtitle display relying on external frame-by-frame reconstruction, and local heat generation caused by long-term operation of a single battery.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] An AR-assisted interactive system for the hearing impaired based on eye-tracking includes: a control circuit, an eye-tracking module, a camera module, and a microphone module;

[0008] The data output terminal of the eye-tracking module is connected to the control circuit via an SPI bus for real-time acquisition of the user's eye focus data.

[0009] The power enable terminal (EN) of the camera module is connected to the control circuit via a GPIO pin;

[0010] The microphone module employs a multi-microphone linear array and integrates an audio digital signal processor (DSP) chip, the configuration register bus of which is connected to the control circuit.

[0011] After the control circuit system is powered on, only the eye-tracking module remains operational. Based on the gaze data it outputs, the user's gaze convergence vector is calculated. When it is determined that the variance of the gaze vector at a certain angle in space is less than a preset threshold of 0.05 and the duration is in the range of 300ms-800ms, it means that the gaze has been fixed in one direction and there is no shaking. Therefore, it is established as the target dialogue orientation θ.

[0012] By pulling the enable pin (EN) of the camera module high via the GPIO pin, the camera module is hardware-wake-up and driven to capture lip image frames only from the central field of the local region of interest (ROI) corresponding to the target dialogue orientation θ. The ROI is a rectangular area with the target dialogue orientation θ as its geometric center and a horizontal and vertical viewing angle range of ±15°. At the same time, beamforming coefficients are written to the audio digital signal processor (DSP) of the microphone module via the bus to lock the dynamic delay parameters of the pickup main lobe within the ±30° angle range of the target dialogue orientation θ, and to shut down the audio channels in other directions.

[0013] Preferably, the eye-tracking module uses a near-infrared global shutter image sensor and is equipped with two infrared LED fill lights with a wavelength of 850nm; the hardware acquisition frame rate of the module is not less than 60Hz and the spatial tracking accuracy is less than 0.5°.

[0014] Preferably, the specific method for the control circuit to calculate the user's line-of-sight convergence vector is as follows:

[0015] The eye-tracking module captures images of the user's eyes under infrared illumination and locally extracts the pupil center coordinates. , ) and the center coordinates of the corneal Purchin spot (infrared reflector) ( , );

[0016] Calculate the two-dimensional difference vector between the pupil center and the corneal reflector. .

[0017] The control circuit employs a quadratic polynomial mapping algorithm to convert a two-dimensional difference vector into a line-of-sight direction vector in three-dimensional space. The mapping formula is as follows: ; ;

[0018] Where (x,y) is the difference vector The amount, and These are the horizontal azimuth and pitch angles in three-dimensional space, respectively, with coefficient a. i and b i(Where i∈{0,1,2,3,4,5}) is obtained in advance by the system through a nine-point calibration method. The control circuit constructs a three-dimensional convergence vector of the user's gaze in real time based on the calculated azimuth and pitch angles. The nine-point calibration method utilizes nine calibration points with known spatial coordinates to establish a mapping model between the difference vector and spatial angles, and solves for the optimal mapping coefficients using the least squares method, thereby quantizing the two-dimensional eye-tracking features into a three-dimensional gaze vector in subsequent real-time tracking.

[0019] Preferably, the multi-microphone linear array of the microphone module is a four-microphone linear array, consisting of four digital MEMS microphones arranged in a straight line at equal intervals, with a center-to-center distance of 20mm between adjacent microphones. The four digital MEMS microphones are fixed to the upper edge of the forehead of the eyeglasses frame. The center-to-center distance between two adjacent microphones is fixed at 20mm, and the total axial length of the entire array is 60mm.

[0020] Preferably, when the control circuit writes beamforming coefficients to the audio digital signal processor (DSP) chip, its core logic is to perform delay compensation on the acquired signals of each channel in the four-microphone linear array, so that the acoustic signals from the target dialogue azimuth θ are completely aligned in phase, and then perform full weighted summation to achieve constructive interference. The formula for calculating the time-domain output response weights in the frequency domain is as follows: ;

[0021] in, Let f(m) be the Fourier transform response of the signal acquired by the m-th microphone (where m∈{1,2,3,4}). Angular frequency, This refers to the output signal response after beamforming. Let be the complex beamforming weight coefficient corresponding to the m-th microphone channel, and its expression is as follows: ;

[0022] Where j is the imaginary unit, This refers to the dynamic delay parameter of the m-th microphone relative to the array's geometric reference point.

[0023] The beamforming coefficient is a coefficient generated based on a delay accumulation algorithm; the dynamic delay parameter is calculated from the target azimuth θ based on a time difference of arrival algorithm, specifically: taking the first microphone in the four-microphone linear array as a spatial geometric reference point, the time difference of arrival delay parameter of each microphone channel relative to this reference point. The specific mathematical calculation formula is as follows:

[0024] ;

[0025] in, This refers to the dynamic delay parameter of the m-th microphone relative to the array's geometric reference point. This is the microphone channel number. The spacing between adjacent microphones is fixed at 20mm. The speed of sound (taken as 340 m / s under standard atmospheric pressure);

[0026] In digital signals, the continuous time delay τm(θ) calculated above is typically a non-integer number of sampling periods. To achieve high-precision time delay compensation in the digital domain, the control circuit employs a Farrow structure fractional-time-delay finite impulse response filter to convert the calculated continuous time delay... Approximating the continuous time delay, corresponding digital delay filter coefficients are generated and written into the configuration register of the audio digital signal processor chip via the I2C bus, thereby locking the main pickup lobe within the ±30° angle range of the target dialogue orientation θ and shutting down audio channels in other directions.

[0027] Preferably, the system further includes a power module, which consists of a first power supply and a second power supply symmetrically built into the left and right temples. The first and second power supplies are two sets of high-energy-density custom-shaped lithium polymer batteries from the same manufacturer and batch. The capacity, physical mass, and internal resistance parameters of the two sets of batteries are completely identical, so that the center of gravity of the whole device falls on the central axis of the glasses.

[0028] The first MOSFET switch (Q1) and the second MOSFET switch (Q2) are respectively connected between the output terminals of the first power supply and the second power supply and the energy storage capacitor assembly of the main power supply rail of the whole machine. The gates of the two MOSFET switches are independently driven by the first GPIO pin and the second GPIO pin of the control circuit. The energy storage capacitor assembly is composed of three multilayer ceramic capacitors (MLCC) connected in parallel.

[0029] The first NTC thermistor (NTC1) is mounted next to the battery on the left temple, and its analog signal output terminal is connected to the first ADC channel of the control circuit; the second NTC thermistor (NTC2) is mounted on the heating area of ​​the right temple, and its analog signal output terminal is connected to the second ADC channel of the control circuit; both thermistors (NTC1 and NTC2) are high-precision, small-package negative temperature coefficient surface mount types.

[0030] Preferably, the control circuit uses the first ADC channel and the second ADC channel to perform high-frequency voltage sampling on the thermistors (NTC1, NTC2) through a voltage divider network, and calculates the temperature T of the left armpiece in real time using an internally integrated calibration table or the Steinhart-Hart formula.左 Temperature T of the main heating area of ​​the right temple 右 This provides a basis for the subsequent temperature rise gradient dT 右 The exact calculus calculations of / dt provide raw data input with zero latency;

[0031] When the temperature rise gradient reaches 0.5°C / s to 1.5°C / s, or the dynamic temperature difference between the left and right temples is greater than 3°C, the power supply switching sequence is triggered: first, the first MOSFET switch (Q1) is fully turned on to maintain the main power supply rail voltage stability, and then the second MOSFET switch (Q2) is turned off to allow the second power supply to enter a physical cooling state; the time interval between the first MOSFET switch (Q1) being fully turned on and the second MOSFET switch (Q2) being turned off is 10μs to 50μs.

[0032] Preferably, the system further includes: an IMU nine-axis inertial sensor, preferably an ICM-20948 low-power nine-axis motion tracking chip (packaged in 3mm×3mm×1mm QFN), whose hardware interrupt pin (INT1) is connected to the external interrupt input terminal (EXINT) of the control circuit, and whose SPI data bus is bound to the DMA channel of the control circuit;

[0033] The LCoS optomechanical / waveguide lens module has its drive input connected to the display data bus of the control circuit. The LCoS optomechanical module integrates a display start row address register and a column offset register. The internal LCoS chip preferably uses a JD7208 silicon-based liquid crystal microdisplay chip. The hardware driver board of the LCoS chip integrates a high-speed display buffer register (FrameBuffer), and its physical address space has an open display start row address register (Start Row Address Register) and a column offset register (Column Offset Register).

[0034] The control circuit is configured such that when the IMU nine-axis inertial sensor detects a sudden change in head angular velocity, its hardware interrupt pin triggers the highest priority interrupt service routine. The control circuit reads angular velocity data through the DMA channel and directly rewrites the display start row address register and column offset register of the LCoS optical engine internal display controller to achieve overall translation compensation of screen pixels. The control circuit (1) has a kernel-level nested vector interrupt controller (NVIC) integrated inside.

[0035] Preferably, the control circuit triggers the highest priority interrupt service routine under the following conditions: the instantaneous angular velocity ω of the current axis > 0.1 rad / s (to filter out minor body tremors caused by human eyes, breathing, or heartbeat), or the time rate of change of angular velocity (i.e., angular acceleration) dω / dt > 0.5 rad / s. 2 (Used to accurately identify real shaking caused by user turning head, pausing while walking, or vibration); the total response time of a single hardware anti-shake compensation is ≤1ms; this dual-index composite triggering mechanism ensures both sensitive capture of shaking and avoids high-frequency false triggering of interruptions caused by slight background noise.

[0036] When the highest priority interrupt is triggered, the control circuit uses the DMA channel to read the horizontal axis angular velocity at time t. angular velocity with vertical axis To achieve anchoring compensation with "equal magnitude and opposite direction," the system employs a geometric acoustic and optical projection mapping formula to calculate the column offset register, thereby achieving overall pixel translation compensation. The pixel translation amount is calculated according to the principle of being equal in magnitude and opposite in direction to the head's angular velocity, specifically using the following formula: ; ;

[0037] in, To calculate the horizontal pixel displacement of the column offset register, To display the vertical pixel displacement of the starting line address register, This is the current image stabilization control period (fixed at 1ms). The negative sign indicates that the pixel translation direction is strictly opposite to the head movement direction. The equivalent combined focal length of the LCoS optical engine projection system and the optical waveguide lens is 18mm. and These are the horizontal and vertical physical pixel sizes of a single pixel in an LCoS display chip, respectively.

[0038] Preferably, the interaction method of the AR hearing-impaired assistive interaction system based on eye-tracking includes the following steps:

[0039] S1: After the system is powered on, the control circuit keeps only the eye-tracking module working, collecting the user's eye focus data in real time and calculating the convergence vector of the user's gaze.

[0040] S2: When it is determined that the variance of the line-of-sight vector at a certain angle in space is less than a preset threshold of 0.05 and the duration reaches 300ms-800ms, the control circuit establishes it as the target dialogue orientation θ;

[0041] S3: The control circuit wakes up the camera module hardware by pulling up the enable pin (EN) of the camera module through the GPIO pin, and drives the camera module to capture lip image frames only in the central field of the local region of interest (ROI) corresponding to the target dialogue orientation θ; image data of non-ROI areas are directly discarded locally and not uploaded to the cloud.

[0042] S4: The control circuit writes beamforming coefficients to the audio digital signal processor (DSP) of the microphone module via the I2C bus, locks the dynamic delay parameter of the pickup main lobe within the ±30° angle range of the target dialogue orientation θ, and shuts down audio channels in other directions.

[0043] S5: The control circuit performs local edge feature extraction on the acquired lip image and directional speech signal to generate a binary image matrix of lip edges and an audio feature packet of Mel-frequency cepstral coefficients (MFCC) within the sound field beam;

[0044] S6: The wireless module uploads the audio feature package and the binary image matrix of the lip edge to the smart mobile application terminal, which then calls the cloud-based large model to perform semantic recognition, context integration, and bidirectional text and speech synthesis.

[0045] S7: The synthesized interactive character stream or audio stream is asynchronously transmitted back to the glasses, where AR text rendering is performed through the LCoS optical engine / waveguide lens module, and directional voice broadcast is given to external interlocutors through the miniature speaker module.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0047] 1. This invention utilizes an eye-tracking module, a camera module, a microphone module, and their connections to the control circuit. Eye-tracking determines the target dialogue based on fluctuation variance, while the camera module captures lip images from the target's location. Simultaneously, the bus locks the microphone module's range and shuts down audio channels in other directions, achieving a control mechanism of "eye-tracking perception - target locking - multimodal directional acquisition." This avoids the high energy consumption caused by all modules operating continuously and in all scenarios, and also reduces signal recognition confusion and transmission false alarms caused by non-target locations in complex environments.

[0048] 2. This invention uses a nine-axis IMU inertial sensor to prevent jitter through hardware interrupts. It reads angular velocity data through a DMA channel and directly performs pixel translation compensation on the internal registers when the device is powered off. Combined with pixel displacement compensation in the opposite direction of the user's head movement, it can anchor the subtitles and objects in the original target position when the user's head moves instantaneously. This eliminates the dynamic ghosting, tearing, and character misalignment of virtual subtitles, and avoids visual dizziness and fatigue for the user.

[0049] 3. The present invention utilizes a power module consisting of a first power supply and a second power supply symmetrically integrated into the left and right temples. The first and second power supplies are two sets of high-energy-density custom-shaped lithium polymer batteries from the same manufacturer and batch. These are combined with an NTC thermistor to monitor the temperature in real time. When the temperature rise gradient of the temple or the dynamic temperature difference between the two sides exceeds a threshold, a switching function is triggered to cut off the battery on the heating side, thus avoiding facial burns caused by unilateral heating. At the same time, the identical configuration on both sides also avoids the mechanical pressure on the bridge of the nose and ears caused by unilateral configuration, improving user comfort. Attached Figure Description

[0050] Figure 1 This is a flowchart of the steps of the AR hearing-impaired assistive interaction method based on eye movement triggering of the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Example 1

[0053] refer to Figure 1 The present invention provides an AR-assisted interactive system for the hearing impaired based on eye movement triggering, comprising: a control circuit, an eye movement capture module, a camera module, and a microphone module;

[0054] The data output of the eye-tracking module is connected to the control circuit via an SPI bus to collect real-time eye focus data. The power enable (EN) of the camera module is connected to the control circuit via a GPIO pin. The microphone module uses a multi-microphone linear array and integrates an audio digital signal processor (DSP) chip. The configuration register bus of the audio digital signal processor (DSP) chip is connected to the control circuit. The control circuit is configured to keep only the eye-tracking module working after the system is powered on, and calculate the user's gaze convergence vector based on its output gaze data. When the variance of the gaze vector at a certain angle in space is less than a preset threshold of 0.05 and the duration is in the range of 300ms-800ms, it means that the gaze has been fixed in one direction and there is no shaking. Therefore, it is established as the target dialogue orientation θ.

[0055] By pulling the enable pin (EN) of the camera module high on the GPIO pin, the camera module is hardware-wake-up and driven to capture lip image frames only from the central field of the local region of interest (ROI) corresponding to the target dialogue orientation θ. The ROI is a rectangular area with the target dialogue orientation θ as the geometric center and a horizontal and vertical viewing angle range of ±15°. At the same time, beamforming coefficients are written to the audio digital signal processor (DSP) of the microphone module through the bus to lock the dynamic delay parameters of the main lobe of the pickup within the ±30° angle range of the target dialogue orientation θ, and to shut down the audio channels in other directions.

[0056] Furthermore, the eye-tracking module employs a near-infrared global shutter image sensor, coupled with two infrared LEDs with a wavelength of 850nm; the hardware acquisition frame rate of this module is no less than 60Hz, and the spatial tracking accuracy is less than 0.5°.

[0057] Furthermore, the specific method for the control circuit to calculate the user's gaze convergence vector is as follows: the eye-tracking module captures the user's eye image under infrared illumination and locally extracts the pupil center coordinates. , ) and the center coordinates of the corneal Purchin spot (infrared reflector) ( , ); Calculate the two-dimensional difference vector between the pupil center and the corneal reflection point. .

[0058] The control circuit employs a quadratic polynomial mapping algorithm to convert a two-dimensional difference vector into a line-of-sight direction vector in three-dimensional space. The mapping formula is as follows: ; ;

[0059] in,( , () is the difference vector The amount, and These are the horizontal azimuth and pitch angles in three-dimensional space, respectively, and their coefficients. and (Where i∈{0,1,2,3,4,5}) is obtained in advance by the system through a nine-point calibration method. The control circuit constructs a three-dimensional convergence vector of the user's gaze in real time based on the calculated azimuth and pitch angles. The nine-point calibration method utilizes nine calibration points with known spatial coordinates to establish a mapping model between the difference vector and spatial angles, and solves for the optimal mapping coefficients using the least squares method, thereby quantizing the two-dimensional eye-tracking features into a three-dimensional gaze vector in subsequent real-time tracking.

[0060] In this embodiment, the system pre-calculates the mapping coefficients using the nine-point calibration method. and The complete calibration process for (where i∈{0,1,2,3,4,5}) is as follows:

[0061] Step 1: After the calibration physical array is established and the gaze guidance system is powered on and enters calibration mode, the control circuit drives the LCoS optomechanical / waveguide lens module to sequentially render and display nine known spatial reference target points P arranged in a 3*3 rectangular topological array on a virtual display plane 1.5m in front of the user. k (where k∈{1, 2,...,9}).

[0062] Normalized spatial coordinates of each calibration point on the display plane ( , ) and its corresponding theoretical three-dimensional target horizontal azimuth angle With pitch angle All values ​​are known preset values ​​of the system. The calibration guidance program controls the nine reference target points to flash sequentially according to a preset time sequence, guiding the user's gaze to stably fixate on each calibration point in turn, with the fixation duration at each calibration point not less than 500ms.

[0063] Step 2: Feature Extraction and Binary Difference Vector Construction When the user gazes at the k-th calibration point At the same time, the eye-tracking module captures high-frequency images of the user's eyes and extracts the coordinates of the pupil center image locally. , And the central image coordinates of the corneal Purchin spot (infrared reflector). , The control circuit calculates the two-dimensional difference vectors between the two on the image plane: ; ;

[0064] Differential calculations eliminate common-mode position deviations caused by minute physical slippage of the eyeglass frame on the user's face.

[0065] Step 3: Constructing the mapping coefficient matrix equation and solving it using least squares;

[0066] After sampling all 9 calibration points, the system obtained 9 sets of known input feature points. , ) and the corresponding theoretical spatial angle target value ( , ).

[0067] The control circuit constructs a quadratic polynomial mapping matrix equation. Taking the horizontal azimuth angle θ as an example, a system of overdeterministic linear equations containing 9 equations is constructed: ;

[0068] In the formula, Let M be a 9-dimensional target angle column vector, M be a 9*6 design matrix composed of 9 sets of difference features, and A be a 6-dimensional mapping coefficient column vector to be determined. Its specific matrix expansion form is as follows: ;

[0069] Similarly, pitch angle The matrix equation is expressed in the form of: ; ;

[0070] The control circuit uses a generalized inverse matrix (least squares method) to solve the above equations, obtaining the optimal mapping coefficient column vectors A and B with the minimum sum of squared residuals: ; ;

[0071] Step 4: Coefficient storage and 3D view convergence vector construction

[0072] The calculated polynomial mapping coefficient matrices ai and bi (i ∈ {0,1,2,3,4,5}) are written into the non-volatile memory (EEPROM / Flash) of the control circuit. In the subsequent real-time operation phase, the control circuit will read the real-time acquired differential components (…). , Substitute the obtained polynomial mapping formula: ; ;

[0073] Then, using the solved horizontal azimuth angle With pitch angle A standardized three-dimensional line-of-sight vector Vgaze is constructed in a three-dimensional Cartesian coordinate system: ;

[0074] After both eyes have completed the above calculations, the control circuit calculates the intersection or center geometric mean of the visual direction vectors of both eyes, thereby establishing the convergence vector of the user's vision in three-dimensional space in real time.

[0075] Furthermore, the microphone module's multi-microphone linear array is a four-microphone linear array, consisting of four digital MEMS microphones arranged in a straight line at equal intervals, with a center-to-center distance of 20mm between adjacent microphones. The four digital MEMS microphones are fixed to the upper edge of the forehead of the eyeglasses frame in a straight line arrangement at equal intervals. The center-to-center distance between two adjacent microphones is fixed at 20mm, and the total axial length of the entire array is 60mm.

[0076] In this embodiment, the hardware selection and electrical parameters are as follows: An ICS-43434 digital microphone is preferably used, supporting direct communication with the control circuit via an I2S digital audio interface. This microphone model has a nominal sensitivity of −26dBFS, a signal-to-noise ratio (SNR) of not less than 64dBA, and a total harmonic distortion (THD) of less than 0.1% across the entire dynamic range. Four digital MEMS microphones are arranged linearly at equal intervals and fixed to the upper edge of the forehead of the eyeglasses frame. The center-to-center distance between two adjacent microphones is fixed at 20mm, and the total axial length of the entire array is 60mm. This geometric spacing matches the mainstream frequency range of human voice signals (300Hz~4000Hz), enabling the audio DSP to obtain optimal directional spatial gain when performing beamforming using the Time Difference of Arrival (TDOA) algorithm, forming an extremely narrow pickup lobe within the horizontal field of view.

[0077] Furthermore, when the control circuit writes beamforming coefficients to the audio digital signal processor (DSP) chip, its core logic is to perform delay compensation on the acquired signals of each channel in the four-microphone linear array, so that the acoustic signals from the target dialogue azimuth θ are completely aligned in phase, and then perform full weighted summation to achieve constructive interference. The formula for calculating the time-domain output response weights in the frequency domain is as follows: ;

[0078] in, Let f(m) be the Fourier transform response of the signal acquired by the m-th microphone (where m∈{1,2,3,4}). Angular frequency, This refers to the output signal response after beamforming. Let be the complex beamforming weight coefficient corresponding to the m-th microphone channel, and its expression is as follows: ;

[0079] Where j is the imaginary unit, This refers to the dynamic delay parameter of the m-th microphone relative to the array's geometric reference point.

[0080] The beamforming coefficients are generated based on a delay accumulation algorithm; the dynamic delay parameters are calculated from the target azimuth θ based on a time difference of arrival algorithm, specifically: taking the first microphone in the four-microphone linear array as the spatial geometric reference point (m=1), the arrival time difference delay parameters of each microphone channel relative to this reference point. The specific mathematical calculation formula is as follows: ;

[0081] in, This refers to the dynamic delay parameter of the m-th microphone relative to the array's geometric reference point. This is the microphone channel number. The spacing between adjacent microphones is fixed at 20mm. The speed of sound (taken as 340 m / s under standard atmospheric pressure);

[0082] In digital signals, the continuous time delay τm(θ) calculated above is typically a non-integer number of sampling periods. To achieve high-precision time delay compensation in the digital domain, the control circuit employs a Farrow structure fractional-time-delay finite impulse response filter to convert the calculated continuous time delay... Approximating the continuous time delay, corresponding digital delay filter coefficients are generated and written into the configuration register of the audio digital signal processor chip via the I2C bus, thereby locking the main pickup lobe within the ±30° angle range of the target dialogue orientation θ and shutting down audio channels in other directions.

[0083] Furthermore, the system also includes: a power module consisting of a first power supply and a second power supply symmetrically built into the left and right temples. The first and second power supplies are two sets of high-energy-density custom lithium polymer irregularly shaped batteries from the same manufacturer and batch. The capacity, physical mass, and internal resistance parameters of the two sets of batteries are completely identical, so that the center of gravity of the whole device falls on the central axis of the glasses.

[0084] The first MOSFET switch (Q1) and the second MOSFET switch (Q2) are respectively connected between the output terminals of the first power supply and the second power supply and the energy storage capacitor assembly of the main power supply rail of the whole machine. The gates of the two MOSFET switches are independently driven by the first GPIO pin and the second GPIO pin of the control circuit. The energy storage capacitor assembly is composed of three multilayer ceramic capacitors (MLCC) connected in parallel.

[0085] The first NTC thermistor (NTC1) is mounted next to the battery on the left temple, and its analog signal output is connected to the first ADC channel of the control circuit; the second NTC thermistor (NTC2) is mounted on the heating area of ​​the right temple, and its analog signal output is connected to the second ADC channel of the control circuit; both thermistors (NTC1 and NTC2) are high-precision, small-package negative temperature coefficient surface mount types.

[0086] In this embodiment, both the first MOSFET switch (Q1) and the second MOSFET switch (Q2) are P-channel enhancement-mode MOSFETs with low conduction loss and low turn-on voltage. Preferably, the AO3401A (in SOT-23 package) is used. Under a drive state with a gate-source voltage VGS = −4.5V, its drain-source on-resistance (maximum RDS(ON)) is less than 44mΩ; its gate threshold voltage (VGS(th)) is within the −0.7V to −1.3V range, matching the 3.3V logic level drive of the GPIO pin of the control circuit. This selection ensures that even under full load operation, the Joule heat loss of the switch itself is negligible. The energy storage capacitor assembly consists of three multilayer ceramic capacitors connected in parallel. The total capacitance of the ceramic capacitor group is 44μF (specifically composed of two 22μF ceramic capacitors in a 0805 package connected in parallel), and its equivalent series resistance (ESR) is strictly less than 5mΩ at a 100kHz operating frequency. This extremely low ESR architecture provides the main power rail of the entire machine with extremely strong transient current response capability.

[0087] Furthermore, the control circuit uses the first and second ADC channels to perform high-frequency voltage sampling on the thermistors (NTC1, NTC2) through a voltage divider network, and calculates the temperature T of the left armpiece in real time using an internally integrated calibration table or the Steinhart-Hart formula. 左 Temperature T of the main heating area of ​​the right temple 右 This provides a basis for the subsequent temperature rise gradient dT 右 The exact calculus calculations of / dt provide raw data input with zero latency;

[0088] When the temperature rise gradient reaches 0.5°C / s~1.5°C / s, or the dynamic temperature difference between the left and right mirror legs is greater than 3°C, the power supply switching sequence is triggered: first, the first MOSFET switch (Q1) is fully turned on to maintain the main power supply rail voltage stability, and then the second MOSFET switch (Q2) is turned off to allow the second power supply to enter the physical cooling state; the time interval between the first MOSFET switch (Q1) being fully turned on and the second MOSFET switch (Q2) being turned off is 10μs~50μs.

[0089] In this embodiment, all thermistors are high-precision, small-package negative temperature coefficient surface-mount types. NCP15XH103F03RC (0402 package) is preferred. Its nominal resistance at 25°C is 10kΩ ± 1%; the thermistor's B constant (within the range of 25°C to 50°C) is strictly fixed at 3380K ± 1%.

[0090] Furthermore, the system also includes: an IMU nine-axis inertial sensor, preferably an ICM-20948 low-power nine-axis motion tracking chip (packaged in 3mm×3mm×1mm QFN), whose hardware interrupt pin (INT1) is connected to the external interrupt input terminal (EXINT) of the control circuit, and whose SPI data bus is bound to the DMA channel of the control circuit;

[0091] The LCoS optomechanical / waveguide lens module has its drive input connected to the display data bus of the control circuit. The LCoS optomechanical module integrates a display start row address register and a column offset register. The internal LCoS chip preferably uses the JD7208 silicon-based liquid crystal microdisplay chip. The hardware driver board of the LCoS chip integrates a high-speed display buffer register (FrameBuffer), and its physical address space has the display start row address register (Start Row Address Register) and column offset register (Column Offset Register) open.

[0092] The control circuit is configured such that when the IMU nine-axis inertial sensor detects a sudden change in head angular velocity, its hardware interrupt pin triggers the highest priority interrupt service routine. The control circuit reads the angular velocity data through the DMA channel and directly rewrites the display start row address register and column offset register of the LCoS optomechanical internal display controller to achieve overall translation compensation of screen pixels. The control circuit integrates a kernel-level nested vector interrupt controller (NVIC).

[0093] In this embodiment, the IMU nine-axis inertial sensor has a range configured to ±2000∘ / s, and its angular velocity scale sensitivity error is strictly controlled within ±1%. Its internal accelerometer has a range configured to ±4g. This high dynamic range ensures that acceleration and angular velocity signals do not experience hardware clipping distortion when the user rapidly turns their head or experiences walking vibrations, resulting in extremely high linearity in the output raw attitude data. The control circuit integrates a kernel-level nested vector interrupt controller (NVIC). During system initialization, the control circuit writes a specific mask to the NVIC's interrupt priority register (IPR) to configure the external interrupt line mapped to the hardware interrupt pin (INT1) of the IMU nine-axis inertial sensor as having the highest preemption priority. When the user's head experiences a momentary high-frequency sway, causing the ICM-20948 to become data ready and pull the INT1 pin high, the control circuit immediately suspends all currently executing low-priority background tasks (such as wireless data packets, UI rendering, etc.) and forcibly enters the hardware interrupt service routine (ISR) with zero delay. Internally, the control circuitry utilizes its dedicated Direct Memory Access (DMA) channel to directly read nine-axis data into the local buffer at a 20MHz SPI bus rate without occupying any CPU computation cycles. This underlying mechanism ensures the highest time determinism of the stabilization system. The LCoS chip's hardware driver board integrates a high-speed display cache register (Frame Buffer), which includes the Start Row Address Register and Column Offset Register in its physical address space. When the highest preemptive priority interrupt is triggered, the control circuitry calculates the reverse angular velocity of the head movement and directly rewrites the reference offset values ​​of these two registers via the parallel display data bus. At this point, without needing to re-render the entire image pixel matrix, the LCoS chip directly performs a physical translation of the entire text pixel matrix before the next hardware refresh cycle based on the modified register address. This hardware register-level reverse displacement compensation is strictly controlled within 1ms, achieving physical anchoring of the subtitles to the real world and avoiding 3D visual dizziness caused by latency.

[0094] Furthermore, the conditions for the control circuit to trigger the highest priority interrupt service routine are: the instantaneous angular velocity ω of the current axis > 0.1 rad / s (to filter out minor body tremors caused by human eyes, breathing, or heartbeat), or the time rate of change of angular velocity (i.e., angular acceleration) dω / dt > 0.5 rad / s. 2(Used to accurately identify real shaking caused by user turning head, pausing while walking, or vibration); the total response time of a single hardware anti-shake compensation is ≤1ms; this dual-index composite triggering mechanism ensures both sensitive capture of shaking and avoids high-frequency false triggering of interruptions caused by slight background noise.

[0095] When the highest priority interrupt is triggered, the control circuit uses the DMA channel to read the horizontal axis angular velocity at time t. angular velocity with vertical axis To achieve anchoring compensation with "equal magnitude and opposite direction," the system employs a geometric acoustic and optical projection mapping formula to calculate the column offset register, thereby achieving overall pixel translation compensation. The pixel translation amount is calculated according to the principle of being equal in magnitude and opposite in direction to the head's angular velocity, specifically using the following formula: ; ;

[0096] in, To calculate the horizontal pixel displacement of the column offset register, To display the vertical pixel displacement of the starting line address register, This is the current image stabilization control period (fixed at 1ms). The negative sign indicates that the pixel translation direction is strictly opposite to the head movement direction. The equivalent combined focal length of the LCoS optical engine projection system and the optical waveguide lens is 18mm. and These are the horizontal and vertical physical pixel sizes of a single pixel in an LCoS display chip, respectively.

[0097] In this embodiment, starting from the IMU pulling up the INT1 pin, the hardware interrupt response delay of the control circuit suspending the background task and entering the ISR is ≤5μs; the delay of transferring the nine-axis raw data to memory via the 20MHz high-speed SPI bus in conjunction with DMA is ≤12μs. In the highest priority interrupt service routine, the time taken for the control circuit to execute the multiplication and division calculations of the aforementioned tangent pixel mapping formula and boundary limiting algorithm is ≤8μs. The delay for the control circuit to write the calculated ΔX and ΔY into the LCoS control chip register via the parallel display data bus and have them latched by the internal hardware counter is ≤15μs. The cumulative absolute delay of the above hardware stages is controlled at around 40μs, which is much smaller than the time window of a single hardware display frame of the LCoS chip at a 90Hz refresh rate (approximately 11.1ms). From the occurrence of physical jitter to the completion of the memory reference address translation, the overall hardware and software response time is ≤1ms (averaging 40μs~50μs), thus ensuring that pixel movement is completed within the human eye's visual persistence time. The LCoS chip has a physical resolution of 1280×720, with physical field of view (FOV) of 36∘ horizontally and 22° vertically. When the virtual subtitles are displayed normally in the center, a maximum allowable margin area is reserved in the video memory canvas. The maximum horizontal translation limit is set to Xmax = ±128 pixels, and the maximum vertical translation limit is set to Ymax = ±72 pixels (corresponding to deformation safety thresholds of approximately ±3.6∘ and ±2.2∘ at the edge of the physical field of view).

[0098] Furthermore, the interaction method of the eye-tracking-triggered AR assistive interaction system for the hearing impaired includes the following steps:

[0099] S1: After the system is powered on, the control circuit keeps only the eye-tracking module working, collecting the user's eye focus data in real time and calculating the convergence vector of the user's gaze.

[0100] S2: When the variance of the line-of-sight vector at a certain angle in space is less than the preset threshold of 0.05 and the duration reaches 300ms-800ms, the control circuit establishes it as the target dialogue orientation θ.

[0101] S3: The control circuit pulls the enable pin (EN) of the camera module high through the GPIO pin to wake up the camera module hardware and drive the camera module to capture lip image frames only for the central field of the target dialogue position θ of the local area of ​​interest (ROI); the image data of the non-ROI area is directly discarded locally and is not uploaded to the cloud.

[0102] S4: The control circuit writes beamforming coefficients to the audio digital signal processor (DSP) of the microphone module via the I2C bus, locks the dynamic delay parameter of the pickup main lobe within the ±30° angle range of the target dialogue azimuth θ, and shuts down the audio channels in other directions.

[0103] S5: The control circuit performs local edge feature extraction on the acquired lip images and directional speech signals to generate a binary image matrix of lip edges and an audio feature packet of Mel-frequency cepstral coefficients (MFCC) within the sound field beam;

[0104] S6: The wireless module uploads the audio feature package and the binarized image matrix of the lip edge to the smart mobile application, which then calls the cloud-based large model to perform semantic recognition, context integration, and bidirectional text and speech synthesis.

[0105] S7: The synthesized interactive character stream or audio stream is asynchronously transmitted back to the glasses, where AR text rendering is performed through the LCoS optical engine / waveguide lens module, and directional voice broadcast is given to external interlocutors through the miniature speaker module.

[0106] Example 2

[0107] An experiment was conducted to investigate the localized heating generated in the temples of eyeglasses due to prolonged high-intensity operation of the core control components and wireless communication modules. The experimental conditions were set as follows: The power supply modules were the first and second power supplies, both using two sets of high-energy-density custom-shaped soft-pack lithium polymer batteries from the same manufacturer and batch. Each battery had a capacity of 350mAh, a nominal voltage of 3.7V, and identical physical mass and internal resistance. The ambient reference temperature was fixed at 25°C ± 0.5°C. Sampling configuration: An NTC thermistor (NCP15XH103F03RC) mounted next to the battery on the inside of the temples monitored the temperature in real time at a sampling rate of 100Hz. An infrared thermal imager monitored the highest temperature on the outer surface of the temples (the side in contact with the skin) in real time. Experimental groups: Five groups of tests were conducted with different discharge rates (C-rate) and chip power consumption combinations to observe the temperature rise gradient d. T右 / dt, the time for the outer surface of the temple to reach 40°C, and the system switching response. Please refer to the table below for specific experimental data: Group 1 (Light Load) 0.5C (175mA), Base A Display 0.12~0.25°C / s ＜0.8°C none No switching was triggered, and natural heat dissipation maintained a balance. Group 2 (Medium Load) 1.0C (350mA), continuous speech recognition 0.35~0.48°C / s ＜1.5°C none No switching was triggered; the temperature is within the normal steady-state heating range. Group 3 (Critical Warning) 1.5C (525mA), large-scale streaming inference 0.52~0.85°C / s 2.1～3.2°C Low Accurately trigger warnings and initiate power routing switching by first turning on and then off the power source. Group 4 (High Load) 2.0C (700mA), full-function concurrent full-load 1.15~1.42°C / s 4.5～5.8°C Medium and high Accurately triggers the warning, completing the bottom layer blockage before heat penetrates to the outer shell. Group 5 (Extreme / Failure) 2.5C (875mA), overload or partial short circuit simulation >1.65°C / s ＞7.2°C High (easily triggers chip HW package) Forcefully trigger main circuit shutdown and switch to single-sided safe power supply mode.

[0108] Experimental data show that when the temperature gradient is below 0.5°C / s (Groups 1 and 2), the Joule heat generated by the system is still within the equilibrium range of natural dissipation of the eye shell, and the unilateral thermal equilibrium has not been broken. At this time, there is no need to switch to avoid unnecessary system operation. When the temperature gradient reaches or exceeds 1.5°C / s (Group 5), the heat will be rapidly conducted to the outer surface of the temple, requiring early intervention. Otherwise, it is very easy to trigger the hardware thermal shutdown protection of the chip or cause burning discomfort to the user's skin. In the range of 0.5°C / s-1.5°C / s (Groups 3 and 4), heat conduction can be blocked before it occurs.

[0109] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0110] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An AR-based hearing-impaired assistive interaction system triggered by eye movements, characterized in that, include: Control circuit, eye-tracking module, camera module, microphone module; The data output terminal of the eye-tracking module is connected to the control circuit via an SPI bus for real-time acquisition of the user's eye focus data. The power enable terminal (EN) of the camera module is connected to the control circuit via a GPIO pin; The microphone module employs a multi-microphone linear array and integrates an audio digital signal processor (DSP) chip, the configuration register bus of which is connected to the control circuit. After the control circuit system is powered on, only the eye-tracking module is kept working. The user's gaze convergence vector is calculated based on the gaze data it outputs. When it is determined that the variance of the gaze vector at a certain angle in space is less than a preset threshold of 0.05 and the duration is in the range of 300ms-800ms, it is established as the target dialogue orientation θ. By pulling the enable pin (EN) of the camera module high via the GPIO pin, the camera module is hardware-wake-up and driven to capture lip image frames only from the central field of the local region of interest (ROI) corresponding to the target dialogue orientation θ. The ROI is a rectangular area with the target dialogue orientation θ as its geometric center and a horizontal and vertical viewing angle range of ±15°. At the same time, beamforming coefficients are written to the audio digital signal processor (DSP) of the microphone module via the bus to lock the dynamic delay parameters of the main lobe of the microphone within the ±30° angle range of the target dialogue orientation θ, and to shut down the audio channels in other directions.

2. The AR-assisted interactive system for the hearing impaired based on eye-tracking as described in claim 1, characterized in that: The eye-tracking module uses a near-infrared global shutter image sensor and is equipped with two infrared LEDs with a wavelength of 850nm. The hardware acquisition frame rate of this module is no less than 60Hz, and the spatial tracking accuracy is less than 0.5°.

3. The AR-assisted interactive system for the hearing impaired based on eye-tracking as described in claim 1, characterized in that: The specific method by which the control circuit calculates the user's line-of-sight convergence vector is as follows: The eye-tracking module captures images of the user's eyes under infrared illumination and locally extracts the pupil center coordinates. , ) and the center coordinates of the corneal Purchin spot (infrared reflector) ( , ); Calculate the two-dimensional difference vector between the pupil center and the corneal reflector. ; The control circuit employs a quadratic polynomial mapping algorithm to convert a two-dimensional difference vector into a line-of-sight direction vector in three-dimensional space. The mapping formula is as follows: ； ； Where (x,y) is the difference vector The amount, and These are the horizontal azimuth and pitch angles in three-dimensional space, respectively, with coefficient a. i and b i (Where i∈{0,1,2,3,4,5}) is obtained in advance by the system through a nine-point calibration method. The control circuit constructs a three-dimensional convergence vector of the user's line of sight in real time based on the calculated azimuth and pitch angles.

4. The AR-assisted interactive system for the hearing impaired based on eye-tracking as described in claim 1, characterized in that: The microphone module's multi-microphone linear array is a four-microphone linear array, consisting of four digital MEMS microphones arranged in a straight line at equal intervals, with a center-to-center distance of 20mm between adjacent microphones.

5. The AR-assisted interactive system for the hearing impaired based on eye-tracking as described in claim 4, characterized in that: The beamforming coefficients written by the control circuit to the audio digital signal processor (DSP) chip are coefficients generated based on a delay accumulation algorithm; the dynamic delay parameters are calculated from the target azimuth θ based on a time difference of arrival algorithm, specifically: taking the first microphone in the four-microphone linear array as a spatial geometric reference point, the time difference of arrival delay parameters of each microphone channel relative to this reference point. The specific mathematical calculation formula is as follows: ； in, This refers to the dynamic delay parameter of the m-th microphone relative to the array's geometric reference point. This is the microphone channel number. The distance between adjacent microphones. Speed ​​of sound; The control circuit employs a Farrow structure fractional time delay finite impulse response filter to calculate the continuous time delay. The coefficients are converted into digital delay filter coefficients and written into the configuration register of the audio digital signal processor chip via the I2C bus.

6. The AR-assisted interactive system for the hearing impaired based on eye-tracking as described in claim 1, characterized in that, Also includes: The power module includes a first power supply and a second power supply symmetrically built into the left and right temples. The battery capacity, physical mass and internal resistance parameters of the two are completely identical, so that the center of gravity of the whole device is located on the central axis of the glasses. The first MOSFET switch (Q1) and the second MOSFET switch (Q2) are respectively connected between the output terminals of the first power supply and the second power supply and the energy storage capacitor assembly of the main power supply rail of the whole machine. The gates of the two MOSFET switches are independently driven by the first GPIO pin and the second GPIO pin of the control circuit. The first NTC thermistor (NTC1) is mounted next to the battery on the left temple, and its analog signal output terminal is connected to the first ADC channel of the control circuit. The second NTC thermistor (NTC2) is mounted on the heating area of ​​the right temple, and its analog signal output terminal is connected to the second ADC channel of the control circuit.

7. The AR-assisted interactive system for the hearing impaired based on eye-tracking as described in claim 6, characterized in that: The control circuit uses the first and second ADC channels to perform high-frequency voltage sampling on the thermistors (NTC1, NTC2) through a voltage divider network, and calculates the temperature T of the left armpiece in real time using an internally integrated calibration table or the Steinhart-Hart formula. 左 Temperature T of the main heating area of ​​the right temple 右 This provides a basis for the subsequent temperature rise gradient dT 右 The exact calculus calculations of / dt provide raw data input with zero latency; When the temperature rise gradient reaches 0.5°C / s to 1.5°C / s, or the dynamic temperature difference between the left and right temples is greater than 3°C, the power supply switching sequence is triggered: first, the first MOSFET switch (Q1) is fully turned on to maintain the main power supply rail voltage stability, and then the second MOSFET switch (Q2) is turned off to allow the second power supply to enter a physical cooling state; the time interval between the first MOSFET switch (Q1) being fully turned on and the second MOSFET switch (Q2) being turned off is 10μs to 50μs.

8. The AR-assisted interactive system for the hearing impaired based on eye-tracking as described in claim 1, characterized in that, Also includes: The IMU nine-axis inertial sensor has its hardware interrupt pin (INT1) connected to the external interrupt input (EXINT) of the control circuit, and its SPI data bus is bound to the DMA channel of the control circuit. The LCoS optomechanical / waveguide lens module has its drive input terminal connected to the display data bus of the control circuit. The LCoS optomechanical module integrates a display start row address register and a column offset register. The control circuit is configured such that when the IMU nine-axis inertial sensor detects a sudden change in head angular velocity, its hardware interrupt pin triggers the highest priority interrupt service routine. The control circuit reads the angular velocity data through the DMA channel and directly rewrites the display start row address register and column offset register of the LCoS optical engine internal display controller to achieve overall translation compensation of the screen pixels.

9. The AR-assisted interactive system for the hearing impaired based on eye-tracking as described in claim 8, characterized in that: The control circuit triggers the highest priority interrupt service routine under the following conditions: the instantaneous angular velocity ω of the current axis > 0.1 rad / s, or the time rate of change of angular velocity (i.e., angular acceleration) dω / dt > 0.5 rad / s. 2 The total response time for a single hardware image stabilization compensation is ≤1ms. When the highest priority interrupt is triggered, the control circuit uses the DMA channel to read the horizontal axis angular velocity at time t. angular velocity with vertical axis The system employs a geometric acoustic and optical projection mapping formula to calculate column offset registers and achieve overall pixel translation compensation. The pixel translation amount is calculated according to the principle that it is equal in magnitude and opposite in direction to the angular velocity of head movement, specifically using the following formula: ； ； in, To calculate the horizontal pixel displacement of the column offset register, To display the vertical pixel displacement of the starting line address register, This is the current image stabilization control period (fixed at 1ms). The negative sign indicates that the pixel translation direction is strictly opposite to the head movement direction. The equivalent combined focal length of the LCoS optical engine projection system and the optical waveguide lens is 18mm. and These are the horizontal and vertical physical pixel sizes of a single pixel in an LCoS display chip, respectively.

10. The interaction method of the AR hearing-impaired assistive interaction system based on eye-tracking triggering according to any one of claims 1 to 9, characterized in that, Includes the following steps: S1: After the system is powered on, the control circuit keeps only the eye-tracking module working, collecting the user's eye focus data in real time and calculating the convergence vector of the user's gaze. S2: When it is determined that the variance of the line-of-sight vector at a certain angle in space is less than a preset threshold of 0.05 and the duration reaches 300ms-800ms, the control circuit establishes it as the target dialogue orientation θ; S3: The control circuit wakes up the camera module hardware by pulling up the enable pin (EN) of the camera module through the GPIO pin, and drives the camera module to capture lip image frames only for the central field of the local region of interest (ROI) corresponding to the target dialogue position θ; the image data of the non-ROI area is directly discarded locally and is not uploaded to the cloud. S4: The control circuit writes beamforming coefficients to the audio digital signal processor (DSP) of the microphone module via the I2C bus, locks the dynamic delay parameter of the pickup main lobe within the ±30° angle range of the target dialogue orientation θ, and shuts down audio channels in other directions. S5: The control circuit performs local edge feature extraction on the acquired lip image and directional speech signal to generate a binary image matrix of lip edges and an audio feature packet of Mel-frequency cepstral coefficients (MFCC) within the sound field beam; S6: The wireless module uploads the audio feature package and the binary image matrix of the lip edge to the smart mobile application terminal, which then calls the cloud-based large model to perform semantic recognition, context integration, and bidirectional text and speech synthesis. S7: The synthesized interactive character stream or audio stream is asynchronously transmitted back to the glasses, where AR text rendering is performed through the LCoS optical engine / waveguide lens module, and directional voice broadcast is given to external interlocutors through the miniature speaker module.

Images