A method, system, and display for interactive display terminals based on multimodal fusion control.
By configuring modulation features for the backlight zones of the display terminal, and using the superposition of low-frequency images and high-frequency beacon signals to drive the near-field electromagnetic induction field, gesture events are calculated in real time and multi-modal fusion verification is performed. This solves the single perception and stability problems of existing display terminal interaction technologies, and achieves a highly intelligent and low-mistouch interaction effect.
Patent Information
- Application Number
- CN202611305916.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-26
- Publication Date
- 2026-11-17
AI Technical Summary
Existing human-computer interaction technologies for display terminals suffer from limited interaction modes, fixed perception dimensions, disconnect between interaction logic and display scenarios, low intelligence, and difficulty in meeting the actual usage needs of multiple scenarios, high precision, and low false touches. Furthermore, external sensing hardware is costly, susceptible to light, occlusion, and environmental electromagnetic interference, has poor recognition stability, and visual sensing devices pose a privacy risk.
By establishing multiple backlight zones for the display terminal and assigning modulation features to each backlight zone, the low-frequency image dimming signal and the high-frequency beacon modulation signal are superimposed to drive each backlight zone, a near-field electromagnetic induction field is constructed, electromagnetic signals are collected by electromagnetic induction coils, disturbance deviations are calculated in real time, gesture events are obtained, multi-modal fusion verification is performed, and interactive commands are output.
It achieves native air gesture interaction with no peripherals, high stability, low false touch, and strong scene adaptability. The overall solution requires zero additional hardware and has a high degree of integration, which greatly improves the intelligence, precision and stability of human-computer interaction on display terminals.
Smart Images

Figure CN123114307A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of display screens, and more specifically, to a display terminal interaction method, system, and display based on multimodal fusion control. Background Technology
[0002] Existing human-computer interaction technologies for display terminals generally suffer from technical defects such as single interaction modes, fixed perception dimensions, and disconnect between interaction logic and display scenarios. The overall interaction architecture is highly homogenized and has a low level of intelligence, making it difficult to meet the actual usage needs of multiple scenarios, high precision, and low accidental touches.
[0003] Most current mainstream gesture interaction solutions rely on independent peripherals such as cameras, infrared arrays, millimeter-wave radar, and external capacitive sensors to complete gesture collection and recognition. Such solutions can only achieve simple gesture judgment based on a single perception dimension. External sensing hardware is expensive, disrupts the overall appearance of the device, and is susceptible to light, obstruction, and environmental electromagnetic interference, resulting in poor recognition stability. Visual sensing devices also pose a privacy risk. Summary of the Invention
[0004] In view of the problems existing in the prior art, this application provides a display terminal interaction method, system and display based on multimodal fusion control.
[0005] A display terminal interaction method based on multimodal fusion control includes:
[0006] Multiple backlight zones are pre-established for the display terminal, and modulation features are assigned to each backlight zone. The source of the electromagnetic field radiated by each backlight zone is identified by the modulation features. By superimposing low-frequency image dimming signals and high-frequency beacon modulation signals through a preset driving circuit, each backlight zone is driven, so that each backlight zone establishes a near-field electromagnetic induction field with various modulation characteristics in the near-field space in front of the screen while displaying the image normally. The superimposed electromagnetic signal formed by the superposition of electromagnetic fields radiated by each backlight partition in the near-field electromagnetic induction field is continuously collected by a preset electromagnetic induction coil, and the channel signal corresponding to each backlight partition is extracted from the superimposed electromagnetic signal according to the modulation characteristics. The disturbance deviation between each channel signal and the preset reference signal is calculated in real time. Based on the temporal variation characteristics of the disturbance deviation on the corresponding channel signal of each backlight partition and the preset spatial mapping characteristics of each backlight partition, the gesture event, including gesture type and hand position, is determined. Acquire display content modal data, perform multimodal fusion verification on the gesture event and the content modal data, and output the interaction command corresponding to the gesture event based on the verification result.
[0007] A display terminal interaction system based on multimodal fusion control, comprising: A partition configuration unit is used to pre-establish multiple backlight partitions for the display terminal and assign modulation features to each backlight partition, thereby identifying the source of the electromagnetic field radiated by each backlight partition through the modulation features; The electromagnetic construction unit is used to superimpose low-frequency image dimming signals and high-frequency beacon modulation signals through a preset driving circuit to drive each backlight zone, so that each backlight zone can establish a near-field electromagnetic induction field with various modulation characteristics in the near-field space in front of the screen while displaying the picture normally. An electromagnetic induction unit is used to continuously collect the superimposed electromagnetic signal formed by the superposition of electromagnetic fields radiated by each backlight partition in the near-field electromagnetic induction field through a preset electromagnetic induction coil, and extract the channel signal corresponding to each backlight partition from the superimposed electromagnetic signal according to the modulation characteristics. The gesture acquisition unit is used to calculate the disturbance deviation between each channel signal and the preset reference signal in real time, and to determine the gesture event, including gesture type and hand position, based on the temporal change characteristics of the disturbance deviation on the corresponding channel signal of each backlight partition and the preset spatial mapping characteristics of each backlight partition. The interaction matching unit is used to acquire display content modal data, perform multimodal fusion verification on the gesture event and the content modal data, and output the interaction command corresponding to the gesture event based on the verification result.
[0008] A display for implementing the multimodal fusion control-based display terminal interaction method described in any of the preceding claims.
[0009] Beneficial Effects: This application proposes a display terminal interaction method, system, and display based on multimodal fusion control. By configuring independent modulation features for each backlight zone, it realizes the frequency division multiplexing superposition driving of image dimming signals and high-frequency beacon modulation signals, enabling the backlight hardware to have dual functions of screen display and distributed electromagnetic beacon radiation. It constructs a near-field electromagnetic induction sensing system that can be traced by zone and can cover the whole area. From the architecture level, it realizes native air gesture interaction with no peripherals, high stability, low false touch, and strong scene adaptability. The overall solution has zero new hardware, high integration, and complete appearance, which greatly improves the intelligence, precision, and stability of human-computer interaction in display terminals.
[0010] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a schematic diagram of the display terminal interaction method of this application; Figure 2 This is a schematic diagram illustrating the principle of the display terminal interaction method of this application; Figure 3 This is a schematic diagram of the enhanced frequency point processing flow of this application; Figure 4 This is a schematic diagram of the display terminal interaction system module of this application; Figure 5 This is a schematic diagram of the display module of this application.
[0013] Figure reference numerals: A1-Partition configuration unit; A2-Electromagnetic construction unit; A3-Electromagnetic induction unit; A4-Gesture acquisition unit; A5-Interaction matching unit; 1-Partition backlight module; 2-High-frequency backlight driving circuit; 3-Electromagnetic induction coil; 4-Signal acquisition and processing module; 5-System control module. Detailed Implementation
[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0015] This application proposes a display terminal interaction method based on multimodal fusion control, which realizes heterogeneous computing power collaboration with perception-driven, timing-matched, and seamless presentation. It achieves excellent image quality, smoothness, and energy efficiency, effectively solving the technical problems of decoupling computing power collaboration from display timing, disconnect between computing power allocation and visual perception, and easily imperceptible image quality upgrades in existing technologies. A schematic diagram of the display terminal interaction method is attached. Figure 1 As shown in the attached diagram, the principle is as follows. Figure 2 As shown, the specific solution is as follows: A display terminal interaction method based on multimodal fusion control includes: 101. Multiple backlight zones are pre-established for the display terminal, and modulation features are assigned to each backlight zone. The source of the electromagnetic field radiated by each backlight zone is identified by the modulation features. 102. By superimposing the low-frequency image dimming signal and the high-frequency beacon modulation signal through the preset driving circuit, each backlight zone is driven, so that each backlight zone establishes a near-field electromagnetic induction field with various modulation characteristics in the near-field space in front of the screen while displaying the normal picture. 103. By continuously collecting the superimposed electromagnetic signals formed by the superposition of electromagnetic fields radiated by each backlight partition in the near-field electromagnetic induction field through a preset electromagnetic induction coil, the channel signal corresponding to each backlight partition is extracted from the superimposed electromagnetic signal based on the modulation characteristics. 104. Calculate the disturbance deviation between each channel signal and the preset reference signal in real time, and determine the gesture event, including gesture type and hand position, based on the temporal change characteristics of the disturbance deviation on the corresponding channel signal of each backlight partition and the preset spatial mapping characteristics of each backlight partition. 105. Obtain the display content modal data, perform multimodal fusion verification on the gesture event and content modal data, and output the interaction command corresponding to the gesture event based on the verification result.
[0016] This embodiment operates on a display terminal equipped with a Mini LED multi-zone backlight structure. The supporting hardware system includes a multi-zone Mini LED backlight module, a high-frequency backlight driving circuit, at least one set of electromagnetic induction coils, a signal acquisition and processing module, and a system control unit. The multi-zone Mini LED backlight module divides the backlight into numerous independent zones, each with an independent driving channel, allowing for individual adjustment of the driving current and output waveform. The high-frequency backlight driving circuit can output composite driving waveforms of different frequency bands. The electromagnetic induction coils are located inside the display bezel, on the back panel, or on the motherboard, responsible for picking up spatially alternating electromagnetic signals. The signal acquisition and processing module integrates amplifiers, filters, analog-to-digital converters, and spectrum analysis units to perform raw signal preprocessing and feature separation operations. The system control unit coordinates the data interaction, timing scheduling, and logical operations of all modules. The entire solution requires no additional active sensors such as cameras, millimeter-wave radar, or external sensing electrodes, relying on the terminal's native hardware to achieve air gesture recognition interaction.
[0017] Step 101: Multiple backlight zones are pre-established for the display terminal, and modulation features are assigned to each backlight zone. The source of the electromagnetic field radiated by each backlight zone is identified by the modulation features.
[0018] Dividing the backlight emission area into multiple backlight zones is a fundamental feature for achieving spatial resolution. The division method employs a rectangular array arrangement, with all backlight zones tightly joined to cover the entire effective backlight emission area. There are no overlapping areas between zones, nor are there any uncovered blank areas. Each backlight zone has an independent electrical drive path, allowing the system to output a drive waveform to any single backlight zone, independently change its current operating state, and independently radiate an alternating electromagnetic field. If a backlight zone cannot be independently controlled, and all backlight zones share the same set of drive signals, the radiated electromagnetic field characteristics of all areas will be completely identical, making it impossible to distinguish signals from different spatial locations and thus preventing positioning. The tight joints, lack of overlapping areas, and absence of uncovered blank areas of the zones ensure that each backlight zone has a unique and definite physical location within the screen plane. In subsequent steps, the system can directly map the detected channel disturbance location to two-dimensional coordinates on the screen, eliminating positioning blind spots or mapping ambiguities.
[0019] Modulation features are the core feature of this step. They are unique identifiers loaded onto the high-frequency drive signals of the backlight zones, allowing the back-end signal processing module to identify and distinguish the signal source. Two implementation methods are available for modulation features: frequency division multiplexing (FDM), which assigns non-overlapping operating frequencies to different backlight zones; and code division multiplexing (CDM), which assigns orthogonal pseudo-random codes to different backlight zones. Both implementations share the common characteristic that the modulation features of any two backlight zones will not be confused. Even after multiple electromagnetic fields are spatially superimposed, the receiver can still rely on the modulation features to trace back to which backlight zone the signal originated from.
[0020] Identifying the source of electromagnetic fields through modulation features is a necessary condition for achieving distributed spatial sensing. Within the space in front of a single display screen, numerous backlight zones radiate electromagnetic fields simultaneously, and these fields superimpose to form a mixed field. Without a unique modulation feature, the acquired mixed electromagnetic signal can only reflect the average electromagnetic variation of the entire space, failing to distinguish which backlight zone the disturbance occurs on, thus preventing the acquisition of precise two-dimensional location information. Pre-completing the zone division and modulation feature binding is a pre-configuration step. This configuration can be completed during the device's factory calibration phase or automatically executed during each device power-on initialization. The configuration information is permanently stored in the terminal storage unit, and this mapping relationship is continuously invoked throughout subsequent signal analysis.
[0021] The binding relationship between modulation features and backlight zones is stored in the terminal's non-volatile storage unit in the form of a mapping table. The mapping table contains at least the following fields: backlight zone number, backlight zone physical location coordinates, assigned modulation feature type, and parameters. In subsequent steps, when the signal acquisition and processing module performs channel separation, and when the system control unit solves for the hand's two-dimensional coordinates, this mapping table must be retrieved to complete the corresponding calculations. If the mapping table is lost or not properly configured, subsequent steps cannot be executed correctly.
[0022] Step 102 uses a preset driving circuit to superimpose the low-frequency image dimming signal and the high-frequency beacon modulation signal to drive each backlight zone, so that each backlight zone can establish a near-field electromagnetic induction field with various modulation characteristics in the near-field space in front of the screen while displaying the image normally.
[0023] The low-frequency image dimming signal and the high-frequency beacon modulation signal are the two core characteristic signals in this step. The low-frequency image dimming signal operates in a frequency band adapted to the backlight brightness adjustment requirements. The signal level changes directly control the backlight output brightness and grayscale level. Its core function is to realize the image display function and ensure that the monitor can output images, videos, operation interfaces, and other screen content normally. The high-frequency beacon modulation signal has a frequency much higher than the human eye's visual cutoff frequency. The human eye cannot perceive the brightness fluctuations caused by this signal. The signal amplitude is constrained by the system to prevent image flickering, uneven brightness, or other image quality damage. The high-frequency beacon modulation signal carries the modulation characteristics set in step 101 and relies on high-frequency alternating current to excite an alternating electromagnetic field.
[0024] The two types of signals are driven using a time-domain superposition and frequency-domain isolation method. Frequency-domain isolation means that the two sets of signals occupy completely non-overlapping frequency ranges, preventing mutual interference and distortion. Time-domain superposition means that the two signals are merged into a single composite drive current input to the backlight LED chip. Under the drive of the composite current, the LED chip simultaneously performs two tasks: first, it adjusts the brightness of the light emission based on the low-frequency component to achieve image display; second, it radiates an alternating electromagnetic field carrying modulation characteristics outward based on the high-frequency alternating component.
[0025] The near-field electromagnetic induction field is the core sensing medium generated in this step. When alternating current flows through the LED chip and backlight traces, a quasi-static alternating electromagnetic field is formed in the near-field region within a limited distance in front of the screen. This electromagnetic field does not belong to the far-field radiation category; its energy is concentrated in the near-field space in front of the screen. The electromagnetic field radiated outward by each backlight zone carries its own corresponding modulation characteristics. After a large number of zones work together, they are integrated to form a continuously distributed, fully covered near-field electromagnetic induction field in front of the screen. When a hand enters the range of this sensing field, the conductivity and dielectric properties of the human body can change the degree of electromagnetic coupling in the local area, causing a change in the electromagnetic signal in the corresponding area, providing a physical basis for gesture detection. If a continuous near-field electromagnetic induction field cannot be constructed, hand movements will not produce recognizable signal changes, and gesture perception cannot be achieved.
[0026] The effective range of the near-field electromagnetic induction field is influenced by a combination of factors, including the amplitude of the backlight zone drive current, the modulation frequency, and the sensitivity of the induction coil. Under typical parameter configurations, the effective sensing distance ranges from several centimeters to tens of centimeters, covering the commonly used distance range for normal user gesture operations. This range can be dynamically adjusted according to actual product requirements by adjusting the amplitude of the high-frequency beacon modulation signal and the gain of the induction coil. An excessively short effective range will limit the gesture operation space, while an excessively long effective range will increase power consumption and electromagnetic compatibility issues. Therefore, a compromise configuration must be made based on the target scenario in actual deployment.
[0027] Step 103 involves continuously collecting superimposed electromagnetic signals formed by the superposition of electromagnetic fields radiated by each backlight partition in the near-field electromagnetic induction field using a preset electromagnetic induction coil, and extracting the channel signal corresponding to each backlight partition from the superimposed electromagnetic signal based on the modulation characteristics.
[0028] The electromagnetic induction coil is the core hardware feature for data acquisition in this step. It utilizes the principle of electromagnetic induction to pick up the energy of alternating electromagnetic fields in space, converting the alternating magnetic field into a voltage signal that can be processed by the circuit. The coil is fixedly installed inside the monitor, requiring no additional accessories from the user. The raw signal acquired by the coil is a superimposed electromagnetic signal, a composite signal formed by the spatial coupling and mixing of the electromagnetic fields radiated by all backlight zones. The raw signal contains electromagnetic components from all zones and lacks zone-specific positioning capabilities; therefore, the location of any disturbance cannot be determined solely by the raw signal.
[0029] After receiving the superimposed electromagnetic signal, the signal acquisition and processing module sequentially performs amplification, filtering, analog-to-digital conversion, and spectrum analysis. The module retrieves the modulation feature lookup table stored in step 101 and uses matched filtering or a spectrum feature recognition algorithm to separate the independent signal components belonging to each backlight zone from the mixed superimposed signal. The separated independent signal components are defined as channel signals, and each channel signal uniquely corresponds to a backlight zone. For example, the modulation feature lookup table is retrieved, and the corresponding separation algorithm is selected based on the implementation form of the modulation feature: if the modulation feature adopts frequency division mode, the channel signal corresponding to each frequency point is separated in the frequency domain using a spectrum analysis algorithm; if the modulation feature adopts code division mode, the corresponding channel signal is despread from the superimposed electromagnetic signal using matched filtering or a correlation despreading algorithm, utilizing the pre-set orthogonal pseudo-random code of each zone. The common principle of both algorithms is to utilize the orthogonality or frequency domain isolation between modulation features to peel off the components corresponding to different backlight zones from the mixed signal one by one.
[0030] A one-to-one correspondence between channel signals establishes a mapping bridge between electromagnetic signals and screen spatial positions. After separation, the system can independently monitor electromagnetic fluctuations in the space above any backlight zone. When a hand approaches a zone, only the channel signal corresponding to that zone shows a significant change, while the other channel signals remain stable. Without feature separation, the system can only observe overall signal changes across the entire area, unable to distinguish the location of disturbances, and can only determine the presence of an object in front, but cannot recognize gesture coordinates and movement trajectories. Continuous acquisition means that the acquisition work continues throughout the entire normal operation of the device, uninterruptedly acquiring real-time electromagnetic states to achieve real-time gesture detection.
[0031] The channel signals separated from the superimposed electromagnetic signals can undergo further filtering and signal-to-noise ratio (SNR) evaluation before being included in the perturbation deviation calculation. For channel signals with an SNR below a preset threshold, the system can mark them as unreliable channels, reduce their weight during gesture recognition, or temporarily exclude them, only to reintegrate them once the SNR recovers. This mechanism ensures that the channel signals relied upon for subsequent gesture recognition have sufficient signal quality, avoiding misjudgments due to deterioration in the quality of individual channel signals.
[0032] Step 104: Calculate the disturbance deviation between each channel signal and the preset reference signal in real time. Based on the temporal variation characteristics of the disturbance deviation on the corresponding channel signal of each backlight partition and the preset spatial mapping characteristics of each backlight partition, determine the gesture event, including gesture type and hand position.
[0033] The preset reference signal is the core reference feature of this step. The reference signal is acquired in a stable environment free from hand contact and dynamic interference. During the equipment initialization phase, all channel steady-state signals are acquired and saved. The reference signal records the standard amplitude and standard phase of each channel under standard conditions, serving as a comparison benchmark. Electromagnetic effects from static metal elements and fixed supports are all included in the reference signal. The reference signal represents the steady-state electromagnetic state without interactive actions. Without a reference signal, the system cannot distinguish between normal environmental electromagnetic fluctuations and electromagnetic changes caused by hand contact.
[0034] The system continuously compares the real-time channel signal corresponding to each backlight zone with a pre-saved reference signal for that channel to obtain the real-time disturbance deviation. The disturbance deviation characterizes the degree of deviation of the near-field electromagnetic environment above the zone from the non-interactive steady-state environment at the current moment. The value of the disturbance deviation dynamically changes with hand movements such as approaching, moving, and moving away. The system continuously records the complete evolution of the disturbance deviation of all backlight zone channels over time, thereby extracting the temporal variation characteristics of the disturbance deviation on each channel. These temporal variation characteristics include the order in which the disturbance deviations appear, the rate of increase, the rate of decrease, the duration of the disturbance, and the transmission order of the disturbance between adjacent zones. Different hand movements will cause the disturbance deviation to exhibit completely different temporal evolution patterns. Multiple zones may experience simultaneous increases or decreases in disturbance deviation, corresponding to the hand approaching or moving away from the screen. If the disturbance deviations of adjacent zones change sequentially in a fixed order, it corresponds to hand swiping or sliding. By relying on the temporal variation characteristics, various hand gestures can be distinguished, and the gesture type can be determined.
[0035] Each backlight zone has a preset spatial mapping feature, which is a correspondence established and fixed in advance during the calibration phase. This records the physical location of each backlight zone, the screen pixel coordinates, and the binding relationship between screen plane areas. When a specific zone channel experiences a valid disturbance deviation, the system uses the spatial mapping feature to convert the zone number of the disturbance into two-dimensional coordinates of the hand on the screen plane. Simultaneously, the system integrates the magnitude of disturbance deviations in multiple adjacent zones and the degree of difference in disturbance deviations between zones to further deduce the distance of the hand in the vertical direction of the screen, thus obtaining complete hand position information.
[0036] The system integrates the gesture types obtained from temporal variation feature analysis with the hand position information obtained from spatial mapping feature analysis to form a complete gesture event. This step strictly follows the previous signal decomposition stage. After obtaining independent channel signals by separating modulation features, it extracts disturbance deviations using a preset reference signal as a steady-state reference. Then, it simultaneously analyzes the action type and spatial position from two dimensions: the temporal evolution law of the disturbance deviation and the spatial correspondence of the partitions, thus fully constructing the gesture event and providing effective input data for subsequent multimodal fusion verification.
[0037] Gesture events are the output results obtained from comprehensive analysis. Each gesture event contains two types of information: gesture type and hand position information. The system can only generate complete and usable gesture events by simultaneously combining temporal features for gesture classification and spatial mapping features for position determination. Obtaining only the position is insufficient to distinguish the type of action, and distinguishing only the gesture type is insufficient to determine the target location; both are indispensable. Real-time calculation of representative signals and continuous feature analysis ensure that gesture recognition has real-time response capabilities.
[0038] The specific logic for 3D distance estimation is as follows: When the hand is close to the screen, the perturbation amplitude of the area directly opposite the hand is the largest, and the perturbation amplitude of adjacent areas decreases rapidly with increasing distance, exhibiting a steep spatial gradient. When the hand is far from the screen, the electromagnetic field's influence range expands, and the difference in perturbation amplitude among multiple adjacent areas decreases, exhibiting a gentle spatial gradient. The system calculates the degree of difference in perturbation amplitude of the corresponding channel signals of adjacent backlight areas within the hand's coverage area, and combines this with the number of currently covered areas and the magnitude of the absolute perturbation amplitude to comprehensively determine the vertical distance between the hand and the screen. When the hand's coverage areas are similar, the larger the absolute perturbation amplitude and the more significant the difference in perturbation amplitude between adjacent areas, the closer the hand is to the screen.
[0039] Step 105: Obtain the display content modal data, perform multimodal fusion verification on the gesture event and content modal data, and output the interaction command corresponding to the gesture event based on the verification result.
[0040] Display content modal data is the core input feature of this step. It is output in real-time by the display control unit and includes the pixel coordinates of all UI controls on the current screen, the boundaries of interactive areas, the range of blank, non-interactive areas, and the current scene type. This data comprehensively describes which areas of the screen support operation and which do not. This modal data is independent of gesture events obtained through electromagnetic sensing and represents a second, independent information source. The display control unit extracts this data from the display buffer or UI control layout description file of the current frame, and its update frequency is synchronized with the screen refresh rate. When the screen content changes (e.g., page switching, pop-up appearance or disappearance, control position movement), the display content modal data is updated synchronously to ensure that the UI layout information used for fusion verification is consistent with the screen currently seen by the user. If the display content modal data update lags behind screen changes, gestures landing on old UI control positions may be misjudged as valid operations, or landing on new UI control positions may be misjudged as invalid operations.
[0041] Multimodal fusion verification is the core logical feature of this step. The first information source is the gesture event obtained by analyzing electromagnetic signals, belonging to the electromagnetic sensing modality; the second information source is screen image information, belonging to the display content modality. Fusion verification is not a simple sequential judgment, but a joint judgment based on mutual constraints between the two sets of information. The system first extracts the hand position within the gesture event and finds the corresponding screen area in the display content modality data. If there are no interactive controls in this area, even if the electromagnetic signal recognizes the gesture, the system determines that the gesture is an invalid disturbance and directly intercepts it without generating an interaction command. If the landing point is in an interactive area, the system then combines the current screen scene to verify whether the gesture is suitable for the current scene's interaction logic, filtering out erroneous operations that do not conform to the scene's usage habits.
[0042] The interaction command is the final output feature of this step. After multimodal fusion verification determines a gesture event to be valid, the system converts the gesture type and hand position into standardized interaction commands that the operating system can recognize. Command forms include clicks, swipes, drags, selections, and page switching. The commands are then output to the operating system or foreground application to complete the interaction response. The core value of setting up dual-modal fusion verification is to address the deficiency of false triggers caused by single electromagnetic sensing. Relying solely on electromagnetic signals cannot distinguish whether a gesture falls within the valid operation area, easily misclassifying hand movements above blank areas of the screen as valid operations. Introducing display content modal joint verification filters out a large number of invalid gestures at the logical level, significantly reducing the probability of accidental touches.
[0043] The conversion of gesture events into interaction commands is performed according to preset gesture-command mapping rules. The same gesture type can be mapped to different interaction commands in different screen scenarios. For example, a swipe gesture maps to a page scrolling command in a document browsing scenario, a fast forward / rewind command in a video playback scenario, and a focus movement command in an icon-based desktop scenario. The gesture-command mapping rules are stored in the system control unit as configuration files and can be customized according to application scenarios. The system encapsulates the hand position coordinates and target UI control information together into the interaction command, enabling the operating system or application to simultaneously know the target object and operation type of the user's intended operation.
[0044] In some embodiments, the backlight zones are assigned two types of modulation features: positioning frequency points and enhancement frequency points. Positioning frequency points are assigned to all backlight zones of the full screen for global spatial perception and two-dimensional positioning in bare-hand mode. Enhancement frequency points are used for finger detection in resonant enhancement mode. The same backlight zone simultaneously carries positioning frequency points and preset enhancement frequency points, which are isolated in the frequency domain.
[0045] Positioning frequencies are allocated to all backlight zones of the screen; there are no backlight zones without positioning frequencies. This ensures that during bare-hand mode operation, an alternating electromagnetic field carrying the positioning frequency characteristics exists in the space above any position on the screen. The sensing object in bare-hand mode is the electromagnetic disturbance directly caused by the human hand. The human hand changes the near-field electromagnetic coupling state through its dielectric and conductivity properties, generating a relatively limited disturbance signal amplitude. Relying on the globally distributed positioning frequencies, the system can collect channel signals corresponding to all zones within the entire area. Based on the temporal characteristics and spatial mapping characteristics of the disturbance deviations in each channel, it completes global spatial perception and two-dimensional hand positioning. The positioning frequencies undertake basic sensing functions. As long as the device's interactive functions are enabled normally, the modulation signal corresponding to the positioning frequency is continuously loaded, ensuring that bare-hand mode can operate independently and completely, achieving basic air gesture recognition without relying on other frequency signals.
[0046] The enhanced frequency, as a second type of modulation feature, primarily serves finger detection in the resonant enhancement mode. The resonant enhancement mode requires the cooperation of a passive resonant accessory. This accessory contains an LC resonant circuit, whose inherent resonant frequency matches the enhanced frequency. When the passive resonant accessory enters the near-field electromagnetic induction field, the LC resonant circuit picks up the electromagnetic energy corresponding to the enhanced frequency, generating a resonant effect and radiating an alternating electromagnetic field of the same frequency. This allows the channel corresponding to the enhanced frequency to generate a characteristic signal with an amplitude much higher than that of a bare hand. Relying on the characteristic signal generated by the enhanced frequency, the system can more accurately locate the finger wearing the passive resonant accessory, improving the positioning signal-to-noise ratio and recognition accuracy. The enhanced frequency does not need to cover the entire screen area; its distribution can be flexibly planned according to product design requirements. It is only necessary to ensure that the modulation signal corresponding to the enhanced frequency is placed in frequently used interactive areas.
[0047] The ability of a single backlight zone to simultaneously carry both positioning and enhancement frequency modulation features, with frequency isolation between the two types of frequencies in the frequency domain, is a core prerequisite for the stable operation of this configuration. Frequency domain isolation means that a sufficient frequency gap is set between the frequency band where the positioning frequency is located and the frequency band where the enhancement frequency is located, and there is no spectral overlap between the two. After the driving circuit superimposes the modulation components corresponding to the two types of frequencies onto the backlight zone driving current, the LED alternating current will synchronously radiate electromagnetic fields of both frequencies. After the electromagnetic induction coil collects the superimposed electromagnetic signal, the signal processing module can use filtering and spectrum analysis to separate the positioning frequency component and the enhancement frequency component, forming the channel signal corresponding to the positioning frequency and the channel signal corresponding to the enhancement frequency, respectively. The two types of channel signals participate in the calculation independently without feature confusion.
[0048] This frequency allocation configuration brings multiple technical benefits. First, a single backlight hardware setup supports both bare-hand sensing and resonance enhancement modes, making the product more flexible. Users can interact directly with their bare hands without accessories, and automatically switch to the high signal-to-noise ratio resonance enhancement recognition mode when wearing a passive resonant ring. Second, the two frequency types each have their own function: the positioning frequency ensures basic interactive functions are not lacking, while the enhancement frequency is responsible for achieving high-precision enhanced sensing, with clear functional boundaries. Third, the same partition can reuse both types of frequency points, eliminating the need for additional independent backlight partitions or independent drive channels, fully utilizing existing hardware resources, and not increasing the overall hardware cost or structural design complexity. Fourth, the frequency domain isolation constraint eliminates crosstalk between the two types of modulation signals from the ground up. The positioning channel signal used for bare-hand recognition will not be interfered with by the resonance signal of the enhancement frequency, and the enhancement frequency signal collected in the resonance enhancement mode will not affect the analysis of the basic bare-hand disturbance signal. The two sensing modes can be enabled independently or simultaneously collected to achieve data fusion.
[0049] During device operation, the signal processing module continuously and in parallel analyzes the positioning frequency channel signal and the enhancement frequency channel signal. The system relies on the positioning frequency channel signal to continuously monitor whether there is hand disturbance across the entire screen area, completing basic wake-up determination and coarse positioning; when the enhancement frequency channel shows a large characteristic disturbance, it is determined that a passive resonant component has entered the sensing area, and the system retrieves the channel signal corresponding to the enhancement frequency to perform refined finger detection.
[0050] In some embodiments, the frequency of the enhanced frequency point matches the resonant frequency of a preset passive resonant ring. When the passive resonant ring enters the near-field electromagnetic induction field, the LC resonant circuit of the passive resonant ring picks up the electromagnetic energy of the corresponding frequency and generates resonance, radiating an electromagnetic field of the same frequency outward, thereby increasing the disturbance amplitude of the channel signal corresponding to the preset enhanced frequency point. When determining a gesture event, hand positioning is preferentially performed based on the disturbance of the channel signal corresponding to the preset enhanced frequency point. The processing flow of the enhanced frequency point is as follows. Figure 3 As shown.
[0051] Matching the frequency of the boosted frequency with the resonant frequency of the LC resonant circuit inside the passive resonant ring is a fundamental condition for the entire collaborative sensing mechanism to function effectively. Frequency matching means that the frequency of the alternating electromagnetic field corresponding to the boosted frequency must be consistent with the inherent resonant frequency of the LC circuit. Only when this condition is met can the LC resonant circuit generate a resonant effect. If there is a significant frequency deviation, the circuit cannot form an effective resonant oscillation, the ring cannot radiate the boosted electromagnetic field, and therefore cannot achieve the signal amplification effect. The passive resonant ring does not have the ability to actively transmit signals; all its working energy is taken from the near-field electromagnetic induction field constructed by the backlight partition in front of the screen. The LC resonant circuit continuously picks up the electromagnetic energy corresponding to the boosted frequency, and after the energy accumulates, it excites a stable resonant oscillation. The resonant oscillation drives the circuit coil to radiate an alternating electromagnetic field of the same frequency outward.
[0052] The electromagnetic field radiated by the ring is superimposed on the near-field electromagnetic induction field originally generated by the backlight partition, changing the electromagnetic coupling state within the frequency band corresponding to the enhanced frequency point. After the electromagnetic induction coil collects the superimposed electromagnetic signal, the signal processing module separates the channel signal corresponding to the enhanced frequency point. Compared to the signal change caused by direct perturbation with a bare hand, the signal fluctuation amplitude brought about by the resonance effect is significantly increased, that is, the perturbation amplitude of the channel signal is enhanced. The increase in perturbation amplitude can effectively improve the signal-to-noise ratio, reduce recognition interference caused by environmental noise and circuit noise, extend the effective interaction distance, and improve the stability of positioning. Bare hand perturbation relies on the dielectric properties of the human body to weakly change the distribution of the electromagnetic field, and the upper limit of the perturbation amplitude is low, making it susceptible to interference from metal supports and environmental clutter. The resonance enhancement mechanism can physically widen the amplitude difference between the effective signal and noise.
[0053] During the phase of simultaneously acquiring the positioning frequency channel signal and the enhanced frequency channel signal, and performing gesture event analysis, the system sets a data usage priority rule, prioritizing hand positioning based on disturbances in the channel signal corresponding to the preset enhanced frequency. This priority rule leverages the high signal-to-noise ratio of the resonant enhanced signal. When the passive resonant ring is within the near-field electromagnetic induction field, the finger position information carried by the enhanced frequency channel is more accurate, and the signal fluctuation characteristics are more easily identified. The positioning frequency channel signal continues to be acquired and processed, serving as auxiliary reference data. Once the enhanced frequency signal weakens below a threshold, the system can smoothly switch to relying on the positioning frequency for sensing.
[0054] The backlight partition continuously loads modulation features with positioning and enhancement frequencies. An electromagnetic induction coil continuously collects superimposed electromagnetic signals across the entire area. The signal processing module separates multiple independent channel signals based on different frequency characteristics. The system continuously calculates the perturbation deviation of each channel signal relative to the reference signal. When a significant perturbation is detected in the enhancement frequency channel, a passive resonant ring is determined to exist within the sensing range. During gesture event parsing, the enhancement frequency channel perturbation data is used as the primary positioning basis. Combined with channel timing features and backlight partition spatial mapping features, the hand position and gesture type are obtained. Subsequently, the display content modal data is retrieved to perform multimodal fusion verification, ultimately outputting the corresponding interaction command.
[0055] In some embodiments, the low-frequency image dimming signal and the high-frequency beacon modulation signal are in non-overlapping frequency bands. The low-frequency image dimming signal is used to achieve screen brightness and grayscale adjustment, and the frequency of the high-frequency beacon modulation signal is higher than the human eye's visual recognition cutoff frequency and the amplitude is controlled. The two signals are directly superimposed in the time domain and isolated from each other in the frequency domain, driving the backlight partition to simultaneously complete the image display and the near-field electromagnetic induction field construction.
[0056] The low-frequency image dimming signal and the high-frequency beacon modulation signal operate in non-overlapping frequency bands, with no spectral overlap in the frequency domain. This is a prerequisite for the two signals to function independently without mutual distortion or interference. The low-frequency image dimming signal operates in a low-frequency range adapted to backlight brightness control; changes in its signal level directly correspond to adjustments in backlight intensity, primarily for adjusting screen brightness and grayscale. The terminal relies on this signal to control the illumination state of each backlight zone, rendering various screen content such as images, videos, and user interfaces to meet normal display requirements.
[0057] The high-frequency beacon modulation signal is selected in a frequency band higher than the human visual recognition cutoff frequency, making rapid brightness fluctuations at this frequency imperceptible to the human visual system. Simultaneously, the amplitude of the high-frequency beacon modulation signal is controlled; the system continuously constrains the upper limit of this component's amplitude in the composite drive signal, preventing large fluctuations in backlight brightness caused by the modulation component, eliminating perceptible flickering and brightness drift, and ensuring that the viewing experience is unaffected by sensory perception. The high-frequency beacon modulation signal carries preset modulation characteristics; the alternating current causes the backlight zones to radiate alternating electromagnetic fields with unique identifiers, providing the physical basis for constructing a near-field electromagnetic induction field.
[0058] The direct superposition of the two signals in the time domain indicates that the high-frequency backlight driver circuit combines the two signals into a single composite drive current before inputting it to the backlight zone. Time-domain superposition does not change the frequency composition of the two signals themselves; it simply outputs two types of control components to the backlight zone synchronously in the same time dimension. Frequency-domain isolation means that even if the two signals are time-domain combined, they can still be completely separated after spectral decomposition, without generating spurious components caused by intermodulation interference.
[0059] The LED devices within the backlight zones respond synchronously to the control effects of two types of signals under the action of a composite driving current. The average luminous intensity of the LEDs dynamically changes with the low-frequency image dimming signal, completing the grayscale and brightness rendering of the image. The high-frequency alternating component superimposed in the current causes the LED operating current to change continuously at a high frequency and periodically, thereby exciting an alternating electromagnetic field in the near-field space in front of the screen. Combined with the unique modulation characteristics of each zone, they together form a near-field electromagnetic induction field that covers the entire area.
[0060] The backlight partition loads a composite driving signal to radiate an electromagnetic field with modulation characteristics. The electromagnetic induction coil collects the superimposed electromagnetic signals in the space. The signal processing module distinguishes the channel components corresponding to different modulation characteristics based on frequency domain differences, continuously calculates the disturbance deviation of the channel signal relative to the reference signal, recognizes gesture events by combining timing characteristics and spatial mapping characteristics, and then performs multimodal fusion verification and outputs interactive commands by combining display content modal data.
[0061] This embodiment offers several technical advantages. Frequency domain isolation fundamentally prevents interference between the two signals, ensuring that the image display function does not damage the electromagnetic beacon characteristics, and that electromagnetic beacon modulation does not distort the image dimming effect. The frequency and amplitude constraints of the high-frequency beacon modulation signal guarantee that the added sensing function will not sacrifice display quality. The time-domain overlay implementation fully reuses existing drive channels, eliminating the need for separate drive hardware for image display and electromagnetic beacon, thus controlling overall hardware costs. Both functions operate synchronously, without the need for time-sharing between display and sensing modes; gesture recognition can continue continuously alongside video playback, eliminating any perception interruption.
[0062] In some embodiments, the acquisition of the preset reference signal includes: acquiring the channel signal corresponding to each backlight partition as the initial reference signal in the absence of hand interference; monitoring the constant offset component of each channel signal in real time, and determining the interference of the constant offset that persists for a long time; if the offset area has no dynamic fluctuation characteristics, it is determined to be static environmental interference and the reference signal is updated adaptively; if the offset area has weak fluctuations, it is determined to be static human interference and the reference signal is not updated.
[0063] The preset reference signal serves as a unified standard for the system to determine whether a valid disturbance has occurred in the channel signal. The value of the reference signal directly determines the calculation result of the disturbance deviation, and the accuracy of the reference signal directly affects the reliability of gesture event recognition. The acquisition of the initial reference signal requires the prerequisite of no hand interference. After the device powers on and completes initialization, the backlight partitions are properly loaded with modulation features, and the near-field electromagnetic induction field is stably established, the system continuously acquires the channel signals corresponding to all backlight partitions over a certain period. After removing transient noise, the steady-state average is calculated, and this set of steady-state channel signals is saved as the initial reference signal. The initial reference signal incorporates the electromagnetic influence from fixed objects such as metal supports and metal desktops that are already present at the time of device startup, eliminating the inherent offset caused by static objects in the initial environment and ensuring a reliable reference for disturbance detection when the device is first started.
[0064] After the device enters normal interactive working state, the system continuously monitors the constant offset component within each channel signal in real time. The constant offset component differs from instantaneous fluctuations; it is characterized by a sustained and stable deviation of the channel signal from the initial reference signal, with the offset amplitude remaining at a fixed value over a long period without rapid fluctuations. The system performs interference type determination for persistent constant offsets, with the core criterion being the presence of dynamic fluctuation characteristics in the offset coverage area. Dynamic fluctuation characteristics refer to small-amplitude periodic fluctuations or slow drift changes in the amplitude and phase of the channel signal in the corresponding area.
[0065] When the monitored area corresponding to a constant offset shows no dynamic fluctuations, it indicates that the object causing the offset is not undergoing any minute movement. This object is a static environmental disturbance, such as a metal component, a wall, or a fixed non-metallic medium. Static environmental disturbances continuously alter the local electromagnetic environment. If the reference signal is not corrected over a long period, subsequent disturbances caused by a hand approaching the area will be partially canceled out, reducing the detection sensitivity. Under these conditions, the system adaptively updates the reference signal of the corresponding channel, incorporating the electromagnetic influence of newly added static objects into the reference system, and continuously maintaining the sensitivity of disturbance detection.
[0066] When a weak fluctuation is detected in the area corresponding to a constant offset, the source of interference is determined to be a stationary human body. Even when a human body remains stationary, breathing movements and slight limb swaying continuously alter the coupling state between the human body and the electromagnetic field, causing weak periodic fluctuations in the channel signal in the corresponding area. This characteristic is the core identifier that distinguishes a stationary human body from a solid, stationary object. In this case, the system will not perform a reference signal update operation. If the reference signal is updated, the system will recognize the electromagnetic offset caused by the human body as the new steady-state reference. Subsequent signal changes caused by gestures such as raising or sliding the hand will be smaller relative to the reference, potentially leading to the system being unable to recognize the gesture event and resulting in interaction failure.
[0067] In some embodiments, when determining a gesture event, the signal strength of a preset enhancement frequency point is evaluated: when the signal of the enhancement frequency point is higher than a preset high threshold, a resonant enhancement mode is adopted, and the hand position is determined by the relationship between resonant coupling strength and distance; when the signal of the enhancement frequency point is lower than a preset low threshold, a bare-hand perception mode is adopted to determine the hand position; when the signal of the preset enhancement frequency point is between the preset high threshold and the preset low threshold, the hand position is determined by fusing the results of the resonant enhancement mode and the bare-hand perception mode.
[0068] During the process of parsing channel signals and determining gesture events, the channel signal strength corresponding to the preset enhanced frequency point is continuously evaluated in real time. The signal strength can be selected as the channel signal amplitude or the effective power of the signal as a quantification indicator. The system pre-defines two sets of judgment thresholds: a high threshold and a low threshold. The two sets of thresholds are obtained through calibration based on the receiving sensitivity of the whole machine's electromagnetic induction coil, the distribution range of the near-field electromagnetic induction field, and the gain characteristics of the passive resonant ring resonator. The high threshold value is greater than the low threshold value, and a transition range is formed between the two thresholds.
[0069] When the signal strength corresponding to the preset enhanced frequency point is higher than the preset high threshold, it indicates that the passive resonant ring is close to the screen, and the ring is in a region with excellent electromagnetic field coupling. The resonance effect is fully utilized, and the enhanced frequency channel possesses signal characteristics with higher signal-to-noise ratio and stronger positional resolution. At this time, the system independently activates the resonance enhancement mode to solve for the hand position. The resonance enhancement mode calculates the distance based on the monotonic relationship between the LC loop resonant coupling strength and the distance between the ring and the screen. It also uses the partition number of the enhanced frequency channel disturbance and spatial mapping characteristics to lock the two-dimensional coordinates. The resonant coupling strength decreases smoothly as the distance increases. This variation can be used to construct a dedicated distance estimation model, which can obtain more accurate positional results compared to the bare-hand sensing mode.
[0070] When the signal strength corresponding to the preset enhancement frequency point is lower than the preset low threshold, there are two typical scenarios. The first scenario is that the user is not wearing a passive resonant ring, and there is no characteristic signal generated by resonance in the space. The second scenario is that the finger wearing the passive resonant ring is far from the screen, the resonant coupling effect is weak, and the resonant characteristic signal of the enhancement frequency point channel is submerged by environmental noise, making it impossible to stably extract effective resonant information. The system determines that it is not suitable to rely on the resonant signal to deduce the position and automatically switches to the bare-hand perception mode. The bare-hand perception mode relies on the channel signal corresponding to the positioning frequency point for calculation. Based on the time-series characteristics of the channel disturbance deviations caused by hand disturbances and the amplitude gradient of disturbances in adjacent partitions, it solves for the two-dimensional position and vertical distance of the hand, ensuring the continued availability of basic gesture recognition functions in accessory-free or long-distance scenarios.
[0071] When the signal strength value corresponding to the preset enhancement frequency point is in the transition range between the high and low thresholds, it indicates that the passive resonant ring is in a medium-distance region. The resonant signal has some effective information, but the gain is insufficient. Relying solely on the resonant enhancement mode or the bare-hand sensing mode can easily lead to increased errors. The system does not directly select a single operating mode but runs two position calculation models simultaneously, obtaining the hand position results from the resonant enhancement mode and the bare-hand sensing mode, respectively. Dynamic weights are then set based on the relative proportion of the enhancement frequency signal strength within the threshold range, and the two sets of position results are weighted and fused to obtain the final hand position. The closer the enhancement frequency signal is to the high threshold, the higher the weight of the resonant enhancement mode calculation result; the closer the enhancement frequency signal is to the low threshold, the higher the weight of the bare-hand sensing mode result. This weighted fusion method ensures a continuous and gradual change in position output, eliminating abrupt errors at mode switching boundaries.
[0072] The backlight partition continuously loads the modulation features corresponding to the positioning frequency and the enhancement frequency. The electromagnetic induction coil continuously collects and superimposes electromagnetic signals. The signal processing module completes frequency domain splitting to obtain the positioning frequency channel signal and the enhancement frequency channel signal respectively. The system combines the reference signal to calculate the disturbance deviation corresponding to each channel. During the gesture event parsing stage, the threshold judgment logic of this embodiment is executed. The system selects a single mode or fuses the results of two modes to obtain the accurate hand position. At the same time, the gesture type is identified by combining the timing features to form a complete gesture event. Subsequently, the display content modal data is retrieved to perform multimodal fusion verification. After the verification is passed, a standardized interaction command is output.
[0073] Two threshold levels are used to classify three operating conditions, enabling automatic adaptation of the two sensing modes to different scenarios. Users do not need to manually switch modes when wearing or removing the ring, or when the distance between their fingers changes. A result fusion strategy is used in the transition zone, abandoning a simple hard-switching mechanism to solve the problems of jitter and coordinate jumps in boundary position recognition, improving the smoothness of air-to-air interaction. The two sensing modes each perform their respective functions: in close-range ring wearing scenarios, the high-precision advantage of the resonance enhancement mode is fully utilized; in scenarios without accessories or at a distance, basic interaction capabilities are maintained by the bare-hand mode, expanding the product's applicable scenarios. Both modes reuse the same backlight emission hardware and the same set of electromagnetic induction acquisition hardware, requiring no hardware architecture modifications. Mode adaptation is achieved solely through algorithmic strategies, controlling hardware upgrade costs. Furthermore, the judgment logic continuously monitors the positioning frequency channel and the enhancement frequency channel in parallel. Effective disturbances in either channel can be captured by the system, eliminating sensing blind spots and ensuring the continuity of interactive recognition.
[0074] In some embodiments, determining the gesture type includes: determining the motion state of the hand relative to the screen based on the changing direction and timing of the perturbation deviation of the channel signals corresponding to different backlight partitions; wherein, if the perturbation deviation of the channel signals corresponding to multiple backlight partitions increases or decreases synchronously, it is determined that the hand is approaching or moving away from the screen; if the perturbation deviation of the channel signals corresponding to backlight partitions with adjacent positional relationships changes sequentially, it is determined that the hand is performing a sweeping motion, and the sweeping direction is determined based on the sequentially changing direction.
[0075] After continuously acquiring the channel signals corresponding to each backlight zone, the system calculates the disturbance deviation of each channel signal relative to a preset reference signal in real time. The disturbance deviation represents the degree of deviation of the current electromagnetic state from a steady-state environment without interaction. The system continuously records the dynamic evolution of the channel disturbance deviation of all backlight zones, extracting two core features: the direction of change and the timing of change. The direction of change refers to whether the disturbance deviation continues to increase or decrease, reflecting whether the local near-field electromagnetic disturbance continues to strengthen or weaken. The timing of change refers to the order in which the disturbance deviation begins to change, the duration of the change, and the order in which the disturbance is transmitted from one zone to adjacent zones. When the hand performs different movements, it changes the electromagnetic coupling conditions in different areas of the near-field space, causing the channel disturbance deviation of the corresponding zone to exhibit differentiated timing evolution patterns. This is the physical basis for distinguishing gesture types based on the disturbance deviation characteristics.
[0076] When the channel disturbance deviations corresponding to multiple backlight zones increase or decrease synchronously, the system determines that the hand is moving closer to or further away from the screen. When the hand moves from a distance towards the screen, the area where the hand couples with the electromagnetic field continuously expands, and the electromagnetic disturbances within this large area synchronously increase, corresponding to a synchronous increase in the channel disturbance deviations of multiple zones, indicating that the hand is approaching the screen. When the hand moves from near to far from the screen, the electromagnetic coupling effect caused by the hand gradually weakens, and the channel disturbance deviations of multiple zones synchronously decrease, indicating that the hand is moving away from the screen. The most crucial characteristic of this action is that the disturbance changes are globally synchronous; there is no obvious order of disturbances between adjacent zones, making it clearly distinguishable from sequentially changing sweeping actions.
[0077] When the channel perturbation deviations corresponding to adjacent backlight zones change sequentially, the system determines that the hand is performing a sweeping motion and determines the sweeping direction based on the order in which the perturbations appear. As the hand slides along a direction parallel to the screen, the area of the hand's action continuously moves, and the perturbation deviations of the backlight zone channels corresponding to the hand's path will change sequentially. If the hand slides from left to right, the perturbation deviation of the left-hand zone channel increases first, followed by the adjacent right-hand zone channel. The sweeping direction is directly determined based on the left-to-right sequence of perturbation appearance. The sequential change in perturbation deviations between adjacent zones is a unique signal characteristic of the sliding sweeping motion, distinguishing it from synchronously changing near-far movements.
[0078] A near-field electromagnetic induction field is constructed by loading modulation features onto the backlight zones. Electromagnetic induction coils collect and superimpose electromagnetic signals. The signal processing module decomposes the signals into channel signals for each backlight zone based on the modulation features. The system compares the real-time channel signals with a preset reference signal to calculate the disturbance deviation. Then, according to the rules of this embodiment, the system analyzes the gesture type based on the direction and timing of the change in the disturbance deviation. Simultaneously, it combines the spatial mapping features of each backlight zone to solve for the hand position. The two are integrated to form a complete gesture event. The system continues to acquire display content modal data to perform multimodal fusion verification. After successful verification, standardized interaction commands are output.
[0079] This embodiment replaces the analysis object with perturbation deviation from the original channel signal, which can eliminate the influence of uneven electromagnetic field strength in different backlight zones, enabling gesture recognition in all areas of the screen to adopt a unified standard and improving the overall recognition balance. The judgment process relies on the comprehensive judgment of continuous multi-frame perturbation deviation time-series data. Instantaneous noise cannot form continuous time-series features, thus possessing natural anti-interference capabilities. This set of judgment rules is compatible with both bare-hand sensing mode and resonance enhancement mode. In bare-hand mode, the channel perturbation deviation at the positioning frequency point is used for calculation, while in the scenario of wearing a passive resonant ring, the channel perturbation deviation at the enhancement frequency point is prioritized for analysis, and the unified algorithm framework reduces development difficulty. The judgment logic is completed based on existing channel data, making full use of the calculation results of the signal acquisition link, without the need for additional hardware and additional acquisition processes, thus controlling the overall cost.
[0080] In some embodiments, the hand position includes the two-dimensional coordinates of the hand in the screen plane and the vertical distance of the hand relative to the screen; the two-dimensional coordinates of the hand in the screen plane are determined based on the backlight partition corresponding to the channel signal with the disturbance deviation, combined with the preset spatial mapping characteristics of each backlight partition; the disturbance deviations of the channel signals corresponding to the multiple backlight partitions currently covered by the hand are obtained, and the distance of the hand relative to the screen is determined based on the degree of difference between the disturbance deviations corresponding to different backlight partitions and the magnitude of the disturbance deviations; wherein, when the partition range covered by the hand is similar, the larger the disturbance deviation and the more obvious the difference in disturbance deviations between different partitions, the closer the hand is to the screen.
[0081] This embodiment defines hand position as comprising two layers of information: the first layer is the two-dimensional coordinates of the hand within the screen plane, and the second layer is the vertical distance of the hand relative to the screen. The two-dimensional coordinates are used to pinpoint the gesture's position on the horizontal and vertical planes of the screen, matching the corresponding interactive control area within the screen interface. The vertical distance characterizes the hand's distance from the screen along the vertical direction, distinguishing between different levels such as distant hovering, medium-distance stopping, and close proximity, supporting layered interaction logic and hierarchical visual feedback. These two types of information work together to constitute complete and usable hand position parameters.
[0082] The system first determines the two-dimensional coordinates of the hand. The signal processing module continuously calculates the perturbation deviation of each channel signal relative to a preset reference signal, filters out the channel signals that generate effective perturbation deviations, and locates the backlight zone corresponding to that channel signal. The system retrieves the spatial mapping features of each backlight zone pre-fixed during the device calibration phase. These spatial mapping features record the one-to-one binding relationship between each backlight zone and the screen's physical coordinates and pixel coordinates. Based on the zone number where the perturbation occurred, the system performs coordinate conversion using the spatial mapping features to obtain the two-dimensional coordinates of the hand on the screen plane. If the perturbation occurs simultaneously in multiple adjacent backlight zones, the system further optimizes the accuracy of the two-dimensional coordinates by combining the centroids of the perturbation distribution across multiple zones.
[0083] The system then calculates the vertical distance between the hand and the screen. It filters out the channel disturbance deviations corresponding to all backlight zones within the hand's current coverage area, extracting two core reference indicators: the overall magnitude of the disturbance deviation and the degree of difference in disturbance deviations between different backlight zones. The human hand has a fixed physical size. When the hand is close to the screen, the coupling effect of the hand on the electromagnetic field is concentrated in a small, localized space. The disturbance deviation of the backlight zone corresponding to the center of the hand reaches a high value, while the coupling effect on adjacent zones farther from the center attenuates rapidly, directly causing a significant difference in disturbance deviation between the central and surrounding zones. As the hand moves further away from the screen, the electromagnetic disturbances it generates spread to a wider area, the overall value of the disturbance deviation continuously decreases, and the difference in disturbance deviations between different zones narrows accordingly.
[0084] The judgment rule sets a pre-constraint condition, namely, that the area covered by the hand is similar. This constraint is used to eliminate interference caused by changes in hand posture, different finger opening degrees, and changes in the size of the hand's planar projection, ensuring the stability of the distance judgment rule. Under the premise of satisfying this constraint, the larger the overall value of the disturbance deviation, and the more obvious the difference in disturbance deviation between different areas, the more concentrated the electromagnetic disturbance is in a local area, indicating that the hand is judged to be closer to the screen. Conversely, if the overall disturbance deviation is small and the difference in disturbance deviation between areas is weak, the hand is judged to be farther from the screen. Using both disturbance amplitude and the degree of difference between areas as dual indicators for judgment can avoid misjudgment caused by relying solely on the amplitude of the disturbance deviation. When relying solely on amplitude for distance recognition, large-scale static environmental disturbances and fluctuations in environmental noise can interfere with the judgment results. Introducing the degree of difference in disturbance deviation between areas can effectively suppress such errors and improve the stability of vertical distance recognition.
[0085] The backlight partition continuously loads modulation features to construct a near-field electromagnetic induction field. Electromagnetic induction coils collect and superimpose electromagnetic signals. The signal processing module separates the channel signals corresponding to each backlight partition based on the modulation features. The system compares the real-time channel signals with a preset reference signal to calculate the disturbance deviation. Then, following the rules of this embodiment, it sequentially solves for the two-dimensional coordinates and vertical distance to form a complete hand position. The system combines the direction and timing of the disturbance deviation change to identify the gesture type, integrating the gesture type and hand position to generate a gesture event. Subsequently, it continues to retrieve display content modal data for multimodal fusion verification. After successful verification, the corresponding interaction command is output.
[0086] This embodiment uses perturbation deviation as a unified analysis object to avoid the reference deviation caused by the inherent inconsistency of electromagnetic field strength in different backlight zones, ensuring that the standard for solving the position of each area of the screen remains consistent. Two-dimensional coordinates are solved based on pre-calibrated spatial mapping features, resulting in a simple algorithm structure that facilitates real-time hardware computation. Vertical distance is identified by the distribution characteristics of the perturbation deviation across the distributed multi-zone area, eliminating the need for an additional depth sensor and fully leveraging the spatial resolution capabilities of the multi-zone backlight array. The entire set of solution rules can be adapted to both bare-hand sensing mode and resonant enhancement mode. In bare-hand mode, the positioning frequency channel perturbation deviation is used for calculation; when wearing a passive resonant ring, the enhanced frequency channel perturbation deviation is prioritized for position determination. The same algorithm framework is compatible with both working modes. The combined output of two-dimensional coordinates and vertical distance enriches the data dimensions of gesture events. The system can set differentiated interaction thresholds based on distance levels to achieve layered interaction functions such as long-distance wake-up and close-distance confirmation, while also supporting graded backlight feedback strategies at different vertical distances.
[0087] A display terminal interaction system based on multimodal fusion control, the system modules are shown in the diagram below. Figure 4 As shown, the system includes: The partition configuration unit A1 is used to pre-establish multiple backlight partitions for the display terminal and assign modulation features to each backlight partition, thereby identifying the source of the electromagnetic field radiated by each backlight partition through the modulation features; The electromagnetic construction unit A2 is used to superimpose the low-frequency image dimming signal and the high-frequency beacon modulation signal through a preset driving circuit to drive each backlight zone, so that each backlight zone can establish a near-field electromagnetic induction field with various modulation characteristics in the near-field space in front of the screen while displaying the picture normally. Electromagnetic induction unit A3 is used to continuously collect the superimposed electromagnetic signal formed by the superposition of electromagnetic fields radiated by each backlight partition in the near-field electromagnetic induction field through a preset electromagnetic induction coil, and extract the channel signal corresponding to each backlight partition from the superimposed electromagnetic signal according to the modulation characteristics. The gesture acquisition unit A4 is used to calculate the disturbance deviation between each channel signal and the preset reference signal in real time, and to determine the gesture event, including gesture type and hand position, based on the temporal change characteristics of the disturbance deviation on the corresponding channel signal of each backlight partition and the preset spatial mapping characteristics of each backlight partition. The interaction matching unit A5 is used to acquire display content modal data, perform multimodal fusion verification on the gesture event and the content modal data, and output the interaction command corresponding to the gesture event based on the verification result.
[0088] A display, characterized by a display terminal interaction method based on multimodal fusion control for implementing any of the above.
[0089] In some embodiments, the display module is illustrated in the appendix. Figure 5 As shown, the display includes: The multi-zone backlight module 1 includes several independently controllable backlight zones, each backlight zone is arranged in a rectangular array and closely spliced together, and each backlight zone has an independent electrical drive path. The high-frequency backlight driving circuit 2 is electrically connected to the multi-zone backlight module and is used to independently load a driving current containing modulation features for each backlight zone. The driving current is formed by superimposing a low-frequency image dimming signal and a high-frequency beacon modulation signal. The modulation features are used to identify the source of the electromagnetic field radiated by each backlight zone. At least one set of electromagnetic induction coils 3 are arranged inside the display terminal to collect the superimposed electromagnetic signals formed by the superposition of electromagnetic fields radiated by each backlight zone in the near field space in front of the screen. The signal acquisition and processing module 4 is electrically connected to the electromagnetic induction coil and is used to receive superimposed electromagnetic signals and separate the channel signals corresponding to each backlight zone from the superimposed electromagnetic signals according to the preset modulation characteristics of each backlight zone. System control module 5, connected to the high-frequency backlight driving circuit and the signal acquisition and processing module respectively, is used to perform the following operations: during the initialization phase, it assigns modulation features to each backlight zone and establishes a reference spectrum for each channel signal under the condition of no hand interference; during the working phase, it determines the gesture event based on the disturbance deviation of each channel signal relative to the reference spectrum, acquires the display content modal data of the current screen, performs multimodal fusion verification on the gesture event and the display content modal data, and outputs the interaction command corresponding to the gesture event based on the verification result.
[0090] Those skilled in the art will understand that the components of this application described above can be implemented using general-purpose computing systems. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage system for execution by the computing system. Alternatively, they can be fabricated as separate integrated circuit components, or multiple components or steps can be fabricated as a single integrated circuit component. Thus, this application is not limited to any particular combination of hardware and software.
[0091] Note that the above description is merely a preferred embodiment and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments. Many other equivalent embodiments may be included without departing from the concept of this application, and the scope of this application is determined by the scope of the appended claims.
[0092] The above disclosures are only a few specific implementation scenarios of this application. However, this application is not limited to these. Any variations that can be conceived by those skilled in the art should fall within the protection scope of this application.
Claims
1. A display terminal interaction method based on multi-modal fusion control, characterized in that, include: Multiple backlight zones are pre-established for the display terminal, and modulation features are assigned to each backlight zone. The source of the electromagnetic field radiated by each backlight zone is identified by the modulation features. By superimposing low-frequency image dimming signals and high-frequency beacon modulation signals through a preset driving circuit, each backlight zone is driven, so that each backlight zone establishes a near-field electromagnetic induction field with various modulation characteristics in the near-field space in front of the screen while displaying the image normally. The superimposed electromagnetic signal formed by the superposition of electromagnetic fields radiated by each backlight partition in the near-field electromagnetic induction field is continuously collected by a preset electromagnetic induction coil, and the channel signal corresponding to each backlight partition is extracted from the superimposed electromagnetic signal according to the modulation characteristics. The disturbance deviation between each channel signal and the preset reference signal is calculated in real time. Based on the temporal variation characteristics of the disturbance deviation on the corresponding channel signal of each backlight partition and the preset spatial mapping characteristics of each backlight partition, the gesture event, including gesture type and hand position, is determined. Acquire display content modal data, perform multimodal fusion verification on the gesture event and the content modal data, and output the interaction command corresponding to the gesture event based on the verification result.
2. The display terminal interaction method according to claim 1, characterized by, The backlight zones are assigned two types of modulation features: positioning frequency points and enhancement frequency points; the positioning frequency points are assigned to all backlight zones of the screen for global spatial perception and two-dimensional positioning in bare-hand mode. The enhanced frequency point is used for finger detection in the resonant enhancement mode; The same backlight zone simultaneously carries the positioning frequency point and the preset enhancement frequency point, which are isolated in the frequency domain.
3. The display terminal interaction method according to claim 2, characterized in that, The frequency of the enhanced frequency point is matched with the resonant frequency of the preset passive resonant ring; When the passive resonant ring enters the near-field electromagnetic induction field, the LC resonant circuit of the passive resonant ring picks up the electromagnetic energy of the corresponding frequency and generates resonance, radiating an electromagnetic field of the same frequency outward, thereby increasing the disturbance amplitude of the channel signal corresponding to the preset enhancement frequency point. When determining the gesture event, hand positioning is preferentially performed based on the disturbance of the channel signal corresponding to the preset enhanced frequency point.
4. The display terminal interaction method of claim 1, wherein, The low-frequency image dimming signal and the high-frequency beacon modulation signal are in non-overlapping frequency bands. The low-frequency image dimming signal is used to adjust the screen brightness and grayscale. The frequency of the high-frequency beacon modulation signal is higher than the human eye's visual recognition cutoff frequency and its amplitude is controlled. The two signals are directly superimposed in the time domain and isolated from each other in the frequency domain, driving the backlight partition to simultaneously complete the image display and the near-field electromagnetic induction field construction.
5. The display terminal interaction method of claim 1, wherein, The acquisition of the preset reference signal includes: In the absence of hand interference, the channel signals corresponding to each backlight zone are collected as the initial reference signals; the constant offset components of each channel signal are monitored in real time, and interference is determined for constant offsets that persist for a long time; if there are no dynamic fluctuations in the offset area, it is determined to be static environmental interference and the reference signal is updated adaptively; if there are weak fluctuations in the offset area, it is determined to be static human interference and the reference signal is not updated.
6. The display terminal interaction method of claim 2, wherein, When determining the gesture event, the signal strength of the preset enhanced frequency point is evaluated: When the signal at the enhanced frequency point is higher than a preset high threshold, a resonant enhancement mode is adopted, and the hand position is determined by the relationship between resonant coupling strength and distance; when the signal at the enhanced frequency point is lower than a preset low threshold, a bare-hand perception mode is adopted to determine the hand position; when the signal at the preset enhanced frequency point is between a preset high threshold and a preset low threshold, the hand position is determined by fusing the results of the resonant enhancement mode and the bare-hand perception mode.
7. The display terminal interaction method of claim 1, wherein, Determining the gesture type includes: judging the movement state of the hand relative to the screen based on the direction and timing of the change in the disturbance deviation of the channel signal corresponding to different backlight zones; wherein, if the disturbance deviation of the channel signal corresponding to multiple backlight zones increases or decreases synchronously, it is determined that the hand is approaching or moving away from the screen; if the disturbance deviation of the channel signal corresponding to the backlight zones with adjacent positional relationships changes sequentially, it is determined that the hand is performing a sweeping motion, and the sweeping direction is determined based on the sequentially changing direction.
8. The display terminal interaction method of claim 1, wherein, The hand position includes the two-dimensional coordinates of the hand in the screen plane and the vertical distance of the hand relative to the screen; Based on the backlight partition corresponding to the channel signal that caused the disturbance deviation, and combined with the preset spatial mapping characteristics of each backlight partition, the two-dimensional coordinates of the hand in the screen plane are determined. The perturbation deviations of the channel signals corresponding to the multiple backlight zones currently covered by the hand are obtained. Based on the degree of difference between the perturbation deviations corresponding to different backlight zones and the magnitude of the perturbation deviations, the distance between the hand and the screen is determined. Among them, when the range of zones covered by the hand is similar, the larger the perturbation deviation and the more obvious the difference in perturbation deviations between different zones, the closer the hand is to the screen.
9. A display terminal interaction system based on multi-modal fusion control, characterized in that, include: A partition configuration unit is used to pre-establish multiple backlight partitions for the display terminal and assign modulation features to each backlight partition, thereby identifying the source of the electromagnetic field radiated by each backlight partition through the modulation features; The electromagnetic construction unit is used to superimpose low-frequency image dimming signals and high-frequency beacon modulation signals through a preset driving circuit to drive each backlight zone, so that each backlight zone can establish a near-field electromagnetic induction field with various modulation characteristics in the near-field space in front of the screen while displaying the picture normally. An electromagnetic induction unit is used to continuously collect the superimposed electromagnetic signal formed by the superposition of electromagnetic fields radiated by each backlight partition in the near-field electromagnetic induction field through a preset electromagnetic induction coil, and extract the channel signal corresponding to each backlight partition from the superimposed electromagnetic signal according to the modulation characteristics. The gesture acquisition unit is used to calculate the disturbance deviation between each channel signal and the preset reference signal in real time, and to determine the gesture event, including gesture type and hand position, based on the temporal change characteristics of the disturbance deviation on the corresponding channel signal of each backlight partition and the preset spatial mapping characteristics of each backlight partition. The interaction matching unit is used to acquire display content modal data, perform multimodal fusion verification on the gesture event and the content modal data, and output the interaction command corresponding to the gesture event based on the verification result.
10. A display, characterized by This method is used to implement the display terminal interaction method based on multimodal fusion control as described in any one of claims 1-8.
