Multi-modal fusion AR glasses intelligent control method and system

Abstract

The invention provides a multi-modal fusion AR glasses intelligent control method and system, and relates to the technical field of artificial intelligence and AR glasses application, and the method comprises the steps: collecting an electroencephalogram signal, an eye movement signal, a heart rate signal and a skin electrical signal of a user, and building a user physiological feature database; the interaction intention of the user is recognized by comparing with the current real-time physiological feature data of the user; an interaction control instruction and content presentation parameters are generated in combination with the sight focus position to adjust spatial distribution of AR glasses display content; according to the visual fatigue degree and the real-time physiological feature data of the user, visual comfort parameters and physiological state parameters are generated; and adjusting optical display parameters of the AR glasses according to a weighting result. By implementing the method, the user can realize natural and stable interaction control of the AR glasses even in a special working environment in which both hands cannot operate and voice interaction is limited, and the practicability of the system in a complex scene is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and AR glasses application technology, and in particular to a multi-modal fusion AR glasses intelligent control method and system. Background Art

[0002] With the rapid development of augmented reality technology, AR glasses are changing how people work and live. In fields such as industrial manufacturing, medical surgery, and education and training, AR glasses can provide users with real-time virtual information overlays and interactive experiences, significantly improving work efficiency. Currently, AR glasses have become a key vehicle for human-computer interaction, and their application scope is continuously expanding.

[0003] In related technologies, interactive control of AR glasses primarily utilizes voice recognition and gesture recognition. By integrating voice recognition modules and gesture recognition sensors into AR glasses, the system can capture the user's voice commands and hand movements, converting them into corresponding control commands after signal processing. Simultaneously, the system projects virtual information into the user's field of view based on preset display parameters, achieving a visual presentation of the information.

[0004] However, in specialized scenarios like industrial environments, ambient noise can severely interfere with speech recognition accuracy, while hand gestures used when operating precision equipment can compromise operational stability and safety. This is especially true in situations where both hands must be focused on the primary task, making it difficult for users to free their hands to interact with AR glasses. This, to a certain extent, limits the effectiveness of AR glasses in specialized work scenarios. Summary of the Invention

[0005] The present application provides a multimodal fusion AR glasses intelligent control method and system for solving the problem of how to achieve precise interactive control of AR glasses when the user cannot operate with both hands and voice interaction is limited.

[0006] In a first aspect, the present application provides an AR glasses intelligent control method based on multimodal fusion, which is applied to an AR glasses control system. The method includes: The multiple physiological sensors on the AR glasses collect the user's EEG signals, eye movement signals, heart rate signals, and skin conduction signals to establish a user physiological feature database; Comparing the physiological feature database with the user's current real-time physiological feature data through a deep learning model to identify the user's interaction intention; Generate spatially aware interactive control instructions and content presentation parameters based on the interaction intention and the user's visual focus position, and perform local image enhancement processing on the visual focus position; Generating visual comfort parameters based on real-time monitoring of the user's visual fatigue level by the mixed reality interaction engine, and generating physiological state parameters based on the real-time physiological characteristic data; According to the interactive control instructions and content presentation parameters, the spatial distribution of the content displayed on the AR glasses is adjusted in real time by a dynamic light field modulator; Adjust the optical display parameters of the AR glasses according to a weighted result of the visual comfort parameter and the physiological state parameter.

[0007] Through the above-mentioned embodiments, the AR glasses control system collects the user's multi-dimensional physiological signals through multiple physiological sensors and analyzes them in combination with a deep learning model, which can accurately identify the user's interaction intention without relying on traditional voice and gesture input. At the same time, by monitoring the user's visual fatigue and physiological state in real time, the display parameters of the AR glasses are dynamically adjusted, ensuring the accuracy of the interaction while providing a good visual experience. This method allows users to achieve natural and stable interactive control of AR glasses even in special working environments where hands cannot be used and voice interaction is limited, thereby improving the practicality of the system in complex scenarios.

[0008] In some embodiments, before the step of generating spatially-aware interaction control instructions and content presentation parameters based on the interaction intention and the user's visual focus position, the method further includes: Collecting the user's historical gaze trajectory data and historical interaction intention data to establish a gaze behavior feature database; Predicting the position information of the user's next gaze area according to the gaze focus prediction model trained with the gaze behavior feature database; The rendering data of the next gaze area is loaded in advance according to the position information.

[0009] Through the above embodiments, the AR glasses control system establishes a gaze behavior feature database and trains a gaze focus prediction model. The system can predict the user's next gaze area in advance and preload the corresponding rendering data. This predictive data processing mechanism reduces system response delays and improves the smoothness of interaction. Especially in work scenarios that require rapid switching of gaze targets, the preloading mechanism can effectively avoid screen freezes or rendering delays, ensuring that users have a coherent and natural visual experience, greatly improving the system's real-time performance and user experience.

[0010] In some embodiments, the step of performing local image enhancement processing on the sight focus position specifically includes: If it is detected that there are multiple interactive targets within the preset range of the sight focus position, obtaining spatial position data corresponding to each of the interactive targets; Calculate the priority weight of each interaction target based on the user's interaction intention strength and historical interaction frequency; The magnitude of the local image enhancement processing performed on each of the interactive targets is determined based on the priority weight.

[0011] Through the above-described embodiment, the AR glasses control system intelligently analyzes the user's interaction intent strength and historical interaction frequency, assigning different priority weights to multiple interactive targets in the field of view and adjusting the degree of local image enhancement accordingly. This not only improves the user's recognition efficiency of important interactive targets, but also avoids visual interference caused by excessive enhancement. Especially in information-intensive work scenarios, this intelligent display optimization strategy can help users locate and identify key information more quickly, significantly improving work efficiency.

[0012] In some embodiments, the step of generating physiological state parameters based on the real-time physiological characteristic data specifically includes: Extracting normal value ranges of physiological parameters under different preset working conditions from the user physiological characteristic database to generate a baseline database of personalized physiological parameters; determining a current usage condition based on the real-time physiological characteristic data, and calculating a deviation between the real-time physiological characteristic data and a normal value range corresponding to the usage condition in the baseline database; The real-time physiological characteristic data and the deviation value are input into a multimodal feature fusion model to generate physiological state parameters. The multimodal feature fusion model is used to fuse multiple physiological characteristic data to generate physiological state parameters.

[0013] Through the above-described embodiments, the AR glasses control system establishes a personalized physiological parameter baseline database, enabling the system to accurately assess the user's current physiological state based on the normal value range under different working conditions. A multimodal feature fusion model is used to comprehensively analyze various types of physiological data, which can more comprehensively and accurately reflect the user's actual state. This adaptive assessment mechanism based on a personalized baseline enables the system to promptly detect abnormal user status and make corresponding adjustments, effectively preventing physiological discomfort caused by long-term use and ensuring user comfort and work safety.

[0014] In some embodiments, before the step of inputting the real-time physiological feature data and the deviation value into the multimodal feature fusion model to generate the physiological state parameter, the method further includes: When it is detected that the real-time physiological characteristic data has missing values, a compensation value is calculated based on the data of the corresponding usage condition in the baseline database; The missing value is determined according to the compensation value.

[0015] Through the above-described embodiment, the AR glasses control system handles the problem of missing physiological characteristic data by calculating compensation values based on the baseline database, ensuring reliable status monitoring even in the event of partial loss of sensor signals. This intelligent data compensation mechanism improves the stability and reliability of the system, avoids monitoring interruptions or misjudgments caused by missing data, ensures continuous and stable operation in complex working environments, and provides users with a more reliable user experience.

[0016] In some embodiments, the step of adjusting the spatial distribution of the content displayed on the AR glasses in real time by using a dynamic light field modulator specifically includes: Obtain content data and depth information to be displayed on AR glasses; mapping the content data to a plurality of virtual focal planes according to the depth information; Determining the distance between adjacent virtual focal planes according to the depth distribution of the content data in the plurality of virtual focal planes, and calculating a transition frame between the virtual focal planes; Generating a continuous switching sequence of the plurality of virtual focal planes by adopting a time-domain interleaved display method based on the transition frames; Multi-focal plane display content is output through a dynamic light field modulator according to a weight coefficient of each virtual focal plane and the continuous switching sequence, wherein the weight coefficient is determined by the gaze depth of the user.

[0017] Through the above-described embodiments, the AR glasses control system can provide a more natural depth perception experience by mapping display content to multiple virtual focal planes and implementing intelligent depth transitions. By dynamically adjusting the weights of each focal plane based on the user's gaze depth, continuous and smooth switching of displayed content is achieved, effectively alleviating the focal plane jump problem in traditional AR displays. This multi-focal plane display technology based on time-domain interleaving improves the spatial perception quality of virtual content, reduces visual fatigue, and provides users with a more comfortable and natural mixed reality experience.

[0018] In some embodiments, after the steps of generating visual comfort parameters by monitoring the user's visual fatigue level in real time according to the mixed reality interaction engine, and generating physiological state parameters according to the real-time physiological characteristic data, the method further includes: When it is detected that the visual comfort parameter is lower than a first preset threshold, switching to an interaction mode with low visual load; When it is detected that the physiological state parameter is lower than a second preset threshold, switching to a simplified interaction mode; Establish an interaction mode evaluation table to record the visual comfort parameters and physiological state parameters before and after switching each interaction mode; Calculating the improvement rate of the interactive mode switching on the visual comfort parameter and the physiological state parameter; When it is detected that the improvement rate is less than a preset target improvement rate, increasing the trigger threshold corresponding to the interaction mode according to a preset interval; When it is detected that the improvement rate is greater than or equal to the expected target improvement rate, the trigger threshold corresponding to the interaction mode is lowered according to a preset interval.

[0019] Through the above-described embodiment, the AR glasses control system dynamically adjusts the trigger threshold for mode switching by establishing an interaction mode evaluation table. This achieves quantitative evaluation and adaptive optimization of the effects of different interaction modes, automatically adjusting system parameters based on actual usage results and continuously optimizing the user experience. Especially in scenarios of prolonged use, the system can intelligently switch between different interaction modes based on the user's actual state, preventing the accumulation of visual and physical fatigue and ensuring optimal system performance in various usage scenarios.

[0020] In a second aspect, the present application provides an AR glasses control system, the AR glasses control system comprising: one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, and the computer program code includes computer instructions. The one or more processors call the computer instructions so that the AR glasses control system can implement a multimodal fusion AR glasses intelligent control method provided in the above embodiment, which will not be repeated here.

[0021] In a third aspect, the present application provides a computer-readable storage medium comprising instructions. When the instructions are executed on an AR glasses control system, the AR glasses control system can implement a multimodal fusion AR glasses intelligent control method provided in the above embodiment, which will not be described in detail here.

[0022] Fourthly, the present application provides a computer program product. When the computer program product runs on an AR glasses control system, the AR glasses control system can implement a multimodal fusion AR glasses intelligent control method provided in the above embodiment, which will not be repeated here.

[0023] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. Multiple physiological sensors collaboratively collect the user's EEG, eye movement, heart rate, and electrodermal signals, and combine them with a deep learning model for multimodal feature fusion analysis, enabling intelligent interactive control without voice or gestures. This multimodal fusion solution based on physiological characteristics not only addresses the limitations of traditional interaction methods in special working environments, but also enables accurate identification of user interaction intent by establishing a personalized physiological parameter baseline database, significantly improving the system's applicability and reliability in complex scenarios.

[0024] 2. By real-time monitoring of the user's visual fatigue and physiological parameters, combined with gaze behavior prediction and dynamic switching of interaction modes, intelligent adjustment of system parameters is achieved. This adaptive optimization solution based on multi-dimensional feedback can promptly identify and respond to changes in user status, automatically adjusting display effects and interaction methods, effectively preventing fatigue caused by long-term use and significantly improving the system's user comfort and sustained performance.

[0025] 3. By intelligently mapping displayed content to multiple virtual focal planes and combining them with time-domain interleaved display technology, continuous depth presentation of content is achieved. This dynamic light field modulation scheme, based on the user's gaze depth, not only solves the focal plane jump problem in traditional AR displays, but also significantly improves the spatial perception quality and recognition efficiency of information through local image enhancement and priority weighting, providing users with a more natural and accurate mixed reality interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of a multimodal fusion AR glasses intelligent control method in an embodiment of the present application; Figure 2 This is another flow chart of a multimodal fusion AR glasses intelligent control method in an embodiment of the present application; Figure 3 This is a schematic diagram of the structure of a physical device of the AR glasses control system in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The terms used in the following examples of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and appended claims of this application, the singular expressions "a," "an," "said," "above," "the," and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in this application refers to any or all possible combinations comprising one or more of the listed items.

[0028] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0029] For ease of understanding, the following describes the process of the method provided by this implementation. Figure 1, which is a flow chart of a multimodal fusion AR glasses intelligent control method in an embodiment of the present application.

[0030] S101. Collect the user's EEG signals, eye movement signals, heart rate signals, and skin conduction signals through multiple physiological sensors carried by AR glasses to establish a user physiological characteristics database.

[0031] The AR glasses control system collects the user's physiological characteristic data through multiple physiological sensors. Optionally, the user's EEG signals are collected through EEG sensors, mainly including EEG activities in different frequency bands such as alpha waves, beta waves, theta waves, and delta waves; eye movement sensors collect eye movement characteristics such as the user's eye movement trajectory, gaze duration, and pupil size; heart rate sensors collect the user's heart rate variability index; and skin conductance sensors collect changes in the user's skin conductance level. The system preprocesses the various collected physiological signals, including noise removal, signal smoothing, and feature extraction. The processed multi-dimensional physiological characteristic data is then stored in the user's physiological characteristic database, and a user portrait model is established to record the distribution pattern of physiological characteristic parameters under different working conditions, providing a data basis for subsequent interaction intention recognition.

[0032] Understandably, in industrial production environments, where operators need to simultaneously operate precision instruments, the collection of these physiological signals will not interfere with their normal work. For example, during semiconductor wafer processing, operators need to observe and adjust the wafer position through a microscope. Traditional voice and gesture interaction methods can affect operational accuracy, while the collection of physiological signals can continue without affecting operations.

[0033] S102. Compare the physiological feature database with the user's current real-time physiological feature data through a deep learning model to identify the user's interaction intention.

[0034] Specifically, the deep learning model adopts a multi-layer neural network structure, including a feature extraction layer, an attention mechanism layer, and a classification layer. In the feature extraction layer, the system performs time-frequency domain analysis on physiological signals of different modalities to extract key feature parameters; in the attention mechanism layer, the system assigns weights according to the importance of different features, highlighting features that have an important impact on the judgment of interaction intentions; in the classification layer, the system matches the fused feature vector with the pre-trained interaction intention model to derive the user's most likely current interaction intention type. For example, in a medical surgery scenario, when a surgeon needs to review a patient's examination report, the system can accurately identify the interaction intention of reviewing the report by analyzing the doctor's EEG activity pattern, eye movement trajectory and other features. The system can conduct a confidence assessment of the recognition results based on the current operating environment and task context to ensure that a high recognition accuracy rate can be maintained even in high-noise and high-interference environments.

[0035] S103: Generate spatially aware interactive control instructions and content presentation parameters based on the interactive intention and the user's visual focus position, and perform local image enhancement processing on the visual focus position.

[0036] The AR glasses control system generates corresponding spatially aware interaction control commands based on the identified interaction intent and the user's gaze focus position. Specifically, the system uses an eye tracking algorithm to precisely locate the coordinates of the user's gaze point and, in combination with scene depth information, constructs a three-dimensional spatial interaction area. After determining the interaction area, the system dynamically generates corresponding control commands and display parameters based on the user's interaction intent type. Simultaneously, the system adaptively enhances the image at the gaze focus position to improve the visual performance of the user's area of interest.

[0037] For example, in industrial quality inspection scenarios, when an inspector focuses on a specific product detail, the system automatically adjusts display parameters such as image contrast and sharpness in that area based on the current interaction intent (such as measuring dimensions or identifying defects), and overlays corresponding virtual information (such as dimension annotations or defect indicators). The system can also dynamically adjust the scope and intensity of the enhancement effect based on user operating habits and scenario requirements, avoiding visual interference caused by excessive enhancement and ensuring that the enhanced image both highlights important information and maintains a natural and comfortable visual experience.

[0038] S104. Generate visual comfort parameters based on real-time monitoring of the user's visual fatigue level by the mixed reality interaction engine, and generate physiological state parameters based on real-time physiological characteristic data.

[0039] The mixed reality interaction engine uses a multi-parameter fusion approach to assess visual fatigue, including objective indicators such as blink frequency, pupil reaction sensitivity, and gaze movement speed. Specifically, the system compares these indicators with preset baseline values to generate standardized visual comfort parameters. The system can also combine real-time biometric data to assess the user's overall physiological state through a multimodal feature fusion model.

[0040] For example, in a medical surgery scenario, when a doctor performs a long minimally invasive surgery, the system can continuously monitor the doctor's visual fatigue and physiological state changes. Specifically, by analyzing changes in the doctor's blinking pattern, the system can promptly detect early signs of visual fatigue; by monitoring the changing trends of heart rate variability and skin conduction levels, the doctor's stress and attention level can be assessed. When the doctor shows obvious signs of fatigue, the system automatically adjusts the brightness, contrast, and spatial layout of the displayed content to reduce the visual burden. This real-time monitoring mechanism based on multi-dimensional physiological indicators enables the system to effectively prevent the accumulation of visual and physical and mental fatigue while ensuring the accuracy of surgical operations.

[0041] S105. According to the interactive control instructions and content presentation parameters, the spatial distribution of the content displayed by the AR glasses is adjusted in real time through the dynamic light field modulator.

[0042] After receiving the interactive control instructions and content presentation parameters, the system determines the spatial position and depth information of the display content based on these parameter information. Then, by using multi-focal plane display technology through a dynamic light field modulator, clear virtual images can be presented on different depth planes. In a specific embodiment, such as in an industrial assembly scenario, when technicians need to perform complex parts assembly, the system can dynamically adjust the spatial position of virtual guidance information according to the needs of the assembly process. For example, during the engine assembly process, the system can distribute part information of different levels on different depth planes to help technicians accurately understand the spatial relationship between parts. By precisely controlling the image clarity and brightness of each depth plane, the system ensures that virtual information is naturally integrated with the actual assembly environment. This spatial display technology based on light field modulation not only provides a more realistic depth perception experience, but also effectively reduces visual discomfort caused by focal plane mismatch, significantly improving work efficiency in complex operating environments.

[0043] S106. Adjust the optical display parameters of the AR glasses based on the weighted results of the visual comfort parameters and the physiological state parameters.

[0044] Specifically, the system uses an adaptive weighting algorithm to assign weight coefficients based on the degree of influence of different parameters on the user experience. For example, in chemical experiment simulation training, the system adjusts the display brightness and contrast of virtual drug labels according to the visual load status of the trainees; according to changes in physiological state parameters, it adjusts the update frequency and complexity of virtual operation guidance information in a timely manner. When it is detected that the trainees are distracted or the number of operational errors increases, the system automatically simplifies the display interface, highlights important information, and adjusts the spatial layout of virtual content to reduce cognitive burden. This display optimization mechanism based on multi-dimensional physiological parameters, by accurately adjusting optical parameters such as brightness, contrast, and color saturation, effectively prevents visual fatigue and operational errors caused by inappropriate display parameter settings while ensuring the display effect, providing users with a more comfortable and efficient learning experience.

[0045] In the above-mentioned embodiment, the AR glasses control system collects the user's multi-dimensional physiological signals through multiple physiological sensors and analyzes them in conjunction with a deep learning model. This allows the system to accurately identify the user's interaction intent without relying on traditional voice and gesture input. Simultaneously, by monitoring the user's visual fatigue and physiological state in real time, it dynamically adjusts the display parameters of the AR glasses, ensuring both interactive accuracy and a good visual experience. This method enables users to achieve natural and stable interactive control of AR glasses even in special working environments where hands are unable to operate and voice interaction is limited, thereby enhancing the system's practicality in complex scenarios.

[0046] The following is a more detailed description of the process of the method provided by this implementation. Figure 2 , which is another flow chart of a multimodal fusion AR glasses intelligent control method in an embodiment of the present application.

[0047] S201: Collect the user's historical gaze trajectory data and historical interaction intention data, and establish a gaze behavior feature database.

[0048] The AR glasses control system continuously tracks the user's gaze movements, recording gaze behavior characteristic data such as dwell time, movement speed, and direction. The system also associates and stores the interaction intent information generated during each interaction with the corresponding gaze behavior data. After time alignment and feature extraction, this data is constructed into a structured gaze behavior feature database. For example, in an industrial assembly scenario, the system records the technician's gaze scan path and gaze patterns of key components during different assembly steps. As technicians assemble complex mechanical equipment, the system captures their gaze movement patterns as they review assembly instructions, locate parts, and perform installation operations. By analyzing this historical data, the system can learn users' visual attention patterns and operational habits during specific tasks, providing reliable training samples for subsequent gaze behavior prediction. Each record in the gaze behavior feature database contains multi-dimensional features such as timestamp, spatial coordinates, gaze duration, and interaction type, enabling the system to perform accurate gaze behavior analysis and pattern recognition.

[0049] S202: Predict the position information of the user's next gaze area based on the gaze focus prediction model trained with the gaze behavior feature database, and load rendering data of the next gaze area in advance.

[0050] The AR glasses control system uses a deep learning algorithm to train a gaze focus prediction model based on the collected gaze behavior feature data. The gaze focus prediction model uses a recurrent neural network structure that can capture the temporal dependencies in the gaze movement sequence. After training, the model can predict in real time the area where the user is most likely to look next based on the current gaze trajectory and task context. Once the next gaze position is predicted, the system immediately starts the data preloading process for that area, including obtaining depth information, preparing rendering materials, and calculating light field parameters. This preloading mechanism significantly reduces system response delays and avoids the screen freeze problem common in traditional AR displays. At the same time, the prediction model also outputs the confidence level of the prediction results. The system can dynamically adjust the allocation of preloaded resources based on the confidence level to ensure the display effect of key areas.

[0051] This predictive data processing mechanism reduces system response latency and improves interactive fluidity. Especially in work scenarios that require rapid switching of gaze targets, the preloading mechanism effectively avoids screen freezes or rendering delays, ensuring a consistent and natural visual experience for users, significantly improving the system's real-time performance and user experience.

[0052] S203: If it is detected that there are multiple interactive targets within the preset range of the sight focus position, obtain spatial position data corresponding to each interactive target.

[0053] The AR glasses control system analyzes the scene information around the user's current line of sight focus through the onboard spatial perception equipment. Specifically, the system defines a dynamic perception range centered on the line of sight focus, which can be adaptively adjusted according to the current task type and environmental complexity. Within this range, the system uses technologies such as depth cameras and structured light scanning to obtain three-dimensional structural information of the scene and identify all potential interactive targets. For example, in an intelligent manufacturing workshop, when inspectors conduct product quality inspections, the system can simultaneously track the position information of multiple components to be inspected. For each identified interactive target, the system will establish a complete spatial description containing parameters such as spatial position, orientation angle, and size. This information provides an accurate spatial reference for subsequent image enhancement processing.

[0054] S204: Calculate the priority weight of each interaction target based on the user's interaction intention strength and historical interaction frequency, and determine the extent of local image enhancement processing for each interaction target.

[0055] The AR glasses control system uses a multi-factor weighted approach to calculate the priority of each interactive target. Specifically, the system analyzes the user's interaction intent strength, including the comprehensive performance of physiological indicators such as gaze duration, pupil reaction, and EEG activity. At the same time, the system establishes an interactive heat map by counting the interaction frequency of each interactive target in historical operations. Based on the historical interaction heat, the system sets different enhancement amplitudes for different interactive targets, realizing the differentiation of the importance of interactive targets. For higher-priority targets, the system can also dynamically adjust the duration and update frequency of the enhancement effect to ensure that key information always maintains optimal visibility.

[0056] In the above embodiment, the AR glasses control system intelligently analyzes the user's interaction intent and historical interaction frequency to assign different priority weights to multiple interaction targets in the field of view and adjust the degree of local image enhancement accordingly. This not only improves the user's recognition efficiency of important interaction targets, but also avoids visual interference caused by excessive enhancement. Especially in information-intensive work scenarios, this intelligent display optimization strategy can help users locate and identify key information more quickly, significantly improving work efficiency.

[0057] S205 , extracting normal value ranges of physiological parameters under different preset working conditions from the user physiological characteristic database to generate a baseline database of personalized physiological parameters.

[0058] The AR glasses control system establishes personalized physiological parameter baselines for different work scenarios by analyzing historical data in the user's physiological characteristics database. Specifically, the system classifies the physiological data in the user's physiological characteristics database into different working conditions, including daily office work, precision operations, high-intensity work, and other different scenarios. Statistical analysis is performed on physiological indicators such as EEG, eye movements, heart rate, and skin electricity under each working condition to establish a multi-dimensional normal value range for parameters. Optionally, the baseline database adopts an adaptive update mechanism that can dynamically adjust the parameter range according to the user's usage habits and physical condition to ensure the personalization and timeliness of the baseline data.

[0059] S206: Determine the current usage condition based on the real-time physiological characteristic data, and calculate the deviation between the real-time physiological characteristic data and the normal value range of the corresponding usage condition in the baseline database.

[0060] The AR glasses control system analyzes the user's physiological data in real time to determine the current working state and calculate parameter deviations. Specifically, the system uses multi-feature fusion to identify the working condition type, combining task information and environmental data for a comprehensive assessment. For example, in industrial quality inspection scenarios, the system can accurately identify whether the worker is in the routine inspection or precision measurement phase based on characteristics such as the worker's eye movement patterns and attention level. After determining the working condition, the system compares the real-time collected physiological data with the normal value range for the corresponding working condition in a baseline database and calculates the degree of deviation for each physiological indicator. For example, if an abnormally elevated EEG theta wave frequency or a significantly decreased heart rate variability is detected, the system can calculate a specific deviation value based on preset thresholds. These deviation values are standardized to ensure that different types of physiological parameters can be uniformly evaluated and compared.

[0061] S207: When it is detected that there are missing values in the real-time physiological characteristic data, a compensation value is calculated based on the data of the corresponding usage condition in the baseline database, and then the missing value is determined.

[0062] The AR glasses control system implements an intelligent data compensation mechanism to address missing physiological characteristic data. When certain sensor signals are temporarily interrupted or the data quality is poor, the system can initiate the data compensation process.

[0063] During implementation, the system determines the type and time period of missing data and then enters the compensation process. The specific implementation steps include: First, the system extracts historical data for the corresponding time period under the current operating conditions from the baseline database and calculates the mean and standard deviation of this type of data under these conditions; second, it analyzes valid data from adjacent time points to determine data change trends; third, combining historical statistical values and change trends, it calculates preliminary compensation values through linear or spline interpolation methods; finally, it makes corrections based on other normally collected physiological parameters to obtain the final compensation value.

[0064] S208: Input the real-time physiological feature data and the deviation value into the multimodal feature fusion model to generate physiological state parameters.

[0065] Among them, the multimodal feature fusion model is a model that integrates multiple physiological characteristic data for comprehensive analysis. It can fuse input real-time physiological characteristic data, such as EEG signals, eye movement signals, heart rate signals, and skin electrodermal signals, as well as corresponding deviation values. This model uses deep learning algorithms and, by learning from a large amount of sample data, can explore potential connections between different physiological characteristics, thereby more accurately reflecting the user's physiological state. For example, it can comprehensively consider the changes in activity in different frequency bands in EEG signals, the gaze duration and pupil size changes in eye movement signals, the fluctuations in heart rate signals, and the changes in skin electrodermal signal levels. Combined with the deviation of these data from the normal value range, it generates a parameter that can fully reflect the user's current physiological state.

[0066] In the above-mentioned embodiment, the AR glasses control system establishes a personalized physiological parameter baseline database, enabling the system to accurately assess the user's current physiological state based on the normal value range under different working conditions. A multimodal feature fusion model is used to comprehensively analyze various types of physiological data, providing a more comprehensive and accurate reflection of the user's actual state. This adaptive assessment mechanism based on a personalized baseline enables the system to promptly detect abnormalities in the user's state and make appropriate adjustments, effectively preventing physiological discomfort caused by long-term use and ensuring user comfort and work safety.

[0067] S209: Obtain content data and depth information to be displayed by the AR glasses, and map the content data to multiple virtual focal planes according to the depth information.

[0068] The AR glasses control system retrieves content data to be displayed from relevant data sources. This data can be virtual information related to the current task, such as assembly instructions for parts or virtual assembly step-by-step guides in industrial assembly scenarios. In educational scenarios, it could be virtual models or knowledge point prompts in teaching materials. The system also uses onboard depth sensors, such as structured light sensors or lidar, to obtain scene depth information, which reflects the distance between different objects in the scene and the user. Based on this depth information, the system maps the content data onto multiple virtual focal planes, simulating the effect of the human eye focusing on different planes at different distances when observing a real scene. This makes the display of virtual content more consistent with human visual habits and enhances spatial perception. For example, content closer to the user is mapped to a closer virtual focal plane, while content farther away is mapped to a farther virtual focal plane.

[0069] S210 : Determine the distance between adjacent virtual focal planes according to the depth distribution of the content data on multiple virtual focal planes, and calculate the transition frame between the virtual focal planes.

[0070] After the content data is mapped to multiple virtual focal planes, the AR glasses control system analyzes the depth distribution of these contents on different virtual focal planes. Based on the difference in depth distribution, the system determines the reasonable spacing between adjacent virtual focal planes. If the content is distributed more evenly in the depth direction, the spacing between adjacent focal planes can be set relatively consistent; if the content is more concentrated in certain depth areas, the spacing of the focal planes in the corresponding areas can be adjusted appropriately to better present the content. After determining the spacing, in order to achieve a smooth transition between virtual focal planes and avoid visual jumps, the system calculates the transition frames between each virtual focal plane. The calculation of specific transition frames can be based on the content data on adjacent focal planes, and through interpolation algorithms and other technologies, image frames in intermediate states are generated, so that when switching from one focal plane to another, the content changes more naturally and smoothly.

[0071] S211 , generating a continuous switching sequence of multiple virtual focal planes by adopting a time-domain interleaved display method based on transition frames.

[0072] After the AR glasses control system obtains the transition frames between each virtual focal plane, it uses a time-domain interleaved display method to generate a continuous switching sequence. Among them, time-domain interleaved display is a technology that arranges and displays image frames at different time points in a specific order. The system interweaves the image frames and transition frames of each virtual focal plane in a certain time sequence. For example, the image frame of the first virtual focal plane is displayed first, followed by the transition frame between the first and second focal planes, and then the image frame of the second virtual focal plane, and so on. In this way, a continuous and smooth display sequence is constructed in the time dimension, so that users can feel that the switching of virtual focal planes is continuous and uninterrupted when watching AR content, effectively improving the visual experience.

[0073] S212 , outputting multi-focal plane display content through a dynamic light field modulator according to a weight coefficient of each virtual focal plane and a continuous switching sequence, wherein the weight coefficient is determined by the user's gaze depth.

[0074] The AR glasses control system determines a weight coefficient for each virtual focal plane based on the user's gaze depth. Optionally, the user's gaze depth can be obtained through eye tracking technology. When the user looks at an object, the system can determine the approximate depth range of the object and assign a higher weight coefficient to the corresponding virtual focal plane. For virtual focal planes corresponding to depth areas that the user does not pay attention to, the weight coefficient is lower. The system combines this information with the previously generated continuous switching sequence and inputs it into the dynamic light field modulator.

[0075] The dynamic light field modulator modulates the images of different virtual focal planes based on weighting factors, allowing the displayed content to appear at different depths, consistent with the user's focus. Virtual focal plane images with high weighting factors appear clearer and brighter, while those with low weighting factors appear less vivid. This simulates the way the human eye focuses on objects at different distances in real scenes, providing a more natural visual experience.

[0076] In the above-mentioned embodiment, the AR glasses control system provides a more natural depth perception experience by mapping the displayed content to multiple virtual focal planes and implementing intelligent depth transitions. By dynamically adjusting the weights of each focal plane based on the user's gaze depth, continuous and smooth switching of displayed content is achieved, effectively alleviating the focal plane jump problem in traditional AR displays. This multi-focal plane display technology based on time-domain interleaving improves the spatial perception quality of virtual content, reduces visual fatigue, and provides users with a more comfortable and natural mixed reality experience.

[0077] S213: When it is detected that the visual comfort parameter is lower than a first preset threshold, switch to a low visual load interaction mode; when it is detected that the physiological state parameter is lower than a second preset threshold, switch to a simplified interaction mode.

[0078] The system presets a first preset threshold and a second preset threshold, which are used to measure the critical values of visual comfort and physiological state, respectively. When the visual comfort parameter is lower than the first preset threshold, it indicates that the user may have visual fatigue, and the system automatically switches to an interactive mode with low visual load. In this mode, the system can reduce the amount of virtual information displayed, reduce image complexity, adjust display brightness and contrast, etc., to reduce the user's visual burden. When the physiological state parameter is detected to be lower than the second preset threshold, it means that the user's physiological state is not good and may be fatigued or stressed. At this time, the system switches to a simplified interaction mode, simplifies the operating process, reduces unnecessary interaction links, and reduces the user's cognitive load.

[0079] S214. Establish an interaction mode evaluation table, record the visual comfort parameters and physiological state parameters before and after each interaction mode switch, and calculate the improvement rate of the interaction mode switch on the visual comfort parameters and physiological state parameters.

[0080] Each time the AR glasses switch interaction modes, the control system creates an interaction mode evaluation table. This table records visual comfort parameters and physiological status parameters before and after the switch. After each mode switch, the system collects these parameters again at a specific interval (e.g., 5 minutes). By comparing the parameter values before and after the switch, the system calculates the improvement rate of the visual comfort parameters and physiological status parameters due to the interaction mode switch. For the visual comfort parameter, the improvement rate can be calculated as: (Visual comfort parameter after the switch - Visual comfort parameter before the switch) / Visual comfort parameter before the switch × 100%; a similar calculation method is used for the physiological status parameter. This quantitative approach allows the system to objectively evaluate the impact of each interaction mode on the user's state, providing data support for subsequent interaction mode optimization.

[0081] S215 : When it is detected that the improvement rate is less than the preset target improvement rate, the trigger threshold of the corresponding interaction mode is increased according to the preset interval; otherwise, the trigger threshold of the corresponding interaction mode is decreased according to the preset interval.

[0082] The AR glasses control system presets a target improvement rate to measure the desired effect of interaction mode switching. The system compares the calculated improvement rate of visual comfort parameters and physiological state parameters with the preset target improvement rate. If the improvement rate is less than the preset target improvement rate, it indicates that the current interaction mode switching effect is not meeting expectations. The system then increases the trigger threshold of the corresponding interaction mode at preset intervals (e.g., by 5% each time). This means that the user's visual or physiological state will need to deteriorate further before the interaction mode switch is triggered, avoiding frequent switching to less effective interaction modes. Conversely, if the improvement rate is greater than or equal to the preset target improvement rate, indicating that the interaction mode switching effect is good, the system will lower the trigger threshold of the corresponding interaction mode at preset intervals (e.g., by 3% each time). This allows the system to switch to the interaction mode more promptly based on changes in user status, improving the user experience.

[0083] In the above-mentioned embodiment, the AR glasses control system dynamically adjusts the trigger thresholds for mode switching by establishing an interaction mode evaluation table. This achieves quantitative evaluation and adaptive optimization of the effects of different interaction modes, automatically adjusting system parameters based on actual usage results to continuously optimize the user experience. In particular, during extended use, the system intelligently switches between different interaction modes based on the user's actual state, preventing the accumulation of visual and physical fatigue and ensuring optimal system performance in various usage scenarios.

[0084] The AR glasses control system of the embodiment of the present invention is an electronic device. Figure 3 A schematic diagram of the architecture of an electronic device suitable for implementing an embodiment of the present invention is shown.

[0085] It should be noted that Figure 3The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0086] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be accomplished by instructions (computer programs) or by controlling related hardware through instructions (computer programs), and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. The electronic device of this embodiment includes a storage medium and a processor, wherein the storage medium stores a plurality of instructions, which can be loaded by the processor to execute any step of the method provided in the embodiment of the present invention.

[0087] Specifically, the storage medium and the processor are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these elements can be electrically connected to each other via one or more signal lines. The storage medium stores computer-executable instructions for implementing the data access control method, including at least one software function module that can be stored in the storage medium in the form of software or firmware. The processor executes various functional applications and data processing by running the software programs and modules stored in the storage medium. The storage medium can be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The storage medium is used to store programs, and the processor executes the programs after receiving the execution instructions.

[0088] Furthermore, the software programs and modules in the above-mentioned storage medium may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components. The processor may be an integrated circuit chip having signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc., which may implement or execute the various methods, steps, and logic flow diagrams disclosed in this embodiment. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0089] Since the instructions stored in the storage medium can execute the steps of any method provided in the embodiments of the present invention, the beneficial effects of any method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0090] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An AR glasses intelligent control method based on multimodal fusion, applied to AR glasses control system, characterized in that: The method comprises: The multiple physiological sensors on the AR glasses collect the user's EEG signals, eye movement signals, heart rate signals, and skin conduction signals to establish a user physiological feature database; Comparing the physiological feature database with the user's current real-time physiological feature data through a deep learning model to identify the user's interaction intention; Generate spatially aware interactive control instructions and content presentation parameters based on the interaction intention and the user's visual focus position, and perform local image enhancement processing on the visual focus position; Generating visual comfort parameters based on real-time monitoring of the user's visual fatigue level by the mixed reality interaction engine, and generating physiological state parameters based on the real-time physiological characteristic data; According to the interactive control instructions and content presentation parameters, the spatial distribution of the content displayed on the AR glasses is adjusted in real time by a dynamic light field modulator; Adjust the optical display parameters of the AR glasses according to a weighted result of the visual comfort parameter and the physiological state parameter.

2. The method according to claim 1, characterized in that Before the step of generating spatially-aware interaction control instructions and content presentation parameters based on the interaction intention and the user's visual focus position, the method further includes: Collecting the user's historical gaze trajectory data and historical interaction intention data to establish a gaze behavior feature database; Predicting the position information of the user's next gaze area according to the gaze focus prediction model trained with the gaze behavior feature database; The rendering data of the next gaze area is loaded in advance according to the position information.

3. The method according to claim 1, characterized in that The step of performing local image enhancement processing on the sight focus position specifically includes: If it is detected that there are multiple interactive targets within the preset range of the sight focus position, obtaining spatial position data corresponding to each of the interactive targets; Calculate the priority weight of each interaction target based on the user's interaction intention strength and historical interaction frequency; The magnitude of the local image enhancement processing performed on each of the interactive targets is determined based on the priority weight.

4. The method according to claim 1, wherein The step of generating physiological state parameters based on the real-time physiological characteristic data specifically includes: Extracting normal value ranges of physiological parameters under different preset working conditions from the user physiological characteristic database to generate a baseline database of personalized physiological parameters; determining a current usage condition based on the real-time physiological characteristic data, and calculating a deviation between the real-time physiological characteristic data and a normal value range corresponding to the usage condition in the baseline database; The real-time physiological characteristic data and the deviation value are input into a multimodal feature fusion model to generate physiological state parameters. The multimodal feature fusion model is used to fuse multiple physiological characteristic data to generate physiological state parameters.

5. The method according to claim 4, characterized in that Before the step of inputting the real-time physiological feature data and the deviation value into the multimodal feature fusion model to generate the physiological state parameter, the method further includes: When it is detected that the real-time physiological characteristic data has missing values, a compensation value is calculated based on the data of the corresponding usage condition in the baseline database; The missing value is determined according to the compensation value.

6. The method according to claim 1, characterized in that The step of adjusting the spatial distribution of the content displayed on the AR glasses in real time by using a dynamic light field modulator specifically includes: Obtain content data and depth information to be displayed on AR glasses; mapping the content data to a plurality of virtual focal planes according to the depth information; Determining the distance between adjacent virtual focal planes according to the depth distribution of the content data in the plurality of virtual focal planes, and calculating a transition frame between the virtual focal planes; Generating a continuous switching sequence of the plurality of virtual focal planes by adopting a time-domain interleaved display method based on the transition frames; Multi-focal plane display content is output through a dynamic light field modulator according to a weight coefficient of each virtual focal plane and the continuous switching sequence, wherein the weight coefficient is determined by the gaze depth of the user.

7. The method according to claim 1, characterized in that After the steps of generating visual comfort parameters by monitoring the user's visual fatigue level in real time according to the mixed reality interaction engine, and generating physiological state parameters according to the real-time physiological characteristic data, the method further includes: When it is detected that the visual comfort parameter is lower than a first preset threshold, switching to an interaction mode with low visual load; When it is detected that the physiological state parameter is lower than a second preset threshold, switching to a simplified interaction mode; Establish an interaction mode evaluation table to record the visual comfort parameters and physiological state parameters before and after switching each interaction mode; Calculating the improvement rate of the interactive mode switching on the visual comfort parameter and the physiological state parameter; When it is detected that the improvement rate is less than a preset target improvement rate, increasing the trigger threshold corresponding to the interaction mode according to a preset interval; When it is detected that the improvement rate is greater than or equal to the expected target, the trigger threshold corresponding to the interaction mode is lowered according to a preset interval.

8. An AR glasses control system, characterized in that: The AR glasses control system includes: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, wherein the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the AR glasses control system to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on an AR glasses control system, the AR glasses control system executes the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product runs on an AR glasses control system, the AR glasses control system is caused to execute the method according to any one of claims 1 to 7.

Images