Multi-modal fusion method and system based on cloud AI model and brain-computer interface

Abstract

The invention discloses a multi-modal fusion method and system based on a cloud AI model and a brain-computer interface, relates to the technical field of intelligent wearable equipment, and aims to solve the technical problem that a brain-computer interface scheme in the aspect of mobile wearable equipment is lacked in the prior art. The method comprises the following steps: acquiring a brain wave signal and an eye muscle signal of a target user; the brain wave signals and the eye muscle signals are subjected to amplification processing and signal conversion processing and then are analyzed, so that the eyeball attention direction of the target user is judged; obtaining a view blind area range based on the eyeball attention direction and the view area of the target user; performing calculation based on the view blind area information and the moving speed and angle of the IMU to obtain target distance information, target range information and target area information; based on the target distance information, the target range information and the target area information, detecting obstacles in the view blind area range, and judging whether a target obstacle appears or not; and if yes, giving an alarm prompt.

Description

Technical Field

[0001] This application relates to the field of smart wearable device technology, specifically to a multimodal fusion method and system based on cloud-based AI models and brain-computer interfaces. Background Technology

[0002] Current applications of brain-computer interface (BCI) technology are significantly limited, primarily concentrated in fixed environments such as medical rehabilitation, smart home control, and virtual reality gaming. In the medical field, invasive BCI solutions are used to assist people with limb disabilities in operating wheelchairs or robotic arms to achieve basic daily living functions; however, these solutions rely on surgical implantation and are only applicable to static scenarios. In the smart home field, non-invasive BCIs control lights or appliance switches by recognizing simple brainwave commands; their interaction modes are limited and cannot adapt to dynamic environmental changes. In virtual reality gaming scenarios, while BCIs can provide an immersive experience, they are limited to pre-set content interaction and lack real-time perception of the real physical environment. It is worth noting that none of the above technologies solve the safety monitoring problem during movement, especially when the user is moving while wearing the device, as they cannot effectively identify potential obstacles in blind spots. Due to the limitations of human physiology, there are natural blind spots in human vision; when a user's attention is focused in a specific direction, the surrounding area easily becomes a visual blind spot, greatly increasing the risk of collisions during walking or movement. Current technologies neither integrate EEG and eye muscle signals for multimodal analysis nor dynamically calculate blind spot ranges using inertial measurement unit data, resulting in a severe lack of safety protection capabilities for mobile wearable devices in complex environments. This technological gap exposes users to unpredictable safety hazards in mobile scenarios such as outdoor activities and industrial inspections, necessitating an innovative solution capable of real-time perception and early warning of blind spots. Summary of the Invention

[0003] The main purpose of this application is to provide a multimodal fusion method and system based on cloud-based AI models and brain-computer interfaces, aiming to solve the technical problem of the lack of brain-computer interface solutions in mobile wearable devices in the existing technology.

[0004] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0005] In a first aspect, embodiments of this application provide a multimodal fusion method based on a cloud-based AI model and a brain-computer interface, applied to a terminal. The method includes the following steps:

[0006] Collect and obtain the target user's brainwave signals and eye muscle signals;

[0007] The brainwave signals and eye muscle signals are amplified and converted before being analyzed to determine the direction of the target user's eye attention.

[0008] Based on the target user's eye attention direction and visual field area, the range of the visual field blind spot is obtained;

[0009] Based on the blind spot information and the IMU's moving speed and angle, the target distance information, target range information, and target area information are calculated.

[0010] Based on the target distance information, the target range information, and the target area information, obstacles within the blind spot are detected, and it is determined whether a target obstacle has appeared.

[0011] If so, an alarm will be issued.

[0012] As some optional embodiments of this application, the amplification processing of the electroencephalogram signal and the eye muscle signal satisfies the following relationship:

[0013] Eamp(t) = GE·{Eraw(t) - Eref(t)}

[0014] EMGamp(t)=GEMG·{EMGraw(t)-EMGref(t)}

[0015] Where: Eraw(t) is the brainwave signal, EMGraw(t) is the eye muscle signal; GE and GEMG are the amplification gains of the brainwave signal and eye muscle signal, respectively; Eref(t) and EMGref(t) are the reference electrode signals of the brainwave signal and eye muscle signal, respectively.

[0016] The signal conversion process satisfies the following relationship:

[0017] E[n] = ADC{Eamp(t)} t = nTs

[0018] EMG[n] = ADC{EMGamp(t)} t = nTs

[0019] Where E[n] is the brainwave signal after signal conversion and processing, EMG[n] is the eye muscle signal after signal conversion and processing, Ts is the sampling period, and n is the sampling point number.

[0020] As some optional embodiments of this application, the step of amplifying and converting the brainwave signals and eye muscle signals before analysis to determine the direction of the target user's eye attention includes:

[0021] Electroocular signal modeling was performed on the ocular muscle signals to obtain the instantaneous pointing angle of the eyeball;

[0022] Time-frequency analysis of EEG information was performed to extract feature vectors related to eye attention and obtain EEG attention weights.

[0023] The instantaneous eye pointing angle and the EEG attention weight are weighted and fused to obtain the self-weighted eye attention direction of the target user.

[0024] As some optional embodiments of this application, the step of performing electrooculography (EOG) signal modeling on the ocular muscle signals to obtain the instantaneous pointing angle of the eyeball includes:

[0025] Let the horizontal electrooculogram (EOG) signal be H[n] and the vertical EOG signal be V[n]; the horizontal deflection angle θh and the vertical deflection angle θv of the eyeball satisfy the following relationship:

[0026] θh=kh·H[n]+bh

[0027] θv=kv·V[n]+bv

[0028] Where: H[n] and V[n] are the signal mean values ​​within the sliding window, used to eliminate baseline drift; kh and kv are the horizontal / vertical sensitivity coefficients; bh and bv are the calibration offsets;

[0029] Therefore, the instantaneous pointing angle of the eyeball is obtained: α=(θh,θv).

[0030] As some optional embodiments of this application, the step of performing time-frequency analysis on EEG information, extracting feature vectors related to eye attention, and obtaining EEG attention weights includes:

[0031] Performing a short-time Fourier transform / wavelet transform on E[n] yields the power spectrum: P(f,n)=STFT(E[n]) 2 ;

[0032] The relative power of the alpha wave (8 Hz–13 Hz) and the beta wave (13 Hz–30 Hz) was extracted.

[0033] Define EEG attention weights: w atte =Pβ / (Pα+ε);

[0034] Where ε is a minimal constant to prevent division by zero, w atte The larger the value, the more focused the user's visual attention.

[0035] As some optional embodiments of this application, the step of weightedly fusing the instantaneous eye pointing angle and the EEG attention weight to obtain the self-weighted eye attention direction of the target user includes:

[0036] An eye pointing vector is constructed, and based on the eye pointing vector, EEG attention weights, and the attention direction of the previous frame, an EEG attention confidence weight is obtained.

[0037] Based on the weighted EEG attention confidence, the fusion vector is normalized and mapped to the user's actual visual field coordinate system to obtain the target user's eye attention direction.

[0038] As some optional embodiments of this application, after issuing an alarm notification if the condition is met, the method further includes:

[0039] The relevant data of the target user is sent to the cloud AI deep learning server so that it can perform individualized learning and continuously improve the model algorithm.

[0040] Secondly, embodiments of this application also provide a multimodal fusion system based on a cloud-based AI model and a brain-computer interface, comprising:

[0041] The cloud-based AI deep learning server is used to provide AI models to the data processing unit for data judgment and to continuously collect terminal data in order to continuously improve the model algorithm.

[0042] The terminal is used to acquire the brainwave signals and eye muscle signals of the target user; after amplifying and converting the brainwave signals and eye muscle signals, it is analyzed to determine the direction of the target user's eye attention; based on the target user's eye attention direction and visual field area, the blind spot range is obtained; based on the blind spot information and the movement speed and angle of the IMU, target distance information, target range information, and target area information are calculated; based on the target distance information, target range information, and target area information, obstacles within the blind spot range are detected, and it is determined whether a target obstacle has appeared; if so, an alarm is issued.

[0043] As some optional embodiments of this application, the terminal includes:

[0044] Electroencephalogram (EEG) signal sensor, used to collect and obtain the EEG signals of the target user;

[0045] An eye muscle signal sensor is used to collect and obtain the eye muscle signals of the target user.

[0046] A signal amplifier is used to amplify the electroencephalogram (EEG) signal and the eye muscle signal.

[0047] ADC is used to perform signal conversion processing on the amplified electroencephalogram (EEG) signal and the eye muscle signal;

[0048] The data processing unit is used to amplify and convert the brainwave signals and eye muscle signals before analysis to determine the direction of the target user's eye attention; based on the target user's eye attention direction and visual field area, to obtain the range of the visual field blind spot; and based on the visual field blind spot information and the movement speed and angle of the IMU, to calculate and obtain target distance information, target range information, and target area information.

[0049] The detection unit includes a radar and a camera, and is used to detect obstacles within the blind spot area based on the target distance information, the target range information, and the target area information, and to determine whether a target obstacle has appeared.

[0050] A speaker is used to issue an alarm when a target obstacle is detected.

[0051] As some optional embodiments of this application, the terminal further includes:

[0052] The storage unit is used to store the target user's personal difference information and to perform personalized difference compensation for the model in specific scenarios.

[0053] Compared to existing technologies, this application, through the aforementioned technical solution, sends relevant data of the target user to a cloud-based AI deep learning server for individualized learning. This enables the system to finely adjust and optimize model parameters based on each user's unique physiological characteristics, behavioral patterns, and environmental interactions. This overcomes the limitations that general models may exhibit when facing individual differences, significantly improving the accuracy of eye attention direction judgment, the precision of blind spot definition, and the reliability of obstacle detection. Continuous improvement of the model algorithm ensures that the system can adapt to changes that may occur during long-term user use, thereby providing more accurate and personalized alarm prompts, effectively reducing the risk of false alarms and missed alarms, and greatly enhancing user experience and security. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the multimodal fusion method based on cloud-based AI models and brain-computer interfaces involved in the embodiments of this application;

[0055] Figure 2 This is a schematic diagram of the multimodal fusion system based on cloud-based AI models and brain-computer interfaces involved in the embodiments of this application. Detailed Implementation

[0056] It should be made clear that the specific implementation examples described herein are for illustrative purposes only and are not intended to limit this application.

[0057] To address the shortcomings of existing technologies, this application proposes a multimodal fusion method based on cloud-based AI models and brain-computer interfaces, applicable to terminals, such as... Figure 1 As shown, the method includes the following steps:

[0058] Collect and obtain the target user's brainwave signals and eye muscle signals;

[0059] The brainwave signal and the eye muscle signal were amplified and converted before being analyzed to determine the direction of the target user's eye attention.

[0060] Based on the target user's eye attention direction and visual field area, the range of the visual blind spot is obtained;

[0061] Based on the blind spot information and the moving speed and angle of the IMU (Inertial Measurement Unit), the target distance information, target range information, and target area information are obtained.

[0062] Based on the target distance information, target range information, and target area information, obstacles within the blind spot are detected to determine whether a target obstacle has appeared.

[0063] If so, an alarm will be issued.

[0064] For ease of understanding, the following explains some key terms in this embodiment:

[0065] Target distance information refers to the real-time relative distance between potential obstacles and the user within the blind spot. Target range information refers to the spatial range, shape, and boundary parameters of detectable obstacles within the blind spot. Target area information refers to the areas within the blind spot that require focused obstacle detection and their detection priority.

[0066] An alarm notification is a warning signal issued by the system to the user when a target obstacle is detected in the blind spot. This notification can be audible, audible, visual, or other forms, designed to alert the user to potential danger so that timely avoidance measures can be taken.

[0067] This embodiment provides a multimodal fusion method based on cloud-based AI models and brain-computer interfaces, the specific implementation of which is as follows:

[0068] First, the target user's electroencephalogram (EEG) and eye muscle signals are acquired. EEG signals are acquired by placing wet electrodes on the user's scalp. Eye muscle signals are acquired by attaching surface electrodes, such as medical patch electrodes, to the skin around the user's eyes. These electrodes are connected to appropriate bioelectrical signal acquisition devices to obtain raw analog signals.

[0069] Next, the EEG and eye muscle signals are amplified and converted before analysis to determine the user's eye attention direction. The acquired EEG and eye muscle signals are typically weak analog signals, requiring initial gain processing via an analog amplifier to increase their amplitude. These amplified analog signals are then fed into an analog-to-digital converter (ADC) to convert them into digital signals for subsequent digital signal processing. After obtaining the digital signals, simple threshold detection or waveform analysis methods, such as detecting peak values ​​or specific waveform patterns in the eye muscle signals, can be used to roughly determine the horizontal and vertical movement trends of the eyes. Simultaneously, simple frequency domain analysis is performed on the EEG signals, such as using Fast Fourier Transform (FFT) to obtain the energy distribution of different frequency bands, to identify rough attention-related features. Finally, these preliminary analysis results are subjected to simple logical judgments or linear combinations to infer the user's eye attention direction.

[0070] Subsequently, based on the target user's eye attention direction and visual field area, the blind spot range is obtained. After obtaining the target user's eye attention direction, a general, standardized visual field area model can be preset. This model is constructed based on average visual data from a large number of users and includes a fixed horizontal viewing angle, vertical viewing angle, and effective viewing distance. Based on the current eye attention direction, a core visual area is defined within this preset visual field area model. This core visual area is then subtracted from the preset complete visual field area, and the remaining portion is defined as the blind spot range.

[0071] Furthermore, based on the blind spot information and the IMU's movement speed and angle, calculations are performed to obtain target distance information, target range information, and target area information. After obtaining the blind spot range, preliminary calculations can be performed using the movement speed and rotation angle data provided by the IMU. For example, it can be assumed that the distance between all points within the blind spot and the user is equal to the average coverage distance of the blind spot, which can be used as the target distance information. The target range information can be simply set as the geometric boundary of the blind spot. The target area information can be roughly divided according to the relative position of the blind spot and the user's movement direction. For example, the blind spot portion located in the direction of movement directly in front of the user can be set as a high-priority area, while other portions are low-priority areas.

[0072] Based on this, after obtaining target distance, target range, and target area information, the blind spot area can be scanned using sensors (such as a wide-angle camera or an ultrasonic sensor). During the scan, any object signal captured by the sensor that exceeds a preset background noise threshold is initially identified as a potential obstacle. Subsequently, through simple image processing algorithms or ultrasonic echo analysis, it is determined whether these potential obstacles are located within the area defined by the target range information, and whether they constitute a target obstacle is determined based on whether their distance from the user is less than a certain fixed threshold.

[0073] Finally, when the system determines that a target obstacle has appeared in the blind spot, it can issue an audible alert through a simple buzzer or a flashing LED indicator.

[0074] As can be seen, the multimodal fusion method proposed in this application, by integrating EEG signals, eye muscle signals, and IMU data, can determine the user's eye attention direction in real time and dynamically identify blind spots in the field of vision. Therefore, in mobile wearable scenarios, this method can effectively detect potential obstacles in blind spots and issue timely alarms when danger occurs, thereby significantly improving user safety during movement and filling the gap in existing technologies for blind spot warning in mobile wearable devices.

[0075] The amplified brainwave signals and eye muscle signals described above satisfy the following relationship:

[0076] Eamp(t)=GE·{Eraw(t)-Eref(t)},

[0077] EMGamp(t)=GEMG·{EMGraw(t)-EMGref(t)};

[0078] Where Eraw(t) is the brainwave signal, EMGraw(t) is the eye muscle signal; GE and GEMG are the amplification gains of the brainwave signal and eye muscle signal, respectively; Eref(t) and EMGref(t) are the reference electrode signals of the brainwave signal and eye muscle signal, respectively.

[0079] Meanwhile, the signal conversion process satisfies the following relationship:

[0080] E[n] = ADC{Eamp(t)} t = nTs,

[0081] EMG[n] = ADC{EMGamp(t)} t = nTs;

[0082] Where E[n] is the brainwave signal after signal conversion and processing, EMG[n] is the eye muscle signal after signal conversion and processing, Ts is the sampling period, and n is the sampling point number.

[0083] This differential amplification method effectively suppresses common-mode noise, such as power line interference and motion artifacts, significantly improving the signal-to-noise ratio. The settings of the EEG signal amplification gain (GE) and the eye muscle signal amplification gain (GEMG) are crucial; they boost weak bioelectrical signals to within the dynamic range that the analog-to-digital converter (ADC) can handle, while avoiding signal saturation or loss.

[0084] In the signal conversion and processing stage, the amplified analog signals Eamp(t) and EMGamp(t) are fed into an analog-to-digital converter (ADC) for digitization. This process converts the continuous analog signal into discrete digital signals E[n] and EMG[n] by sampling at discrete time points t=nTs. Here, Ts represents the sampling period, and its reciprocal is the sampling frequency. According to the Nyquist sampling theorem, the sampling frequency should be at least twice the highest frequency of the signal to avoid aliasing and ensure the complete preservation of the original signal information. The resolution of the ADC (e.g., 12-bit, 16-bit, or 24-bit) also directly affects the accuracy of the digitized signal; a high-resolution ADC can capture more subtle signal changes.

[0085] This application further proposes to analyze the amplified and signal-converted EEG signals and eye muscle signals to determine the eye attention direction of the target user, including: performing electrooculography (EOG) signal modeling on the eye muscle signals to obtain the instantaneous eye pointing angle; performing time-frequency analysis on the EEG information to extract feature vectors related to eye attention and obtain EEG attention weights; and performing weighted fusion of the instantaneous eye pointing angle and the EEG attention weights to obtain the self-weighted eye attention direction of the target user.

[0086] Specifically, electrooculography (EOG) signal modeling of eye muscle signals is performed to obtain the instantaneous eye pointing angle. This process involves processing the signal-transformed eye muscle signals, such as the horizontal EOG signal H[n] and the vertical EOG signal V[n], using a pre-established mathematical model. This model is typically based on the linear relationship between eye movement and EOG signals, and determines the horizontal sensitivity coefficient kh, the vertical sensitivity coefficient kv, and the corresponding calibration offsets bh and bv through calibration. In practical applications, H[n] and V[n] can be data processed by averaging the signals within a sliding window to effectively eliminate possible baseline drift in the signals and ensure the accuracy of the eye deflection angle calculation. Using these parameters, the horizontal deflection angle θh and the vertical deflection angle θv of the eye can be calculated, thus obtaining the instantaneous eye pointing angle α=(θh,θv). This instantaneous pointing angle characterizes the physical direction of the user's eye in space and is the basis for determining the direction of attention.

[0087] This application also proposes time-frequency analysis of EEG information to extract feature vectors related to eye attention and obtain EEG attention weights. Specifically, time-frequency analysis is performed on the EEG signal E[n] after signal conversion processing, for example, using short-time Fourier transform or wavelet transform, to obtain its power spectrum P(f,n). Based on this, the relative power of specific frequency bands can be extracted, such as the relative power of alpha waves in the range of 8 Hz–13 Hz and beta waves in the range of 13 Hz–30 Hz. EEG activity in these frequency bands is closely related to visual attention states. By defining EEG attention weights w... atte =Pβ / (Pα+ε), where ε is a very small constant to prevent division by zero, which can quantify the user's current level of visual attention. atte The larger the value, the more focused the user's visual attention is, which provides a cognitive basis for subsequent attention direction fusion.

[0088] Building upon this, this application further proposes a weighted fusion of the instantaneous eye pointing angle and the EEG attention weight to obtain the target user's self-weighted eye attention direction. This fusion process first constructs an eye pointing vector, which can be represented in three-dimensional space based on the instantaneous pointing angle α=(θh,θv). Then, based on the eye pointing vector, the EEG attention weight, and the attention direction of the previous frame, an EEG attention confidence weight is obtained. This confidence weight can be dynamically adjusted according to the magnitude of the EEG attention weight. For example, when the EEG attention weight is high, it indicates that the user's attention is highly focused, and the confidence of the eye pointing vector is higher; conversely, when the attention weight is low, it may be necessary to refer more to the attention direction of the previous frame for smoothing or correction to improve stability. Finally, based on the EEG attention confidence weight, the fused vector is normalized and mapped to the user's actual visual field coordinate system, thereby obtaining the target user's final eye attention direction.

[0089] This application also proposes a more refined method for obtaining the blind spot range, which includes the following steps:

[0090] First, the terminal's pre-set visual calibration module personalizes the target user's individual visual field area to eliminate the impact of individual visual differences on visual field judgment. The visual calibration module is a built-in functional unit on the terminal, designed to acquire and record each target user's unique visual physiological parameters through interactive guidance. This includes the user's horizontal and vertical visual boundaries, as well as effective viewing distance, at different eye-turning angles. During calibration, the terminal guides the user to sequentially focus on multiple preset reference points on the screen, covering the extreme visual points in both the horizontal and vertical directions. The system combines this with previously acquired basic data on the user's eye attention direction to accurately record the user's visual boundaries at different eye-turning angles, thereby determining the user's complete visual field area. This visual field area, with the user's eyeball as the origin, forms a three-dimensional visual space containing three core parameters: horizontal viewing angle, vertical viewing angle, and effective viewing distance. Simultaneously, this process eliminates natural visual limitations caused by individual eye physiological structures (such as eyelid occlusion and retinal light-sensing range), ultimately generating a unique user visual field area parameter library for subsequent blind spot calculations. This parameter library is personalized and can effectively eliminate the impact of visual differences between different users on the judgment of the field of view, providing reliable benchmark data for subsequent accurate calculation of blind spots.

[0091] Secondly, based on the target user's eye attention direction obtained in the previous step, the terminal tracks the dynamic changes of this attention direction in real time to determine the current direction of the user's eye, i.e., the core direction of attention focus. Simultaneously, the system records the eye turning angles corresponding to this attention direction, such as horizontal and vertical turning angles. This real-time data is then matched to a pre-defined user visual field parameter library to determine the specific location of the current attention direction within the complete visual field. Based on this, the system can delineate the current user's attention focus range. This range is the core visible part within the user's complete visual field, the area the user is currently actively focusing on and can clearly perceive; its boundaries are determined by the turning limits of the eye attention direction and the natural visible boundaries of the visual field. This step ensures accurate identification of the user's real-time focus area, providing a dynamic basis for subsequent blind spot delineation.

[0092] Next, combining the user's complete visual field and the current focus of attention, the terminal data processing module completes the division and definition of the blind spot. The core definition of a blind spot is: within the user's complete visual field, the area beyond the current focus of attention, which the user cannot perceive through eye movement or peripheral vision; it also includes spatial areas outside the complete visual field that the user cannot naturally see. Blind spots are subdivided into "relative blind spots" and "absolute blind spots." Relative blind spots are the non-focused areas within the complete visual field; that is, although the user has visual ability, because their current attention is focused on the core direction, they do not actively pay attention to this area, resulting in an inability to perceive changes in objects or the environment in real time. The extent of this area adjusts synchronously with the dynamic changes in the direction of eye attention. For example, when the user's attention is focused directly in front, parts of the visible area on the left and right sides horizontally, and parts of the visible area on the up and down sides vertically, are all considered relative blind spots. An absolute blind spot is a spatial area beyond the user's complete field of vision. It is a region that cannot be seen regardless of eye movement due to the user's eye's physiological structure and the limits of their visual field. The extent of this area is relatively fixed and only relates to the previously defined boundaries of the user's visual field. For example, areas beyond the limits of the user's horizontal and vertical visual angles, as well as areas beyond the effective viewing distance, all fall under the category of absolute blind spots. This classification helps in a more precise understanding and handling of different types of blind spots.

[0093] Finally, the terminal integrates the divided relative and absolute blind zones, eliminating the overlapping portions of the two types of blind zones to clearly define the overall range of the visual blind zone. Simultaneously, the system records the core parameters of the blind zone, including its spatial orientation, coverage angle, coverage distance, and size, and converts them into a standardized data format that the terminal can recognize and use for subsequent calculations, thus completing the acquisition of the visual blind zone range. This blind zone range data will be synchronously transmitted to the next step, providing accurate basic data for calculating target distance information, target range information, and target area information based on IMU data, as well as for subsequent obstacle detection. This ensures the accuracy of subsequent alarm prompts and avoids missed or false detections due to deviations in the blind zone range definition.

[0094] This application further proposes a method for calculating target distance information, target range information, and target area information based on the aforementioned blind spot information and the IMU's moving speed and angle, specifically including the following steps:

[0095] First, the preprocessing and synchronization calibration of the basic data are completed. This step aims to provide accurate and consistent input data for subsequent distance, range, and area calculations. The terminal first retrieves the complete information of the previously determined blind spots, including the spatial orientation, coverage angle, coverage distance, and range size of the blind spots, as well as the classification and definition results of relative and absolute blind spots. Simultaneously, it reads two types of core sensor data transmitted by the IMU in real time—movement speed and rotation angle. The movement speed covers the linear and curved movement speeds of the terminal (and the target user wearing the terminal), while the rotation angle covers the horizontal turning angle and the vertical pitch angle. For the read IMU data, the terminal uses built-in filtering algorithms, such as Kalman filtering or complementary filtering, to remove abnormal fluctuations caused by device jitter and environmental electromagnetic interference, ensuring the stability and accuracy of the speed and angle data. Subsequently, timestamp synchronization technology is used to align the blind spot information with the processed IMU data in terms of time dimension, ensuring that all data involved in the calculation correspond to the same time node and avoiding calculation deviations caused by data asynchrony. At the same time, the angle data of the IMU is mapped to the user's field of view coordinate system that was calibrated in the early stage, so as to achieve dimensional unification of the two types of data and lay the foundation for subsequent spatial calculations.

[0096] Secondly, the target distance information is calculated. The core of this step is determining the real-time relative distance and its dynamic changes between the area containing potential obstacles within the blind spot and the user (terminal). During the calculation, using the user's eyeball as the spatial origin and combining the already defined basic coverage distance parameters of the blind spot, the initial distance range between the blind spot and the user is first delineated. Then, based on the real-time movement speed output by the IMU, the user's current movement state is determined. If the user is stationary, the initial distance range remains unchanged; if the user is moving at a constant or accelerating speed, the relative distance between the blind spot and the user is updated in real-time, taking into account the movement speed and the time interval between two adjacent calculations. Simultaneously, by combining the IMU's rotation angle data, the spatial orientation offset of the blind spot caused by the user's body turning and head rotation is corrected, thereby accurately calculating the real-time distance between different points within the blind spot and the user. Finally, these distance data are integrated to form complete target distance information, clearly distinguishing the distance boundaries of the near blind spot (closer to the user, high collision risk) and the long blind spot (farther from the user, low collision risk), providing a basis for subsequent risk assessment.

[0097] Next, the target range information is calculated. This step aims to further refine the blind spot range, focusing on clarifying the spatial range, shape, and boundary parameters of detectable obstacles within the blind spot, providing support for subsequent obstacle localization. During calculation, the previously determined basic blind spot range (including the coverage angle and size of relative and absolute blind spots) is used as a benchmark, combined with real-time rotation angle changes captured by the IMU, to synchronously adjust the spatial boundary of the blind spot. When the user turns their head or body, the IMU provides real-time feedback of the corresponding angle data, and the terminal dynamically corrects the orientation and coverage of the blind spot accordingly, ensuring that the target range information remains synchronized with the user's current spatial posture and eye attention direction. Simultaneously, combined with the calculated target distance information, the specific size of the blind spot spatial range is refined based on the differences in blind spot coverage angles at different distances. For example, near-range blind spots are limited by the field of view angle, resulting in a relatively concentrated coverage area and smaller size; long-range blind spots have a relatively wide coverage area and larger size. Finally, standardized target range information is output, clarifying the spatial boundary thresholds of detectable obstacles within the blind spot range.

[0098] Next, the target area information is calculated. This step integrates and optimizes target distance and target range information, focusing on identifying key obstacle detection areas and their priorities within the blind spot to improve the efficiency and accuracy of subsequent obstacle detection. During the calculation, the terminal combines the movement speed and rotation angle data fed back by the IMU to predict the user's movement trend, such as moving forward / backward or turning left / right. Combined with target distance information, areas within the blind spot that are close to the user and move in the same direction are designated as high-priority detection areas, as these areas have the highest risk of collision with the user. Blind spot areas that are far from the user and deviate from the user's movement direction are designated as low-priority detection areas. Simultaneously, based on the classification of blind spots, areas in the absolute blind spot that exceed the user's visual limits and cannot be avoided by turning are separately designated as key warning areas. Finally, the orientation, detection priority, distance, and range parameters of all areas are integrated to form complete target area information, guiding subsequent obstacle detection strategies.

[0099] Finally, the calculation results are verified and output. This step aims to ensure the accuracy and consistency of the calculation results. The terminal synchronously verifies the target distance information, target range information, and target area information obtained from the above calculations, checking the consistency between the three types of information. For example, the distance range of the target area must match the target distance information, and the range parameters must be consistent with the target range information. Through this cross-validation, abnormal parameters caused by data interference or calculation deviations can be eliminated. After the verification is passed, the three types of information are converted into a standardized data format that the terminal can recognize and directly use for the next obstacle detection step, and transmitted in real time to the subsequent detection steps. This provides accurate and comprehensive basic data for obstacle detection and target obstacle judgment within the blind spot area, ensuring the coherence and feasibility of the entire technical solution, and fully adapting to the application requirements of real-time data processing of the terminal.

[0100] In some of the embodiments described above in this application, although the blind spot range can be obtained based on the target user's eye attention direction and visual field area, and the target distance information, target range information, and target area information can be calculated by combining the IMU's movement speed and angle, in practical applications, how to efficiently and accurately detect obstacles within the blind spot and reliably determine whether there are truly target obstacles that require warning, so as to avoid false alarms or missed alarms, remains a problem that needs to be solved. To address this, this application further proposes a method for detecting obstacles within the blind spot range and determining whether a target obstacle has appeared based on the aforementioned target distance information, target range information, and target area information, specifically including the following steps:

[0101] First, the detection parameters are initialized and the detection area is located. The terminal first calls the three types of core basic data acquired in the early stage: target distance information (including the real-time relative distance between each point in the blind zone and the user, and the distance boundaries of near and far blind zones), target range information (including the spatial range, shape, and boundary parameters of the blind zone), and target area information (including detection priority and the division of key warning areas). Then, the obstacle detection parameters are initialized and set according to the three types of data, including detection sensitivity, detection frequency, and obstacle judgment threshold. The detection frequency is kept synchronized with the data acquisition frequency of the IMU to ensure real-time detection. The detection sensitivity is dynamically adjusted in combination with the target distance information—higher sensitivity is set for near blind zones and high-priority detection areas, and the sensitivity is appropriately reduced for far blind zones and low-priority detection areas to balance detection accuracy and terminal power consumption. At the same time, based on the target area information and target range information, the detection range of the blind zone is accurately located, and the specific spatial coordinates of high-priority detection areas, low-priority detection areas, and key warning areas are determined, so that the detection focus is prioritized on high-risk areas to improve detection efficiency.

[0102] The initialization of detection parameters aims to set the basic operating conditions and judgment criteria for the subsequent obstacle detection process. Detection sensitivity can be dynamically adjusted based on target distance information. For example, for near-field blind spots or high-priority detection areas close to the user, a higher detection sensitivity can be set to ensure rapid response to potential hazards and detection of subtle obstacles. Conversely, for distant near-field blind spots or low-priority detection areas, the sensitivity can be appropriately reduced to minimize unnecessary interference and computational energy consumption. The detection frequency should be synchronized with the IMU's data acquisition frequency. For instance, if the IMU acquires data at 100Hz, the detection frequency should also be set to 100Hz to achieve rapid response to environmental changes and real-time monitoring. The obstacle judgment threshold can be preset with a series of parameters, such as minimum obstacle size (e.g., objects smaller than 5 cm can be ignored), minimum reflection intensity (e.g., signals below a certain decibel value can be considered noise), and minimum duration (e.g., signals lasting less than a certain millisecond can be considered transient interference), used for initial screening of potential obstacles. Detection area localization involves dividing blind spots into different detection priority areas based on target area and target range information. For example, high-priority detection areas might correspond to the blind spots in front of the user's direction of movement and at close range; key warning areas might correspond to parts of the absolute blind spot that exceed the user's visual limits and cannot be avoided by turning; low-priority detection areas are other relative blind spots. The terminal can store the spatial coordinates of these areas as 3D bounding boxes or polygonal regions for subsequent scanning and detection, ensuring that detection resources are effectively allocated to the most critical areas.

[0103] Secondly, comprehensive obstacle scanning and detection is conducted within the blind spot area. The terminal, through its built-in sensing components (working in conjunction with the brain-computer interface and IMU), combined with target distance information, performs a comprehensive scan of the area within the blind spot. The scanning method employs a partitioned scanning mode, prioritizing high-priority detection areas and key warning areas, followed by lower-priority detection areas, avoiding missed detections, false detections, or detection delays caused by disordered scanning. During the scanning process, the reflected signals, contour features, and spatial location information of various potential obstacles within the blind spot are captured in real time. Simultaneously, combined with target distance information, the real-time relative distance between each potential obstacle and the user is recorded. By combining this with target range information, it determines whether potential obstacles are within the boundary of the blind spot, eliminating interference signals outside the blind spot to ensure targeted detection.

[0104] The sensing and detection components may include, but are not limited to, radar (such as millimeter-wave radar), lidar (LiDAR), ultrasonic sensors, and cameras (combined with image recognition algorithms). These sensors work together to acquire information such as the distance, speed, angle, size, and shape of obstacles. The zoned scanning mode refers to the sensing and detection components allocating resources and adjusting the scanning order according to the priority of pre-defined detection areas during scanning. For example, the scanning beam of radar or lidar can be guided or focused on high-priority areas, or sensor data from these areas can be processed first during data processing. Cameras can prioritize image analysis of these areas. This mode concentrates limited computing resources and sensor capabilities on the areas most likely to be dangerous, improving detection efficiency and response speed. During scanning, the reflected signals, contour features, and spatial location information of potential obstacles are captured in real time; these are raw data directly or indirectly acquired by the sensors. For example, radar can provide reflected signal intensity and distance, lidar can provide point cloud data to form contours and spatial locations, and cameras can provide image pixel data for contour recognition. Simultaneously, combined with target distance information, the real-time relative distance between each potential obstacle and the user is recorded, which helps in subsequent distance risk assessment. By combining target range information, it is determined whether potential obstacles are within the boundary of the blind spot. Interference signals beyond the blind spot are eliminated to ensure the targeting of detection and avoid processing irrelevant objects.

[0105] Next, feature extraction and preliminary screening of potential obstacles are completed. Based on the information about potential obstacles captured during the scanning detection process, the terminal detection module performs multi-dimensional feature extraction. The core features extracted include the obstacle's outline size, spatial posture, movement state (stationary or moving), and reflection intensity. Simultaneously, combined with target distance information, the distance change trend between the potential obstacle and the user is extracted. Combined with target area information, the detection area priority of the potential obstacle is recorded. After feature extraction, based on the preset obstacle judgment threshold and the previously initialized detection parameters, potential obstacles are initially screened, eliminating interference items that do not meet the judgment criteria—for example, tiny objects with excessively small outline sizes or reflection intensities below the threshold, interference signals exceeding the near-far blind zone boundaries in the target distance information, and signals that are outside the blind zone but mistakenly captured. Potential obstacles that meet the judgment criteria are retained after screening and proceed to the next step of precise judgment.

[0106] The multi-dimensional feature extraction process includes extracting features such as obstacle outline dimensions (calculated from sensor data to determine the obstacle's length, width, height, or area), spatial orientation (the obstacle's orientation or tilt angle in three-dimensional space), movement state (determining whether the obstacle is stationary or moving, and its speed and direction, based on continuous frames of sensor data), reflection intensity (the signal strength received by the sensor, which can be used to distinguish obstacles of different materials), distance change trends (determining whether the obstacle is approaching, moving away, or remaining relatively stationary based on continuous distance measurement data), and detection area priority (recording the area where the obstacle is located, such as high priority, low priority, or key warning). The initial screening is based on preset obstacle judgment thresholds and initialized detection parameters to quickly filter potential obstacles after feature extraction. For example, if an object's outline dimensions are smaller than a preset minimum obstacle size threshold (e.g., less than 5 cm), or its reflection intensity is lower than an environmental noise threshold, it can be initially judged as a non-obstacle or interference signal and eliminated. This step aims to quickly eliminate a large number of irrelevant or minor interferences, reduce the computational burden of subsequent complex judgments, and improve system efficiency.

[0107] Then, a multi-dimensional comprehensive judgment is conducted to determine whether a target obstacle has appeared. Combining the characteristic information of potential obstacles screened in the early stages, along with target distance information, target range information, and target area information, a multi-dimensional comprehensive judgment is carried out. The core judgment dimensions include three aspects to ensure the accuracy of the judgment results. First, positional conformity judgment: determining whether the potential obstacle is within the blind spot and whether it is within the detection area defined by the target area information, with a focus on checking whether it is in a high-priority detection area or a key warning area. If it is outside the blind spot or a non-detection area, it is judged as a non-target obstacle and excluded. Second, distance risk judgment: combining target distance information, judging the real-time relative distance between the potential obstacle and the user and the distance change trend. If the potential obstacle is in the near blind spot and the distance to the user is continuously decreasing, there is a collision risk, and it is prioritized as a high-risk potential obstacle. If it is in the far blind spot... If the obstacle is in a blind spot and the distance does not change significantly, it is temporarily identified as a low-risk potential obstacle. Third, feature matching judgment: the extracted features of the potential obstacle are matched with the target obstacle feature library preset by the terminal. The target obstacle feature library contains common obstacle features that may pose a collision risk to the user (such as walls, tables and chairs, moving pedestrians or objects, etc.). If the feature matching degree of the potential obstacle reaches the preset threshold, and the first two judgments meet the requirements, it is initially identified as a target obstacle. If the feature matching degree does not reach the threshold, or the position and distance judgments do not meet the requirements, it is identified as a non-target obstacle and is excluded.

[0108] Multi-dimensional comprehensive judgment is the core decision-making process. Location conformity judgment compares the spatial coordinates of potential obstacles with the boundary coordinates of the blind spot, high-priority detection area, and key warning area to determine if it falls within the effective detection area. For example, if the obstacle's coordinates fall within the high-priority detection area, its location conformity is high. Distance risk judgment combines target distance information. If the real-time relative distance between the potential obstacle and the user is less than a preset near-blind spot distance threshold, and the distance trend shows a continuous approach, it is judged as having a high collision risk. Conversely, if the distance is far and there is no obvious approach trend, the risk is low. Feature matching judgment matches the extracted features of potential obstacles with a preset target obstacle feature library on the terminal. The target obstacle feature library can pre-store typical feature models of various common obstacles (such as pedestrians, vehicles, walls, pillars, tables, chairs, etc.), including their average size, shape, material reflectivity, etc. Using machine learning or pattern recognition algorithms, the extracted potential obstacle features are compared with the models in the library to calculate the matching degree. For example, if the matching degree is higher than 80%, it is considered likely to be a target obstacle. Only when all three judgment dimensions meet the preset conditions can it be initially determined as a target obstacle.

[0109] Finally, the judgment results are verified and confirmed. To avoid misjudgments caused by a single-dimensional judgment, the terminal performs a secondary verification of the initially identified target obstacle. The verification includes: rechecking whether its position is within the blind spot, whether the distance change trend poses a collision risk, and whether the feature matching degree is stable. Simultaneously, combining the IMU's movement speed and rotation angle data, it predicts whether the user's subsequent movement trajectory intersects with the potential obstacle's movement trajectory. If an intersection exists, it is further confirmed as a target obstacle. If the verification reveals a deviation in the initial judgment, or if the potential obstacle has exceeded the blind spot and the distance is no longer decreasing, the judgment result is corrected, and it is determined to be a non-target obstacle. After verification, the terminal explicitly outputs the final judgment result: if, after comprehensive judgment and secondary verification, an obstacle meeting all judgment criteria exists, it is judged as "a target obstacle has appeared"; if, after scanning, detection, and screening, no obstacle meeting the judgment criteria is found, or all potential obstacles are excluded, it is judged as "no target obstacle has appeared," and the judgment result is transmitted in real time to the next step, providing accurate basis for whether to issue an alarm.

[0110] The secondary verification process further validates the initial assessment, aiming to improve the accuracy and robustness of the judgment. Verification includes rechecking whether the obstacle's position remains within the blind spot, whether the distance between the obstacle and the user still poses a collision risk, and whether its feature matching remains stable. More importantly, by combining IMU data on movement speed and rotation angle, the system can predict whether the user's subsequent movement trajectory intersects with the trajectory of potential obstacles. If the predicted trajectory intersects, the collision risk is further confirmed, thus confirming it as a target obstacle. If the secondary verification reveals a deviation in the initial judgment—for example, the obstacle has moved out of the blind spot, the distance is no longer decreasing, or the feature matching has decreased—it is corrected to a non-target obstacle. This step is the final decision point, ensuring that only obstacles that have undergone rigorous multi-dimensional judgment and secondary verification are confirmed as "target obstacles," thereby triggering an alarm.

[0111] In some of the embodiments described above in this application, due to differences in individual physiological characteristics, behavioral habits, and environmental factors, a single general-purpose model may struggle to maintain optimal detection accuracy and adaptability, potentially affecting the accuracy or timeliness of alarms and failing to fully meet the personalized needs of users. Therefore, this application further proposes that, after issuing an alarm, if the stated condition is met, the relevant data of the target user should be sent to a cloud-based AI deep learning server for individualized learning, continuously improving the model algorithm.

[0112] Specifically, the relevant data of the target user may include, but is not limited to: raw or preprocessed electroencephalogram (EEG) signals, eye muscle signals, inertial measurement unit (IMU) data, eye attention direction determined by the system, calculated blind spot range, target distance information, target range information, target area information, and features extracted during obstacle detection and the final detection judgment results. In addition, it may also include user feedback data on alarm prompts (if any) and relevant data about the terminal's environment (such as light intensity, ambient noise, etc.). This data is transmitted to a cloud-based AI deep learning server in encrypted and / or anonymized form via the terminal's built-in communication module, such as through wireless communication methods like Wi-Fi or cellular networks (e.g., 4G / 5G). The cloud-based AI deep learning server typically consists of a high-performance computing cluster equipped with a graphics processing unit (GPU) or tensor processor (TPU) to support the training and inference of large-scale deep learning models.

[0113] The personalized learning refers to the cloud-based AI deep learning server making personalized adjustments and optimizations to a pre-set general AI model based on relevant data uploaded by a specific target user, so that it can more accurately adapt to the user's physiological characteristics, behavioral patterns and specific usage scenarios.

[0114] Furthermore, embodiments of this application propose a multimodal fusion system based on a cloud-based AI model and a brain-computer interface, such as... Figure 2 As shown, it includes:

[0115] The cloud-based AI deep learning server is used to provide AI models to the data processing unit for data judgment and to continuously collect terminal data in order to continuously improve the model algorithm.

[0116] The terminal is used to collect the target user's brainwave signals and eye muscle signals; after amplifying and converting the brainwave signals and eye muscle signals, it is analyzed to determine the target user's eye attention direction; based on the target user's eye attention direction and visual field area, the blind spot range is obtained; based on the blind spot information and the IMU's movement speed and angle, the target distance information, target range information, and target area information are calculated; based on the target distance information, target range information, and target area information, obstacles within the blind spot range are detected to determine whether a target obstacle has appeared; if so, an alarm is issued.

[0117] The core innovation of this embodiment lies in the systematic integration of a cloud-based AI deep learning server with a terminal's multimodal signal acquisition and analysis module. This enables dynamic identification of blind spots and real-time obstacle detection in mobile wearable scenarios, thereby enhancing mobile safety. Specifically, the terminal simultaneously acquires EEG signals and eye muscle signals, combining them with IMU motion data to construct a complete dynamic model of the blind spot. Based on refined calculations of target distance, target range, and target area information, it accurately locates high-risk areas. Compared to traditional solutions that rely on a single sensor or static visual field model, this system effectively solves the problem of visual field calibration deviation caused by individual differences through multi-source signal fusion and collaborative optimization of the cloud-based AI model. It also avoids the risk of missed or false detections due to data asynchrony.

[0118] In its implementation, the terminal first acquires raw physiological signals through EEG signal sensors and eye muscle signal sensors. After preprocessing by signal amplifiers and ADCs, the data processing unit analyzes the direction of eye attention in real time. Based on the user's personalized visual field calibration results, the system dynamically divides relative and absolute blind spots, and combines the movement speed and angle data fed back by the IMU to accurately calculate target distance, target range, and target area information. The detection unit, based on this information, prioritizes scanning high-priority areas, identifies target obstacles through multi-dimensional feature matching and secondary verification mechanisms, and finally issues tiered alarm prompts via speakers. This technical solution not only achieves proactive warnings of obstacles in visual blind spots but also continuously collects terminal data through a cloud-based AI deep learning server for model iteration, ensuring the system's adaptability and reliability in different scenarios, thereby significantly improving the safety performance of mobile wearable devices in complex environments.

[0119] This application further describes the specific composition of the aforementioned terminal, which includes an electroencephalogram (EEG) signal sensor, an eye muscle signal sensor, a signal amplifier, an ADC, a data processing unit, a detection unit, and a speaker.

[0120] Among them, the electroencephalogram (EEG) signal sensor is used to acquire the brainwave signals of the target user. EEG signal sensors typically employ electrode arrays, capturing the electrical activity of neurons in the brain through contact with the scalp. These electrodes can be wet electrodes (requiring conductive gel) or dry electrodes (not requiring gel), and their arrangement and number affect the spatial resolution and coverage of the signal. The acquired EEG signals are weak bioelectrical signals, which are important indicators for assessing the user's cognitive state and attention level.

[0121] Eye muscle signal sensors are used to acquire eye muscle signals from a target user. These sensors typically detect changes in electrical potential (EOG) generated during eye movements by placing electrodes on the skin around the eyes. These electrodes can be non-invasive surface electrodes, usually located around the eye socket, such as the inner and outer corners of the eyes and above and below them. By analyzing these signals, horizontal and vertical eye movements can be accurately tracked, thereby determining the user's gaze point and instantaneous eye pointing.

[0122] The signal amplifier is used to amplify the aforementioned electroencephalogram (EEG) and eye muscle signals. Since bioelectrical signals are typically very weak and susceptible to noise interference, a high-gain signal amplifier is needed to amplify them, improving the signal-to-noise ratio (SNR) to achieve the voltage level required by subsequent processing circuits. The signal amplifier typically employs a differential amplifier or instrumentation amplifier, possessing high common-mode rejection ratio (CMRR) and low noise characteristics to ensure signal accuracy and integrity.

[0123] An analog-to-digital converter (ADC) is used to convert the amplified EEG and eye muscle signals into digital signals for analysis by the digital processing unit. The amplified analog bioelectrical signals need to be converted into digital signals before they can be analyzed by the digital processing unit. The ADC converts continuous analog signals into discrete digital sequences through sampling and quantization. The ADC's sampling rate and resolution are key parameters, determining the fidelity of the digital signal to the original analog signal. High sampling rates and high resolutions help preserve detailed information about the signal, providing a foundation for subsequent accurate analysis.

[0124] The data processing unit amplifies and converts the aforementioned EEG and eye muscle signals before analyzing them to determine the target user's eye attention direction. Based on the target user's eye attention direction and visual field area, it obtains the blind spot range. Based on the blind spot information and the IMU's movement speed and angle, it calculates target distance, target range, and target area information. The data processing unit is the core computing module of the terminal, typically composed of a microcontroller, digital signal processor (DSP), or embedded system, responsible for executing complex algorithms. Its functions include time-frequency analysis, feature extraction, and modeling fusion of digitized EEG and eye muscle signals to accurately determine the user's eye attention direction. Based on this, the data processing unit dynamically calculates and defines the blind spot range by combining preset or calibrated user visual field parameters. Furthermore, it receives and processes movement speed and angle data from the IMU, and through fusion calculation, obtains target distance, target range, and target area information within the blind spot in real time, providing comprehensive spatial perception data for subsequent obstacle detection.

[0125] The detection unit, comprising radar and a camera, is used to detect obstacles within the aforementioned blind spot based on the target distance, target range, and target area information, and to determine whether a target obstacle exists. The detection unit is a key hardware component for achieving obstacle perception in blind spots. The radar detects obstacles by emitting and receiving radio waves, providing accurate distance, speed, and angle information, and is unaffected by environmental conditions such as lighting and smoke, making it suitable for all-weather operation. The camera acquires visual images through optical imaging, providing rich texture, color, and shape information, which helps identify the type and fine features of obstacles. The integrated use of radar and camera achieves complementary advantages: radar provides accurate distance and motion data, while the camera provides rich visual details, jointly improving the accuracy and robustness of obstacle detection, especially in complex and changing blind spot environments.

[0126] The speaker is used to issue an alarm when a target obstacle is detected. As a human-computer interaction output device, the speaker can promptly alert the user by emitting a clear audible alarm when a potential target obstacle is detected, thereby avoiding potential collision risks. The alarm can be a preset buzzer, voice prompt, or other audible signal, and its volume and frequency can be adjusted according to actual needs to ensure that the user can perceive and react in a timely manner.

[0127] Through the aforementioned technical solution, the clearly defined hardware component division and functional configuration within the terminal effectively solve the problems of complex system integration, low data acquisition and processing efficiency, and difficulty in guaranteeing the accuracy and real-time performance of obstacle detection. EEG signal sensors and eye muscle signal sensors ensure high-quality biosignal acquisition, providing reliable raw data for subsequent attention direction determination. The signal chain composed of signal amplifiers and ADCs ensures signal integrity and the accuracy of digital processing. The data processing unit, as the core computing platform, efficiently performs complex tasks such as multimodal signal fusion, blind spot definition, and target information calculation. More importantly, the collaborative work of radar and camera in the detection unit enables multi-dimensional, high-precision perception of obstacles within the blind spot. Radar excels in providing accurate distance and speed information, while the camera performs exceptionally well in providing rich visual details and identifying obstacles. Their combination significantly improves the accuracy and robustness of obstacle detection, effectively compensating for the limitations of a single sensor in complex environments. When a target obstacle is detected, the speaker can promptly issue an alarm, ensuring a rapid user response, thereby significantly improving the overall system performance and user safety.

[0128] This application further proposes that the terminal also includes a storage unit for storing the target user's personal difference information and performing personal difference compensation for the model in specific scenarios.

[0129] Specifically, the storage unit is a hardware or software module within the terminal used for persistent data storage. It can utilize non-volatile storage media such as built-in flash memory, solid-state drives, or embedded multimedia cards, or leverage file systems or database services provided by the terminal's operating system. The core function of this storage unit is to provide the data foundation for subsequent personalized compensation, ensuring that user-specific information can be stored long-term and accessed at any time, avoiding repeated retrieval or calculation each time it is used.

[0130] The stored personal difference information of the target user refers to data reflecting the unique attributes of the target user in terms of physiology, behavior, cognition, etc. This information may include the user's physiological parameters, such as physiological structural data of the eyeball (e.g., eyeball diameter, pupil size, degree of eyelid occlusion), retinal light-sensitive range, baseline characteristics of electroencephalogram (EEG), amplitude range of eye muscle signals, etc. This data can be calibrated by the terminal when the user first uses it or modeled through long-term data collection. In addition, user behavioral habit data can also be stored, such as the user's eye movement habits in different situations, attention concentration patterns, reaction time to specific stimuli, and visual perception preferences at different movement speeds and angles. This data can be learned and accumulated through the user's interaction history with the terminal, task completion status, etc.

[0131] The aforementioned personalized compensation for models in specific scenarios refers to the system dynamically adjusting and optimizing general model or algorithm parameters based on stored individual differences of the target user in specific application scenarios to adapt to the user's individual characteristics and improve system performance. For example, when determining the user's eye attention direction, the system can correct the calculation of the instantaneous eye pointing angle α=(θh,θv) based on stored personalized calibration parameters such as kh, kv, bh, and bv, making it more accurately reflect the user's actual eye pointing. Simultaneously, by combining the user's historical EEG attention weight watte data, the system adjusts the EEG attention confidence weighted fusion process to adapt to the user's unique EEG attention patterns. When obtaining the visual field blind spot range, the system can call upon the stored user's personal visual field parameter library (including horizontal viewing angle, vertical viewing angle, and effective viewing distance) instead of using general preset values, thereby more accurately defining the user's complete visual field and attention focus range, and thus accurately dividing relative and absolute blind spots. For example, for users with significant eyelid occlusion, the lower limit of the vertical viewing angle can be appropriately adjusted. When detecting obstacles, the detection sensitivity and obstacle determination threshold can be adjusted based on the user's perception habits of obstacles at different distances and of different types. For example, for users who react slowly to near obstacles, the detection sensitivity and alarm lead time for near-range blind spots can be appropriately increased. When determining whether a target obstacle has appeared, feature matching can be optimized by incorporating the user's preference for recognizing specific obstacles. This compensation mechanism is key to solving the problem of insufficient accuracy of general models in the face of individual differences. It enables the system to transform from a "one-size-fits-all" general solution to a personalized service that is "tailored to each individual," significantly improving the accuracy, reliability, and user experience of the entire multimodal fusion method.

[0132] In some embodiments described above, the system detects obstacles in blind spots using a multimodal fusion method and issues an alarm. However, due to significant individual differences among users in terms of EEG signals, eye muscle signals, eye attention direction judgment, and visual field calibration, using a uniform model or parameters may lead to a decrease in the system's accuracy in judging specific users, or even misjudgment or missed judgment, affecting the accuracy of alarm prompts and user experience.

[0133] Through the aforementioned technical solutions, the system can break free from excessive reliance on general models. By introducing storage units and storing the individual differences of target users, the data processing unit can utilize this personalized information to provide differentiated compensation to the AI ​​model in specific scenarios. This allows for more accurate adaptation to the physiological and behavioral characteristics of the target user in determining eye attention direction, acquiring blind spot range, calculating target distance, target range, and target area information, as well as ultimately detecting obstacles and providing alarm prompts. This significantly improves the system's adaptability and accuracy to individual users, effectively avoiding misjudgments and omissions caused by individual differences, and enhancing system reliability and user experience. For example, for users whose eye movement habits or EEG characteristics deviate from the average level, the system no longer simply applies general parameters but optimizes based on their specific data to ensure that their attention direction is accurately identified and blind spots are precisely defined.

[0134] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A multimodal fusion method based on cloud-based AI models and brain-computer interfaces, characterized in that, Applied to a terminal, the method includes the following steps: Collect and obtain the target user's brainwave signals and eye muscle signals; The brainwave signals and eye muscle signals are amplified and converted before being analyzed to determine the direction of the target user's eye attention. Based on the target user's eye attention direction and visual field area, the range of the visual field blind spot is obtained; Based on the blind spot information and the IMU's moving speed and angle, the target distance information, target range information, and target area information are calculated. Based on the target distance information, the target range information, and the target area information, obstacles within the blind spot are detected, and it is determined whether a target obstacle has appeared. If so, an alarm will be issued.

2. The multimodal fusion method based on cloud-based AI models and brain-computer interfaces according to claim 1, characterized in that, The amplification of the brainwave signal and the eye muscle signal satisfies the following relationship: Eamp(t) = GE·{Eraw(t) - Eref(t)} EMGamp(t)=GEMG·{EMGraw(t)-EMGref(t)} Where: Eraw(t) is the brainwave signal, EMGraw(t) is the eye muscle signal; GE and GEMG are the amplification gains of the brainwave signal and eye muscle signal, respectively; Eref(t) and EMGref(t) are the reference electrode signals of the brainwave signal and eye muscle signal, respectively. The signal conversion process satisfies the following relationship: E[n] = ADC{Eamp(t)} t = nTs EMG[n] = ADC{EMGamp(t)} t = nTs Where E[n] is the brainwave signal after signal conversion and processing, EMG[n] is the eye muscle signal after signal conversion and processing, Ts is the sampling period, and n is the sampling point number.

3. The multimodal fusion method based on cloud-based AI models and brain-computer interfaces according to claim 2, characterized in that, The step of amplifying and converting the brainwave signals and eye muscle signals before analyzing them to determine the direction of the target user's eye attention includes: Electroocular signal modeling was performed on the ocular muscle signals to obtain the instantaneous pointing angle of the eyeball; Time-frequency analysis of EEG information was performed to extract feature vectors related to eye attention and obtain EEG attention weights. The instantaneous eye pointing angle and the EEG attention weight are weighted and fused to obtain the self-weighted eye attention direction of the target user.

4. The multimodal fusion method based on cloud-based AI models and brain-computer interfaces according to claim 3, characterized in that, The process of modeling the electrooculogram (EOG) signals of the eye muscles to obtain the instantaneous pointing angle of the eyeball includes: Let the horizontal electrooculogram (EOG) signal be H[n] and the vertical EOG signal be V[n]; the horizontal deflection angle θh and the vertical deflection angle θv of the eyeball satisfy the following relationship: θh=kh·H[n]+bh θv=kv·V[n]+bv Where: H[n] and V[n] are the signal mean values ​​within the sliding window, used to eliminate baseline drift; kh and kv are the horizontal / vertical sensitivity coefficients; bh and bv are the calibration offsets; Therefore, the instantaneous pointing angle of the eyeball is obtained: α=(θh,θv).

5. The multimodal fusion method based on cloud-based AI models and brain-computer interfaces as described in claim 3, characterized in that, The step of performing time-frequency analysis on EEG information, extracting feature vectors related to eye attention, and obtaining EEG attention weights includes: Performing a short-time Fourier transform / wavelet transform on E[n] yields the power spectrum: P(f,n)=STFT(E[n]) 2 ; The relative power of the alpha wave (8 Hz–13 Hz) and the beta wave (13 Hz–30 Hz) was extracted. Define EEG attention weights: w atte =Pβ / (Pα+ε); Where ε is a minimal constant to prevent division by zero, w atte The larger the value, the more focused the user's visual attention.

6. The multimodal fusion method based on cloud-based AI models and brain-computer interfaces as described in claim 3, characterized in that, The step of weightedly fusing the instantaneous eye pointing angle and the EEG attention weight to obtain the self-weighted eye attention direction of the target user includes: An eye pointing vector is constructed, and based on the eye pointing vector, EEG attention weights, and the attention direction of the previous frame, an EEG attention confidence weight is obtained. Based on the weighted EEG attention confidence, the fusion vector is normalized and mapped to the user's actual visual field coordinate system to obtain the target user's eye attention direction.

7. The multimodal fusion method based on cloud-based AI models and brain-computer interfaces according to claim 1, characterized in that, If so, after issuing the alarm, the following also includes: The relevant data of the target user is sent to the cloud AI deep learning server so that it can perform individualized learning and continuously improve the model algorithm.

8. A multimodal fusion system based on cloud-based AI models and brain-computer interfaces, characterized in that, include: The cloud-based AI deep learning server is used to provide AI models to the data processing unit for data judgment and to continuously collect terminal data in order to continuously improve the model algorithm. The terminal is used to collect the brainwave signals and eye muscle signals of the target user; after amplifying and converting the brainwave signals and eye muscle signals, the signals are analyzed to determine the direction of the target user's eye attention. Based on the target user's eye attention direction and visual field area, the range of the visual field blind spot is obtained; Based on the blind spot information and the IMU's moving speed and angle, the target distance information, target range information, and target area information are calculated. Based on the target distance information, the target range information, and the target area information, obstacles within the blind spot are detected, and it is determined whether a target obstacle has appeared. If so, an alarm will be issued.

9. The multimodal fusion system based on cloud-based AI models and brain-computer interfaces according to claim 8, characterized in that, The terminal includes: Electroencephalogram (EEG) signal sensor, used to collect and obtain the EEG signals of the target user; An eye muscle signal sensor is used to collect and obtain the eye muscle signals of the target user. A signal amplifier is used to amplify the electroencephalogram (EEG) signal and the eye muscle signal. ADC is used to perform signal conversion processing on the amplified electroencephalogram (EEG) signal and the eye muscle signal; The data processing unit is used to amplify and convert the brainwave signals and eye muscle signals before analysis to determine the direction of the target user's eye attention; based on the target user's eye attention direction and visual field area, to obtain the range of the visual field blind spot; and based on the visual field blind spot information and the movement speed and angle of the IMU, to calculate and obtain target distance information, target range information, and target area information. The detection unit includes a radar and a camera, and is used to detect obstacles within the blind spot area based on the target distance information, the target range information, and the target area information, and to determine whether a target obstacle has appeared. A speaker is used to issue an alarm when a target obstacle is detected.

10. The multimodal fusion system based on cloud-based AI models and brain-computer interfaces according to claim 9, characterized in that, The terminal also includes: The storage unit is used to store the target user's personal difference information and to perform personalized difference compensation for the model in specific scenarios.

Images