A multi-modal state inference system and method based on electroencephalogram signals

Abstract

The application discloses a multi-modal state inference system and method based on electroencephalogram signals, which comprises the following steps: collecting multi-modal data of a user mainly in the form of electroencephalogram signals on the terminal side, extracting features of each mode by a modal feature encoder, fusing features of various modes based on fusion weights by a cross-modal attention fusion module to obtain fused features, rapidly inferring the fused features by an inference module to obtain a predicted state and a confidence, and uploading intermediate features in a data segment to an edge side or a cloud side when the confidence is lower than a threshold value, so as to continue inferring the intermediate features in the data segment to obtain a state inference result. As can be seen, the cloud side is trained, the fusion weights are shared and fused on the edge side, the terminal side power and bandwidth consumption are greatly reduced, the terminal side local rapid inference is adopted, and only deep features instead of original data are uploaded when the confidence is insufficient, so that the inference real-time performance and efficiency are improved, and the privacy is protected, thereby improving the prediction and inference efficiency of the user state in a heterogeneous environment.

Description

Technical Field

[0001] This application relates to the field of physiological signal processing technology, and more specifically, to a multimodal state reasoning system and method based on electroencephalogram (EEG) signals. Background Technology

[0002] With the development of brain-computer interfaces and multimodal perception technologies, user state reasoning based on EEG signals is widely used in smart education, driver monitoring, and medical rehabilitation. Cloud-edge-device collaboration and federated learning support large-scale, privacy-compliant state computation, enabling multi-institutional data collaboration and model generalization optimization. Existing systems mostly adopt a layered architecture of edge data acquisition, edge preprocessing, and cloud training and inference. By improving state recognition accuracy through multimodal fusion, and relying on data encryption and localized processing to meet privacy compliance requirements, this has become the mainstream technology direction for user state computation.

[0003] In real-world mobile or wearable applications, edge devices (such as EEG headbands) have extremely limited computing power, power, and bandwidth resources, making it difficult to run complex deep learning models independently. Furthermore, edge devices struggle to collaborate with the edge and cloud in heterogeneous environments, resulting in low efficiency in predicting and inferring user states.

[0004] Therefore, how to achieve intelligent collaboration of cloud, edge, and terminal computing tasks in heterogeneous environments with limited computing power, power, bandwidth, and other resources, in order to improve the efficiency of predictive reasoning about user status, is an issue that needs attention. Summary of the Invention

[0005] In view of the above problems, this application provides a multimodal state reasoning system and method based on EEG signals to achieve intelligent collaboration of cloud, edge and terminal computing tasks in heterogeneous environments with limited computing power, power, bandwidth and other resources, thereby improving the efficiency of predictive reasoning of user state.

[0006] To achieve the above objectives, the following specific solutions are proposed:

[0007] A multimodal state reasoning system based on electroencephalogram (EEG) signals includes an edge device, an edge device, and a cloud device. The edge device deploys an edge multimodal reasoning model, which is a lightweight model derived from the edge multimodal reasoning model after local sample fine-tuning. The edge multimodal reasoning model includes a modal feature encoder, a cross-modal attention fusion module, and a reasoning module. The cloud device distributes fusion weights for feature fusion to the edge device, and the edge device downloads the fusion weights from the edge device.

[0008] The edge collects multimodal data of the user, with EEG signals as the main modality. The modality feature encoder extracts features of each modality of the multimodal data. The cross-modal attention fusion module fuses the features of various modalities based on the fusion weight to obtain fused features. The inference module quickly infers the fused features to obtain the user's predicted state and its confidence level. When the confidence level is lower than the threshold, the intermediate features of the data segment are uploaded to the edge or the cloud.

[0009] The edge side or the cloud performs reasoning on the intermediate features of the data segment to obtain a state reasoning result. The intermediate features of the data segment are the modal feature vectors extracted by the edge side and the deep semantic embedding vectors generated after integration by the lightweight cross-modal fusion module.

[0010] Optionally, multiple edge sides are connected to the cloud, and the cloud deploys a super network that generates modal heterogeneous sub-networks. The cloud distributes the modal heterogeneous sub-networks to each edge side.

[0011] Each of the edge sides uploads its model architecture descriptor to the cloud;

[0012] The cloud utilizes the hypernetwork to generate initial parameters for the heterogeneous subnetwork of each edge side based on the model architecture descriptor of each edge side;

[0013] Each edge side updates the modal heterogeneous subnetwork according to its initial parameters to obtain the updated modal heterogeneous subnetwork.

[0014] Optionally, the edge side determines the model update amount of the modal heterogeneous subnetwork and extracts the knowledge features of the modal heterogeneous subnetwork;

[0015] The cloud uses the hypernetwork to distill the knowledge features of each edge side into the modal feature sharing layer of the hypernetwork. By aggregating the model update amounts of each edge side, the cloud updates the generation parameters of the hypernetwork. The cloud uses the hypernetwork to generate customized subnetwork parameters based on the model architecture descriptors of each edge side, and then distributes the customized subnetwork parameters to each edge side.

[0016] Each edge side adjusts its modal heterogeneous subnetwork based on the customized subnetwork parameters to obtain the adjusted modal heterogeneous subnetwork.

[0017] Optionally, a unified state space is deployed in the cloud;

[0018] After the edge device extracts the features of each modality through the modality feature encoder, it maps the features of each modality to the unified state space through the mapping layer of that modality to obtain an independent representation of the user's state for that modality.

[0019] Optionally, the cloud deploys a task ontology knowledge base associated with the unified state space. The task ontology knowledge base includes multiple pieces of knowledge information. Each piece of knowledge information contains the user state and its state label in each scenario, as well as the state result of associating the state label based on a pre-built user state analysis knowledge graph.

[0020] When the edge receives a request status result task, it maps the task tag of the request status result task to the unified state space to output the status result in the task ontology knowledge base corresponding to the task tag of the request status result task. The request status result task includes the status tags that exist in the task ontology knowledge base.

[0021] Optionally, the cloud-based deployment and maintenance of a multimodal plugin library, which contains modal plugins corresponding to each modality;

[0022] When the cloud receives a modal analysis request sent by the edge side, it sends modal plugins for several target modalities corresponding to the modal analysis request, as well as fusion strategies for each target modality, to the edge side.

[0023] Optionally, each of the modal plugins is configured with a modal confidence estimator;

[0024] The modality confidence evaluator for each target modality calculates the quality score of the target modality based on the signal-to-noise ratio and signal distribution of the modal signal, and adjusts the attention weight for fusing the target modality in the fusion strategy based on the quality score of the target modality.

[0025] Optionally, a global modal model is deployed in the cloud, and the cloud distributes a global multimodal fusion model dominated by EEG modality to multiple edge sides, with each edge side deploying multiple modal models;

[0026] Each edge side performs local independent training for each modal model, and adjusts the global multimodal fusion model through multimodal fusion data to obtain the model adjustment amount of the global multimodal fusion model. The multimodal fusion data is the data after fusing the modal data output by each modal model.

[0027] The cloud platform updates the global modal model by aggregating the model adjustment amounts of each edge side, obtains global model parameters, and then sends the global model parameters to each edge side.

[0028] Each edge side updates the initial parameters of the global multimodal fusion model based on the global model parameters to obtain the updated global multimodal fusion model, and sends the end-side multimodal inference model that is consistent with the feature representation of the global multimodal fusion model to the end side communicating with that edge side.

[0029] Optionally, the lightweight model is a model that is trained on the cloud and then distributed to the edge side. After feature extraction from other corresponding modal models on the edge side, multimodal fusion is performed. Following local data training and fine-tuning, and lightweight adjustments, the model is then distributed to the edge side. The training process of the backbone model includes:

[0030] Obtain multiple user state inference samples;

[0031] Using the inference process of simulating the inference samples of each user state as the adaptation task, and with the training objective of finding the initial parameter space element with high gradient sensitivity, the basic inference model is trained to obtain the backbone model with the best initial state. When a new user accesses the edge side, the edge side uses fewer than a preset number of label samples of the new user to quickly update the gradient of the backbone model, and combines it with other modal models for training. After extracting features, the modal features are fused and trained in the multimodal fusion model to obtain the edge side multimodal model.

[0032] A multimodal state reasoning method based on electroencephalogram (EEG) signals is applied to the edge side of a multimodal state reasoning system based on EEG signals. The state reasoning system further includes an edge side and a cloud side. The edge side deploys an edge-side multimodal reasoning model, which is a lightweight model downloaded from the edge-side multimodal reasoning model after local sample fine-tuning. The edge-side multimodal reasoning model includes a modal feature encoder, a cross-modal attention fusion module, and a reasoning module. The cloud side downloads fusion weights for feature fusion to the edge side, and the edge side downloads the fusion weights from the edge side.

[0033] The method includes:

[0034] Collect multimodal data from users, primarily based on electroencephalogram (EEG) signals;

[0035] The modality feature encoder extracts features for each modality of the multimodal data;

[0036] The fused features are obtained by fusing features from various modalities through the cross-modal attention fusion module and based on the fusion weights.

[0037] The inference module rapidly infers the fused features to obtain the user's predicted state and its confidence level.

[0038] When the confidence level is lower than the threshold, the intermediate features of the data segment are uploaded to the edge side or the cloud, so that the edge side or the cloud can perform reasoning on the intermediate features of the data segment to obtain the state reasoning result. The intermediate features of the data segment are the modal feature vectors extracted by the edge side and the deep semantic embedding vectors generated after integration by the lightweight cross-modal fusion module.

[0039] Using the above technical solution, this application collects multimodal data of users, primarily based on EEG signals, from the device side. A modal feature encoder extracts features for each modality of the multimodal data. A cross-modal attention fusion module fuses the features of various modalities based on fusion weights to obtain fused features. An inference module rapidly infers from these fused features to obtain the user's predicted state and its confidence level. When the confidence level is below a threshold, the intermediate features of the data segment are uploaded to the edge or cloud for inference by the edge or cloud to obtain the state inference result. The intermediate features are the feature vectors of each modality extracted from the device side and the deep semantic embedding vector generated after integration by the lightweight cross-modal fusion module. Therefore, by using cloud training, lightweight distribution, and edge-side shared fusion weights, the computational power, power consumption, and bandwidth consumption of the device side are significantly reduced, adapting to resource-constrained scenarios for wearable devices. Employing fast local inference on the device side, uploading deep features only when the confidence level is insufficient instead of the original data improves the real-time performance and efficiency of inference while protecting privacy. The clear division of labor between the cloud, edge, and device achieves a balance between accuracy and efficiency in state inference in a heterogeneous environment. Attached Figure Description

[0040] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0041] Figure 1 A system architecture diagram for implementing multimodal state reasoning based on electroencephalogram (EEG) signals is provided in the embodiments of this application;

[0042] Figure 2 A schematic diagram illustrating a cloud-edge-device three-layer collaborative architecture for implementing multimodal state reasoning based on EEG signals, provided in an embodiment of this application;

[0043] Figure 3This is a schematic diagram illustrating a process for implementing multimodal state reasoning based on electroencephalogram (EEG) signals on the edge, as provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] Figure 1 An optional system architecture for implementing multimodal state reasoning based on electroencephalogram (EEG) signals, as provided in the embodiments of this application, is as follows: Figure 1 As shown, the system architecture may include: a terminal 10, an edge 20, and a cloud 30. There can be multiple edge 20s and terminal 10s, while there is only one cloud 30 as the core. The cloud 30 communicates with the terminal 10 through the edge 20s. Each edge 20 can communicate with multiple terminal 10s, and the cloud 30 can communicate with multiple edge 20s.

[0046] Specifically, the edge device 10, serving as the entry point for data acquisition and preliminary processing, can be equipped with various wearable sensing devices, such as lightweight electroencephalography (EEG) headsets (e.g., EEG rings, EEG glasses, headbands), smartwatches, and cameras, microphones, heart rate monitors, eye trackers, etc., or terminal PCs connected to these devices for acquiring data from other modalities. These devices on the edge device 10 can have built-in low-power processing units (such as MCUs or dedicated AI chips) responsible for performing tasks such as signal acquisition and synchronization, data preprocessing, and lightweight feature extraction and inference.

[0047] Specifically, the signal acquisition and synchronization process may include: real-time acquisition of the user's multimodal data, and ensuring time synchronization of cross-modal data through a precise timestamp protocol.

[0048] The data preprocessing process may specifically include: filtering the original signal (such as bandpass filtering and notch filtering), artifact removal (such as eye movement and electromyography interference), and segmentation.

[0049] The lightweight feature extraction and inference process can specifically include: configuring a corresponding lightweight feature encoder for each access modality (such as facial expression, voice, eye movement, heart rate, etc.), and configuring an edge-side multimodal inference model. All edge-side multimodal inference models follow the principles of low power consumption, lightweight design, and high compression to adapt to edge computing power limitations. The edge-side multimodal inference model is deployed on edge 10 and is a lightweight model that is fine-tuned from the edge-side multimodal inference model 20 using local samples. The edge-side multimodal inference model includes a modal feature encoder, a cross-modal attention fusion module, and an inference module. The EEG modality deploys lightweight core models (such as EEGNetv4 and Mini-DeepConvNet) to handle core feature extraction and basic state inference on the edge. Other modalities, such as facial expressions, speech, eye tracking, and heart rate, each deploy dedicated lightweight models (e.g., a simplified version of MobileNetV2 for facial expressions, MFCC + a lightweight RNN for speech, a lightweight CNN for keypoint detection for eye tracking, and a PPG feature extraction MLP for heart rate). All non-EEG modal models are packaged as standardized modal plugins, which can be loaded / unloaded on the edge as needed. In the lightweight cross-modal attention fusion module of the edge multimodal inference model, the output features of each lightweight model are initially fused, and rapid inference is completed in the inference module. The fusion weights are trained on the cloud (30) and then distributed to the edge (20). After local fine-tuning on the edge (20), they are downloaded to the lightweight fusion module on the edge (10) to achieve the minimum necessary parameter set for fusion. The fusion weights can be fine-tuned locally.

[0050] Edge computing systems can be deployed within the organization where the data originates, such as hospitals, schools, research centers, or corporate fleets, typically as small servers or edge computing clusters. Their core function is to undertake regional computing and management tasks, including data aggregation and management, local model training and federated collaboration, task ontology mapping, and relay inference, all while ensuring the data remains within the organization.

[0051] Specifically, the data aggregation and management process may include: aggregating intermediate features or preprocessed data uploaded by multiple end devices within the organization, and performing unified formatting and label alignment.

[0052] The process of local model training and federated collaboration can specifically include: using the organization's data to train or fine-tune the global model distributed from the cloud 30 locally, calculating model gradients or updating weights. Each edge device 20 maintains local encoder model parameters and multimodal fusion model parameters for all connected modalities. The model parameters correspond one-to-one with the modal plugins / global fusion models on the cloud 30 and can be flexibly maintained according to the organization's modal configuration. Model parameters are not loaded for modalities that are not connected, saving storage and computing resources on the edge device 20.

[0053] The process of task ontology mapping can specifically include mapping specific task tags within the organization (such as "distracted in class" or "drowsy while driving") to a unified state space defined in the cloud.

[0054] The relay inference process may specifically include: when the confidence level of the preliminary prediction inference result of the end side 10 is insufficient to assess the reliability of the result, the edge side 20 can provide the end side 10 with low latency and high accuracy verification inference service.

[0055] Cloud30, serving as the brain of the entire system, consists of a large-scale server cluster and is responsible for global model maintenance, federated learning coordination, and platform management. Specifically, it can perform global model maintenance, federated learning coordination, multimodal plugin management, privacy and security auditing, and unified state space and task ontology knowledge base management.

[0056] Specifically, the process of maintaining the global model may include maintaining a cross-organizational and cross-user general EEG backbone model and a multimodal fusion model.

[0057] The coordination process of federated learning can specifically include: acting as the central server for federated learning, it is responsible for aggregating encrypted or noisy model updates uploaded by each edge node (such as using aggregation strategies like FedAvg, Fed-Dyn, and Federated MAML), and distributing the updated global model. The cloud does not access any raw data, therefore it does not perform model training based on raw data. However, it is not simply a matter of updating parameters by adding them together, but rather a complex training operation centered around the aggregation and optimization of the global model. This includes fine-tuning the training by combining the aggregated model parameters with synthetic data or publicly available datasets without privacy risks; performing optimization operations such as regularization / early stopping / hyperparameter tuning on the aggregated model; and conducting federated MAML outer loop meta-training for the two-layer modeling framework to optimize the initial parameters of the global model.

[0058] The process of managing multimodal plugins can specifically include: managing feature encoder plugins of various modalities and providing on-demand configuration and distribution functions.

[0059] The privacy and security audit process may specifically include: managing access control policies across the entire chain and logging all operations for auditing and accountability.

[0060] The process of managing a unified state space and task ontology knowledge base can specifically include: defining and maintaining a standardized user state space that is unified across all modalities, and a task ontology knowledge base built on ontology and knowledge graphs to achieve multi-task knowledge transfer and semantic unification of task tags.

[0061] The unified state space is a multi-dimensional, numerical, globally unified state space constructed in the cloud. Its core dimensions include core user state indicators such as valence, arousal, emotional intensity, cognitive load, attention, and fatigue. Each dimension has a clear numerical range and semantic definition (e.g., valence: -1 to 1, negative for negative emotions, positive for positive emotions), serving as a common language for all modalities. All modalities accessed by the platform (EEG / facial / voice / eye movement / heart rate, etc.) ultimately have their feature representations mapped to this unified state space: each modality's feature encoder extracts its original features and maps the feature vectors to the unified state space through a modality-specific mapping layer, obtaining an independent representation of the user state for that modality; then, an EEG-led attention fusion network fuses the representations mapped to the state space from each modality, obtaining a comprehensive multi-modal user state representation. This representation remains within the unified state space, and the final user state prediction results are output from this space. All application scenarios follow the same set of state space mapping rules, ensuring the comparability and transferability of user state calculation results across scenarios.

[0062] The task ontology knowledge base is a cross-institutional and cross-scenario task tag semantic knowledge base built on ontology and knowledge graphs. It serves as a bridge connecting the personalized task tags of various institutions with the unified state space in the cloud. Its core functions are reflected in three aspects: semantic unification, knowledge transfer, and task standardization. Semantic unification: It maps personalized task tags from different institutions and scenarios (such as "classroom inattentiveness" in schools and "driving drowsiness" in motorcades) to unified semantic concepts (such as "low attention" and "high fatigue"), solving the problem of semantic silos in task tags; Knowledge transfer: Through the semantic relationships in the ontology knowledge base (such as "yawning" → "fatigue" → "reduced cognitive load"), it enables zero-sample / few-sample knowledge transfer of models across tasks and scenarios; Task standardization: It provides standardized semantic templates for task access in new institutions / new scenarios, reducing the model adaptation and integration costs for new tasks and enabling the platform to be plug-and-play.

[0063] The task ontology knowledge base is associated with a unified state space. The task ontology knowledge base includes multiple pieces of knowledge information, each containing the user's state and its state label for each scenario, as well as the state result associated with the state label based on a pre-built user state analysis knowledge graph. Therefore, when the edge device 20 receives a request for a state result task, it can map the task label of the request for a state result task to the unified state space to output the state result corresponding to the task label of the request for a state result task in the task ontology knowledge base. The request for a state result task includes the state labels existing in the task ontology knowledge base.

[0064] The task ontology knowledge base can adopt a layered construction strategy of top-level standardization + bottom-level personalization. It uses OWL (Web Ontology Language) for formal description and Neo4j to build a knowledge graph for semantic storage and retrieval. Specific construction steps include:

[0065] First, define the core ontology concept layer (cloud-based standardization): Based on the core needs of user state computation in the cloud, define three categories of top-level core concepts. All core concepts are assigned a unique semantic ID and a clear definition, forming a standardized top-level framework for the ontology knowledge base. Among them, the user state category includes emotions, cognitive load, attention, fatigue, etc. (corresponding one-to-one with the dimensions of the unified state space); the behavioral characteristics category includes yawning, blinking frequency, abnormal EEG rhythms, etc. (observable behavioral / physiological characteristics of each modality); and the application scenario category includes smart education, driving safety, medical rehabilitation, etc. (the core application scenarios of the platform).

[0066] Second, define the semantic relationships between concepts (cloud-based 30 standardization): use OWL relational attributes to define standardized semantic relationships for core concepts. Core relationships include causal relationships, mapping relationships, subordinate relationships, and association relationships. All semantic relationships are assigned a weight value of 0 to 1 to indicate the strength of the association. The relationships are labeled by domain experts and optimized through model iteration.

[0067] Third, edge-side personalized task tag mapping (edge-side localization): Each edge side maps its own personalized task tags / observation indicators to the core ontology concept layer defined in the cloud. The edge side only needs to complete the mapping once, and the cloud will synchronize the mapping relationship to the ontology knowledge base, forming a two-layer ontology structure of core concepts + edge-side personalized tags.

[0068] Fourth, instantiation of the ontology knowledge base and construction of the knowledge graph: Based on the above concepts and relationships, a knowledge graph is used to realize the instantiation and storage of the ontology knowledge base. Nodes are core ontology concepts, edge-side personalized task tags, and user status indicators, while edges are semantic associations and weight values ​​between nodes. The storage engine uses the Neo4j graph database, which supports efficient semantic query and association reasoning. At the same time, the ontology knowledge base supports incremental updates. When a new edge / task is accessed, only new nodes and edges need to be added, without reconstructing the entire ontology knowledge base.

[0069] Fifth, dynamic optimization and self-learning of the ontology knowledge base: During platform operation, the ontology knowledge base is dynamically optimized through model feedback and a semantic auto-mapping engine based on the Large Language Model (LLM). The model feedback compares the user state prediction results of the multimodal model with the semantic associations of the ontology knowledge base. If there is a discrepancy, the relation weights are automatically adjusted. The semantic auto-mapping engine based on the Large Language Model (LLM) periodically reviews and corrects the mapping relationships, ultimately achieving self-learning and self-optimization of the ontology knowledge base to adapt to the task requirements of different scenarios.

[0070] In actual use, the edge side 20 can upload local task tags to the cloud 30 → the cloud 30 performs semantic queries in the ontology knowledge base → matches the corresponding core user state concepts → maps to a unified state space → guides the feature extraction and fusion training of the model, realizing cross-task knowledge transfer.

[0071] Understandably, the unified state space (quantization benchmark) and the task ontology knowledge base (semantic association) provide semantic directions and training objectives for feature extraction of each modal encoder model, and provide semantic basis for fusion weights and output normalization standard for multimodal fusion models. This enables the training, inference, and federated collaboration of edge / cloud three-level models to have a unified benchmark, ultimately achieving cross-institutional, cross-task, and cross-modal knowledge transfer, while ensuring the platform's scalability, generalization, and collaboration.

[0072] For example Figure 2 , Figure 2 This paper illustrates a three-layer collaborative architecture of cloud, edge, and device to achieve multimodal state reasoning based on EEG signals. The device 10 is equipped with a lightweight wearable sensing terminal, responsible for EEG-based and multimodal-assisted data acquisition, preprocessing, lightweight feature extraction, and local rapid reasoning. Only intermediate features are uploaded, while the raw data is stored locally. The edge 20 acts as a regional federated client, aggregating device data, performing local model training and fine-tuning, completing task label ontology mapping, and distributing models and fusion weights to the device. The cloud 30 serves as the system hub, maintaining a unified state space and task ontology knowledge base, coordinating federated learning aggregation, managing multimodal plugins, supporting heterogeneous architecture collaboration through a topology-aware hypernetwork, and coordinating resource scheduling and end-to-end privacy and security auditing. This three-layer linkage achieves privacy-compliant and highly efficient collaborative user state computation.

[0073] based on Figure 1 The system architecture shown and Figure 2 The diagram illustrates the architecture of a cloud-edge-device three-layer collaborative solution for multimodal state reasoning based on EEG signals, along with a description of the respective functions performed by the device (device 10), edge (edge ​​20), and cloud (cloud 30). Figure 3 This illustration shows a flowchart of an edge-side implementation of a multimodal state reasoning method based on electroencephalogram (EEG) signals, as provided in an embodiment of this application. (Refer to...) Figure 3The process may include:

[0074] Step S101: Collect multimodal data of the user with EEG signals as the main modality.

[0075] Specifically, with the user's authorization and consent to collect biometric information, the edge device 10 can collect the user's raw EEG signals in real time through a lightweight EEG head-mounted device, and simultaneously call peripherals such as cameras, microphones, heart rate sensors, and eye trackers to collect auxiliary modal data such as facial expressions, voice, heart rate, and eye movement trajectories; a unified timestamp protocol is used to complete the accurate alignment of multimodal data, ensuring that the complete data stream of EEG as the main modality and other modalities as auxiliary pluggable inputs is acquired synchronously.

[0076] Step S102: Extract the features of each mode of the multimodal data using a modal feature encoder.

[0077] Specifically, the edge device 10 can independently extract features from different modalities of data, such as EEG, facial expressions, speech, heart rate, and eye movements, by calling the corresponding modal feature encoders. EEG signals are the core dominant modality, employing a lightweight dedicated encoder to extract key features such as rhythm, energy, and time-frequency domain characteristics. Other auxiliary modalities use corresponding standardized plug-in encoders to extract their own specific features. Each encoder follows a unified interface specification, outputting modal feature vectors aligned with both dimensions and semantics, providing standardized input for subsequent cross-modal attention fusion.

[0078] Furthermore, after extracting the features of each modality through the modality feature encoder, the edge device 10 can map the features of each modality to a unified state space through the mapping layer of that modality to obtain an independent representation of the user's state for that modality.

[0079] Step S103: The features of various modalities are fused by the cross-modal attention fusion module and based on the fusion weights to obtain the fused features.

[0080] The fusion weights can be distributed by the cloud 30 after global analysis. Specifically, the cloud 30 can distribute the fusion weights used for feature fusion to the edge side, and the edge 10 downloads the fusion weights from the edge side 20.

[0081] Specifically, through the cross-modal attention fusion module, EEG modality features are used as the core dominant channel. Based on the fusion weights downloaded from the edge side 20, dynamic weighting and attention interaction calculations are performed on each modality feature. The cross-modal attention fusion module automatically evaluates the confidence of each auxiliary modality feature, assigns feature importance according to weight, maps multi-path heterogeneous modality features to a unified state space for semantic alignment and global fusion, and finally outputs high-dimensional fusion features for state reasoning.

[0082] Step S104: Quickly infer the fused features through the inference module to obtain the user's predicted state and its confidence level.

[0083] Specifically, the inference module receives unified representation features obtained through cross-modal attention fusion, performs fast forward inference calculations using a lightweight neural network structure, and completes classification and regression calculations of user states such as emotion, fatigue, cognitive load, and attention level on the edge side based on model parameters distributed from the cloud and optimized locally. It simultaneously outputs the corresponding predicted state results and confidence scores, providing a basis for subsequent hierarchical scheduling.

[0084] Step S105: When the confidence level is lower than the threshold, the intermediate features of the data segment are uploaded to the edge side or the cloud so that the edge side or the cloud can perform inference on the intermediate features of the data segment and obtain the state inference result.

[0085] Among them, the intermediate features of the data segment can be the modal feature vectors extracted by the edge 10 and the deep semantic embedding vectors generated after integration by the lightweight cross-modal fusion module. The intermediate features of the data segment are located at the hierarchical exit of the segmented inference architecture. While retaining the key state information related to the task, it realizes the physical desensitization and dimensional compression of the original physiological waveform, thereby supporting cloud or edge nodes to perform high-computation-intensive subsequent verification calculations while maintaining the continuity of the inference context.

[0086] It is important to note that highly sensitive individual information contained on the edge side 10 will not be uploaded to the edge side 20 or the cloud 30. Instead, it will be permanently stored in a local adaptation layer with a very small number of parameters to achieve a balance between optimizing computing resources and protecting privacy.

[0087] Specifically, when the confidence level of the user state prediction output by the inference module of the edge 10 is lower than the preset threshold, the system automatically starts the hierarchical review and inference mechanism, encrypts and uploads the intermediate features of the data segment corresponding to the current time period to the relevant edge 20 or uploads them to the cloud 30 through the edge 20. The edge 20 or the cloud 30 then uses a model with higher precision and greater computing power to complete the deep inference and output the final reliable state inference result.

[0088] For example, if the confidence level is higher than the preset high threshold (0.95), the result is considered reliable, and it is directly adopted and responded to locally to achieve the lowest latency; if the confidence level is lower than the preset low threshold (0.7), the result is considered unreliable, and the intermediate features of the data segment are uploaded to edge nodes with stronger computing power or the cloud, and a more complex model is called for high-precision verification; if the confidence level is between the two, it is decided whether to upload based on the current resource status of the device (battery power, network status).

[0089] This embodiment provides a multimodal state reasoning method based on EEG signals. It collects multimodal data from the user, primarily EEG signals, on the edge. A modal feature encoder extracts features for each modality of the multimodal data. A cross-modal attention fusion module fuses these features based on fusion weights to obtain fused features. The reasoning module rapidly infers from these fused features to obtain the user's predicted state and its confidence level. When the confidence level is below a threshold, the intermediate features of the data segment are uploaded to the edge or cloud for further reasoning to obtain the state reasoning result. The intermediate features are deep semantic embedding vectors generated after pre-training feature extraction operators on the EEG signals. Therefore, by using cloud training, lightweight distribution, and edge-side shared fusion weights, the method significantly reduces edge-side computing power, power consumption, and bandwidth consumption, making it suitable for resource-constrained scenarios in wearable devices. Employing fast local reasoning on the edge, uploading deep features only when the confidence level is insufficient, rather than the original data, improves reasoning real-time performance and efficiency while protecting privacy. The clear division of labor between the cloud, edge, and edge achieves a balance between accuracy and efficiency in state reasoning in a heterogeneous environment.

[0090] In some embodiments of this application, the lightweight backbone model mentioned in the foregoing embodiments is further described. The lightweight backbone model can be a lightweight model that is trained on the cloud 30 and then distributed to the edge side 20, and the backbone model is adjusted on the edge side 20 and then distributed to the end side 10.

[0091] Specifically, the training process of the backbone model includes:

[0092] Obtain multiple user state inference samples.

[0093] The adaptation task is to simulate the reasoning process of various user state inference samples. The training objective is to find the initial parameter space element with high gradient sensitivity. The basic inference model is trained to obtain the backbone model with the best initial state. When a new user accesses the edge side 20, the edge side 20 uses the new user's label samples (less than a preset number) to quickly update the gradient of the backbone lightweight model, thus obtaining a new edge-side multimodal inference model adapted to the new user.

[0094] Specifically, in Cloud30, the backbone model no longer employs traditional aggregation methods like FedAvg, which only aim to minimize the global average loss. Instead, it introduces a meta-learning paradigm (such as a federated variant of the Model-Agnostic Meta-Learning (MAML) algorithm) to train a general EEG backbone model. The core idea is to treat the data from each client (or a group of similar clients) as an independent task. The optimization goal of the Cloud30 server is no longer to learn a model that performs best on average across all data, but rather to learn a meta-model with optimal initialization. The training objective is to optimize an initialization parameter space theta with extremely high gradient sensitivity by simulating massive individual adaptation tasks. This initialization parameter space theta encapsulates the meta-features of EEG signals across individual dimensions, representing a general neurophysiological representation stripped of individual physical differences. This training objective allows the platform to quickly converge along the pre-defined optimal adaptation path using this theta as a starting point when facing unknown new users, through a very small number of unsupervised or supervised samples. The model state with this initialization parameter space theta represents the optimal initialization state. This meta-model possesses high generalization and plasticity, serving as an excellent starting point. When facing any new user (or task), it can quickly converge to a high-performance personalized model with only one or a few gradient descent runs using a minimal amount of local data. This learning-to-adapt mechanism solves the Non-IID problem of EEG data across users because it actively learns how to overcome differences in data distributions during the training phase, thereby acquiring robust cross-subject representation capabilities.

[0095] On each user's endpoint 10 or its associated edge endpoint 20, a lightweight set of individual adaptation parameters is maintained for that user. This adaptation layer can be one or more fully connected layers, an affine transformation module, or a more advanced batch-specific batch normalization layer. Its function is to accurately map the shared feature representation extracted by the general backbone model to the user's unique state representation space. The individual adaptation process is as follows: When a new user joins, the pre-trained meta-model (backbone model) is first downloaded from the cloud. Subsequently, the user only needs to complete a short calibration task containing a small number of labeled samples. The system uses these samples to perform rapid gradient updates on the individual adaptation layer (or jointly fine-tuning the top layer of the backbone model) locally.

[0096] Understandably, since the metamodel already possesses rapid adaptability, this process typically requires only a few iterations, significantly reducing reliance on locally labeled samples, substantially lowering the calibration cost for new users, and improving user experience. The parameters of this adaptation layer are updated locally only and do not participate in any cross-institutional parameter sharing, thus fundamentally protecting the privacy of individual characteristics. For systems without initial labeled data or requiring initial setup, the system can first initiate a self-supervised alignment mechanism that integrates perception and adaptation. That is, in the initial stages of user device use (e.g., the first 3-5 seconds), the system collects unlabeled raw physiological signals in real time and uses self-supervised contrastive learning or Domain Adversarial Neural Networks (DANNs) to quickly map the new user's signal distribution to a global general feature space. This ensures that the system has completed preliminary basic adaptation before the user performs any explicit calibration tasks, achieving cold start optimization with zero calibration.

[0097] In some embodiments of this application, the cloud 30 of the above-mentioned multimodal state reasoning system based on EEG signals is further described. Specifically, the cloud 30 can also deploy and maintain a multimodal plugin library, which can contain modal plugins corresponding to each modality.

[0098] Specifically, Cloud30 can define a unified plugin interface specification for all supported modalities (such as EEG, facial images / videos, speech, eye tracking, heart rate / PPG, functional near-infrared spectroscopy fNIRS, etc.). This specification details the input data format, sampling rate, time synchronization protocol, preprocessing flow, and the dimensions and semantics of the output feature vectors. Each modality corresponds to an independent feature encoder (such as a CNN for images, an RNN for speech, or a general-purpose Transformer). These encoders are encapsulated as modality plugins and are uniformly managed and distributed by Cloud30.

[0099] Based on this, when the cloud 30 receives a modal analysis request sent by the edge side, it can send modal plugins for several target modalities corresponding to the modal analysis request, as well as fusion strategies for each target modality, to the edge side 20.

[0100] Specifically, after receiving the modality configuration and analysis request sent by the edge device 20 in real time, the cloud device 30 parses and verifies the scenario type, task requirements, device capabilities, and modality access list carried in the request. Based on the request, it matches the modality plugin library and fusion strategy template library uniformly maintained by the cloud device 30, automatically selects the EEG dominant modality and several target modalities that are suitable for the task, and encrypts and encapsulates the standardized modality plugins, modality encoder parameters, feature mapping rules, and interface call specifications for the corresponding target modality. Simultaneously, it generates and distributes a multimodal fusion strategy adapted to the task, including the initial value of fusion weight, cross-modal attention allocation rules, modality confidence assessment mechanism, and modality failure smoothing degradation strategy. After receiving the request, the edge device 20 completes the plugin loading, parameter initialization, and fusion strategy configuration, and can quickly enable the corresponding multimodal analysis capabilities, realizing on-demand modality access, plug-and-play functionality, and flexible expansion.

[0101] Based on this, and considering the need to improve the system's perception performance in extreme environments, each modal plugin can be configured with a modal confidence evaluator. Specifically, the modal confidence evaluator of each target modal plugin can calculate the quality score of the target modality based on the signal-to-noise ratio and the degree of signal distribution offset of the modal signal of the target modality, and adjust the attention weight of the target modality in the fusion strategy based on the quality score of the target modality.

[0102] For example, when the confidence level of a certain modality (such as eye movement) decreases due to the influence of ambient light, the fusion network automatically reduces the attention weight of that modality and shifts the prediction energy to the EEG backbone channel. This mechanism ensures that even in extreme environments (such as insufficient lighting or noisy backgrounds), the system can achieve a smooth and imperceptible performance degradation from multimodal fusion to the EEG-dominant core modality.

[0103] The multimodal state reasoning method based on EEG signals provided in this application constructs a flexible and scalable multimodal platform centered on EEG. Standardized plug-in modal interfaces and an EEG-led intelligent fusion strategy enable the platform to combine multiple modalities on demand according to specific application scenarios, significantly improving the accuracy and dimensionality of user state recognition. Its support for personalized federated learning architecture further enables it to flexibly handle complex situations involving modal heterogeneity and missing features in the real world. Through the generation of heterogeneous subnetworks via hypernetworks and knowledge distillation, efficient knowledge sharing between edge devices of different modalities and topologies is achieved.

[0104] In some embodiments of this application, the process of performing task-level multimodal configuration and personalized federated learning on the above-mentioned multimodal state reasoning system based on electroencephalogram (EEG) signals is described. Specifically, this process may include:

[0105] 1) First, cloud 30 deploys a super network for generating modal heterogeneous subnetworks, and cloud 30 distributes the modal heterogeneous subnetworks to each edge side 20.

[0106] Specifically, Cloud30 allows administrators to flexibly configure the required modal combinations and corresponding fusion strategies according to the needs of different application tasks. EEG is the dominant modality, requiring all edge devices accessing this platform to deploy an EEG encoder, while non-EEG modal encoders are selected as needed based on the task. For example, the "driver monitoring" task can be configured as "EEG + eye movement + facial expression", while the "online learning attention analysis" task can be configured as "EEG + heart rate".

[0107] Furthermore, in the process of multimodal federated learning, a Topology-aware HyperNetwork (TAHN) can be introduced, enabling different edge sides to have heterogeneous model architectures due to their different available modalities, while still achieving effective knowledge sharing and collaborative training among them. The THAN hypernetwork, deployed in the cloud, is a core component for solving the problem of heterogeneous model architectures caused by modal heterogeneity on the edge side. Its core design involves the hypernetwork generating heterogeneous subnetworks and topology-aware knowledge distillation. TAHN is a large-scale generative neural network containing a modality feature sharing layer and a topology-aware generation layer. The modality feature sharing layer learns the general feature representations of all modalities and serves as the core knowledge carrier across edge sides and architectures. The topology-aware generation layer contains various model topology generation parameters (layer width, depth, number of attention heads, fusion strategy, etc.) and can generate subnetwork parameters adapted to the heterogeneous architecture of the edge side based on its architecture descriptor.

[0108] 2) Each edge side 20 uploads its model architecture descriptor to the cloud 30.

[0109] Specifically, the model architecture descriptor can include topology information such as access modality, number of model layers, layer width, and fusion strategy. The edge device 20 can encrypt and upload its own model architecture descriptor to the cloud 30.

[0110] 3) Cloud 30 uses the hypernetwork to generate the initial parameters of the heterogeneous subnetwork of each edge side 20 based on the model architecture descriptor of each edge side 20.

[0111] Specifically, the cloud-based TAHN supernetwork can generate customized initial parameters for each edge side 20 of the heterogeneous subnetwork based on this descriptor.

[0112] 4) Each edge side 20 updates the modal heterogeneous subnetwork according to its initial parameters to obtain the updated modal heterogeneous subnetwork.

[0113] Specifically, after receiving the customized sub-network parameters generated by TAHN, the edge device 20 can combine the customized sub-network parameters with local multimodal data for local training, thereby updating the modal heterogeneous sub-network.

[0114] Furthermore, considering the further training of the modal heterogeneous subnetworks of the edge side 20 of the multimodal state reasoning system based on EEG signals for edge-cloud federated learning, the specific scheme continues to include:

[0115] 5) On the edge side 20, determine the model update amount of the modal heterogeneous subnetwork and extract the knowledge features of the modal heterogeneous subnetwork.

[0116] Specifically, during the training and updating of the modal heterogeneous subnetwork, the edge side 20 can simultaneously obtain the model update amount of the modal heterogeneous subnetwork and extract the knowledge features of the modal heterogeneous subnetwork (such as intermediate layer features and prediction probability distribution).

[0117] 6) The cloud 30 distills the knowledge features of each edge side 20 into the modal feature sharing layer of the super network through the super network. By aggregating the model update amount of each edge side 20, the generation parameters of the super network are updated. The super network generates customized sub-network parameters based on the model architecture descriptor of each edge side 20, and distributes the customized sub-network parameters to each edge side 20.

[0118] Specifically, the TAHN supernetwork on the cloud (30 layers) distills the knowledge features of heterogeneous models on each edge side into the modal feature sharing layer of TAHN through topology-aware knowledge distillation, enabling knowledge extraction and sharing between heterogeneous architectures. Simultaneously, the cloud (30 layers) aggregates the model updates from each edge side (20 layers) to optimize the generation parameters of TAHN, making the subsequently generated subnetworks more suitable for the scenarios of each edge side (20 layers). The cloud-optimized TAHN supernetwork then regenerates updated customized subnetwork parameters based on the model architecture descriptors of each edge side (20 layers) and distributes them to each edge side (20 layers) for the next round of federated training.

[0119] 7) Each edge side 20 adjusts its modal heterogeneous subnetwork based on customized subnetwork parameters to obtain the adjusted modal heterogeneous subnetwork.

[0120] In some embodiments of this application, the process of cloud-edge federated collaborative training is described. During cloud-edge federated collaborative training, the edge device 10 only performs inference and feature extraction and does not participate in training. The cloud device 30 deploys a global modality model and distributes a global multimodal fusion model to multiple edge devices 20. Each edge device 20 deploys multiple modality models (each modality model corresponds to one modality). Among them, the cloud maintains a multi-layer model, which includes a meta-model layer (containing an EEG meta-learning backbone model, responsible for extracting cross-individual common features), a generation layer (containing a TAHN super network, responsible for generating adapted sub-models for edge devices with different computing power), a fusion layer (containing a cross-modal attention fusion model, responsible for global modality weight strategy management), a plug-in layer (containing a pre-trained model library of heterogeneous modalities (face, voice, eye tracking, etc.)), and a cognitive layer (responsible for semantic mapping of labels). The global multimodal fusion model distributed by the cloud device 30 to multiple edge devices 20 is the fusion layer of this multi-layer model. The cloud-edge federated collaborative training process can include modality-independent training and multimodal fusion training, specifically including:

[0121] S1. Each edge side 20 performs local independent training for each modal model, and adjusts the global multimodal fusion model through multimodal fusion data to obtain the model adjustment amount of the global multimodal fusion model.

[0122] The multimodal fusion data refers to the fused modal data output from each modal model. Local independent training can be gradient descent training over a limited number of rounds (e.g., 1-5 rounds). Model tuning can include the difference between the gradient tuning and weight tuning of the global multimodal fusion model.

[0123] Specifically, during the modality-independent training process on the edge side 20, each modality (EEG / face / speech / eye tracking) is trained locally on the edge side. Due to the different rules for the acquisition and preprocessing of modal data, the models of each modality do not interfere with each other during training and only learn the feature representation of their own modality. After completing the local model training for each modality, the edge side 20 uses the multimodal fusion data of its own institution to fine-tune the global multimodal fusion model (EEG-led attention fusion network) distributed by the cloud 30, and learns the modality fusion weights in the context of its own institution.

[0124] S2 and Cloud 30 update the global modal model by aggregating the model adjustment amounts of each edge side 20, obtain the global model parameters, and then distribute the global model parameters to each edge side 20.

[0125] Specifically, the cloud 30 collects the encrypted model updates from each edge 20, and uses aggregation strategies such as FedAvg / Fed-Dyn / Federated MAML for global aggregation to update the parameters of the global modal model / fusion model. The cloud 30 then encrypts and distributes the updated global model parameters to all edge 20s.

[0126] S3. Each edge side 20 updates the initial parameters of the global multimodal fusion model based on the global model parameters to obtain the updated global multimodal fusion model, and sends the end side multimodal inference model consistent with the feature representation of the global multimodal fusion model to the end side 10 that communicates with the edge side 20.

[0127] Specifically, edge side 20 synchronously updates the initial parameters of its local global multimodal fusion model and enters the next round of federated training. At the same time, the edge side 10's edge multimodal inference model is updated by its corresponding edge side 20. Edge side 20 quantizes / compresses the edge side 10's edge multimodal inference model based on its local training results and then pushes it to ensure the consistency of feature representation between the edge side 10's edge multimodal inference model and the edge side 20's global multimodal fusion model.

[0128] In some embodiments of this application, considering that the endpoint 10 and edge 20 may encounter resource pressure when inferring and predicting the user's state, this application introduces a scheme for resource perception and dynamic task scheduling in the above-mentioned multimodal state reasoning system based on EEG signals. This scheme may specifically include:

[0129] Both the endpoint 10 and edge 20 can periodically monitor their own resource status (CPU / GPU load, remaining power, network bandwidth, and latency) and report this status information to the resource scheduling module in the cloud 30. The scheduler in the cloud 30 can dynamically select the optimal operating mode for each endpoint 10 and every edge 20 based on a global resource view and task priorities. The operating module may include:

[0130] Pure Local Mode: When network conditions are poor or to save power, all calculations are performed locally.

[0131] Edge-cloud collaborative mode: When higher accuracy is required and network conditions permit, hierarchical inference is adopted or part of the training task is uploaded to the edge side 20 or the cloud 30.

[0132] Federated learning scheduling: During the training phase, the cloud-based scheduler can dynamically select a subset of clients to participate in this round of federated learning based on the resource status and data quality of the nodes, and adjust the number of iterations of their local training to optimize overall training efficiency and fairness.

[0133] The multimodal reasoning method based on EEG signals provided in this application achieves intelligent allocation of computing tasks among the terminal, edge, and cloud through hierarchical reasoning and resource-aware dynamic scheduling mechanism. It effectively balances system response latency, device energy consumption, and computing accuracy, enabling it to be widely adapted to various application environments from high-performance servers to low-power wearable devices.

[0134] In some embodiments of this application, a scheme for unified management of tasks, organizations, and devices in the above-mentioned multimodal state reasoning system based on EEG signals is introduced. Specifically, the scheme may include task and tag ontology management and unified access to heterogeneous devices and sensors.

[0135] In terms of task and tag ontology management, semantic modeling can be performed based on task tag systems under different institutions and experimental paradigms. For example, "driver yawning" and "student dozing off in class" can both be mapped to the concept of "fatigue" in the ontology library, enabling the model to understand the semantic relationships between different task tags, thereby achieving zero-shot or few-shot knowledge transfer across tasks.

[0136] The task ontology library can incorporate a semantic auto-mapping engine based on Large Language Model (LLM). When a new organization joins the platform, the LLM engine can automatically perform contextual semantic parsing on the organization's custom non-standard tags (such as "daydreaming" behavior defined by a school). The engine automatically associates these tags with top-level core concepts in the ontology library (such as "attention deficit") and generates corresponding mapping weight values, which are then stored in the Neo4j knowledge graph. This lowers the barrier to entry for new tasks and enables rapid knowledge transfer with "zero semantic burden" across industry scenarios.

[0137] Regarding the unified access of heterogeneous devices and sensors, the system can be configured to uniformly encapsulate sensors from different manufacturers and of different types (such as EEG+EOG combinations or EEG+fNIRS combinations). Through standardized data interfaces, time synchronization protocols, and feature alignment strategies, these heterogeneous devices are seamlessly integrated into a unified state computing framework, greatly enhancing the system's compatibility and scalability.

[0138] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0139] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0140] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal state reasoning system based on electroencephalogram (EEG) signals, characterized in that, The system includes a device side, an edge side, and a cloud side. The device side is equipped with a device side multimodal inference model, which is a lightweight model that is fine-tuned locally by the edge side multimodal inference model. The device side multimodal inference model includes a modal feature encoder, a cross-modal attention fusion module, and an inference module. The cloud side distributes fusion weights for feature fusion to the edge side, and the device side downloads the fusion weights from the edge side. The edge collects multimodal data of the user, with EEG signals as the main modality. The modality feature encoder extracts features of each modality of the multimodal data. The cross-modal attention fusion module fuses the features of various modalities based on the fusion weight to obtain fused features. The inference module quickly infers the fused features to obtain the user's predicted state and its confidence level. When the confidence level is lower than the threshold, the intermediate features of the data segment are uploaded to the edge or the cloud. The edge side or the cloud performs reasoning on the intermediate features of the data segment to obtain a state reasoning result. The intermediate features of the data segment are the modal feature vectors extracted by the edge side and the deep semantic embedding vectors generated after integration by the lightweight cross-modal fusion module.

2. The system according to claim 1, characterized in that, Multiple edge sides are connected to the cloud, and the cloud deploys a super network that generates modal heterogeneous sub-networks. The cloud distributes the modal heterogeneous sub-networks to each edge side. Each of the edge sides uploads its model architecture descriptor to the cloud; The cloud utilizes the hypernetwork to generate initial parameters for the heterogeneous subnetwork of each edge side based on the model architecture descriptor of each edge side; Each edge side updates the modal heterogeneous subnetwork according to its initial parameters to obtain the updated modal heterogeneous subnetwork.

3. The system according to claim 2, characterized in that, The edge side determines the model update amount of the modal heterogeneous sub-network and extracts the knowledge features of the modal heterogeneous sub-network; The cloud uses the hypernetwork to distill the knowledge features of each edge side into the modal feature sharing layer of the hypernetwork. By aggregating the model update amounts of each edge side, the cloud updates the generation parameters of the hypernetwork. The cloud uses the hypernetwork to generate customized subnetwork parameters based on the model architecture descriptors of each edge side, and then distributes the customized subnetwork parameters to each edge side. Each edge side adjusts its modal heterogeneous subnetwork based on the customized subnetwork parameters to obtain the adjusted modal heterogeneous subnetwork.

4. The system according to claim 1, characterized in that, The cloud deployment has a unified state space; After the edge device extracts the features of each modality through the modality feature encoder, it maps the features of each modality to the unified state space through the mapping layer of that modality to obtain an independent representation of the user's state for that modality.

5. The system according to claim 4, characterized in that, The cloud-based system is equipped with a task ontology knowledge base associated with the unified state space. The task ontology knowledge base includes multiple pieces of knowledge information. Each piece of knowledge information contains the user state and its state label under each scenario, as well as the state result of associating the state label based on a pre-built user state analysis knowledge graph. When the edge receives a request status result task, it maps the task tag of the request status result task to the unified state space to output the status result in the task ontology knowledge base corresponding to the task tag of the request status result task. The request status result task includes the status tags that exist in the task ontology knowledge base.

6. The system according to claim 1, characterized in that, The cloud-based deployment and maintenance of a multimodal plugin library, which contains modal plugins corresponding to each modality; When the cloud receives a modal analysis request sent by the edge side, it sends modal plugins for several target modalities corresponding to the modal analysis request, as well as fusion strategies for each target modality, to the edge side.

7. The system according to claim 6, characterized in that, Each of the modal plugins is configured with a modal confidence estimator; The modality confidence evaluator for each target modality calculates the quality score of the target modality based on the signal-to-noise ratio and signal distribution of the modal signal, and adjusts the attention weight for fusing the target modality in the fusion strategy based on the quality score of the target modality.

8. The system according to claim 1, characterized in that, The cloud deploys a global modal model, and the cloud sends global multimodal fusion models dominated by EEG modality to multiple edge sides, with each edge side deploying multiple modal models; Each edge side performs local independent training for each modal model, and adjusts the global multimodal fusion model through multimodal fusion data to obtain the model adjustment amount of the global multimodal fusion model. The multimodal fusion data is the data after fusing the modal data output by each modal model. The cloud platform updates the global modal model by aggregating the model adjustment amounts of each edge side, obtains global model parameters, and then sends the global model parameters to each edge side. For each edge side, the edge side updates the initial parameters of the global multimodal fusion model based on the global model parameters to obtain the updated global multimodal fusion model, and sends the end-side multimodal inference model that is consistent with the feature representation of the global multimodal fusion model to the end side that communicates with the edge side.

9. The system according to any one of claims 1-8, characterized in that, The lightweight model is a model that is trained on the cloud and then deployed to the edge side. On the edge side, features from other corresponding modal models are extracted and multimodal fusion is performed. After fine-tuning with local data and undergoing lightweight processing, the model is deployed to the edge side. The training process of the backbone model includes: Obtain multiple user state inference samples; Using the inference process of simulating the inference samples of each user state as the adaptation task, and with the training objective of finding the initial parameter space element with high gradient sensitivity, the basic inference model is trained to obtain the backbone model with the best initial state. When a new user accesses the edge side, the edge side uses fewer than a preset number of label samples of the new user to quickly update the gradient of the backbone model, and combines it with other modal models for training. After extracting features, the modal features are fused and trained in the multimodal fusion model to obtain the edge side multimodal model.

10. A multimodal state reasoning method based on electroencephalogram (EEG) signals, characterized in that, An edge-side application is used in a multimodal state reasoning system based on EEG signals. The state reasoning system also includes an edge side and a cloud. The edge side deploys an edge-side multimodal reasoning model, which is a lightweight model that is fine-tuned locally by the edge-side multimodal reasoning model. The edge-side multimodal reasoning model includes a modal feature encoder, a cross-modal attention fusion module, and a reasoning module. The cloud sends fusion weights for feature fusion to the edge side, and the edge side downloads the fusion weights from the edge side. The method includes: Collect multimodal data from users, primarily based on electroencephalogram (EEG) signals; The modality feature encoder extracts features for each modality of the multimodal data; The fused features are obtained by fusing features from various modalities through the cross-modal attention fusion module and based on the fusion weights. The inference module rapidly infers the fused features to obtain the user's predicted state and its confidence level. When the confidence level is lower than the threshold, the intermediate features of the data segment are uploaded to the edge side or the cloud, so that the edge side or the cloud can perform reasoning on the intermediate features of the data segment to obtain the state reasoning result. The intermediate features of the data segment are the modal feature vectors extracted by the edge side and the deep semantic embedding vectors generated after integration by the lightweight cross-modal fusion module.

Images