A multi-sided federated computing system based on physiological signals

By using a multi-sided federated computing system based on physiological signals, and utilizing a topology-aware hypernetwork control module and a parameter reflection mapping module, knowledge flow and global model updates are realized in a heterogeneous edge hardware environment. This solves the problem of existing technologies being unable to adapt to edge hardware and improves the efficiency and accuracy of cross-node knowledge collaborative training.

CN122432611BActive Publication Date: 2026-11-10DONGGUAN ZKTECO ELECTRONICS TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202610912306.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-06-24
Publication Date
2026-11-10
Estimated Expiration
2046-06-24

AI Technical Summary

Technical Problem

The existing federated architecture cannot adapt to edge hardware with varying specifications, cannot achieve cross-node knowledge alignment and collaborative training, and is difficult to realize bidirectional knowledge flow between different sides and global model updates and evolution.

Method used

A multi-sided federated computing system based on physiological signals is adopted. The topology-aware supernetwork control module perceives the hardware topology of the edge side, dynamically generates heterogeneous sub-models, and performs bidirectional flow of gradient information and global model updates between the cloud and the edge side. The parameter reflection mapping module and meta-learning evolution module are used to adapt and evolve the model.

Benefits of technology

It enables bidirectional knowledge flow and global model updates and evolution in heterogeneous edge hardware environments, improves the efficiency and accuracy of cross-node knowledge collaborative training, and adapts to edge devices with different hardware specifications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122432611B_ABST
    Figure CN122432611B_ABST
Patent Text Reader

Abstract

This application discloses a multi-sided federated computing system based on physiological signals. A topology-aware supernetwork control module perceives the hardware topology description of the edge side, dynamically samples and organizes optimal parameter slices from the global supernetwork matrix, generating heterogeneous sub-models. The cloud distributes these heterogeneous sub-models to the edge side, where they undergo adaptive training based on a small number of local samples to obtain adapted heterogeneous sub-models. The cloud receives the heterogeneous gradients of each adapted heterogeneous sub-model, and a parameter reflection mapping module embeds these heterogeneous gradients of different dimensions into the matrix. The matrix is ​​then updated based on these heterogeneous gradients. Therefore, even with inconsistent edge hardware specifications, the topology-aware supernetwork control module can slice out optimal parameters from the matrix, generating heterogeneous sub-models adapted to the edge hardware. Simultaneously with the training of the heterogeneous sub-models, the matrix is ​​also updated through heterogeneous gradients, enabling collaborative training between the matrix and the heterogeneous sub-models. This achieves bidirectional flow between the cloud and the edge side, and facilitates the updating and evolution of the global model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of biosignal processing technology, and more specifically, to a multi-sided federated computing system based on physiological signals. Background Technology

[0002] With the development of brain-computer interfaces and multimodal perception technologies, user physiological signal inference, primarily based on electroencephalogram (EEG) signals, is widely used in smart education, driver monitoring, and medical rehabilitation. Multi-side collaboration and federated learning support large-scale, privacy-compliant state computation, enabling multi-institutional data collaboration and model generalization optimization. Existing systems often employ a layered architecture of edge-side data acquisition, edge preprocessing, and cloud-based training and inference, improving state recognition accuracy through multimodal fusion.

[0003] Traditional federated architectures use a fixed-structure global model for training, which cannot adapt to edge hardware with varying specifications. They cannot achieve cross-node knowledge alignment and collaborative training through a single fixed-structure global model, and the local knowledge generated by each edge node cannot be effectively aggregated, making it difficult to achieve cross-device knowledge transfer.

[0004] Therefore, how to achieve bidirectional flow of knowledge and global model updates and evolution in a heterogeneous environment with multiple edge hardware is a problem that needs attention. Summary of the Invention

[0005] In view of the above problems, this application provides a multi-sided federated computing system based on physiological signals to realize bidirectional flow of knowledge between different sides and the updating and evolution of the global model in an environment with heterogeneous hardware on multiple edge sides.

[0006] To achieve the above objectives, the following specific solutions are proposed:

[0007] A multi-sided federated computing system based on physiological signals includes a cloud and multiple edge sides. The cloud includes a global supernetwork matrix, a topology-aware supernetwork control module, and a parameter reflection mapping module. The global supernetwork matrix is ​​a parameter storage matrix based on a weight-sharing mechanism that integrates various physiological signal modal encoders and cross-modal interaction tensors.

[0008] The topology-aware hypernetwork control module senses the hardware topology description of each edge side, and dynamically samples and organizes the optimal parameter slices from the global hypernetwork matrix according to the hardware topology description of each edge side, and generates a heterogeneous sub-model corresponding to each edge side based on the optimal parameter slices.

[0009] The cloud sends the heterogeneous sub-model to the edge side;

[0010] The edge side performs adaptive training on the heterogeneous sub-model based on a local label sample of less than a preset number, to obtain an adapted heterogeneous sub-model.

[0011] The cloud receives the heterogeneous gradient information of the adapted heterogeneous sub-model uploaded by each of the edge sides;

[0012] The parameter reflection mapping module embeds the heterogeneous gradient information of each of the adapted heterogeneous sub-models in different dimensions into the corresponding coordinates of the global supernetwork parent body according to the sampling index recorded by the topology-aware supernetwork control module.

[0013] The global supernetwork parent is updated based on the heterogeneous gradient information of each of the adapted heterogeneous sub-models to complete global evolution.

[0014] Optionally, the cloud also includes a knowledge distillation coordinator.

[0015] The knowledge distillation coordinator is used to lightweight the heterogeneous sub-model after the topology-aware supernetwork control module generates the heterogeneous sub-model, so as to obtain a lightweight heterogeneous sub-model.

[0016] Optionally, the cloud also includes a meta-learning evolution module.

[0017] The meta-learning evolution module is used to store and update the meta-parameters of the global supernetwork parent, and the heterogeneous sub-model carries the meta-parameters produced by the meta-learning evolution module when it is distributed.

[0018] The edge side performs adaptation training on the lightweight heterogeneous sub-model based on the meta-parameters and the label samples to obtain the adapted heterogeneous sub-model.

[0019] Optionally, the system also includes an edge side, each of the edge sides being communicatively connected to several edge sides, the edge side including a relay inference module, and the edge side including a segmented feature encoder and a confidence evaluation feedback unit;

[0020] The segmented feature encoder includes feature encoders for various physiological signal modalities. Each feature encoder is used to acquire its corresponding target physiological signal and extract the pre-features of the target physiological signal.

[0021] The confidence assessment feedback unit is used to upload a desensitized vector generated based on the physiological signal to the edge side when the confidence of the state reasoning result based on the physiological signal on the edge side is lower than a preset confidence threshold.

[0022] The relay inference module is used to perform attention fusion calculation based on the desensitized vector uploaded from the endpoint to obtain a new state inference result.

[0023] Optionally, the edge side also includes a federated fine-tuning engine, which is used to train the heterogeneous sub-model and adjust the fusion weights using local private data.

[0024] Optionally, the system also includes privacy enhancement components;

[0025] The privacy enhancement component is used to perform privacy enhancement on the desensitized vector before the confidence assessment feedback unit uploads the desensitized vector to obtain a security-reinforced desensitized vector, and to perform privacy enhancement on the heterogeneous gradient information before the heterogeneous gradient information is uploaded on the edge side to obtain a security-reinforced heterogeneous gradient information.

[0026] Optionally, the cloud also includes a task ontology library and a unified state space;

[0027] The topology-aware hypernetwork control module generates the heterogeneous sub-model under the constraints of the task ontology library;

[0028] The feature vectors extracted from each plug-in modality accessed by the edge side to the cloud are mapped in the unified state space.

[0029] Optionally, the system also includes a modal plugin management library, which contains multiple non-core modal plugins and records the interface definition, access status, and weight information of each non-core modal plugin.

[0030] Optionally, the system also includes a resource monitoring scheduler, which collects hardware information from each edge side in real time, including CPU load, GPU usage, memory bandwidth, and device power consumption.

[0031] The resource monitoring scheduler is used to schedule the target computing task to an adjacent idle edge side if it detects that the hardware information representation load of the main edge side exceeds the standard when the main edge side is executing the target computing task.

[0032] Optionally, the edge side includes an offline caching module and a breakpoint resume module;

[0033] The offline caching module is used to temporarily store the federated training gradient information locally on the edge side when the connection between the edge side and the cloud is interrupted.

[0034] The breakpoint resume module is used to determine the training information of the local heterogeneous sub-model based on the federated training gradient information and a reflection mapping algorithm after the connection between the edge side and the cloud is restored from interruption, and then upload the training information to the cloud.

[0035] Using the above technical solution, this application perceives the hardware topology description of each edge side through the topology-aware hypernetwork control module, and dynamically samples and organizes optimal parameter slices from the global hypernetwork matrix based on the hardware topology description of each edge side. Based on the optimal parameter slices, a heterogeneous sub-model corresponding to that edge side is generated. The cloud sends the heterogeneous sub-model to the edge side, and the edge side performs adaptation training on the heterogeneous sub-model based on a local number of label samples less than a preset number, to obtain an adapted heterogeneous sub-model. The cloud receives the heterogeneous gradient information of the adapted heterogeneous sub-models uploaded by each edge side. The parameter reflection mapping module embeds the heterogeneous gradient information of each adapted heterogeneous sub-model in different dimensions into the corresponding coordinates of the global hypernetwork matrix according to the sampling index recorded by the topology-aware hypernetwork control module. The global hypernetwork matrix is ​​updated based on the heterogeneous gradient information of each adapted heterogeneous sub-model, completing global evolution. This demonstrates that even with varying edge hardware specifications, the cloud-based topology-aware hypernetwork control module can slice out the optimal parameters from the parent model, thereby generating heterogeneous sub-models adapted to the edge hardware. While the heterogeneous sub-models are being trained, the parent model is also updated through heterogeneous gradient information, enabling collaborative training between the parent model and the heterogeneous sub-models. This achieves bidirectional flow between the cloud and the edge, as well as the updating and evolution of the global model. Attached Figure Description

[0036] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0037] Figure 1 An overall architecture diagram of a multi-sided federated computing system provided for embodiments of this application;

[0038] Figure 2 A schematic diagram of the cloud structure of a multi-sided federated computing system provided in this application embodiment;

[0039] Figure 3 A schematic diagram of a first process for implementing multi-sided federated computing based on physiological signals on the end side, as provided in an embodiment of this application;

[0040] Figure 4 A schematic diagram illustrating a second process for implementing multi-sided federated computing based on physiological signals on the end side, as provided in an embodiment of this application;

[0041] Figure 5 This is a schematic diagram of a third process for implementing multi-sided federated computing based on physiological signals on the end side, as provided in an embodiment of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] Figure 1 An optional system architecture for implementing multi-sided federated computing based on physiological signals, as provided in the embodiments of this application, is as follows: Figure 1 As shown, the overall architecture of the system can include: a terminal 10, an edge 20, and a cloud 30. There can be multiple edge 20s and terminal 10s, while there is only one cloud 30 as the core. The cloud 30 communicates with the terminal 10 through the edge 20s. Each edge 20 can communicate with multiple terminal 10s, and the cloud 30 can communicate with multiple edge 20s.

[0044] Specifically, the edge device 10, serving as the entry point for data acquisition and preliminary processing, can be equipped with various wearable sensing devices, such as lightweight electroencephalography (EEG) headsets (e.g., EEG rings, EEG glasses, headbands), smartwatches, and cameras, microphones, heart rate monitors, eye trackers, etc., or terminal PCs connected to these devices for acquiring data from other modalities. These devices on the edge device 10 can have built-in low-power processing units (such as MCUs or dedicated AI chips) responsible for performing tasks such as signal acquisition and synchronization, data preprocessing, and lightweight feature extraction and inference.

[0045] Specifically, the signal acquisition and synchronization process may include: real-time acquisition of the user's multimodal data, and ensuring time synchronization of cross-modal data through a precise timestamp protocol.

[0046] The data preprocessing process may specifically include: filtering the original signal (such as bandpass filtering and notch filtering), artifact removal (such as eye movement and electromyography interference), and segmentation.

[0047] The edge device 10 may include a segmented feature encoder and a confidence assessment feedback unit. The segmented feature encoder contains feature encoders for various physiological signal modalities. Each feature encoder is used to acquire its corresponding target physiological signal and extract the pre-features of the target physiological signal. The confidence assessment feedback unit is used to upload a desensitized vector generated based on the physiological signal to the edge device 20 when the confidence of the state reasoning result based on the physiological signal by the edge device 10 is lower than a preset confidence threshold.

[0048] The lightweight feature extraction and inference process can specifically include: configuring a corresponding lightweight feature encoder for each access modality (such as facial expression, voice, eye movement, heart rate, etc.), and configuring an edge-side multimodal inference model. All edge-side multimodal inference models follow the principles of low power consumption, lightweight design, and high compression to adapt to edge computing power limitations. The edge-side multimodal inference model is deployed on edge 10 and is a lightweight model that is fine-tuned from the edge-side multimodal inference model 20 using local samples. The edge-side multimodal inference model includes a modal feature encoder, a cross-modal attention fusion module, and an inference module. The EEG modality deploys lightweight core models (such as EEGNetv4 and Mini-DeepConvNet) to handle core feature extraction and basic state inference on the edge. Other modalities, such as facial expressions, speech, eye tracking, and heart rate, each deploy dedicated lightweight models (e.g., a simplified version of MobileNetV2 for facial expressions, MFCC + a lightweight RNN for speech, a lightweight CNN for keypoint detection for eye tracking, and a PPG feature extraction MLP for heart rate). All non-EEG modal models are packaged as standardized modal plugins, which can be loaded / unloaded on the edge as needed. In the lightweight cross-modal attention fusion module of the edge multimodal inference model, the output features of each lightweight model are initially fused, and rapid inference is completed in the inference module. The fusion weights are trained on the cloud (30) and then distributed to the edge (20). After local fine-tuning on the edge (20), they are downloaded to the lightweight fusion module on the edge (10) to achieve the minimum necessary parameter set for fusion. The fusion weights can be fine-tuned locally.

[0049] Edge computing systems can be deployed within the organization where the data originates, such as hospitals, schools, research centers, or corporate fleets, typically as small servers or edge computing clusters. Their core function is to undertake regional computing and management tasks, including data aggregation and management, local model training and federated collaboration, task ontology mapping, and relay inference, all while ensuring the data remains within the organization.

[0050] Specifically, the data aggregation and management process may include: aggregating intermediate features or preprocessed data uploaded by multiple end devices within the organization, and performing unified formatting and label alignment.

[0051] The process of local model training and federated collaboration can specifically include: using the organization's own data to train or fine-tune heterogeneous sub-models distributed from the cloud 30 locally, and calculating model gradients or updating weights. Each edge device 20 maintains its own local encoder model parameters and a multi-modal fusion model parameter set for all connected modalities. These model parameters correspond one-to-one with the modal plugins / global fusion models on the cloud 30 and can be flexibly maintained according to the organization's modal configuration. Model parameters are not loaded for unconnected modalities, saving storage and computing resources on the edge device 20. The edge device 20 also includes a federated fine-tuning engine for training heterogeneous sub-models and adjusting fusion weights using local private data.

[0052] The process of task ontology mapping can specifically include mapping specific task tags within the organization (such as "distracted in class" or "drowsy while driving") to a unified state space defined in the cloud.

[0053] The relay inference module on the edge side can specifically include: performing attention fusion calculation based on the desensitized vector uploaded by the end side 10 to obtain a new state inference result, and the edge side 20 can provide the end side 10 with low latency and high accuracy verification inference service.

[0054] Cloud30, serving as the brain of the entire system, consists of a large-scale server cluster and is responsible for global model maintenance, federated learning coordination, and platform management. Specifically, it can perform global model maintenance, federated learning coordination, multimodal plugin management, privacy and security auditing, and unified state space and task ontology knowledge base management.

[0055] Specifically, such as Figure 2 As shown, the cloud 30 may specifically include: a global supernetwork parent 31, a topology-aware supernetwork control module 32, and a parameter reflection mapping module 33.

[0056] Among them, the global supernetwork matrix 31 is a parameter storage matrix based on a weight sharing mechanism that integrates various physiological signal modal encoders and cross-modal interaction tensors.

[0057] Specifically, the global supernetwork matrix 31 can be a high-dimensional parameter storage matrix based on a weight-sharing mechanism. It integrates the EEG backbone encoder, various plug-in modal encoders (facial, speech, etc.), and cross-modal interaction tensors (such as multimodal attention parameters). The global supernetwork matrix 31 stores full-scale general knowledge and serves as the matrix for all heterogeneous sub-models, ensuring that models of different specifications have homogeneity in underlying logic and parameter indexing.

[0058] The process of multi-sided federated computation based on physiological signals to achieve cross-node knowledge collaborative training can be referenced. Figure 3 Specifically, it includes:

[0059] Step S110: The topology-aware hypernetwork control module senses the hardware topology description of each edge side, and dynamically samples and organizes the optimal parameter slices from the global hypernetwork matrix according to the hardware topology description of each edge side, and generates a heterogeneous sub-model corresponding to each edge side based on the optimal parameter slices.

[0060] Specifically, the topology-aware hypernetwork control module is a logic control engine deployed in the cloud. It is responsible for sensing the hardware topology description (such as computing power, memory, and bandwidth) fed back from the edge side. Based on the hardware constraints of the edge side 20, the topology-aware hypernetwork control module can dynamically sample and organize the optimal parameter slices from the global hypernetwork matrix 31, thereby generating customized heterogeneous subnetworks for edge side 20 devices with different performance.

[0061] Step S120: The cloud sends the heterogeneous sub-model to the edge.

[0062] Specifically, the cloud 30 can distribute different heterogeneous sub-models to their corresponding edge sides 20.

[0063] Step S130: The edge side performs adaptation training on the heterogeneous sub-model based on less than a preset number of local label samples to obtain the adapted heterogeneous sub-model.

[0064] It is understandable that since the heterogeneous sub-model is generated based on the hardware topology description of edge side 20, it is already highly adapted to edge side 20 when given to edge side 20. Therefore, only a small number of local samples are needed to train the adapted heterogeneous sub-model.

[0065] Step S140: The cloud receives the heterogeneous gradient information of the adapted heterogeneous sub-model uploaded by each edge side.

[0066] In step S150, the parameter reflection mapping module embeds the heterogeneous gradient information of each adapted heterogeneous sub-model in different dimensions into the corresponding coordinates of the global supernetwork mother body according to the sampling index recorded by the topology-aware supernetwork control module.

[0067] Understandably, when gradients are propagated back from the edge sides with multiple dimensions and heterogeneous structures, this module uses the original mapping table of the topology-aware supernetwork control module to accurately reflect these fragmented heterogeneous gradient information back to the corresponding positions in the supernetwork parent body.

[0068] Step S160: The global supernetwork parent body is updated based on the heterogeneous gradient information of each adapted heterogeneous sub-model to complete the global evolution.

[0069] It is understandable that the global supernetwork matrix 31 can complete global evolution by absorbing the heterogeneous experience of various adapted heterogeneous sub-models.

[0070] The multi-sided federated computing system based on physiological signals provided in this embodiment perceives the hardware topology description of each edge side through a topology-aware supernetwork control module. Based on the hardware topology description of each edge side, it dynamically samples and organizes optimal parameter slices from the global supernetwork matrix. Based on the optimal parameter slices, it generates heterogeneous sub-models corresponding to each edge side. The cloud distributes the heterogeneous sub-models to the edge sides. The edge sides perform adaptation training on the heterogeneous sub-models based on fewer than a preset number of local labeled samples to obtain adapted heterogeneous sub-models. The cloud receives the heterogeneous gradient information of the adapted heterogeneous sub-models uploaded by each edge side. The parameter reflection mapping module embeds the heterogeneous gradient information of each adapted heterogeneous sub-model in different dimensions into the corresponding coordinates of the global supernetwork matrix according to the sampling index recorded by the topology-aware supernetwork control module. The global supernetwork matrix is ​​updated based on the heterogeneous gradient information of each adapted heterogeneous sub-model, completing global evolution. This demonstrates that even with varying edge hardware specifications, the cloud-based topology-aware hypernetwork control module can slice out the optimal parameters from the parent model, thereby generating heterogeneous sub-models adapted to the edge hardware. While the heterogeneous sub-models are being trained, the parent model is also updated through heterogeneous gradient information, enabling collaborative training between the parent model and the heterogeneous sub-models. This achieves bidirectional flow between the cloud and the edge, as well as the updating and evolution of the global model.

[0071] In some embodiments of this application, the cloud 30 mentioned in the foregoing embodiments will be further described, from... Figure 2 It is known that the cloud 30 also includes a knowledge distillation coordinator 34. After the topology-aware supernetwork control module generates heterogeneous sub-models, the knowledge distillation coordinator 34 can lightweight the heterogeneous sub-models to obtain lightweight heterogeneous sub-models.

[0072] Based on this, the process of achieving cross-node knowledge collaborative training through multi-sided federated computing based on physiological signals can be referenced. Figure 4 Specifically, it includes:

[0073] Step S210: The topology-aware hypernetwork control module senses the hardware topology description of each edge side, and dynamically samples and organizes the optimal parameter slices from the global hypernetwork matrix according to the hardware topology description of each edge side, and generates a heterogeneous sub-model corresponding to each edge side based on the optimal parameter slices.

[0074] Specifically, step S210 corresponds one-to-one with step S110 in the aforementioned embodiment, as detailed above, and will not be repeated here.

[0075] Step S220: After the topology-aware supernetwork control module generates the heterogeneous sub-model, the knowledge distillation coordinator performs lightweighting on the heterogeneous sub-model to obtain a lightweight heterogeneous sub-model, which serves as a new heterogeneous sub-model.

[0076] Understandably, the knowledge distillation coordinator 34 uses distillation technology to inject the deep semantic logic of the global supernetwork mother 31 into the simplified sub-model, ensuring that the small model does not lose the brainwave-driven recognition accuracy while reducing its size.

[0077] Among them, the global supernetwork mother 31 can act as a teacher model, which can guide the heterogeneous subnetworks to perform feature alignment, ensuring that the simplified model still retains the brainwave-dominated logical core.

[0078] Step S230: The cloud sends the heterogeneous sub-model to the edge side.

[0079] Step S240: The edge side performs adaptation training on the heterogeneous sub-model based on less than a preset number of local label samples to obtain the adapted heterogeneous sub-model.

[0080] Step S250: The cloud receives the heterogeneous gradient information of the adapted heterogeneous sub-model uploaded by each edge side.

[0081] In step S260, the parameter reflection mapping module embeds the heterogeneous gradient information of each adapted heterogeneous sub-model in different dimensions into the corresponding coordinates of the global supernetwork mother body according to the sampling index recorded by the topology-aware supernetwork control module.

[0082] Step S270: The global supernetwork parent body is updated based on the heterogeneous gradient information of each adapted heterogeneous sub-model to complete the global evolution.

[0083] Specifically, steps S230 to S270 correspond one-to-one with steps S120 to S160 in the aforementioned embodiments, as detailed above, and will not be repeated here.

[0084] In some embodiments of this application, the cloud 30 mentioned in the foregoing embodiments will be further described, from... Figure 2 It is known that the cloud 30 also includes a meta-learning evolution module 35, which stores and updates the meta-parameters of the global hypernetwork parent 31.

[0085] Based on this, the process of achieving cross-node knowledge collaborative training through multi-sided federated computing based on physiological signals can be referenced. Figure 5 Specifically, it includes:

[0086] Step S310: The topology-aware hypernetwork control module senses the hardware topology description of each edge side, and dynamically samples and organizes the optimal parameter slices from the global hypernetwork matrix according to the hardware topology description of each edge side, and generates a heterogeneous sub-model corresponding to each edge side based on the optimal parameter slices.

[0087] Step S320: After the topology-aware supernetwork control module generates the heterogeneous sub-model, the knowledge distillation coordinator performs lightweighting on the heterogeneous sub-model to obtain a lightweight heterogeneous sub-model, which serves as a new heterogeneous sub-model.

[0088] Specifically, steps S310 to S320 correspond one-to-one with steps S210 to S220 in the aforementioned embodiments, as detailed above, and will not be repeated here.

[0089] Step S330: The cloud sends the heterogeneous sub-model to the edge side. When the heterogeneous sub-model is sent, it carries the meta-parameters produced by the meta-learning evolution module.

[0090] Specifically, the meta-parameters represent an easily adaptable state, enabling the heterogeneous sub-models distributed from the cloud to have extremely strong generalization capabilities.

[0091] Step S340: Based on meta-parameters and fewer than a preset number of local labeled samples, the edge side performs adaptation training on the heterogeneous sub-model to obtain the adapted heterogeneous sub-model.

[0092] Understandably, the meta-parameters carried by the distributed heterogeneous sub-models give the models extremely strong generalization capabilities, thus requiring only a small amount of local data to quickly adapt the state of new users.

[0093] Step S350: The cloud receives the heterogeneous gradient information of the adapted heterogeneous sub-model uploaded by each edge side.

[0094] In step S360, the parameter reflection mapping module embeds the heterogeneous gradient information of each adapted heterogeneous sub-model in different dimensions into the corresponding coordinates of the global supernetwork mother body according to the sampling index recorded by the topology-aware supernetwork control module.

[0095] Step S370: The global supernetwork parent body is updated based on the heterogeneous gradient information of each adapted heterogeneous sub-model to complete the global evolution.

[0096] Specifically, steps S350 to S370 correspond one-to-one with steps S250 to S270 in the aforementioned embodiments, as detailed above, and will not be repeated here.

[0097] In some embodiments of this application, the cloud 30 mentioned in the foregoing embodiments will be further described, from... Figure 2 It can be seen that the cloud 30 also includes a unified state space 36 and a task ontology library 37.

[0098] The process of managing a unified state space and task ontology library can specifically include: defining and maintaining a standardized user state space that is unified across all modalities, and a task ontology knowledge library built on ontology and knowledge graphs to achieve multi-task knowledge transfer and semantic unification of task tags.

[0099] The unified state space is a multi-dimensional, numerical, globally unified state space constructed in the cloud. Its core dimensions include core user state indicators such as valence, arousal, emotional intensity, cognitive load, attention, and fatigue. Each dimension has a clear numerical range and semantic definition (e.g., valence: -1 to 1, negative for negative emotions, positive for positive emotions), serving as a common language for all modalities. All modalities accessed by the platform (EEG / facial / voice / eye movement / heart rate, etc.) ultimately have their feature representations mapped to this unified state space: each modality's feature encoder extracts its original features and maps the feature vectors to the unified state space through a modality-specific mapping layer, obtaining an independent representation of the user state for that modality; then, an EEG-led attention fusion network fuses the representations mapped to the state space from each modality, obtaining a comprehensive multi-modal user state representation. This representation remains within the unified state space, and the final user state prediction results are output from this space. All application scenarios follow the same set of state space mapping rules, ensuring the comparability and transferability of user state calculation results across scenarios.

[0100] The task ontology library is a cross-institutional, cross-scenario task tag semantic knowledge base built on ontology and knowledge graphs. It serves as a bridge connecting the personalized task tags of various institutions with the unified state space in the cloud. Its core functions are reflected in three aspects: semantic unification, knowledge transfer, and task standardization. Semantic unification: It maps personalized task tags from different institutions and scenarios (such as "classroom inattentiveness" in schools and "driving drowsiness" in motorcades) to unified semantic concepts (such as "low attention" and "high fatigue"), solving the problem of semantic silos in task tags; Knowledge transfer: Through the semantic relationships in the ontology knowledge base (such as "yawning" → "fatigue" → "reduced cognitive load"), it enables zero-sample / few-sample knowledge transfer of models across tasks and scenarios; Task standardization: It provides standardized semantic templates for task access in new institutions / new scenarios, reducing the model adaptation and integration costs for new tasks and enabling the platform to be plug-and-play.

[0101] The task ontology is associated with a unified state space. The task ontology includes multiple pieces of knowledge information, each containing the user's state and its state label for each scenario, as well as state results associated with the state labels based on a pre-built user state analysis knowledge graph. Therefore, when the edge device 20 receives a request for a state result task, it can map the task label of the request for a state result task to the unified state space to output the state result corresponding to the task label in the task ontology. The request for a state result task includes the state labels existing in the task ontology knowledge base.

[0102] It is understandable that the topology-aware hypernetic network control module needs to generate heterogeneous sub-models under the constraints of the task ontology library, and the feature vectors extracted by each plug-in modality of the edge side accessing the cloud all have mappings in a unified state space.

[0103] The task ontology library can adopt a layered construction strategy of top-level standardization + bottom-level customization. It uses OWL (Web Ontology Language) for formal description and Neo4j to build a knowledge graph for semantic storage and retrieval. Specific construction steps include:

[0104] First, define the core ontology concept layer (cloud-based standardization): Based on the core needs of user state computation in the cloud, define three categories of top-level core concepts. All core concepts are assigned a unique semantic ID and a clear definition, forming a standardized top-level framework for the ontology library. Among them, the user state category includes emotions, cognitive load, attention, fatigue, etc. (corresponding one-to-one with the dimensions of the unified state space); the behavioral characteristics category includes yawning, blinking frequency, abnormal EEG rhythms, etc. (observable behavioral / physiological characteristics of each modality); and the application scenario category includes smart education, driving safety, medical rehabilitation, etc. (the core application scenarios of the platform).

[0105] Second, define the semantic relationships between concepts (cloud-based 30 standardization): use OWL relational attributes to define standardized semantic relationships for core concepts. Core relationships include causal relationships, mapping relationships, subordinate relationships, and association relationships. All semantic relationships are assigned a weight value of 0 to 1 to indicate the strength of the association. The relationships are labeled by domain experts and optimized through model iteration.

[0106] Third, edge-side personalized task tag mapping (edge-side localization): Each edge side 20 maps its own personalized task tags / observation indicators to the core ontology concept layer defined by the cloud 30. The edge side 20 only needs to complete the mapping once, and the cloud 30 will synchronize the mapping relationship to the ontology library, forming a two-layer ontology structure of core concepts + edge-side personalized tags.

[0107] Fourth, the instantiation of the ontology and the construction of the knowledge graph: Based on the above concepts and relationships, a knowledge graph is used to realize the instantiation and storage of the ontology knowledge base. The nodes are the core ontology concepts, the personalized task tags of the edge side, and the user status indicators. The edges are the semantic associations between nodes plus weight values. The storage engine adopts the Neo4j graph database, which supports efficient semantic query and association reasoning. At the same time, the ontology supports incremental updates. When a new edge side or a new task is connected, only new nodes and edges need to be added, without reconstructing the entire ontology knowledge base.

[0108] Fifth, dynamic optimization and self-learning of the ontology: During platform operation, the ontology knowledge base is dynamically optimized through model feedback and a semantic auto-mapping engine based on the Large Language Model (LLM). The model feedback compares the user state prediction results of the multimodal model with the semantic associations of the ontology. If there is a discrepancy, the relation weights are automatically adjusted. The semantic auto-mapping engine based on the Large Language Model (LLM) periodically reviews and corrects the mapping relationships, ultimately achieving self-learning and self-optimization of the ontology to adapt to the task requirements of different scenarios.

[0109] In actual use, the edge side 20 can upload local task tags to the cloud 30 → the cloud 30 performs semantic queries in the ontology knowledge base → matches the corresponding core user state concepts → maps to a unified state space → guides the feature extraction and fusion training of the model, realizing cross-task knowledge transfer.

[0110] Understandably, the unified state space (quantization benchmark) and task ontology library (semantic association) provide semantic directions and training objectives for feature extraction for each modal encoder model, and semantic basis for fusion weights and output normalization standard for multimodal fusion models. This enables the training, inference, and federated collaboration of edge / cloud three-level models to have a unified benchmark, ultimately achieving cross-institutional, cross-task, and cross-modal knowledge transfer, while ensuring the platform's scalability, generalization, and collaboration.

[0111] In some embodiments of this application, the multi-sided federated computing system based on physiological signals mentioned in the above embodiments is further described. The system may also include privacy enhancement components, modal plugin management library and resource monitoring scheduler.

[0112] Specifically, the privacy enhancement component can be used to enhance the privacy of the de-identified vector before it is uploaded by the confidence assessment feedback unit, resulting in a security-hardened de-identified vector. The privacy enhancement component can also be used to enhance the privacy of heterogeneous gradient information before it is uploaded from the edge side 20, resulting in security-hardened heterogeneous gradient information.

[0113] The modality plugin management library contains multiple non-core modality plugins. The cloud supports hot-swapping of each non-core modality plugin, allowing the architecture to dynamically adjust the input dimensions of the fusion network based on the actual hardware configuration of 20 edge-side plugins / 10 endpoint-side plugins. The modality plugin management library records the interface definition, access status, and weight information of each non-core modality plugin. Non-core modalities refer to physiological signal modalities other than EEG signal modalities. When a non-core modality fails (e.g., poor ambient light causing facial recognition failure), the management library notifies the topology-aware hypernetwork control module to adjust the organizational framework, achieving a smooth degradation of system performance and ensuring uninterrupted core EEG inference.

[0114] The resource monitoring and scheduler collects hardware information from each edge device in real time, including CPU load, GPU usage, memory bandwidth, and device power consumption. If the resource monitoring and scheduler detects that the main edge device's hardware information indicates an excessive load while executing a target computing task, it will schedule the target computing task to a nearby idle edge device to ensure computing efficiency in high-concurrency scenarios.

[0115] In some embodiments of this application, the edge side 20 mentioned in the above embodiments will be further described. The edge side 20 may also include an offline caching module and a breakpoint resume module.

[0116] Understandably, unstable mobile networks are common in EEG applications. Therefore, the offline caching module can temporarily store the federated training gradient information locally on the edge device 20 when the connection between the edge device 20 and the cloud device 30 is interrupted. After the connection between the edge device 20 and the cloud device 30 is restored, the breakpoint resume module determines the training information of the local heterogeneous sub-model based on the federated training gradient information using a reflection mapping algorithm, and uploads the training information to the cloud device 30, ensuring the continuity and knowledge integrity of the global hypernetwork evolution process.

[0117] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0118] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0119] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-sided federated computing system based on physiological signals, characterized in that, It includes the cloud and multiple edge sides. The cloud includes a global super network matrix, a topology-aware super network control module and a parameter reflection mapping module. The global super network matrix is ​​a parameter storage matrix based on a weight sharing mechanism that integrates various physiological signal modal encoders and cross-modal interaction tensors. The topology-aware hypernetwork control module senses the hardware topology description of each edge side, and dynamically samples and organizes the optimal parameter slices from the global hypernetwork matrix according to the hardware topology description of each edge side, and generates a heterogeneous sub-model corresponding to each edge side based on the optimal parameter slices. The cloud sends the heterogeneous sub-model to the edge side; The edge side performs adaptive training on the heterogeneous sub-model based on a local label sample of less than a preset number, to obtain an adapted heterogeneous sub-model. The cloud receives the heterogeneous gradient information of the adapted heterogeneous sub-model uploaded by each of the edge sides; The parameter reflection mapping module embeds the heterogeneous gradient information of each of the adapted heterogeneous sub-models in different dimensions into the corresponding coordinates of the global supernetwork parent body according to the sampling index recorded by the topology-aware supernetwork control module. The global supernetwork parent is updated based on the heterogeneous gradient information of each of the adapted heterogeneous sub-models to complete global evolution.

2. The multi-sided federated computing system based on physiological signals according to claim 1, characterized in that, The cloud also includes a knowledge distillation coordinator. The knowledge distillation coordinator is used to lightweight the heterogeneous sub-model after the topology-aware supernetwork control module generates the heterogeneous sub-model, so as to obtain a lightweight heterogeneous sub-model.

3. The multi-sided federated computing system based on physiological signals according to claim 2, characterized in that, The cloud also includes a meta-learning evolution module. The meta-learning evolution module is used to store and update the meta-parameters of the global supernetwork parent, and the heterogeneous sub-model carries the meta-parameters produced by the meta-learning evolution module when it is distributed. The edge side performs adaptation training on the lightweight heterogeneous sub-model based on the meta-parameters and the label samples to obtain the adapted heterogeneous sub-model.

4. The multi-sided federated computing system based on physiological signals according to claim 1, characterized in that, It also includes an edge side, each of the edge sides being communicatively connected to several edge sides, the edge side including a relay inference module, and the edge side including a segmented feature encoder and a confidence evaluation feedback unit; The segmented feature encoder includes feature encoders for various physiological signal modalities. Each feature encoder is used to acquire its corresponding target physiological signal and extract the pre-features of the target physiological signal. The confidence assessment feedback unit is used to upload a desensitized vector generated based on the physiological signal to the edge side when the confidence of the state reasoning result based on the physiological signal on the edge side is lower than a preset confidence threshold. The relay inference module is used to perform attention fusion calculation based on the desensitized vector uploaded from the endpoint to obtain a new state inference result.

5. The multi-sided federated computing system based on physiological signals according to claim 4, characterized in that, The edge side also includes a federated fine-tuning engine, which is used to train the heterogeneous sub-model and adjust the fusion weights using local private data.

6. The multi-sided federated computing system based on physiological signals according to claim 4, characterized in that, It also includes privacy enhancement components; The privacy enhancement component is used to perform privacy enhancement on the desensitized vector before the confidence assessment feedback unit uploads the desensitized vector to obtain a security-reinforced desensitized vector, and to perform privacy enhancement on the heterogeneous gradient information before the heterogeneous gradient information is uploaded on the edge side to obtain a security-reinforced heterogeneous gradient information.

7. The multi-sided federated computing system based on physiological signals according to claim 1, characterized in that, The cloud also includes a task ontology library and a unified state space; The topology-aware hypernetwork control module generates the heterogeneous sub-model under the constraints of the task ontology library; The feature vectors extracted from each plug-in modality accessed by the edge side to the cloud are mapped in the unified state space.

8. The multi-sided federated computing system based on physiological signals according to claim 1, characterized in that, It also includes a modal plugin management library, which contains multiple non-core modal plugins. The modal plugin management library records the interface definition, access status and weight information of each non-core modal plugin.

9. The multi-sided federated computing system based on physiological signals according to claim 1, characterized in that, It also includes a resource monitoring scheduler, which collects hardware information from each edge side in real time, including CPU load, GPU usage, memory bandwidth and device power consumption. The resource monitoring scheduler is used to schedule the target computing task to an adjacent idle edge side if it detects that the hardware information representation load of the main edge side exceeds the standard when the main edge side is executing the target computing task.

10. The multi-sided federated computing system based on physiological signals according to claim 1, characterized in that, The edge side includes an offline caching module and a breakpoint resume module; The offline caching module is used to temporarily store the federated training gradient information locally on the edge side when the connection between the edge side and the cloud is interrupted. The breakpoint resume module is used to determine the training information of the local heterogeneous sub-model based on the federated training gradient information and a reflection mapping algorithm after the connection between the edge side and the cloud is restored from interruption, and then upload the training information to the cloud.

Citation Information

Patent Citations

  • Embedded neural network quantification method for unmanned aerial vehicle end side target detection and identification

    CN120526134A

  • Hardware computing power perceived model adaptive dynamic compression and distribution system

    CN120893521A