Federated Learning-Based Mental State Assessment Methods and Devices

Abstract

This application discloses a method and apparatus for assessing mental states based on federated learning, relating to the field of artificial intelligence technology. The method includes: initializing a particle swarm on a central server, selecting the globally optimal particle as the dominant model for this round, and distributing it to participating clients; training the dominant model locally on the participating clients and uploading the local knowledge to the central server; weighting and aggregating the local knowledge on the central server to generate aggregated knowledge; using the aggregated knowledge to update the entire particle swarm and the globally optimal particle on the central server, and returning to the dominant model selection step; repeating the above steps until a global mental state assessment model is output and deployed to all participating clients; and using the global mental state assessment model to perform mental state assessment on the participating clients. This method solves the problems of low data privacy and security, data silos, and static model fusion defects in existing technologies.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for assessing mental state based on federated learning. Background Technology

[0002] Mental health has become a major global public health issue. Utilizing artificial intelligence (AI) technology for large-scale, automated assessment of mental states has become an important research direction. Existing methods typically rely on centralized machine learning models, requiring the aggregation of massive amounts of user data from various sources (such as social media text, voice, video, and physiological signals) onto a central server for training. However, this approach suffers from the following fatal flaws:

[0003] 1) Low data privacy and security: User data related to psychological state (such as conversation content, emotional expression, physiological indicators, etc.) is highly sensitive personal privacy. Centralizing this data to a central server for model training poses a significant risk of data leakage and does not comply with increasingly stringent data protection regulations.

[0004] 2) Data silo problem: Data is scattered across different medical institutions, schools, mobile application platforms, etc., forming "data silos". Due to privacy and commercial competition, data is difficult to share, resulting in limited sample size and data dimensionality for models trained by a single institution, leading to insufficient evaluation accuracy and generalization ability.

[0005] 3) Static model fusion limitations: Most existing multimodal fusion models employ static or simple weighting strategies, failing to grasp the relative importance of each modality in different contexts. For example, late-night social media behavior and facial expressions during daytime meetings have entirely different meanings when assessing depressive tendencies; static models cannot capture these dynamic changes.

[0006] Therefore, designing a psychological state assessment scheme that can strictly protect data privacy, effectively integrate multi-source heterogeneous data, intelligently adapt to different situations, and ensure system security is a problem that urgently needs to be solved in the current technology field. Summary of the Invention

[0007] This invention provides a method and apparatus for assessing mental state based on federated learning, which solves the problems of low data privacy and security, data silos, and static model fusion defects in existing technologies.

[0008] In a first aspect, embodiments of the present invention provide a method for assessing mental states based on federated learning, the method comprising:

[0009] The central server initializes the particle swarm, selects the globally optimal particle in the particle swarm as the dominant model for this round, and distributes the dominant model and the public unlabeled dataset to at least one participating client.

[0010] On the participating client side, based on local multimodal user data, the dominant model is trained locally using a context-aware and dynamically weighted adaptive multimodal fusion model, and the obtained local knowledge is uploaded to the central server.

[0011] The central server receives local knowledge from each participating client and performs weighted aggregation based on the dynamic credibility scores of each participating client to generate aggregated knowledge.

[0012] The central server uses the aggregated knowledge to update the entire particle swarm, guides all particles to find the best solution in the solution space, updates the globally optimal particle, and returns to the dominant model selection step.

[0013] Repeat the above steps until the globally optimal particle in the particle swarm satisfies the preset convergence condition. Finally, output the globally optimal particle as the global psychological state evaluation model and deploy it to all participating client devices.

[0014] The participating client uses a global psychological state assessment model to assess the psychological state of the collected real-time multimodal user data, and obtains the psychological state assessment results of the users.

[0015] The technical solution provided in this application has at least the following beneficial effects:

[0016] Through the federated learning framework, the original multimodal user data of each participating client is kept locally, and only the soft labels (local knowledge) generated by the model prediction are uploaded. This fundamentally avoids the risk of leakage of sensitive privacy data. The knowledge distillation mechanism further reduces the exchanged content from parameters to soft labels, and combined with the trustworthiness awareness mechanism to resist malicious attacks, it builds a comprehensive privacy and security barrier from data to algorithm. The knowledge distillation mechanism transmits the "knowledge learned" by the model rather than the "model itself", which effectively overcomes the data heterogeneity problem caused by differences in user groups, and makes the global model have stronger generalization ability. The client adopts a "context-aware and dynamically weighted adaptive multimodal fusion model", which can dynamically adjust the weight of different modal data according to the specific context of the user (such as time, location, activity status), generate more accurate and representative fusion features, and significantly improve the expressive power of the local model. The central server only aggregates model knowledge (such as soft labels) that does not contain original information, thereby ensuring that the highly sensitive privacy of users is effectively protected at the legal and technical levels, and fully complies with regulatory requirements. The federated learning framework enables previously isolated "data silos" to collaboratively model without sharing data, greatly expanding the training sample size and data diversity of the model, and significantly improving the accuracy and generalization ability of the mental state assessment model. By combining the particle swarm optimization algorithm with the federated learning process, each particle in the particle swarm represents a potential global model solution. Guided by aggregated knowledge, the entire particle swarm performs parallel and efficient intelligent optimization in a complex solution space. Compared with the problem that traditional federated averaging algorithms may get stuck in local optima, this application can converge to a better-performing global model more quickly and accurately, greatly improving the training efficiency of federated learning and the upper limit of the final model.

[0017] In one optional implementation, a particle swarm is initialized at the central server, the globally optimal particle in the swarm is selected as the dominant model for this round, and the dominant model and a public unlabeled dataset are distributed to at least one participating client, including:

[0018] Define a particle representation scheme on the central server, initialize the particle swarm according to the particle representation scheme, and set the key parameters of the particle swarm optimization algorithm.

[0019] Establish a credibility record table for all registered participating clients, and initialize a baseline credibility score for each participating client in the credibility record table;

[0020] Based on the validation dataset, the fitness value of each particle in the particle swarm is obtained using a preset fitness function, and the particle with the best fitness value in this round is taken as the global optimal particle, thus obtaining the dominant model of this round of federated training.

[0021] The dominant model and the public unlabeled dataset will be distributed via an encrypted channel to at least one of the selected participating clients in this round.

[0022] In one optional implementation, the participating client trains the dominant model locally based on local multimodal user data using a context-aware and dynamically weighted adaptive multimodal fusion model, and uploads the obtained local knowledge to the central server, including:

[0023] Several raw multimodal user data are collected from the participating client, and the raw multimodal user data is standardized and preprocessed to generate several local standard multimodal user data.

[0024] Local standard multimodal user data is input into a context-aware and dynamically weighted adaptive multimodal fusion model to generate local fusion features;

[0025] Based on several local fusion features, the local model is trained locally, and the local multimodal user data is input into the master model issued by the central server for synchronous training to obtain the updated local model.

[0026] The public unlabeled dataset distributed by the central server is input into the updated local model for prediction to obtain predicted soft labels, and the predicted soft labels are uploaded to the central server as local knowledge.

[0027] In one alternative implementation, multimodal user data includes text data, voice data, facial expression data, eye movement and physiological signals, social media usage data, sleep data, and motion data.

[0028] In one alternative implementation, the context-aware and dynamically weighted adaptive multimodal fusion model includes a text feature extractor based on the BERT algorithm, a speech feature extractor based on the CRNN algorithm, a facial expression feature extractor based on the CNN+LSTM algorithm, an eye-tracking and physiological signal feature extractor based on the LSTM algorithm, other feature extractors based on the FCN algorithm, a context-aware module based on the MLP algorithm, and a dynamically weighted module based on the Attention mechanism.

[0029] In one alternative implementation, a central server receives local knowledge from each participating client and performs weighted aggregation based on the dynamic trust scores of each participating client to generate aggregated knowledge, including:

[0030] The central server receives local knowledge from each participating client and uses the verification dataset to evaluate the quality of each piece of local knowledge, obtaining the corresponding quality indicators.

[0031] Based on the quality indicators, update the baseline credibility score of each participating client in the credibility record table to obtain the updated credibility score;

[0032] The updated credibility score is used as the weight to perform a weighted average of the local knowledge of all participating clients, resulting in aggregated knowledge.

[0033] In one alternative implementation, the central server uses the aggregated knowledge to update the entire particle swarm, guiding all particles to optimize in the solution space, updating the globally optimal particle, and returning the dominant model selection and local training steps, including:

[0034] The aggregated knowledge is used as pseudo-labels on the central server. Supervised training is performed on the global optimal particle corresponding to the dominant model in this round on the public dataset. The trained parameters are used to cover the global optimal particle to obtain the global optimal particle of the previous round.

[0035] Based on the global optimal particle from the previous round, a convergence factor and a Levy flight mechanism are introduced to guide all particles in the swarm, except for the global optimal particle from the previous round, to search for optimization in the solution space, thus obtaining an updated swarm.

[0036] Based on the validation dataset, the fitness function is used to obtain the fitness value of each updated particle in the updated particle swarm, and the updated particle with the best fitness value in this round is updated as the global best particle in this round.

[0037] Based on the globally optimal particle in this round, return to the dominant model selection step.

[0038] In one alternative implementation, the above steps are repeated until the globally optimal particle in the particle swarm satisfies a preset convergence condition. Finally, the globally optimal particle is output as the global psychological state evaluation model and deployed to all participating clients, including:

[0039] Repeat the dominant model selection, local training, weighted aggregation, and particle swarm update steps to obtain the latest globally optimal particle in the particle swarm.

[0040] If the fitness value of the global best particle increases less than the preset improvement threshold for several consecutive rounds, or if the fitness value of the latest global best particle meets the fitness value threshold or reaches the maximum number of iterations, the latest global best particle will be used as the global psychological state evaluation model.

[0041] Freeze the global psychological state assessment model on the central server and deploy the frozen global psychological state assessment model to all participating client terminals.

[0042] In one optional implementation, a global psychological state assessment model is used on the participating client to assess the psychological state of the collected real-time multimodal user data, obtaining the user's psychological state assessment results, including:

[0043] Real-time multimodal user data is collected from the participating client and standardized preprocessed to generate real-time standard multimodal user data.

[0044] Real-time standard multimodal user data is input into a context-aware and dynamically weighted adaptive multimodal fusion model to generate real-time fusion features;

[0045] The real-time fused features are input into the global psychological state assessment model to assess the psychological state and obtain the user's psychological state assessment results.

[0046] Secondly, embodiments of the present invention provide a mental state assessment device based on federated learning for implementing a mental state assessment method. The device includes:

[0047] The dominant model selection unit is used to initialize the particle swarm on the central server, select the globally optimal particle in the particle swarm as the dominant model for this round, and distribute the dominant model and the public unlabeled dataset to at least one participating client.

[0048] The local training unit is used to train the dominant model locally on the participating client based on local multimodal user data, using a context-aware and dynamically weighted adaptive multimodal fusion model, and then upload the obtained local knowledge to the central server.

[0049] The weighted aggregation unit is used to receive local knowledge from each participating client at the central server and perform weighted aggregation based on the dynamic credibility scores of each participating client to generate aggregated knowledge.

[0050] The particle swarm update unit is used to update the entire particle swarm using the aggregated knowledge at the central server, guiding all particles to seek optimization in the solution space and updating the globally optimal particle.

[0051] The global model deployment unit is used to output the global optimal particle as a global psychological state evaluation model and deploy it to all participating clients when the global optimal particle in the particle swarm satisfies the preset convergence condition.

[0052] The global model application unit is used to evaluate the psychological state of users by using the global psychological state assessment model on the participating client to obtain the psychological state assessment results of the users.

[0053] A third aspect of this invention provides an electronic device, which includes:

[0054] At least one processor; and a memory communicatively connected to the at least one processor; wherein,

[0055] The memory stores instructions that can be executed by at least one processor, such that the at least one processor can perform the method proposed in the first aspect of the present invention.

[0056] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first aspect of the present invention. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention;

[0058] Figure 2 This is a flowchart illustrating the steps of a mental state assessment method based on federated learning provided in an embodiment of the present invention.

[0059] Figure 3 This is a functional unit diagram of a psychological state assessment device based on federated learning provided in an embodiment of the present invention. Detailed Implementation

[0060] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0061] The present invention will be further described below with reference to the accompanying drawings.

[0062] Reference Figure 1 , Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention.

[0063] like Figure 1As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0064] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0065] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and an electronic program for a mental state assessment device based on federated learning.

[0066] exist Figure 1 In the electronic device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device. The electronic device calls the electronic program of the federated learning-based psychological state assessment device stored in the memory 1005 through the processor 1001 and executes the federated learning-based psychological state assessment method provided in the embodiment of the present invention.

[0067] Reference Figure 2 The present invention provides a method for assessing mental state based on federated learning, the method comprising:

[0068] S201: Initialize the particle swarm on the central server, select the globally optimal particle in the particle swarm as the dominant model for this round, and distribute the dominant model and the public unlabeled dataset to at least one participating client.

[0069] S202: On the participating client, based on local multimodal user data, the dominant model is trained locally using a context-aware and dynamically weighted adaptive multimodal fusion model, and the obtained local knowledge is uploaded to the central server.

[0070] S203: The central server receives local knowledge from each participating client and performs weighted aggregation based on the dynamic credibility scores of each participating client to generate aggregated knowledge;

[0071] S204: The central server uses the aggregated knowledge to update the entire particle swarm, guides all particles to find the best solution in the solution space, updates the globally optimal particle, and returns to the dominant model selection step.

[0072] S205: Repeat the above steps until the global optimal particle in the particle swarm satisfies the preset convergence condition, and finally output the global optimal particle as the global psychological state evaluation model and deploy it to all participating client.

[0073] S206: Use a global psychological state assessment model on the participating client to assess the psychological state of the collected real-time multimodal user data and obtain the user's psychological state assessment results.

[0074] The technical solution provided in this application has at least the following beneficial effects:

[0075] Through the federated learning framework, the original multimodal user data of each participating client is kept locally, and only the soft labels (local knowledge) generated by the model prediction are uploaded. This fundamentally avoids the risk of leakage of sensitive privacy data. The knowledge distillation mechanism further reduces the exchanged content from parameters to soft labels, and combined with the trustworthiness awareness mechanism to resist malicious attacks, it builds a comprehensive privacy and security barrier from data to algorithm. The knowledge distillation mechanism transmits the "knowledge learned" by the model rather than the "model itself", which effectively overcomes the data heterogeneity problem caused by differences in user groups, and makes the global model have stronger generalization ability. The client adopts a "context-aware and dynamically weighted adaptive multimodal fusion model", which can dynamically adjust the weight of different modal data according to the specific context of the user (such as time, location, activity status), generate more accurate and representative fusion features, and significantly improve the expressive power of the local model. The central server only aggregates model knowledge (such as soft labels) that does not contain original information, thereby ensuring that the highly sensitive privacy of users is effectively protected at the legal and technical levels, and fully complies with regulatory requirements. The federated learning framework enables previously isolated "data silos" to collaboratively model without sharing data, greatly expanding the training sample size and data diversity of the model, and significantly improving the accuracy and generalization ability of the mental state assessment model. By combining the particle swarm optimization algorithm with the federated learning process, each particle in the particle swarm represents a potential global model solution. Guided by aggregated knowledge, the entire particle swarm performs parallel and efficient intelligent optimization in a complex solution space. Compared with the problem that traditional federated averaging algorithms may get stuck in local optima, this application can converge to a better-performing global model more quickly and accurately, greatly improving the training efficiency of federated learning and the upper limit of the final model.

[0076] In one optional implementation, a particle swarm is initialized at the central server, the globally optimal particle in the swarm is selected as the dominant model for this round, and the dominant model and a public unlabeled dataset are distributed to at least one participating client, including:

[0077] S2011: Define a particle representation scheme on the central server, initialize the particle swarm according to the particle representation scheme, and set the key parameters of the particle swarm optimization algorithm (including: inertia weight, individual learning factor, social learning factor, maximum number of iterations, etc.).

[0078] Option A (Model Parameter Space): All trainable parameters (weights and biases) of a complete global mental state assessment model; in this case, particle swarm optimization searches directly in the model parameter space.

[0079] Option B (Hyperparameter policy space): A set of hyperparameters controlling the federated learning process, such as the temperature parameter in knowledge distillation, the balance coefficient between hard and soft loss in the loss function, the client's local learning rate, and the parameters in the confidence update function. In this case, the particle swarm search for the optimal cooperative configuration in the policy space;

[0080] Scheme C (Hybrid Space): A particle simultaneously contains the initial perturbation vector of the model parameters and a set of hyperparameters, enabling more complex joint optimization;

[0081] This embodiment prefers scheme A to achieve end-to-end model optimization;

[0082] The central server sets the particle swarm size N (e.g., N=50) and initializes the particle swarm using the Tent chaotic mapping sequence.

[0083] The formula is:

[0084]

[0085] In the formula, The first in the particle swarm i One particle; For the first i One chaotic variable; These are the upper and lower bounds of the search space; i For particle indication;

[0086]

[0087] In the formula, For the first i+ One chaotic variable; compared with random initialization, chaotic initialization can ensure that the population is evenly distributed in the solution space, thus enhancing diversity.

[0088] S2012: Establish a credibility record table for all registered participating clients and initialize a baseline credibility score for each participating client in the credibility record table;

[0089] S2013: Based on the validation dataset, use the preset fitness function to obtain the fitness value of each particle in the particle swarm, and take the particle with the best fitness value in this round as the global optimal particle to obtain the dominant model of this round of federated training.

[0090] The formula is:

[0091]

[0092] In the formula, For particles X fitness value; X For particle reference parameters; For particles X The accuracy of the corresponding model on the validation dataset; For particles X The expected communication cost associated with the corresponding model (e.g., the larger the model, the higher the cost). For using particles in the previous round X When used as the dominant model, the variance of local performance of each client (used to measure fairness). For fitness weighting coefficients;

[0093] The server records the particle with the highest fitness in the current round and marks its position (model parameters) as the global optimal position. For each particle, it compares its fitness in the current round with its own historical best fitness. If it is better in the current round, it updates its individual historical best position.

[0094] S2014: Distribute the dominant model and a public, unlabeled dataset containing no sensitive information to at least one of the selected participating clients in this round via an encrypted channel.

[0095] In one optional implementation, the participating client trains the dominant model locally based on local multimodal user data using a context-aware and dynamically weighted adaptive multimodal fusion model, and uploads the obtained local knowledge to the central server, including:

[0096] S2021: Collect some raw multimodal user data from the participating party's client, and perform standardized preprocessing on the raw multimodal user data to generate some local standard multimodal user data.

[0097] In this embodiment, multimodal user data includes text data, voice data, facial expression data, eye movement and physiological signals, social media usage data, sleep data, and motion data;

[0098] Raw data is often heterogeneous, high-dimensional, noisy, and in various formats. Directly inputting this raw data into the model will not only lead to low training efficiency and difficulty in convergence, but will also seriously affect the accuracy and reliability of the final evaluation results. Therefore, "standardized preprocessing" is an indispensable bridge between raw data and advanced models. Its core goal is to transform raw data from different sources and in different formats into a unified, regular, and information-dense standard data format for subsequent feature extraction and model fusion.

[0099] Standardization preprocessing of text data:

[0100] Data cleaning:

[0101] Removing irrelevant information: Delete HTML tags, URLs, special symbols, emojis (unless the emojis themselves are the analysis objects), etc.;

[0102] Standardizing the format: Convert full-width characters to half-width characters and unify the English case;

[0103] Handling noise: Correct spelling mistakes and remove repeated or meaningless words (such as "ah", "um");

[0104] Word segmentation and tokenization:

[0105] Split continuous text strings into independent lexical or sub-word units (tokens); for example, split the sentence "I'm in a very bad mood today" into ["I", "today", "mood", "very", "bad"];

[0106] Removing stop words:

[0107] Remove words that contribute little to semantic expression but have a high frequency of occurrence, such as "de", "le", "shi", etc. This helps reduce the data dimension and highlight the core information;

[0108] Standardizing the output:

[0109] Finally, convert the processed text into a sequence of integer IDs, where each ID corresponds to a word in the dictionary. For example, ["I", "today", "mood", "very", "bad"] may be converted to [101, 102, 103, 104, 105]; this fixed-length numerical sequence is the standardized text data and can be directly input into the text feature extractor;

[0110] Standard preprocessing of speech data:

[0111] [[ID=~30]]Data sampling and quantization:

[0112] Sampling: Sample the analog speech signal at a fixed frequency (such as 16 kHz or ~44.1 kHz) and convert it into a discrete time series;

[0113] Quantization: Map the amplitude value of each sampling point to a finite number of discrete levels;

[0114] Pre-emphasis:

[0115] Amplify the high-frequency part of the speech signal through a high-pass filter to compensate for the energy attenuation of the high-frequency part during the generation of the speech signal and make the spectrum flatter;

[0116] Framing and windowing:

[0117] Note: There seems to be an error in the original text where "44.1kHz" is written as "~44.1kHz" in the "Sampling" section. It is assumed that the "~" is a typo and should be removed in the translation. If this is not the case, please clarify the meaning. Also, the number "30" in the "Data sampling and quantization" section has a tilde (~) in the original text which is likely an error and has been removed in the translation. If this is incorrect, please provide the correct information.Since the speech signal is stationary for a short period of time (about 10-30ms), it needs to be divided into a series of short frames;

[0118] To reduce spectral leakage at frame boundaries, each frame of data needs to be multiplied by a window function (such as a Hamming window or a Hanning window).

[0119] Noise suppression and silence detection:

[0120] Use algorithms such as spectral subtraction and Wiener filtering to remove background noise;

[0121] Silence segments are detected and removed using features such as energy or zero-crossing rate, retaining only the valid speech portion;

[0122] Standardized output:

[0123] Each frame of the processed speech signal is converted into acoustic features, such as Mel frequency cepstral coefficients or Mel spectrograms. These features are fixed-dimensional vectors or two-dimensional matrices that can effectively characterize the acoustic content of speech and serve as the standard input for the speech feature extractor.

[0124] Standardized preprocessing of facial expression data:

[0125] Face detection and alignment:

[0126] Use algorithms (such as YOLO) to locate face regions in image or video frames;

[0127] Alignment is performed based on key points such as eyes, nose, and corners of the mouth, and the face is rotated and scaled to a standard pose to eliminate the effects of changes in head posture and position.

[0128] Clipping and Normalization:

[0129] Cropping out the aligned face area and scaling it uniformly to a fixed size (e.g., 224x224 pixels).

[0130] Normalize pixel values, for example, scale the range of pixel values ​​in the range of [0, 255] to the range of [0, 1] or [-1, 1].

[0131] Data augmentation (optional):

[0132] To increase the model's generalization ability, operations such as random flipping, slight rotation, and color jitter can be performed;

[0133] Standardized output:

[0134] The final result is a tensor of a fixed size (e.g., 3x224x224), where 3 represents the three RGB channels. This tensor is the standard input for the facial expression feature extractor.

[0135] Standardized preprocessing of eye movement and physiological signals:

[0136] Signal denoising:

[0137] Use low-pass, high-pass, or band-pass filters to remove noise such as power frequency interference, baseline drift, and electromyography.

[0138] Signal segmentation:

[0139] Divide a continuous stream of signals into data segments based on events (such as starting a task) or fixed time windows (such as every 30 seconds);

[0140] Outlier Detection and Handling:

[0141] Identify and handle abrupt changes caused by equipment malfunctions or abnormal user activity (such as loss of eye-tracking data or sudden spikes in heart rate); interpolation, smoothing, or direct removal methods can be used.

[0142] Standardization and normalization:

[0143] Z-score standardization: For each data segment, calculate its mean (μ) and standard deviation (σ), and convert the data point x to (x-μ) / σ; this can eliminate the huge differences in the dimensions of different physiological indicators (such as heart rate, skin conductance);

[0144] Min-Max normalization: linearly maps data to the interval [0, 1] or [-1, 1].

[0145] Standardized output:

[0146] It outputs a series of time series data with fixed length and normalized numerical range, which can be directly input into the eye movement and physiological signal feature extractor;

[0147] Standardized preprocessing of social media usage data:

[0148] Feature extraction:

[0149] Extract meaningful statistical features from the raw logs, such as: daily usage time, frequency of app opening, number of posts / likes / comments, active time periods, and social network size;

[0150] Data transformation:

[0151] Perform one-hot encoding on categorized data (such as App type);

[0152] Timestamp data can be converted into periodic features such as "time of day" and "day of the week";

[0153] Standardized output:

[0154] This results in a fixed-dimensional numerical feature vector. This vector is then normalized using Min-Max or Z-score to make it comparable to the data scale of other modalities, and used as input to other feature extractors.

[0155] Standardized preprocessing of sleep and exercise data:

[0156] Data Analysis:

[0157] Parse the proprietary data format from equipment manufacturers and extract key metrics;

[0158] Sleep data: total sleep duration, deep sleep / light sleep / REM duration, sleep efficiency, sleep latency, number of awakenings during the night, etc.

[0159] Exercise data: steps, calories burned, activity duration, type of exercise, heart rate zone distribution, etc.

[0160] Time alignment and aggregation:

[0161] The data is aggregated by day or week to generate daily / weekly sleep and activity profiles;

[0162] Missing value handling:

[0163] For data loss caused by users not wearing devices, interpolation, filling with default values ​​(such as 0), or model prediction can be used.

[0164] Standardized output:

[0165] Similar to social media data, it is converted into a fixed-dimensional numerical feature vector and normalized before being input into other feature extractors.

[0166] S2022: Input local standard multimodal user data into a context-aware and dynamically weighted adaptive multimodal fusion model to generate local fusion features;

[0167] In this embodiment, the context-aware and dynamically weighted adaptive multimodal fusion model includes a text feature extractor based on the BERT algorithm, a speech feature extractor based on the CRNN algorithm, a facial expression feature extractor based on the CNN+LSTM algorithm, an eye movement and physiological signal feature extractor based on the LSTM algorithm, other feature extractors based on the FCN algorithm, a context-aware module based on the MLP algorithm, and a dynamically weighted module based on the Attention mechanism.

[0168] Text feature extractor: Employs the Bidirectional Encoder Representations from Transformers (BERT) algorithm to deeply understand text semantics;

[0169] Speech feature extractor: Employs a Convolutional Recurrent Neural Network (CRNN) algorithm, combining a convolutional neural network to extract local acoustic features and a recurrent neural network to capture temporal dependencies;

[0170] Facial expression feature extractor: It adopts the Convolutional Neural Network (CNN) + Long Short Term Memory Network (LSTM) algorithm. CNN is used to extract facial key point features, and LSTM is used to analyze the dynamic changes of facial expressions.

[0171] Eye movement and physiological signal feature extractor: The LSTM algorithm is used to process the time series of eye movement trajectories and physiological signals;

[0172] Other feature extractors: Employ fully convolutional networks (FCNs) to process structured data such as social media, sleep, and motion.

[0173] Context-aware module: A module based on the Multilayer Perceptron (MLP) algorithm. It receives external metadata and low- or mid-level features of each modality, rather than high-level features after deep abstraction. It takes current context information as input. For example, it can receive word frequency statistics of text, volume / speed of speech, activation intensity of facial action units (AU), gaze distribution of eye movements, and timestamps (such as weekday / weekend, day / night). It outputs a context vector, which is a mathematical representation of the context in which the current data is located. For example, it may encode complex contexts such as "late night - alone - high mobile phone use - low social interaction" or "daytime - meeting - high vocal activity - positive facial expressions".

[0174] Dynamic weighting module: An attention-based module that receives a context vector and dynamically generates a set of attention weights for each modality's feature extractor. Through training, the network learns the complex mapping relationship between context and modality importance. For example, when a user is moving, physiological and motion data will have higher weights; when a user is chatting socially, text and voice data will have higher weights. Finally, this module outputs a weighted and fused local feature vector.

[0175] S2023: Based on several local fusion features, the local model is trained locally, and the local multimodal user data is input into the dominant model issued by the central server for synchronous training to obtain the updated local model;

[0176] In this embodiment, local multimodal user data is input into the dominant model distributed by the central server for synchronous training to obtain its teacher soft label prediction results. ;

[0177] The model outputs not hard labels (such as 0 or 1), but probability distributions, i.e. "soft labels". For example, for depression risk assessment, the output may be [0.1, 0.8, 0.1], representing the probability of no, mild, and moderate to severe depression, respectively.

[0178] The formula for the total loss function of the local model is:

[0179]

[0180] In the formula, This represents the total loss value. This is the balance coefficient; Prediction results for student models With real labels The cross-entropy loss is used, and the student model is a local model; Soft label prediction results for student models Teacher soft label prediction results The KL divergence loss between the two models is used, with the teacher model being the dominant model.

[0181] If the total loss value meets the total loss value threshold, the local knowledge distillation training is completed, and the updated local model is obtained.

[0182] S2024: Input the public unlabeled dataset issued by the central server into the updated local model for prediction, obtain the predicted soft label, and upload the predicted soft label as local knowledge to the central server;

[0183] In this embodiment, after training locally, participating clients use a small portion of the local validation set to evaluate their own model performance (such as accuracy), and upload this performance value along with the soft label to the central server for the central server to calculate the variance of each client's local performance.

[0184] In one alternative implementation, a central server receives local knowledge from each participating client and performs weighted aggregation based on the dynamic trust scores of each participating client to generate aggregated knowledge, including:

[0185] S2031: The central server receives local knowledge from each participating party's client and uses the verification dataset to evaluate the quality of each piece of local knowledge, obtaining the corresponding quality indicators.

[0186] The validation dataset consists of input data and true labels:

[0187] For each participating client, the central server uses its uploaded local knowledge to "predict" the input portion of the validation dataset, detailed as follows:

[0188] The key point is that local knowledge is a soft label for the model on a public dataset; it is a model itself, not a direct prediction function. To use local knowledge to predict validation datasets, a knowledge transfer hypothesis and operation are needed. The most direct and effective method is:

[0189] Training a proxy model: Using local knowledge as a supervision signal, quickly train a lightweight proxy model on a public dataset. The proxy model can have a very simple structure, such as one or two fully connected layers.

[0190] Training objective: Minimize the discrepancy between the agent model's predictions on public datasets and local knowledge;

[0191] Once trained, this proxy model can be considered an "embodiment" of local knowledge; the central server uses the input from the validation dataset to make predictions and obtain the predicted soft labels:

[0192] Based on the predicted soft labels, one or more quality metrics can be calculated, including accuracy, F1 score, and cross-entropy loss. You can choose one of the above metrics or combine multiple metrics into a comprehensive quality score.

[0193] S2032: Based on the quality indicators, update the baseline credibility score of each participating client in the credibility record table to obtain the updated credibility score;

[0194] The formula is:

[0195]

[0196] In the formula, For the first j Credibility score of updates from participating client devices; For the first j The benchmark credibility score of the participating client; For the first j Quality metrics for participating client applications; It is a smoothing factor between 0 and 1, used to balance historical performance and current performance; j The amount indicated by the participating client;

[0197] S2033: Using the updated credibility score as weight, perform a weighted average of the local knowledge of all participating clients to obtain aggregated knowledge;

[0198] The formula is:

[0199]

[0200] In the formula, To aggregate knowledge; For local knowledge; J The total number of participating clients in federated learning;

[0201] In one alternative implementation, the central server uses the aggregated knowledge to update the entire particle swarm, guiding all particles to optimize in the solution space, updating the globally optimal particle, and returning the dominant model selection and local training steps, including:

[0202] S2041: The aggregated knowledge is used as a pseudo-label on the central server. Supervised training is performed on the global optimal particle corresponding to the dominant model in this round on the public dataset. The trained parameters are used to cover the global optimal particle to obtain the global optimal particle of the previous round.

[0203] S2042: Based on the global optimal particle of the previous round, a convergence factor and Levy flight mechanism are introduced to guide all particles in the particle swarm except the global optimal particle of the previous round to search for optimization in the solution space, and obtain an updated particle swarm.

[0204] The formula is:

[0205]

[0206] In the formula, Number of iterations t+ 1 of i The search direction for an updated particle; The convergence factor; Number of iterations t The iThe updated historical best position of the particle; Number of iterations t The globally optimal particle; Number of iterations t The i An updated particle; For individual learning factors and social learning factors, both are random numbers between [0,1]. Inertial weights; for Levy Distributed random numbers; b for Levy Step length, and b ∈[1,2]; t This represents the current iteration number; This is the step size parameter;

[0207]

[0208] In the formula, These are the maximum and minimum values ​​of the convergence factor; This represents the maximum number of iterations. t This represents the current iteration number; , To adjust the parameters; It is the hyperbolic tangent function;

[0209]

[0210] In the formula, Number of iterations t+ 1 of i An updated particle;

[0211] S2043: Based on the validation dataset, use the fitness function to obtain the fitness value of each updated particle in the updated particle swarm, and update the updated particle with the best fitness value in this round as the global best particle in this round.

[0212] S2044: Based on the globally optimal particle in this round, return to the dominant model selection step.

[0213] In one alternative implementation, the above steps are repeated until the globally optimal particle in the particle swarm satisfies a preset convergence condition. Finally, the globally optimal particle is output as the global psychological state evaluation model and deployed to all participating clients, including:

[0214] S2051: Repeat the dominant model selection, local training, weighted aggregation and particle swarm update steps to obtain the latest global optimal particle in the particle swarm.

[0215] S2052: If the fitness value of the global best particle increases less than the preset improvement threshold for several consecutive rounds, or if the fitness value of the latest global best particle meets the fitness value threshold or reaches the maximum number of iterations, the latest global best particle will be used as the global psychological state evaluation model.

[0216] S2053: Freeze the global psychological state assessment model of the central server and deploy the frozen global psychological state assessment model to all participating client clients.

[0217] In one optional implementation, a global psychological state assessment model is used on the participating client to assess the psychological state of the collected real-time multimodal user data, obtaining the user's psychological state assessment results, including:

[0218] S2061: Collect real-time multimodal user data from participating clients and perform standardized preprocessing on the real-time multimodal user data to generate real-time standard multimodal user data;

[0219] S2062: Input real-time standard multimodal user data into a context-aware and dynamically weighted adaptive multimodal fusion model to generate real-time fusion features;

[0220] S2063: Input real-time fused features into the global psychological state assessment model to evaluate the user's psychological state, obtaining the user's psychological state assessment results (such as depression risk level, anxiety index, emotional state, etc.). This result can be presented to the user or, with the user's authorization, provided to a health consultant or doctor as a basis for auxiliary diagnosis and intervention.

[0221] This invention also provides a mental state assessment device based on federated learning, referring to... Figure 3 The diagram shows a functional unit diagram of a mental state assessment device 300 based on federated learning according to the present invention. The device may include the following units:

[0222] The dominant model selection unit 301 is used to initialize the particle swarm on the central server, select the globally optimal particle in the particle swarm as the dominant model for this round, and distribute the dominant model and the public unlabeled dataset to at least one participating client.

[0223] The local training unit 302 is used to train the dominant model locally on the participant's client based on local multimodal user data through a context-aware and dynamically weighted adaptive multimodal fusion model, and upload the obtained local knowledge to the central server.

[0224] The weighted aggregation unit 303 is used to receive local knowledge from each participating client at the central server and perform weighted aggregation based on the dynamic credibility scores of each participating client to generate aggregated knowledge.

[0225] Particle swarm update unit 304 is used to update the entire particle swarm using the aggregated knowledge in the central server, guide all particles to seek optimization in the solution space, and update the globally optimal particle.

[0226] The global model deployment unit 305 is used to output the global optimal particle as a global psychological state evaluation model and deploy it to all participating client when the global optimal particle in the particle swarm satisfies the preset convergence condition.

[0227] The global model application unit 306 is used to evaluate the psychological state of the collected real-time multimodal user data on the participant's client using the global psychological state assessment model, and obtain the user's psychological state assessment results.

[0228] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0229] Memory, used to store computer programs;

[0230] The processor, when executing a program stored in memory, implements the federated learning-based mental state assessment method of the present invention.

[0231] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0232] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0233] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a computer-readable storage medium storing a computer program that, when executed by a processor, implements the federated learning-based mental state assessment method of the present invention.

[0234] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable hardware devices (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0235] The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0236] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0237] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0238] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "And / or" indicates that either one or both can be chosen. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0239] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for assessing mental state based on federated learning, characterized in that, The method includes: The central server initializes the particle swarm, selects the globally optimal particle in the swarm as the dominant model for this round, and distributes the dominant model and the public unlabeled dataset to at least one participating client, including: Define a particle representation scheme on the central server, initialize the particle swarm according to the particle representation scheme, and set the key parameters of the particle swarm optimization algorithm. Establish a credibility record table for all registered participating clients, and initialize a baseline credibility score for each participating client in the credibility record table; Based on the validation dataset, the fitness value of each particle in the particle swarm is obtained using a preset fitness function, and the particle with the best fitness value in this round is taken as the global optimal particle, thus obtaining the dominant model of this round of federated training. The dominant model and the public unlabeled dataset will be distributed to at least one selected participant client in this round via an encrypted channel; On the participating client side, based on local multimodal user data, the dominant model is trained locally using a context-aware and dynamically weighted adaptive multimodal fusion model, and the obtained local knowledge is uploaded to the central server. The central server receives local knowledge from each participating client and performs weighted aggregation based on the dynamic credibility scores of each participating client to generate aggregated knowledge. The central server uses the aggregated knowledge to update the entire particle swarm, guides all particles to find the best solution in the solution space, updates the globally optimal particle, and returns to the dominant model selection step. Repeat the above steps until the globally optimal particle in the particle swarm satisfies the preset convergence condition. Finally, output the globally optimal particle as the global psychological state evaluation model and deploy it to all participating client devices. The participating client uses a global psychological state assessment model to assess the psychological state of the collected real-time multimodal user data, and obtains the psychological state assessment results of the users.

2. The method for assessing mental state based on federated learning according to claim 1, characterized in that, On the participating client side, based on local multimodal user data, the dominant model is trained locally using a context-aware and dynamically weighted adaptive multimodal fusion model, and the resulting local knowledge is uploaded to the central server, including: Several raw multimodal user data are collected from the participating client, and the raw multimodal user data is standardized and preprocessed to generate several local standard multimodal user data. Local standard multimodal user data is input into a context-aware and dynamically weighted adaptive multimodal fusion model to generate local fusion features; Based on several local fusion features, the local model is trained locally, and the local multimodal user data is input into the master model issued by the central server for synchronous training to obtain the updated local model. The public unlabeled dataset distributed by the central server is input into the updated local model for prediction to obtain predicted soft labels, and the predicted soft labels are uploaded to the central server as local knowledge.

3. The method for assessing mental state based on federated learning according to claim 2, characterized in that, The multimodal user data includes text data, voice data, facial expression data, eye movement and physiological signals, social media usage data, sleep data, and motion data.

4. The method for assessing mental state based on federated learning according to claim 3, characterized in that, The context-aware and dynamically weighted adaptive multimodal fusion model includes a text feature extractor based on the BERT algorithm, a speech feature extractor based on the CRNN algorithm, a facial expression feature extractor based on the CNN+LSTM algorithm, an eye movement and physiological signal feature extractor based on the LSTM algorithm, other feature extractors based on the FCN algorithm, a context-aware module based on the MLP algorithm, and a dynamically weighted module based on the Attention mechanism.

5. The method for assessing mental state based on federated learning according to claim 4, characterized in that, The central server receives local knowledge from each participating client and performs weighted aggregation based on the dynamic credibility scores of each participating client to generate aggregated knowledge, including: The central server receives local knowledge from each participating client and uses the verification dataset to evaluate the quality of each piece of local knowledge, obtaining the corresponding quality indicators. Based on the quality indicators, update the baseline credibility score of each participating client in the credibility record table to obtain the updated credibility score; The updated credibility score is used as the weight to perform a weighted average of the local knowledge of all participating clients, resulting in aggregated knowledge.

6. The method for assessing mental state based on federated learning according to claim 5, characterized in that, The central server uses the aggregated knowledge to update the entire particle swarm, guiding all particles to find the best solution in the solution space, updating the globally optimal particle, and returning the dominant model selection and local training steps, including: The aggregated knowledge is used as pseudo-labels on the central server. Supervised training is performed on the global optimal particle corresponding to the dominant model in this round on the public dataset. The trained parameters are used to cover the global optimal particle to obtain the global optimal particle of the previous round. Based on the global optimal particle from the previous round, a convergence factor and a Levy flight mechanism are introduced to guide all particles in the swarm, except for the global optimal particle from the previous round, to search for optimization in the solution space, thus obtaining an updated swarm. Based on the validation dataset, the fitness function is used to obtain the fitness value of each updated particle in the updated particle swarm, and the updated particle with the best fitness value in this round is updated as the global best particle in this round. Based on the globally optimal particle in this round, return to the dominant model selection step.

7. The method for assessing mental state based on federated learning according to claim 6, characterized in that, Repeat the above steps until the globally optimal particle in the particle swarm satisfies the preset convergence condition. Finally, output the globally optimal particle as the global psychological state evaluation model and deploy it to all participating clients, including: Repeat the dominant model selection, local training, weighted aggregation, and particle swarm update steps to obtain the latest globally optimal particle in the particle swarm. If the fitness value of the global best particle increases less than the preset improvement threshold for several consecutive rounds, or if the fitness value of the latest global best particle meets the fitness value threshold or reaches the maximum number of iterations, the latest global best particle will be used as the global psychological state evaluation model. Freeze the global psychological state assessment model on the central server and deploy the frozen global psychological state assessment model to all participating client terminals.

8. The method for assessing mental state based on federated learning according to claim 7, characterized in that, The participating client uses a global psychological state assessment model to assess the psychological state of the collected real-time multimodal user data, obtaining the user's psychological state assessment results, including: Real-time multimodal user data is collected from the participating client and standardized preprocessed to generate real-time standard multimodal user data. Real-time standard multimodal user data is input into a context-aware and dynamically weighted adaptive multimodal fusion model to generate real-time fusion features; The real-time fused features are input into the global psychological state assessment model to assess the psychological state and obtain the user's psychological state assessment results.

9. A psychological state assessment device based on federated learning, used to implement the psychological state assessment method as described in any one of claims 1-8, characterized in that, The device includes: The dominant model selection unit is used to initialize a particle swarm on the central server, select the globally optimal particle in the particle swarm as the dominant model for this round, and distribute the dominant model and the public unlabeled dataset to at least one participating client. This includes: defining a particle representation scheme on the central server, initializing the particle swarm according to the scheme, and setting key parameters of the particle swarm optimization algorithm; establishing a credibility record table for all registered participating clients and initializing a baseline credibility score for each participating client in the credibility record table; obtaining the fitness value of each particle in the particle swarm using a preset fitness function based on the validation dataset, and selecting the particle with the best fitness value in this round as the globally optimal particle, thus obtaining the dominant model for this round of federated training; and distributing the dominant model and the public unlabeled dataset to at least one selected participating client in this round through an encrypted channel. The local training unit is used to train the dominant model locally on the participating client based on local multimodal user data, using a context-aware and dynamically weighted adaptive multimodal fusion model, and then upload the obtained local knowledge to the central server. The weighted aggregation unit is used to receive local knowledge from each participating client at the central server and perform weighted aggregation based on the dynamic credibility scores of each participating client to generate aggregated knowledge. The particle swarm update unit is used to update the entire particle swarm using the aggregated knowledge at the central server, guiding all particles to seek optimization in the solution space and updating the globally optimal particle. The global model deployment unit is used to output the global optimal particle as a global psychological state evaluation model and deploy it to all participating clients when the global optimal particle in the particle swarm satisfies the preset convergence condition. The global model application unit is used to evaluate the psychological state of users by using the global psychological state assessment model on the participating client to obtain the psychological state assessment results of the users.

Images