Human factor intelligent state prediction method and device, edge computing equipment and medium

Abstract

Embodiments of the invention provide a human factor intelligent state prediction method and apparatus, an edge computing device and a medium. The method comprises the steps of obtaining first physiological signal data; performing individual difference signal removal on the first physiological signal data to obtain second physiological signal data; and performing state prediction of the user based on the second physiological signal data. According to the embodiment of the invention, the accuracy of prediction results is improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence and human factors engineering technology, and specifically relates to a human factors intelligence state prediction method, device, edge computing equipment and medium. Background Art

[0002] With the development of computer science and technology, the use of objective measurement indicators to monitor the status of individuals in real time has had a huge impact on the fields of traffic human factors, aerospace, industrial construction, etc. For example, by modeling the driver's EEG signals, early warning of the driver's fatigue state and dangerous driving behavior can be issued to reduce the accident rate.

[0003] In the existing technology, a prediction model is trained based on training samples to obtain a trained prediction model, and user status is predicted based on the trained prediction model.

[0004] However, the accuracy of the prediction results of the state prediction using the prediction model of the existing technology is not high. Summary of the invention

[0005] The embodiments of the present application provide a human factors intelligent state prediction method, device, edge computing device and medium to solve the problem that the prediction results are not accurate when the state prediction model of the prior art is used for state prediction.

[0006] In a first aspect, an embodiment of the present application provides a method for predicting a human intelligence state, comprising:

[0007] Acquire first physiological signal data, where the first physiological signal data includes multimodal physiological signal data of a user;

[0008] performing individual difference signal removal on the first physiological signal data to obtain second physiological signal data;

[0009] Based on the second physiological signal data, the state of the user is predicted.

[0010] Optionally, removing individual difference signals from the first physiological signal data to obtain second physiological signal data includes:

[0011] Preprocessing the multimodal physiological data included in the first physiological signal data to obtain preprocessed first physiological signal data, wherein the preprocessing includes at least one of the following: filtering, artifact removal and independent component analysis; the multimodal physiological signal data includes at least one of the following: electroencephalogram (EEG), heart rate variability (HRV), skin electrode activity (EDA), respiratory signal (RESP), photoplethysmogram (PPG), skin temperature (SKT), blood oxygen saturation (SPO2), electrocardiogram (ECG), electromyogram (EMG) and human body nine-axis posture data;

[0012] Performing feature extraction on the preprocessed first physiological signal data to obtain a user feature value corresponding to the first physiological signal, wherein the user feature value includes: a time domain feature value and a frequency domain feature value, the time domain feature value includes at least one of the following: a maximum value, a minimum value, and a standard deviation, and the frequency domain feature value includes: a power spectrum density of different frequency bands;

[0013] Based on the global difference space matrix and the individual vector, individual difference signal removal processing is performed on the user characteristic value to obtain second physiological signal data.

[0014] Optionally, the predicting the state of the user based on the second physiological signal data includes:

[0015] The second physiological signal data is input into a prediction model to predict the state of the user to obtain a prediction result, which includes at least one of the following: a cognitive load prediction result, an emotional state prediction result, a stress prediction result, and a fatigue prediction result.

[0016] Optionally, performing individual difference signal removal processing on the user characteristic value based on the global difference space matrix and the individual vector to obtain the second physiological signal data includes:

[0017] Based on the difference space matrix and the individual vector of the user, obtaining the individual difference contribution value of the user;

[0018] Based on the user characteristic value and the individual difference contribution value, second physiological signal data is obtained.

[0019] Optionally, before performing individual difference signal removal processing on the user characteristic value based on the global difference space matrix and the individual vector to obtain the second physiological signal data, the method further includes:

[0020] Acquire the global difference space matrix and the individual vector through training;

[0021] The step of obtaining the global difference space matrix and the individual vector through training includes:

[0022] Obtain a training sample data set, perform feature extraction on each training sample data, and obtain sample features;

[0023] Construct a global difference space matrix and initialize the parameters of the global difference space matrix;

[0024] Updating the parameters of the global difference space matrix based on the sample features and an iterative algorithm;

[0025] Based on the global difference space matrix, the individual vectors are obtained.

[0026] Optionally, updating the parameters of the global difference space matrix based on the sample features and an iterative algorithm includes:

[0027] Based on the sample characteristics, obtain the Baum-Welch statistic of each sample user;

[0028] Based on the Baum-Welch statistic of each sample user, the posterior distribution of the latent variable is calculated, and the posterior distribution includes: mean and expectation;

[0029] According to the posterior distribution of the latent variables, the parameters of the global difference space matrix are updated using maximum likelihood estimation;

[0030] Determine whether the global difference space matrix converges, if not, return to execute the Baum-Welch statistic based on each sample user, calculate the posterior distribution of the latent variable, the posterior distribution includes: mean and expectation, until the global difference space matrix converges.

[0031] In a second aspect, an embodiment of the present application provides a human intelligence state prediction device, comprising:

[0032] An acquisition module, configured to acquire first physiological signal data, wherein the first physiological signal data includes multimodal physiological signal data of a user;

[0033] A processing module, used for removing individual difference signals from the first physiological signal data to obtain second physiological signal data;

[0034] A prediction module is used to predict the state of the user based on the second physiological signal data.

[0035] Optionally, the processing module is specifically used to preprocess the multimodal physiological data included in the first physiological signal data to obtain the preprocessed first physiological signal data, wherein the preprocessing includes at least one of the following: filtering, artifact removal and independent component analysis; the multimodal physiological signal data includes at least one of the following: electroencephalogram (EEG), heart rate variability (HRV), skin electrodermal activity (EDA), respiratory signal (RESP), photoplethysmogram (PPG), skin temperature (SKT), blood oxygen saturation (SPO2), electrocardiogram (ECG), electromyogram (EMG) and human nine-axis posture data; feature extraction is performed on the preprocessed first physiological signal data to obtain user characteristic values ​​corresponding to the first physiological signal, wherein the user characteristic values ​​include: time domain characteristic values ​​and frequency domain characteristic values, the time domain characteristic values ​​include at least one of the following: maximum value, minimum value and standard deviation, and the frequency domain characteristic values ​​include: power spectral density of different frequency bands; based on the global difference space matrix and the individual vector, the user characteristic values ​​are subjected to individual difference signal removal processing to obtain the second physiological signal data.

[0036] Optionally, the prediction module is specifically used to input the second physiological signal data into a prediction model to predict the user's state and obtain a prediction result, which includes at least one of the following: a cognitive load prediction result, an emotional state prediction result, a stress prediction result, and a fatigue prediction result.

[0037] Optionally, the processing module is specifically used to obtain the individual difference contribution value of the user based on the difference space matrix and the individual vector of the user; and obtain the second physiological signal data based on the user characteristic value and the individual difference contribution value.

[0038] Optionally, the processing module is further used to obtain the global difference space matrix and the individual vector through training;

[0039] The processing module is specifically used to obtain a set of training sample data, perform feature extraction on each training sample data, and obtain sample features; construct a global difference space matrix and initialize the parameters of the global difference space matrix; update the parameters of the global difference space matrix based on the sample features and an iterative algorithm; and obtain the individual vector based on the global difference space matrix.

[0040] Optionally, the processing module is specifically used to obtain the Baum-Welch statistic of each sample user based on the sample features; calculate the posterior distribution of the latent variable based on the Baum-Welch statistic of each sample user, and the posterior distribution includes: mean and expectation; update the parameters of the global difference space matrix using maximum likelihood estimation according to the posterior distribution of the latent variable; determine whether the global difference space matrix converges, and if not, return to execute the Baum-Welch statistic based on each sample user to calculate the posterior distribution of the latent variable, and the posterior distribution includes: mean and expectation, until the global difference space matrix converges.

[0041] In a third aspect, an embodiment of the present application provides an edge computing device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the human intelligence state prediction method as described in any one of the first aspects are implemented.

[0042] Optionally, also include:

[0043] A physiological signal data acquisition unit, used to acquire first physiological signal data; and / or,

[0044] The physiological signal data acquisition interface unit is used to connect to the physiological signal acquisition unit to acquire the first physiological signal data.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the human intelligence state prediction method as described in any one of the first aspects are implemented.

[0046] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed by a processor, implements the steps of the human intelligence state prediction method as described in the first aspect above.

[0047] The human factors intelligent state prediction method, apparatus, edge computing device and medium provided in the embodiments of the present application obtain first physiological signal data, wherein the first physiological signal data includes multimodal physiological signal data of the user; remove individual difference signals from the first physiological signal data to obtain second physiological signal data; and predict the state of the user based on the second physiological signal data. When predicting the state of the user, the embodiments of the present application use the second physiological signal data with the individual difference signals removed to predict the state of the user, which is conducive to reducing the influence of the individual difference signals on the prediction results and improving the accuracy of the prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of a process flow of a method for predicting human intelligence status provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of a process flow of another method for predicting human intelligence status provided in an embodiment of the present application;

[0050] Figure 3 A flowchart of another method for predicting human intelligence status provided in an embodiment of the present application;

[0051] Figure 4 A flowchart of another method for predicting human intelligence status provided in an embodiment of the present application;

[0052] Figure 5 A flowchart of another method for predicting human intelligence status provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of the structure of a human factors intelligence state prediction device provided in an embodiment of the present application;

[0054] Fig. 7A A schematic diagram of the structure of an edge computing device provided in an embodiment of the present application;

[0055] Figure 7B A schematic diagram of the structure of another edge computing device provided in an embodiment of the present application;

[0056] Figure 7C A schematic diagram of the structure of another edge computing device provided in an embodiment of the present application;

[0057] Fig.7D A schematic diagram of the structure of another edge computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0059] The terms "first", "second", etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of one type, and the number of objects is not limited, for example, the first object can be one or more. In addition, "or" in the present application represents at least one of the connected objects. For example, "A or B" covers three schemes, namely, Scheme 1: including A but not including B; Scheme 2: including B but not including A; Scheme 3: including both A and B. The character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0060] The term "indication" in this application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, operations to be performed, or request results in the sent indication; an indirect indication can be understood as the receiver determining the corresponding information based on the indication sent by the sender, or making a judgment and determining the operation to be performed or the request result based on the judgment result.

[0061] Human intelligence is based on human factors engineering technology combined with artificial intelligence (AI) technology. Human factors engineering technology is the application of human psychological and physiological principles to the engineering and design of products, processes and systems. In other words, it is the technology of designing and improving the human-machine-environment system according to human characteristics. Combining this technology with artificial intelligence technology can enable machines to interact with people more intelligently. Due to the differences in individual physiological signals, the differences in individual physiological signals are not considered in the prior art, resulting in low accuracy of prediction in the prior art. This application adopts individual difference removal technology to normalize the features, thereby achieving the removal of individual differences in physiological signals and improving the accuracy of signal representation status.

[0062] The physiological signals involved in the embodiments of the present application include but are not limited to: electroencephalogram (EEG), heart rate variability (HRV), electrodermal activity (EDA), respiratory signal (RESP), photoplethysmogram (PPG), skin temperature (SKT), blood oxygen saturation (SPO2), electrocardiogram (ECG), electromyography (EMG) and nine-axis posture data of the human body.

[0063] Among them, EEG is electroencephalogram, which is a method of recording the brain's electrical activity by placing electrodes on the scalp. EEG signals usually contain brain waves of different frequencies, such as delta waves (1-4HZ), theta waves (4-8HZ), alpha waves (8-12HZ), beta waves (13-30HZ) and gamma waves (30-100HZ). Changes in EEG waveforms can reflect different states of the brain. For example, alpha waves are usually associated with a relaxed state, while beta waves are associated with an alert or anxious state.

[0064] HRV is heart rate variability, which refers to the variation in the time intervals between consecutive heartbeats. The degree of HRV can reflect the regulation of the heart by the autonomic nervous system. For example, high HRV is generally associated with better stress resilience and health status, while low HRV may be associated with stress, disease or poor health status.

[0065] EDA is the electrical skin activity, also known as skin conductivity, which measures changes in the electrical conductivity of the skin surface. It is a non-specific stress response that can reflect emotional arousal, concentration, or stress levels. For example, when an individual is stressed or excited, sweat gland activity increases, leading to an increase in skin conductance.

[0066] RESP is a breathing signal, a method of monitoring breathing patterns and rates, which can reflect an individual's physiological state, such as relaxation, tension, or sleep. For example, rapid shallow breathing may be associated with anxiety or panic, while slow and deep breathing may be associated with relaxation or meditation.

[0067] PPG is a photoplethysmogram, a technology that uses optical sensors to measure changes in blood volume, and is commonly used to monitor heart rate and blood flow. PPG sensors work by emitting light and detecting changes in the absorption of light by the blood, which are synchronized with the heart cycle. PPG is widely used in health and fitness trackers and can provide estimates of heart rate, heart rate variability, and blood oxygen saturation.

[0068] SKT is skin temperature, which refers to the temperature of the skin surface and can reflect the body's internal temperature regulation mechanism and external environmental conditions. Changes in skin temperature may be related to emotional state, physiological activity or health status. For example, stress or anxiety may cause the body to release heat, thereby increasing skin temperature.

[0069] SPO2 is blood oxygen saturation, a measure of how well oxygen is bound to hemoglobin in the blood. SPO2 is usually measured using a pulse oximeter, a device that sends light through the blood and measures the absorption of different wavelengths of light. Low levels of SPO2 may indicate inadequate oxygen delivery.

[0070] ECG is an electrocardiogram, a medical test that records the electrical activity of the heart. By placing electrodes on the surface of the body, ECG can detect and record the heart's electrical signals, which reflect the heart's rhythm, frequency, and conduction characteristics.

[0071] EMG is electromyography, a technique that measures the electrical signals produced by muscles when they are active. By placing electrodes on the surface of the skin or inserted into the muscles, EMG can assess the activity level and functional status of the muscles.

[0072] The nine-axis posture data of the human body is usually collected by a sensor suite including accelerometers, gyroscopes and magnetometers to monitor and analyze the movement and direction changes of the human body. The nine-axis sensor integrates the acceleration and angular velocity information of three spatial axes and the direction of the geomagnetic field, which can be used to determine the precise direction and movement trajectory of the device or human body.

[0073] In an embodiment of the present application, the physiological signal data of the user is first obtained, and then individual difference signals are removed from the physiological signal data. For the convenience of description, in the following embodiments, the acquired physiological signal data of the user is described as the first physiological signal data, and the physiological signal data after individual difference signals are removed is described as the second physiological signal data.

[0074] A human intelligence state prediction method provided in an embodiment of the present application can be used for driving tasks, and the driver's fatigue level can be identified in real time during driving by collecting the driver's EEG data in real time.

[0075] The following describes the technical solution of the user status prediction method of the application with several specific embodiments:

[0076] Figure 1 A flowchart of a method for predicting human intelligence status provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method of this embodiment is as follows:

[0077] S11: Acquire first physiological signal data.

[0078] Optionally, the first physiological signal data includes multimodal physiological signal data of the user, wherein the multimodal physiological signal data includes at least one of the following: EEG, HRV, EDA, RESP, PPG, SKT, SPO2, ECG, EMG and nine-axis posture data of the human body.

[0079] There are many ways to obtain the first physiological signal data of the user. For example, the EEG brain wave signal of the driver is collected during a driving task. The drivers are required to perform the same repetitive task under different fatigue states. The EEG brain wave signals of multiple drivers under different fatigue states are collected using a dry electrode portable electroencephalograph with a sampling rate of 256 Hz.

[0080] S12: performing individual difference signal removal on the first physiological signal data to obtain second physiological signal data.

[0081] Among them, individual difference signals refer to the fact that even in the same environment and state, the physiological signal characteristic values ​​exhibited by different users will be different due to differences in physiological and psychological characteristics. These differences may be due to various factors such as genetics, living habits, and health conditions. For example, the HRV baseline level of different users may be different, some users may have a higher HRV, while others may have a lower HRV. Similarly, the amplitude and frequency of EEG signals may also vary from individual to individual.

[0082] Optionally, after obtaining the first physiological signal data, it is first preprocessed by filtering, artifact removal, independent component analysis, etc.; then, feature extraction is performed on the preprocessed first physiological signal data to obtain user feature values ​​corresponding to the first physiological signal; finally, based on the global difference space matrix and the individual vector, individual difference signal removal processing is performed on the user feature values ​​to obtain the second physiological signal data.

[0083] Among them, for the preprocessing, for example, the collected driver's EEG brain wave signal is subjected to high-pass filtering, low-pass filtering and notch filtering, which can effectively eliminate the interference of the mains, motion artifacts and other interferences of the EEG signal; independent component analysis can be performed to identify and remove artifact components such as electrooculography and electromyography. For the feature extraction of the preprocessed first physiological signal data, the user characteristic value corresponding to the first physiological signal is obtained. For example, the extraction of the user characteristic value includes the frequency domain characteristic value, that is, the extraction of the average power of the sine waves of 5 different frequencies of δ, θ, α, β, and γ of the driver's EEG brain wave signal. The average power can reflect the energy of the signal within a specific frequency range. For the individual difference signal removal processing of the user characteristic value, the second physiological signal data is obtained. For example, in the driving task, the two drivers are required to be in an alert state. The characteristic values ​​of the EEG brain wave signals corresponding to each of them in this state are significantly different. The individual difference removal and adjustment of their characteristic values ​​can make the characteristic values ​​of the two closer, and the alert state of the two drivers can be more accurately evaluated.

[0084] S13: Predicting the state of the user based on the second physiological signal data.

[0085] Optionally, the second physiological signal data is input into a prediction model to predict the user's state to obtain a prediction result, and the prediction result includes at least one of the following: a cognitive load prediction result, an emotional state prediction result, a stress prediction result, and a fatigue prediction result.

[0086] For example, in a driving task, the driver's fatigue state is classified based on the EEG signal with individual differences removed, and the level of fatigue state is obtained, such as high, medium, and low. Different measures are taken for the driver according to the fatigue level.

[0087] In this embodiment, the user's first physiological signal data is obtained; the individual difference signal is removed from the first physiological signal data to obtain the second physiological signal data; and the user's state is predicted based on the second physiological signal data. That is, when predicting the user's state, the second physiological signal data with the individual difference signal removed is used to predict the user's state, which is conducive to reducing the influence of the individual difference signal on the prediction result and improving the accuracy of the prediction result.

[0088] Figure 2 A flowchart of another method for predicting human intelligence status provided in an embodiment of the present application is shown below. Figure 2 is Figure 1 Based on the embodiment shown, a possible implementation of S12 is further described as follows: Figure 2 As shown, the method of this embodiment is as follows:

[0089] S121: Preprocess the first physiological signal data to obtain preprocessed first physiological signal data, wherein the preprocessing includes at least one of the following: filtering, artifact removal, and independent component analysis.

[0090] Among them, the filtering methods include: high-pass filtering, low-pass filtering and notch filtering, etc. High-pass filtering can remove low-frequency noise and DC components, such as slow fluctuations caused by physiological processes such as breathing, heartbeat, and body temperature changes; low-pass filtering can remove high-frequency noise and smooth the signal to extract the outline of the signal, such as high-frequency noise caused by the subject's head or body movement; notch filtering can remove interference at specific frequencies, such as power frequency interference caused by power supply frequency. The artifact removal methods include: visual inspection, filter and independent component analysis, etc. For example, the high-frequency noise caused by the subject's head or body movement is a motion artifact, which is mainly removed by a low-pass filter. The independent component analysis is a statistical technique that can separate the signal sources of independent components in physiological signals, including various noise sources, such as electrooculographic artifacts caused by eye movements and electromyographic artifacts caused by muscle activity. Each component represents a statistically independent source in the signal. By identifying and removing noise components, the purity of the signal can be improved.

[0091] For example, in the driving task, for the acquired EEG signal, setting the high-pass filter to 0.5Hz, the low-pass filter to 70Hz, and the notch filter to 50Hz can effectively remove the electromyographic interference caused by the driver's breathing in the EEG signal, and can also remove the motion artifacts caused by the driver's head or body movement and the 50Hz power frequency interference; through independent component analysis, it is possible to remove artifacts such as electrooculography and electromyography in the EEG signal by observing the time domain waveform, spectral characteristics and spatial distribution of the independent components.

[0092] S122: Perform feature extraction on the preprocessed first physiological signal data to obtain user feature values ​​corresponding to the first physiological signal, wherein the user feature values ​​include: time domain feature values ​​and frequency domain feature values, the time domain feature values ​​include at least one of the following: maximum value, minimum value and standard deviation, and the frequency domain feature values ​​include: power spectrum density of different frequency bands.

[0093] The feature extraction adopts a sliding window method, and a sliding window is given, for example, 5s is used as a window, to extract user feature values ​​of the first physiological signal, including: extraction of time domain feature values ​​and time domain feature values, and features can be extracted according to different channels of the first physiological signal data.

[0094] The time domain characteristic values ​​include: the maximum value, minimum value, range, median, peak value, root mean square, standard deviation, variance, etc. of the signal waveform in the sliding window. Among them, the maximum value, minimum value, range, median and peak value can be observed in a given sliding window, indicating the dynamic range of the signal; the root mean square refers to the square root of the average of all instantaneous amplitude squares of the signal in a given sliding window, indicating the measure of signal power and used to describe the strength of the signal; the standard deviation refers to the square root of the average of the squares of the deviations of the signal amplitude value from its average value, indicating the dispersion of the signal amplitude value; the variance refers to the average of the squares of the deviations of the signal amplitude value from its average value, which is the square of the standard deviation, and describes the fluctuation size of the signal.

[0095] The frequency domain eigenvalues ​​include: average power and median power of the five different frequency bands of δ, θ, α, β, and γ. The average power refers to the average value of the power spectrum density, which reflects the energy of the signal within a specific frequency range; the median frequency refers to the frequency point where the power of all frequencies in the signal power spectrum is accumulated to half, which is a statistic used to describe the center position of the signal power spectrum.

[0096] S123: Based on the global difference space matrix and the individual vector, perform individual difference signal removal processing on the user characteristic value to obtain second physiological signal data.

[0097] First, features are extracted from training data, the global difference space matrix is ​​estimated, and individual vectors are obtained. Then, individual differences are removed from the input data to obtain normalized feature data, i.e., the second physiological signal data.

[0098] In this embodiment, after acquiring the first physiological signal data, it is first preprocessed by filtering, artifact removal, independent component analysis, etc.; then, feature extraction is performed on the preprocessed first physiological signal data to obtain user feature values ​​corresponding to the first physiological signal; finally, based on the global difference space matrix and the individual vector, individual difference signal removal processing is performed on the user feature values ​​to obtain the second physiological signal data, thereby realizing the removal of individual differences in physiological signals. In this process, multiple different types of physiological signals are supported, thereby improving the generalization ability of the model.

[0099] Figure 3 A flowchart of another method for predicting human intelligence status provided in an embodiment of the present application is shown below. Figure 3 is Figure 2Based on the embodiment shown, a possible implementation method of obtaining the global difference space matrix and the individual vector by training before S123 is further described, as follows: Figure 3 As shown, the method of this embodiment is as follows:

[0100] S31: Obtain a training sample data set, perform feature extraction on each training sample data, and obtain sample features.

[0101] A training sample data set of the first physiological signal is obtained, and feature extraction is performed on all training data. For example, assuming that X channels are extracted from each data, and each channel has Y feature values, XY sample features are extracted from each data.

[0102] S32: construct a global difference space matrix and initialize the parameters of the global difference space matrix.

[0103] After obtaining the sample features, for each channel of the signal data, the correlation with all other channels is calculated to form a covariance matrix. For example, if there are X channels, the global difference space matrix can be represented as an X×X matrix, where the element C ij is the correlation coefficient between the i-th channel and the j-th channel; then initialize the parameters of the global difference space matrix.

[0104] S33: Update the parameters of the global difference space matrix based on the sample features and the iterative algorithm.

[0105] The parameters of the global difference space matrix are updated based on the sample features and iterative algorithm to obtain the Baum-Welch statistic of each sample user, and then the expectation-maximization algorithm (EM) steps are performed, including: E ​​step, i.e., the expectation step: calculating the posterior distribution of the hidden variable, M step, i.e., the maximization step: maximum likelihood value estimation, obtaining the updated global difference space matrix, and finally looping the EM step, and finally the final global difference space matrix training is completed. The hidden variable is the global difference factor.

[0106] S34: Based on the global difference space matrix, obtain the individual vector.

[0107] Given a sample user s and his first physiological signal feature vector sequence Y1, Y2, ..., Y N, assuming that the Gaussian Mixture Model-Universal Background Model (GMM-UBM) has C Gaussian classifications, each Gaussian component c, after obtaining the global difference space matrix, for a given sample user s and its h-th first physiological signal data feature, the GMM mean supervector obtained by UBM is expressed as:

[0108] M s,h =m u +Tω s,h (1)

[0109] Among them, M s,h Represents the GMM mean supervector associated with a given sample user s and its h-th first physiological signal, m u represents the mean supervector independent of the given sample user s and its h-th first physiological signal data feature, that is, the mean supervector of the UBM. Assuming that the UBM contains C Gaussian mixture components g, then m u It is the combination of all mixed component mean vectors, T is the global difference space matrix, s,h represents the hth first physiological signal data feature of sample user s, ω s,h is a hidden variable and also a global difference factor, representing the weight of the sample user s and its hth first physiological signal data feature in the global difference space. According to formula (1), ω s,h Calculate ω by taking into account the weighted average of all possible values s,h The posterior mean of is the individual vector of the user.

[0110] In this embodiment, by acquiring a training sample data set, extracting features from each training sample data, obtaining sample features, constructing a global difference space matrix, and initializing the parameters of the global difference space matrix, the parameters of the global difference space matrix are updated based on the sample features and an iterative algorithm, and the individual vector is obtained based on the global difference space matrix. Thus, the global difference space matrix and the individual vector are obtained.

[0111] Figure 4 A flowchart of another method for predicting human intelligence status provided in an embodiment of the present application is shown below. Figure 4 is Figure 3 Based on the embodiment shown, a possible implementation of S33 is further described as follows: Figure 4 As shown, the method of this embodiment is as follows:

[0112] S331: Based on the sample characteristics, obtain the Baum-Welch statistic of each sample user.

[0113] The Baum-Welch statistic of the sample user reflects the probability distribution characteristics of the first physiological signal feature of the sample user.

[0114] Given a sample user s and his first physiological signal feature vector sequence Y1, Y2, ..., Y N , assuming that GMM-UBM has C Gaussian classifications, each Gaussian component c, the Baum-Welch statistic of sample user s is obtained as follows:

[0115] N c (s)=∑ t γ t (c)(2)

[0116] Fc c (s)=∑ t γ t (c)Y t (3)

[0117]

[0118] Among them, ∑ t represents the sum of all features of sample user s, that is, the total number of frames, t represents one of the total number of frames, and γ t (c) represents the posterior probability of the cth Gaussian component at observation time t, that is, the probability that the tth observation data comes from the cth sub-model under the current model parameters, which is called the responsiveness of sub-model c to observation time t, N c (s) is the zero-order statistic calculated from the first physiological signal data feature of sample user s, representing the posterior probability γ on all time frames t t The sum of (c) is the responsiveness of the Gaussian mixture model (GMM) to the characteristics of each sample user; Y t represents the feature vector at time t, F c (s) is the first-order statistic obtained by further analyzing the first physiological signal feature of the sample user s, indicating that features are extracted for each frame. It is the average value of the feature vector weighted by the probability that it belongs to the Gaussian component c, reflecting the direction of the mean; Yes t The conjugate transpose of , diag(·) function represents taking the diagonal matrix, S c (s) represents the characteristic vector Y for all t t and its conjugate transpose The outer product of is weighted summed, and then the diagonal elements are taken to obtain the second-order statistics, which reflects the degree of dispersion of the features.

[0119] γ t The calculation formula of (c) is as follows:

[0120]

[0121] Among them, ωc represents the mixed weight of Gaussian component c, C is the total number of Gaussian classification categories, and p c (Y t ) is the probability density function of the multivariate Gaussian distribution, ωj represents the weight of category j, and p j (Y t ) means that under the condition of category j, the observed Y t The probability density function of .

[0122] In the Baum-Welch statistic, we can define and are the first-order and second-order central statistics:

[0123]

[0124] Among them, mc represents the mean vector corresponding to the Gaussian component c, Y t represents the feature vector at time t, γ t (c) represents the posterior probability of the cth Gaussian component at the observation time t, and the diag(·) function represents a diagonal matrix.

[0125] In order to reduce the influence of noise and estimate the central moment of the Gaussian component more accurately, the statistics of the first physiological signal feature subtract the mean vector of each Gaussian component, and the first-order and second-order central statistics are expressed by Baum-Welch statistics:

[0126]

[0127]

[0128] S332: Based on the Baum-Welch statistic of each sample user, calculate the posterior distribution of the latent variable, where the posterior distribution includes: mean and expectation.

[0129] Among them, calculating the posterior distribution of the latent variable is the expected E step in the EM algorithm. First, for each sample user s's h-th first physiological signal data, a matrix l is introduced as:

[0130] l T (s)=I+T*Σ- 1 N(s)T(10)

[0131] Among them, l T (s) represents the covariance matrix of the first physiological signal data of sample user s, N(s) represents the zero-order statistics of the first physiological signal data of sample user s, I represents the identity matrix, T represents the global difference space matrix, ∑-1 represents the inverse of the covariance matrix ∑.

[0132] Then, given the feature vector of the first physiological signal data of the sample user s and the parameter set (V, ∑), the posterior distribution of the latent variable is The covariance matrix is ​​l- 1 The Gaussian distribution of (s) is expressed in expected form as:

[0133]

[0134] Among them, E[ω s,h ] and E[ω s,h ω s,h ] represents the expected value of the posterior distribution of the latent variable, represents the expectation of the complex conjugate of the latent variable, ω s,h represents the hidden variable, i.e., the global difference factor, l -1 (s) is the matrix l T The inverse matrix of (s), Represents the first-order central statistic of the h-th first physiological signal data of user s.

[0135] S333: Based on the posterior distribution of the latent variables, use maximum likelihood estimation to update the parameters of the global difference space matrix.

[0136] Among them, the maximum likelihood estimation is the maximization M step in the EM step, and the following statistics are calculated through the training set:

[0137]

[0138] Among them, φ c represents the evaluation function, and formula 13 is used to find the parameter N that maximizes the evaluation function. c.h (s), Ω represents the estimated value of the global difference space matrix, N c.h (s) represents the parameter that maximizes the evaluation function, represents the first-order central statistic of the h-th physiological signal data of user s, represents the complex conjugate of the latent variable, represents the expectation of the complex conjugate of the latent variable, ∑ s represents the sum of sample users s, ∑ h represents the sum of the h-th first physiological signal feature data of each sample user s.

[0139] Similarly, for each Gaussian component c=1...C and each dimension of the feature parameter f=1...P, let i=(c-1)P+f, then the update formula of the global difference space matrix is ​​as follows:

[0140] T iφ c =Ω i ,(i=1,...,CP) (15)

[0141] Among them, T i φ c represents the updated global difference space matrix, Ω i Represents the updated estimate of the global difference space matrix.

[0142] S334: Determine whether the global difference space matrix converges. If not, return to execute S332. If converged, execute S335.

[0143] If the global difference space matrix does not converge, return to execute the Baum-Welch statistic based on each sample user to calculate the posterior distribution of the latent variable, the posterior distribution includes: mean and expectation, until the global difference space matrix converges, then the global difference space matrix T is trained.

[0144] S335: The global difference space converges and a global difference space matrix is ​​obtained.

[0145] In this embodiment, the Baum-Welch statistic of each sample user is obtained based on the sample features, and the posterior distribution of the latent variable is calculated based on the Baum-Welch statistic of each sample user, and the posterior distribution includes: mean and expectation. According to the posterior distribution of the latent variable, the parameters of the global difference space matrix are updated using the maximum likelihood estimation to determine whether the global difference space matrix converges. If not, return to execute the Baum-Welch statistic based on each sample user to calculate the posterior distribution of the latent variable, and the posterior distribution includes: mean and expectation until the global difference space matrix converges. Thus, the parameters of the global difference space matrix are continuously updated to achieve convergence of the global difference space matrix.

[0146] Figure 5 A flowchart of a method for predicting human intelligence status provided in an embodiment of the present application is shown in FIG. Figure 5 is Figure 2 Based on the embodiment shown, a possible implementation of S123 is further described as follows: Figure 4 As shown, the method of this embodiment is as follows:

[0147] S1231: Based on the difference space matrix and the individual vector of the user, obtain the individual difference contribution value of the user.

[0148] Optionally, the product of the difference space matrix and the individual vector of the user is the individual difference contribution value of the user.

[0149] S1232: Obtain second physiological signal data based on the user characteristic value and the individual difference contribution value.

[0150] Optionally, the difference between the user characteristic value and the individual difference contribution value is the second physiological signal data.

[0151] In this embodiment, the individual difference contribution value of the user is obtained based on the difference space matrix and the individual vector of the user, and the second physiological signal data is obtained based on the user characteristic value and the individual difference contribution value. Thus, the individual difference signal is removed by the user characteristic value to obtain the second physiological signal data.

[0152] Figure 6 A schematic diagram of the structure of a human intelligence state prediction device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device of this embodiment includes an acquisition module 601, a processing module 602 and a prediction module 603, wherein the acquisition module 601 is used to acquire first physiological signal data, and the first physiological signal data includes multimodal physiological signal data of the user;

[0153] A processing module 602 is used to remove individual difference signals from the first physiological signal data to obtain second physiological signal data;

[0154] The prediction module 603 is used to predict the state of the user based on the second physiological signal data.

[0155] Optionally, the processing module 602 is specifically used to preprocess the multimodal physiological data included in the first physiological signal data to obtain preprocessed first physiological signal data, wherein the preprocessing includes at least one of the following: filtering, artifact removal and independent component analysis; the multimodal physiological signal data includes at least one of the following: electroencephalogram (EEG), heart rate variability (HRV), skin electrodermal activity (EDA), respiratory signal (RESP), photoplethysmogram (PPG), skin temperature (SKT), blood oxygen saturation (SPO2), electrocardiogram (ECG), electromyogram (EMG) and human nine-axis posture data; feature extraction is performed on the preprocessed first physiological signal data to obtain user characteristic values ​​corresponding to the first physiological signal, wherein the user characteristic values ​​include: time domain characteristic values ​​and frequency domain characteristic values, the time domain characteristic values ​​include at least one of the following: maximum value, minimum value and standard deviation, and the frequency domain characteristic values ​​include: power spectral density of different frequency bands; based on the global difference space matrix and the individual vector, the user characteristic values ​​are subjected to individual difference signal removal processing to obtain second physiological signal data.

[0156] Optionally, the prediction module 603 is specifically used to input the second physiological signal data into a prediction model to predict the user's state and obtain a prediction result, which includes at least one of the following: a cognitive load prediction result, an emotional state prediction result, a stress prediction result, and a fatigue prediction result.

[0157] Optionally, the processing module 602 is specifically used to obtain the individual difference contribution value of the user based on the difference space matrix and the individual vector of the user; and obtain the second physiological signal data based on the user characteristic value and the individual difference contribution value.

[0158] Optionally, the processing module 602 is further used to obtain the global difference space matrix and the individual vector through training;

[0159] The processing module 602 is specifically used to obtain a set of training sample data, perform feature extraction on each training sample data, and obtain sample features; construct a global difference space matrix and initialize the parameters of the global difference space matrix; update the parameters of the global difference space matrix based on the sample features and an iterative algorithm; and obtain the individual vector based on the global difference space matrix.

[0160] Optionally, the processing module 602 is specifically used to obtain the Baum-Welch statistic of each sample user based on the sample characteristics; calculate the posterior distribution of the latent variable based on the Baum-Welch statistic of each sample user, and the posterior distribution includes: mean and expectation; update the parameters of the global difference space matrix using maximum likelihood estimation according to the posterior distribution of the latent variable; determine whether the global difference space matrix converges, and if not, return to execute the Baum-Welch statistic based on each sample user to calculate the posterior distribution of the latent variable, and the posterior distribution includes: mean and expectation, until the global difference space matrix converges.

[0161] The device of this embodiment can be used to execute the technical solutions of the above-mentioned method embodiments accordingly. Its implementation principles and technical effects are similar and will not be repeated here.

[0162] Fig. 7A A structural diagram of an edge computing device provided in an embodiment of the present application, the edge computing device of this embodiment includes a processor 701 and a memory 702, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of any of the above-mentioned human factors intelligent state prediction method embodiments are implemented.

[0163] Optional, Figure 7B A schematic diagram of the structure of another edge computing device provided in an embodiment of the present application, in Figure 7B is Fig. 7A On the basis of the embodiment shown, further, it also includes:

[0164] The physiological signal data acquisition unit 703 is used to acquire first physiological signal data.

[0165] Among them, the physiological signal data acquisition unit is directly responsible for the component of collecting physiological signals, which is used to collect data from the user's physiological activities and convert them into digital signals through an analog-to-digital converter for processing by the processor.

[0166] Optional, Figure 7C A schematic diagram of the structure of another edge computing device provided in an embodiment of the present application, Figure 7C is Fig. 7A On the basis of the embodiment shown, further, it also includes:

[0167] The physiological signal data acquisition interface unit 704 is used to connect to the physiological signal acquisition unit to acquire the first physiological signal data.

[0168] The physiological signal data acquisition interface unit is responsible for receiving and forwarding the first physiological signal data acquired by the physiological signal data acquisition unit.

[0169] Optional, Fig.7D A schematic diagram of the structure of another edge computing device provided in an embodiment of the present application, Fig.7D is Figure 7B On the basis of the embodiment shown, further, it also includes:

[0170] The physiological signal data acquisition interface unit 704 is used to connect to the physiological signal acquisition unit to acquire the first physiological signal data.

[0171] The physiological signal data acquisition interface unit is responsible for receiving and forwarding the first physiological signal data acquired by the physiological signal data acquisition unit.

[0172] An embodiment of the present application also provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of any of the above-mentioned human intelligence state prediction method embodiments are implemented.

[0173] An embodiment of the present application also provides a computer program product, which, when executed by a processor, implements the steps of any of the above-mentioned human intelligence state prediction method embodiments.

[0174] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0175] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general hardware platform, and of course, can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, disk, CD, etc.), including several instructions to enable a terminal or a network-side device to execute the methods described in each embodiment of the present application.

[0176] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of the present application and the scope of protection of the claims, and these implementation methods are all within the protection of the present application.

Claims

1. A method for predicting human intelligence status, characterized in that: include: Acquire first physiological signal data, where the first physiological signal data includes multimodal physiological signal data of a user; performing individual difference signal removal on the first physiological signal data to obtain second physiological signal data; Based on the second physiological signal data, the state of the user is predicted.

2. The method according to claim 1, characterized in that The step of removing individual difference signals from the first physiological signal data to obtain second physiological signal data includes: Preprocessing the multimodal physiological data included in the first physiological signal data to obtain preprocessed first physiological signal data, wherein the preprocessing includes at least one of the following: filtering, artifact removal and independent component analysis; the multimodal physiological signal data includes at least one of the following: electroencephalogram (EEG), heart rate variability (HRV), skin electrode activity (EDA), respiratory signal (RESP), photoplethysmogram (PPG), skin temperature (SKT), blood oxygen saturation (SPO2), electrocardiogram (ECG), electromyogram (EMG) and human body nine-axis posture data; Performing feature extraction on the preprocessed first physiological signal data to obtain a user feature value corresponding to the first physiological signal, wherein the user feature value includes: a time domain feature value and a frequency domain feature value, the time domain feature value includes at least one of the following: a maximum value, a minimum value, and a standard deviation, and the frequency domain feature value includes: a power spectrum density of different frequency bands; Based on the global difference space matrix and the individual vector, individual difference signal removal processing is performed on the user characteristic value to obtain second physiological signal data.

3. The method according to claim 2, characterized in that The predicting the state of the user based on the second physiological signal data includes: The second physiological signal data is input into a prediction model to predict the state of the user to obtain a prediction result, which includes at least one of the following: a cognitive load prediction result, an emotional state prediction result, a stress prediction result, and a fatigue prediction result.

4. The method according to claim 2, characterized in that: The step of performing individual difference signal removal processing on the user characteristic value based on the global difference space matrix and the individual vector to obtain the second physiological signal data includes: Based on the difference space matrix and the individual vector of the user, obtaining the individual difference contribution value of the user; Based on the user characteristic value and the individual difference contribution value, second physiological signal data is obtained.

5. The method according to claim 4, characterized in that Before performing individual difference signal removal processing on the user characteristic value based on the global difference space matrix and the individual vector to obtain the second physiological signal data, the method further includes: Acquire the global difference space matrix and the individual vector through training; The step of obtaining the global difference space matrix and the individual vector through training includes: Obtain a training sample data set, perform feature extraction on each training sample data, and obtain sample features; Construct a global difference space matrix and initialize the parameters of the global difference space matrix; Updating the parameters of the global difference space matrix based on the sample features and an iterative algorithm; Based on the global difference space matrix, the individual vectors are obtained.

6. The method according to claim 5, characterized in that The updating of the parameters of the global difference space matrix based on the sample features and the iterative algorithm comprises: Based on the sample characteristics, obtain the Baum-Welch statistic of each sample user; Based on the Baum-Welch statistic of each sample user, the posterior distribution of the latent variable is calculated, and the posterior distribution includes: mean and expectation; According to the posterior distribution of the latent variables, the parameters of the global difference space matrix are updated using maximum likelihood estimation; Determine whether the global difference space matrix converges, if not, return to execute the Baum-Welch statistic based on each sample user, calculate the posterior distribution of the latent variable, the posterior distribution includes: mean and expectation, until the global difference space matrix converges.

7. A human intelligence state prediction device, characterized in that: include: An acquisition module, configured to acquire first physiological signal data, wherein the first physiological signal data includes multimodal physiological signal data of a user; A processing module, used for removing individual difference signals from the first physiological signal data to obtain second physiological signal data; A prediction module is used to predict the state of the user based on the second physiological signal data.

8. An edge computing device, characterized in that: It includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the human intelligence state prediction method as described in any one of claims 1 to 6 are implemented.

9. The edge computing device according to claim 8, characterized in that: Also includes: A physiological signal data acquisition unit, used for acquiring first physiological signal data; and / or, The physiological signal data acquisition interface unit is used to connect to the physiological signal acquisition unit to acquire the first physiological signal data.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the human intelligence state prediction method as described in any one of claims 1 to 6 are implemented.

Images