Personnel state identification method and device based on multi-modal data, equipment and medium
Through multimodal data processing, physiological data features are extracted using GAN, GCN and CNN networks, and transfer learning is performed through domain adaptation networks, which solves the problem of large differences in single-modal data dependence and individuals in the existing technology, and achieves a more accurate and stable state recognition effect.
Patent Information
- Application Number
- CN202411833909.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-12
- Publication Date
- 2025-04-08
AI Technical Summary
The existing state recognition methods rely on single mode data, resulting in weak generalization capabilities of the model, large individual differences, and easy to be affected by noise, making it difficult to achieve accurate state recognition.
The state recognition method based on multimodal data is adopted, sample physiological data is enhanced through the GAN network, multimodal graph features are extracted, single-modal spatial features are extracted, and transfer learning is performed in combination with the domain adaptation network, objective physiological data features independent of individuals are extracted, and finally input the state classifier for state recognition.
Through multimodal data processing, data richness and feature accuracy can be enhanced, individual differences can be overcome, and the accuracy and stability of state recognition can be improved.
Smart Images
Figure CN119783023A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of intelligent individual status recognition, and in particular to a method, device, equipment and medium for personnel status recognition based on multimodal data. Background Art
[0002] State recognition using physiological and psychological signals has a wide range of applications in healthcare, traffic driving, military and national defense, user experience, etc., and individual differences are one of the important challenges facing this field. Individual differences lead to a decrease in the prediction accuracy of the model for new individuals, limiting the practical application and development of physiological and psychological signal state recognition. In order to reduce the impact of individual differences on state recognition, researchers have proposed various methods, such as data calibration, feature extraction, transfer learning, adversarial learning, etc., so that the model can better adapt to the changes in physiological and psychological characteristics of different individuals.
[0003] However, the current state recognition methods have the following problems: 1. Due to the limitations of application scenarios, most existing technologies can only collect a small amount of baselines and are difficult to expand. As a result, the model's generalization ability is weakened due to the lack of big data.
[0004] 2. The data source only has a single physiological signal, which usually can only reflect specific physiological activities and may vary greatly between individuals, especially in baseline states, emotional responses or physiological conditions, and is more susceptible to noise.
[0005] 3. Over-reliance on single-modal data, complex and unstable domain adversarial neural networks, and large domain differences will affect model performance.
[0006] These factors highlight the importance of using multimodal psychophysiological models to reduce individual differences in state recognition. Therefore, how to reduce individual differences in state recognition methods is a technical problem that needs to be solved urgently. Summary of the invention
[0007] In order to reduce the impact of individual differences on the state recognition process, the present application provides a method, device, equipment and medium for personnel state recognition based on multimodal data.
[0008] In a first aspect, the present application provides a method for identifying personnel status based on multimodal data, which adopts the following technical solution: Obtain multi-modal sample physiological data from multiple subjects in each state; Obtain multimodal physiological data of the target person to be analyzed; Applying a GAN network to enhance the sample physiological data to obtain amplified baseline data; normalizing the baseline data; Applying a GCN network to extract multimodal graph features of the baseline data and determine state features related to the state; The baseline data is used as a source domain, the physiological data to be analyzed is used as a target domain, a domain adaptation network is used to perform migration from the source domain to the target domain, and objective physiological data features independent of the individual in the physiological data to be analyzed are extracted based on the state features; The objective physiological data features are input into a preset state classifier to determine the state of the target person.
[0009] By adopting the above technical solution and applying the GAN network, the original sample physiological data can be enhanced to generate more diverse data, solve the problem of insufficient sample size or data imbalance that may be encountered in the process of physiological data collection, and use the GCN network to extract the multimodal graph features of the baseline data, which can capture the complex associations and interactions between physiological data of different modalities, and can more accurately reflect the user's physiological state. By taking the baseline data as the source domain and the physiological data to be analyzed as the target domain, and using the domain adaptation network to perform transfer learning from the source domain to the target domain, the individual differences in physiological data between different users can be overcome, and objective physiological data features independent of the individual can be extracted. Finally, the objective physiological data features are input into the preset state classifier to achieve accurate identification of the target person's state.
[0010] Furthermore, the state characteristics include multimodal graph characteristics and unimodal spatial characteristics. Before taking the baseline data as the source domain and the physiological data to be analyzed as the target domain, and using a domain adaptation network to perform migration from the source domain to the target domain, and extracting physiological data characteristics independent of the individual in the physiological data to be analyzed based on the state characteristics, it also includes: applying a CNN network to extract each unimodal spatial feature of the baseline data to determine the state characteristics related to the state.
[0011] By adopting the above technical solution, the CNN network, with its powerful feature extraction capability, can deeply capture the spatial features of each unimodal physiological data, and can further screen out features closely related to the state from the unimodal spatial features, namely, state features. These features can more accurately reflect the user's physiological state. In the previous step, the multimodal graph features have been extracted by the GCN network, and now, the unimodal spatial features extracted by the CNN network can provide additional information for the multimodal feature fusion. These two features have different focuses and advantages in describing physiological data, and their combination can more comprehensively reflect the user's physiological state.
[0012] Furthermore, the step of obtaining multimodal sample physiological data of multiple subjects in each state includes: Acquiring first multimodal physiological data of a plurality of subjects in a first preset time period in each state; intercepting second multimodal physiological data of a second preset time period based on the first multimodal physiological data, wherein the second preset time period is shorter than the first preset time period; Dividing the second multimodal physiological data according to a preset length and a preset overlap rate to obtain a plurality of multimodal physiological data segments; The multimodal physiological data segments are subjected to noise reduction to obtain sample physiological data.
[0013] By adopting the above technical solution, the analysis units can be further refined by dividing the data into segments, so that the research can explore the local characteristics and change patterns of physiological data in more depth. The setting of preset length and overlap rate helps to balance the detail level and computational cost of the data, ensuring the comprehensiveness and accuracy of the analysis. Noise reduction is an important step in data preprocessing, which aims to eliminate or weaken the noise and interference factors in the data and improve the purity and quality of the data.
[0014] Furthermore, the application of the GAN network to enhance the sample physiological data to obtain amplified baseline data includes: Construct a generator corresponding to each single modality, wherein the generator includes a convolutional layer and an activation function in sequence; Construct a discriminator corresponding to each single modality, wherein the discriminator includes a fully connected layer and an elimination layer is introduced in the first layer; The generator performs feature mapping and transformation on random noise, and outputs new sample data similar to the sample physiological data of each single modality; The generator and the discriminator perform adversarial training according to the preset learning speed. After reaching the number of iterations, the updated new sample data corresponding to each single modality is obtained; The sample physiological data and new sample data corresponding to each single modality are used as the amplified multimodal baseline data.
[0015] By adopting the above technical solution, the generator can convert random noise into new sample data similar to real physiological data through feature mapping and transformation, thereby realizing data amplification. The discriminator contains a fully connected layer and introduces an elimination layer in the first layer, which helps to improve the discriminator's ability to distinguish between true and false data. After multiple iterative training, the generator can generate new sample data that is highly similar to real physiological data, and combine the sample physiological data corresponding to each single modality with the new sample data to form amplified multimodal baseline data, providing rich data resources for subsequent analysis and research.
[0016] Furthermore, the applying of the GCN network to extract multimodal graph features of the baseline data includes: Extracting features of the baseline data, and building a connectivity graph according to the features of the baseline data; The input layer of the GCN network obtains the node feature matrix X and the adjacency matrix A in the connected graph, the node feature matrix X is an N×M matrix, N is the total number of nodes, M is the feature dimension of each node, and the adjacency matrix A is an N×N matrix; The hidden layer of the GCN network updates the node feature matrix layer by layer through multiple graph convolutions to generate new node representations. The propagation formula of the hidden layer is: in, is the adjacency matrix of a connected graph with additional self-connections, I is the identity matrix; for The degree matrix of is the symmetric normalized Laplace matrix of A; H l is the feature matrix of the lth layer, that is, X of the input layer; σ is the activation function; W l is the weight matrix of the lth layer; The output layer of the GCN network is classified according to the latest node feature matrix to obtain the connectivity features of the baseline data.
[0017] By adopting the above technical solution, features are extracted from the baseline data, and a connectivity graph is constructed based on these features. The complex relationships and information in the baseline data can be intuitively represented in the form of a graph. The introduction of the node feature matrix X and the adjacency matrix A enables the GCN network to simultaneously consider the features of the nodes and the connection relationships between the nodes, thereby more accurately capturing the deep features of the baseline data. The hidden layer updates the node feature matrix layer by layer through multiple graph convolution operations to generate new node representations. This process can gradually extract the deep features in the baseline data and improve the expressiveness and discrimination of the features. The output layer performs classification according to the latest node feature matrix to obtain the connectivity features of the baseline data, which can convert the deep features extracted by the GCN network into specific classification results, providing useful information for subsequent analysis and research.
[0018] In a second aspect, the present application provides a personnel status recognition device based on multimodal data, which adopts the following technical solution: A sample physiological data acquisition module is used to acquire multimodal sample physiological data of multiple subjects in each state; A module for acquiring physiological data to be analyzed, used to acquire multimodal physiological data to be analyzed of a target person; An enhancement module, used for enhancing the sample physiological data using a GAN network to obtain amplified baseline data; A normalization module, used for normalizing the baseline data; A state feature determination module, used to extract multimodal graph features of the baseline data using a GCN network, and determine state features related to the state; An objective physiological data feature extraction module, used to use the baseline data as a source domain and the physiological data to be analyzed as a target domain, use a domain adaptation network to perform migration from the source domain to the target domain, and extract objective physiological data features independent of the individual in the physiological data to be analyzed based on the state features; The state recognition module is used to input the objective physiological data characteristics into a preset state classifier to determine the state of the target person.
[0019] By adopting the above technical solution and applying the GAN network, the original sample physiological data can be enhanced to generate more diverse data, solve the problem of insufficient sample size or data imbalance that may be encountered in the process of physiological data collection, and use the GCN network to extract the multimodal graph features of the baseline data, which can capture the complex associations and interactions between physiological data of different modalities, and can more accurately reflect the user's physiological state. By taking the baseline data as the source domain and the physiological data to be analyzed as the target domain, and using the domain adaptation network to perform transfer learning from the source domain to the target domain, the individual differences in physiological data between different users can be overcome, and objective physiological data features independent of the individual can be extracted. Finally, the objective physiological data features are input into the preset state classifier to achieve accurate identification of the target person's state.
[0020] In a third aspect, the present application provides an electronic device, which adopts the following technical solution: An electronic device, comprising: at least one processor; at least one memory; At least one computer program, wherein the at least one computer program is stored in the memory and configured to be executed by the at least one processor, the at least one computer program being configured to: execute the method as described in any one of the first aspects.
[0021] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium stores a computer program that can be loaded by a processor and executes the method as described in any one of the first aspects.
[0022] In summary, the present application includes at least one of the following beneficial technical effects: 1. By applying the GAN network, the original sample physiological data can be enhanced to generate more diverse data, solving the problem of insufficient sample size or data imbalance that may be encountered during the physiological data collection process; 2. Using the GCN network to extract multimodal graph features of baseline data can capture the complex associations and interactions between physiological data of different modalities, and can more accurately reflect the user's physiological state; 3. The unimodal spatial features extracted by the CNN network can provide additional information for the fusion of multimodal features. These two features have different focuses and advantages in describing physiological data. Their combination can more comprehensively reflect the user's physiological state. 4. By taking the baseline data as the source domain and the physiological data to be analyzed as the target domain, and using the domain adaptation network to perform transfer learning from the source domain to the target domain, we can overcome the individual differences in physiological data between different users and extract objective physiological data features independent of the individual. Finally, the objective physiological data features are input into the preset state classifier to achieve accurate identification of the target person's state. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 It is a flow chart of a method for identifying personnel status based on multimodal data in an embodiment of the present application.
[0024] Figure 2 It is a schematic diagram of the structure of the GCN network in the embodiment of the present application.
[0025] Figure 3 It is a structural block diagram of a personnel status recognition device based on multimodal data in an embodiment of the present application.
[0026] Figure 4 It is a structural block diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION
[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0028] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.
[0029] The present application embodiment discloses a method for identifying a person's status based on multimodal data. Figure 1, executed by an electronic device, which can be a server or a terminal device, wherein the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smart phone, a tablet computer, a desktop computer, etc., but is not limited thereto. It includes (steps S101 to S108): Step S101: Acquire multi-modal sample physiological data of multiple subjects in each state.
[0030] Step S102: Acquire multimodal physiological data to be analyzed of the target person.
[0031] Specifically, the electronic device first trains a model for identifying the user's state so that the model can extract physiological data of the target person independent of the individual. Therefore, the electronic device obtains sample physiological data before obtaining the multimodal physiological data to be analyzed of the user.
[0032] The sample physiological data comes from the data of multiple subjects in each state, where the state type can be set according to the application of state recognition. For example, if state recognition is applied to driving fatigue monitoring, the data of the subjects in fatigue driving and non-fatigue driving states are obtained respectively; if state recognition is applied to user emotion recognition, the data of the subjects in each emotion are obtained respectively; if state recognition is applied to operator distraction recognition, the data of the subjects in the state of concentrating on work and distracted work are obtained respectively.
[0033] In order to improve the accuracy of identifying user status through physiological data, this application adopts multimodal sample physiological data, and the electronic device communicates with the physiological signal acquisition device to obtain the physiological data of the subject or user. The electronic device supports the common acquisition of physiological signals of the subject or user, and the information collected by each product corresponds to a single-modal sample data. The electronic device and the information acquisition device are time-synchronized to facilitate the simultaneous acquisition of single-modal data sent by each information acquisition device in different states. The electronic device further synchronizes multiple single-modal data to obtain multimodal data. And the modality of the target person's physiological data to be analyzed is the same as the modality type of the sample physiological data.
[0034] Physiological signals are data that can reflect the physiological and psychological state of a subject or user, such as: EEG (Electroencephalogram), fNIRS (functional near-infrared spectroscopy), PPG (Photo PlethysmoGraphy), EDA (Electrodermal activity), RESP (Respiration), SKT (skin temperature), SPO2 (oxygen saturation), ECG (electrocardiogram), EMG (electromyelogram), and HRV (Heart Rate Variability), etc. In the embodiment of the present application, the subject's EEG, heart rate, and skin electricity data in each state can be obtained.
[0035] Further, in order to improve the credibility of the sample physiological data and to facilitate reducing individual differences in model training, step S101 includes (steps S1011 to S1014) when obtaining multimodal sample physiological data of multiple subjects in each state; Step S1011: obtaining first multimodal physiological data of a plurality of subjects in a first preset time period in each state.
[0036] Specifically, the electronic device first collects the first multimodal physiological data within a first preset time period, and then performs an enhancement operation based on the first multimodal physiological data to obtain a large amount of multimodal physiological data, enriching the sample data while reducing the difficulty of collecting the sample data. As the above example, the electronic device of the present application can obtain 10 minutes of EEG, heart rate and skin conduction data of multiple subjects in a negative state and a positive state respectively.
[0037] Step S1012: extracting second multimodal physiological data of a second preset time period based on the first multimodal physiological data, where the second preset time period is shorter than the first preset time period.
[0038] Furthermore, in the first multimodal physiological data, data fluctuations may exist due to environmental influences. For example, in the first segment of the data, the subject's mind is not fully focused on watching the video, and in the last segment of the data, the subject's attention is diverted, which may reduce the accuracy of the data. Therefore, the electronic device obtains the second multimodal physiological data of the second preset time period. If the first preset time period is 10 minutes, the second preset time period can be 5 minutes.
[0039] Step S1013: dividing the second multimodal physiological data according to a preset length and a preset overlap rate to obtain a plurality of multimodal physiological data segments.
[0040] For example, the electronic device divides the second multimodal physiological data into segments with a length of 10 seconds and an overlap rate of 50%.
[0041] Step S1014: denoising the multimodal physiological data segments to obtain sample physiological data.
[0042] Specifically, the electronic device performs targeted noise reduction processing on each single-modal physiological data segment, such as performing notch filtering on EEG data and smoothing and noise reduction on heart rate and skin conductance data, thereby obtaining sample physiological data.
[0043] Step S103: applying the GAN network to enhance the sample physiological data to obtain the amplified baseline data, including (steps S1031 to S1035): Step S1031: construct a generator corresponding to each single modality, and the generator includes a convolutional layer and an activation function in sequence.
[0044] Step S1032: construct a discriminator corresponding to each single modality, the discriminator includes a fully connected layer, and a random elimination layer is introduced in the first layer.
[0045] Step S1033: The generator performs feature mapping and transformation on the random noise, and outputs new sample data that is similar to the sample physiological data of each single modality.
[0046] Step S1034: The generator and the discriminator perform adversarial training according to a preset learning speed. After reaching the number of iterations, new sample data corresponding to each single modality is obtained after the update.
[0047] Step S1035: taking the sample physiological data corresponding to each single modality and the new sample data as the amplified multimodal baseline data.
[0048] Specifically, the network structure of GAN (Generative Adversarial Networks) is mainly composed of two key components, namely the generator and the discriminator. The electronic device establishes a corresponding GAN network for each single modality to amplify the physiological data of each modality.
[0049] The generator is responsible for generating new data samples similar to real data. Multiple convolutional layers can be set, and the number of convolution kernels in each convolutional layer increases step by step. The convolutional layer performs feature mapping and transformation on random noise, and then introduces nonlinear activation functions, so that the generator obtains rich new sample data. Among them, if there are four convolutional layers, the number of convolution kernels in each convolutional layer can be 64, 128, 256 and 512 respectively. Common activation functions include ReLU, Leaky ReLU, Tanh, etc.
[0050] The discriminator is responsible for distinguishing whether the input data is real data or fake data generated by the generator. The number of fully connected layers is consistent with the number of convolutional layers, and elimination layers, such as Dropout layers, are randomly introduced in the first layer to prevent overfitting of the model.
[0051] The generator and the discriminator are trained adversarially. During the training process, the discriminator receives the real sample physiological data and the new sample data and tries to distinguish them. During the training phase of the generator, the parameters of the discriminator are fixed and unchanged, and only the parameters of the generator are optimized to make the new sample data generated by the generator closer to the real sample physiological data. Similarly, during the training phase of the discriminator, the parameters of the generator are fixed and unchanged, and only the parameters of the discriminator are optimized to make the discriminator correctly distinguish the real sample physiological data from the new sample data.
[0052] The training of the generator and the discriminator is performed alternately. In each iteration, the generator is trained first and then the discriminator. This process stops when the number of iterations is reached or the samples generated by the generator are realistic enough that the discriminator cannot distinguish them. At this time, the electronic device obtains updated new sample data. The updated new sample data is already very similar to the sample physiological data. The electronic device collectively refers to the multimodal sample physiological data and the new sample data as baseline data, which can represent the standard physiological data of the user in each state. Through the above-mentioned enhancement operations, the amount of sample physiological data is expanded. The preset learning speed and number of iterations may be set according to actual needs. For example, the preset learning speed may be 0.001, and the number of iterations may be 500.
[0053] Step S104: normalize the baseline data.
[0054] Specifically, normalizing the baseline data after GAN network enhancement can significantly improve the stability of training and the quality of new sample data. Commonly used normalization methods include BatchNorm, LayerNorm, InstanceNorm, WeightNorm, Spectral Normalization, Gradient Normalization, and Min-Max Normalization.
[0055] Step S105: Apply the GCN network to extract multimodal graph features of the baseline data and determine state features related to the state.
[0056] Specifically, the baseline data is the multimodal physiological data of the subjects. In order to facilitate the analysis of the relationship and combination of physiological data between modalities, common features independent of individuals are extracted, and a connectivity graph of the baseline data is established, so that the GCN network performs feature analysis based on the connectivity graph.
[0057] The GCN network (Graph Convolutional Network) utilizes the adjacency matrix and node feature matrix in the graph structure and adopts spectral graph theory to aggregate node information to achieve efficient processing of graph data.
[0058] Step S105 includes (step S1051 to step S1054): Step S1051: extract the features of the baseline data, and build a connectivity graph based on the features of the baseline data.
[0059] Specifically, a connected graph consists of two basic elements, namely nodes and edges, which can be represented by G=(V, E), where G represents a connected graph, V represents a set of nodes, and E represents a set of edges connecting nodes.
[0060] Step S1052: The input layer of the GCN network obtains the node feature matrix X and the adjacency matrix A in the connected graph. The node feature matrix X is an N×M matrix, where N is the number of nodes and M is the feature dimension of each node. The adjacency matrix A is an N×N matrix.
[0061] Specifically, taking node features as an example, for the i-th node, it has feature X i , the total node features can be expressed as matrix X N×M Represented by , where N represents the total number of nodes and M represents the number of features of each node (the dimension of the feature vector).
[0062] The adjacency matrix of a connected graph describes the connection relationship between each node, usually expressed as A N×N Indicates, for example, A ij=1 means that the i-th node and the j-th node are connected. If A ij =0, it means that the i-th node and the j-th node are not connected.
[0063] Therefore, assuming that the connectivity graph has 18 nodes, a time-space domain connectivity graph with n layers of time domain features is extracted for 18 channel signals and represented by embedding feature node features, then its size is 18×n.
[0064] Step S1053: The hidden layer of the GCN network updates the node feature matrix layer by layer through multiple graph convolutions to generate new node representations. The propagation formula of the hidden layer is: in, is the adjacency matrix of a connected graph with additional self-connections, I is the identity matrix; for The degree matrix of is the symmetric normalized Laplace matrix of A; H l is the feature matrix of the lth layer, that is, X of the input layer; σ is the activation function; W l is the weight matrix of the lth layer.
[0065] Step S1054: The output layer of the GCN network is classified according to the latest node feature matrix to obtain the connectivity features of the baseline data.
[0066] Specifically, the output layer of the GCN network includes a connected BN layer, a Dropout layer, a Flatten layer, and an activation function.
[0067] Among them, the BN layer (Batch Normalization Layer) accelerates the convergence speed of the GCN network, reduces the model's dependence on the initial parameters, and improves the robustness of the model.
[0068] The activation function determines how neurons perform nonlinear transformations on input signals. Commonly used activation functions include the sigmoid function, Tanh function, ReLU (Rectified Linear Unit) function, ELU (Exponential Linear Unit) function, Softmax function, etc.
[0069] The Dropout layer is a regularization technique that aims to reduce overfitting and improve the generalization ability of the GCN network.
[0070] The Flatten layer is used to convert multi-dimensional input data (such as the output of the convolutional layer) into a one-dimensional vector for subsequent fully connected layer processing.
[0071] In summary, each layer structure can be selected and used according to actual needs. Figure 2 In the embodiment of the present application, the activation function of the network can be set to the ReLU function and the Softmax function, and the calculation formula for the forward propagation of the GCN network as a whole is: Among them, Z is the newly obtained feature representation, that is, the connectivity feature of the baseline data, X is the input node feature matrix, and A is the input adjacency matrix.
[0072] Step S106: Apply the CNN network to extract each single-modal spatial feature of the baseline data and determine the state features related to the state.
[0073] Specifically, the electronic device extracts the graph features of the baseline data through the GCN network to obtain the connectivity between the multimodal data of the baseline data, and each unimodal physiological data also has corresponding time series features. In order to analyze the characteristics of each unimodal data, the present application also uses a CNN network (Convolutional Neural Networks) to extract the features of the baseline data.
[0074] The baseline data input to the CNN network has been preprocessed, and then the CNN network model is constructed, where the size of the input layer is the same as the dimension of the preprocessed baseline data, and then multiple convolutional layers are used to extract local features in the baseline data, and activation functions are used to increase nonlinearity. The pooling layer is applied to reduce the dimension of the feature map, and then the fully connected layer is applied to integrate the features and classify them so that the output layer outputs the features of the baseline data as state features related to the state.
[0075] Step S107: using the baseline data as the source domain and the physiological data to be analyzed as the target domain, using the domain adaptation network to migrate from the source domain to the target domain, and extracting objective physiological data features independent of the individual in the physiological data to be analyzed based on the state features.
[0076] Specifically, the state features of the electronic device obtained through the GCN network are placed in the adaptation layer of the graph space, and the state features obtained through the CNN network are placed in the adaptation layer of the Euclidean space. Then the source domain and the target domain are determined, the migration from the source domain to the target domain is completed, the feature distribution distance between the source domain and the target domain is shortened, and the objective physiological data features independent of the individual are extracted from the physiological data to be analyzed.
[0077] Among them, objective physiological data characteristics are universal physiological signal characteristics that are independent of the individual user to be analyzed, reducing the impact of personal factors on the accuracy of state analysis.
[0078] Step S108: Input the objective physiological data features into a preset state classifier to determine the state of the target person.
[0079] Specifically, the state classifier is set according to the type of state to be analyzed. For example, if state recognition is applied to user emotion recognition, the preset state classifier is used to distinguish whether physiological data features independent of an individual belong to positive emotions or negative emotions.
[0080] Common classifiers include logistic regression, support vector machine, decision tree, random forest, K nearest neighbor, linear discriminant analysis, etc. The most appropriate classifier is selected according to the application scenario of state analysis.
[0081] It should be noted that step S105 can be executed before step S106, or after step S106, or simultaneously with step S106. This is currently only one possible execution method and is not intended to be a limitation of the embodiments of the present application.
[0082] In order to better implement the above method, the present application embodiment also provides a personnel status recognition device based on multimodal data, referring to Figure 3 , the personnel status recognition device 200 based on multimodal data includes: A sample physiological data acquisition module 201 is used to acquire multimodal sample physiological data of multiple subjects in each state; The physiological data acquisition module 202 is used to acquire the multi-modal physiological data of the target person to be analyzed; An enhancement module 203 is used to enhance the sample physiological data using a GAN network to obtain amplified baseline data; A normalization module 204, used to normalize the baseline data; A state feature determination module 205 is used to extract multimodal graph features of baseline data using a GCN network to determine state features related to the state; The objective physiological data feature extraction module 206 is used to use the baseline data as the source domain and the physiological data to be analyzed as the target domain, use the domain adaptation network to migrate from the source domain to the target domain, and extract objective physiological data features independent of the individual in the physiological data to be analyzed based on the state features; The state recognition module 207 is used to input objective physiological data features into a preset state classifier to determine the state of the target person.
[0083] Furthermore, the state feature determination module 205 is also used to: apply the CNN network to extract each single-modal spatial feature of the baseline data, and determine the state feature related to the state.
[0084] Furthermore, the sample physiological data acquisition module 201 is specifically used for: Acquiring first multimodal physiological data of a plurality of subjects in a first preset time period in each state; intercepting second multimodal physiological data of a second preset time period based on the first multimodal physiological data, where the second preset time period is shorter than the first preset time period; The second multimodal physiological data is divided according to a preset length and a preset overlap rate to obtain a plurality of multimodal physiological data segments; and noise reduction is performed on the multimodal physiological data segments to obtain sample physiological data.
[0085] Furthermore, the enhancement module 203 is specifically used for: Construct a generator corresponding to each single modality, which includes a convolutional layer and an activation function in sequence; Construct a discriminator corresponding to each single modality. The discriminator includes a fully connected layer and an elimination layer is introduced in the first layer. The generator performs feature mapping and transformation on random noise, and outputs new sample data similar to the sample physiological data of each single modality; The generator and the discriminator perform adversarial training according to the preset learning speed. After reaching the number of iterations, the updated new sample data corresponding to each single modality is obtained; The sample physiological data and new sample data corresponding to each single modality are used as the expanded multimodal baseline data.
[0086] Furthermore, when the state feature determination module 205 applies the GCN network to extract the multimodal graph features of the baseline data, it is specifically used for: Extract the features of the baseline data and build a connectivity graph based on the features of the baseline data; The input layer of the GCN network obtains the node feature matrix X and adjacency matrix A in the connected graph. The node feature matrix X is an N×M matrix, where N is the total number of nodes and M is the feature dimension of each node. The adjacency matrix A is an N×N matrix. The hidden layer of the GCN network updates the node feature matrix layer by layer through multiple graph convolutions to generate new node representations. The propagation formula of the hidden layer is: in, is the adjacency matrix of a connected graph with additional self-connections, I is the identity matrix; for The degree matrix of is the symmetric normalized Laplace matrix of A; H l is the feature matrix of the lth layer, that is, X of the input layer; σ is the activation function; W l is the weight matrix of the lth layer; The output layer of the GCN network is classified according to the latest node feature matrix to obtain the connectivity features of the baseline data.
[0087] The various variations and specific examples of the methods in the aforementioned embodiments are also applicable to the personnel status recognition device based on multimodal data in the present embodiment. Through the aforementioned detailed description of the personnel status recognition method based on multimodal data, those skilled in the art can clearly understand the implementation method of the personnel status recognition device based on multimodal data in the present embodiment. Therefore, for the sake of brevity of the specification, it will not be described in detail here.
[0088] In order to better implement the above method, the present application embodiment provides an electronic device, referring to Figure 4 , the electronic device 300 includes: a processor 301, a memory 303 and a display screen 305. Among them, the memory 303 and the display screen 305 are connected to the processor 301, such as through a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in actual applications, the transceiver 304 is not limited to one, and the structure of the electronic device 300 does not constitute a limitation on the embodiments of the present application.
[0089] Processor 301 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0090] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 302 may be divided into an address bus, a data bus, a control bus, etc.
[0091] The memory 303 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.
[0092] The memory 303 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the contents shown in the above method embodiment.
[0093] Figure 4 The electronic device 300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0094] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, the personnel status recognition method based on multimodal data provided in the above embodiment is implemented. By applying the GAN network, the original sample physiological data can be enhanced to generate more diversified data, thereby solving the problem of insufficient sample size or data imbalance that may be encountered in the process of physiological data collection. The multimodal graph features of the baseline data are extracted by the GCN network, which can capture the complex associations and interactions between physiological data of different modalities, and can more accurately reflect the user's physiological state. By taking the baseline data as the source domain and the physiological data to be analyzed as the target domain, and using the domain adaptation network to perform transfer learning from the source domain to the target domain, the individual differences in physiological data between different users can be overcome, and objective physiological data features independent of the individual can be extracted. Finally, the objective physiological data features are input into a preset state classifier to achieve accurate recognition of the target person's state.
[0095] In this embodiment, the computer-readable storage medium may be a tangible device that holds and stores instructions used by the instruction execution device. The computer-readable storage medium may be, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof. Specifically, the computer-readable storage medium may be a portable computer disk, a hard disk, a USB flash drive, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, an optical disk, a magnetic disk, a mechanical encoding device, and any combination thereof.
[0096] The computer program in this embodiment includes program codes for executing all the aforementioned methods, and the program codes may include instructions corresponding to the steps of the methods provided in the above embodiments. The computer program can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The computer program can be executed entirely on a user's computer or as an independent software package.
[0097] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.
[0098] In addition, it is to be understood that relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprises," "comprising," or any other variation thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or elements that are inherent to such process, method, article, or device.
Claims
1. A method for identifying personnel status based on multimodal data, characterized in that: include: Obtain multi-modal sample physiological data from multiple subjects in each state; Obtain multimodal physiological data of the target person to be analyzed; Applying a GAN network to enhance the sample physiological data to obtain amplified baseline data; normalizing the baseline data; Applying a GCN network to extract multimodal graph features of the baseline data and determine state features related to the state; The baseline data is used as a source domain, the physiological data to be analyzed is used as a target domain, a domain adaptation network is used to perform migration from the source domain to the target domain, and objective physiological data features independent of the individual in the physiological data to be analyzed are extracted based on the state features; The objective physiological data features are input into a preset state classifier to determine the state of the target person.
2. The method according to claim 1, characterized in that The state features include multimodal graph features and unimodal spatial features. Before taking the baseline data as a source domain and the physiological data to be analyzed as a target domain, using a domain adaptation network to perform migration from the source domain to the target domain, and extracting physiological data features independent of an individual in the physiological data to be analyzed based on the state features, the method further includes: A CNN network is applied to extract each single-modal spatial feature of the baseline data and determine the state features related to the state.
3. The method according to claim 1, characterized in that The obtaining of multi-modal sample physiological data of multiple subjects in each state includes: Acquiring first multimodal physiological data of a plurality of subjects in a first preset time period in each state; intercepting second multimodal physiological data of a second preset time period based on the first multimodal physiological data, wherein the second preset time period is shorter than the first preset time period; Dividing the second multimodal physiological data according to a preset length and a preset overlap rate to obtain a plurality of multimodal physiological data segments; The multimodal physiological data segments are subjected to noise reduction to obtain sample physiological data.
4. The method according to claim 1, characterized in that: The applying of the GAN network to enhance the sample physiological data to obtain the amplified baseline data includes: Construct a generator corresponding to each single modality, wherein the generator includes a convolutional layer and an activation function in sequence; Construct a discriminator corresponding to each single modality, wherein the discriminator includes a fully connected layer and an elimination layer is introduced in the first layer; The generator performs feature mapping and transformation on random noise, and outputs new sample data similar to the sample physiological data of each single modality; The generator and the discriminator perform adversarial training according to the preset learning speed. After reaching the number of iterations, the updated new sample data corresponding to each single modality is obtained; The sample physiological data and new sample data corresponding to each single modality are used as the amplified multimodal baseline data.
5. The method according to claim 1, characterized in that The applying of the GCN network to extract the multimodal graph features of the baseline data includes: Extracting features of the baseline data, and building a connectivity graph according to the features of the baseline data; The input layer of the GCN network obtains the node feature matrix X and the adjacency matrix A in the connected graph, the node feature matrix X is an N×M matrix, N is the total number of nodes, M is the feature dimension of each node, and the adjacency matrix A is an N×N matrix; The hidden layer of the GCN network updates the node feature matrix layer by layer through multiple graph convolutions to generate new node representations. The propagation formula of the hidden layer is: in, is the adjacency matrix of a connected graph with additional self-connections, I is the identity matrix; for The degree matrix of is the symmetric normalized Laplace matrix of A; H l is the feature matrix of the lth layer, that is, X of the input layer; σ is the activation function; W l is the weight matrix of the lth layer; The output layer of the GCN network is classified according to the latest node feature matrix to obtain the connectivity features of the baseline data.
6. The method according to claim 5, characterized in that The output layer of the GCN network includes a connected BN layer, a Dropout layer, a Flatten layer, and an activation function.
7. A personnel status recognition device based on multimodal data, characterized in that: include: A sample physiological data acquisition module is used to acquire multimodal sample physiological data of multiple subjects in each state; A module for acquiring physiological data to be analyzed, used to acquire multimodal physiological data to be analyzed of a target person; An enhancement module, used for enhancing the sample physiological data using a GAN network to obtain amplified baseline data; A normalization module, used for normalizing the baseline data; A state feature determination module, used to extract multimodal graph features of the baseline data using a GCN network, and determine state features related to the state; A physiological data feature extraction module independent of an individual, used to use the baseline data as a source domain, the physiological data to be analyzed as a target domain, use a domain adaptation network to perform migration from the source domain to the target domain, and extract objective physiological data features independent of an individual from the physiological data to be analyzed based on the state features; The state recognition module is used to input the objective physiological data characteristics into a preset state classifier to determine the state of the target person.
8. An electronic device, characterized in that: include: at least one processor; at least one memory; At least one computer program, wherein the at least one computer program is stored in the memory and configured to be executed by the at least one processor, the at least one computer program being configured to: execute the method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 6.