Multi-mode human factor intelligent personnel state recognition method and device, edge computing equipment and medium

Abstract

The embodiment of the invention provides a multi-mode human factor intelligent personnel state recognition method and device, edge computing equipment and a medium, and the method comprises the steps: obtaining physiological signal data, which comprises multi-mode physiological signal data of personnel; performing feature extraction on the multi-mode physiological signal data to obtain target feature vectors corresponding to multiple modes respectively; the target feature vectors corresponding to the multiple modes are input into a multi-mode bottleneck transformation network for attention calculation, values of multiple fusion bottleneck tokens are obtained, and the multi-mode bottleneck transformation network comprises multiple fusion bottleneck token units; and performing personnel state identification based on the values of the fusion bottleneck tokens. Namely, the multi-modal physiological signals are adopted to obtain values of a plurality of fusion bottleneck tokens through attention calculation for personnel state recognition, the fusion bottleneck unit improves the fusion performance of different modal information, and the accuracy of the recognition result is improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence and human factors engineering technology, and specifically relates to a multimodal human factors intelligent personnel status recognition method, device, edge computing equipment and medium. Background Art

[0002] With the development of computer science and technology, how to align, fuse and classify physiological signal data of different modalities is of great help to the identification of various states of individuals. For example, the data source feature values ​​of the driver's multimodal physiological signals can be extracted, and the optimization model can be established to realize real-time recognition of various different states of the driver, such as cognitive load, emotional state, stress and fatigue. Summary of the invention

[0003] Embodiments of the present application provide a multimodal human factors intelligent personnel status recognition method, apparatus, edge computing device and medium.

[0004] In a first aspect, an embodiment of the present application provides a multimodal human factor intelligent personnel status recognition method, comprising:

[0005] Acquiring physiological signal data, wherein the physiological signal data includes multimodal physiological signal data of a person;

[0006] Performing feature extraction on the multi-modal physiological signal data to obtain target feature vectors corresponding to the multi-modalities respectively;

[0007] Inputting the target feature vectors corresponding to the multi-modalities into a multi-modal bottleneck transformation network for attention calculation to obtain values ​​of multiple fused bottleneck tokens, wherein the multi-modal bottleneck transformation network includes multiple fused bottleneck token units;

[0008] Personnel status recognition is performed based on the values ​​of the multiple fused bottleneck tokens.

[0009] Optionally, the step of inputting the target feature vectors corresponding to the multi-modalities into a multi-modal bottleneck transformation network for attention calculation to obtain values ​​of multiple fused bottleneck tokens includes:

[0010] For each fused bottleneck token unit, obtain the target attention score for each modality;

[0011] Based on the target attention scores of multiple modalities, the value of the fused bottleneck token is obtained.

[0012] Optionally, obtaining the attention score of each modality for each fusion bottleneck token unit includes:

[0013] For the fused bottleneck token unit, based on the initial value of the fused bottleneck token unit and a plurality of target sub-feature vectors of each modality, calculating an initial attention score of each target sub-feature vector;

[0014] Normalize the initial attention scores of multiple target sub-feature vectors to obtain the weight of each target sub-feature vector;

[0015] Based on the weight of each target sub-feature vector, the multiple target sub-feature vectors are weighted summed, and the obtained sum is used as the target attention score of the modality.

[0016] Optionally, obtaining the attention score of each modality for each fusion bottleneck token unit includes:

[0017] For the fused bottleneck token unit, based on the initial value of the fused bottleneck token unit and a plurality of target sub-feature vectors of each modality, calculating an initial attention score of each target sub-feature vector;

[0018] Normalize the initial attention scores of multiple target sub-feature vectors to obtain the weight of each target sub-feature vector;

[0019] Based on the weight of each target sub-feature vector, the multiple target sub-feature vectors are weighted and averaged, and the obtained average value is used as the target attention score of the modality.

[0020] Optionally, the identifying of personnel status based on the values ​​of the multiple fused bottleneck tokens includes:

[0021] The values ​​of the multiple fused bottleneck tokens are fused and input into the classification network to perform personnel status recognition to obtain a recognition result.

[0022] Optionally, the extracting features of the multi-modal physiological signal data to obtain target feature vectors corresponding to the multi-modalities respectively includes:

[0023] Inputting the multimodal physiological signal data of the person into the corresponding feature extraction network to extract features, and generating initial feature vectors corresponding to the multiple modalities;

[0024] Signal projection is performed on the initial feature vectors respectively corresponding to the multiple modes to obtain target feature vectors respectively corresponding to the multiple modes, and the target feature vectors respectively corresponding to the multiple modes have the same dimension.

[0025] Optionally, the acquiring of physiological signal data includes:

[0026] The multimodal physiological data included in the physiological signal data is preprocessed to obtain preprocessed physiological signal data, wherein the preprocessing includes at least one of the following: filtering, artifact removal and independent component analysis; the multimodal physiological signal data includes at least one of the following: electroencephalogram (EEG), heart rate variability (HRV), skin electrodermal activity (EDA), respiratory signal (RESP), photoplethysmogram (PPG), skin temperature (SKT), blood oxygen saturation (SPO2), electrocardiogram (ECG), electromyogram (EMG) and human body nine-axis posture data.

[0027] In a second aspect, an embodiment of the present application provides a multimodal human factor intelligent personnel status recognition device, comprising:

[0028] An acquisition module, used to acquire physiological signal data, wherein the physiological signal data includes multi-modal physiological signal data of a person;

[0029] A feature extraction module is used to extract features from the multi-modal physiological signal data to obtain target feature vectors corresponding to the multi-modal data;

[0030] A processing module, used for inputting the target feature vectors corresponding to the multi-modalities into a multi-modal bottleneck transformation network for attention calculation to obtain values ​​of multiple fused bottleneck tokens, wherein the multi-modal bottleneck transformation network includes multiple fused bottleneck token units;

[0031] An identification module is used to identify the status of a person based on the values ​​of the multiple fused bottleneck tokens.

[0032] Optionally, the processing module is specifically used to obtain a target attention score for each modality for each fused bottleneck token unit; and obtain a value of the fused bottleneck token based on the target attention scores of multiple modalities.

[0033] Optionally, the processing module is specifically used to calculate, for the fused bottleneck token unit, an initial attention score of each target sub-feature vector based on the initial value of the fused bottleneck token unit and multiple target sub-feature vectors of each modality; normalize the initial attention scores of the multiple target sub-feature vectors to obtain the weight of each target sub-feature vector; and perform weighted summation of the multiple target sub-feature vectors based on the weight of each target sub-feature vector, and use the obtained sum as the target attention score of the modality.

[0034] Optionally, the processing module is specifically used to calculate, for the fused bottleneck token unit, an initial attention score of each target sub-feature vector based on the initial value of the fused bottleneck token unit and multiple target sub-feature vectors of each modality; normalize the initial attention scores of the multiple target sub-feature vectors to obtain the weight of each target sub-feature vector; and based on the weight of each target sub-feature vector, weightedly average the multiple target sub-feature vectors, and use the obtained average value as the target attention score of the modality.

[0035] Optionally, the recognition module is specifically used to fuse the values ​​of the multiple fused bottleneck tokens and input them into a classification network to perform personnel status recognition to obtain a recognition result.

[0036] Optionally, the feature extraction module is specifically used to input the multimodal physiological signal data of the person into the corresponding feature extraction network for feature extraction, and obtain initial feature vectors corresponding to multiple modalities; perform signal projection on the initial feature vectors corresponding to the multiple modalities to obtain target feature vectors corresponding to the multiple modalities, and the dimensions of the target feature vectors corresponding to the multiple modalities are the same.

[0037] Optionally, the acquisition module is specifically used to preprocess the multimodal physiological data included in the physiological signal data to obtain preprocessed physiological signal data, wherein the preprocessing includes at least one of the following: filtering, artifact removal and independent component analysis; the multimodal physiological signal data includes at least one of the following: electroencephalogram (EEG), heart rate variability (HRV), skin electrodermal activity (EDA), respiratory signal (RESP), photoplethysmogram (PPG), skin temperature (SKT), blood oxygen saturation (SPO2), electrocardiogram (ECG), electromyogram (EMG) and nine-axis posture data of the human body.

[0038] In a third aspect, an embodiment of the present application provides an edge computing device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the multimodal human factors intelligent personnel status recognition method as described in any one of the first aspects are implemented.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the multimodal human factors intelligent personnel status recognition method as described in any one of the first aspects are implemented.

[0040] The multimodal human factors intelligent personnel status recognition method, device, edge computing device and medium provided in the embodiment of the present application obtain physiological signal data, the physiological signal data includes multimodal physiological signal data of personnel; feature extraction is performed on the multimodal physiological signal data to obtain target feature vectors corresponding to the multimodal respectively; the target feature vectors corresponding to the multimodal respectively are input into the multimodal bottleneck transformation network for attention calculation to obtain the values ​​of multiple fused bottleneck tokens, the multimodal bottleneck transformation network includes multiple fused bottleneck token units; personnel status recognition is performed based on the values ​​of the multiple fused bottleneck tokens. That is, the values ​​of multiple fused bottleneck tokens obtained by attention calculation using multimodal physiological signals are used to identify the status of personnel, and the fused bottleneck unit improves the fusion performance of different modal information, which is conducive to improving the accuracy of the recognition result. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of a multimodal human factors intelligent personnel status recognition method provided in an embodiment of the present application;

[0042] Figure 2 A flowchart of another multimodal human factors intelligent personnel status recognition method provided in an embodiment of the present application;

[0043] Figure 3 A flowchart of another multimodal human factors intelligent personnel status recognition method provided in an embodiment of the present application;

[0044] Figure 4 A flowchart of another multimodal human factors intelligent personnel status recognition method provided in an embodiment of the present application;

[0045] Figure 5 A flowchart of another multimodal human factors intelligent personnel status recognition method provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of the structure of a multi-modal human factors intelligent personnel status recognition device provided in an embodiment of the present application;

[0047] Figure 7 A schematic diagram of the structure of an edge computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0049] The terms "first", "second", etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of one type, and the number of objects is not limited, for example, the first object can be one or more. In addition, "or" in the present application represents at least one of the connected objects. For example, "A or B" covers three schemes, namely, Scheme 1: including A but not including B; Scheme 2: including B but not including A; Scheme 3: including both A and B. The character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0050] The term "indication" in this application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, operations to be performed, or request results in the sent indication; an indirect indication can be understood as the receiver determining the corresponding information based on the indication sent by the sender, or making a judgment and determining the operation to be performed or the request result based on the judgment result.

[0051] Human intelligence is based on human factors engineering technology combined with artificial intelligence (AI) technology. Human factors engineering technology is the application of human psychological and physiological principles to the engineering and design of products, processes and systems. In other words, it is the technology of designing and improving the human-machine-environment system according to human characteristics. Combining this technology with artificial intelligence technology can enable machines to interact with people more intelligently.

[0052] The multimodal physiological signals involved in the embodiments of the present application include but are not limited to at least one of the following or a combination: electroencephalogram (EEG), heart rate variability (HRV), electrodermal activity (EDA), respiratory signal (RESP), photoplethysmogram (PPG), skin temperature (SKT), blood oxygen saturation (SPO2), electrocardiogram (ECG), electromyography (EMG) and nine-axis posture data of the human body.

[0053] Among them, EEG is electroencephalogram, which is a method of recording the brain's electrical activity by placing electrodes on the scalp. EEG signals usually contain brain waves of different frequencies, such as delta waves (1-4HZ), theta waves (4-8HZ), alpha waves (8-12HZ), beta waves (13-30HZ) and gamma waves (30-100HZ). Changes in EEG waveforms can reflect different states of the brain. For example, alpha waves are usually associated with a relaxed state, while beta waves are associated with an alert or anxious state.

[0054] HRV is heart rate variability, which refers to the variation in the time intervals between consecutive heartbeats. The degree of HRV can reflect the regulation of the heart by the autonomic nervous system. For example, high HRV is generally associated with better stress resilience and health status, while low HRV may be associated with stress, disease or poor health status.

[0055] EDA is the electrical skin activity, also known as skin conductivity, which measures changes in the electrical conductivity of the skin surface. It is a non-specific stress response that can reflect emotional arousal, concentration, or stress levels. For example, when an individual is stressed or excited, sweat gland activity increases, leading to an increase in skin conductance.

[0056] RESP is a breathing signal, a method of monitoring breathing patterns and rates, which can reflect an individual's physiological state, such as relaxation, tension, or sleep. For example, rapid shallow breathing may be associated with anxiety or panic, while slow and deep breathing may be associated with relaxation or meditation.

[0057] PPG is a photoplethysmogram, a technology that uses optical sensors to measure changes in blood volume, and is commonly used to monitor heart rate and blood flow. PPG sensors work by emitting light and detecting changes in the absorption of light by the blood, which are synchronized with the heart cycle. PPG is widely used in health and fitness trackers and can provide estimates of heart rate, heart rate variability, and blood oxygen saturation.

[0058] SKT is skin temperature, which refers to the temperature of the skin surface and can reflect the body's internal temperature regulation mechanism and external environmental conditions. Changes in skin temperature may be related to emotional state, physiological activity or health status. For example, stress or anxiety may cause the body to release heat, thereby increasing skin temperature.

[0059] SPO2 is blood oxygen saturation, a measure of how well oxygen is bound to hemoglobin in the blood. SPO2 is usually measured using a pulse oximeter, a device that sends light through the blood and measures the absorption of different wavelengths of light. Low levels of SPO2 may indicate inadequate oxygen delivery.

[0060] ECG is an electrocardiogram, a medical test that records the electrical activity of the heart. By placing electrodes on the surface of the body, ECG can detect and record the heart's electrical signals, which reflect the heart's rhythm, frequency, and conduction characteristics.

[0061] EMG is electromyography, a technique that measures the electrical signals produced by muscles when they are active. By placing electrodes on the surface of the skin or inserted into the muscles, EMG can assess the activity level and functional status of the muscles.

[0062] The nine-axis posture data of the human body is usually collected by a sensor suite including accelerometers, gyroscopes and magnetometers to monitor and analyze the movement and direction changes of the human body. The nine-axis sensor integrates the acceleration and angular velocity information of three spatial axes and the direction of the geomagnetic field, which can be used to determine the precise direction and movement trajectory of the device or human body.

[0063] The human status involved in the embodiments of the present application includes but is not limited to the following: cognitive load, emotional state, fatigue state, etc.

[0064] Among them, cognitive load refers to the amount of information that the brain needs to process when completing a task. Excessive cognitive load may make it difficult for individuals to process information effectively, affecting learning efficiency and the quality of task completion.

[0065] Emotional state refers to the emotional response experienced by an individual in a specific situation. It can be short-lived, such as happiness, sadness, anger or fear, or it can be a more lasting emotional tendency, such as optimism or pessimism. Emotional state not only affects an individual's mental health, but also affects his or her behavior and social interaction.

[0066] Fatigue refers to a feeling of exhaustion after long-term mental or physical labor, which manifests as decreased energy, difficulty concentrating, and reduced work efficiency. Fatigue not only affects the work and quality of life of an individual, but may also increase the risk of accidents.

[0067] In an embodiment of the present application, multimodal physiological signal data of a person is first obtained, and then feature extraction is performed on the physiological signal data. The obtained feature vector is subjected to attention calculation to obtain the values ​​of multiple fusion bottleneck tokens. For the convenience of description, in the following embodiments of the present application, the corresponding feature vector obtained by feature extraction of the multimodal physiological signal data is described as an initial feature vector, the feature vector obtained by signal projection of the initial feature vector is described as a target feature vector, the sub-feature vector in the target feature vector is described as a target sub-feature vector, the attention score calculated for each target sub-feature vector is described as an initial attention score, and the attention score obtained by weighted summation or weighted averaging of multiple target sub-feature vectors is described as a target attention score.

[0068] A multimodal human factors intelligent personnel status recognition method provided in an embodiment of the present application can be used for driving tasks, for example, by real-time collection of the driver's EEG signal data and PPG signal data and multimodal alignment and fusion, the driver's cognitive load and other personnel status can be identified during the driving process.

[0069] The following describes the technical solution of the multi-modal human factor intelligent personnel status recognition method of the application with several specific embodiments:

[0070] Figure 1 A flowchart of a multimodal human factors intelligent personnel status recognition method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method of this embodiment is as follows:

[0071] S11: Acquire physiological signal data, where the physiological signal data includes multimodal physiological signal data of a person.

[0072] Optionally, the multimodal physiological data included in the physiological signal data is preprocessed to obtain preprocessed physiological signal data, wherein the preprocessing includes at least one of the following: filtering, artifact removal and independent component analysis. The multimodal physiological signal data includes at least one of the following: electroencephalogram (EEG), heart rate variability (HRV), skin electrode activity (EDA), respiratory signal (RESP), photoplethysmogram (PPG), skin temperature (SKT), blood oxygen saturation (SPO2), electrocardiogram (ECG), electromyogram (EMG) and human body nine-axis posture data.

[0073] There are many ways to obtain multimodal physiological signal data of personnel. For example, in a driving task, the driver is required to perform tasks under different load conditions, and the driver's EEG signal data and PPG signal data need to be collected. The EEG signal data is collected by placing electrodes on the scalp to record the electrical activity generated by the brain. These electrodes can detect tiny voltage changes, which reflect the activity of neuronal groups. Collecting EEG signal data of multiple drivers under different load conditions includes but is not limited to the following two possible collection methods:

[0074] One possible method for collecting EEG signal data is to use a multi-channel electrode cap with a sampling rate of 256 Hz. Specifically, the multi-channel electrode cap usually contains 19, 32 or more electrodes. The electrode cap is worn from back to front and ensured to be worn correctly. The central midline (CZ) point, that is, the midpoint of the line segment connecting the nasion and the external occipital protuberance is located in the center of the head and is symmetrical on the left and right. The electrodes are in contact with the driver's scalp through conductive gel or paste to improve signal quality.

[0075] Another possible way to collect EEG signal data is to use the ErgoLAB dry electrode portable EEG device with a sampling rate of 256 Hz. Specifically, the dry electrodes are usually integrated into a headband or hat to place the dry electrode sensors in symmetrical positions on the driver's prefrontal, frontal, parietal, temporal and occipital cortices. The dry electrodes do not require conductive gel and are in direct contact with the skin through the special design of the electrode surface. The position of the dry electrodes is appropriately adjusted according to the driver's head shape and hair condition to ensure good contact between the dry electrodes and the scalp.

[0076] The collection of PPG signal data is mainly achieved through optical sensors, which usually contain a light source and a photosensitive element. When the light emitted by the light source passes through the tissue or is reflected back to the photosensitive element, the change in blood flow caused by the heart beat causes the amount of light absorbed or scattered to change, thereby indirectly measuring physiological parameters such as blood pressure, heart rate and blood oxygen saturation. The collection of PPG signals of multiple drivers under different load conditions includes but is not limited to the following two possible collection methods:

[0077] A possible way to collect PPG signal data is to use a pulse oximeter with a sampling rate of 64Hz. Specifically, a clip is clamped on the driver's index finger or middle finger. The built-in light emitting diode (LED) emits red light or infrared light. The photosensitive element detects the change in light intensity passing through the finger and calculates the blood oxygen saturation based on the absorption characteristics of hemoglobin to light of different wavelengths.

[0078] Another possible way to collect PPG signal data is to use an ear clip heart rate monitor with a sampling rate of 64Hz. Specifically, a small photoelectric sensor is clamped on the earlobe in the form of an ear clip. The carotid artery blood flow system runs vertically along the ear canal area, and there is also a capillary system passing through the antitragus and earlobe. The light source and photosensor work closely together to detect changes in blood flow.

[0079] Further, the acquired multimodal physiological signal data is preprocessed to obtain preprocessed physiological signal data, wherein the filtering methods include: high-pass filtering, low-pass filtering and notch filtering, etc. High-pass filtering can remove low-frequency noise and DC components, such as slow fluctuations caused by physiological processes such as breathing, heartbeat, and body temperature changes; low-pass filtering can remove high-frequency noise and smooth the signal to extract the outline of the signal, such as high-frequency noise caused by the subject's head or body movement; notch filtering can remove interference of specific frequencies, such as power frequency interference caused by power supply frequency. The artifact removal methods include: visual inspection, filter and independent component analysis, etc. For example, the high-frequency noise caused by the subject's head or body movement is a motion artifact, which is mainly removed by a low-pass filter. The independent component analysis is a statistical technique that can separate the signal sources of independent components in physiological signals, including various noise sources, such as electrooculographic artifacts caused by eye movements and electromyographic artifacts caused by muscle activity. Each component represents a statistically independent source in the signal. By identifying and removing noise components, the purity of the signal can be improved.

[0080] Specifically, for example, in a driving task, for the acquired EEG signal data, the cutoff frequency of the high-pass filter is set to 0.5 Hz, the cutoff frequency of the low-pass filter is set to 70 Hz, and the center frequency of the notch filter is set to 50 Hz. This can effectively remove the electromyographic interference caused by the driver's breathing in the EEG signal, and can also remove motion artifacts caused by the driver's head or body movement and 50 Hz power frequency interference. Through independent component analysis, it is possible to remove artifacts such as electrooculography and electromyography in the EEG brain wave signal by observing the time domain waveform, spectral characteristics and spatial distribution of the independent components.

[0081] S12: Extracting features from the multi-modal physiological signal data to obtain target feature vectors corresponding to the multi-modal data.

[0082] The acquired physiological signal data of each modality is first processed in blocks before feature extraction. For example, for EEG signals, a fixed-length window is used to extract data of different frequency bands, including: θ wave (4-8Hz), α wave (8-12Hz), γ wave (25-45Hz), lowβ (12-16Hz) and highβ (16-25Hz). Five time-frequency graphs are generated for each frame, and then 3×3=9 patches are generated to input the feature extraction network. For PPG signals, a time-frequency graph is generated, and then 3×3=9 patches are generated to input the feature network. The physiological signal data of each modality after block processing are extracted through their corresponding feature extraction networks. Due to different network parameter settings, the initial feature vectors obtained are of different dimensions. The initial target feature vectors of each modality are then linearly mapped to obtain the target feature vectors corresponding to the multi-modalities of the same dimension.

[0083] S13: Inputting the target feature vectors corresponding to the multimodalities into the multimodal bottleneck transformation network for attention calculation to obtain values ​​of multiple fused bottleneck tokens, wherein the multimodal bottleneck transformation network includes multiple fused bottleneck token units.

[0084] The Multimodal Bottleneck Transformer (MBT) is based on the Transformer model architecture and introduces fusion bottlenecks to perform modal fusion at multiple levels. Its basic structure includes a feature extraction layer, an attention layer, and a multimodal fusion layer. The target feature vectors of multiple modal physiological signals extracted by the feature extraction layer are used as data sets to train the multimodal bottleneck transformer network model, and multiple fusion bottleneck token values ​​are obtained, and then the status of the personnel is identified by training the classification network.

[0085] Among them, the feature extraction layer is used to extract the target feature vectors of multiple modal physiological signals. The structure of the feature extraction layer is different for physiological signals of different modalities, such as Convolutional Neural Networks (CNN) of different structures or different complexities. The multimodal fusion layer is a multimodal attention bottleneck formed by multiple fusion bottleneck tokens (FSN Token) included in the MBT model. It improves the efficiency and performance of the model by limiting the cross-modal attention flow and forcing the collection and compression of modal information of multimodal physiological signal data.

[0086] Among them, the fusion bottleneck token unit integrates and aligns information between different modalities. FSN Tokens restricts direct interactions between different modalities, forcing the model to organize and condense the key information of each modality before transmitting information, thereby effectively capturing and fusing features from different modalities. By performing attention calculations on the target feature vectors corresponding to multiple modalities and multiple fusion bottleneck token units, the value of the fusion bottleneck token containing the fusion information of multiple modalities can be obtained. In other words, each FSN Token updates its representation based on its similarity with the target feature vectors of different modalities, and finally forms a fused feature representation for subsequent personnel status recognition.

[0087] S14: Perform personnel status identification based on the values ​​of the multiple fused bottleneck tokens.

[0088] Optionally, the values ​​of the multiple fused bottleneck tokens are fused and input into a classification network to perform personnel status recognition to obtain a recognition result.

[0089] The classification network is a linear classifier, which maps input data to different personnel status categories through a linear function. The linear function can be expressed as:

[0090] f(x)=wx+b

[0091] Among them, f(x) represents the score or probability that the result belongs to a certain category, x represents the value of multiple fused bottleneck tokens input, w represents the weight vector, and b represents the bias term. The principle of the linear classifier is to make classification decisions based on the linear combination of the input data vector and the weight vector. Specifically, the algorithm calculates the dot product of the value x of the input multiple fused bottleneck tokens and the weight vector w, and adds the bias term b. The result f(x) is used to determine which personnel status category the input data belongs to. For example, in the driving task, the driver's cognitive load is identified. Assuming that the driver's cognitive load is divided into three levels: low, medium and high, corresponding to different cognitive load assessment thresholds, the level of cognitive load is classified and identified according to the relationship between the value of f(x) output by the classification network and the size of each threshold.

[0092] In the embodiment of the present application, physiological signal data is obtained, and the physiological signal data includes multimodal physiological signal data of a person; feature extraction is performed on the multimodal physiological signal data to obtain target feature vectors corresponding to the multimodal respectively; the target feature vectors corresponding to the multimodal respectively are input into the multimodal bottleneck transformation network for attention calculation to obtain the values ​​of multiple fused bottleneck tokens, and the multimodal bottleneck transformation network includes multiple fused bottleneck token units; and the state of a person is identified based on the values ​​of the multiple fused bottleneck tokens. That is, the state of a person is identified by using the values ​​of multiple fused bottleneck tokens obtained by attention calculation using multimodal physiological signals, and the fused bottleneck unit improves the fusion performance of different modal information, which is conducive to improving the accuracy of the recognition result.

[0093] Figure 2 A flowchart of another multimodal human factors intelligent personnel status recognition method provided in an embodiment of the present application is provided. Figure 2 is Figure 1 Based on the embodiment shown, a possible implementation of S13 is further described as follows: Figure 2 As shown, the method of this embodiment is as follows:

[0094] S131: For each fused bottleneck token unit, obtain the target attention score of each modality.

[0095] Among them, attention calculation is performed on multiple target sub-feature vectors in the target feature vector corresponding to each fused bottleneck token unit and multiple modalities, so as to obtain the initial attention score of each target sub-feature vector, that is, the similarity between each fused bottleneck token unit and each target sub-feature vector; the weight of each target sub-feature vector can be obtained by normalizing the initial attention score of each fused bottleneck token unit; based on the weight of each target sub-feature vector, the weights of multiple target sub-feature vectors are weighted summed or weighted averaged to obtain the target attention score of each modality.

[0096] Specifically, assuming that there are two modalities of physiological signal data, namely EEG signal and PPG signal, the driver's cognitive load is identified in the driving task through the multimodal fusion of these two modalities of physiological signal data. Assume that there are two fusion bottleneck token units, denoted as fsn1 and fsn2, and that there are three target sub-feature vectors in the EEG signal target feature vector, denoted as eeg1, eeg2 and eeg3, and there are three target sub-feature vectors in the PPG signal target feature vector, denoted as ppg1, ppg2 and ppg3. Among them, for the EEG signal, the initial attention scores of fsn1 and fsn2 with eeg1, eeg2 and eeg3 are calculated respectively, and then the initial attention scores are normalized to obtain the weights of eeg1, eeg2 and eeg3 for fsn1. and the weight for fsn2 The weight and is a vector whose element square sum is 1. Finally, based on the weight of the target sub-feature vector of each EEG signal, eeg1, eeg2 and eeg3 are weighted summed to obtain the target attention score of the EEG signal. and For PPG signals, calculate fsn1 and fsn2 with ppg1 and ppg respectively. z The initial attention scores of ppg1, ppg2 and ppg3 are then normalized to obtain the weights of ppg1, ppg2 and ppg3. and Finally, based on the weight of the target sub-feature vector of each PPG signal, ppg1, ppg2 and ppg3 are weighted summed to obtain the target attention score of the PPG signal. and

[0097] S132: Based on the target attention scores of multiple modalities, obtain the value of the fusion bottleneck token.

[0098] Optionally, the target attention scores of the multiple modalities are fused to obtain the value of the fused bottleneck token.

[0099] Among them, based on the above example, for the fusion bottleneck token unit fsn1, the target attention scores of its corresponding EEG signal and PPG signal are and By fusing, that is, summing and averaging, we can get the updated value of the fused bottleneck token, which can be expressed as:

[0100]

[0101] in, represents the attention score of the EEG signal and PPG signal fused by fsn1, fsn′1 represents the value of the fusion bottleneck token updated by fsn1, represents the target attention score of the EEG signal for fsn1, represents the target attention score of the PPG signal for fsn1.

[0102] For the fusion bottleneck token unit fsn2, the target attention scores of its corresponding EEG signal and PPG signal are and By fusing, that is, summing and averaging, we can get the updated value of the fused bottleneck token, which can be expressed as:

[0103]

[0104] in, represents the attention score of the EEG signal and the PPG signal fused by fsn2, fsn′2 represents the value of the fusion bottleneck token updated by fsn2, represents the target attention score of the EEG signal for fsn2, represents the target attention score of the PPG signal for fsn2.

[0105] Specifically, assuming that the target attention scores of each modality have been calculated, for example, the target attention scores of the EEG signal are and The target attention scores of the PPG signals are and Use the fused target attention score to update the value of the fused bottleneck token, that is, update the value of the fused bottleneck token by the corresponding multimodal target attention score. The updated fsn1 is fsn′1=[0.35,0.35,0.3], and the updated fsn2 is fsn′2=[0.4,0.35,0.25].

[0106] In this embodiment, the target attention score of each modality is obtained for each fusion bottleneck token unit, and the value of the fusion bottleneck token is obtained based on the target attention scores of multiple modalities. Thus, the fusion of multiple modal physiological signals is achieved.

[0107] Figure 3 A flowchart of another multimodal human factors intelligent personnel status recognition method provided in an embodiment of the present application is provided. Figure 3 is Figure 2 Based on the embodiment shown, a possible implementation of S131 is further described as follows: Figure 3 As shown, the method of this embodiment is as follows:

[0108] S1311: For the fused bottleneck token unit, based on the initial value of the fused bottleneck token unit and multiple target sub-feature vectors of each modality, calculate the initial attention score of each target sub-feature vector.

[0109] Among them, based on the initial value of the fused bottleneck token unit and multiple target sub-feature vectors of each modality, the initial attention score of each target sub-feature vector is calculated, that is, the similarity between each fused bottleneck token unit and each target sub-feature vector is calculated.

[0110] Specifically, based on the above example, using dot product as the similarity metric, for example, for the fused bottleneck token unit fsn1, the initial attention score of the target sub-feature vector eeg1 of the EEG signal can be expressed as:

[0111]

[0112] in, It represents the initial attention score for the target sub-feature vector eeg1 of fsn1, that is, the similarity between fsn1 and eeg1, which is represented by the dot product of fsn1 and eeg1.

[0113] The initial attention score of the target sub-feature vector ppg1 of the PPG signal can be expressed as:

[0114]

[0115] in, It represents the initial attention score for the target sub-feature vector ppg1 of fsn1, that is, the similarity between fsn1 and ppg1, which is represented by the dot product of fsn1 and ppg1.

[0116] Similarly, for other fused bottleneck token units and target sub-feature vectors of physiological signals of each modality, dot product calculation is also used to obtain their respective initial attention scores.

[0117] S1312: Normalize the initial attention scores of multiple target sub-feature vectors to obtain the weight of each target sub-feature vector.

[0118] Among them, for each fusion bottleneck token unit, the initial attention scores of all target sub-feature vectors corresponding to each modality are normalized to obtain the weight of each target sub-feature vector.

[0119] Specifically, based on the aforementioned example, for example, for the fused bottleneck token unit fsn1, the weight of each target sub-feature vector of the EEG signal can be expressed as:

[0120]

[0121] in, represents the weight of each target sub-feature vector of the EEG signal for fsn1, and softmax() represents the normalization process. represents the initial attention score for the fsn1 target sub-feature vector eeg1, represents the initial attention score for the fsn1 target sub-feature vector eeg2, Represents the initial attention score for the fsn1 target sub-feature vector eeg3.

[0122] For the fused bottleneck token unit fsn1, the weight of each target sub-feature vector of the PPG signal can be expressed as:

[0123]

[0124] in, represents the weight of each target sub-feature vector of the EEG signal for fsn1, and softmax() represents the normalization process. represents the initial attention score for the fsn1 target sub-feature vector ppg1, represents the initial attention score for the fsn1 target sub-feature vector ppg2, Represents the initial attention score for the fsn1 target sub-feature vector ppg3.

[0125] Similarly, for other fused bottleneck token units, softmax normalization is also used to process the initial attention score of each target sub-feature vector of each modality to obtain the weight of each target sub-feature vector of each modality.

[0126] S1313: Based on the weight of each target sub-feature vector, perform weighted summation on the multiple target sub-feature vectors, and use the obtained sum as the target attention score of the modality.

[0127] Among them, for each fusion bottleneck token unit, based on the weight of each target sub-feature vector of each modality, all target sub-feature vectors corresponding to each modality are weighted summed to obtain the target attention score of each modality.

[0128] Specifically, based on the aforementioned example, for example, for the fused bottleneck token unit fsn1, the target attention score of the EEG signal can be expressed as:

[0129]

[0130] in, represents the target attention score of the EEG signal for fsn1, represents the weight of each target sub-feature vector of the EEG signal for fsn1, and eeg1, eeg2 and eeg3 represent the three target sub-feature vectors of the EEG signal.

[0131] For the fused bottleneck token unit fsn1, the target attention score of the PPG signal can be expressed as:

[0132]

[0133] in, represents the target attention score of the PPG signal for fsn1, represents the weight of each target sub-feature vector of the PPG signal for fsn1, and ppg1, ppg2 and ppg3 represent the three target sub-feature vectors of the PPG signal.

[0134] Similarly, for other fused bottleneck token units, the weights of each target sub-feature vector of each corresponding modality and each target sub-feature vector of each modality are used to calculate the weighted sum to obtain the target attention score of each modality.

[0135] In this embodiment, for the fused bottleneck token unit, based on the initial value of the fused bottleneck token unit and multiple target sub-feature vectors of each modality, the initial attention score of each target sub-feature vector is calculated; the initial attention scores of the multiple target sub-feature vectors are normalized to obtain the weight of each target sub-feature vector; based on the weight of each target sub-feature vector, the multiple target sub-feature vectors are weighted summed, and the obtained sum is used as the target attention score of the modality. Thus, the target attention score of each modality is obtained in a way, which is conducive to realizing the fusion of multi-modal physiological signals through attention calculation.

[0136] Figure 4 A flowchart of another multimodal human factors intelligent personnel status recognition method provided in an embodiment of the present application is provided. Figure 4 is Figure 2 Based on the embodiment shown, another possible implementation of S131 is further described as follows: Figure 4 As shown, the method of this embodiment is as follows:

[0137] S1314: For the fused bottleneck token unit, based on the initial value of the fused bottleneck token unit and multiple target sub-feature vectors of each modality, calculate an initial attention score of each target sub-feature vector.

[0138] For this step, please refer to the detailed description of S1311, which will not be repeated here.

[0139] S1315: Normalize the initial attention scores of multiple target sub-feature vectors to obtain the weight of each target sub-feature vector.

[0140] This step can be referred to the detailed description of S1312 and will not be repeated here.

[0141] S1316: Based on the weight of each target sub-feature vector, weighted average is performed on the multiple target sub-feature vectors, and the obtained average value is used as the target attention score of the modality.

[0142] Among them, for each fusion bottleneck token unit, based on the weight of each target sub-feature vector of each modality, all target sub-feature vectors corresponding to each modality are weighted and averaged to obtain the target attention score of each modality.

[0143] Specifically, based on the aforementioned example, for example, for the fused bottleneck token unit fsn1, the target attention score of the EEG signal can be expressed as:

[0144]

[0145] in, represents the target attention score of the EEG signal for fsn1, represents the weight of each target sub-feature vector of the EEG signal for fsn1, and eeg1, eeg2 and eeg3 represent the three target sub-feature vectors of the EEG signal.

[0146] For the fused bottleneck token unit fsn1, the target attention score of the PPG signal can be expressed as:

[0147]

[0148] in, represents the target attention score of the PPG signal for fsn1, represents the weight of each target sub-feature vector of the PPG signal for fsn1, and ppg1, ppg2 and ppg3 represent the three target sub-feature vectors of the PPG signal.

[0149] Similarly, for other fused bottleneck token units, the weights of each target sub-feature vector of each corresponding modality and each target sub-feature vector of each modality are used to calculate the weighted average to obtain the target attention score of each modality.

[0150] In this embodiment, the initial attention score of each target sub-feature vector is calculated based on the initial value of the fused bottleneck token unit and multiple target sub-feature vectors of each modality for the fused bottleneck token unit; the initial attention scores of multiple target sub-feature vectors are normalized to obtain the weight of each target sub-feature vector; based on the weight of each target sub-feature vector, the multiple target sub-feature vectors are weighted and averaged, and the obtained average value is used as the target attention score of the modality. Thus, the target attention score of each modality physiological signal is obtained in another way, which is conducive to realizing the fusion of multi-modal physiological signals through attention calculation.

[0151] Figure 5 A flowchart of another multimodal human factors intelligent personnel status recognition method provided in an embodiment of the present application is provided. Figure 5 is Figure 1 Based on the embodiment shown, a possible implementation of S12 is further described as follows: Figure 5 As shown, the method of this embodiment is as follows:

[0152] S121: Inputting the multimodal physiological signal data of the person into the corresponding feature extraction network to extract features, and obtaining initial feature vectors corresponding to the multiple modalities.

[0153] Among them, the physiological signal data of each modality is input into the corresponding feature extraction network for feature extraction. These networks are specially designed for each modality. Each feature extraction network outputs the initial feature vectors of the physiological signal data of each modality. The dimensions of these initial feature vectors may be different.

[0154] Specifically, for example, the EEG signal may be processed by a two-dimensional CNN network, while the PPG signal may be processed by another network. The dimension of the initial feature vector of the EEG signal output by the feature extraction network of the EEG signal is determined by the sampling rate and window length, the number of leads, and the size of the batch. It is generally a three-dimensional matrix [B, C, T], where B represents batch_size, C represents channel (number of leads), and T is the time length, that is, the window length multiplied by the sampling rate. The dimension of the initial feature vector of the EEG signal output by the feature extraction network of the PPG signal is determined by the sampling rate and window length, and the size of the batch. It is generally a two-dimensional matrix, [B, T], where B represents batch_size, and T is the time length, that is, the window length multiplied by the sampling rate.

[0155] S122: Perform signal projection on the initial feature vectors corresponding to the multiple modes respectively to obtain target feature vectors corresponding to the multiple modes respectively, and the target feature vectors corresponding to the multiple modes respectively have the same dimension.

[0156] In order to enable the feature vectors of different modalities to be compared and fused in the same framework, the initial feature vector needs to be projected to the target feature vector with the same dimension, that is, multimodal alignment is performed, which can be achieved through a variety of methods, including but not limited to: Principal Components Analysis (PCA) or linear mapping.

[0157] Among them, taking linear mapping as an example, assuming that the dimension of the initial feature vector is d1 and the target dimension is d2, the linear mapping can be expressed as:

[0158] F=g(f;E)=Ef+b1

[0159] Among them, F represents the target feature vector, g(;) represents the linear mapping, f represents the initial feature vector, E represents the linear mapping matrix, which is a d2×d1 weight matrix, and b1 represents a d2×1 bias vector.

[0160] Specifically, assume that the initial feature vector of the EEG signal data is represented by eeg 初始 =[1.2,3.4,2.1], the initial feature vector of PPG signal data is expressed as ppg 初始 =[0.5,1.2], through signal projection, the initial feature vectors with different dimensions are converted into target feature vectors with the same dimension, then the target feature vector of EEG signal data is expressed as eeg 目标 =[0.6,,0.8,0.5], the initial feature vector of PPG signal data is expressed as ppg 目标 =[0.4, 0.6, 0.3]. Therefore, the target feature vectors of EEG signals and PPG signals have the same dimension and can be used for subsequent multimodal fusion.

[0161] In this embodiment, the multimodal physiological signal data of the person are respectively input into the corresponding feature extraction network for feature extraction, and the initial feature vectors corresponding to the multiple modalities are respectively obtained; the initial feature vectors corresponding to the multiple modalities are respectively projected to obtain the target feature vectors corresponding to the multiple modalities, and the target feature vectors corresponding to the multiple modalities have the same dimension. Thus, the features of the physiological signals of multiple modalities with the same dimension are obtained, the alignment of the physiological signals of multiple modalities is realized, and the accuracy of the fusion of the multimodal physiological signals is improved.

[0162] In the above embodiment, the values ​​of multiple fused bottleneck tokens are fused and then input into the classification network for personnel status recognition. One simple fusion method is splicing, that is, splicing the values ​​of multiple fused bottleneck tokens into a longer feature vector.

[0163] Specifically, based on the previous example, assuming that fsn′1=[0.45, 0.25, 0.3], fsn′2=[0.3, 0.45, 0.25], the fused feature vector is expressed as features=[fsn′1, fsn′2]=[0.45, 0.25, 0.3, 0.3, 0.45, 0.25], and then the fused feature vector is input into a trained classification network, i.e., a linear classifier, and the classification network identifies the physiological state of the person based on the input feature vector.

[0164] Figure 6 A schematic diagram of the structure of a multi-modal human factors intelligent personnel status recognition device provided in an embodiment of the present application, such as Figure 6 As shown, the device of this embodiment includes an acquisition module 601, a feature extraction module 602, a processing module 603 and an identification module 604, wherein the acquisition module 601 is used to acquire physiological signal data, and the physiological signal data includes multi-modal physiological signal data of a person;

[0165] A feature extraction module 602 is used to extract features from the multi-modal physiological signal data to obtain target feature vectors corresponding to the multi-modal data;

[0166] A processing module 603 is used to input the target feature vectors corresponding to the multimodalities into a multimodal bottleneck transformation network for attention calculation to obtain values ​​of multiple fused bottleneck tokens, wherein the multimodal bottleneck transformation network includes multiple fused bottleneck token units;

[0167] The identification module 604 is used to identify the status of the personnel based on the values ​​of the multiple fused bottleneck tokens.

[0168] Optionally, the processing module 603 is specifically used to obtain a target attention score of each modality for each fused bottleneck token unit; and obtain a value of the fused bottleneck token based on the target attention scores of multiple modalities.

[0169] Optionally, the processing module 603 is specifically used to calculate, for the fused bottleneck token unit, an initial attention score of each target sub-feature vector based on the initial value of the fused bottleneck token unit and multiple target sub-feature vectors of each modality; normalize the initial attention scores of the multiple target sub-feature vectors to obtain the weight of each target sub-feature vector; and perform weighted summation of the multiple target sub-feature vectors based on the weight of each target sub-feature vector, and use the obtained sum as the target attention score of the modality.

[0170] Optionally, the processing module 603 is specifically used to calculate, for the fused bottleneck token unit, an initial attention score of each target sub-feature vector based on the initial value of the fused bottleneck token unit and multiple target sub-feature vectors of each modality; normalize the initial attention scores of the multiple target sub-feature vectors to obtain the weight of each target sub-feature vector; and based on the weight of each target sub-feature vector, weightedly average the multiple target sub-feature vectors, and use the obtained average value as the target attention score of the modality.

[0171] Optionally, the identification module 604 is specifically used to fuse the values ​​of the multiple fused bottleneck tokens and input them into a classification network to perform personnel status identification to obtain a recognition result.

[0172] Optionally, the feature extraction module 602 is specifically used to input the multimodal physiological signal data of the person into the corresponding feature extraction network for feature extraction, and obtain initial feature vectors corresponding to multiple modalities; perform signal projection on the initial feature vectors corresponding to the multiple modalities to obtain target feature vectors corresponding to the multiple modalities, and the dimensions of the target feature vectors corresponding to the multiple modalities are the same.

[0173] Optionally, the acquisition module 601 is specifically used to preprocess the multimodal physiological data included in the physiological signal data to obtain preprocessed physiological signal data, wherein the preprocessing includes at least one of the following: filtering, artifact removal and independent component analysis; the multimodal physiological signal data includes at least one of the following: electroencephalogram EEG, heart rate variability HRV, skin electrodermal activity EDA, respiratory signal RESP, photoplethysmogram PPG, skin temperature SKT, blood oxygen saturation SPO2, electrocardiogram ECG, electromyogram EMG and human body nine-axis posture data.

[0174] The device of this embodiment can be used to execute the technical solutions of the above-mentioned method embodiments accordingly. Its implementation principles and technical effects are similar and will not be repeated here.

[0175] Figure 7 A schematic diagram of the structure of an edge computing device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the edge computing device of this embodiment includes a processor 701 and a memory 702, wherein the memory 702 stores programs or instructions that can be run on the processor 701, and when the program or instructions are executed by the processor 701, the steps of any of the above-mentioned multimodal human factors intelligent personnel status recognition method embodiments are implemented.

[0176] An embodiment of the present application also provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of any of the above-mentioned multimodal human factors intelligent personnel status recognition method embodiments are implemented.

[0177] An embodiment of the present application also provides a computer program product, which, when executed by a processor, implements the steps of any of the above-mentioned multi-modal human factor intelligent personnel status recognition method embodiments.

[0178] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0179] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general hardware platform, and of course, can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, disk, CD, etc.), including several instructions to enable a terminal or a network-side device to execute the methods described in each embodiment of the present application.

[0180] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of the present application and the scope of protection of the claims, and these implementation methods are all within the protection of the present application.

Claims

1. A multimodal human factors intelligent personnel status recognition method, characterized in that: include: Acquiring physiological signal data, wherein the physiological signal data includes multimodal physiological signal data of a person; Performing feature extraction on the multi-modal physiological signal data to obtain target feature vectors corresponding to the multi-modalities respectively; Inputting the target feature vectors corresponding to the multi-modalities into a multi-modal bottleneck transformation network for attention calculation to obtain values ​​of multiple fused bottleneck tokens, wherein the multi-modal bottleneck transformation network includes multiple fused bottleneck token units; Personnel status recognition is performed based on the values ​​of the multiple fused bottleneck tokens.

2. The method according to claim 1, characterized in that The target feature vectors corresponding to the multi-modalities are input into the multi-modal bottleneck transformation network for attention calculation to obtain the values ​​of multiple fusion bottleneck tokens, including: For each fused bottleneck token unit, obtain the target attention score for each modality; Based on the target attention scores of multiple modalities, the value of the fused bottleneck token is obtained.

3. The method according to claim 2, characterized in that For each fusion bottleneck token unit, the attention score of each modality is obtained, including: For the fused bottleneck token unit, based on the initial value of the fused bottleneck token unit and a plurality of target sub-feature vectors of each modality, calculating an initial attention score of each target sub-feature vector; Normalize the initial attention scores of multiple target sub-feature vectors to obtain the weight of each target sub-feature vector; Based on the weight of each target sub-feature vector, the multiple target sub-feature vectors are weighted summed, and the obtained sum is used as the target attention score of the modality.

4. The method according to claim 2, characterized in that: For each fusion bottleneck token unit, the attention score of each modality is obtained, including: For the fused bottleneck token unit, based on the initial value of the fused bottleneck token unit and a plurality of target sub-feature vectors of each modality, calculating an initial attention score of each target sub-feature vector; Normalize the initial attention scores of multiple target sub-feature vectors to obtain the weight of each target sub-feature vector; Based on the weight of each target sub-feature vector, the multiple target sub-feature vectors are weighted and averaged, and the obtained average value is used as the target attention score of the modality.

5. The method according to any one of claims 1 to 4, characterized in that: The performing personnel status recognition based on the values ​​of the multiple fused bottleneck tokens includes: fusing the values ​​of the multiple fused bottleneck tokens and inputting them into a classification network to perform personnel status recognition to obtain a recognition result.

6. The method according to any one of claims 1 to 4, characterized in that: The feature extraction of the multimodal physiological signal data to obtain target feature vectors corresponding to the multimodalities includes: inputting the multimodal physiological signal data of the person into the corresponding feature extraction network for feature extraction, and obtaining initial feature vectors corresponding to the multiple modalities; performing signal projection on the initial feature vectors corresponding to the multiple modalities to obtain target feature vectors corresponding to the multimodalities, and the dimensions of the target feature vectors corresponding to the multiple modalities are the same.

7. The method according to any one of claims 1 to 4, characterized in that: The step of acquiring physiological signal data comprises: The multimodal physiological data included in the physiological signal data is preprocessed to obtain preprocessed physiological signal data, wherein the preprocessing includes at least one of the following: filtering, artifact removal and independent component analysis; the multimodal physiological signal data includes at least one of the following: electroencephalogram (EEG), heart rate variability (HRV), skin electrodermal activity (EDA), respiratory signal (RESP), photoplethysmogram (PPG), skin temperature (SKT), blood oxygen saturation (SPO2), electrocardiogram (ECG), electromyogram (EMG) and human body nine-axis posture data.

8. A human factors intelligent personnel status recognition device, characterized in that: include: An acquisition module, used to acquire physiological signal data, wherein the physiological signal data includes multi-modal physiological signal data of a person; A feature extraction module is used to extract features from the multi-modal physiological signal data to obtain target feature vectors corresponding to the multi-modal data; A processing module, used for inputting the target feature vectors corresponding to the multi-modalities into a multi-modal bottleneck transformation network for attention calculation to obtain values ​​of multiple fused bottleneck tokens, wherein the multi-modal bottleneck transformation network includes multiple fused bottleneck token units; An identification module is used to identify the status of a person based on the values ​​of the multiple fused bottleneck tokens.

9. An edge computing device, characterized in that: It includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the multimodal human factor intelligent personnel status recognition method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the multimodal human factors intelligent personnel status recognition method as described in any one of claims 1 to 7 are implemented.

Images