Training method and apparatus for human-factor intelligence state monitoring model, and human-factor intelligence state monitoring method and apparatus

Abstract

Provided in the embodiments of the present application are a training method and apparatus for a human-factor intelligence state monitoring model, and a human-factor intelligence state monitoring method and apparatus. The training method comprises: collecting physiological signals and video signals of persons to construct a training sample set, wherein an initial neural network model comprises a data preprocessing module, a common feature extraction module, a multi-head feature extraction module, etc.; the data preprocessing module preprocessing the physiological signals and the video signals to generate two-dimensional spatial data, and fusing the two-dimensional spatial data; the common feature extraction module performing feature extraction and fusion on a fused signal to generate feature maps on multiple scales, and summarizing the feature maps on multiple scales; the multi-head feature extraction module fusing the feature maps on multiple scales to obtain a high-dimensional feature map, and generating a prediction result on the basis of the high-dimensional feature map; and using the training sample set to train the initial model, so as to finally obtain a human-factor intelligence state monitoring model. In the present application, multivariate time-series data is fused with two-dimensional image data to construct and train a human-factor intelligence state monitoring model, so that faster and more accurate monitoring is implemented.

Description

Human factors intelligent state monitoring model training method, monitoring method and device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on August 7, 2023, with application number 2023109878379 and invention name “Human Factor Intelligent State Monitoring Model Training Method, Monitoring Method and Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a human factors intelligent state monitoring model training method, monitoring method and device with multimodal synchronous data fusion. Background Art

[0004] With the development of the times and the continuous improvement of people's quality of life, cars have gradually become a commonplace and a primary means of transportation. However, while cars bring convenience and improve travel efficiency, frequent traffic accidents also lead to heavy economic losses and pose a threat to people's lives and property. Statistics show that among the various causes of traffic accidents, fatigue and distracted driving account for a high proportion. Therefore, if the human factor intelligent driving status can be timely monitored, traffic accidents caused by poor driving status can be effectively avoided.

[0005] Currently, human intelligence status monitoring relies primarily on sensors and deep learning models. Various deep learning models, such as recurrent neural networks (RNNs), temporal convolutional networks (TCNs), and transformer networks, are widely used to analyze the time series signals of human intelligence status obtained through monitoring and extract corresponding features to determine the human intelligence status.

[0006] RNN and TCN mainly focus on capturing changes between adjacent moments, and are prone to gradient explosion and gradient vanishing problems. Therefore, they cannot effectively capture long-term temporal dependencies, resulting in large errors in prediction results and low monitoring accuracy. In addition, RNN has a slow computing speed and low monitoring efficiency. The improved recurrent neural network - Long Short-Term Memory Network (LSTM) has alleviated gradient descent or gradient vanishing to a certain extent, but it has many parameters, a large model, and a blocking calculation method (sequential processing over time). It cannot be calculated in parallel like Transformer, and the computational cost is high. Although Transformer has a natural advantage in modeling long-term dependencies and can be calculated in parallel, due to the extremely complex temporal changes in actual situations, it is difficult to mine reliable temporal dependencies by relying solely on attention between discrete time points.

[0007] The introduction of the TimesNet network has greatly improved this situation. It uses the natural multi-periodicity of time series and the fast Fourier transform to identify the two types of temporal changes in time series, namely intra-periodic and inter-cycle changes, to obtain better prediction results. However, this method targets multiple one-dimensional time series as the original data and cannot process two-dimensional data such as images or videos.

[0008] Summary of the Invention

[0009] The embodiments of the present application provide a multimodal synchronous data fusion human factors intelligent state monitoring model training method, monitoring method and device to eliminate or improve one or more defects existing in the prior art, and solve the problems of low monitoring efficiency, low accuracy, high computational cost and unsuitability for two-dimensional data such as images or videos in existing human factors intelligent state monitoring solutions.

[0010] On the one hand, the present application provides a multi-modal synchronous data fusion human factors intelligent state monitoring model training method, characterized in that the method comprises the following steps:

[0011] Constructing a training sample set, wherein the training sample set includes multiple samples, each sample includes a collected physiological signal and video signal of a person, wherein the physiological signal includes an electrocardiogram signal, an electroencephalogram signal, and an electromyography signal; adding a true label to each sample;

[0012] Constructing an initial neural network model, the initial neural network model includes a data preprocessing module, a data fusion module, a common feature extraction module, a multi-head feature extraction module and a prediction output layer; the common feature extraction module includes a convolution layer, a dense fusion layer and a spatial pyramid pooling layer;

[0013] The data preprocessing module performs a first data preprocessing operation on the physiological signal in each sample to obtain a two-dimensional physiological signal; uses a preset deep learning method to extract a key frame of the video signal in each sample, and performs a second data preprocessing operation on the key frame to obtain a two-dimensional video signal;

[0014] The data fusion module fuses the two-dimensional physiological signal and the two-dimensional video signal in the channel direction to generate a three-dimensional fusion signal;

[0015] The common feature extraction module uses the convolution layer and the dense fusion layer to extract and fuse features of the three-dimensional fusion signal to generate feature maps of different scales, and uses the spatial pyramid pooling layer to aggregate the feature maps through pyramid pooling operations of different scales;

[0016] The multi-head feature extraction module fuses the feature maps of different scales obtained by the common feature extraction module to obtain a high-dimensional feature map;

[0017] The prediction output layer generates a prediction result of the corresponding sample according to the high-dimensional feature map;

[0018] The initial neural network model is trained using the training sample set, and a loss between the predicted result and the true label is constructed. The parameters of the initial neural network model are adjusted using the loss, and finally a human factors intelligent state monitoring model with multimodal synchronous data fusion is obtained.

[0019] In some embodiments of the present application, the data preprocessing module performs a first data preprocessing operation on the physiological signal in each sample to obtain a two-dimensional physiological signal, and further includes:

[0020] Performing fast Fourier transform on the electrocardiogram (ECG) signal, the electroencephalogram (EEG) signal, and the electromyography (EMG) signal in the physiological signals to obtain corresponding ECG amplitude-frequency characteristic graphs, EEG amplitude-frequency characteristic graphs, and EMG amplitude-frequency characteristic graphs;

[0021] Extracting the first preset number of frequencies with the largest amplitudes from the ECG amplitude-frequency characteristic graph, the EEG amplitude-frequency characteristic graph, and the EMG amplitude-frequency characteristic graph respectively to obtain the ECG frequency, the EEG frequency, and the EMG frequency;

[0022] The corresponding ECG cycle, EEG cycle and EMG cycle are set according to the ECG frequency, the EEG frequency and the EMG frequency; the ECG signal is decomposed into multiple cycles using the ECG cycle to generate an ECG decomposition result, the EEG signal is decomposed into multiple cycles using the EEG cycle to generate an EEG decomposition result, and the EMG signal is decomposed into multiple cycles using the EMG cycle to generate an EMG decomposition result;

[0023] The ECG decomposition results, the EEG decomposition results and the EMG decomposition results are respectively subjected to data dimensionality upgrading. The data dimensionality upgrading refers to stacking the results of multi-cycle decomposition on a two-dimensional plane. Since different cycles correspond to different data dimensions, the data missing caused by different data lengths during stacking are filled with the number 0. After data dimensionality upgrading, two-dimensional ECG signals, two-dimensional EEG signals and two-dimensional EMG signals are respectively obtained.

[0024] In some embodiments of the present application, performing a second data preprocessing operation on the key frame to obtain a two-dimensional video signal further includes:

[0025] Converting the key frame into a grayscale image to obtain a grayscale key frame;

[0026] The grayscale key frame is cut so that the dimension of the cut grayscale key frame is the same as the dimension of the two-dimensional electrocardiogram signal, the two-dimensional electroencephalogram signal, and the two-dimensional electromyography signal.

[0027] In some embodiments of the present application, the common feature extraction module uses the convolution layer to extract features from the input image to generate a first intermediate feature map, and the size of the first intermediate feature map is:

[0028] Among them, A represents the height or width of the first intermediate feature map; W represents the height or width of the input image; K represents the convolution kernel size of the convolution layer; P represents the expanded pixels of the convolution layer; S represents the jump step size of the convolution layer.

[0029] In some embodiments of the present application, the dense fusion layer includes a segmentation module, a bottleneck module, a channel merging module and a 1×1 convolution block.

[0030] In some embodiments of the present application, inputting the first intermediate feature map generated by the convolution layer into the dense fusion layer for feature fusion to generate a second intermediate feature map includes the following steps:

[0031] The segmentation module segments the first intermediate feature map into two first segmented images and a second segmented image of the same size;

[0032] The bottleneck module performs two consecutive bottleneck operations on the first segmented image to obtain a first bottleneck image and a first secondary bottleneck image; the bottleneck module performs two consecutive bottleneck operations on the second segmented image to obtain a second bottleneck image and a second secondary bottleneck image;

[0033] The channel merging module sequentially performs channel fusion on the first secondary bottleneck image, the first bottleneck image, the first segmented image, the second secondary bottleneck image, the second bottleneck image, and the second segmented image to obtain a fused feature map;

[0034] The 1×1 convolution block performs a convolution operation on the fused feature map to adjust the size.

[0035] In some embodiments of the present application, the multi-head feature extraction module also includes the dense fusion layer. The multi-head feature extraction module performs an upsampling operation on the feature map summarized by the common feature extraction module, performs a channel merging operation with feature maps of other different scales, uses the dense fusion layer to perform feature fusion operations, and uses a preset convolutional layer to perform convolution operations. Based on the above operations, multiple high-dimensional feature maps of different scales are finally obtained.

[0036] In some embodiments of the present application, the prediction output layer generates a prediction result of the corresponding sample according to the high-dimensional feature map, further comprising:

[0037] The high-dimensional feature map is average-pooled and activated using the Softmax function to obtain the corresponding prediction result.

[0038] On the other hand, the present application provides a multi-modal synchronous data fusion human factors intelligent state monitoring method, the method comprising the following steps:

[0039] Collecting physiological signals and video signals of personnel, wherein the physiological signals include electrocardiogram signals, electroencephalogram signals and electromyography signals;

[0040] The physiological signal and the video signal are input into the multimodal synchronous data fusion human factors intelligence state monitoring model obtained by the multimodal synchronous data fusion human factors intelligence state monitoring model training method as mentioned above, and the corresponding prediction results are generated to judge the human factors intelligence state.

[0041] On the other hand, the present application provides a multi-modal synchronous data fusion human factors intelligent state monitoring model training device, the device comprising:

[0042] A sample set construction module is used to collect and organize physiological signals and video signals of multiple people to construct a training sample set and add a label to each sample in the training sample set;

[0043] The model training module is used to use the training sample set and train the initial neural network model based on the multimodal synchronous data fusion human factors intelligent state monitoring model training method described above to obtain a modal data fusion human factors intelligent state monitoring model.

[0044] On the other hand, the present application provides a multi-modal synchronous data fusion human factors intelligent state monitoring device, the device comprising:

[0045] Signal acquisition module, used to collect physiological signals and video signals of personnel;

[0046] A signal analysis module, wherein the signal analysis module includes a multimodal synchronous data fusion human factors intelligent state monitoring model trained based on the multimodal synchronous data fusion human factors intelligent state monitoring model training method described above; the signal analysis module is used to input the collected physiological signals and the video signals into the multimodal synchronous data fusion human factors intelligent state monitoring model to obtain the human factors intelligent state.

[0047] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods mentioned above.

[0048] On the other hand, the present application also provides a device comprising: a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the steps of implementing any one of the methods mentioned above.

[0049] On the other hand, the present application also provides an apparatus, comprising: a processor, configured to call and run a computer program from a memory, so that a device equipped with the apparatus executes the steps of implementing any one of the methods mentioned above.

[0050] On the other hand, the present application also provides a computer program product, comprising computer program instructions, which enable a computer to execute the steps of any one of the methods mentioned above.

[0051] The beneficial effects of this application are at least:

[0052] The present application provides a multi-modal synchronous data fusion human factors intelligent state monitoring model training method, monitoring method and device, including: collecting physiological signals and video signals of personnel to construct a training sample set; the initial neural network model includes a data preprocessing module, a data fusion module, a common feature extraction module, a multi-head feature extraction module, etc.; the data preprocessing module preprocesses the physiological signals and video signals to generate two-dimensional spatial data, and fuses them through the data fusion module; the common feature extraction module extracts and fuses features of the fused signals to generate multi-scale feature maps, and summarizes them; the multi-head feature extraction module fuses the multi-scale feature maps to obtain high-dimensional feature maps, and generates prediction results based on the high-dimensional feature maps; the initial model is trained using the training sample set to ultimately obtain a human factors intelligent state monitoring model. The training method provided in this application constructs a training sample set based on electrocardiogram signals, electroencephalogram signals, electromyography signals and video signals, and the data is comprehensive; combines one-dimensional waveform data with two-dimensional image (video) data, and fuses them in the channel direction to generate three-dimensional data, so that the model can use multimodal synchronous data for monitoring, thereby improving monitoring accuracy; the constructed initial network model is provided with a common feature extraction module and a multi-head feature extraction module, which can accurately and comprehensively extract features, further improving monitoring accuracy, and the constructed initial network model has a faster computing speed than existing models such as recurrent neural networks, thereby improving monitoring efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] FIG1 is a schematic diagram of the steps of a method for training a human factors intelligent state monitoring model using multimodal synchronous data fusion in one embodiment of the present application.

[0054] FIG2 is a structural flow chart of a human factors intelligent state monitoring model for multimodal synchronous data fusion in one embodiment of the present application.

[0055] FIG3 is a waveform diagram of multi-cycle decomposition of an electrocardiogram signal in an embodiment of the present application.

[0056] FIG4 is a schematic diagram of a two-dimensional electrocardiogram signal waveform in an embodiment of the present application.

[0057] FIG5 is a structural flow chart of a dense fusion layer in one embodiment of the present application.

[0058] FIG6 is an execution flow chart of the common feature extraction module in one embodiment of the present application.

[0059] FIG7 is a structural diagram of a human factors intelligent state monitoring model training device with multimodal synchronous data fusion in one embodiment of the present application.

[0060] FIG8 is a structural diagram of a human factors intelligent status monitoring device with multi-modal synchronous data fusion in one embodiment of the present application. DETAILED DESCRIPTION

[0061] The following will describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of the embodiments. With respect to the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0062] In order to solve the problems of low monitoring efficiency, low accuracy, high computational cost, and unsuitability for two-dimensional data such as images or videos in existing human factors intelligent state monitoring solutions, this application provides a human factors intelligent state monitoring model training method using multimodal synchronous data fusion. As shown in FIG1 , the method includes the following steps S101 to S103:

[0063] Step S101: Construct a training sample set. The training sample set contains multiple samples, each of which includes physiological signals and video signals collected from a person. Physiological signals include electrocardiogram (ECG), electroencephalogram (EEG), and electromyography (EMG). Add a true label to each sample.

[0064] Step S102: constructing an initial neural network model, which includes a data preprocessing module, a data fusion module, a common feature extraction module, a multi-head feature extraction module and a prediction output layer, wherein the common feature extraction module further includes a convolution layer, a dense fusion layer and a spatial pyramid pooling layer.

[0065] The data preprocessing module performs a first data preprocessing operation on the physiological signal in each sample to obtain a two-dimensional physiological signal; uses a preset deep learning method to extract the key frame of the video signal in each sample, and performs a second data preprocessing operation on the key frame to obtain a two-dimensional video signal.

[0066] The data fusion module fuses the two-dimensional physiological signal and the two-dimensional video signal in the channel direction to generate a three-dimensional fusion signal.

[0067] The common feature extraction module uses convolutional layers and dense fusion layers to extract and fuse features of the three-dimensional fusion signal to generate feature maps of different scales, and uses the spatial pyramid pooling layer to aggregate the feature maps through pyramid pooling operations of different scales.

[0068] The multi-head feature extraction module fuses the feature maps of different scales obtained by the common feature extraction module to obtain a high-dimensional feature map.

[0069] The prediction output layer generates the prediction results of the corresponding samples based on the high-dimensional feature map.

[0070] Step S103: Use the training sample set to train the initial neural network model, and construct the loss between the predicted results and the true labels. Use the loss to adjust the parameters of the initial neural network model, and finally obtain a multimodal synchronous data fusion human factors intelligent state monitoring model.

[0071] In step S101, a training sample set for model training is first constructed.

[0072] In some embodiments, a portable biometric information collection device is used to synchronously collect physiological signals from a person, and a camera is used to collect video signals from the person. Physiological signals include electrocardiogram (ECG) signals, electroencephalogram (EEG) signals, and electromyography (EMG) signals. For example, X1 represents an ECG signal, X2 represents an EEG signal, and X3 represents an EMG signal. Video signals include facial expressions, facial movements, and body movements, which are characteristic of human intelligence.

[0073] In some embodiments, each sample is labeled normal or abnormal. When the driver is in a normal driving state, the corresponding sample is labeled 0, and when the driver is in an abnormal driving state, the corresponding sample is labeled 1. Abnormal driving states include fatigue driving, distracted driving, and other situations.

[0074] In step S102, an initial neural network model is constructed. As shown in FIG2 , the initial neural network model includes a data preprocessing module, a data fusion module, a common feature extraction module, a multi-head feature extraction module and a prediction output layer.

[0075] In the data preprocessing module, the physiological and video signals in the sample are preprocessed to improve signal quality and convert them into data of the required dimensions. The data preprocessing module can be further divided into a waveform data normalization module and a video data keyframe extraction module, which perform data preprocessing on physiological and video signals, respectively.

[0076] Specifically, the waveform data normalization processing module performs a first data preprocessing operation on the physiological signal in each sample to obtain a two-dimensional physiological signal:

[0077] Perform fast Fourier transform on the ECG signal X1, EEG signal X2 and EMG signal X3 in the physiological signal to obtain the corresponding ECG amplitude-frequency characteristic diagram, EEG amplitude-frequency characteristic diagram and EMG amplitude-frequency characteristic diagram. Extract the preset number (denoted as N) frequencies with the largest amplitude in the ECG amplitude-frequency characteristic diagram, EEG amplitude-frequency characteristic diagram and EMG amplitude-frequency characteristic diagram respectively to obtain the ECG frequency K 11 、H 12 ,...,H 1N , EEG frequency H 21 , K 22 ,...,K 2N and EMG frequency K 31, K 32 ,...,K 3N . Assume that the ECG frequency K 11 , K 12 ,...,K 1N The corresponding periods are T 11 、T 12 ,...,T 1N , then use the ECG cycle T 11 、T 12 ,...,T 1N Perform multi-cycle decomposition on the ECG signal X1 to generate the ECG decomposition result, as shown in Figure 3; suppose the EEG frequency K 21 , K 22 ,...,K 2N The corresponding periods are T 21 、T 22 ,...,T 21 , then use the EEG cycle T 21 、T 22 ,...,T 2N Perform multi-cycle decomposition on the EEG signal X2 to generate the EEG decomposition result; set the EMG frequency K 31 , K 32 ,...,K 31 The corresponding periods are T 31 、T 32 ,...,T 3N , then use the myoelectric period T 31 、T 32 ,...,T 31 Perform multi-cycle decomposition on the electromyographic signal X3 to generate the electromyographic decomposition results. Perform data dimensionality upgrade on the electrocardiographic decomposition results, the electroencephalographic decomposition results, and the electromyographic decomposition results, respectively, from one-dimensional space to two-dimensional space, to obtain a two-dimensional electrocardiographic signal P1, a two-dimensional electroencephalographic signal P2, and a two-dimensional electromyographic signal P3.

[0078] Data dimensionality increase involves stacking the results of multi-period decomposition on a two-dimensional plane. Because different periods correspond to different data dimensions, missing data due to varying data lengths are padded with zeros during stacking. After data dimensionality increase, the resulting signals are a two-dimensional electrocardiogram (ECG), a two-dimensional electroencephalogram (EEG), and a two-dimensional electromyography (EMG). For example, the two-dimensional ECG signal P1 is shown in Figure 4.

[0079] The video data keyframe extraction module uses a preset deep learning method to extract keyframes from the video signal of each sample. For example, the deep learning method selected is Deep Keyframe Detection in Human Action Video to extract M keyframes. A second data preprocessing operation is performed on the extracted keyframes to obtain a two-dimensional video signal:

[0080] The keyframes are converted into grayscale images to obtain grayscale keyframes; the grayscale keyframes are cut so that the dimensions of the cut grayscale keyframes are the same as those of the two-dimensional electrocardiogram signal P1, the two-dimensional electroencephalogram signal P2, and the two-dimensional electromyography signal P3. The cut grayscale keyframes are denoted as F1, F2, ..., F M , F1, F2, ..., F M All are two-dimensional spatial data.

[0081] In the data fusion module, the two-dimensional physiological signals P1, P2, P3 and the two-dimensional video signals F1, X2, ..., F M Fusion is performed in the channel direction to generate a three-dimensional fusion signal. Assume that the size of the three-dimensional fusion signal is H×W×G, where H represents height, W represents width, and C represents channel.

[0082] The common feature extraction module includes convolutional layers, dense fusion layers, and spatial pyramid pooling layers.

[0083] In some embodiments, after being processed by the convolution layer, the image size changes. The common feature extraction module uses the convolution layer to extract features from the input image and generates a first intermediate feature map, wherein the size calculation formula of the first intermediate feature map is shown in formula (1):

[0084] Among them, A represents the height (or width) of the first intermediate feature map; W represents the height (or width) of the input image; K represents the convolution kernel size of the convolution layer; P represents the expanded pixels of the convolution layer; S represents the jump step size of the convolution layer.

[0085] It has been verified that when the stride S is 2, the convolution operation can reduce the size of the image to half of the original image size, that is, the width becomes half of the original and the height becomes half of the original.

[0086] Dense fusion layer (DC2F) is an optimization solution proposed based on C2F to solve the gradient divergence problem of deep networks.

[0087] In some embodiments, the dense fusion layer further includes a segmentation module, a bottleneck module (BottleNeck), a channel merging module and a 1×1 convolution block.

[0088] In some embodiments, as shown in FIG5 , the dense fusion layer performs feature fusion on the first intermediate feature map generated by the convolutional layer to generate a second intermediate feature map, including the following steps:

[0089] The segmentation module splits the first intermediate feature map into two equally sized first and second segmented images. For example, if the size of the first intermediate feature map is [L, M, N], then the sizes of the first and second segmented images are [L, M, N / 2]. The first segmented image is denoted as PL, and the second segmented image is denoted as PR. In convolutional neural networks in computer vision, feature map sizes are typically expressed as [L, M, N], where L, M, and N represent the length, width, and depth of the feature map, respectively.

[0090] The bottleneck module performs two consecutive bottleneck operations on the first segmented image PL to obtain a first bottleneck image PL2 and a first secondary bottleneck image PL3; the bottleneck module performs two consecutive bottleneck operations on the second segmented image PR to obtain a second bottleneck image PR2 and a second secondary bottleneck image PR3.

[0091] The channel merging module performs channel fusion on the first secondary bottleneck image PL3, the first bottleneck image PL2, the first segmented image PL, the second secondary bottleneck image PR3, the second bottleneck image PR2 and the second segmented image PR in sequence to obtain a fused feature map.

[0092] A 1×1 convolution block performs a convolution operation on the fused feature map, where the convolution block stride is 1, the expansion pixel is 0, and the number of convolution kernels is C, so that the second intermediate feature map output by the dense fusion layer has the same size as the first intermediate feature map input.

[0093] In the common feature extraction module, the convolution layer and the dense fusion layer are used to extract and fuse the features of the 3D fusion signal to generate feature maps of different scales. The spatial pyramid pooling layer is then used to aggregate the feature maps through pyramid pooling operations of different scales. As shown in FIG6 , the module specifically includes the following steps S102101 to S102111:

[0094] Step S102101: Receive a three-dimensional fusion signal input by a data fusion module, record the three-dimensional fusion signal as C0, and the size of C0 is [H, W, C].

[0095] Step S102102: Use a convolutional layer to perform a convolution operation on the 3D fused signal to obtain C1, where the size of C1 is [H / 2, W / 2, C×2]. The convolutional layer parameters are Conv: (K=3, S=2, P=1), where K represents the convolution kernel size of 3×3; S represents the jump step size; and P represents the expanded pixels. The same principle applies to the following and will not be further explained.

[0096] Step S102103: Use a convolutional layer to perform a convolution operation on C1 to obtain C2, where the size of C2 is [H / 4, W / 4, C×4]. The convolutional layer parameters are Conv: (K=3, S=2, P=1).

[0097] Step S102104: Use the dense fusion layer to perform a DC2F operation on C2 to obtain C3, where the size of C3 is [H / 4, W / 4, C×4].

[0098] Step S102105: Use a convolutional layer to perform a convolution operation on C3 to obtain C4, where the size of C4 is [H / 8, W / 8, C×8]. The convolutional layer parameters are Conv: (K=3, S=2, P=1).

[0099] Step S102106: Use the dense fusion layer to perform a DC2F operation on C4 to obtain C5, where the size of C5 is [H / 8, W / 8, C×8].

[0100] Step S102107: Use a convolutional layer to perform a convolution operation on C5 to obtain C6, where the size of C6 is [H / 16, W / 16, C×16]. The convolutional layer parameters are Conv: (K=3, S=2, P=1).

[0101] Step S102108: Use the dense fusion layer to perform a DC2F operation on C6 to obtain C7, where the size of C7 is [H / 16, W / 16, C×16].

[0102] Step S102109: Use a convolutional layer to perform a convolution operation on C7 to obtain C8, where the size of C8 is [H / 32, W / 32, C×16]. The convolutional layer parameters are Conv: (K=3, S=2, P=1).

[0103] Step S102110: Use the dense fusion layer to perform a DC2F operation on C8 to obtain C9, where the size of C9 is [H / 32, W / 32, C×16].

[0104] Step S102111: Use the spatial pyramid pooling layer to perform a spatial pyramid pooling fusion operation on C9 to obtain C 10 , C 10 The dimensions are [H / 32,W / 32,C×16].

[0105] The multi-head feature extraction module also has a dense fusion layer. The multi-head feature extraction module performs an upsampling operation on the feature map obtained by the common feature extraction module, performs a channel merging operation with feature maps of other different scales, uses a dense fusion layer to perform feature fusion operations, and uses a preset convolution layer to perform convolution operations. Based on the above operations, multiple high-dimensional feature maps of different scales are finally obtained. Specifically, the following steps are included: S102201 to S102215:

[0106] Step S102201: Summarize the feature graph C obtained by the common feature extraction module 10 Perform upsampling operation to obtain C11 , C 11 The dimensions are [H / 16, W / 16, C×16].

[0107] Step S102202: C 11 The channel merging operation is performed on C7 extracted from the common feature extraction module to obtain C 12 , C 12 The dimensions are [H / 16, W / 16, C×32].

[0108] Step S102203: Use dense fusion layer to fusion C 12 Perform DC2F operation to obtain C 13 , C 13 The dimensions are [H / 16, W / 16, C×32].

[0109] Step S102204: C 13 Perform convolution operation to get C 14 , C 14 The size of the network is [H / 16, W / 16, C×8]. The convolution parameters are Conv: (K=1, S=1, P=0).

[0110] Step S102205: C 14 Perform upsampling operation to obtain C 15 , C 15 The dimensions are [H / 8,W / 8,C×8].

[0111] Step S102206: C 15 The channel merging operation is performed on C5 extracted from the common feature extraction module to obtain C 16 , C 16 The dimensions are [H / 8,W / 8,C×16].

[0112] Step S102207: Use dense fusion layer to fusion C 16 Perform DC2F operation to obtain C 17 , C 17 The dimensions are [H / 8,W / 8,C×16].

[0113] Step S102208: C 17 Perform convolution operation to get C 18 , C 18 The size of the network is [H / 8, W / 8, C×8]. The convolution parameters are Conv: (K=1, S=1, P=0).

[0114] Step S102209: C 18 Perform convolution operation to get C 19 , C19 The size of the network is [H / 16, W / 16, C×8]. The convolution parameters are Conv: (K=3, S=2, P=1).

[0115] Step S102210: C 19 and C 14 Perform channel merging operation to obtain C 20 , C 20 The dimensions are [H / 16, W / 16, C× 16].

[0116] Step S102211: Use dense fusion layer to fusion C 20 Perform DC2F operation to obtain C 21 , C 21 The dimensions are [H / 16, W / 16, C×16].

[0117] Step S102212: For C 21 Perform convolution operation to get C 22 , C 22 The size of the network is [H / 32, W / 32, C×16]. The convolution parameters are Conv: (K=3, S=2, P=1).

[0118] Step S102213: C 22 and the feature map C obtained by summarizing the common feature extraction module 10 Perform channel merging operation to obtain C 23 , C 23 The dimensions are [H / 32,W / 32,C×32].

[0119] Step S102214: C 23 Perform convolution operation to get C 24 , C 24 The size of the network is [H / 32, W / 32, C×16]. The convolution parameters are Conv: (K=1, S=1, P=0).

[0120] Step S102215: Use dense fusion layer to fusion C 24 Perform DC2F operation to obtain C 25 , C 25 The dimensions are [H / 32,W / 32,C×16].

[0121] Finally, the multi-head feature extraction module outputs C 18 、C 21 、C 25 High-dimensional feature maps at three scales.

[0122] In the prediction output layer, the prediction results of the corresponding samples are generated based on the high-dimensional feature maps.

[0123] In some embodiments, the high-dimensional feature map is average pooled and activated using the Softmax function to obtain the corresponding prediction result, that is, the prediction result is C 18 、C 21 、C 25 The predicted voting results can be used to determine the person's intelligence status.

[0124] In step S103, the initial neural network model is trained using the training sample set, and the loss between the predicted results and the true labels is constructed. The parameters of the initial neural network model are adjusted using the loss, and finally a human factors intelligent state monitoring model with multimodal synchronous data fusion is obtained.

[0125] In some embodiments, the loss is a binary cross entropy loss.

[0126] Those skilled in the art will appreciate that the human-based intelligent state in this application includes the human-based intelligent driving state, such as the driver's driving state, the human-vehicle interactive driving state, and the unmanned driving state, and this application does not limit this. It should be clarified that this application is not limited to the driving state, but also includes other fields, such as classroom concentration, emotional state, fatigue state, stress state, etc.

[0127] The present application also provides a multi-modal synchronous data fusion human factors intelligent state monitoring method, including steps S201 to S202:

[0128] Step S201: collecting physiological signals and video signals of a person, wherein the physiological signals include electrocardiogram signals, electroencephalogram signals and electromyography signals.

[0129] Step S202: Input the physiological signal and the video signal into the multimodal synchronous data fusion human factors intelligence state monitoring model obtained by the multimodal synchronous data fusion human factors intelligence state monitoring model training method as described above to generate corresponding prediction results to judge the human factors intelligence state.

[0130] Accordingly, the present application provides a multimodal synchronous data fusion human factors intelligent state monitoring model training device. As shown in FIG7 , the multimodal synchronous data fusion human factors intelligent state monitoring model training device includes a sample set construction module 710 and a model training module 720. Specifically:

[0131] The sample set construction module 710 is used to collect and organize physiological signals and video signals of multiple persons to construct a training sample set and add a label to each sample in the training sample set.

[0132] The model training module 720 is used to use the training sample set constructed by the sample set construction module 710, and train the initial neural network model based on the multimodal synchronous data fusion human factors intelligent state monitoring model training method described above to obtain a modal data fusion human factors intelligent state monitoring model.

[0133] Accordingly, the present application provides a multi-modal synchronous data fusion human factors intelligent state monitoring device. As shown in FIG8 , the multi-modal synchronous data fusion human factors intelligent state monitoring device includes a signal acquisition module 810 and a signal analysis module 820. Specifically:

[0134] The signal acquisition module 810 is used to acquire physiological signals and video signals of personnel.

[0135] The signal analysis module 820 includes a multimodal synchronous data fusion human factors intelligent state monitoring model 821 trained based on the multimodal synchronous data fusion human factors intelligent state monitoring model training method described above; the signal analysis module 820 is used to input the collected physiological signals and video signals into the multimodal synchronous data fusion human factors intelligent state monitoring model 821 to obtain the human factors intelligent state.

[0136] It should be noted that the description of the device embodiment corresponds to the description of its corresponding method embodiment. Therefore, for parts that are not described in detail, reference can be made to the above method embodiment.

[0137] The present application also provides a device comprising: a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the steps of implementing the aforementioned multimodal synchronous data fusion human factors intelligent state monitoring model training method or multimodal synchronous data fusion human factors intelligent state monitoring method.

[0138] The present application also provides a device, including: a processor, for calling and running a computer program from a memory, so that a device installed with the device executes the steps of implementing the aforementioned multimodal synchronous data fusion human factors intelligent state monitoring model training method or multimodal synchronous data fusion human factors intelligent state monitoring method.

[0139] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned multimodal synchronous data fusion human factors intelligent state monitoring model training method or multimodal synchronous data fusion human factors intelligent state monitoring method. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0140] The present application also provides a computer program product, including computer program instructions, which enable a computer to execute the steps of implementing the aforementioned multimodal synchronous data fusion human factors intelligent state monitoring model training method or multimodal synchronous data fusion human factors intelligent state monitoring method.

[0141] In summary, the present application provides a human factors intelligent state monitoring model training method, monitoring method and device with multimodal synchronous data fusion, including: collecting physiological signals and video signals of personnel to construct a training sample set; the initial neural network model includes a data preprocessing module, a data fusion module, a common feature extraction module, a multi-head feature extraction module, etc.; the data preprocessing module preprocesses the physiological signals and video signals to generate two-dimensional spatial data, and fuses them through the data fusion module; the common feature extraction module extracts and fuses the features of the fused signals to generate a multi-scale feature map, and summarizes them; the multi-head feature extraction module fuses the multi-scale feature maps to obtain a high-dimensional feature map, and generates a prediction result based on the high-dimensional feature map; the initial model is trained using the training sample set, and finally a human factors intelligent state monitoring model is obtained. The training method provided in this application constructs a training sample set based on ECG signals, EEG signals, EMG signals and video signals, and the data is comprehensive; combines one-dimensional waveform data (multivariate time series data, including ECG waveform data and EEG waveform data) with two-dimensional image (video) data, and fuses them in the channel direction to generate three-dimensional data, so that the model can use multimodal synchronous data for monitoring, thereby improving monitoring accuracy; the constructed initial network model is provided with a common feature extraction module and a multi-head feature extraction module, which can accurately and comprehensively extract features, further improving monitoring accuracy, and the constructed initial network model has a faster computing speed than existing models such as recurrent neural networks, thereby improving monitoring efficiency.

[0142] It should be understood that the processor of the embodiments of the present application may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by hardware integrated logic circuits in the processor or software instructions. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0143] It is understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0144] It should be understood that the above-mentioned memories are exemplary but not restrictive. For example, the memories in the embodiments of the present application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM RAM (DR RAM), etc. In other words, the memories in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0145] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0146] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0147] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0150] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0151] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. In view of this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0152] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multi-modal synchronous data fusion human factor intelligent state monitoring model training method, characterized in that: The method comprises the following steps: Constructing a training sample set, wherein the training sample set includes a plurality of samples, each sample includes a collected physiological signal and a video signal of a person, wherein the physiological signal includes an electrocardiogram signal, an electroencephalogram signal, and an electromyography signal; adding a true label to each sample; Constructing an initial neural network model, the initial neural network model includes a data preprocessing module, a data fusion module, a common feature extraction module, a multi-head feature extraction module and a prediction output layer; the common feature extraction module includes a convolution layer, a dense fusion layer and a spatial pyramid pooling layer; The data preprocessing module performs a first data preprocessing operation on the physiological signal in each sample to obtain a two-dimensional physiological signal; extracts a key frame of the video signal in each sample using a preset deep learning method, and performs a second data preprocessing operation on the key frame to obtain a two-dimensional video signal; The data fusion module fuses the two-dimensional physiological signal and the two-dimensional video signal in the channel direction to generate a three-dimensional fusion signal; The common feature extraction module uses the convolution layer and the dense fusion layer to perform feature extraction and fusion on the three-dimensional fusion signal to generate feature maps of different scales, and uses the spatial pyramid pooling layer to aggregate the feature maps through pyramid pooling operations of different scales; The multi-head feature extraction module fuses the feature maps of different scales obtained by the common feature extraction module to obtain a high-dimensional feature map; The prediction output layer generates a prediction result of the corresponding sample according to the high-dimensional feature map; The initial neural network model is trained using the training sample set, and the loss between the prediction result and the true label is constructed. The parameters of the initial neural network model are adjusted using the loss, and finally a human factors intelligent state monitoring model with multimodal synchronous data fusion is obtained.

2. The method for training a human factors intelligent state monitoring model based on multi-modal synchronous data fusion according to claim 1 is characterized in that: The data preprocessing module performs a first data preprocessing operation on the physiological signal in each sample to obtain a two-dimensional physiological signal, and further includes: Performing fast Fourier transform on the electrocardiogram, electroencephalogram and electromyography signals in the physiological signals respectively to obtain corresponding electrocardiogram amplitude-frequency characteristic graph, electroencephalogram amplitude-frequency characteristic graph and electromyography amplitude-frequency characteristic graph; Respectively extracting the first preset number of frequencies with the largest amplitudes in the ECG amplitude-frequency characteristic diagram, the EEG amplitude-frequency characteristic diagram, and the EMG amplitude-frequency characteristic diagram to obtain the ECG frequency, the EEG frequency, and the EMG frequency; The corresponding ECG cycle, EEG cycle and EMG cycle are set according to the ECG frequency, the EEG frequency and the EMG frequency; the ECG signal is decomposed into multiple cycles using the ECG cycle to generate an ECG decomposition result, the EEG signal is decomposed into multiple cycles using the EEG cycle to generate an EEG decomposition result, and the EMG signal is decomposed into multiple cycles using the EMG cycle to generate an EMG decomposition result; The electrocardiogram (ECG) decomposition result, the electroencephalogram (EEG) decomposition result and the electromyography (EMG) decomposition result are respectively subjected to data dimension upgrading to obtain a two-dimensional electrocardiogram (ECG) signal, a two-dimensional electroencephalogram (EEG) signal and a two-dimensional electromyography (EMG) signal.

3. The multi-modal synchronous data fusion human factors intelligent state monitoring model training method according to claim 2 is characterized in that: Performing a second data preprocessing operation on the key frame to obtain a two-dimensional video signal also includes: Converting the key frame into a grayscale image to obtain a grayscale key frame; The grayscale key frame is cut so that the dimension of the cut grayscale key frame is the same as the dimension of the two-dimensional electrocardiogram signal, the two-dimensional electroencephalogram signal and the two-dimensional electromyography signal.

4. The multi-modal synchronous data fusion human factors intelligent state monitoring model training method according to claim 1 is characterized in that: The common feature extraction module uses the convolution layer to extract features from the input image to generate a first intermediate feature map. The size of the first intermediate feature map is: Among them, A represents the height or width of the first intermediate feature map; W represents the height or width of the input image; K represents the convolution kernel size of the convolution layer; P represents the expanded pixels of the convolution layer; S represents the jump step size of the convolution layer.

5. The method for training a human factors intelligent state monitoring model based on multi-modal synchronous data fusion according to claim 4 is characterized in that: The dense fusion layer includes a segmentation module, a bottleneck module, a channel merging module and a 1×1 convolution block.

6. The multi-modal synchronous data fusion human factor intelligent state monitoring model training method according to claim 5 is characterized in that: Inputting the first intermediate feature map generated by the convolution layer into the dense fusion layer for feature fusion to generate a second intermediate feature map, comprising the following steps: The segmentation module segments the first intermediate feature map into two first segmented images and second segmented images of the same size; The bottleneck module continuously performs two bottleneck operations on the first segmented image to obtain a first bottleneck image and a first secondary bottleneck image; the bottleneck module continuously performs two bottleneck operations on the second segmented image to obtain a second bottleneck image and a second secondary bottleneck image; The channel merging module performs channel fusion on the first secondary bottleneck image, the first bottleneck image, the first segmented image, the second secondary bottleneck image, the second bottleneck image and the second segmented image in sequence to obtain a fused feature map; The 1×1 convolution block performs a convolution operation on the fused feature map to adjust the size.

7. The method for training a human factors intelligent state monitoring model based on multi-modal synchronous data fusion according to claim 1, characterized in that: The multi-head feature extraction module also includes the dense fusion layer. The multi-head feature extraction module performs an upsampling operation on the feature map summarized by the common feature extraction module, performs a channel merging operation with feature maps of other different scales, uses the dense fusion layer to perform a feature fusion operation, and uses a preset convolutional layer to perform a convolution operation. Based on the above operations, multiple high-dimensional feature maps of different scales are finally obtained.

8. The multi-modal synchronous data fusion human factor intelligent state monitoring model training method according to claim 7 is characterized in that: The prediction output layer generates a prediction result of the corresponding sample according to the high-dimensional feature map, and also includes: The high-dimensional feature map is average-pooled and activated using the Softmax function to obtain the corresponding prediction result.

9. A human factors intelligent state monitoring method based on multi-modal synchronous data fusion, characterized in that: The method comprises the following steps: Collecting physiological signals and video signals of personnel, wherein the physiological signals include electrocardiogram signals, electroencephalogram signals and electromyography signals; The physiological signal and the video signal are input into the human factors intelligence state monitoring model with multimodal synchronous data fusion obtained by the human factors intelligence state monitoring model training method with multimodal synchronous data fusion as described in any one of claims 1 to 8, and corresponding prediction results are generated to judge the human factors intelligence state.

10. A device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory, and executing the method according to any one of claims 1 to 8, or executing the method according to claim 9.

11. A device, characterized in that: include: A processor, configured to call and run a computer program from a memory, so that a device equipped with the apparatus executes the method according to any one of claims 1 to 8, or executes the method according to claim 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8, or implements the steps of the method according to claim 9.

13. A computer program product, characterized in that The method comprises computer program instructions, wherein the computer program instructions enable a computer to execute the method according to any one of claims 1 to 8, or the method according to claim 9.

Images