Human factor intelligent state monitoring model training method, monitoring method and device

Abstract

This invention provides a training method, monitoring method, and device for a human-centric intelligent state monitoring model. It involves collecting physiological and video signals from individuals to construct a training sample set. The initial neural network model includes a data preprocessing module, a common feature extraction module, and a multi-head feature extraction module. The data preprocessing module preprocesses the physiological and video signals to generate two-dimensional spatial data and then fuses them. The common feature extraction module extracts and fuses features from the fused signals to generate multi-scale feature maps, which are then summarized. The multi-head feature extraction module fuses the multi-scale feature maps to obtain a high-dimensional feature map, and generates prediction results based on the high-dimensional feature map. The initial model is trained using the training sample set to ultimately obtain the human-centric intelligent state monitoring model. This invention fuses multivariate time-series data with two-dimensional image data to construct and train a human-centric intelligent state monitoring model, resulting in faster and more accurate monitoring.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a training method, monitoring method and device for a human-caused intelligent state monitoring model based on multimodal synchronous data fusion. Background Technology

[0002] With the development of the times and the continuous improvement of people's living standards, cars have gradually entered thousands of households and become one of the main means of transportation. However, while cars bring convenience and improve travel efficiency, frequent traffic accidents have also caused heavy economic losses and threatened people's lives and property. Statistics show that among various causes of traffic accidents, fatigue driving and distracted driving account for a relatively high proportion. Therefore, if the human-induced intelligent driving state can be monitored in a timely manner, traffic accidents caused by poor driving conditions can be effectively avoided.

[0003] Currently, the monitoring of human intelligence status relies heavily on sensors and deep learning models. Among these, various deep learning models are widely used to analyze the time-series signals of human intelligence status obtained from monitoring, extract corresponding features, and determine the human intelligence status. Examples include Recurrent Neural Networks (RNNs), Temporal Convolutional Networks (TCNs), and Transformers.

[0004] RNNs and TCNs primarily focus on capturing changes between nearby time points, making them prone to gradient explosion and vanishing problems. Therefore, they cannot effectively capture long-term temporal dependencies, resulting in large prediction errors and low monitoring accuracy. Furthermore, RNNs are computationally slow, leading to low monitoring efficiency. The improved recurrent neural network—Long Short-Term Memory Network (LSTM)—alleviates gradient descent or vanishing to some extent, but it has many parameters, a large model size, and its computation method remains blocking (processing sequentially over time), unlike Transformers which can be computed in parallel, resulting in high computational costs. While Transformers have a natural advantage in modeling long-term dependencies and can be computed in parallel, the extremely complex temporal changes in reality make it difficult to uncover reliable temporal dependencies using only discrete-time attention.

[0005] The introduction of the TimesNet network greatly improved this situation by utilizing the inherent multi-periodicity of time series and the Fast Fourier Transform to identify the two types of temporal changes presented in the time series: intra-periodic and inter-periodic, in order to obtain better prediction results. However, this method is designed for multiple one-dimensional time series as raw data and cannot process two-dimensional data such as images or videos. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a training method, monitoring method and device for a human-caused intelligent state monitoring model based on multimodal synchronous data fusion, so as to eliminate or improve one or more defects existing in the prior art, and solve the problems of low monitoring efficiency, low accuracy, high computational cost and unsuitability for two-dimensional data such as images or videos in existing human-caused intelligent state monitoring schemes.

[0007] On one hand, the present invention provides a training method for a human-centric intelligent state monitoring model based on multimodal synchronous data fusion, characterized in that the method includes the following steps:

[0008] A training sample set is constructed, which contains multiple samples. Each sample includes collected physiological and video signals of the personnel, including electrocardiogram (ECG), electroencephalogram (EEG), and electromyogram (EMG) signals. Real labels are added to each sample.

[0009] An initial neural network model is constructed, which includes a data preprocessing module, a data fusion module, a common feature extraction module, a multi-head feature extraction module, and a prediction output layer; the common feature extraction module includes a convolutional layer, a dense fusion layer, and a spatial pyramid pooling layer.

[0010] The data preprocessing module performs a first data preprocessing operation on the physiological signals in each sample to obtain a two-dimensional physiological signal; it then uses a preset deep learning method to extract key frames from the video signals in each sample and performs a second data preprocessing operation on the key frames to obtain a two-dimensional video signal.

[0011] The data fusion module fuses the two-dimensional physiological signal and the two-dimensional video signal in the channel direction to generate a three-dimensional fused signal;

[0012] The common feature extraction module uses the convolutional layer and the dense fusion layer to extract and fuse features from the three-dimensional fusion signal, generating feature maps at different scales. The spatial pyramid pooling layer then aggregates the feature maps through pyramid pooling operations at different scales.

[0013] The multi-head feature extraction module fuses feature maps of different scales obtained by the common feature extraction module to obtain a high-dimensional feature map;

[0014] The prediction output layer generates prediction results for the corresponding samples based on the high-dimensional feature map;

[0015] The initial neural network model is trained using the training sample set, and a loss is constructed between the predicted result and the true label. The parameters of the initial neural network model are adjusted using the loss, and finally a human-caused intelligent state monitoring model with multimodal synchronous data fusion is obtained.

[0016] In some embodiments of the present invention, the data preprocessing module performs a first data preprocessing operation on the physiological signals in each sample to obtain two-dimensional physiological signals, and further includes:

[0017] Fast Fourier transform was performed on the electrocardiogram (ECG), electroencephalogram (EEG), and electromyogram (EMG) signals in the physiological signals to obtain the corresponding ECG amplitude-frequency characteristic maps, EEG amplitude-frequency characteristic maps, and EMG amplitude-frequency characteristic maps.

[0018] Extract the first preset number of frequencies with the largest amplitude from the electrocardiogram amplitude-frequency characteristic map, the electroencephalogram amplitude-frequency characteristic map, and the electromyogram amplitude-frequency characteristic map respectively to obtain the electrocardiogram frequency, the electroencephalogram frequency, and the electromyogram frequency;

[0019] The ECG frequency, EEG frequency, and EMG frequency are used to set corresponding ECG cycles, EEG cycles, and EMG cycles; the ECG signal is decomposed into multiple cycles using the ECG cycle to generate ECG decomposition results; the EEG signal is decomposed into multiple cycles using the EEG cycle to generate EEG decomposition results; and the EMG signal is decomposed into multiple cycles using the EMG cycle to generate EMG decomposition results.

[0020] The electrocardiogram (ECG), electroencephalogram (EEG), and electromyogram (EMG) decomposition results are respectively subjected to data dimensionality upscaling. Data dimensionality upscaling refers to stacking the results of multi-cycle decomposition on a two-dimensional plane. Since the data dimensions corresponding to different cycles are different, data gaps caused by different data lengths are filled with 0s during stacking. After data dimensionality upscaling, two-dimensional ECG signals, two-dimensional EEG signals, and two-dimensional EMG signals are obtained respectively.

[0021] In some embodiments of the present invention, performing a second data preprocessing operation on the keyframes to obtain a two-dimensional video signal further includes:

[0022] The keyframes are converted into grayscale images to obtain grayscale keyframes;

[0023] The grayscale keyframes are segmented so that the dimensions of the segmented grayscale keyframes are the same as the dimensions of the two-dimensional electrocardiogram signal, the two-dimensional electroencephalogram signal, and the two-dimensional electromyogram signal.

[0024] In some embodiments of the present invention, the common feature extraction module uses the convolutional layer to extract features from the input image and generate a first intermediate feature map, wherein the size of the first intermediate feature map is:

[0025]

[0026] Where A represents the height or width of the first intermediate feature map; W represents the height or width of the input image; K represents the kernel size of the convolutional layer; P represents the expanded pixels of the convolutional layer; and S represents the stride of the convolutional layer.

[0027] In some embodiments of the present invention, the dense fusion layer includes a segmentation module, a bottleneck module, a channel merging module, and a 1×1 convolutional block.

[0028] In some embodiments of the present invention, the first intermediate feature map generated by the convolutional layer is input into the dense fusion layer for feature fusion to generate a second intermediate feature map, including the following steps:

[0029] The segmentation module divides the first intermediate feature map into two segmented images of the same size: a first segmented image and a second segmented image.

[0030] The bottleneck module performs two consecutive bottleneck operations on the first segmented image to obtain a first bottleneck image and a first secondary bottleneck image; the bottleneck module performs two consecutive bottleneck operations on the second segmented image to obtain a second bottleneck image and a second secondary bottleneck image.

[0031] The channel merging module performs channel fusion on the first secondary bottleneck image, the first bottleneck image, the first segmented image, the second secondary bottleneck image, the second bottleneck image, and the second segmented image in sequence to obtain a fused feature map;

[0032] The 1×1 convolutional block performs a convolution operation on the fused feature map and adjusts its size.

[0033] In some embodiments of the present invention, the multi-head feature extraction module further includes the dense fusion layer. The multi-head feature extraction module performs upsampling operation on the feature map obtained by the common feature extraction module, performs channel merging operation with other feature maps of different scales, performs feature fusion operation using the dense fusion layer, and performs convolution operation using a preset convolution layer. Based on the above operations, multiple high-dimensional feature maps of different scales are finally obtained.

[0034] In some embodiments of the present invention, the prediction output layer generates prediction results for corresponding samples based on the high-dimensional feature map, and further includes:

[0035] The high-dimensional feature map is subjected to average pooling and activated using the Softmax function to obtain the corresponding prediction results.

[0036] On the other hand, the present invention provides a human-centric intelligent state monitoring method based on multimodal synchronous data fusion, the method comprising the following steps:

[0037] The system collects physiological and video signals from personnel, including electrocardiogram (ECG), electroencephalogram (EEG), and electromyogram (EMG) signals.

[0038] The physiological signal and the video signal are input into the human-caused intelligent state monitoring model obtained by any of the multimodal synchronous data fusion training methods mentioned above, and corresponding prediction results are generated to determine the human-caused intelligent state.

[0039] On the other hand, the present invention provides a training device for a human-centric intelligent state monitoring model based on multimodal synchronous data fusion, the device comprising:

[0040] The sample set construction module is used to collect and organize physiological signals and video signals from multiple individuals to construct a training sample set, and to add labels to each sample in the training sample set.

[0041] The model training module is used to train the initial neural network model using the training sample set and based on the multimodal synchronous data fusion human-cause intelligent state monitoring model training method described above, so as to obtain the modal data fusion human-cause intelligent state monitoring model.

[0042] On the other hand, the present invention provides a human-centric intelligent state monitoring device with multimodal synchronous data fusion, the device comprising:

[0043] The signal acquisition module is used to collect physiological and video signals from personnel.

[0044] The signal analysis module includes a human-caused intelligent state monitoring model trained based on the multimodal synchronous data fusion human-caused intelligent state monitoring model training method described above. The signal analysis module is used to input the collected physiological signals and video signals into the multimodal synchronous data fusion human-caused intelligent state monitoring model to obtain the human-caused intelligent state.

[0045] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods mentioned above.

[0046] The beneficial effects of the present invention are at least as follows:

[0047] This invention provides a training method, monitoring method, and apparatus for a human-caused intelligent state monitoring model using multimodal synchronous data fusion, comprising: collecting physiological signals and video signals from personnel to construct a training sample set; an initial neural network model including a data preprocessing module, a data fusion module, a common feature extraction module, and a multi-head feature extraction module; the data preprocessing module preprocesses the physiological signals and video signals to generate two-dimensional spatial data, which is then fused by the data fusion module; the common feature extraction module extracts and fuses features from the fused signals to generate multi-scale feature maps, which are then summarized; the multi-head feature extraction module fuses the multi-scale feature maps to obtain a high-dimensional feature map, and generates prediction results based on the high-dimensional feature map; the initial model is trained using the training sample set to finally obtain the human-caused intelligent state monitoring model. The training method provided by this invention constructs a training sample set based on electrocardiogram (ECG), electroencephalogram (EEG), electromyogram (EMG), and video signals, providing comprehensive data. It combines one-dimensional waveform data with two-dimensional image (video) data and fuses them in the channel direction to generate three-dimensional data, enabling the model to use multimodal synchronous data for monitoring and improving monitoring accuracy. The constructed initial network model has a common feature extraction module and a multi-head feature extraction module, which can accurately and comprehensively extract features, further improving monitoring accuracy. Moreover, the constructed initial network model has a faster computation speed than existing recurrent neural network models, improving monitoring efficiency.

[0048] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0049] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0050] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0051] Figure 1 This is a schematic diagram illustrating the steps of a human-centric intelligent state monitoring model training method for multimodal synchronous data fusion in one embodiment of the present invention.

[0052] Figure 2 This is a flowchart illustrating the structure of a human-centric intelligent state monitoring model for multimodal synchronous data fusion in one embodiment of the present invention.

[0053] Figure 3This is a waveform diagram illustrating multi-cycle decomposition of an electrocardiogram signal in one embodiment of the present invention.

[0054] Figure 4 This is a schematic diagram of a two-dimensional electrocardiogram signal waveform in one embodiment of the present invention.

[0055] Figure 5 This is a flowchart illustrating the structure of a densely fused layer in one embodiment of the present invention.

[0056] Figure 6 This is an execution flowchart of the common feature extraction module in one embodiment of the present invention.

[0057] Figure 7 This is a structural diagram of a human-caused intelligent state monitoring model training device for multimodal synchronous data fusion in one embodiment of the present invention.

[0058] Figure 8 This is a structural diagram of a human-centric intelligent state monitoring device for multimodal synchronous data fusion in one embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0060] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0061] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0062] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0063] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0064] It should be emphasized here that the step markers mentioned below are not a limitation on the order of the steps, but should be understood as meaning that the steps can be executed in the order mentioned in the embodiments, or in a different order than in the embodiments, or several steps can be executed simultaneously.

[0065] To address the problems of low monitoring efficiency, low accuracy, high computational cost, and unsuitability for two-dimensional data such as images or videos in existing human-caused intelligent state monitoring schemes, this invention provides a multimodal synchronous data fusion method for training a human-caused intelligent state monitoring model, such as... Figure 1 As shown, the method includes the following steps S101 to S103:

[0066] Step S101: Construct a training sample set, which contains multiple samples. Each sample includes collected physiological and video signals from the personnel. The physiological signals include electrocardiogram (ECG), electroencephalogram (EEG), and electromyography (EMG) signals. Add a ground truth label to each sample.

[0067] Step S102: Construct an initial neural network model, which includes a data preprocessing module, a data fusion module, a common feature extraction module, a multi-head feature extraction module, and a prediction output layer. The common feature extraction module further includes a convolutional layer, a dense fusion layer, and a spatial pyramid pooling layer.

[0068] The data preprocessing module performs a first data preprocessing operation on the physiological signals in each sample to obtain a two-dimensional physiological signal; it then uses a preset deep learning method to extract key frames from the video signal in each sample and performs a second data preprocessing operation on the key frames to obtain a two-dimensional video signal.

[0069] The data fusion module fuses two-dimensional physiological signals and two-dimensional video signals in the channel direction to generate a three-dimensional fused signal.

[0070] The common feature extraction module uses convolutional layers and dense fusion layers to extract and fuse features from the 3D fusion signal, generating feature maps at different scales. The spatial pyramid pooling layer then aggregates the feature maps through pyramid pooling operations at different scales.

[0071] The multi-head feature extraction module fuses feature maps of different scales obtained from the common feature extraction module to obtain a high-dimensional feature map.

[0072] The prediction output layer generates prediction results for the corresponding samples based on the high-dimensional feature map.

[0073] Step S103: Train the initial neural network model using the training sample set, construct the loss between the prediction result and the real label, adjust the parameters of the initial neural network model using the loss, and finally obtain the human-caused intelligent state monitoring model with multimodal synchronous data fusion.

[0074] In step S101, a training sample set for model training is first constructed.

[0075] In some embodiments, a portable bio-information acquisition device is used to synchronously acquire physiological signals of the person; a camera is used to acquire video signals of the person. The physiological signals include electrocardiogram (ECG), electroencephalogram (EEG), and electromyogram (EMG) signals. For example, X1 represents the ECG signal, X2 represents the EEG signal, and X3 represents the EMG signal. The video signals include facial expressions, facial movements, and limb movements of the human body.

[0076] In some embodiments, each sample is labeled with a normal or abnormal true label. When a person is in a normal driving state, the corresponding sample is marked as 0, and when a person is in an abnormal driving state, the corresponding sample is marked as 1. Abnormal driving states include situations such as fatigued driving and distracted driving.

[0077] In step S102, an initial neural network model is constructed, such as... Figure 2 As shown, the initial neural network model includes a data preprocessing module, a data fusion module, a common feature extraction module, a multi-head feature extraction module, and a prediction output layer.

[0078] In the data preprocessing module, physiological and video signals in the samples are preprocessed to improve signal quality and transform them into data of the required dimensions. The data preprocessing module can be further divided into a waveform data standardization module and a video data keyframe extraction module, which perform data preprocessing on the physiological and video signals respectively.

[0079] Specifically, the waveform data standardization module performs a first data preprocessing operation on the physiological signals in each sample to obtain two-dimensional physiological signals:

[0080] Fast Fourier Transform (FFT) was performed on the electrocardiogram (ECG) signal X1, electroencephalogram (EEG) signal X2, and electromyogram (EMG) signal X3 from the physiological signals to obtain the corresponding amplitude-frequency characteristic maps of ECG, EEG, and EMG, respectively. The frequencies with the largest amplitudes, denoted as N, were extracted from each of the ECG, EEG, and EMG amplitude-frequency characteristic maps to obtain the ECG frequency K. 11 K 12 ... K 1N EEG frequency K 21 K 22 ... K 2N and electromyographic frequency K 31 K 32 ... K 3N Let the electrocardiogram frequency be K. 11 K 12 ... K 1N The corresponding periods are T 11 T 12 ... T 1N Then use the ECG cycle T11 T 12 ... T 1N The ECG signal X1 is decomposed into multiple cycles to generate the ECG decomposition results, such as... Figure 3 As shown; let the brainwave frequency K 21 K 22 ... K 2N The corresponding periods are T 21 T 22 ... T 2N Then use EEG cycle T 21 T 22 ... T 2N The EEG signal X2 is decomposed into multiple cycles to generate the EEG decomposition results; let the electromyography frequency K be... 31 K 32 ... K 3N The corresponding periods are T 31 T 32 ... T 3N Then use electromyography cycle T 31 T 32 ... T 3N The electromyography (EMG) signal X3 was decomposed into multiple cycles to generate EMG decomposition results. The EMG decomposition results from ECG, EEG, and EMG were then upscaled from one-dimensional space to two-dimensional space to obtain two-dimensional ECG signal P1, two-dimensional EEG signal P2, and two-dimensional EMG signal P3.

[0081] Data dimensionality upscaling refers to stacking the results of multi-period decomposition on a two-dimensional plane. Since different periods correspond to different data dimensions, missing data due to varying data lengths is padded with zeros during stacking. After dimensionality upscaling, two-dimensional electrocardiogram (ECG), electroencephalogram (EEG), and electromyographic (EMG) signals are obtained. For example, the two-dimensional ECG signal P1... Figure 4 As shown.

[0082] The video data keyframe extraction module uses a preset deep learning method to extract keyframes from the video signal in each sample. For example, the deep learning method chosen is Deep Keyframe Detection in Human Action Video to extract M keyframes. The extracted keyframes undergo a second data preprocessing operation to obtain a two-dimensional video signal.

[0083] The keyframes are converted to grayscale images to obtain grayscale keyframes. These grayscale keyframes are then segmented so that their dimensions are the same as those of the two-dimensional ECG signal P1, the two-dimensional EEG signal P2, and the two-dimensional EMG signal P3. The segmented grayscale keyframes are denoted as F1, F2, ..., F... M F1, F2, ..., F MAll data are in two-dimensional space.

[0084] In the data fusion module, the two-dimensional physiological signals P1, P2, P3 and the two-dimensional video signals F1, F2, ..., F3 are combined. M The fusion is performed in the channel direction to generate a three-dimensional fused signal. Let the size of the three-dimensional fused signal be H×W×C, where H represents the height, W represents the width, and C represents the channel.

[0085] The common feature extraction module includes convolutional layers, dense fusion layers, and spatial pyramid pooling layers.

[0086] In some embodiments, the image size changes after processing by the convolutional layer. The common feature extraction module uses the convolutional layer to extract features from the input image and generate a first intermediate feature map, wherein the size of the first intermediate feature map is calculated as shown in formula (1):

[0087]

[0088] Where A represents the height (or width) of the first intermediate feature map; W represents the height (or width) of the input image; K represents the kernel size of the convolutional layer; P represents the expanded pixels of the convolutional layer; and S represents the stride of the convolutional layer.

[0089] It has been verified that when the stride S is 2, the convolution operation can reduce the size of the image to half of the original image size, that is, the width becomes half of the original and the height becomes half of the original.

[0090] Dense Fusion Layer (DC2F) is an optimized solution proposed based on C2F to address the gradient divergence problem in deep networks.

[0091] In some embodiments, the dense fusion layer further includes a segmentation module, a bottleneck module, a channel merging module, and a 1×1 convolutional block.

[0092] In some embodiments, such as Figure 5 As shown, the dense fusion layer performs feature fusion on the first intermediate feature map generated by the convolutional layer to generate a second intermediate feature map, including the following steps:

[0093] The segmentation module divides the first intermediate feature map into two identical segmented images: a first segmented image and a second segmented image. For example, if the size of the first intermediate feature map is [L, M, N], then the sizes of the first and second segmented images are [L, M, N / 2]. The first segmented image is denoted as PL, and the second segmented image as PR. In convolutional neural networks in computer vision, feature map sizes are typically represented by [L, M, N], where L, M, and N represent the length, width, and depth of the feature map, respectively.

[0094] The bottleneck module performs two consecutive bottleneck operations on the first segmented image PL to obtain the first bottleneck image PL2 and the first secondary bottleneck image PL3; the bottleneck module performs two consecutive bottleneck operations on the second segmented image PR to obtain the second bottleneck image PR2 and the second secondary bottleneck image PR3.

[0095] The channel merging module performs channel fusion on the first and second bottleneck images PL3, PL2, PL, PR3, PR2 and PR in sequence to obtain a fused feature map.

[0096] A 1×1 convolutional block performs a convolution operation on the fused feature map, where the convolutional block stride is 1, the expansion pixel is 0, and the number of convolutional kernels is C, so that the second intermediate feature map output by the dense fusion layer has the same size as the input first intermediate feature map.

[0097] In the common feature extraction module, convolutional layers and dense fusion layers are used to extract and fuse features from the 3D fusion signal, generating feature maps at different scales. Then, spatial pyramid pooling layers are used to aggregate these feature maps through pyramid pooling operations at different scales. Figure 6 As shown, the specific steps include S102101 to S102111:

[0098] Step S102101: Receive the three-dimensional fusion signal input from the data fusion module, and denote the three-dimensional fusion signal as C0, where the dimensions of C0 are [H, W, C].

[0099] Step S102102: Perform a convolution operation on the 3D fused signal using a convolutional layer to obtain C1, where the size of C1 is [H / 2, W / 2, C×2]. The convolutional layer parameters are Conv: (K=3, S=2, P=1), where K represents the kernel size (3×3), S represents the stride, and P represents the expanded pixels. The same principle applies below and will not be further explained.

[0100] Step S102103: Perform a convolution operation on C1 using a convolutional layer to obtain C2, where the size of C2 is [H / 4, W / 4, C×4]. The convolutional layer parameters are Conv: (K=3, S=2, P=1).

[0101] Step S102104: Perform DC2F operation on C2 using a dense fusion layer to obtain C3, the size of C3 is [H / 4, W / 4, C×4].

[0102] Step S102105: Perform a convolution operation on C3 using a convolutional layer to obtain C4, where the size of C4 is [H / 8, W / 8, C×8]. The convolutional layer parameters are Conv: (K=3, S=2, P=1).

[0103] Step S102106: Perform DC2F operation on C4 using a dense fusion layer to obtain C5, the size of C5 is [H / 8, W / 8, C×8].

[0104] Step S102107: Perform a convolution operation on C5 using a convolutional layer to obtain C6, where the size of C6 is [H / 16, W / 16, C×16]. The convolutional layer parameters are Conv: (K=3, S=2, P=1).

[0105] Step S102108: Perform DC2F operation on C6 using a dense fusion layer to obtain C7, the size of C7 is [H / 16, W / 16, C×16].

[0106] Step S102109: Perform a convolution operation on C7 using a convolutional layer to obtain C8, where the size of C8 is [H / 32, W / 32, C×16]. The convolutional layer parameters are Conv: (K=3, S=2, P=1).

[0107] Step S102110: Perform DC2F operation on C8 using a dense fusion layer to obtain C9, the size of C9 is [H / 32, W / 32, C×16].

[0108] Step S10211l: Perform spatial pyramid pooling fusion operation on C9 using a spatial pyramid pooling layer to obtain C 101 C 10 The dimensions are [H / 32, W / 32, C×16].

[0109] The multi-head feature extraction module also includes a dense fusion layer. This module upsamples the feature maps obtained from the common feature extraction module, merges channels with feature maps of different scales, performs feature fusion using the dense fusion layer, and performs convolution operations using a pre-defined convolutional layer. Based on these operations, multiple high-dimensional feature maps of different scales are finally obtained. Specifically, this includes the following steps S102201~102215:

[0110] Step S102201: Summarize the feature maps C obtained from the common feature extraction module. 10 Perform an upsampling operation to obtain C 11 C 11 The dimensions are [H / 16, W / 16, C×16].

[0111] Step S102202: Perform a channel merging operation on C11 and C7 extracted from the common feature extraction module to obtain C. 12 C 12 The dimensions are [H / 16, W / 16, C×32].

[0112] Step S102203: Use a dense fusion layer to apply C 12 Perform a DC2F operation to obtain C. 13 C 13 The dimensions are [H / 16, W / 16, C×32].

[0113] Step S102204: For C 13 Perform a convolution operation to obtain C 14 C 14 The dimensions are [H / 16, W / 16, C×8]. The convolution parameters are Conv: (K=1, S=1, P=0).

[0114] Step S102205: For C 14 Perform an upsampling operation to obtain C 15 C 15 The dimensions are [H / 8, W / 8, C×8].

[0115] Step S102206: Place C 15 The channel merging operation is performed on C5 obtained from the common feature extraction module to obtain C. 16 C 16 The dimensions are [H / 8, W / 8, C×16].

[0116] Step S102207: Use a dense fusion layer to apply C 16 Perform a DC2F operation to obtain C. 17 C 17 The dimensions are [H / 8, W / 8, C×16].

[0117] Step S102208: For C 17 Perform a convolution operation to obtain C 18 C 18 The dimensions are [H / 8, W / 8, C×8]. The convolution parameters are Conv: (K=1, S=1, P=0).

[0118] Step S102209: For C 18 Perform a convolution operation to obtain C 19 C 19 The dimensions are [H / 16, W / 16, C×8]. The convolution parameters are Conv: (K=3, S=2, P=1).

[0119] Step S102210: Place C 19 and C 14 Perform a channel merging operation to obtain C 20 C 20 The dimensions are [H / 16, W / 16, C×16].

[0120] Step S102211: Use a dense fusion layer to apply C 20 Perform a DC2F operation to obtain C. 21 C 21 The dimensions are [H / 16, W / 16, C×16].

[0121] Step S102212: For C 21 Perform a convolution operation to obtain C 22 C 22 The dimensions are [H / 32, W / 32, C×16]. The convolution parameters are Conv: (K=3, S=2, P=1).

[0122] Step S102213: Place C 22 The feature map C obtained by summarizing the features from the common feature extraction module 10 Perform a channel merging operation to obtain C 23 C 23 The dimensions are [H / 32, W / 32, C×32].

[0123] Step S102214: For C 23 Perform a convolution operation to obtain C 24 C 24 The dimensions are [H / 32, W / 32, C×16]. The convolution parameters are Conv: (K=1, S=1, P=0).

[0124] Step S102215: Use a dense fusion layer to apply C 24 Perform a DC2F operation to obtain C. 25 C 25 The dimensions are [H / 32, W / 32, C×16].

[0125] The final multi-head feature extraction module outputs C 18 C 21 C 25 High-dimensional feature maps at three scales.

[0126] In the prediction output layer, prediction results for the corresponding samples are generated based on the high-dimensional feature map.

[0127] In some embodiments, the high-dimensional feature map is subjected to average pooling and activated using the Softmax function to obtain the corresponding prediction result, i.e., the prediction result is C. 18 C 21 C 25 The predicted voting results can be used to determine a person's intellectual state.

[0128] In step S103, the initial neural network model is trained using the training sample set, and the loss between the prediction result and the real label is constructed. The parameters of the initial neural network model are adjusted using the loss, and finally a human-caused intelligent state monitoring model with multimodal synchronous data fusion is obtained.

[0129] In some embodiments, the loss is a binary cross-entropy loss.

[0130] Those skilled in the art will understand that the human-caused intelligent state in this invention includes human-caused intelligent driving states, such as driver-driven states, human-vehicle interactive driving states, and driverless states, etc., and this invention is not limited thereto. It should be clarified that this invention is not limited to driving states, but can also include states in other fields, such as focused classroom states, emotional states, fatigue states, and stress states.

[0131] The present invention also provides a human-centric intelligent state monitoring method for multimodal synchronous data fusion, comprising steps S201 to S202:

[0132] Step S201: Collect physiological and video signals from the personnel, including electrocardiogram (ECG), electroencephalogram (EEG), and electromyogram (EMG) signals.

[0133] Step S202: Input physiological signals and video signals into the human-caused intelligent state monitoring model obtained by the multimodal synchronous data fusion human-caused intelligent state monitoring model training method described above, so as to generate corresponding prediction results and determine the human-caused intelligent state.

[0134] Accordingly, the present invention provides a training device for a human-centric intelligent state monitoring model based on multimodal synchronous data fusion, such as... Figure 7 As shown, the training device for the human-caused intelligent state monitoring model, which involves multimodal synchronous data fusion, includes a sample set construction module 710 and a model training module 720. Specifically:

[0135] The sample set construction module 710 is used to collect and organize physiological and video signals from multiple individuals to construct a training sample set and add labels to each sample in the training sample set.

[0136] The model training module 720 is used to train the initial neural network model based on the training sample set constructed by the sample set construction module 710 and the human factors intelligent state monitoring model training method of multimodal synchronous data fusion described above, so as to obtain the human factors intelligent state monitoring model of modal data fusion.

[0137] Accordingly, the present invention provides a human-centric intelligent state monitoring device with multimodal synchronous data fusion, such as... Figure 8As shown, the multimodal synchronous data fusion human-centric intelligent state monitoring device includes a signal acquisition module 810 and a signal analysis module 820, specifically:

[0138] The signal acquisition module 810 is used to acquire physiological signals and video signals from personnel.

[0139] The signal analysis module 820 includes a human-caused intelligent state monitoring model 821 trained based on the human-caused intelligent state monitoring model training method described above using multimodal synchronous data fusion. The signal analysis module 820 is used to input the collected physiological signals and video signals into the human-caused intelligent state monitoring model 821 to obtain the human-caused intelligent state.

[0140] It should be noted that the description of the device embodiment corresponds to the description of its corresponding method embodiment. Therefore, any parts not described in detail can be referred to the method embodiment above.

[0141] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a training method for a human-caused intelligent state monitoring model based on multimodal synchronous data fusion and a human-caused intelligent state monitoring method based on multimodal synchronous data fusion.

[0142] Corresponding to the above method, the present invention also provides an apparatus comprising a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the apparatus performs the steps of the method as described above.

[0143] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0144] In summary, this invention provides a training method, monitoring method, and apparatus for a human-caused intelligent state monitoring model using multimodal synchronous data fusion, comprising: collecting physiological signals and video signals from personnel to construct a training sample set; an initial neural network model including a data preprocessing module, a data fusion module, a common feature extraction module, and a multi-head feature extraction module; the data preprocessing module preprocesses the physiological signals and video signals to generate two-dimensional spatial data, which is then fused by the data fusion module; the common feature extraction module extracts and fuses features from the fused signals to generate multi-scale feature maps, which are then summarized; the multi-head feature extraction module fuses the multi-scale feature maps to obtain a high-dimensional feature map, and generates prediction results based on the high-dimensional feature map; the initial model is trained using the training sample set to finally obtain the human-caused intelligent state monitoring model. The training method provided by this invention constructs a training sample set based on electrocardiogram (ECG), electroencephalogram (EEG), electromyogram (EMG), and video signals, providing comprehensive data. It combines one-dimensional waveform data (multivariate time-series data, including ECG and EEG waveforms) with two-dimensional image (video) data and fuses them along the channel direction to generate three-dimensional data. This allows the model to utilize multimodal synchronous data for monitoring, improving monitoring accuracy. The constructed initial network model includes a common feature extraction module and a multi-head feature extraction module, which can accurately and comprehensively extract features, further improving monitoring accuracy. Moreover, the constructed initial network model has a faster computation speed compared to existing recurrent neural networks and other models, improving monitoring efficiency.

[0145] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0146] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0147] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0148] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A training method for a human-centric intelligent state monitoring model using multimodal synchronous data fusion, characterized in that, The method includes the following steps: A training sample set is constructed, which contains multiple samples. Each sample includes collected physiological and video signals of the personnel, including electrocardiogram (ECG), electroencephalogram (EEG), and electromyogram (EMG) signals. Real labels are added to each sample. An initial neural network model is constructed, which includes a data preprocessing module, a data fusion module, a common feature extraction module, a multi-head feature extraction module, and a prediction output layer; the common feature extraction module includes a convolutional layer, a dense fusion layer, and a spatial pyramid pooling layer. The data preprocessing module performs a first data preprocessing operation on the physiological signals in each sample to obtain a two-dimensional physiological signal; it then extracts keyframes from the video signal in each sample using a preset deep learning method, and performs a second data preprocessing operation on the keyframes to obtain a two-dimensional video signal. The first data preprocessing operation includes: performing Fast Fourier Transform on the electrocardiogram (ECG), electroencephalogram (EEG), and electromyogram (EMG) signals in the physiological signal to obtain corresponding ECG amplitude-frequency characteristic maps, EEG amplitude-frequency characteristic maps, and EMG amplitude-frequency characteristic maps; and extracting the largest amplitude from each of the ECG, EEG, and EMG amplitude-frequency characteristic maps. The frequency of electrocardiogram (ECG), electroencephalogram (EEG), and electromyography (EMG) is determined by counting a number of frequencies. Corresponding ECG, EEG, and EMG cycles are then set based on these frequencies. The ECG signal is decomposed into multiple cycles using the ECG cycles to generate ECG decomposition results, the EEG signal is decomposed into multiple cycles using the EEG cycles to generate EEG decomposition results, and the EMG signal is decomposed into multiple cycles using the EMG cycles to generate EMG decomposition results. The ECG, EEG, and EMG decomposition results are then subjected to data dimensionality upscaling to obtain two-dimensional ECG, two-dimensional EEG, and two-dimensional EMG signals. The data fusion module fuses the two-dimensional physiological signal and the two-dimensional video signal in the channel direction to generate a three-dimensional fused signal; The common feature extraction module uses the convolutional layer and the dense fusion layer to extract and fuse features from the three-dimensional fusion signal, generating feature maps at different scales. The spatial pyramid pooling layer then aggregates the feature maps through pyramid pooling operations at different scales. The multi-head feature extraction module fuses feature maps of different scales obtained by the common feature extraction module to obtain a high-dimensional feature map; The prediction output layer generates prediction results for the corresponding samples based on the high-dimensional feature map; The initial neural network model is trained using the training sample set, and a loss is constructed between the predicted result and the true label. The parameters of the initial neural network model are adjusted using the loss, and finally a human-caused intelligent state monitoring model with multimodal synchronous data fusion is obtained.

2. The training method for a human-centric intelligent state monitoring model based on multimodal synchronous data fusion according to claim 1, characterized in that, The process of performing a second data preprocessing operation on the keyframes to obtain a two-dimensional video signal further includes: The keyframes are converted into grayscale images to obtain grayscale keyframes; The grayscale keyframes are segmented so that the dimensions of the segmented grayscale keyframes are the same as the dimensions of the two-dimensional electrocardiogram signal, the two-dimensional electroencephalogram signal, and the two-dimensional electromyogram signal.

3. The training method for a human-centric intelligent state monitoring model based on multimodal synchronous data fusion according to claim 1, characterized in that, The common feature extraction module uses the convolutional layer to extract features from the input image and generates a first intermediate feature map. The size of the first intermediate feature map is: ； in, This indicates the height or width of the first intermediate feature map; This indicates the height or width of the input image; This indicates the kernel size of the convolutional layer; This represents the extended pixels of the convolutional layer; This indicates the step size of the convolutional layer.

4. The training method for a human-centric intelligent state monitoring model based on multimodal synchronous data fusion according to claim 3, characterized in that, The dense fusion layer includes a segmentation module, a bottleneck module, a channel merging module, and a 1×1 convolutional block.

5. The training method for a human-centric intelligent state monitoring model based on multimodal synchronous data fusion according to claim 4, characterized in that, The first intermediate feature map generated by the convolutional layer is input into the dense fusion layer for feature fusion to generate a second intermediate feature map, including the following steps: The segmentation module divides the first intermediate feature map into two segmented images of the same size: a first segmented image and a second segmented image. The bottleneck module performs two consecutive bottleneck operations on the first segmented image to obtain a first bottleneck image and a first secondary bottleneck image; the bottleneck module performs two consecutive bottleneck operations on the second segmented image to obtain a second bottleneck image and a second secondary bottleneck image. The channel merging module performs channel fusion on the first secondary bottleneck image, the first bottleneck image, the first segmented image, the second secondary bottleneck image, the second bottleneck image, and the second segmented image in sequence to obtain a fused feature map; The 1×1 convolutional block performs a convolution operation on the fused feature map and adjusts its size.

6. The training method for a human-centric intelligent state monitoring model based on multimodal synchronous data fusion according to claim 1, characterized in that, The multi-head feature extraction module also includes the dense fusion layer. The multi-head feature extraction module performs upsampling on the feature map obtained by the common feature extraction module, performs channel merging on the feature map with other feature maps of different scales, performs feature fusion on the dense fusion layer, and performs convolution on the preset convolution layer. Based on the above operations, multiple high-dimensional feature maps of different scales are finally obtained.

7. The training method for a human-centric intelligent state monitoring model based on multimodal synchronous data fusion according to claim 6, characterized in that, The prediction output layer generates prediction results for the corresponding samples based on the high-dimensional feature map, and further includes: The high-dimensional feature map is subjected to average pooling and activated using the Softmax function to obtain the corresponding prediction results.

8. A human-centric intelligent state monitoring method based on multimodal synchronous data fusion, characterized in that, The method includes the following steps: The system collects physiological and video signals from personnel, including electrocardiogram (ECG), electroencephalogram (EEG), and electromyogram (EMG) signals. The physiological signal and the video signal are input into the human-caused intelligent state monitoring model obtained by the multimodal synchronous data fusion human-caused intelligent state monitoring model training method as described in any one of claims 1 to 7, and corresponding prediction results are generated to determine the human-caused intelligent state.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images