Fatigue state monitoring method and device based on multi-modal data, equipment and medium

Abstract

The invention provides a fatigue state monitoring method, device, equipment and system based on multi-modal data and a medium, and the method comprises the steps: obtaining the physiological data and working data of an operator, the working data being used for representing the working data affecting the fatigue state of the operator; preprocessing the physiological data and the working data; performing index extraction and data fusion on the preprocessed physiological data and working data to obtain a fusion feature vector; and inputting the fusion feature vector into a trained deep learning model to carry out fatigue state identification so as to obtain the fatigue state of the operator. According to the method provided by the invention, the fatigue state of the operator can be comprehensively and accurately identified and monitored.

Description

Technical Field

[0001] The present application relates to the field of fatigue state monitoring, and in particular to a fatigue state monitoring method, device, equipment, system and medium based on multimodal data. Background Art

[0002] In related technologies, the fatigue status monitoring of operators mainly relies on simple physical and mental load tests, such as subjective questionnaires, interviews, and single skin electricity, electrocardiogram, and electromyography analysis. However, these methods can only provide single-dimensional data, which makes it difficult to fully and accurately reflect the actual fatigue status of operators, resulting in large errors in the monitoring results. Summary of the invention

[0003] The present application provides a fatigue status monitoring method, device, equipment, system and medium based on multimodal data, which helps to comprehensively and accurately identify and monitor the fatigue status of operators.

[0004] In a first aspect, the present application provides a fatigue state monitoring method based on multimodal data, comprising: Acquire the physiological data and work data of the operator, where the work data is used to characterize the work data that affects the fatigue state of the operator; Preprocess physiological data and working data; Extract indicators and fuse data from preprocessed physiological data and working data to obtain fused feature vectors; The fused feature vector is input into the trained deep learning model for fatigue state recognition to obtain the fatigue state of the operator.

[0005] In one possible implementation, the work data includes work time, work intensity and work environment data, and the work environment data includes work environment temperature and weather conditions; Physiological data include electromyographic signals, heart rate variability data, respiratory rate data, electrocardiographic signals and electroencephalographic signals.

[0006] In one possible implementation, the physiological data and the working data are preprocessed, including: Process physiological data and work data at a unified sampling rate; The sliding window technology is used to divide the physiological data and working data processed at a uniform sampling rate into N segments of data.

[0007] In one possible implementation, the preprocessed physiological data and working data are subjected to index extraction and data fusion, including: Extracting physiological data indicators and work data indicators based on the data of each segment respectively, and obtaining N feature vectors, each feature vector includes the physiological data indicators and work data indicators of the corresponding segment, and the physiological data indicators and work data indicators are used to characterize indicators related to the fatigue state of the operator; The N feature vectors are fused to obtain a fused feature vector.

[0008] In one possible implementation, the training of a deep learning model includes: Constructing a training sample set, the training sample set includes multiple samples, each sample includes collected physiological data and working data; Preprocess the physiological data and working data of each sample in the training sample set; Perform index extraction and data fusion on the physiological data and working data of each sample after preprocessing to obtain a fusion feature vector set; Add a true label to each sample based on the fused feature vector set; A deep learning model is constructed using a convolutional neural network and a long short-term memory network. The convolutional neural network includes a convolutional layer, a pooling layer, and a first fully connected layer, and the long short-term memory network includes a long short-term memory layer and a second fully connected layer. The convolutional layer is used to extract local features, the pooling layer is used to reduce data dimensions, the first fully connected layer is used to integrate features and output a first feature vector, and the long short-term memory layer is used to capture long-term dependencies in the first feature vector. The second fully connected layer is used to output the prediction result of the fatigue state. The deep learning model is trained using the fused feature vector set to obtain the prediction results of the fatigue state of each sample; The loss between the predicted result of the fatigue state of each sample and the corresponding true label is calculated, and the loss is used to adjust the parameters of the deep learning model.

[0009] One possible implementation method is to add a true label to each sample based on the fused feature vector set, including: The number of fatigue state levels k is calculated based on the fused feature vector set using the Bayesian information criterion algorithm; Based on the number of fatigue state levels k, a time series segmentation clustering algorithm is used to classify and label the fused feature vector set to obtain the true label corresponding to each sample.

[0010] In one possible implementation, the method further includes: An early warning reminder is provided based on the fatigue status of the operator, and the early warning reminder includes a vibration reminder, and / or a sound reminder, and / or a visual reminder.

[0011] In one possible implementation, the fatigue status includes no fatigue, low fatigue, medium fatigue and high fatigue, and the warning is performed based on the fatigue status of the operator, including: When the fatigue status of the operator is moderate or high, an early warning reminder is issued to the operator.

[0012] In one possible implementation, the physiological data indicators include time domain indicators and frequency domain indicators, the time domain indicators at least include mean value, maximum value, minimum value, variance and standard deviation, and the frequency domain indicators at least include total power, average power, median power and power peak; Work data indicators include at least working hours, work intensity level, average, maximum and minimum working environment temperature, and weather classification.

[0013] In a second aspect, the present application provides a fatigue status monitoring device based on multimodal data, comprising: An acquisition module, used for acquiring physiological data and work data of the operator, wherein the work data is used for representing work data that affects the fatigue state of the operator; A preprocessing module, used for preprocessing physiological data and working data; The index extraction and data fusion module is used to extract the index and fuse the data of the preprocessed physiological data and working data to obtain the fusion feature vector; The state recognition module is used to input the fused feature vector into the trained deep learning model for fatigue state recognition to obtain the fatigue state of the operator. In a third aspect, the present application provides an electronic device, including: a processor and a memory, the memory is used to store a computer program; the processor is used to run the computer program to implement the fatigue state monitoring method based on multimodal data as shown in the first aspect.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is executed on a computer, the computer implements the fatigue state monitoring method based on multimodal data as shown in the first aspect.

[0015] In the fifth aspect, the present application provides a fatigue status monitoring system based on multimodal data, including: an electronic device and a cloud server as shown in the third aspect; the cloud server is used to train or update a deep learning model, and send the trained or updated deep learning model to the electronic device.

[0016] In one possible implementation, the electronic device is used to upload the operator's physiological data, work data, and corresponding fatigue status to a cloud server; the cloud server is also used to generate charts and analysis reports based on the operator's physiological data, work data, and corresponding fatigue status.

[0017] Compared with the prior art, this application has at least the following technical effects: The present application provides a fatigue status monitoring method based on multimodal data, by obtaining the physiological data and work data of the operator, preprocessing the physiological data and work data, extracting indicators and data fusion of the preprocessed physiological data and work data, obtaining a fused feature vector, and inputting the fused feature vector into a trained deep learning model for fatigue status identification to obtain the fatigue status of the operator. In the fatigue status identification, the present application adds actual working environment factors such as working hours, work intensity, temperature and weather to improve the accuracy of the monitoring results. Combining physiological data and work data helps to comprehensively and accurately identify and monitor the fatigue status of operators. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of the structure of a fatigue status monitoring system based on multimodal data provided in an embodiment of the present application; Figure 2 A schematic diagram of a process flow of a fatigue state monitoring method based on multimodal data provided in an embodiment of the present application; Figure 3 A schematic diagram of a fatigue state recognition process of a deep learning model provided in an embodiment of the present application; Figure 4 A schematic diagram of the training process of the deep learning model provided in the embodiment of the present application; Figure 5 A schematic diagram of a process for adding a true label to each sample provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of a fatigue status monitoring device based on multimodal data provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In the embodiments of the present application, unless otherwise specified, the character " / " indicates that the objects before and after the association are in an or relationship. For example, A / B can represent A or B. "And / or" describes the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone.

[0020] It should be pointed out that the words "first", "second", etc. involved in the embodiments of the present application are only used to distinguish the description purpose, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated, nor can they be understood as indicating or implying order.

[0021] In the embodiments of the present application, "at least one" means one or more, and "more" means two or more. In addition, "at least one of the following" or similar expressions refers to any combination of these items, which may include any combination of single items or plural items. For example, at least one of A, B, or C may represent: A, B, C, A and B, A and C, B and C, or A, B and C. Among them, each of A, B, and C may be an element itself, or a set containing one or more elements.

[0022] In the embodiments of the present application, "exemplary", "in some embodiments", "in another embodiment", etc. are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a concrete way.

[0023] In the embodiments of the present application, "of", "corresponding", and "corresponding" can sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings to be expressed are consistent. In the embodiments of the present application, communication and transmission can sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings to be expressed are consistent. For example, transmission can include sending and / or receiving, which can be a noun or a verb.

[0024] The equal to involved in the embodiments of the present application can be used in conjunction with greater than, and is applicable to the technical solution adopted when greater than, and can also be used in conjunction with less than, and is applicable to the technical solution adopted when less than. It should be noted that when equal to is used in conjunction with greater than, it cannot be used in conjunction with less than; when equal to is used in conjunction with less than, it cannot be used in conjunction with greater than.

[0025] In related technologies, fatigue status monitoring of operators usually only focuses on a single physiological or psychological indicator, which makes it difficult to provide a comprehensive and accurate fatigue status assessment. In addition, most current monitoring equipment and systems fail to fully integrate multiple factors in the actual working environment, such as working hours, work intensity, temperature and weather conditions, which results in large errors in the monitoring results and makes it impossible to effectively identify and monitor the fatigue status of operators in a complex working environment.

[0026] Based on the above problems, the embodiment of the present application proposes a fatigue status monitoring method based on multimodal data, which helps to comprehensively and accurately identify and monitor the fatigue status of operators.

[0027] Now combined Figure 1-Figure 5 The fatigue status monitoring method based on multimodal data provided in an embodiment of the present application is described.

[0028] Figure 1 This is a schematic diagram of the structure of a fatigue status monitoring system based on multimodal data provided in an embodiment of the present application. Figure 1 As shown, the fatigue state monitoring system based on multimodal data includes a cloud server and an electronic device, and the electronic device and the cloud server can perform real-time data interaction. The number of electronic devices can be one or more. Figure 1 Only one electronic device is taken as an example, and the present application does not limit the number of electronic devices.

[0029] In this application, the cloud server is used to train or update the deep learning model, and send the trained or updated deep learning model to the electronic device. The cloud server uses a complex deep learning model to globally train and optimize the data, which is suitable for long-term trend analysis. In addition, the cloud server regularly updates the model and sends the updated model to the electronic device, so that the fatigue recognition ability of the electronic device is continuously optimized and the prediction accuracy is improved.

[0030] Electronic devices collect the physiological data and work data of workers through multimodal data sensors. For example, they collect the physiological data of workers in real time through electromyography sensors, pulse sensors (Photoplethysmography, PPG), breathing sensors, electrocardiogram sensors and electroencephalograms, record working time through timestamps, evaluate work intensity through accelerometers and other motion sensors, and record working environment temperature through temperature sensors. After preprocessing, index extraction and data fusion of the physiological data and work data collected in real time, the electronic devices input the processed data into the trained deep learning model for real-time identification and classification of fatigue status. Furthermore, electronic devices can provide early warning reminders based on the fatigue status of workers. By analyzing the real-time monitoring data of workers, electronic devices can identify fatigue status and provide feedback in a timely manner.

[0031] Optionally, the electronic device can temporarily store key data, such as detailed data fragments when an early warning occurs, for subsequent analysis. The electronic device can also upload the operator's physiological data and work data, as well as the corresponding fatigue status, to a cloud server via a wireless network (such as Wi-Fi, Long Term Evolution (LTE)). The cloud server stores, manages, and backs up the received data, such as storing the data in a cloud database (such as MySQL, MongoDB), implementing data cleaning and archiving strategies, and ensuring data integrity and security.

[0032] The cloud server can also generate charts and analysis reports based on the physiological data, work data, and corresponding fatigue status of the operators. For example, the cloud server can generate a detailed fatigue status analysis report based on the saved historical data, and can also generate trend charts, statistical charts, and warning records, etc., and then display real-time data, fatigue status, and warning information through a dashboard, or use a chart library (such as D3.js, Chart.js) and a data visualization platform (such as Tableau) for display.

[0033] Through the multimodal data sensors of electronic equipment, work environment information and personnel physiological data are collected and indicators are extracted to provide real-time and comprehensive work environment status and personnel data. Combined with real-time data visualization and historical data report presentation, it supports fatigue monitoring and early warning feedback of operators to ensure work efficiency and safety.

[0034] Optionally, the real-time data can be comprehensively analyzed on the cloud server to predict the future fatigue status of the operators.

[0035] Optionally, when an electronic device uploads data to a cloud server, an encrypted communication protocol (such as Hypertext Transfer Protocol Secure (HTTPS) and Message Queuing Telemetry Transport (MQTT)) can be used to ensure data security, and data can be transmitted in JSON or XML format.

[0036] Optionally, the electronic device may be a smart bracelet, a smart watch, a head-mounted device, or an embedded computing module, etc. The embodiment of the present application does not limit the type of the electronic device.

[0037] This application shortens the data processing chain, provides real-time feedback and warnings, and reduces latency by performing data processing and fatigue identification on electronic devices. Electronic devices are responsible for real-time data collection, preprocessing, and preliminary analysis, while cloud servers are responsible for deep learning model training and advanced data analysis, forming an efficient layered architecture. The cloud server continuously optimizes the model based on historical data and regularly updates the model, so that the system maintains a dynamic balance between real-time and accuracy.

[0038] Figure 2 The flowchart of the fatigue state monitoring method based on multimodal data provided in the embodiment of the present application specifically includes the following steps: Step S21, acquiring physiological data and work data of the operator, wherein the work data is used to characterize work data that affects the fatigue state of the operator.

[0039] In some embodiments, the work data includes working time, work intensity and work environment data, the work environment data includes working environment temperature and weather conditions; the physiological data includes electromyography (EMG), heart rate variability data (HRV), respiration rate data (RESP), electrocardiogram (ECG) and electroencephalogram (EEG).

[0040] Through multimodal data sensors, such as electromyography sensors, pulse sensors, respiratory sensors, electrocardiogram sensors and electroencephalograms, the physiological data of operators are collected in real time. These data include electromyography signals, heart rate variability data, respiratory rate data, electrocardiogram signals and electroencephalogram signals, etc. At the same time, work data including working hours, work intensity, temperature and weather conditions are monitored. Physiological data and work data are transmitted to electronic devices through wireless communication modules (such as Bluetooth and Wi-Fi).

[0041] Specifically, the electromyography sensor is used to collect electromyography signals. The electromyography sensor is attached to the surface of the muscle to collect electrical signals of muscle activity; the pulse sensor is used to collect heart rate variability data and obtains heart rate data through photoelectric volumetric pulse wave technology; the respiratory sensor is used to collect respiratory frequency data. The respiratory sensor is worn on the chest or abdomen to record the breathing condition in real time; the electrocardiogram sensor is used to collect electrocardiogram signals. The electrocardiogram sensor is attached to the chest to record the electrical activity of the heart; the electroencephalograph is used to collect electroencephalogram signals. The electroencephalograph is worn on the head to record the electrical activity emitted by brain neurons.

[0042] The collection of work data can specifically include: recording the start and end time of work through timestamps; using accelerometers and other motion sensors to evaluate work intensity; using temperature sensors to record ambient temperature; and obtaining weather conditions through weather stations or related equipment.

[0043] In other embodiments, physiological data may also include blood pressure, blood oxygen content, electrodermal activity (EDA), etc., work data may also include work category and conditions, work speed, etc., and work environment data may also include lighting conditions, noise, dust and air quality, etc.

[0044] The fatigue state of the operator is not only related to a single physiological data, but also to other physiological data as well as the worker's working hours, work intensity, working environment, etc. Compared with the method of using single-dimensional physiological data (such as skin electricity or electrocardiography or electromyography) to monitor the fatigue state of the operator, this application proposes to monitor the fatigue state based on physiological data and work data. On the one hand, actual working environment factors such as working hours, work intensity, temperature and weather are added. On the other hand, multimodal physiological data is added, including electromyography, electrocardiography, electroencephalography, respiratory rate, etc., and fatigue state is identified based on the collected multimodal data, which helps to comprehensively and accurately identify and monitor the fatigue state of the operator and improve the accuracy of the monitoring results.

[0045] Step S22, pre-processing the physiological data and the working data.

[0046] In some embodiments, the preprocessing includes: performing unified sampling rate processing on the physiological data and the working data; and dividing the physiological data and the working data processed at the unified sampling rate into N segments of data using a sliding window technology.

[0047] When the data sampling rates of different sensors are inconsistent, upsampling or downsampling technology is used to unify all data to the same sampling rate to ensure data consistency. Specifically, the upsampling formula and downsampling formula are used to unify the data sampling rate. The upsampling formula and downsampling formula are as follows: Upsampling formula: (Interpolation method) Downsampling formula:

[0048] Among them, x is the original signal, y is the upsampled or downsampled signal, L is the sampling factor, and n is the time domain subscript.

[0049] Next, according to the actual working hours of the operators, the physiological data and work data processed at the unified sampling rate are segmented according to time using the sliding window technology, and divided into N segments of data. Specifically, the window size and step size are set, such as setting the window size to 10 minutes and the step size to 1 minute, and dividing the data into N 10-minute groups of data.

[0050] In other optional embodiments, preprocessing may also include data cleaning and artifact removal. Data cleaning includes processing missing values ​​and outliers. For missing values, a filling method may be used, such as using the mean, median or mode, etc.; for outliers, a deletion or replacement method may be used. Artifact removal includes using an independent component analysis algorithm to separate independent signal sources from multivariate signals to remove electrooculographic artifacts, etc.

[0051] Step S23, extracting indicators and fusing data on the preprocessed physiological data and working data to obtain a fused feature vector.

[0052] Specifically, the indicators related to the fatigue state of the operator are extracted from the pre-processed physiological data and work data, including physiological data indicators and work data indicators.

[0053] Optionally, the physiological data indicators include time domain indicators and frequency domain indicators, the time domain indicators include at least the mean, maximum, minimum, variance and standard deviation, and the frequency domain indicators include at least the total power, average power, median power and power peak; the work data indicators include at least the working time, work intensity level, the average, maximum and minimum values ​​of the working environment temperature, and the weather classification.

[0054] For example, if the physiological data is an electromyographic signal, the extracted physiological data indicators include the time domain indicators and frequency domain indicators of the electromyographic signal. The time domain indicators include mean, maximum, minimum, variance, root mean square (RMS), integrated electromyogram (iEMG), mean absolute value, standard deviation, etc. The frequency domain indicators include median frequency, mean frequency, etc.

[0055] The physiological data is HRV, and the extracted physiological data indicators include time domain indicators and frequency domain indicators of HRV. The time domain indicators include average heart rate (AVHR), standard deviation (SDNN), root mean square deviation (RMSSD), etc. The frequency domain indicators include total power (Total power), high frequency power (HF power), low frequency power (LF power), low frequency / high frequency ratio (LF / HF), etc.

[0056] The physiological data is respiratory rate data, and the extracted physiological data indicators include time domain indicators and frequency domain indicators of respiratory rate data, among which the time domain indicators include average respiratory rate (AVRESP), standard deviation (Std), maximum value (Max), minimum value (Min), range (Range), etc., and the frequency domain indicators include power (Power), peak value (Peak), etc.

[0057] The physiological data is an electroencephalogram signal, and the extracted physiological data indicators include the time domain indicators and frequency domain indicators of the electroencephalogram signal. The time domain indicators include mean, maximum, minimum, etc., and the frequency domain indicators include α wave power, β wave power, γ wave power, etc.

[0058] Work data indicators include working hours calculated based on the start and end time of work, the intensity of exercise obtained by monitoring workers using motion capture, accelerometers, gyroscopes and other motion sensors, the average, maximum and minimum values ​​of the working environment temperature obtained by real-time monitoring using temperature sensors, and weather classification (such as sunny, rainy, cloudy, etc.) obtained using weather station data or third-party application programming interfaces (APIs). Work data indicators can also include environmental parameters such as humidity, wind speed, and rainfall.

[0059] In some embodiments, physiological data indicators and work data indicators are extracted based on the data of each segment respectively to obtain N feature vectors, each feature vector includes the physiological data indicators and work data indicators of the corresponding segment, and the physiological data indicators and work data indicators are used to characterize indicators related to the fatigue state of the operator; the N feature vectors are fused to obtain a fused feature vector.

[0060] Specifically, the physiological data index and the working data index are extracted from the data of each segment (including physiological data and working data) to obtain N feature vectors. Each feature vector includes the physiological data index and the working data index of the corresponding segment. The feature vector is expressed as follows: X t =[EMG t ,HRV t ,RESP t ,ECG t ,EEG t ,WorkTime t ,WorkIntensity t ,Temperature t ,Weather t ] Wherein, t represents the time segment number, indicating the t-th segment, and the value range of t is 1-N.

[0061] For example, when t=4, the eigenvector is expressed as: X4=[EMG4,HRV4,RESP4,ECG4,EEG4,WorkTime4,WorkIntensity4,Temperature4,Weather4] The feature vector includes the physiological data index and the work data index of the corresponding fourth segment.

[0062] Next, the N feature vectors are fused to obtain a fused feature vector. The fused feature is expressed as: P = [X1, X2, …, X N ].

[0063] It is understandable that the physiological data indicators and work data indicators contained in the feature vector are only for illustrative purposes, and other physiological data indicators and work data may also be included, for example, indicators extracted from blood pressure, blood oxygen content, skin electrical activity, lighting conditions, etc.

[0064] The original physiological data and work data contain information that is irrelevant to the fatigue state. By extracting indicators, the dimension of the data can be reduced, the complexity of subsequent processing can be reduced, and the most important information can be retained. It is also helpful to extract the features that best represent the fatigue state from the complex physiological data and work data, thereby improving the accuracy and efficiency of fatigue state identification. Furthermore, different types of data (physiological data and work data) are integrated into a unified data structure (i.e., fused feature vector) for easy analysis and processing.

[0065] Step S24: input the fused feature vector into a trained deep learning model to perform fatigue state recognition to obtain the fatigue state of the operator.

[0066] Among them, the trained deep learning model is composed of a convolutional neural network (CNN) and a long short-term memory network (LSTM), such as Figure 3 As shown, Figure 3 A schematic diagram of the fatigue state recognition process of the deep learning model provided in the embodiment of the present application, wherein the convolutional neural network includes a convolutional layer, a pooling layer and a first fully connected layer ( Figure 3 The CNN fully connected layer in the CNN), the long short-term memory network includes the long short-term memory layer and the second fully connected layer ( Figure 3 The fused feature vector P is input into the convolutional layer to extract local features, then into the pooling layer to reduce the data dimension, then into the first fully connected layer to integrate features and output the first feature vector. The first feature vector output by the CNN network is input into the long short-term memory layer to process the time series features, capture the long-term dependency in the first feature vector, and then into the second fully connected layer to process and output the classification result of the fatigue state.

[0067] The fatigue state recognition model based on the CNN+LSTM deep learning algorithm can effectively classify and predict fatigue caused by load accumulation.

[0068] In some embodiments, Figure 4 A schematic diagram of the training process of the deep learning model provided in the embodiment of the present application, such as Figure 4 As shown in Figure 1, the training process of the deep learning model specifically includes the following steps: Step S41, constructing a training sample set, wherein the training sample set includes a plurality of samples, each sample including collected physiological data and working data.

[0069] Specifically, a multimodal data sensor is used to collect physiological data and work data of multiple operators as data of multiple samples. The specific implementation process is shown in step S21, which will not be described in detail here.

[0070] Step S42, preprocessing the physiological data and working data of each sample in the training sample set.

[0071] Specifically, the data collected by different sensors are processed at a unified sampling rate, and the data processed at the unified sampling rate is segmented according to time using a sliding window technology, or data cleaning and artifact removal are performed. For details, refer to step S22 and will not be described here.

[0072] Step S43, performing index extraction and data fusion on the preprocessed physiological data and working data of each sample to obtain a fused feature vector set.

[0073] For each sample, the physiological data and working data are subjected to index extraction and data fusion to obtain the fused feature vector of the sample. The specific implementation process is shown in step S23 and will not be described in detail here. The fused feature vector set is composed of the fused feature vectors corresponding to the multiple samples.

[0074] Step S44, adding a true label to each sample based on the fused feature vector set.

[0075] Specifically, Figure 5 As shown, Figure 5 The flowchart of adding a true label to each sample provided in the embodiment of the present application, that is, step S44 specifically includes the following steps: Step S441, using the Bayesian Information Criterion algorithm based on the fused feature vector set to calculate the number k of fatigue state levels.

[0076] Specifically, the Bayesian Information Criterion (BIC) algorithm is used to calculate the number of levels of fatigue status. BIC is used for model selection to evaluate the applicability of different cluster numbers. The specific steps are as follows: Input: A fused feature vector set containing physiological data indicators and work data indicators.

[0077] The process includes model selection: maximum likelihood estimation (L) for each possible number of clusters (k); and calculation of BIC values: for each k, the corresponding BIC value is calculated.

[0078] Specifically, 1. Calculate the maximum likelihood estimate (L): Perform the maximum likelihood estimate for each possible number of clusters (k).

[0079] 2. Calculate the BIC value: For each k, calculate the corresponding BIC value.

[0080]

[0081] Among them, L is the maximum likelihood estimate, k is the number of model parameters, and N is the number of samples.

[0082] 3. Select the optimal model: Select k with the smallest BIC value as the optimal number of clusters.

[0083] Output: The optimal number of clusters k, that is, the number of levels k of fatigue status.

[0084] Step S442: Based on the number k of the fatigue state levels, a time series segmentation clustering algorithm is used to perform data classification and labeling on the fused feature vector set to obtain a true label corresponding to each sample.

[0085] Specifically, the time series segmentation clustering algorithm (Toeplitz Inverse Covariance-Based Clustering, TICC) is used to cluster and label the data in the fusion feature vector set, and the data is divided into k types of fatigue levels, so as to obtain the true label corresponding to each sample.

[0086] Input: the number of fatigue state levels k, the fused feature vector set.

[0087] process: 1. Data segmentation: The time series data in the fused feature vector set is segmented into fixed lengths (for example, every 10 minutes is a segment).

[0088] 2. Selection of initial cluster centers: Randomly select the initial cluster centers or use other methods to determine the initial centers.

[0089] 3. EM algorithm iterative update: Step E: Calculate the probability that each data segment belongs to each cluster.

[0090] Step M: Update the cluster center to minimize the distance between the segmented data and the cluster center.

[0091] 4. Iterative convergence: Repeat steps E and M until the clustering results converge.

[0092] Output: Clustering result, which divides the data into k fatigue levels, so as to obtain the true label corresponding to each sample.

[0093] For example, according to the BIC algorithm, k=4 is obtained, that is, the fatigue levels are divided into four categories (such as no fatigue, low fatigue, medium fatigue, and high fatigue). Then, according to the TICC algorithm, the data in the fused feature vector set are clustered and labeled, and the data in the fused feature vector set are assigned to four different fatigue levels.

[0094] The BIC algorithm is used to determine the number of clusters of fatigue states, and the TICC algorithm is used to classify fatigue levels. The optimal number of clusters k calculated by the BIC algorithm helps the TICC algorithm determine the most appropriate number of clusters. This k value is the result of the best balance between model complexity and data fitting, which helps the TICC algorithm to cluster and label data more accurately, thereby more effectively identifying and classifying different fatigue levels. The combination of the BIC algorithm and the TICC algorithm can improve the accuracy and reliability of cluster labeling.

[0095] Step S45, constructing a deep learning model using a convolutional neural network and a long short-term memory network.

[0096] Specifically, the convolutional neural network includes a convolutional layer, a pooling layer and a first fully connected layer, and the long short-term memory network includes a long short-term memory layer and a second fully connected layer; the convolutional layer is used to extract local features, the pooling layer is used to reduce data dimensions, the first fully connected layer is used to integrate features and output a first feature vector, and the long short-term memory layer is used to capture long-term dependencies in the first feature vector; the second fully connected layer is used to output the prediction results of the fatigue state.

[0097] Step S46: Use the fused feature vector set to train the deep learning model to obtain a prediction result of the fatigue state of each sample.

[0098] Specifically, the fused feature vector set (each sample has a true label) is input into the deep learning model for training. The CNN processing steps include: 1. Input the fused feature vector set (including preprocessed data).

[0099] 2. Use convolutional layers to extract local features.

[0100] Convolutional layer formula:

[0101] Among them, x is the input data, w is the convolution kernel, b is the bias term, and y is the output feature map.

[0102] 3. Use pooling layers to reduce data dimensions.

[0103] Pooling layer formula:

[0104] 4. Use the fully connected layer to output the feature vector.

[0105] Fully connected layer formula:

[0106] Among them, W is the weight matrix, x is the input vector, y is the output vector, b is the bias term, and ReLU is the activation function.

[0107] LSTM processing steps: 1. Input the feature vector output by CNN.

[0108] 2. Use the LSTM layer to process time series features and capture long-term dependencies.

[0109] Among them, the LSTM layer is composed of multiple LSTM units, each LSTM unit is responsible for processing the data at a time point in the sequence. The calculation formula of the LSTM unit is as follows:

[0110] Among them, f t It is the forget gate, t Input gate, is the candidate memory, C t is a memory unit, o t is the output gate, h t is a hidden state.

[0111] 3. Use the fully connected layer to output the classification results of fatigue status.

[0112] Fully connected layer formula:

[0113] Where W is the weight matrix, h t is the hidden state of LSTM, b is the bias term, and Softmax is the activation function for multi-classification problems.

[0114] Step S47, calculating the loss between the predicted result of the fatigue state of each sample and the corresponding true label, and using the loss to adjust the parameters of the deep learning model.

[0115] Specifically, the deep learning model is trained using the fused feature vector set to obtain the predicted result of the fatigue state of each sample, the loss between the fatigue state result and the true label is calculated, and the parameters of the deep learning model are adjusted using the loss to finally obtain a trained deep learning model.

[0116] In some embodiments, the fatigue status monitoring method based on multimodal data provided by the present application further includes: providing an early warning reminder based on the fatigue status of the operator, and the early warning reminder includes a vibration reminder, and / or a sound reminder, and / or a visual reminder. Among them, the vibration reminder can be issued by a wearable device; the sound reminder can be issued by headphones or speakers; and the visual reminder can be issued by a display screen or an LED light on the wearable device.

[0117] Optionally, the fatigue status includes no fatigue, low fatigue, medium fatigue and high fatigue. When the fatigue status of the operator is medium fatigue or high fatigue, an early warning reminder is issued to the operator. By analyzing the real-time monitoring data of the operator, the fatigue status is timely identified. When the system detects the "medium fatigue" or "high fatigue" status, an alarm is immediately triggered and a timely warning is issued. Prevent the operator from reducing work efficiency, reducing work quality or increasing accident risks due to excessive fatigue.

[0118] Optionally, the fatigue state can be divided into first-level fatigue, second-level fatigue, third-level fatigue, fourth-level fatigue, fifth-level fatigue, etc. according to the degree of fatigue, wherein the first-level fatigue corresponds to no fatigue. Those skilled in the art can classify according to actual conditions, and this application does not limit it.

[0119] Furthermore, the system can also provide feedback on the response time and respond to changes in fatigue status in real time, keeping the delay at the millisecond level to ensure that operators are reminded in a timely manner.

[0120] This application obtains the physiological data and work data of the operator, pre-processes the physiological data and work data, extracts indicators and fuses the pre-processed physiological data and work data to obtain a fused feature vector, and inputs the fused feature vector into a trained deep learning model to identify the fatigue state and obtain the fatigue state of the operator. In the fatigue state identification, this application adds actual working environment factors such as working hours, work intensity, temperature and weather to improve the accuracy of the monitoring results. Combining physiological data and work data helps to comprehensively and accurately identify and monitor the fatigue state of the operator and improve the work safety of the operator.

[0121] In power grid emergency maintenance tasks, operators often face long hours of high-intensity work, which may lead to fatigue, thereby affecting work efficiency and safety. The following uses monitoring the fatigue state of power grid maintenance personnel as an example to illustrate the method provided in this application.

[0122] 1. Collect physiological data and work data Power grid maintenance personnel wear electronic terminals (such as smart bracelets) to collect data such as heart rate, electromyography signals, ambient temperature, and work intensity data.

[0123] For example, the collection of physiological data includes: attaching electromyographic sensors to the forearm muscles of maintenance personnel to collect electromyographic signals (EMG) in real time. The strength of the electromyographic signals can reflect the degree of muscle fatigue in the arm. Installing a pulse sensor on the wrist to measure heart rate variability (HRV) through photoelectric volume pulse wave technology, the fluctuation of heart rate can provide an indirect indicator of fatigue. Wearing a respiratory sensor on the chest to record the respiratory rate (RESP), abnormal fluctuations in respiratory rate may be related to fatigue. Attaching an electrocardiogram sensor to the chest to record electrocardiogram signals (ECG), changes in heart rate can provide further signs of fatigue. Wearing an electroencephalogram on the head to record electroencephalogram signals (EEG), changes in alpha waves and beta waves can reflect the degree of attention and fatigue in the working state.

[0124] The collection of work data includes real-time monitoring of working hours, work intensity, ambient temperature and weather conditions. Each collection indicator is acquired through special sensors and equipment.

[0125] Specifically, working time can be collected using a time recording device (or automatically recorded by the operating system). The timer starts when the recorder starts the work and stops when the work is completed, and the total working time is recorded. The collected data type is timestamp data in the format of yyyy-mm-dd hh:mm:ss. An example of recorded working time: start time: 2024-09-01 08:00:00, end time: 2024-09-01 14:30:00, working time: 6.5 hours.

[0126] Using motion capture, wearable accelerometers, gyroscopes and other motion sensors to monitor the operator's exercise intensity, the system collects the acceleration changes in the maintenance personnel's body movements in real time, especially the activity intensity of the hands or torso during operation, and integrates the acceleration signal to obtain the work intensity. The recorded data type is three-axis acceleration data (unit: m / s²). The calculation formula for work intensity is as follows:

[0127] Where I represents the intensity of exercise and a(t) represents the function of acceleration changing with time.

[0128] Examples of recorded data: Average acceleration: 0.8 m / s²; Maximum acceleration: 2.5 m / s²; Work intensity level: Moderate.

[0129] The system uses temperature sensors (such as thermistors, digital temperature sensors) to monitor the ambient temperature in real time and records the temperature changes of the working environment regularly (such as every minute). The unit of temperature data is degrees Celsius (℃). Examples of recorded temperature data are current temperature: 30℃; minimum temperature: 28℃; maximum temperature: 32℃; average temperature: 30.5℃.

[0130] The system obtains local weather data of multiple categories regularly (e.g., every hour) through external meteorological equipment (such as weather stations) or third-party APIs, such as humidity (%), rainfall (mm), wind speed (m / s), and weather conditions (sunny, rainy, cloudy, etc.). An example of weather data is the current weather: sunny; humidity: 60%; rainfall: 0mm; wind speed: 5m / s.

[0131] 2. Data preprocessing Data preprocessing includes: unified data sampling rate and time series segmentation. Among them, unified data sampling rate is to process data through linear interpolation (up sampling or down sampling) or timestamp technology to unify different sensor data to the same sampling rate. For example, the temperature sensor records data once a minute, and the motion sensor records data once a second. Up sampling or down sampling is used to ensure data synchronization. Time series segmentation uses sliding window technology to segment all data by time. For example, if the window size is set to 10 minutes and the step size is 1 minute, the data from 10:00 to 10:10 is regarded as a time segment, and the physiological data and work data in this time period are regarded as a segment of data.

[0132] 3. Data indicator extraction and data fusion Specifically, extract the time domain indicators and frequency domain indicators of physiological data (including electromyographic signals, heart rate variability data, respiratory rate data, electrocardiogram signals and electroencephalogram signals). For the specific implementation steps and technical principles, please refer to the description of the above embodiments, which will not be repeated here. According to the work data (including working hours, work intensity, working environment temperature, weather conditions), extract the working hours, work intensity level, average value, maximum value and minimum value of the working environment temperature, and weather classification. Among them, the total working time is calculated by the start and end timestamps (working time = end time - start time); the average acceleration and maximum acceleration of the acceleration sensor are used to judge the work intensity; the average value, maximum value and minimum value of the ambient temperature are extracted according to the recorded working environment temperature; the weather information of the current time period is extracted using the weather station data or the weather conditions obtained by the third-party API, such as sunny, rainy and other classification variables, and the humidity, wind speed, etc. are recorded.

[0133] The extracted physiological data indicators and work data indicators are fused to obtain a fused feature vector.

[0134] 4. Input the fused feature vector into the trained deep learning model for fatigue state recognition to obtain the fatigue state of the power grid maintenance personnel.

[0135] The specific implementation steps and technical principles can be found in the description of the above embodiments and will not be elaborated here.

[0136] Furthermore, the work data collected in real time can be displayed through a dashboard or monitoring system. For example, the working time is displayed through a progress bar, such as the current working time is displayed as 4 hours and 30 minutes; the work intensity is displayed through an accelerometer data chart, such as the acceleration change curve on the timeline; the temperature is displayed through a thermometer icon to display the current temperature and its changing trend; weather information is displayed through icons to display the current weather (such as sunny, cloudy, etc.) and specific data (such as humidity, wind speed, etc.).

[0137] All work data can also be further summarized into tables or reports for subsequent analysis and recording. For example, the work time record table: displays the start and end time of each time period; the work intensity report: displays the average acceleration, maximum acceleration, and determines the work intensity level according to the predetermined standards; the temperature and weather change table: displays the change curves of environmental parameters such as temperature, humidity, and wind speed to assist in analyzing their impact on fatigue.

[0138] Furthermore, when the electronic device detects that the fatigue state reaches a warning threshold (such as medium fatigue or high fatigue state), a real-time reminder is given through vibration, sound or visual signals, and the alarm information is transmitted to the cloud server for recording and subsequent analysis.

[0139] Based on the same technical principle, the present application also provides a fatigue status monitoring device based on multimodal data, such as Figure 6 As shown, the fatigue state monitoring device 60 based on multimodal data includes: An acquisition module 61 is used to acquire physiological data and work data of the operator, where the work data is used to characterize work data that affects the fatigue state of the operator; A preprocessing module 62, for preprocessing physiological data and working data; The index extraction and data fusion module 63 is used to extract the index and fuse the data of the pre-processed physiological data and working data to obtain a fusion feature vector; The state recognition module 64 is used to input the fused feature vector into the trained deep learning model for fatigue state recognition to obtain the fatigue state of the operator.

[0140] In one possible implementation, the work data includes working hours, work intensity and work environment data, and the work environment data includes working environment temperature and weather conditions; the physiological data includes electromyography signals, heart rate variability data, respiratory rate data, electrocardiogram signals and electroencephalogram signals.

[0141] In one possible implementation, the preprocessing module 62 is further used to: perform unified sampling rate processing on the physiological data and the working data; and divide the physiological data and the working data processed at the unified sampling rate into N segments of data using a sliding window technology.

[0142] In one possible implementation, the indicator extraction and data fusion module 63 is also used to: extract physiological data indicators and work data indicators based on the data of each segment, respectively, to obtain N feature vectors, each feature vector includes the physiological data indicators and work data indicators of the corresponding segment, the physiological data indicators and work data indicators are used to characterize indicators related to the fatigue state of the operator; perform data fusion on the N feature vectors to obtain a fused feature vector.

[0143] In one possible implementation, the training of the deep learning model includes: constructing a training sample set, the training sample set includes multiple samples, each sample includes collected physiological data and work data; preprocessing the physiological data and work data of each sample in the training sample set; extracting indicators and fusing data for the physiological data and work data of each sample after preprocessing to obtain a fused feature vector set; adding a true label to each sample based on the fused feature vector set; constructing a deep learning model using a convolutional neural network and a long short-term memory network, the convolutional neural network includes a convolutional layer, a pooling layer and a first fully connected layer, and the long short-term memory network includes a long short-term memory layer and a second fully connected layer; the convolutional layer is used to extract local features, the pooling layer is used to reduce data dimensions, the first fully connected layer is used to integrate features and output a first feature vector, and the long short-term memory layer is used to capture long-term dependencies in the first feature vector; the second fully connected layer is used to output a prediction result of a fatigue state; training the deep learning model using the fused feature vector set to obtain a prediction result of a fatigue state for each sample; calculating the loss between the prediction result of the fatigue state of each sample and the corresponding true label, and using the loss to adjust the parameters of the deep learning model.

[0144] In one possible implementation, a true label is added to each sample based on the fused feature vector set, including: using the Bayesian Information Criterion algorithm based on the fused feature vector set to calculate the number of levels k of fatigue status; and using the time series segmentation and clustering algorithm based on the number of levels k of fatigue status to classify and label the fused feature vector set to obtain the true label corresponding to each sample.

[0145] In one possible implementation, the device 60 further includes: an early warning module for providing early warning reminders based on the fatigue status of the operator, wherein the early warning reminders include vibration reminders, and / or sound reminders, and / or visual reminders.

[0146] In one possible implementation, the fatigue status includes no fatigue, low fatigue, medium fatigue and high fatigue, and the early warning module is further used to send an early warning reminder to the operator when the fatigue status of the operator is medium fatigue or high fatigue.

[0147] In one possible implementation, the physiological data indicators include time domain indicators and frequency domain indicators. The time domain indicators include at least the average value, maximum value, minimum value, variance and standard deviation, and the frequency domain indicators include at least the total power, average power, median power and power peak; the work data indicators include at least the working time, work intensity level, the average value, maximum value and minimum value of the working environment temperature, and the weather classification.

[0148] Figure 6 The fatigue state monitoring device 60 based on multimodal data provided in the illustrated embodiment can be used to execute the technical solution of the method embodiment shown in the present application, and its implementation principle and technical effect can be further referred to the relevant description in the method embodiment. The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when it is run on a computer, the computer executes the fatigue state monitoring method based on multimodal data provided in the embodiment shown in the present application.

[0149] Combine the following Figure 7 The exemplary electronic device provided in the embodiments of the present application is further introduced. Figure 7 A schematic structural diagram of an electronic device 7000 is shown.

[0150] The electronic device 7000 may include: at least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the fatigue status monitoring method based on multimodal data provided in the embodiment shown in the present application.

[0151] Figure 7 A block diagram of an exemplary electronic device 7000 suitable for implementing the embodiments of the present application is shown. Figure 7 The electronic device 7000 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0152] like Figure 7As shown, the electronic device 7000 is in the form of a general computing device. The components of the electronic device 7000 may include, but are not limited to: one or more processors 7010, a memory 7020, a communication bus 7040 connecting different system components (including the memory 7020 and the processor 7010), and a communication interface 7030.

[0153] Communication bus 7040 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnection (PCI) bus.

[0154] The electronic device 7000 typically includes a variety of computer system readable media, which can be any available media that can be accessed by the electronic device, including volatile and nonvolatile media, removable and non-removable media.

[0155] The memory 7020 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Figure 7Not shown, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a compact disc read only memory (CD-ROM), a digital versatile disc read only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to the communication bus 7040 via one or more data medium interfaces. The memory 7020 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0156] A program / utility having a set (at least one) of program modules may be stored in the memory 7020, such program modules including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules generally perform the functions and / or methods of the embodiments described herein.

[0157] The electronic device 7000 may also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via the communication interface 7030. Furthermore, the electronic device 7000 may also communicate with the network adapter ( Figure 7 The network adapter may communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via the communication bus 7040. It should be understood that although Figure 7 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 7000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Drives; hereinafter referred to as: RAID) systems, tape drives, and data backup storage systems.

[0158] The processor 7010 executes various functional applications and data processing by running the programs stored in the memory 7020, such as implementing the method provided in the embodiment of the present application.

[0159] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 7000. In other embodiments of the present application, the electronic device 7000 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0160] In the above embodiments, the processor involved may include, for example, a CPU, a DSP, a microcontroller or a digital signal processor, and may also include a GPU, an embedded neural network processor (Neural-network Process Units; hereinafter referred to as: NPU) and an image signal processor (Image Signal Processing; hereinafter referred to as: ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASIC, or one or more integrated circuits for controlling the execution of the program of the technical solution of the present application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.

[0161] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0163] In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk, and other media that can store program codes.

[0164] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A fatigue state monitoring method based on multimodal data, characterized in that: The method comprises: Acquiring physiological data and work data of an operator, wherein the work data is used to characterize work data that affects a fatigue state of the operator; Preprocessing the physiological data and the working data; Extract indicators and fuse data from preprocessed physiological data and working data to obtain fused feature vectors; The fused feature vector is input into a trained deep learning model to perform fatigue state recognition to obtain the fatigue state of the operator.

2. The method according to claim 1, characterized in that The working data includes working time, working intensity and working environment data, and the working environment data includes working environment temperature and weather conditions; The physiological data includes electromyographic signals, heart rate variability data, respiratory rate data, electrocardiographic signals and electroencephalographic signals.

3. The method according to claim 1, characterized in that The preprocessing of the physiological data and the working data includes: Performing unified sampling rate processing on the physiological data and the working data; The sliding window technology is used to divide the physiological data and working data processed at a uniform sampling rate into N segments of data.

4. The method according to claim 3, characterized in that The index extraction and data fusion of the pre-processed physiological data and working data include: Extracting physiological data indicators and work data indicators based on the data of each segment respectively to obtain N feature vectors, each feature vector including the physiological data indicators and work data indicators of the corresponding segment, the physiological data indicators and the work data indicators being used to characterize indicators related to the fatigue state of the operator; The N feature vectors are subjected to data fusion to obtain the fused feature vector.

5. The method according to claim 1, characterized in that The training of the deep learning model includes: Constructing a training sample set, wherein the training sample set includes a plurality of samples, each sample including collected physiological data and working data; Preprocessing the physiological data and working data of each sample in the training sample set; Perform index extraction and data fusion on the physiological data and working data of each sample after preprocessing to obtain a fusion feature vector set; Adding a true label to each sample based on the fused feature vector set; A deep learning model is constructed using a convolutional neural network and a long short-term memory network, wherein the convolutional neural network includes a convolutional layer, a pooling layer and a first fully connected layer, and the long short-term memory network includes a long short-term memory layer and a second fully connected layer; the convolutional layer is used to extract local features, the pooling layer is used to reduce data dimensions, the first fully connected layer is used to integrate features and output a first feature vector, and the long short-term memory layer is used to capture long-term dependencies in the first feature vector; the second fully connected layer is used to output a prediction result of a fatigue state; The deep learning model is trained using the fused feature vector set to obtain a prediction result of the fatigue state of each sample; The loss between the predicted result of the fatigue state of each sample and the corresponding true label is calculated, and the parameters of the deep learning model are adjusted using the loss.

6. The method according to claim 5, characterized in that The adding a true label to each sample based on the fused feature vector set comprises: The number k of levels of fatigue state is calculated based on the fused feature vector set using a Bayesian information criterion algorithm; Based on the number k of levels of the fatigue state, a time series segmentation clustering algorithm is used to perform data classification and labeling on the fused feature vector set to obtain a true label corresponding to each sample.

7. The method according to claim 1, characterized in that The method further comprises: An early warning reminder is provided based on the fatigue state of the operator, and the early warning reminder includes a vibration reminder, and / or a sound reminder, and / or a visual reminder.

8. The method according to claim 7, characterized in that The fatigue status includes no fatigue, low fatigue, medium fatigue and high fatigue. The reminder and warning based on the fatigue status of the operator includes: When the fatigue state of the operator is medium fatigue or high fatigue, an early warning reminder is issued to the operator.

9. The method according to claim 4, characterized in that The physiological data indicators include time domain indicators and frequency domain indicators, the time domain indicators at least include mean value, maximum value, minimum value, variance and standard deviation, and the frequency domain indicators at least include total power, average power, median power and power peak; The work data indicators include at least working hours, work intensity level, average value, maximum value and minimum value of working environment temperature, and weather classification.

10. A fatigue status monitoring device based on multimodal data, characterized in that: The device comprises: An acquisition module, used for acquiring physiological data and work data of an operator, wherein the work data is used for representing work data that affects a fatigue state of the operator; A preprocessing module, used for preprocessing the physiological data and the working data; The index extraction and data fusion module is used to extract the index and fuse the data of the preprocessed physiological data and working data to obtain the fusion feature vector; The state recognition module is used to input the fused feature vector into a trained deep learning model to perform fatigue state recognition to obtain the fatigue state of the operator.

11. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store a computer program; the processor is used to run the computer program to implement the fatigue state monitoring method based on multimodal data as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a computer, the fatigue state monitoring method based on multimodal data as described in any one of claims 1 to 9 is implemented.

13. A fatigue status monitoring system based on multimodal data, characterized in that: comprising the electronic device and the cloud server as claimed in claim 11; The cloud server is used to train or update the deep learning model, and send the trained or updated deep learning model to the electronic device.

14. The system according to claim 13, characterized in that The electronic device is used to upload the physiological data and work data of the operator, and the corresponding fatigue status to the cloud server; The cloud server is also used to generate charts and analysis reports based on the operator's physiological data, work data, and corresponding fatigue status.

Images