Method, system, edge computing terminal and medium for monitoring health status of cabin workers based on multimodal data

Abstract

The present invention provides a method, system, edge computing terminal and medium for monitoring the health status of cabin workers based on multimodal data. The method includes: obtaining multimodal physiological data of cabin workers and environmental data in the cabin collected in real time by multimodal data sensors; performing data preprocessing including data cleaning and data standardization on the multimodal physiological data and environmental data; extracting physiological index features from the preprocessed multimodal physiological data, and extracting environmental index features from the preprocessed environmental data; using the extracted physiological index features and environmental index features as input data of a pre-trained deep learning network model to obtain a real-time predicted output of health status monitoring results; uploading the collected multimodal physiological data and environmental data to a cloud server, and receiving updates to the deep learning network model from the cloud server. The present invention can realize the identification and monitoring of health status in a classified prediction manner.

Description

Technical Field

[0001] The present invention relates to the field of intelligent health status monitoring technology, and in particular to a method, system, edge computing terminal and medium for monitoring the health status of cabin workers based on multimodal data. Background Art

[0002] Cabin types include spacecraft cabins, ship cabins, and aviation cabins. The cabin environment is often relatively closed and may contain a variety of potential risk factors, and the health status of operators directly affects their work efficiency. By identifying the health status of operators, we can understand whether the cabin environment is normal and whether the working status of operators in the cabin is normal. Therefore, the health status identification technology of operators in the cabin environment is of great importance.

[0003] In the prior art, a cabin environment control method and system customized for multiple seafarers are proposed, and the cabin environment can be controlled by a combination of control parameters. The prior art includes the following steps: obtaining multiple temperature adjustment threshold sub-values, humidity adjustment threshold sub-values, and wind speed adjustment threshold sub-values; arbitrarily combining each temperature adjustment threshold sub-value, humidity adjustment threshold sub-value, and wind speed adjustment threshold sub-value to obtain multiple cabin environment control parameter combinations; regulating the cabin environment, monitoring the overall average physical signs of all seafarers corresponding to each cabin environment control parameter combination within a preset time period; selecting the cabin environment control parameter combination corresponding to the minimum total error of the physical signs as the optimal cabin environment control parameter combination.

[0004] However, in the existing technology, cabin environment monitoring technology mainly relies on a single environmental sensor (for example, temperature and humidity sensor), which cannot fully monitor the health status of operators, lacks real-time feedback, and does not consider the influence of multiple environmental factors. In addition, most of the existing monitoring equipment and systems fail to fully combine multiple factors in the actual working environment, such as working hours, work intensity, temperature and weather conditions, which makes the monitoring results have large errors and cannot effectively identify and monitor the health status of operators in a complex working environment.

[0005] Therefore, how to provide a method for identifying the health status of workers in a cabin environment and the corresponding hardware entity is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a method, system, edge computing terminal and medium for monitoring the health status of cabin workers based on multimodal data to eliminate or improve one or more defects existing in the prior art.

[0007] One aspect of the present invention provides a method for monitoring the health status of cabin workers based on multimodal data, which is implemented using an edge computing terminal and includes the following steps: obtaining multimodal physiological data of cabin workers and environmental data in the cabin collected in real time by a multimodal data sensor; performing data preprocessing including data cleaning and data standardization on the multimodal physiological data and environmental data; extracting physiological indicator features from the preprocessed multimodal physiological data, and extracting environmental indicator features from the preprocessed environmental data; using the extracted physiological indicator features and environmental indicator features as input data of a pre-trained deep learning network model to obtain a real-time predicted output of a health status monitoring result; uploading the collected multimodal physiological data and environmental data to a cloud server, and receiving an update of the deep learning network model from the cloud server; wherein, during the training process of the deep learning network model, a clustering model with different numbers of clusters is fitted, the number of clusters is determined by calculating and comparing the BIC value, the input containing the multimodal physiological data and environmental data is clustered and health level labeled by using a time series segmentation clustering method, and the deep learning network model is trained to learn the relationship between the multimodal physiological data and environmental data and the health level labeling.

[0008] In some embodiments of the present invention, the multimodal data sensor includes multiple types of electromyography sensors EMG, pulse sensors PPG, respiratory sensors RESP, electrocardiogram sensors ECG, electroencephalograms EEG and environmental sensors; the multimodal physiological data includes multiple types of EMG electrical signals, HRV collected by PPG, respiratory rate and respiratory depth monitored by RESP, ECG electrocardiogram data and EEG electroencephalogram data; the physiological indicator characteristics include multiple types of EMG amplitude, time domain indicator characteristics and frequency domain indicator characteristics of HRV, respiratory rate and rhythm of RESP, heart rate and heart rate variability of ECG electrocardiogram data, and band activity of EEG electroencephalogram data; the environmental indicator characteristics include multiple types of temperature, humidity, air pressure and carbon dioxide concentration in the cabin.

[0009] In some embodiments of the present invention, the data cleaning includes filtering denoising and outlier processing, and the data standardization includes signal amplification processing and standardization processing for format and sampling rate.

[0010] In some embodiments of the present invention, the step of determining the number of clusters of clustering by calculating and comparing BIC values ​​includes: calculating the log-likelihood value of each fitted clustering model; calculating the BIC value of each clustering model in combination with the calculated log-likelihood value; selecting the number of clusters of the clustering model that can minimize the BIC value as the number of clusters of clustering, and performing time series segmentation clustering according to the selected number of clusters of clustering.

[0011] In some embodiments of the present invention, a time series segmentation and clustering method is used to cluster and annotate the health level of inputs containing multimodal physiological data and environmental data, and the step of training the deep learning network model to learn the relationship between the multimodal physiological data and environmental data and the health level annotations includes: clustering the multimodal physiological data and environmental data in a time series segmentation and clustering manner, and the number of clusters is the number of clusters that can minimize the BIC value; annotating health level labels for different clusters obtained by clustering; and using a training set containing multimodal physiological data and environmental data and health level annotations to train the deep learning network model, so that the trained deep learning network model can predict the health level based on the input multimodal physiological data and environmental data.

[0012] In some embodiments of the present invention, the method for monitoring the health status of cabin workers is implemented based on an edge computing terminal, which obtains multimodal physiological data and environmental data collected in real time by a multimodal data sensor and uploads them to a cloud server; for the cloud server, the method also includes: the cloud server uses the multimodal physiological data and environmental data received in real time from the edge computing terminal to further train and update the deep learning network model.

[0013] In some embodiments of the present invention, for the real-time output health status monitoring results obtained, the method also includes: displaying the health status monitoring results or the cabin operator health status monitoring report containing the health status monitoring results in real time through a user interface; receiving control instructions for the edge computing terminal or multimodal data sensor input by the operator in real time through the user interface, so as to adjust the working status of the edge computing terminal or the multimodal data sensor according to the control instructions.

[0014] Corresponding to the above method, the present invention also provides an edge computing terminal, including a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the edge computing terminal implements the steps of the method described in any one of the above embodiments.

[0015] Corresponding to the above method, the present invention also provides a cabin worker health status monitoring system based on multimodal data, the system comprising: a multimodal data sensor, used to collect multimodal physiological data of cabin workers and environmental data in the cabin in real time; a wireless communication module, used to transmit the multimodal physiological data and environmental data collected by the multimodal data sensor to an edge computing terminal; an edge computing terminal, used to implement cabin worker health status monitoring according to the steps of the method described in any one of claims 1-7; a cloud server, used to receive the collected multimodal physiological data and environmental data from the edge computing terminal, further train and optimize the clustering model, and update the optimized clustering model to the edge computing terminal.

[0016] Corresponding to the above method, the present invention also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method described in any one of the above embodiments are implemented.

[0017] Corresponding to the above method, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any one of the above embodiments when executed by a processor.

[0018] The method, system, edge computing terminal and medium for monitoring the health status of cabin workers based on multimodal data proposed in the present invention can use the BIC algorithm to perform feature selection and model parameter optimization on multimodal sensor data to reduce feature dimensions and improve the generalization ability of the model, and can use time series segmentation and clustering to process multivariate time series data. By analyzing the time dependence of sensor data, patterns and trends related to health status can be identified, and deep learning network models can be used for learning, thereby realizing the identification and monitoring of health status in a classified and predicted manner, thereby realizing accurate health status monitoring of cabin workers.

[0019] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to a person skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0020] Those skilled in the art will appreciate that the objectives and advantages that can be achieved by the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0022] Figure 1 The present invention is a flow chart of a method for monitoring the health status of cabin workers in one embodiment of the present invention.

[0023] Figure 2 A technical roadmap for designing a method for monitoring the health status of cabin workers in one embodiment of the present invention.

[0024] Figure 3 A schematic diagram of the structure of the computer equipment included in the edge computing terminal. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0026] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0027] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0028] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0029] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0030] In order to overcome the problems existing in the prior art, the present invention proposes a method, system, edge computing terminal and medium for monitoring the health status of cabin workers based on multimodal data. The method uses multimodal data and edge computing to perform real-time monitoring and intelligent identification of the health status of workers in special cabin environments, and combines the BIC algorithm, TICC algorithm and deep learning network model to identify the health status of workers, so as to improve work efficiency and ensure work safety.

[0031] Figure 1 The present invention is a flowchart of a method for monitoring the health status of cabin workers based on multimodal data in one embodiment of the present invention.

[0032] The method includes:

[0033] Step S110: Acquire multimodal physiological data of cabin workers and environmental data in the cabin collected in real time by multimodal data sensors.

[0034] Step S120: performing data preprocessing including data cleaning and data standardization on the multimodal physiological data and environmental data.

[0035] Step S130: extracting physiological index features from the preprocessed multimodal physiological data, and extracting environmental index features from the preprocessed environmental data.

[0036] Step S140: Using the extracted physiological indicator characteristics and environmental indicator characteristics as input data for the pre-trained deep learning network model to obtain real-time predicted output of health status monitoring results.

[0037] Step S150: Upload the collected multimodal physiological data and environmental data to the cloud server, and receive updates to the deep learning network model from the cloud server.

[0038] Among them, in the training process of the deep learning network model, clustering models with different numbers of clusters are fitted, the number of clusters is determined by calculating and comparing the BIC value, and the time series segmentation clustering method is used to cluster and label the health level of the input containing multimodal physiological data and environmental data, and the deep learning network model is trained to learn the relationship between multimodal physiological data and environmental data and health level labels.

[0039] The method for monitoring the health status of cabin workers based on multimodal data proposed in the present invention can use the BIC algorithm to perform feature selection and model parameter optimization on multimodal sensor data to reduce feature dimensions and improve the generalization ability of the model, and can process multivariate time series data by time series segmentation and clustering. By analyzing the time dependence of sensor data, patterns and trends related to health status can be identified, and deep learning network models can be used for learning, thereby realizing the identification and monitoring of health status in a classified and predicted manner, thereby realizing accurate health status monitoring of cabin workers.

[0040] In some embodiments of the present invention, multimodal data and equipment may be: (1) multimodal data sensors include multiple types of electromyography sensors EMG, pulse sensors PPG, respiratory sensors RESP, electrocardiogram sensors ECG, electroencephalograms EEG, and environmental sensors; (2) multimodal physiological data include multiple types of EMG electrical signals, HRV collected by PPG, respiratory rate and respiratory depth monitored by RESP, ECG electrocardiogram data, and EEG electroencephalogram data; (3) physiological indicator characteristics include multiple types of EMG amplitude, time domain indicator characteristics and frequency domain indicator characteristics of HRV, respiratory rate and rhythm of RESP, heart rate and heart rate variability of ECG electrocardiogram data, and band activity of EEG electroencephalogram data; (4) environmental indicator characteristics include multiple types of temperature, humidity, air pressure, and carbon dioxide concentration in the cabin.

[0041] By adopting this embodiment of the invention, it is possible to realize intelligent identification of the status of the operating personnel in the cabin through multimodal physiological data and environmental data.

[0042] In some embodiments of the present invention, data cleaning includes filtering, denoising and outlier processing, and data standardization includes signal amplification and standardization of format and sampling rate.

[0043] By adopting this embodiment of the invention, the quality of the characteristic data used for intelligent identification of the status of the operating personnel in the cabin can be guaranteed.

[0044] In some embodiments of the present invention, the step of determining the number of clusters of clustering by calculating and comparing BIC values ​​includes: (1) calculating the log-likelihood value of each fitted clustering model; (2) calculating the BIC value of each clustering model in combination with the calculated log-likelihood value; (3) selecting the number of clusters of the clustering model that can minimize the BIC value as the number of clusters of clustering, and performing time series segmentation clustering according to the selected number of clusters.

[0045] By adopting the embodiment of the invention, the BIC algorithm can be used to perform feature selection and model parameter optimization on multimodal sensor data to reduce feature dimensions and improve the generalization ability of the model, and the BIC algorithm is used to determine the number of clusters used for clustering.

[0046] In some embodiments of the present invention, a time series segmentation and clustering method is used to cluster and annotate the health level of inputs containing multimodal physiological data and environmental data, and the step of training the deep learning network model to learn the relationship between the multimodal physiological data and environmental data and the health level annotations includes: (1) clustering the multimodal physiological data and environmental data in a time series segmentation and clustering manner, and the number of clusters is the number of clusters that can minimize the BIC value; (2) annotating health level labels for different clusters obtained by clustering; (3) using a training set containing multimodal physiological data and environmental data and health level annotations to train the deep learning network model, so that the trained deep learning network model can predict the health level based on the input multimodal physiological data and environmental data.

[0047] By adopting the embodiment of the invention, multivariate time series data is processed by time series segmentation and clustering. By analyzing the time dependence of sensor data, patterns and trends related to health status are identified, the collected physiological data characteristic indicators and environmental data characteristic indicators are clustered, and each clustering result is labeled, so as to be used for the deep learning model to identify the health status of workers.

[0048] In some embodiments of the present invention, the method for monitoring the health status of cabin workers based on multimodal data is implemented based on an edge computing terminal, and the edge computing terminal obtains multimodal physiological data and environmental data collected in real time by a multimodal data sensor and uploads it to a cloud server. For the cloud server, the method further includes: the cloud server further trains and updates the deep learning network model using the multimodal physiological data and environmental data received in real time from the edge computing terminal.

[0049] By adopting the embodiment of the invention, it is possible to utilize the cloud service platform to realize end-cloud collaboration, utilize the edge end for real-time data collection and preliminary analysis, and utilize the cloud to realize deep learning, model training, and dynamic model updating, so as to ensure the real-time, accuracy, and robustness of the system.

[0050] In some embodiments of the present invention, for the real-time output health status monitoring results obtained, the method also includes: (1) displaying the health status monitoring results or the cabin operator health status monitoring report containing the health status monitoring results in real time through a user interface; (2) receiving control instructions for the edge computing terminal or multimodal data sensor input by the operator in real time through the user interface, so as to adjust the working state of the edge computing terminal or the multimodal data sensor according to the control instructions.

[0051] In other embodiments of the present invention, the execution entity may also be an edge computing processing unit, the edge computing terminal sends the collected data or the pre-processed collected data to the edge computing processing unit, and the wireless communication module transmits the data to the edge computing processing unit for data preprocessing and preliminary status identification.

[0052] By adopting the embodiment of the invention, real-time visualization and real-time hardware control can be realized by using the user interface.

[0053] Corresponding to the above method, the present invention also provides a cabin worker health status monitoring system based on multimodal data, the system comprising: (1) a multimodal data sensor, used to collect multimodal physiological data of cabin workers and environmental data in the cabin in real time; (2) a wireless communication module, used to transmit the multimodal physiological data and environmental data collected by the multimodal data sensor to an edge computing terminal; (3) an edge computing terminal, used to implement cabin worker health status monitoring according to the steps of the method described in any one of claims 1 to 7; (4) a cloud server, used to receive the collected multimodal physiological data and environmental data from the edge computing terminal, further train and optimize the clustering model, and update the optimized clustering model to the edge computing terminal.

[0054] Figure 2 The technical roadmap for designing a method for monitoring the health status of cabin workers in one embodiment of the present invention is as follows: The construction idea of ​​the method for monitoring the health status of cabin workers based on multimodal data proposed by the present invention is described in detail.

[0055] The components of the health status identification system for workers in a cabin environment include but are not limited to: (1) edge computing terminal; (2) multimodal data sensor; (3) wireless communication module; (4) cloud server; and (5) user interface.

[0056] The functions of its various components are introduced as follows: (1) Edge computing terminal: refers to the terminal device, including smart bracelets, head-mounted devices or embedded computing modules. The edge computing terminal directly interacts with the user for data collection and preliminary processing. (2) Wireless communication module: Uses Bluetooth, Wi-Fi and other technologies to transmit data to the edge computing terminal. (3) Cloud server: Responsible for data storage, deep learning model training, historical data management, report generation and model update. (4) User interface: Provides real-time data display and health status report. (5) Multimodal data sensor: Electromyography sensor (EMG): Attached to the forearm muscle to collect electromyography signals in real time. Pulse rate sensor (PPG): Installed on the wrist to measure heart rate variability (HRV). Respiration sensor (RESP): Worn on the chest to record breathing rate. Electrocardiogram sensor (ECG): Attached to the chest to record electrocardiogram signals. Electroencephalogram (EEG): Worn on the head to record electroencephalogram signals.

[0057] In some embodiments of the present invention, the system may also include an edge computing processing unit (ECU), which is a hardware configuration that drives edge computing and includes a CPU, memory, storage, and other parts. It is usually located at the edge of facilities such as routers, switches, and gateways and is used for data preprocessing and preliminary status identification.

[0058] The process of identifying the health status of workers in the cabin environment includes data collection, data preprocessing, indicator feature extraction, data analysis and model building, real-time detection and feedback of workers' health status using models, and remote monitoring and maintenance of edge computing terminals by cloud servers.

[0059] The following is an introduction to each stage of the process of identifying the health status of operators in the cabin environment:

[0060] Phase 1: Data collection phase.

[0061] The data collection phase includes three parts: sensor initialization, physiological data collection, and environmental data collection. First, ensure that all sensors are deployed correctly and synchronized. During the identification of the health status of operators, physiological data including EMG, HRV, RESP, ECG, and EEG are collected, and environmental data including working hours, work intensity, temperature, and weather conditions are monitored and collected in real time through a variety of sensors. Among them, EMG refers to the electrical signal generated by muscle activity, HRV refers to the heart rate variability data collected through the PPG sensor, RESP refers to the monitoring of respiratory rate and breathing depth, ECG refers to the electrocardiogram data that records the electrical activity of the heart, and EEG refers to the electroencephalogram data that monitors brain activity.

[0062] In the specific implementation process, taking the application scenario in the submarine cabin as an example, the edge computing terminal smart bracelet worn by the workers in the submarine cabin integrates a variety of physiological sensors, including electromyography sensor (EMG), pulse sensor (PPG), respiration sensor (RESP), electrocardiogram sensor (ECG) and electroencephalogram (EEG). These sensors collect physiological data of workers in real time, such as electromyography signals, heart rate variability, respiratory rate, electrocardiogram signals and electroencephalogram signals. At the same time, environmental sensors monitor the ambient temperature, humidity, carbon dioxide concentration and other data in the cabin.

[0063] Phase 2: Data preprocessing phase.

[0064] The data preprocessing stage includes data cleaning and data standardization. Data cleaning refers to removing noise and outliers, and data standardization refers to unifying data formats and sampling rates. Preprocessing the collected physiological and environmental data can facilitate subsequent processing.

[0065] During the specific implementation process, the collected physiological data can be preprocessed by the bracelet and realized through the built-in edge computing module of the bracelet. The preprocessing includes: filtering and denoising, signal amplification and normalization processing (standardization processing) to improve the quality and availability of the data.

[0066] When the monitored health status is a fatigue state of the user, physiological and environmental indicators related to fatigue may be extracted from the processed data, such as the mean, standard deviation, maximum and minimum values ​​of the heart rate.

[0067] Phase 3: indicator feature extraction phase.

[0068] The indicator feature extraction stage includes physiological indicator feature extraction and environmental indicator feature extraction. Physiological indicator feature extraction refers to extracting key indicator features from raw physiological data, such as EMG amplitude, HRV time domain indicator features and frequency domain indicator features, RESP frequency and rhythm, ECG heart rate and heart rate variability, and EEG band activity. Environmental indicator feature extraction refers to extracting indicators related to the working environment, such as temperature, humidity, air pressure, etc.

[0069] Phase 4: Data analysis and model building phase.

[0070] In the data analysis and model building stage, the Bayesian Information Criterion algorithm (BIC algorithm) model is used to determine the number of clusters for the health status of cabin operators, and the time series segmentation clustering algorithm (TICC clustering algorithm model, Toeplitz Inverse Covariance-Based Clustering) is applied to cluster and label physiological data and environmental data, classify the data into different health levels, and use the trained deep learning network model to perform real-time health level classification. This step is completed on the edge computing terminal to reduce the delay of data transmission and provide quick feedback and early warning.

[0071] In the specific implementation process, the health status can be specifically reflected as the fatigue status of the user, that is, the fatigue level of the cabin workers is monitored using this method, the health level is replaced by the fatigue level, and the fatigue level classification is realized.

[0072] The process of BIC algorithm selecting the best clustering model is as follows:

[0073] (1) Fitting clustering models with different numbers of clusters.

[0074] The main use of the BIC algorithm in statistics is to select the best model from a limited set of models. For each cluster number k, fit a clustering model. Taking K-means as an example, you need to calculate the sum of the squares of the distances from each data point to its nearest cluster center, that is, the total internal sum of squares (SSE):

[0075]

[0076] Where n is the total number of data points. k is the number of clusters. i is the data point x i The cluster to which it belongs. i =j) is the indicator function, if x i If it belongs to cluster j, it is 1, otherwise it is 0. j is the center of cluster j. ||x i -μ j || 2 is the data point x i To cluster center μ j The square of the Euclidean distance.

[0077] Among them, n is the number of data samples, k is the number of parameters of the model, which is usually related to the number of clusters in clustering. For K-means, k is the number of clusters. is the maximum likelihood estimate of the model, here we use the log-likelihood of SSE.

[0078] (2) Calculate the log-likelihood value of each fitted clustering model.

[0079] Log-likelihood It can be approximately calculated as follows:

[0080]

[0081] Among them, SSE i is the sum of the squares of the distances from the ith data point to its nearest cluster center.

[0082] (3) Calculate the BIC value of each fitted clustering model.

[0083] The BIC value for each model was calculated using the BIC formula:

[0084]

[0085] Where n is the number of data samples and k is the number of parameters of the model, which is usually related to the number of clusters in clustering. is the maximum likelihood estimate of the model.

[0086] (4) Select the model with the smallest BIC value.

[0087] Compare the BIC values ​​of different k values, and select the k with the smallest BIC value as the optimal number of clusters. This is because the model with the smallest BIC value finds the best balance between model complexity and data fitting.

[0088] The TICC algorithm is an algorithm for efficiently segmenting and clustering multivariate time series. The establishment of the TICC algorithm model includes the following steps:

[0089] (1) Data Preparation:

[0090] TICC computes a separate precision matrix for each time delay within the time window. For a window size of w, we compute w-1 separate precision matrices: one for the points at time t and t-1, another for the points at time T and t-2, and so on, until the points at time t and t-w+1. Where subsequence represents, which is an n×w dimensional vector.

[0091] X t =[x t-w+1 ,x t-w ,…,x t ];

[0092] (2) Initialization:

[0093] Initialize the cluster assignments by computing a Gaussian mixture model of the stacked data, using the same number of mixture model components as the number of clusters created by TICC.j is the mixing coefficient of the jth cluster, μ j is the mean vector, ∑ j is the covariance matrix.

[0094]

[0095] (3) Update Cluster Description:

[0096] The principle formula in this step is as follows:

[0097]

[0098] In the stage of updating cluster descriptions, each cluster j has an inverse covariance matrix Θ j The inverse covariance matrix is ​​a block Toeplitz matrix where each block is of n×n dimensions.

[0099] (4) Compute Labeling Costs:

[0100] During the labeling cost phase, we compute the (log) likelihood of each point with respect to each cluster.

[0101]

[0102] Where β is a parameter used to measure the penalty when adjacent signal segments are of different classes.

[0103] Iterate until the cluster assignments stop changing.

[0104] In the iteration phase, the TICC problem can be formulated as an optimization problem, where Π is the indicator function, if X i and X i+1 If they are not in the same cluster, then it is 1, otherwise it is 0.

[0105]

[0106] During the model training process, a convolutional neural network model (CNN) is trained on a cloud server using historical data to predict health status (including fatigue status).

[0107] Phase 5: Use the model to detect and provide feedback on the health status of workers in real time.

[0108] This stage includes two steps: real-time data processing and health status identification. (1) In the real-time data processing step, the program inputs the physiological data and environmental data collected in real time into the trained deep learning model; (2) In the health status identification step, the pre-trained model is used to analyze the input data and identify the fatigue status of the workers.

[0109] In the specific implementation process, the physiological data of the working personnel can be collected in real time and input into the trained deep learning model to realize the real-time identification and classification of fatigue status. When the system detects that the fatigue status reaches the warning threshold, it will immediately remind the working personnel in real time through vibration, sound or visual signals to ensure that the working personnel can take timely measures to avoid excessive fatigue. The calculation of this process can be realized by the edge computing terminal built into the bracelet worn by the operator. In addition, the system adopts data synchronization and fault-tolerant technology to ensure that the integrity and accuracy of the data can be maintained even in the complex electromagnetic environment of the submarine, thereby ensuring the reliability of the monitoring data. This real-time monitoring and feedback mechanism, combined with the advantage of data stability in the special environment of the submarine, provides a strong guarantee for the health and safety of submarine working personnel.

[0110] Phase 6: Cloud servers remotely monitor and maintain the edge computing terminal’s capabilities.

[0111] The cloud server has the following functions during the implementation of the entire solution: (1) Data upload: The edge computing terminal uploads the collected data to the cloud server in real time; (2) Data storage and analysis: The server stores data and conducts in-depth analysis to optimize the model and provide insights; (3) Model update: The model is updated based on new data and analysis results to improve recognition accuracy.

[0112] The cloud server plays a core role in this solution. It not only provides remote monitoring and maintenance functions, but also enables technical operators to perform necessary maintenance and upgrades on the edge computing bracelet in a timely manner after the submarine returns to the base, ensuring the long-term stable operation of the equipment. This function is especially important for special environments such as submarines, because submarines are often far away from the base when performing missions, and timely maintenance after returning can ensure the smooth progress of the next mission.

[0113] In addition, the cloud server's deep analysis capabilities provide strong data support for the health and safety of submarine workers. It can receive a large amount of physiological and environmental data uploaded from edge computing terminals in real time, and use convolutional neural networks (CNN) and long short-term memory networks (LSTM) models for deep analysis to accurately predict fatigue status. This deep analysis can not only generate detailed analysis reports to help formulate targeted preventive measures, but also optimize work processes and reduce health risks for submarine workers.

[0114] Among them, the extracted features can be deeply learned through convolutional neural network (CNN), and the relationship between multimodal physiological data and environmental data and the health status labels of the operators obtained by clustering can be learned, which can be implemented in various ways such as prediction model and classification model. The health status in the embodiment of the present invention can specifically refer to fatigue status only, or other operator status.

[0115] In a complex electromagnetic environment such as a submarine, the cloud server can not only process and store real-time data from edge computing terminals, but also display real-time data and historical trends through a visual dashboard, providing intuitive health monitoring information for submarine workers. This real-time data flow and in-depth analysis capabilities make the cloud server a powerful tool for health monitoring of submarine workers. Even in the face of extreme environmental challenges, it can maintain data continuity and accuracy, providing solid technical support for the successful completion of submarine missions.

[0116] Phase 7: User interaction phase.

[0117] The user interaction phase includes two steps: interface display and report generation. (1) Interface display: The user interface displays real-time monitoring data and health status, including physiological parameters and environmental conditions. (2) Report generation: Users can generate detailed health status reports and historical trend analysis as needed.

[0118] Phase 8: System optimization and maintenance.

[0119] The system optimization and maintenance phase includes two steps: system performance monitoring and system update. (1) Performance monitoring: Regularly check system performance to ensure data accuracy and real-time performance. (2) System update: Update system software and algorithms based on new research results and technological advances.

[0120] Corresponding to the above method, the present invention also provides an edge computing terminal, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the edge computing terminal implements the steps of the method described above.

[0121] Figure 3 A schematic diagram of the structure of the computer equipment included in the edge computing terminal.

[0122] See also Figure 3The computer device 00 includes: a processor 01, a memory 02, and a computer program stored in the memory 02 and executable on the processor 01. When the processor 01 executes the computer program, the human factor data server access control method provided in the above method embodiment is implemented.

[0123] Among them, the processor 01 is connected to the memory 02, such as through a bus 03. The processor 01 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in combination with the contents disclosed in this application. The processor 01 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of DSP and microprocessors, etc. The bus 03 may include a path to transmit information between the above components. The bus 03 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 130 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The memory 02 is used to store a computer program corresponding to the human factor data server access control method of the above embodiment of the present application, and the computer program is controlled and executed by the processor 01. The processor 01 is used to execute the computer program stored in the memory 02 to implement the content shown in the above method embodiment.

[0124] Corresponding to the above method, the present invention also provides a computer-readable storage medium, on which a computer program / instruction is stored, and when the computer program / instruction is executed by a processor, the steps of the method described in any one of the above embodiments are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0125] Corresponding to the above method, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any one of the above embodiments when executed by a processor.

[0126] It should be understood by ordinary technical personnel in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but this implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier.

[0127] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and technical operators in this field can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0128] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0129] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For technical personnel in the field, the embodiments of the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for monitoring the health status of cabin workers based on multimodal data, characterized in that: include: Acquire multimodal physiological data of cabin operators and environmental data in the cabin collected in real time by multimodal data sensors; Perform data preprocessing including data cleaning and data standardization on multimodal physiological data and environmental data; Extracting physiological index features from the preprocessed multimodal physiological data, and extracting environmental index features from the preprocessed environmental data; The extracted physiological index characteristics and environmental index characteristics are used as input data for the pre-trained deep learning network model to obtain real-time prediction output of health status monitoring results; Uploading the collected multimodal physiological data and environmental data to a cloud server, and receiving updates to the deep learning network model from the cloud server; In which, during the training process of the deep learning network model, clustering models with different numbers of clusters are fitted, the number of clusters is determined by calculating and comparing the BIC value, and the input containing multimodal physiological data and environmental data is clustered and health level labeled by time series segmentation clustering, and the deep learning network model is trained to learn the relationship between multimodal physiological data and environmental data and health level labels.

2. The method according to claim 1, characterized in that The multimodal data sensor includes multiple types of an electromyographic sensor EMG, a pulse sensor PPG, a respiratory sensor RESP, an electrocardiographic sensor ECG, an electroencephalometer EEG and an environmental sensor; The multimodal physiological data includes multiple types of EMG electrical signals, HRV collected by PPG, respiratory rate and respiratory depth monitored by RESP, ECG electrocardiogram data and EEG electroencephalogram data; The physiological index characteristics include multiple ones of the amplitude of EMG, the time domain index characteristics and frequency domain index characteristics of HRV, the respiratory frequency and rhythm of RESP, the heart rate and heart rate variability of ECG data, and the band activity of EEG data; The environmental indicator characteristics include multiple ones of the temperature, humidity, air pressure and carbon dioxide concentration in the cabin.

3. The method according to claim 1, characterized in that The data cleaning includes filtering, denoising and outlier processing, and the data standardization includes signal amplification processing and standardization processing for format and sampling rate.

4. The method according to claim 1, characterized in that: The step of determining the number of clusters of clustering by calculating and comparing BIC values ​​comprises: Calculate the log-likelihood value for each cluster model fitted; The BIC value of each clustering model was calculated by combining the calculated log-likelihood values; The number of clusters of the clustering model that can minimize the BIC value is selected as the number of clusters, and the time series segmentation clustering is performed according to the selected number of clusters.

5. The method according to claim 1, characterized in that The steps of clustering and health level labeling the input including multimodal physiological data and environmental data by using time series segmentation and clustering, and training the deep learning network model to learn the relationship between the multimodal physiological data and environmental data and the health level labeling include: The multimodal physiological data and environmental data are clustered in the manner of time series segmentation and clustering, and the number of clusters is the number of clusters that can minimize the BIC value; Label the health level of different clusters obtained by clustering; A deep learning network model is trained using a training set containing multimodal physiological data, environmental data and health level annotations, so that the trained deep learning network model can predict the health level based on the input multimodal physiological data and environmental data.

6. The method according to claim 1, characterized in that The method for monitoring the health status of cabin workers is implemented based on an edge computing terminal, which obtains multimodal physiological data and environmental data collected in real time by a multimodal data sensor and uploads the data to a cloud server; For the cloud server, the method also includes: the cloud server uses the multimodal physiological data and environmental data received in real time from the edge computing terminal to further train and update the deep learning network model.

7. The method according to claim 1, characterized in that For the obtained real-time output health status monitoring result, the method further includes: Display health status monitoring results or cabin operator health status monitoring reports containing health status monitoring results in real time through a user interface; The control instructions for the edge computing terminal or the multimodal data sensor input by the operator are received in real time through the user interface to adjust the working state of the edge computing terminal or the multimodal data sensor according to the control instructions.

8. An edge computing terminal, comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the edge computing terminal implements the steps of the method as described in any one of claims 1 to 7.

9. A cabin operator health status monitoring system based on multimodal data, characterized in that: The system includes: Multimodal data sensors are used to collect multimodal physiological data of cabin operators and environmental data in the cabin in real time; A wireless communication module, used to transmit multimodal physiological data and environmental data collected by the multimodal data sensor to the edge computing terminal; An edge computing terminal, used to implement health status monitoring of cabin workers according to the steps of the method described in any one of claims 1 to 7; The cloud server is used to receive the multimodal physiological data and environmental data collected from the edge computing terminal, further train and optimize the clustering model, and update the optimized clustering model to the edge computing terminal.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 7 are implemented.

Images