Fatigue monitoring method and system integrating face and ultrasonic bracelet

Abstract

The invention provides a face and ultrasonic bracelet integrated fatigue monitoring method and system, and belongs to the technical field of fatigue monitoring, and the method comprises the steps: obtaining a face video stream of a to-be-monitored object, and blood vessel imaging, hemodynamics and body temperature data provided by an ultrasonic bracelet; dynamic changes of facial feature points are extracted through a facial key point tracking algorithm, a facial motion feature group is constructed, time sequence features are embedded through position coding, and facial motion feature vectors are formed; according to the blood vessel imaging data, the hemodynamics and the body temperature data, an ultrasonic data feature group is constructed, and an ultrasonic physiological feature vector is formed. And through a pre-trained multi-modal fatigue monitoring neural network, performing fatigue degree identification based on the ultrasonic physiological feature vector and the facial motion feature vector. According to the method, the ultrasonic physiological data and the facial movement data are combined for fatigue monitoring, the time sequence and mutual relevance of the ultrasonic physiological data and the facial movement data are considered, and the accuracy of fatigue judgment is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of fatigue monitoring technology, specifically to a fatigue monitoring method and system that integrates facial features with an ultrasonic wristband. Background Technology

[0002] During transportation or flight, the fatigue level of drivers or pilots has a crucial impact on safety. Fatigue not only significantly reduces reaction speed and judgment but can also lead to distraction, decision-making errors, and slow reactions, thereby increasing the risk of accidents. Therefore, ensuring that drivers or pilots maintain sufficient alertness and concentration during their work is critical to ensuring transportation and flight safety. To promptly detect and intervene in driver or pilot fatigue, fatigue monitoring systems have emerged. The main purpose of fatigue monitoring is to identify fatigue levels in a timely manner by monitoring the physical and mental state of drivers or pilots in real time and to take necessary early warning or intervention measures. These measures may include automatically reminding drivers to rest, providing appropriate auxiliary control systems, or activating autopilot mode in extreme situations. By monitoring fatigue, operational errors caused by fatigue can be effectively avoided, reducing the occurrence of traffic and flight accidents. Furthermore, fatigue monitoring can help companies and airlines optimize work schedules, scientifically arrange the working hours of drivers and pilots, and ensure they maintain optimal condition during peak mission periods, thereby improving the efficiency and safety of transportation or flight operations.

[0003] Facial changes are often the preferred method for assessing fatigue, especially in real-time monitoring using video data acquisition and facial motion analysis. Changes in facial expressions, such as blinking frequency, eyelid drooping, and the degree of facial muscle relaxation, can reflect the fatigue level of a pilot or operator. The correlation between facial feature changes and fatigue makes facial video analysis an effective means of fatigue detection, particularly during long-duration driving or flight missions, enabling timely detection of signs of fatigue through dynamic monitoring and early warning.

[0004] Relying solely on facial features for fatigue assessment has limitations. Fatigue is a comprehensive physiological and psychological state, manifested not only in changes in facial expressions but also in fluctuations in physiological indicators such as heart rate, skin temperature, and respiratory rate. These physiological changes are often precursory signals of fatigue, indicating a pilot's or driver's fatigue state before obvious facial changes appear. However, both facial features and physiological indicators have limitations in their application. Emotional fluctuations or the specificity of physiological data can easily lead to difficulties in accurately pinpointing the onset of fatigue, potentially resulting in delayed or inaccurate fatigue assessments. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a fatigue monitoring method that integrates facial and ultrasonic wristband data. This method combines ultrasound physiological data and facial motion data for fatigue monitoring, taking into account the temporal sequence and interrelationship between the two, thereby effectively improving the accuracy of fatigue assessment.

[0006] To achieve the above objectives, the present invention provides the following technical solution.

[0007] A fatigue monitoring method integrating facial and ultrasonic wristbands includes the following steps: The morphology of blood vessels under different degrees of fatigue was detected by an ultrasound wristband, and time-series data of two-dimensional blood vessel images were obtained; time-series data of blood vessel diameter were obtained from the two-dimensional blood vessel images. The time it takes for ultrasound waves to travel through the wrist is determined by the time the ultrasound bracelet receives the echo, thus determining the speed of ultrasound propagation. Based on the mathematical model of blood viscosity and ultrasound propagation speed attenuation obtained from the experiment, an estimated value of blood viscosity is obtained. Time series data of blood flow resistance changes are obtained based on the blood vessel diameter and the current estimated value of blood viscosity. For time series data of blood vessel diameter and blood flow resistance, first-order difference is used to obtain the rate of change of blood vessel diameter and blood flow resistance in each time period. The sliding window method is used to smooth the data and obtain the trend of change of the rate of change in each time window, and to locate abnormal periods where the trend of change of blood flow resistance or blood vessel diameter exceeds a preset threshold. Based on facial video streams under different degrees of fatigue, facial motion feature data and facial temperature data were obtained concurrently with the abnormal time period. A training set was constructed, including facial motion feature data, temperature, blood vessel diameter and blood viscosity change rates during abnormal periods under different levels of fatigue. A multimodal neural network was trained to obtain a monitoring model for predicting fatigue state.

[0008] Preferably, obtaining time-series data of blood vessel diameter from two-dimensional blood vessel images includes the following steps: Two-dimensional blood vessel images of consecutive frames are grayscaled and denoised to obtain preprocessed two-dimensional blood vessel images; Edge extraction of two-dimensional blood vessel images is performed using a threshold segmentation method, and noise is removed by dilation and erosion to obtain the blood vessel region; The vascular region is refined into a centerline using a skeletonization algorithm, and the vascular diameter is estimated using a radial projection method along the centerline direction. ; In the formula, For the location of blood vessels The diameter at that point For position The distance from the blood vessel wall, The angle along the centerline; The average of the blood vessel diameters at multiple locations is used as the blood vessel diameter at the current time point. D ; Each frame of the two-dimensional blood vessel image corresponds to a time point, and the diameter of the blood vessels in each frame of the two-dimensional blood vessel image is extracted. D We obtained time-series data of blood vessel diameters and smoothed them using Kalman filtering.

[0009] Preferably, obtaining the estimated blood viscosity value based on the mathematical model of blood viscosity change and ultrasound propagation velocity attenuation obtained from experiments includes the following steps: The time it takes for ultrasound waves to travel through the wrist is determined by the time the ultrasound bracelet receives the echo, thus obtaining the ultrasound propagation speed. Based on the average propagation speed obtained after the ultrasound bracelet receives the echo under different blood viscosities, a mathematical model of blood viscosity and ultrasound propagation speed attenuation is determined, and the model is fitted experimentally. ; In the formula, The speed of ultrasonic wave propagation. It is a constant. Blood viscosity; The time it takes for ultrasound waves to travel through the wrist is determined by the time the ultrasound bracelet receives the echo, thus determining the speed of ultrasound propagation; blood viscosity is determined by a mathematical model of blood viscosity and the attenuation of ultrasound propagation speed.

[0010] Preferably, the step of obtaining time-series data on changes in blood flow resistance based on vessel diameter and current blood viscosity estimates includes the following steps: Estimate the length of the vessel segment to be measured based on the centerline length obtained from the 2D vascular image. ; Based on Poiseuille's law, blood flow resistance is determined by estimating blood viscosity and vessel radius: ; in, It is the radius of the blood vessel; By analyzing the resistance data from multiple measurements, time-series data of changes in blood flow resistance were obtained.

[0011] Preferably, the time series data of vessel diameter and blood flow resistance are processed by using first-order difference to obtain the rate of change of vessel diameter and blood flow resistance in each time period, smoothing the data using a sliding window method and obtaining the trend of change of the rate of change in each time window, and locating abnormal periods in which the trend of change of blood flow resistance or vessel diameter exceeds a preset threshold, including the following steps: Time series data for determining blood vessel diameter Time series data of blood flow resistance ,in, t These are time points in a time series; Changes in vessel diameter and blood flow resistance were calculated using the first-order difference method. and The rate of change of vessel diameter and the rate of change of blood flow resistance were determined by the time difference between adjacent time points, and the rate of change was smoothed by the sliding window method to obtain the rate of change of vessel diameter. and rate of change of blood flow resistance ; The size of a preset time window is determined, and the rate of change of the diameter of the first and last blood vessels and the rate of change of blood flow resistance within the window are obtained. The trend of the rate of change is determined. When the trend of change exceeds the preset value, the time window is an abnormal time window, and multiple adjacent abnormal time windows are taken as abnormal time periods.

[0012] Preferably, obtaining facial motion feature data concurrent with the abnormal time period based on facial video streams under different degrees of fatigue includes the following steps: Acquire multiple facial image frames simultaneously in an abnormal time period from a facial video stream; For each facial image frame, a facial key point tracking algorithm is used to extract the dynamic changes of facial feature points and obtain facial motion feature data such as blink frequency, number and duration of yawns, changes in head posture angle, and rate of change of facial expression. Each frame of the image corresponds to a time point. The time series data of facial motion feature data of each facial image frame is extracted and smoothed by Kalman filtering.

[0013] Preferably, the multimodal neural network comprises four feature extraction modules, a feature fusion module, and a fully connected layer; wherein, the four feature extraction modules include three CNN-based feature extraction modules and one LSTM-based feature extraction module, which respectively extract features from the rate of change of temperature, blood vessel diameter, and blood viscosity, as well as facial motion feature data, and perform feature fusion through the feature fusion module to obtain fused features; based on the fused features, the corresponding fatigue level is mapped and output through the fully connected layer.

[0014] This invention also proposes a fatigue monitoring system integrating facial recognition and an ultrasonic wristband, the system comprising: An ultrasonic wristband, with ultrasonic transmitting patches and ultrasonic receiving patches respectively arranged at both ends of its inner side; The time-series data processing module is used to detect the morphology of blood vessels under different degrees of fatigue through an ultrasonic wristband, obtain time-series data of two-dimensional blood vessel images, and obtain time-series data of blood vessel diameter based on the two-dimensional blood vessel images. The blood flow resistance analysis module is used to determine the time it takes for ultrasound waves to travel through the wrist based on the time it takes for the ultrasound bracelet to receive the echo, and to determine the speed of ultrasound propagation; based on the mathematical model of blood viscosity and ultrasound propagation speed attenuation obtained from experiments, it obtains an estimated value of blood viscosity; and based on the blood vessel diameter and the current estimated value of blood viscosity, it obtains time series data of blood flow resistance changes. The abnormal time period analysis module uses first-order difference to obtain the rate of change of blood vessel diameter and blood flow resistance in each time period for time series data of blood vessel diameter and blood flow resistance. It uses the sliding window method to smooth the data and obtain the rate of change in each time window. Through the rising trend identification model of the rate of change, it locates abnormal time periods when the trend of change of blood flow resistance or blood vessel diameter exceeds the preset threshold. The facial motion feature analysis module is used to obtain facial motion feature data that is synchronous with the abnormal time period based on facial video streams under different degrees of fatigue. The monitoring model construction module is used to train a multimodal neural network based on the training set to obtain a monitoring model for predicting fatigue state; the training set includes facial movement feature data, temperature, blood vessel diameter and blood viscosity change rates during abnormal periods under different degrees of fatigue state.

[0015] The beneficial effects of this invention are: This invention proposes a fatigue monitoring method integrating facial and ultrasonic wristband data. This method combines ultrasound physiological data with facial motion data to form a multimodal fatigue monitoring system. Through multi-dimensional analysis of vascular morphology, blood flow resistance, and facial motion characteristics, it fully considers the temporal sequence and interrelationships of these data, effectively improving the accuracy of fatigue state assessment. Specifically, the ultrasonic wristband technology is used to acquire changes in physiological parameters such as vascular morphology and blood flow resistance, while facial motion analysis captures subtle changes in facial expressions to reflect the fatigue level of a driver or pilot. Since fatigue is a dynamic process, single physiological data or facial feature data is often insufficient to comprehensively and accurately assess the degree of fatigue. Combining these two types of data provides more precise fatigue identification. This integrated monitoring method effectively avoids the misjudgment problems of single monitoring methods and improves the robustness and sensitivity of the system.

[0016] By monitoring blood flow resistance and the rate of change in vessel diameter in real time, combined with temporal data of facial movement characteristics, the system can issue early warnings when the fatigue level of a driver or pilot reaches a critical point. Furthermore, based on the joint analysis of the ultrasound wristband and facial video stream, this method can dynamically adjust the criteria for fatigue assessment to adapt to differences in individual physiological responses, improving the personalization and adaptability of fatigue detection. Therefore, this multimodal data fusion method not only improves the accuracy of assessments but also provides a more reliable basis for fatigue intervention and early warning. Attached Figure Description

[0017] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0018] Figure 2 This is a flowchart illustrating the process of obtaining the blood vessel diameter according to an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] Example 1 When the human body experiences prolonged activity or fatigue, local muscles produce metabolic waste products, such as lactic acid, leading to increased blood flow resistance. Simultaneously, blood vessels dilate in an attempt to increase blood flow to supply more oxygen and nutrients and help remove waste. The wrist, as one of the most important areas for blood circulation, shows changes in blood vessel diameter and blood flow resistance that can indirectly reflect the level of fatigue. For example, in a state of fatigue, the diameter of blood vessels in the wrist may increase, while blood flow resistance may rise, indicating restricted blood flow and potentially insufficient oxygen and nutrient supply. Real-time monitoring of these changes can help identify early signs of fatigue, providing pilots with a scientific basis to adjust rest and training intensity in a timely manner and avoid flight accidents. Therefore, changes in wrist blood flow resistance and blood vessel diameter have become effective physiological parameters for assessing and monitoring fatigue, especially in the fatigue recovery process after high-intensity training. To this end, this embodiment proposes a fatigue monitoring method integrating facial and ultrasonic wristbands. Based on the monitoring results of changes in blood flow resistance and blood vessel diameter combined with facial movement feature data, fatigue judgment is made. The specific steps are as follows: Figure 1 As shown, it includes:

[0021] S1: The time series data of two-dimensional vascular images are obtained by detecting the vascular morphology under different degrees of fatigue using an ultrasound wristband; the time series data of vascular diameter are obtained based on the two-dimensional vascular images.

[0022] S2: Determine the time it takes for ultrasound waves to travel through the wrist based on the time the ultrasound bracelet receives the echo, and determine the speed of ultrasound propagation; obtain an estimated value of blood viscosity based on the mathematical model of blood viscosity and ultrasound propagation speed attenuation obtained from the experiment; obtain time series data of blood flow resistance changes based on blood vessel diameter and the current estimated value of blood viscosity.

[0023] S3: For time series data of blood vessel diameter and blood flow resistance, first-order difference is used to obtain the rate of change of blood vessel diameter and blood flow resistance in each time period. The sliding window method is used to smooth the data and obtain the trend of change of the rate of change in each time window, and to locate abnormal periods when the trend of change of blood flow resistance or blood vessel diameter exceeds the preset threshold.

[0024] S4: Based on facial video streams under different levels of fatigue, obtain facial motion feature data and facial temperature data that are concurrent with the abnormal time period.

[0025] S5: Construct a training set, including facial motion feature data, temperature, blood vessel diameter and blood viscosity change rates during abnormal periods under different levels of fatigue, train a multimodal neural network, and obtain a monitoring model for predicting fatigue state.

[0026] This invention, based on the intervention of physiological data, correlates the superficial data (facial movement feature data) caused by fatigue with physiological data in a temporal sequence, avoiding the influence of individual differences in physiological state on the assessment of fatigue status. Secondly, this invention uses the changing trend of blood flow resistance or the rate of change of blood vessel diameter as an indicator, which can more accurately and objectively reflect changes in vascular regulation and metabolic load in the body under fatigue state.

[0027] First, from an accuracy perspective, changes in blood flow resistance and vessel diameter are jointly regulated by the autonomic nervous system and local metabolic factors, reflecting the cardiovascular system's physiological response to fatigue stimuli in real time and avoiding the uncertainty of subjective fatigue assessment. Second, from a representativeness perspective, the response of peripheral blood vessels, such as those in the wrist, to fatigue is highly representative—under high load, the contraction and dilation of peripheral blood vessels are closely related to overall circulatory regulation, reflecting the body's energy consumption and metabolic recovery status. From the perspective of aligning with fatigue manifestations, fatigue is characterized by increased sympathetic nerve excitation, elevated vascular tension, and restricted local blood flow, while the recovery period is marked by vasodilation and decreased resistance. This dynamic process closely matches the body's physiological rhythm from tension to recovery, and capturing the specificity of its short-term trends better characterizes changes in fatigue status. Therefore, by analyzing the trends in the rate of change of blood flow resistance or vessel diameter, the onset, development, and recovery stages of fatigue can be captured with high sensitivity, serving as an accurate indicator for fatigue assessment.

[0028] Furthermore, such as Figure 2 The diagram shows the flowchart for obtaining blood vessel diameter. Obtaining time-series data of blood vessel diameter from a two-dimensional blood vessel image includes the following steps: S1.2: Perform grayscale conversion and denoising on consecutive frames of two-dimensional blood vessel images to obtain preprocessed two-dimensional blood vessel images.

[0029] S1.3: Edge extraction of two-dimensional blood vessel images is performed using a threshold segmentation method, and noise is removed by dilation and erosion to obtain the blood vessel region.

[0030] S1.4: The vascular region is refined into a centerline using a skeletonization algorithm, and the vascular diameter is estimated using a radial projection method along the centerline direction. ; In the formula, For the location of blood vessels The diameter at that point For position The distance from the blood vessel wall, The angle is along the centerline.

[0031] S1.5: Calculate the average of the blood vessel diameters at multiple locations as the blood vessel diameter at the current time point. D .

[0032] S1.6: Corresponding each frame of the two-dimensional vascular image to a time point, extract the vascular diameter from each frame of the two-dimensional vascular image. D We obtained time-series data of blood vessel diameters and smoothed them using Kalman filtering.

[0033] This invention uses the centerline projection method to determine the morphology of blood vessels, and then uses the average diameter at multiple locations to determine the diameter of the blood vessels. The above solution method is more accurate and can basically know the diameter change status of blood vessels in the current time period.

[0034] The blood viscosity estimation method of this invention is based on obtaining a corresponding linear law by considering the influence of different blood viscosities on the ultrasonic wave velocity. Blood, as a non-Newtonian fluid, has viscosity affected by factors such as blood composition, temperature, and flow rate. When ultrasonic waves pass through blood, their propagation speed changes due to variations in blood viscosity. Generally, higher blood viscosity results in slower ultrasonic wave propagation, while lower blood viscosity leads to faster propagation. By experimentally measuring the ultrasonic wave velocity under different viscosity conditions and establishing a linear relationship, a quantitative estimation of blood viscosity can be achieved. The core of this estimation method lies in measuring the time delay or frequency shift of ultrasonic wave propagation and, combined with a known linear relationship model, converting it into a blood viscosity value. This method offers high accuracy and real-time performance, providing non-invasive and rapid blood viscosity monitoring.

[0035] Specifically, the following steps are included: S2.1: The time it takes for ultrasound waves to travel through the wrist is determined by the time the ultrasound bracelet receives the echo, thus obtaining the ultrasound propagation speed. Based on the average propagation speed obtained after the ultrasound bracelet receives the echo under different blood viscosities, a mathematical model of blood viscosity and ultrasound propagation speed attenuation is determined, and the model is fitted experimentally. ; In the formula, The speed of ultrasonic wave propagation. It is a constant. This refers to blood viscosity.

[0036] S2.2: Determine the time it takes for the ultrasound to travel through the wrist based on the time it takes for the ultrasound bracelet to receive the echo, and determine the speed of ultrasound propagation; determine the blood viscosity based on the mathematical model of blood viscosity and the attenuation of ultrasound propagation speed.

[0037] Furthermore, the process of obtaining time-series data on changes in blood flow resistance based on vessel diameter and current blood viscosity estimates includes the following steps: Estimate the length of the vessel segment to be measured based on the centerline length obtained from the 2D vascular image. ; Based on Poiseuille's law, blood flow resistance is determined by estimating blood viscosity and vessel radius: ; in, It is the radius of the blood vessel; By analyzing the resistance data from multiple measurements, time-series data of changes in blood flow resistance were obtained.

[0038] Because the short-term trends in blood vessel diameter and blood flow resistance are more specific in characterizing changes in fatigue state, this invention determines abnormal periods based on abnormal short-term changes in blood vessel diameter and blood flow resistance to reflect the actual fatigue state. Specifically: For time-series data of vessel diameter and blood flow resistance, the rate of change of vessel diameter and blood flow resistance within each time period is obtained using first-order difference. The data is smoothed using a sliding window method to obtain the trend of change of the rate of change within each time window. Abnormal periods in which the trend of change of blood flow resistance or vessel diameter exceeds a preset threshold are located, including the following steps: First, determine the time series data of blood vessel diameter. Time series data of blood flow resistance ,in, t These are time points in a time series.

[0039] Next, the changes in vessel diameter and blood flow resistance were calculated using the first-order difference method. and The rate of change of vessel diameter and the rate of change of blood flow resistance were determined by the time difference between adjacent time points, and the rate of change was smoothed by the sliding window method to obtain the rate of change of vessel diameter. and rate of change of blood flow resistance .

[0040] Finally, the size of the preset time window is determined, and the rate of change of the diameter of the first and last blood vessels and the rate of change of blood flow resistance within the window are obtained to determine the trend of the rate of change. When the trend of change exceeds the preset value, the time window is an abnormal time window, and multiple adjacent abnormal time windows are taken as abnormal time periods.

[0041] Furthermore, facial fatigue features are still obtained using traditional facial video streams, capturing facial motion feature data concurrent with the abnormal time period, including the following steps: S4.1: Obtain multiple facial image frames simultaneously in the abnormal time period of the facial video stream.

[0042] S4.2: For each facial image frame, a facial key point tracking algorithm is used to extract the dynamic changes of facial feature points and obtain facial motion feature data such as blink frequency, number and duration of yawns, changes in head posture angle, and rate of facial expression changes.

[0043] S4.3: For each frame of image, a time point is assigned. The time series data of facial motion feature data of each facial image frame is extracted and smoothed by Kalman filtering.

[0044] The above steps are existing facial landmark tracking methods, and will not be elaborated upon in this invention. Facial temperature data is obtained using an infrared camera.

[0045] Finally, this invention employs a multimodal neural network as the fatigue level recognition model. The multimodal neural network includes four feature extraction modules, a feature fusion module, and a fully connected layer. The four feature extraction modules include three CNN-based and one LSTM-based feature extraction module, which extract features from temperature, blood vessel diameter, blood viscosity change rates, and facial motion data, respectively. These features are then fused using the feature fusion module to obtain fused features. Based on these fused features, the corresponding fatigue level is mapped and output through the fully connected layer. The training set is obtained by labeling physiological and facial data from pilots' daily work processes.

[0046] This invention is based on the joint analysis of an ultrasonic wristband and facial video streams. This method can also dynamically adjust the criteria for fatigue assessment to adapt to differences in physiological responses among individuals, thus improving the personalization and adaptability of fatigue detection. Therefore, this multimodal data fusion method not only improves the accuracy of assessment but also provides a more reliable basis for fatigue intervention and early warning.

[0047] The above is one embodiment of the fatigue monitoring method that integrates facial features and an ultrasonic wristband. Based on the same idea, this embodiment also provides a corresponding fatigue monitoring system that integrates facial features and an ultrasonic wristband. The ultrasonic wristband has ultrasonic transmitting patches and ultrasonic receiving patches arranged at both ends of its inner side.

[0048] The time-series data processing module is used to detect the morphology of blood vessels under different degrees of fatigue through an ultrasonic wristband, and obtain time-series data of two-dimensional blood vessel images; and obtain time-series data of blood vessel diameter based on the two-dimensional blood vessel images.

[0049] The blood flow resistance analysis module is used to determine the time it takes for ultrasound waves to travel through the wrist based on the time it takes for the ultrasound bracelet to receive the echo, and to determine the speed of ultrasound propagation; based on the mathematical model of blood viscosity and ultrasound propagation speed attenuation obtained from experiments, it obtains an estimated value of blood viscosity; and based on the blood vessel diameter and the current estimated value of blood viscosity, it obtains time series data of blood flow resistance changes.

[0050] The abnormal period analysis module uses first-order difference to obtain the rate of change of blood vessel diameter and blood flow resistance in each time period for time series data of blood vessel diameter and blood flow resistance. It uses the sliding window method to smooth the data and obtain the rate of change in each time window. Through the rising trend identification model of the rate of change, it locates abnormal periods when the trend of change of blood flow resistance or blood vessel diameter exceeds the preset threshold.

[0051] The facial motion feature analysis module is used to obtain facial motion feature data that occurs simultaneously with abnormal time periods based on facial video streams under different degrees of fatigue.

[0052] The monitoring model construction module is used to train a multimodal neural network based on the training set to obtain a monitoring model for predicting fatigue state; the training set includes facial movement feature data, temperature, blood vessel diameter and blood viscosity change rates during abnormal periods under different degrees of fatigue state.

[0053] This embodiment also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The provided fatigue monitoring method combines facial and ultrasonic wristband technology.

[0054] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0055] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A fatigue monitoring method integrating facial recognition and an ultrasonic wristband, characterized in that, Includes the following steps: The morphology of blood vessels under different degrees of fatigue was detected by an ultrasound wristband, and time-series data of two-dimensional blood vessel images were obtained; time-series data of blood vessel diameter were obtained from the two-dimensional blood vessel images. The time it takes for ultrasound waves to travel through the wrist is determined by the time the ultrasound bracelet receives the echo, thus determining the speed of ultrasound propagation. Based on the mathematical model of blood viscosity and ultrasound propagation speed attenuation obtained from the experiment, an estimated value of blood viscosity is obtained. Time series data of blood flow resistance changes are obtained based on the blood vessel diameter and the current estimated value of blood viscosity. For time series data of blood vessel diameter and blood flow resistance, first-order difference is used to obtain the rate of change of blood vessel diameter and blood flow resistance in each time period. The sliding window method is used to smooth the data and obtain the trend of change of the rate of change in each time window, and to locate abnormal periods where the trend of change of blood flow resistance or blood vessel diameter exceeds a preset threshold. Based on facial video streams under different degrees of fatigue, facial motion feature data and facial temperature data were obtained concurrently with the abnormal time period. A training set was constructed, including facial motion feature data, facial temperature data, blood vessel diameter change rate data, and blood flow resistance change rate data during abnormal periods under different levels of fatigue. A multimodal neural network was trained to obtain a monitoring model for predicting fatigue state.

2. The fatigue monitoring method combining facial recognition and an ultrasonic wristband as described in claim 1, characterized in that, The process of obtaining time-series data on blood vessel diameter from two-dimensional blood vessel images includes the following steps: Two-dimensional blood vessel images of consecutive frames are grayscaled and denoised to obtain preprocessed two-dimensional blood vessel images; Edge extraction of two-dimensional blood vessel images is performed using a threshold segmentation method, and noise is removed by dilation and erosion to obtain the blood vessel region; The vascular region is refined into a centerline using a skeletonization algorithm, and the vascular diameter is estimated using a radial projection method along the centerline direction. ； In the formula, For the location of blood vessels The diameter at that point For position The distance from the blood vessel wall, The angle along the centerline; The average of the blood vessel diameters at multiple locations is used as the blood vessel diameter at the current time point. D ; Each frame of the two-dimensional blood vessel image corresponds to a time point, and the diameter of the blood vessels in each frame of the two-dimensional blood vessel image is extracted. D We obtained time-series data of blood vessel diameters and smoothed them using Kalman filtering.

3. The fatigue monitoring method combining facial recognition and an ultrasonic wristband as described in claim 1, characterized in that, The process of obtaining an estimated blood viscosity value based on a mathematical model of blood viscosity change and ultrasound propagation velocity attenuation obtained from experiments includes the following steps: The time it takes for ultrasound waves to travel through the wrist is determined by the time the ultrasound bracelet receives the echo, thus obtaining the ultrasound propagation speed. Based on the average propagation speed obtained after the ultrasound bracelet receives the echo under different blood viscosities, a mathematical model of blood viscosity and ultrasound propagation speed attenuation is determined, and the model is fitted experimentally. ； In the formula, The speed of ultrasonic wave propagation. It is a constant. Blood viscosity; The time it takes for ultrasound waves to travel through the wrist is determined by the time the ultrasound bracelet receives the echo, thus determining the speed of ultrasound propagation; blood viscosity is determined by a mathematical model of blood viscosity and the attenuation of ultrasound propagation speed.

4. The fatigue monitoring method combining facial recognition and an ultrasonic wristband as described in claim 3, characterized in that, The process of obtaining time-series data on changes in blood flow resistance based on estimated vessel diameter and current blood viscosity includes the following steps: Estimate the length of the vessel segment to be measured based on the centerline length obtained from the 2D vascular image. ; Based on Poiseuille's law, blood flow resistance is determined by estimating blood viscosity and vessel radius: ; in, It is the radius of the blood vessel; By analyzing the resistance data from multiple measurements, time-series data of changes in blood flow resistance were obtained.

5. The fatigue monitoring method combining facial recognition and an ultrasonic wristband as described in claim 1, characterized in that, The time-series data of vessel diameter and blood flow resistance are processed by using first-order difference to obtain the rate of change of vessel diameter and blood flow resistance within each time period, smoothing the data using a sliding window method and obtaining the trend of change of the rate of change within each time window, and locating abnormal periods where the trend of change of blood flow resistance or vessel diameter exceeds a preset threshold, including the following steps: Time series data for determining blood vessel diameter Time series data of blood flow resistance ,in, t These are time points in a time series; Changes in vessel diameter and blood flow resistance were calculated using the first-order difference method. and The rate of change of vessel diameter and the rate of change of blood flow resistance were determined by the time difference between adjacent time points, and the rate of change was smoothed by the sliding window method to obtain the rate of change of vessel diameter. and rate of change of blood flow resistance ; The size of a preset time window is determined, and the rate of change of the diameter of the first and last blood vessels and the rate of change of blood flow resistance within the window are obtained. The trend of the rate of change is determined. When the trend of change exceeds the preset value, the time window is an abnormal time window, and multiple adjacent abnormal time windows are taken as abnormal time periods.

6. The fatigue monitoring method combining facial recognition and an ultrasonic wristband as described in claim 1, characterized in that, The step of obtaining facial motion feature data concurrent with abnormal time periods from facial video streams under different degrees of fatigue includes the following steps: Acquire multiple facial image frames simultaneously in an abnormal time period from a facial video stream; For each facial image frame, a facial key point tracking algorithm is used to extract the dynamic changes of facial feature points and obtain facial motion feature data such as blink frequency, number and duration of yawns, changes in head posture angle, and rate of change of facial expression. Each frame of the image corresponds to a time point. The time series data of facial motion feature data of each facial image frame is extracted and smoothed by Kalman filtering.

7. The fatigue monitoring method combining facial recognition and an ultrasonic wristband as described in claim 1, characterized in that, The multimodal neural network consists of four feature extraction modules, a feature fusion module, and a fully connected layer. The four feature extraction modules include three CNN-based feature extraction modules and one LSTM-based feature extraction module, which extract features from the rate of change of temperature, blood vessel diameter, and blood viscosity, as well as facial motion feature data. The feature fusion module then performs feature fusion to obtain fused features. Based on the fused features, the corresponding fatigue level is mapped and output through the fully connected layer.

8. A fatigue monitoring system integrating facial recognition and an ultrasonic wristband, characterized in that, The system includes: An ultrasonic wristband, with ultrasonic transmitting patches and ultrasonic receiving patches respectively arranged at both ends of its inner side; The time-series data processing module is used to detect the morphology of blood vessels under different degrees of fatigue through an ultrasonic wristband, and obtain two-dimensional time-series data of blood vessels; and obtain time-series data of blood vessel diameter based on the two-dimensional blood vessel images. The blood flow resistance analysis module is used to determine the time it takes for ultrasound waves to travel through the wrist based on the time it takes for the ultrasound bracelet to receive the echo, and to determine the speed of ultrasound propagation; based on the mathematical model of blood viscosity and ultrasound propagation speed attenuation obtained from experiments, it obtains an estimated value of blood viscosity; and based on the blood vessel diameter and the current estimated value of blood viscosity, it obtains time series data of blood flow resistance changes. The abnormal time period analysis module uses first-order difference to obtain the rate of change of blood vessel diameter and blood flow resistance in each time period for time series data of blood vessel diameter and blood flow resistance. It uses the sliding window method to smooth the data and obtain the rate of change in each time window. Through the rising trend identification model of the rate of change, it locates abnormal time periods when the trend of change of blood flow resistance or blood vessel diameter exceeds the preset threshold. The facial motion feature analysis module is used to obtain facial motion feature data and facial temperature data in sequence with abnormal time periods based on facial video streams under different degrees of fatigue. The monitoring model construction module is used to train a multimodal neural network based on the training set to obtain a monitoring model for predicting fatigue state; the training set includes facial movement feature data, temperature, blood vessel diameter and blood viscosity change rates during abnormal periods under different degrees of fatigue state.

Images