Method and equipment for continuous fatigue monitoring by using intelligent equipment

Abstract

Methods, apparatus, and processor-readable media are described for predicting a fatigue level (270) of a user based on a photoplethysmogram, PPG, signal. In various examples, a device-side method is described. A pair of valid PPG segments (230) is acquired from a continuous PPG signal (210). A plurality of PPG features are extracted for each active PPG fragment (230) in the active PPG fragment pair. Then, a plurality of paired features (640) are extracted from the plurality of PPG features of each valid PPG segment (230), and a fatigue level (270) is predicted from the paired features (640). Optionally, the predicted fatigue level (270) is compared to predefined criteria, and a fatigue alert (290) is issued to the user in accordance with the comparison. In an example, the disclosed method may help overcome challenges related to fatigue prediction in a real environment where the PPG signal (210) may be noisy and affected by subject variability.

Description

Technical Field

[0001] The present invention relates to the field of continuous biological signal monitoring using smart devices, and more particularly to a system and method for monitoring fatigue. Background Art

[0002] Fatigue can be a persistent feeling of physical and / or mental tiredness or weakness in an individual. There are many factors that can cause fatigue, including but not limited to: disease, lifestyle, diet, work style, and psychology. For example, many pathological conditions such as thyroid disease, heart disease, or diabetes can cause fatigue in patients. A lifestyle that includes poor sleep habits (such as not getting enough sleep or sleeping too much) may cause excessive daytime tiredness. Alcohol or drug abuse may stimulate the nervous system and disrupt sleep patterns, thereby causing fatigue. In many cases, work conditions or requirements, including the need to continuously focus on repetitive tasks, may also cause fatigue. For example, long-distance drivers who drive for long periods of time will often feel tired. Other factors that cause fatigue may include long working hours, physical and mental overload, and irregular work schedules (such as shift work). Finally, individuals experiencing anxiety, depression, or high stress also commonly report chronic fatigue.

[0003] Fatigue can negatively impact an individual's physical or mental performance in a variety of ways. Fatigue can increase the likelihood that an individual will lose focus, feel drowsy, have headaches or dizziness, suffer impaired coordination (e.g., hand-eye coordination), and have impaired decision-making. Fatigued individuals are less efficient at performing tasks and are less productive. In addition, fatigue significantly increases the risk of workplace accidents and injuries, such as those related to driving or other vehicle operation, machine operation, construction, or mining. Traffic accidents are the leading cause of death, with an estimated 1.3 million deaths each year, and fatigue is responsible for up to 20% of road accidents.

[0004] Therefore, it would be useful to provide a solution that enables continuous fatigue monitoring of individuals in a real-world setting. Summary of the invention

[0005] The present invention describes devices, methods, and processor-readable media for predicting a user's fatigue level based on a photoplethysmogram (PPG) signal. In various examples, the present invention describes a device-side method. A valid PPG segment pair is obtained from a continuous PPG signal. A plurality of PPG features are extracted for each valid PPG segment in the valid PPG segment pair. Then, a plurality of paired features are extracted from the plurality of PPG features for each valid PPG segment, and a fatigue level is predicted based on the paired features. Optionally, the predicted fatigue level is compared with a predefined criterion, and a fatigue warning is issued to the user based on the comparison. In an example, the disclosed method can help overcome challenges associated with fatigue prediction in a real environment, where the PPG signal may be noisy and subject to subject variability.

[0006] The disclosed system and method can use valid PPG signals that are not contaminated by motion artifacts (MA) or other noise for fatigue level prediction. This can improve fatigue prediction performance in real-world conditions where the user may be moving and performing daily activities.

[0007] In various examples, the present invention provides a technical effect of predicting fatigue level for a user based on comparison of PPG data between users. Input obtained from a PPG sensor is validated to remove motion artifacts (MA) or other noise, and PPG features related to the user's heart rate and heart rate variability are input into a progressive feature combination to predict the user's fatigue level.

[0008] In an exemplary aspect, the present invention describes a method for predicting a user's fatigue level. The method includes: obtaining a valid PPG segment pair from a PPG signal; extracting a plurality of PPG features of each valid PPG segment in the valid PPG segment pair; extracting a plurality of paired features from the plurality of PPG features of each valid PPG segment; and predicting the user's fatigue level based on the paired features.

[0009] In the above exemplary aspects of the method, obtaining a valid PPG segment pair from the PPG signal includes: extracting an original PPG segment pair from the PPG signal; preprocessing each original PPG segment in the original PPG segment pair to generate a processed PPG segment pair; and verifying each processed PPG segment in the processed PPG segment pair using a trained PPG signal verifier to obtain the valid PPG segment pair.

[0010] In any of the above exemplary aspects of the method, extracting multiple PPG features for each valid PPG segment in the valid PPG segment pair includes: detecting multiple peaks in each valid PPG segment in the valid PPG segment pair; acquiring multiple NN intervals for each valid PPG segment in the valid PPG segment pair based on the corresponding multiple peaks; and extracting multiple PPG features for each valid PPG segment in the valid PPG segment pair based on the corresponding multiple NN intervals.

[0011] In some exemplary aspects of the method, preprocessing each raw PPG segment in the raw PPG segment pair to generate a processed PPG segment pair includes: filtering each raw PPG segment in the raw PPG segment pair using a bandpass filter with a bandpass frequency of 0.6-8.0 Hz; and normalizing each raw PPG segment in the raw PPG segment pair so that the mean is zero and the standard deviation is 1.

[0012] In some exemplary aspects of the method, the trained PPG signal verifier is a multi-layer perceptron network (MLP) based on fast Fourier transform (FFT).

[0013] In some exemplary aspects of the method, the trained PPG signal verifier is a temporal convolutional network (TCN).

[0014] In any of the above exemplary aspects of the method, predicting the fatigue level includes: inputting the paired features into a progressive feature search tree to obtain a predicted fatigue level, the predicted fatigue level including an odds ratio and a probability of experiencing a change in fatigue level.

[0015] In the above exemplary aspect of the method, before inputting the pair of features into the progressive feature search tree: constructing the progressive feature search tree by calculating one or more relationships between corresponding one or more pair of features and corresponding fatigue levels.

[0016] In the above exemplary aspects of the method, the one or more relationships between the corresponding one or more pairs of features and the corresponding fatigue levels are calculated using Fisher's exact test.

[0017] In the above-mentioned exemplary aspect of the method, the corresponding one or more paired features correspond to corresponding hierarchical levels, and the progressive feature search tree is constructed by progressively calculating the one or more relationships between the corresponding one or more paired features and the corresponding fatigue levels according to the corresponding hierarchical levels.

[0018] In some exemplary aspects of the method, the method further includes: before extracting the plurality of pairwise features from the plurality of PPG features of each valid PPG segment: classifying the valid PPG segment pairs using a trained differential network, the classification describing the similarity between the valid PPG segment pairs; and in response to the valid PPG segment pairs being classified as having low similarity, extracting the plurality of pairwise features from the plurality of PPG features of each valid PPG segment in the valid PPG segment pairs.

[0019] In any of the above exemplary aspects of the method, the method further includes: comparing the fatigue level with a predefined standard; and issuing a fatigue warning to the user based on the comparison.

[0020] In an exemplary aspect, the present invention describes a system. The system includes: a PPG sensor; one or more memories storing executable instructions; one or more processors coupled to the PPG sensor and the one or more memories, wherein the executable instructions are used to cause the one or more processors to perform the following operations: obtaining a valid PPG segment pair from a PPG signal; extracting a plurality of PPG features of each valid PPG segment in the valid PPG segment pair; extracting a plurality of paired features from the plurality of PPG features of each valid PPG segment; and predicting a user's fatigue level based on the paired features.

[0021] In the above exemplary aspects of the system, when executed by the one or more processors to obtain a valid PPG segment pair from the PPG signal, the executable instructions further cause the system to perform the following operations: extract an original PPG segment pair from the PPG signal; pre-process each original PPG segment in the original PPG segment pair to generate a processed PPG segment pair; and verify each processed PPG segment in the processed PPG segment pair using a trained PPG signal verifier to obtain the valid PPG segment pair.

[0022] In any of the above exemplary aspects of the system, when executed by the one or more processors to extract multiple PPG features for each valid PPG segment in the valid PPG segment pair, the executable instructions further cause the system to perform the following operations: detect multiple peaks in each valid PPG segment in the valid PPG segment pair; obtain multiple NN intervals for each valid PPG segment in the valid PPG segment pair based on the corresponding multiple peaks; and extract multiple PPG features for each valid PPG segment in the valid PPG segment pair based on the corresponding multiple NN intervals.

[0023] In any of the above-mentioned exemplary aspects of the system, when executed by the one or more processors to predict the fatigue level, the executable instructions also cause the system to perform the following operations: input the paired features into a progressive feature search tree to obtain a predicted fatigue level, wherein the predicted fatigue level includes an odds ratio and a probability of experiencing a change in fatigue level.

[0024] In the above-mentioned exemplary aspects of the system, when executed by the one or more processors, the executable instructions also cause the system to perform the following operations: before inputting the paired features into the progressive feature search tree: constructing a progressive feature search tree by calculating one or more relationships between the corresponding one or more paired features and the corresponding fatigue levels, and the one or more relationships between the corresponding one or more paired features and the corresponding fatigue levels are calculated using Fisher's exact test.

[0025] In the above exemplary aspects of the system, when executed by the one or more processors, the executable instructions further cause the system to perform the following operations: before extracting the plurality of pairwise features from the plurality of PPG features of each valid PPG segment: classifying the valid PPG segment pairs using a trained differential network, the classification describing the similarity between the valid PPG segment pairs; in response to the valid PPG segment pairs being classified as having low similarity, extracting the plurality of pairwise features from the plurality of PPG features of each valid PPG segment in the valid PPG segment pairs.

[0026] In the above exemplary aspects of the system, when executed by the one or more processors, the executable instructions further cause the system to perform the following operations: compare the fatigue level with a predefined standard; and issue a fatigue warning to the user based on the comparison.

[0027] In some aspects, the present invention describes a non-transitory computer-readable medium having machine-executable instructions stored thereon, which, when executed by one or more processors of a computing system, cause the computing system to perform the following operations: obtain valid PPG segment pairs from a PPG signal; extract multiple PPG features for each valid PPG segment in the valid PPG segment pairs; extract multiple paired features from the multiple PPG features for each valid PPG segment; and predict a user's fatigue level based on the paired features. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Reference will now be made by way of example to the accompanying drawings which show exemplary embodiments of the present application, in which:

[0029] Figure 1 A block diagram of an exemplary hardware structure of a computing system provided for an example of the present invention, the computing system being usable to implement a method for predicting fatigue level;

[0030] Figure 2 A block diagram of an exemplary architecture of a fatigue prediction system provided for an exemplary embodiment of the present invention;

[0031] Figure 3 A block diagram of an exemplary architecture of a PPG pre-processor provided for exemplary embodiments of the present invention;

[0032] Figure 4A A schematic diagram of an exemplary continuous raw PPG signal and raw PPG segments associated with respective time steps in a series of time steps provided for an exemplary embodiment of the present invention;

[0033] Figure 4B Schematic diagram of an exemplary processed PPG segment and a corresponding PPG segment provided for an exemplary embodiment of the present invention;

[0034] Figure 5 A block diagram of an exemplary architecture of a PPG feature extractor provided for exemplary purposes of the present invention;

[0035] Figure 6 A block diagram of an exemplary architecture of a predictor provided for examples of the present invention;

[0036] 7A to 7D A schematic diagram of an exemplary heart rate pattern provided for an example of the present invention;

[0037] Figure 8 An exemplary progressive feature search tree that may be used is provided for examples of the present invention;

[0038] Fig. 9A schematic diagram of exemplary continuous raw PPG signals associated with respective time steps in a series of time steps and corresponding PPG segments forming two consecutive PPG segment pairs provided for exemplary embodiments of the present invention;

[0039] Fig.10 A flow chart of steps of an exemplary method 1000 for predicting fatigue level 270 is provided for an example of the present invention. DETAILED DESCRIPTION

[0040] The exemplary technical solutions of the present invention are described below in conjunction with the accompanying drawings.

[0041] The present invention describes devices, methods, and processor-readable media for predicting a user's fatigue level based on a photoplethysmogram (PPG) signal. In various examples, the present invention describes a device-side method. A valid PPG segment pair is obtained from a continuous PPG signal. A plurality of PPG features are extracted for each valid PPG segment in the valid PPG segment pair. Then, a plurality of paired features are extracted from the plurality of PPG features for each valid PPG segment, and a fatigue level is predicted based on the paired features. Optionally, the predicted fatigue level is compared with a predefined criterion, and a fatigue alert is issued to the user based on the comparison. In an example, the disclosed method can help overcome challenges associated with fatigue prediction in a real environment, where the PPG signal may be noisy and subject to subject variability. The disclosed system can also help overcome challenges associated with fatigue monitoring performance using smart devices (e.g., using a smart watch).

[0042] To help understand the present invention, some prior art techniques for fatigue monitoring are now discussed.

[0043] In order to reduce the risk of fatigue, it is important to design an automatic system that can continuously monitor the fatigue level for fatigue management. Existing fatigue monitoring methods can be roughly divided into two categories: (1) camera-based methods and (2) smartwatch-based methods.

[0044] Camera-based approaches use computer vision techniques to extract behavioral signs such as yawning and eyelid closure from video streams to detect fatigue. Vehicle manufacturers have deployed camera-based approaches to detect driver fatigue. However, the use of cameras in these approaches introduces several limitations: The first limitation includes the usage scenario. The camera needs to be installed in a fixed position, and fatigue cannot be continuously monitored when the user performs various tasks in different positions. Another limitation includes the difficulty of cameras in performing early detection. Often, users show physical or behavioral signs of fatigue such as yawning and eyelid closure after fatigue has already occurred. The manifestation of physical or behavioral signs of fatigue may indicate that the user has experienced a high level of fatigue. Therefore, detecting physical or behavioral signs may be performed too late to avoid fatigue-related risks. Another limitation includes the sensitivity of computer vision techniques to environmental factors. The detection performance of behavioral signs of fatigue can be affected by environmental factors such as light and reflections from glasses. Finally, camera-based approaches introduce privacy issues. Users may request to turn off the fatigue monitoring system to address privacy concerns.

[0045] Smartwatch-based methods collect physiological signals from the user's wrist to detect fatigue. The collected physiological signals may include a photoplethysmogram (PPG). The PPG signal is obtained from an optical sensor, which consists of a light emitter and a detector, wherein the light emitter continuously emits light to the skin and the detector absorbs the reflected light. As the blood volume changes during the cardiac cycle, the amount of light absorbed also changes, so the peak of the PPG signal corresponds to the cardiac cycle. The interval between two peaks reflects the interval between two heartbeats. Therefore, the PPG signal can be used to extract heart rate (i.e., HR) and heart rate variability (i.e., HRV). Since fatigue causes changes in HR and HRV, wearable devices can detect fatigue early. Wearable devices can achieve continuous fatigue monitoring in an unobtrusive manner. Therefore, the smartwatch-based method can overcome some of the challenges associated with the camera-based method described previously.

[0046] However, existing smartwatch-based fatigue prediction methods are developed using PPG signals collected from controlled environments in laboratory settings. Therefore, existing smartwatch-based methods perform poorly on real-world data. There may be several reasons for the poor performance: First, PPG signals are noisy. Motion artifacts (MA) may introduce noise into PPG signals, thereby interfering with the estimation of heart rate (HR) and heart rate variability (HRV) and affecting fatigue prediction. Second, fatigue is not the only factor that causes changes in PPG signals. PPG signals may be affected by many other factors, such as emotions and pathological factors, and it may be difficult to distinguish PPG changes directly caused by fatigue. Finally, HR and HRV vary significantly between individuals, so methods that predict fatigue based on HR or HRV information for one individual may perform poorly for another individual.

[0047] The present invention describes examples that may help address some or all of the above-mentioned shortcomings of the prior art.

[0048] In the present invention, "fatigue level" may refer to a measure or level of fatigue experienced by an individual, for example, a level based on the Stanford Sleepiness Scale.

[0049] In the present invention, "Stanford Sleepiness Scale (SSS)" refers to a self-assessment questionnaire for measuring an individual's fatigue or sleepiness level or alertness level. The individual classifies his or her fatigue level into 1 to 7 levels according to the following descriptions: 1-"Feeling active, energetic, alert or fully awake."; 2-"Functioning at a high level, but not optimal; able to concentrate."; 3-"Awake but relaxed; responsive but not fully alert."; 4-"A little dazed, depressed."; 5-"Confused; lost interest in staying awake; slow to react."; 6-"Sleepy, dizzy, resisting sleepiness; wanting to lie down."; 7-"No longer resisting sleepiness, falling asleep quickly; not conscious."

[0050] In the present invention, "cardiac cycle" may refer to: the heart activity from the start of one heartbeat to the start of the next heartbeat. The cardiac cycle consists of two phases: the first phase, the heart is filled with blood (such as diastole); the second phase, the heart contracts and pumps blood into the circulatory system (such as systole). During each cardiac cycle, pressure pulses related to the contraction of the heart's chambers can be detected in the circulatory system.

[0051] In the present invention, a "photoplethysmogram (PPG)" is a plethysmogram obtained using optical techniques. In an example, a PPG can be obtained using a pulse oximeter, which measures changes in the blood volume circulating in the subcutaneous tissue of an individual by illuminating the skin and measuring changes in light absorption. Due to changes in blood volume in the skin during each cardiac cycle, changes in the amount of light detected by the photodiode can be observed. In an example, the changes observed in each cardiac cycle may resemble a periodic function, where a peak in the amplitude of the change (followed by a valley) can be sampled and recorded as a time series PPG signal.

[0052] In the present invention, "heart rate (HR)" may refer to: the number of heart beats of an individual measured over a period of time. For example, HR may be expressed as the number of heart beats measured per minute (eg, beats / minute or bpm).

[0053] In the present invention, "inter-beat interval (IBI)" or "NN interval" may refer to: the duration between each heartbeat. IBI is usually measured as the interval between R peaks (RR) in the QRS complex of the traditional electrocardiogram (ECG) heartbeat waveform or the interval between peaks in the PPG signal, or the time interval (NN) between normalized R peaks in the PPG signal.

[0054] In the present invention, "heart rate variability (HRV)" may refer to: a measurement of the variation in the time intervals between consecutive heartbeats, or simply the variation in IBI. In an example, HRV may be affected by a variety of factors, including fatigue, emotion, exercise, or pathological factors.

[0055] In the present invention, a "PPG segment" may refer to a subset of a PPG signal that has been extracted or "clipped" from a larger or continuous data stream. In some examples, a PPG segment may be a portion of an original or processed PPG signal. In some examples, a PPG segment may have a duration of 5 minutes, or a PPG segment may have a longer or shorter duration. In some examples, a PPG segment may be an original PPG segment (e.g., a subset of an original PPG signal), a processed PPG segment (e.g., a subset of a processed PPG signal), or a valid PPG segment (e.g., a subset of a processed PPG signal that has been classified as a valid PPG signal).

[0056] In the present invention, "a pair of PPG segments" or "PPG segment pair" may mean that the two PPG segments correspond to two different time steps in a series of time steps. In an example, each PPG segment in the PPG segment pair may include an identifier or a label to efficiently distinguish one PPG segment from another PPG segment. In an example, the PPG segment pair may be a labeled PPG segment pair, and may be labeled with true fatigue level information for model training. In an example, a pair of PPG segments may be used to assess the fatigue level of the user, for example, using information obtained from each PPG segment in the pair of PPG segments, including NN intervals, PPG features, or paired features, etc.

[0057] In the present invention, a "NN interval series" may refer to a set of NN intervals extracted from a valid PPG segment and concatenated into a time series having the same duration (eg, 5 minutes) as the PPG segment.

[0058] In the present invention, "PPG feature" may refer to: a feature that describes the characteristics of a PPG signal within a specified duration. For example, the PPG feature may be related to HR (e.g., including maximum HR, minimum HR, average HR, etc.), or may be related to HRV (e.g., including the percentage of NN interval differences greater than a threshold, the RMS of the square of the NN interval, the standard deviation of the NN interval, or the standard deviation of a series of NN intervals, etc.). In the present invention, the percentage of NN interval differences greater than a threshold may represent: in a series of NN intervals, the number of adjacent NN interval pairs that differ by more than a specified threshold (e.g., 50 ms) relative to the total number of adjacent NN interval pairs in the series of NN intervals.

[0059] In the present invention, "paired features" may refer to features that describe the change of PPG features between two time steps. For example, the paired features may include the change of the average HR between time step t and time step t-1, the change of the NN interval or NN interval difference between time step t and time step t-1, or the change of the minimum and maximum HR between time step t and time step t-1 that is visually similar to a defined heart rate pattern, etc.

[0060] In the present invention, "Fisher's exact test" is a statistical significance test used with categorical data to test the significance of the association (or chance) between two categorical variables. In the example, Fisher's test uses a contingency table to summarize the relationship between categorical variables, for example, by displaying the frequency distribution of the variables.

[0061] Other terms used in the present invention may be introduced and defined in the following description.

[0062] Figure 1A block diagram of an exemplary hardware structure of a computing system 100 provided for examples of the present invention, which may be used to implement a method for predicting a fatigue level. Examples of the present invention may be implemented in other computing systems, which may include components different from those discussed below. The computing system 100 may be used to execute instructions for predicting a user's fatigue level 270 using any of the examples described herein. The computing system 100 may also be used to train blocks of the fatigue prediction system 200, or blocks of the fatigue prediction system 200 may be trained by another computing system.

[0063] Although Figure 1 A single instance of each component is shown, but multiple instances of each component may exist in the computing system 100. In addition, although the computing system 100 is shown as a single block, the computing system 100 can be a single physical machine or device (e.g., implemented as a single computing device, such as a single workstation, a single end-user device, a single server, etc.), and can include mobile communication devices (smartphones), laptops, tablet computers, desktop computers, vehicle driver assistance systems, wearable devices (e.g., smart watches), assistive technology devices, medical diagnostic devices, virtual reality devices, augmented reality devices, and Internet of Things (IoT) devices, etc.

[0064] The computing system 100 includes at least one processor 102, such as a central processing unit, a microprocessor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated logic circuit, a dedicated artificial intelligence processor unit, a graphics processing unit (GPU), a tensor processing unit (TPU), a neural processing unit (NPU), a hardware accelerator, or a combination thereof.

[0065] The computing system 100 may include input / output (I / O) interfaces 104 that may support connections with input devices 106 and / or optional output devices 110. In the example shown, the input devices 106 (e.g., keyboard, mouse, microphone, touch screen, and / or keypad) may also include a PPG sensor 108. In the example shown, the optional output devices 110 (e.g., display, speaker, and / or printer) are shown as optional and external to the computing system 100. In other exemplary embodiments, there may not be any input devices 106 and output devices 110, in which case the I / O interfaces 104 may not be required.

[0066] The computing system 100 may include an optional communication interface 112 for wired or wireless communication with other computing systems (e.g., other computing systems in a network). The communication interface 112 may include a wired link (e.g., an Ethernet cable) and / or a wireless link (e.g., one or more antennas) for communication within and / or between networks.

[0067] The computing system 100 may include one or more memories 114 (collectively referred to as "memory 114"), which may include volatile or non-volatile memory (e.g., flash memory, random access memory (RAM) and / or read-only memory (ROM). The non-transitory memory 114 may store instructions executed by the processor 102, for example, to perform the examples described in the present invention. For example, the memory 114 may store instructions for implementing any of the networks and methods disclosed herein. The memory 114 may include other software instructions, for example, to implement an operating system (OS), and other applications / functions. The instructions may include instructions for implementing and operating the following references. Figure 2 Instructions 200 -I of fatigue prediction system 200 are described.

[0068] The memory 114 may also store other data 116, information, rules, policies, and machine executable instructions described herein, including PPG features, such as the PPG features at the current time step t. t 250, or the PPG feature of the previous time step t–1 t–1 255, these features can be generated by the fatigue prediction system 200 and stored in the memory 114. The memory 114 can also store the NN interval sequence including the current time step t t 530 and the NN interval sequence of the previous time step t–1 t–1 535 , which may have been generated by the fatigue prediction system 200 .

[0069] In some examples, computing system 100 may also include one or more electronic storage units (not shown), such as a solid-state drive, a hard disk drive, a magnetic disk drive, and / or an optical disk drive. In some examples, data and / or instructions may be provided by an external memory (e.g., an external drive in wired or wireless communication with computing system 100), or by a transient or non-transient computer-readable medium. Examples of non-transient computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable memory. Storage units and / or external memory may be used in conjunction with memory 114 to implement data storage, retrieval, and caching functions of computing system 100. For example, components of computing system 100 may communicate with each other via a bus.

[0070] Figure 2 A block diagram of an exemplary architecture of a fatigue prediction system 200 is provided for an exemplary embodiment of the present invention.

[0071] In some examples, the fatigue prediction system 200 may receive the raw continuous PPG signal 210 and output a predicted fatigue level 270 for the user. In an example, an electronic device (e.g., a smart watch fixed on the wrist of the individual and in contact with the skin of the individual) may be used to monitor the continuous biosignal (e.g., PPG signal) of the individual. In an example, the smart watch may include a PPG sensor 108 (e.g., a pulse oximeter) for measuring the raw continuous PPG signal 210. In an example, when acquiring the raw continuous PPG signal 210, the PPG sensor 108 may emit light close to the user's skin and may detect absorbed or reflected light from the skin. In an example, the intensity of the recorded raw PPG signal representing the change in blood volume in the subcutaneous tissue may be observed to vary with each heartbeat.

[0072] Although the methods and apparatus are described herein in connection with the use of a smartwatch, it should be understood that other wearable electronic devices may also be used to monitor and record an individual's PPG signals.

[0073] In an example, the PPG preprocessor 220 of the fatigue prediction system 200 may receive the original continuous PPG signal 210 and may output a valid PPG segment corresponding to a corresponding time step t t 230, as follows Figure 3 In this example, the valid PPG fragment t230 may be a pre-processed PPG signal having a predefined duration, which may be used to predict fatigue of the user.

[0074] Figure 3 A block diagram of an exemplary architecture of the PPG preprocessor 220 is provided for an exemplary embodiment of the present invention. In the example, the PPG segment extractor 310 of the PPG preprocessor 220 receives the continuous raw PPG signal 210 and extracts the raw PPG segments corresponding to the corresponding time step t. t 320. In the example, the original PPG segment t 320 may represent a portion of the continuous raw PPG signal 210 having a predetermined duration (eg, a duration of 5 minutes) that has been extracted or “clipped” from the raw continuous PPG signal 210 at a corresponding time step t.

[0075] Figure 4A A schematic diagram of an exemplary continuous raw PPG signal 210 and raw PPG segments 320 associated with a corresponding time step t in a series of time steps provided for an exemplary embodiment of the present invention. In an example, the raw PPG segments 320a, 320b, 320c ... 320n may be recorded at regular time step intervals 410 corresponding to the corresponding time steps t, t+1, t+2 ... t+n. In an example, each raw PPG segment 320 may have a predefined segment duration 420 corresponding to the corresponding time step t. In some embodiments, for example, the time step interval 410 may be 15 minutes and the predefined segment duration 420 may be 5 minutes.

[0076] Return to Figure 3 , the original PPG fragment can be t 320 is input to processor 330 to generate a processed PPG segment t 340. It should be appreciated that the PPG signal may be noisy, for example due to motion artifacts (MA), for example caused by ambient light leaking into the gap between the PPG sensor 108 and the user's skin surface. In this regard, the raw PPG segments t 320 may include a plurality of valid PPG signals with clear peaks and a plurality of invalid PPG signals contaminated by motion artifacts. In an example, processor 330 may process the raw PPG segment t 320 to help remove some of the noise associated with MA and other noise sources in the signal. In an example, processor 330 can convert the original PPG segment t 320 is divided into a plurality of non-overlapping PPG segments, each segment having a predetermined segment duration. In some embodiments, for example, the original PPG segment t320 can be segmented into 60 non-overlapping PPG segments, each of which has a duration of 5 seconds. In an example, the processor 330 can process the raw PPG segments, for example, using a bandpass filter with a bandpass frequency of 0.6-8.0 Hz. t 320. In an example, processor 330 may then process the original PPG segment t Each filtered 5-second PPG segment of 320 was normalized to have a mean of 0 and a standard deviation of 1.

[0077] In the example, the processed PPG segment can be t 340 is input to the PPG signal validator 350 to convert the processed PPG fragments t Each 5-second PPG segment of 340 is classified as valid or invalid, and a valid PPG segment for fatigue prediction is generated t 230.

[0078] Figure 4B Example processed PPG segments provided for example embodiments of the present invention t 340 and a schematic diagram of a corresponding PPG segment. In an example, each PPG segment may be classified as a valid PPG segment 430 or an invalid PPG segment 440, wherein a PPG segment may be considered a valid PPG segment 430 if the PPG segment satisfies a set of criteria for classifying a PPG segment as valid. In an example, the criteria for classifying a PPG segment as a valid PPG segment 430 may include: (1) having clear peaks in the PPG segment signal, (2) the duration between peaks is consistent with a typical heart rate (i.e., 0.5 Hz–3 Hz). In an example, an invalid PPG segment 440 may represent a PPG signal that has been contaminated by MA and may exhibit invisible peaks that follow a typical heart rate frequency, etc. In some embodiments, for example, only valid PPG signals may be used to predict fatigue, while invalid PPG signals may be discarded. In some embodiments, for example, the PPG signal validator 350 may be configured to identify the PPG segments from the processed PPG segments by connecting the valid PPG segments 430 and discarding the invalid PPG segments 440. t 340 Generate valid PPG fragment t 230. For example, according to the number of invalid PPG segments 440 that are discarded, the valid PPG segments t The duration of 230 can be longer than the processed PPG segment t 340 short.

[0079] In an example, the PPG signal validator 350 may be a trained machine learning model, such as a signal classifier. In an example, the PPG signal validator 350 may be trained by supervised machine learning using a labeled dataset. In an example, when combining the training dataset, a collection of 35,800 PPG segments (e.g., 5-second PPG segments) was manually inspected and labeled as "valid" or "invalid" based on the quality of the segment. For example, of the 35,800 labeled PPG segments, 15,800 were labeled as "valid" and the remaining 20,000 were labeled as "invalid".

[0080] In one embodiment, for example, the PPG signal validator 350 can be a Fourier transform-based multi-layer perception network (FFT). In an example, the FFT method uses a fast Fourier transform to convert the PPG segment from the time domain to the frequency domain and derive the frequency power of the PPG segment. In an example, the frequency power can then be fed into a trained multi-layer perception network for classifying the signal as a valid PPG segment 430 or an invalid PPG segment 440.

[0081] In another embodiment, for example, the PPG signal validator 350 may be a temporal convolutional network (TCN). In an example, the TCN method may input each PPG segment of the processed PPG segment 340 into a trained temporal convolutional network as a valid PPG segment 430 or an invalid PPG segment 440 for signal classification.

[0082] In the example, both the FFT and TCN classification methods show good performance in accurately predicting the valid PPG segments 430. Both the FFT and TCN methods can classify 5 seconds of samples in less than 0.1 ms. Although the TCN method shows slightly higher accuracy than the FFT method, the FFT method is much more efficient because the PPG segments in the frequency domain have lower dimensions compared to the PPG segments in the time domain.

[0083] Return to Figure 2 , the effective PPG fragment can be t 230 is input into the PPG feature extractor 240 to obtain a set of PPG features at the current time step t t 250, as follows Figure 5 described.

[0084] Figure 5 A block diagram of an exemplary architecture of the PPG feature extractor 240 is provided for exemplary embodiments of the present invention. In the example, the valid PPG segment can be t230 is input to the PPG feature extractor 240, and can be used to extract the valid PPG fragments. t 230 Perform peak detection 510 to identify valid PPG segments t 230. For example, in some embodiments, the peak value can be detected by finding the local maximum in the signal by comparison with adjacent values, or another method can be used. An example of a peak detection algorithm that can be implemented in an exemplary embodiment is described in: Elgendi, Mohamed, et al., "Systolic peak detection in acceleration photoplethysmograms measuredfrom emergency responders in tropical conditions", PloS one 8.10 (2013): e76585, the entire contents of which are incorporated herein by reference. Another example of a peak detection algorithm that can be implemented in an exemplary embodiment is described in: Kazemi, Kianoosh, et al., "Robust PPG Peak Detection Using Dilated Convolutional Neural Networks", Sensors 22.16 (2022): 6054, the entire contents of which are incorporated herein by reference.

[0085] In the example, the detected peaks can be used to identify valid PPG segments. t 230 extracts a sequence of NN intervals 515, where the NN interval represents the duration between consecutive peaks. In an example, the sequence of NN intervals 515 for each valid PPG segment 430 can then be input into a concatenator 520 to generate a sequence of NN intervals corresponding to time step t. t 530, comprising a series of concatenated NN intervals representing a predefined segment duration 420 (e.g., a duration of 5 minutes). To illustrate the action of the concatenator 520, an example is now described. Assuming a valid PPG segment t 230 includes 30 valid PPG segments 430, each of which has a duration of 5 seconds. In this regard, the valid PPG segments t230 may only have a duration of 2.5 minutes, which is shorter than the predefined segment duration 420 of 5 minutes. Therefore, the concatenator 520 may concatenate the sequence of NN intervals 515 corresponding to 30 valid PPG segments 430, and may continue to sequentially append NN intervals 515 corresponding to valid PPG segments 430 until a sequence of NN intervals corresponding to the predefined segment duration 420 (e.g., 5 minutes) is generated. t 530. Can output NN interval sequence t 530 for use by the predictor 260, as described below with respect to Figure 6 Optionally, the NN spacer sequence t 530 can be stored in the NN interval sequence database 560 for retrieval as needed during prediction operations.

[0086] In the example, the NN interval sequence can be t 530 Generate a set of heart rate (HR) features t 540. In the example, the following equation can be used for the NN interval sequence t Each NN interval 515 in 530 calculates the heart rate:

[0087]

[0088] Here, the duration of the NN interval represents the duration in seconds.

[0089] In the example, the HR feature t 540 may include: a maximum HR (Max HR) within a 5-minute duration, a minimum HR (Min HR) within a 5-minute duration, and a mean HR (Mean HR) within a 5-minute duration, etc.

[0090] In the example, the NN interval sequence can be t 530 Generate a set of heart rate variability (HRV) features t 560. In the example, the HRV feature t 550 may include: the percentage of NN interval differences greater than 20ms (PNN20), the percentage of NN interval differences greater than 50ms (PNN50), the square root mean of NN interval differences (RMSSD), the standard deviation of NN interval differences (SDSD), and the standard deviation of the NN interval sequence (SDNN), etc.

[0091] In the example, the HR feature t 540 and HRV characteristics t550 can be combined to form a set of PPG features corresponding to time step t t 250. Optionally, a set of PPG features t 250 may be stored in the PPG signature database 570 for retrieval as needed during prediction operations.

[0092] Figure 6 A block diagram of an exemplary architecture of the predictor 260 provided for an example of the present invention. In an example, the predictor 260 uses a PPG feature pair 625 obtained at two different time steps in a series of time steps to infer changes in HR and HRV features that may indicate changes in fatigue level 270. In an example, the PPG feature pair 625 may include a PPG feature associated with the current time step t t 250 and the PPG features associated with the previous time step t–1 t–1 255.

[0093] In an example, in order to effectively infer the change of fatigue level 270 from PPG feature pair 625, the PPG signals obtained for the user at two different time steps in a series of time steps must show significant differences. In order to confirm that the PPG feature pair 625 shows significant differences, the differential network 610 can receive the NN interval sequence pair 605 corresponding to the PPG feature pair 625, and can output a similarity classification 620, wherein the similarity classification 620 describes the degree of similarity between the NN interval sequence pair 605. In an example, the NN interval sequence pair 605 may include the NN interval sequence associated with the current time step t t 530 and the NN interval sequence associated with the previous time step t–1 t–1 535. In some embodiments, for example, the NN interval sequence can be respectively t 530 and NN interval sequence t–1HR and HRV are extracted in 535 and input into two identical neural networks to learn hidden representations. For example, in some embodiments, the two identical neural networks can each be a trained multilayer perceptron (MLP) network. In the example, the outputs of the two MLP networks can be compared using a contrastive loss 620. An example of a contrastive loss method that can be implemented in an example embodiment is described in: "Siamese neural networks for one-shot image recognition" by Koch, Gregory et al. in the ICML Deep Learning Symposium Vol. 2.2015, the entire contents of which are incorporated herein by reference. In the example, the similarity classification 620 may indicate whether the NN interval sequence pair 605 is "similar" or whether the NN interval sequence pair 605 is "different", where the "different" NN interval sequence pair 605 can be used by the predictor 260 to predict changes in the user's fatigue level 270.

[0094] In the example, the differential network 610 may include two trained machine learning models, such as two MLP networks. In the example, the differential network 610 may be trained by supervised machine learning using a labeled dataset. In the example, when assembling a labeled training dataset, the user may be asked to complete a fatigue level self-assessment (e.g., according to the Stanford Sleepiness Scale) at each time step t in the time step sequence and at each time step t thereafter as ground truth fatigue level information. In the example, the Stanford Sleepiness Scale uses a ranking of 1 to 7 levels, corresponding to the user's sense of fatigue. In the example, 1 point represents that the user feels active, energetic, alert, or fully awake, while 7 points represent that the user no longer struggles to wake up, sleep will soon occur, or wants to dream. Using this approach, we created a training dataset comprising a set of 4540 labeled NN interval sequence pairs from two different time steps, each labeled NN interval sequence pair being labeled with corresponding ground truth fatigue level information. In addition, a subset of 607 labeled NN interval sequence pairs from two different time steps were randomly selected from the training dataset and manually labeled as “similar” or “different” based on three factors: (1) observed HR difference, (2) observed HRV difference, and (3) changes in user-reported fatigue ratings.

[0095] In the example, to train the differentiation network 610, HR and HRV are extracted from each labeled NN interval series pair in a subset of 607 labeled NN interval series pairs, respectively, and fed into the corresponding MLP network of the differentiation network 610. The outputs of the two identical neural networks are compared using a contrastive loss function in order to maximize the difference in the learned hidden representations of the labeled NN interval sequence pairs labeled as "different" and minimize the difference in the learned hidden representations of the labeled NN segment pairs labeled as "similar".

[0096] In an example, once it has been confirmed that the PPG feature pair 625 shows a significant difference, the pairwise feature extractor 630 of the predictor 260 may receive the PPG feature pair 625, including the PPG feature associated with the current time step t in the series of time steps. t 250, and the PPG features associated with the previous time step t–1 t–1 255, and the associated paired features 640 can be extracted. In an example, the paired features 640 may include features describing the change in the value of the PPG feature between two time steps, for example, the change in the average HR between time step t and time step t-1, the change in the NN interval or the NN interval difference, or the change in the minimum and maximum HR visually similar to the defined heart rate pattern 700, as described below with reference to FIG. 7A to FIG. 7D As described. In an example, the paired features 640 may show an increase in value between two time steps or a decrease in value between two time steps. In an example, the increase in the value of the paired features 640 between two time steps may be described by an "UP" pattern descriptor, and the decrease in the value of the paired features 640 between two time steps may be described by a "DOWN" pattern descriptor. In an example, the importance of the paired features 640 may also be represented by classification in one of multiple layers, wherein the first layer paired features 640 may be very important in the prediction of the fatigue level 270, while the fifth layer paired features 640 may be less important in the prediction of the fatigue level 270. A summary of exemplary extracted paired features 640 and related layers is provided in Table 1 below.

[0097] Table 1: Exemplary extracted pairs of features 640, pattern descriptors, and assigned layers

[0098]

[0099]

[0100] FIG. 7A to FIG. 7DSchematic diagram of an exemplary heart rate pattern 700 provided for an example of the present invention. In some embodiments, for example, the heart rate pattern 700 may describe the relationship between HR features 540, such as the difference in minimum HR 710, average HR 720, and maximum HR 730 between two time steps in a series of time steps (e.g., current time step t and previous time step t-1).

[0101] Fig. 7A An “UP” heart rate pattern 700a is shown, where both the minimum HR 710 and the maximum HR 730 increase between time step t−1 and time step t. Figure 7B A “downward” heart rate pattern 700b is shown, where both the minimum HR 710 and the maximum HR 730 decrease between time step t−1 and time step t. Figure 7C A “shrinking” heart rate pattern 700c is shown, where the minimum HR 710 increases and the maximum HR 730 decreases between time step t−1 and time step t. Fig.7D A “zoomed-in” heart rate pattern 700d is shown, where the minimum HR 710 decreases and the maximum HR 730 increases between time step t−1 and time step t.

[0102] Return to Figure 6 , the pair of features 640 may be progressively combined using a progressive feature search tree 650 to predict the fatigue level 270. In the example, the progressive feature search tree 650 is constructed by deriving the relationship between the progressive layers of the pair of features 640 and the fatigue level 270 using Fisher's exact test. An example of a Fisher's test that may be implemented in an exemplary embodiment is described in Fisher, Ronald A, "On the interpretation of χ2 from contingency tables, and the calculation of P", Journal of the Royal Statistical Society, 85.1 (1922): 87-94, the entire contents of which are incorporated herein by reference.

[0103] In an example, to derive a relationship between a given paired feature 640 and a fatigue level 270, a dataset of labeled paired features 640 is obtained, comprising a set of corresponding PPG segment pairs labeled with corresponding ground truth fatigue level information (e.g., fatigue level self-assessments corresponding to two associated time steps). The PPG segment pairs are divided into two groups: (1) a control group, and (2) an experimental group. For a particular value of a given paired feature 640 (e.g., an "up" heart rate pattern), the control group contains labeled PPG segment pairs that display the value of the given paired feature 640, and the experimental group contains all remaining PPG segment pairs in the dataset. The following null hypothesis H0 is tested for a given value of the paired feature 640: "There is no difference in the proportion of PPG segment pairs that experience an increase in fatigue level in the experimental and control groups."

[0104] In the example, the null hypothesis H0 is evaluated using Fisher's exact test with a 99% confidence level (e.g., p-value < 0.01). In the example, an odds ratio (OR) is calculated to measure the likelihood of experiencing an increase in fatigue level in the experimental group. If OR = 1, the likelihood of an increase or decrease in fatigue level in each group is equal. If OR > 1, the PPG segment pairs in the experimental group are more likely to experience an increase in fatigue level, and if OR < 1, the PPG segment pairs in the experimental group are less likely to experience an increase in fatigue level.

[0105] It is now demonstrated that the relationship between a given pair of features 640 (e.g., an "upward" heart rate pattern) and the fatigue level 270 can be derived using Fisher's exact test. In the example, a dataset of labeled pair of features 640 including a set of PPG segment pairs labeled with corresponding ground truth fatigue level information is obtained, and the PPG segment pairs are separated into a control group and an experimental group according to the value of the given pair of features 640. In the example, the number of instances of increased fatigue level and decreased fatigue level in each group is calculated, as shown in Table 2 below.

[0106] Table 2: Contingency table of “upward” heart rate pattern

[0107] Increase fatigue level Reduce fatigue level "Upward" heart rate pattern (control group) 159 77 Others (experimental group) 463 627

[0108] In the example, using Fisher's exact test, the odds ratio for the "UP" heart rate pattern is 2.8, with a significant p-value (p-value = 4.7e-12 < 0.01). The results indicate that when the PPG segment pair exhibits an "UP" heart rate pattern, the individual is 2.8 times more likely to experience a decrease in fatigue level, and the probability of experiencing a change in fatigue level (e.g., a decrease in fatigue level) is 67%.

[0109] In an example, the above method can be repeated for each value of the first layer paired features to further derive the relationship between the paired features 640 and the fatigue level 270. For example, a contingency table can be generated for the remaining values ​​of the heart rate pattern feature, including a "downward" heart rate pattern, a "shrinking" heart rate pattern, and an "expanding" heart rate pattern, and the odds ratio and the probability of experiencing a change in fatigue level can be calculated for each value, respectively. In an example, additional paired features 640 (e.g., second layer paired features) can be progressively combined with the first layer features, and the relationship between the combined features and the fatigue level can be derived using Fisher's exact test. A second example is provided to illustrate the derivation of the relationship between the selected first layer paired features 640 of the "upward" heart rate pattern and the selected second layer paired features 640 of the "upward" average heart rate and the fatigue level 270. In an example, a data set including a set of PPG segment pairs labeled with corresponding ground truth fatigue level information is obtained, and the PPG segment pairs are separated into a control group and an experimental group. In an example, the number of instances of increased fatigue level and decreased level in each group is calculated, as shown in Table 3 below. In the example, using Fisher's exact test, the odds ratio of the layer 1 "up" heart rate pattern signature to the layer 2 "up" mean heart rate signature and the probability of a change in fatigue level can be calculated.

[0110] Table 3: Contingency table of “upward” heart rate pattern and “upward” average heart rate

[0111] Increase fatigue level Reduce fatigue level "Upward" heart rate pattern + "upward" average heart rate (control group) 146 72 Others (experimental group) 476 632

[0112] In an example, the above method may be repeated for each value of each pair of features 640, proceeding sequentially based on the layer, e.g., adding an additional layer 2 pair of features 640, followed by a layer 3 pair of features 640, and so on, in all layers of the pair of features 640 until all combinations are calculated. In this regard, a progressive feature search tree 650 is constructed to query the prediction results of a given PPG segment pair while performing Fisher's exact test on all progressive combinations of features.

[0113] Figure 8An exemplary progressive feature search tree 650 that can be used is provided for examples of the present invention. In the example, the progressive feature search tree 650 includes a plurality of nodes 800 (e.g., represented by circles) arranged in a plurality of layers 805, wherein each layer corresponds to a separate layer of paired features 640 shown in Table 1. Each node represents a progressive pair of feature 640 combination and includes an associated OR and a probability that the user experiences a change in fatigue level. In constructing the progressive feature search tree 650, a maximum layer depth is reached when adding more feature layers does not produce a significant association with a change in fatigue level 270. In the example, the progressive feature search tree 650 is constructed to query the prediction results of a pair of PPG segments. In order to predict the fatigue level 270 based on a pair of PPG segments, the paired features 640 of the pair of PPG segments are sequentially and progressively input into the search tree 650 according to the associated layers until the maximum layer depth, and for the combination of paired features 640 that show a significant association with fatigue, the corresponding OR and the probability of a change in fatigue level 270 for the feature combination are obtained. In the example, if all feature combinations in a pair of PPG segments are not significantly associated with changes in fatigue levels, no prediction can be made for the sample.

[0114] In the example, the progressive feature search tree 650 starts at node 810. Node 810 includes four branches in the first layer of the search tree according to the first layer paired features of "heart rate pattern". In the example, nodes 820, 822, 824, and 826 may correspond to the values ​​of the heart rate pattern including "up", "down", "zoom out", and "zoom in", respectively. In the example, the next layer corresponding to the second layer paired features "average heart rate" may be added, for example, represented by nodes 830 and 832. In the example, node 830 may correspond to the "up" average heart rate, and node 832 may correspond to the "down" average heart rate. In the example, the progressive feature search tree 650 may include other layers, such as shown in nodes 840, 850, and 860. In some embodiments, for example, nodes 840, 850, and 860 may correspond to values ​​associated with the third layer, the fourth layer, and the fifth layer paired features, respectively.

[0115] Return to Figure 6 , the fatigue level 270 may be predicted at the first time step t in a series of time steps, and may be repeated at each time step t thereafter. In an example, each fatigue level 270 prediction may include an odds ratio 272 obtained from the progressive feature search tree 650 and a probability of change in the fatigue level 274. Optionally, the fatigue level 270 prediction may also include a sleep scale indicator 276, which may indicate whether the user's fatigue level 270 relative to the Stanford Sleepiness Scale is trending upward, downward, or remaining the same.

[0116] Return to Figure 2, once the fatigue level 270 has been predicted, the evaluator block 280 may optionally evaluate the fatigue level 270 prediction and determine the risk level associated with the fatigue level 270. In an example, the evaluator may compare the fatigue level 270 with a predefined criterion (e.g., a threshold). In other examples, the evaluator may evaluate the trend of the fatigue level 270 and may compare the trend with a predefined criterion (e.g., a value of a slope or another threshold). In an example, the evaluator 280 may issue a fatigue alert 290 to the user based on the comparison, for example, to draw attention to the predicted fatigue level 270. In an example scenario, the user may demonstrate an increase in fatigue level 270 while working, such as while driving or operating machinery. Accordingly, the first fatigue alert 290a may issue an alert or notification to the user to draw attention to the increased fatigue or sleepiness, or prompt the user to pause and rest, or take an action such as initiating exercise or drinking coffee. In another example scenario, in response to resting or engaging in certain actions after receiving the first fatigue alert 290a, the user may exhibit a reduced fatigue level (e.g., increased alertness). Therefore, a second fatigue alert 290b may be issued to the user, indicating that they can return to work. In another exemplary scenario, the user may exhibit a reduced fatigue level (e.g., increased alertness) during a time when the user should not be alert (e.g., late at night when they should be sleeping). Accordingly, the fatigue alert 290 may issue an alert or notification to the user to draw attention to the increased alertness, or prompt the user to go to bed.

[0117] In another embodiment of the present invention, fatigue level prediction 270 may be performed using two or more pairs of consecutive PPG segments instead of one pair of PPG segments. In an example, a first pair of PPG segments corresponding to time steps t and t-1 in a series of time steps may be used to generate fatigue level prediction. In an example, a second pair of PPG segments corresponding to time steps t+1 and t may be used to further refine or improve fatigue level prediction, for example, by examining changes in HR and HRV between two or more pairs of consecutive PPG segments, as described below with respect to Fig. 9 described.

[0118] Fig. 9 Schematic diagram of exemplary continuous raw PPG signals 210 and corresponding PPG segments 910 forming two pairs of continuous PPG segments 940 for predicting a user's fatigue level 270 provided for an exemplary embodiment of the present invention. In an example, capturing changes in HR and HRV over two pairs of continuous PPG segments 940 may provide an improved predicted fatigue level 270 for the user. In an example, the two pairs of continuous PPG segments 940 may include a first pair of PPG segments 920 and a second pair of PPG segments 930.

[0119] In the example, the first pair of PPG segments 920 includes PPG segments 910a and 910b corresponding to time steps t-1 and t, respectively, in a series of time steps, and the second pair of PPG segments 930 can be obtained, where PPG segments 910b and 910c corresponding to time steps t and t+1, respectively, in the sequence of time steps. In the example, for example, a pair of features 640 can be generated for each pair of PPG segments 920 and 930, as previously described with respect to Figure 6 described.

[0120] In the example, the previous discussion on Figure 8 The method described herein builds a progressive feature search tree 650 starting with the first level paired features 640 (e.g., heart rate patterns) and proceeding to the second level features, and so on, for consecutive pairs of PPG segments 940. In the example, each combination of values ​​of the paired features 640 across two consecutive pairs of PPG segments 940 is examined using Fisher's exact test to assess the degree of association with the fatigue level 270. In the example, Fisher's exact test is performed on the progressively combined paired features 640 of consecutive pairs of PPG segments 940 to build the progressive feature search tree 650. In the example, the method described previously with respect to the fatigue level 270 can then be used. Figure 6 The progressive feature search tree 650 predicts the fatigue level 270 for two pairs of consecutive PPG segments 940 .

[0121] Fig.10 A flow chart of steps of an exemplary method 1000 for predicting fatigue level 270 is provided for an example of the present invention. Method 1000 may be performed by computing system 100. For example, processor 102 may execute computer readable instructions (which may be stored in memory 114) to cause computing system 100 to perform method 1000.

[0122] The method 1000 begins at step 1002, where valid pairs of PPG segments 230 are obtained from the continuous raw PPG signal 210. In an example, when generating valid PPG segment pairs, the raw PPG segment pairs are extracted from the continuous raw PPG signal 210 and pre-processed, for example, to remove motion artifacts or other noise. In an example, the processed PPG segment pairs can be input into the trained PPG signal validator 350 to generate valid PPG segment pairs.

[0123] In step 1004, a plurality of PPG features may be extracted for each valid segment 230 in the valid PPG segment pair. In an example, peak detection 510 may be performed on each valid PPG segment 230 in the valid PPG segment pair to detect a plurality of peaks in each valid PPG segment 230. In an example, a plurality of NN intervals 515 may be obtained for each valid PPG segment 230 in the valid PPG segment pair based on the corresponding plurality of peaks. In an example, a plurality of PPG features may be extracted for each valid PPG segment 230 in the valid PPG segment pair based on the corresponding plurality of NN intervals 515.

[0124] In step 1006 , a plurality of paired features 640 may be extracted from the plurality of PPG features of each valid PPG segment 230 in the pair of valid PPG segments.

[0125] In step 1008, fatigue level 270 may be predicted based on the paired features 640. In an example, the paired features 640 may be input into a progressive feature search tree 650 to obtain fatigue level prediction 270. In an example, the paired features 640 may be progressively input into the progressive feature search tree 650 in sequence according to the associated layers of each paired feature 640, wherein the first layer paired features 640 are input first, then the second layer paired features 640 are input, and so on, until the maximum layer depth. In an example, if a node in the progressive search tree 650 determines that the progressive paired features 640 have a significant association with fatigue, an odds ratio 272 and a probability 274 of experiencing a change in fatigue level may be displayed.

[0126] Optionally, in step 1010 , fatigue level 270 may be compared to a predefined standard, and in step 1012 , fatigue warning 290 may be issued to the user based on the comparison.

[0127] In some examples, fatigue level 270 may be output to an application on an electronic device (e.g., a software application executed by computing system 100) to issue fatigue warning 290 to an individual. In other examples, fatigue level 270 may be output to an application to be executed by an onboard computing system or heavy machinery operating system to serve fatigue warning 290 to an individual operating a vehicle or operating machinery. In situations where an individual operating a vehicle or machinery is expected to experience increasing fatigue, the vehicle or machinery safety system may issue a notification or warning to the operator of the vehicle or machinery to prompt them to take safety action, such as moving the vehicle off the road or shutting down the machinery.

[0128] Although the present invention describes the method and process in a specific order, one or more steps of the method and process may be omitted or modified as appropriate. One or more steps may be performed in sequence, but not in the order described (as appropriate).

[0129] Although the present invention has been described, at least in part, in terms of methods, it will be understood by those skilled in the art that the present invention also relates to various components for performing at least some aspects and features of the described methods by hardware components, software, or any combination of the two. Accordingly, the technical solution of the present invention can be embodied in the form of a software product. Suitable software products can be stored in a pre-recorded storage device or other similar non-volatile or non-transient computer-readable medium, such as a DVD, CD-ROM, USB flash drive, removable hard disk or other storage medium. The software product includes instructions stored thereon, which enable a processing device (such as a personal computer, server or network device) to perform examples of the methods disclosed herein. Machine executable instructions can be in the form of code sequences, configuration information, or other data, which, when executed, cause a machine (e.g., a processor or other processing device) to perform steps in the method according to the examples of the present invention.

[0130] The present invention may be embodied in other specific forms without departing from the subject matter of the claims. The described exemplary embodiments are merely illustrative and not restrictive in all respects. Selected features from one or more of the above embodiments may be combined to create alternative embodiments not explicitly described, and it is understood that features suitable for such combinations are within the scope of the present invention.

[0131] All values ​​and subranges within the disclosed range are also disclosed. In addition, although the systems, devices, and processes disclosed and shown herein may include a specific number of elements / components, the systems, devices, and assemblies may be modified to include more or fewer elements / components in such elements / components. For example, although any element / component disclosed may be cited as a singular, the embodiments disclosed herein may be modified to include a plurality of such elements / components. The subject matter described herein is intended to cover and encompass all appropriate technical changes.

Claims

1. A method for predicting a user's fatigue level, characterized in that: The method comprises: Obtaining valid PPG segment pairs from a photoplethysmogram (PPG) signal; extracting a plurality of PPG features of each valid PPG segment in the pair of valid PPG segments; extracting a plurality of pairwise features from the plurality of PPG features of each valid PPG segment; The fatigue level of the user is predicted according to the paired features.

2. The method according to claim 1, characterized in that: The obtaining of a valid PPG segment pair from the PPG signal comprises: extracting raw PPG segment pairs from the PPG signal; preprocessing each of the original PPG segment pairs to generate processed PPG segment pairs; Each processed PPG segment in the processed PPG segment pair is verified using a trained PPG signal verifier to obtain the valid PPG segment pair.

3. The method according to claim 1 or 2, characterized in that: The extracting a plurality of PPG features of each valid PPG segment in the valid PPG segment pair comprises: detecting a plurality of peaks in each valid PPG segment of the pair of valid PPG segments; Obtaining a plurality of NN intervals of each valid PPG segment in the pair of valid PPG segments according to the corresponding plurality of peaks; A plurality of PPG features are extracted for each valid PPG segment in the pair of valid PPG segments according to the corresponding plurality of NN intervals.

4. The method according to claim 2, characterized in that: The preprocessing of each original PPG segment in the original PPG segment pair to generate a processed PPG segment pair comprises: filtering each raw PPG segment in the pair of raw PPG segments using a bandpass filter having a bandpass frequency of 0.6-8.0 Hz; Each raw PPG segment in the raw PPG segment pair is normalized to have a mean of zero and a standard deviation of 1.

5. The method according to claim 2, characterized in that: The trained PPG signal verifier is a multi-layer perceptron network (MLP) based on fast Fourier transform (FFT).

6. The method according to claim 2, characterized in that The trained PPG signal verifier is a temporal convolutional network (TCN).

7. The method according to any one of claims 1 to 6, characterized in that The predicting of the fatigue level comprises: The pair-wise features are input into a progressive feature search tree to obtain a predicted fatigue level including an odds ratio and a probability of experiencing a change in fatigue level.

8. The method according to claim 7, characterized in that Before inputting the pair of features into the progressive feature search tree: A progressive feature search tree is constructed by computing one or more relationships between corresponding one or more pairwise features and corresponding fatigue levels.

9. The method according to claim 8, characterized in that The one or more relationships between the corresponding one or more paired features and the corresponding fatigue level are calculated using Fisher's exact test.

10. The method according to claim 8, characterized in that The corresponding one or more paired features correspond to corresponding hierarchical levels, and the progressive feature search tree is constructed by progressively calculating the one or more relationships between the corresponding one or more paired features and the corresponding fatigue levels according to the corresponding hierarchical levels.

11. The method according to any one of claims 1 to 10, characterized in that Also includes: Before extracting a plurality of pairwise features from the plurality of PPG features of each valid PPG segment: classifying the valid PPG segment pairs using a trained differential network, wherein the classification describes the similarity between the valid PPG segment pairs; In response to the valid PPG segment pair being classified as having a low similarity, the plurality of pair-wise features are extracted from the plurality of PPG features of each valid PPG segment in the valid PPG segment pair.

12. The method according to any one of claims 1 to 11, characterized in that Also includes: comparing the fatigue level to a predefined standard; A fatigue warning is issued to the user based on the comparison.

13. A system, characterized in that: include: PPG sensor; one or more memories storing executable instructions; One or more processors coupled to the PPG sensor and the one or more memories, the executable instructions being configured to cause the one or more processors to perform the following operations: Obtain valid PPG segment pairs from the PPG signal; extracting a plurality of PPG features of each valid PPG segment in the pair of valid PPG segments; extracting a plurality of pairwise features from the plurality of PPG features of each valid PPG segment; The user's fatigue level is predicted according to the paired features.

14. The system according to claim 13, characterized in that When executed by the one or more processors to obtain valid PPG segment pairs from the PPG signal, the executable instructions further cause the system to: extracting raw PPG segment pairs from the PPG signal; preprocessing each of the original PPG segment pairs to generate processed PPG segment pairs; Each processed PPG segment in the processed PPG segment pair is verified using a trained PPG signal verifier to obtain the valid PPG segment pair.

15. The system according to claim 13 or 14, characterized in that When executed by the one or more processors to extract a plurality of PPG features for each valid PPG segment in the pair of valid PPG segments, the executable instructions further cause the system to: detecting a plurality of peaks in each valid PPG segment of the pair of valid PPG segments; Obtaining a plurality of NN intervals of each valid PPG segment in the pair of valid PPG segments according to the corresponding plurality of peaks; A plurality of PPG features are extracted for each valid PPG segment in the pair of valid PPG segments according to the corresponding plurality of NN intervals.

16. The system according to any one of claims 13 to 15, characterized in that When executed by the one or more processors to predict the fatigue level, the executable instructions further cause the system to: The pair-wise features are input into a progressive feature search tree to obtain a predicted fatigue level including an odds ratio and a probability of experiencing a change in fatigue level.

17. The method according to claim 16, characterized in that When executed by the one or more processors, the executable instructions further cause the system to: Before inputting the pair of features into the progressive feature search tree: A progressive feature search tree is constructed by calculating one or more relationships between corresponding one or more pairwise features and corresponding fatigue levels, wherein the one or more relationships between corresponding one or more pairwise features and corresponding fatigue levels are calculated using Fisher's exact test.

18. The system according to claim 13, characterized in that When executed by the one or more processors, the executable instructions further cause the system to: Before extracting a plurality of pairwise features from the plurality of PPG features of each valid PPG segment: classifying the valid PPG segment pairs using a trained differential network, wherein the classification describes the similarity between the valid PPG segment pairs; In response to the valid PPG segment pair being classified as having a low similarity, the plurality of pair-wise features are extracted from the plurality of PPG features of each valid PPG segment in the valid PPG segment pair.

19. The system according to any one of claims 14 to 18, characterized in that When executed by the one or more processors, the executable instructions further cause the system to: comparing the fatigue level to a predefined standard; A fatigue warning is issued to the user based on the comparison.

20. A non-transitory computer readable medium, characterized in that The non-transitory computer-readable medium has machine-executable instructions stored thereon, which, when executed by one or more processors of a computing system, cause the computing system to perform the following operations: Obtain valid PPG segment pairs from the PPG signal; extracting a plurality of PPG features of each valid PPG segment in the pair of valid PPG segments; extracting a plurality of pairwise features from the plurality of PPG features of each valid PPG segment; The user's fatigue level is predicted according to the paired features.

Images