Fatigue analysis method and system based on face recognition

Abstract

The invention provides a fatigue analysis method and system based on face recognition. The method comprises the following steps: acquiring a face video, and intercepting and recognizing the face video, so as to extract image features and stack the image features to form a feature map; performing deep learning on the feature map to obtain a physiological index; and fusing the physiological indexes through a fatigue index analysis model to obtain a fatigue index.

Description

Technical Field

[0001] The present invention relates to the field of civil aircraft test flight and human factors engineering technology, and more specifically, to a fatigue analysis method and system based on face recognition. Background Art

[0002] In many high-risk industries and fields, especially in key and important positions, managers pay special attention to the control of employee fatigue status. For example, test flight is a high-risk activity that requires personnel in key test flight positions to make accurate and quick decisions in a complex environment, and experimental flights may involve long hours of work and flight operations, which can easily lead to fatigue and decreased concentration. Therefore, it is of great significance for personnel in key test flight positions to conduct full-process status detection during work and provide active abnormal warnings at critical moments. For example, in the field of civil aircraft test and flight testing, test pilots, test flight engineers, maintenance personnel, test modification implementation positions, etc., employees in these positions often need to maintain a high level of attention and vigilance, and the safety risks caused by fatigue work are often unacceptable.

[0003] By monitoring the changes in physiological indicators, it is technically possible to monitor the fatigue status of key employees, so that managers can timely understand the fatigue status of key employees, thereby playing a positive role in adjusting plans and workloads, preventing employees from working in fatigue mode, and reducing the risk of human factors. Combining the current status of domestic and international research on fatigue monitoring, the methods for monitoring fatigue values ​​include emotion mapping, EEG signal mapping, ECG signal mapping, physiological indicator mapping, etc. These signal acquisition techniques and methods for calculating fatigue index are different.

[0004] However, existing technologies such as EEG signal acquisition and emotion signal acquisition often rely on professional medical equipment or distributed cameras in all scenarios, which often require a lot of money to build and maintain. In addition, the existing algorithms for calculating fatigue values ​​based on physiological indicators mainly focus on correlation calculations through individual experiments, which can cause fatigue detection results to vary greatly with individual changes. Therefore, there is a need for improved fatigue analysis and detection technology in this field. Summary of the invention

[0005] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] One of the purposes of the present invention is to provide a fatigue analysis method based on face recognition. The fatigue analysis method extracts multiple health-related physiological indicators from face videos based on the slight changes in skin color caused by pulsation signals, and then calculates and analyzes these physiological indicators through a fatigue index analysis model and converts them into fatigue indexes, thereby helping users to better monitor and manage the health of relevant staff.

[0007] According to one aspect of the present disclosure, a fatigue analysis method based on face recognition is provided, the method comprising: obtaining a face video, intercepting and identifying the face video, thereby extracting image features and stacking them to form a feature map; performing deep learning on the feature map to obtain physiological indicators; and fusing the physiological indicators through a fatigue index analysis model to obtain a fatigue index.

[0008] In one embodiment of the present disclosure, acquiring the face video further includes: setting a threshold time; and capturing the face video by a camera for a time greater than the threshold time.

[0009] In another embodiment of the present disclosure, extracting image features and stacking them to form a feature map further includes: using a 34-layer ResNet model to process the face video to generate the feature map.

[0010] In another embodiment of the present disclosure, performing deep learning on the feature graph to obtain the physiological indicator further includes: performing deep learning on the feature graph using a sequence-to-sequence model, wherein the sequence-to-sequence model includes an encoder component and a decoder component, the encoder component is used to input a plurality of time series signals constituted by the feature graph, and the decoder component is used to output the physiological indicator, and the sequence lengths of both the plurality of time series signals and the physiological indicator are variable.

[0011] In yet another embodiment of the present disclosure, the physiological indicator comprises at least one of heart rate, heart rate variability, respiratory rate, blood oxygen saturation, body temperature, systolic blood pressure, and diastolic blood pressure.

[0012] In yet another embodiment of the present disclosure, the fatigue index analysis model uses contrastive learning to train physiological indicators, determines the fatigue level of the user corresponding to the face video, and converts the fatigue level into a fatigue index.

[0013] In a further embodiment of the present disclosure, using contrastive learning to train physiological indicators further includes: constructing contrast pairs for the physiological indicators; and designing a loss function, and using gradient descent or a variant of gradient descent to optimize the loss function.

[0014] In a further embodiment of the present disclosure, optimizing the loss function using gradient descent or a variant of gradient descent further includes: updating parameters of the fatigue index analysis model using a stochastic gradient descent optimization algorithm.

[0015] According to another aspect of the present disclosure, a fatigue analysis system based on face recognition is provided, which includes: a memory; a communication interface; and at least one controller communicatively coupled to the memory and the communication interface, and the at least one controller is configured to execute the fatigue analysis method based on face recognition of the present invention.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable medium storing computer-executable instructions is provided. When the computer-executable instructions are executed by a processor, the fatigue analysis method based on face recognition of the present invention is performed.

[0017] These and other features and advantages will become apparent by reading the following detailed description and by reference to the associated drawings.It is to be understood that the foregoing general description and the following detailed description are illustrative only and are not restrictive of the aspects of what is claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to understand the manner in which the above features of the present invention are used in detail, the above briefly summarized contents can be described in more detail with reference to various embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the accompanying drawings only show some typical aspects of the present invention and should not be considered to limit its scope, because the description may allow for other equally effective aspects.

[0019] Figure 1 The figure is a flow chart of a fatigue analysis method based on face recognition according to an embodiment of the present invention.

[0020] Figure 2 It is a schematic diagram of a 34-layer ResNet model according to an embodiment of the present invention.

[0021] Figure 3 is a schematic diagram of a sequence-to-sequence model according to an embodiment of the present invention.

[0022] Figure 4a-4g It is a timing diagram of actually measured physiological indicators according to an embodiment of the present invention.

[0023] Figure 5 is a schematic diagram of a fatigue analysis system based on face recognition according to an embodiment of the present invention.

[0024] Figure 6 is a schematic diagram of a fatigue analysis architecture based on face recognition according to an embodiment of the present invention.

[0025] In the drawings, the drawings are not drawn to scale. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0027] In the description of the present disclosure, it should be noted that, unless otherwise specified, "multiple" means more than two; the terms "upper", "lower", "left", "right", "inside", "outside", etc. indicate positions or positional relationships only for the convenience of describing the present disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present disclosure. In addition, the terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. "Vertical" is not strictly vertical, but is within the allowable error range. "Parallel" is not strictly parallel, but is within the allowable error range.

[0028] The directional words appearing in the following description are all directions shown in the figures, and do not limit the specific structure of the present disclosure. In the description of the present disclosure, it should also be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present disclosure can be understood according to the specific circumstances.

[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present disclosure. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] In the description of the embodiments of the present disclosure, the term "and / or" is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0031] Figure 1A flow chart of a fatigue analysis method based on face recognition according to an embodiment of the present invention is shown.

[0032] like Figure 1 As shown in , in step 102, a face video may be acquired, and the face video may be intercepted and recognized, thereby extracting image features and stacking them to form a feature map.

[0033] Specifically, for example, subtle color changes in facial areas (especially areas of high blood perfusion such as the forehead or cheeks) are caused by changes in vascular volume associated with the cardiac cycle. During each cardiac cycle, the volume of blood in facial blood vessels changes as the heart contracts and relaxes. During systole (the contraction phase), when blood flow increases, more oxygenated blood increases, resulting in a slight increase in the absorption of light. Conversely, during diastole (the relaxation phase), blood flow decreases, resulting in a decrease in the absorption of light. These changes in light absorption correspond to subtle changes in skin color. Therefore, facial videos can be processed using specific algorithms to extract pulsation signals associated with heart rate. These algorithms typically involve tracking specific facial areas, isolating color changes, and applying signal processing techniques to enhance the pulsation signal.

[0034] In a non-limiting example, acquiring a face video may further include: setting a threshold time; and capturing a face video through a camera for a time greater than the threshold time. For example, a user's face video may be captured through a camera of a mobile phone, computer, smart watch or other device, that is, the user may use a camera to shoot a face video for more than a threshold time, thereby completing the capture of the face video. Optionally, in order to ensure that the video has sufficient resolution and frame rate, a high-definition camera device may be required. It can be appreciated that the present invention does not limit the specific method of capturing face videos. In addition, these videos may be in the visible light range or in the infrared range.

[0035] In another non-limiting example, capturing and identifying a face video may further include: capturing the acquired face video frame by frame for image recognition, extracting optical signals in the image, and capturing tiny color changes on the skin surface in the image.

[0036] In another non-limiting example, extracting image features and stacking them to form a feature map may further include: using a 34-layer ResNet model to process the face video to generate the feature map. Specifically, in order to capture subtle facial color changes, the ResNet model may be used to extract image features.

[0037] ResNet is a network model proposed by He Kaiming in 2015. Generally, the deeper the number of layers of a neural network, the stronger its expressive power is and the more complex information it can extract. However, when the number of layers reaches a certain limit, "degeneration" will occur. In order to alleviate this problem, the ResNet model proposes the following Figure 2 The short cut design shown. The stacked layers of networks are called a module. For a certain module, the function that can be fitted is F(x). If the actual mapping is H(x), instead of letting F(x) directly learn H(x), it is better to learn the residual H(x)-x, that is, F(x):=H(x)-x. In this way, the original forward path becomes to use F(x)+x to fit H(x). This may be easier to optimize because it is relatively easier to fit F(x) to 0. Therefore, for redundant modules, only F(x)→0 is required to obtain the identical mapping without reducing performance. The ResNet model includes models with 18, 34, 50, 101, and 152 layers. After weighing the amount of computation and the effect, the present invention adopts a 34-layer model (i.e., the ResNet34 model), which includes convolution and pooling operations, and uses Relu as the activation function.

[0038] The ResNet34 model includes 1 base layer, 16 residual structures, and 1 fully connected layer. The entire network has a total of 1+2x3+2x4+2x6+2x3+1=34 layers. Table 1 shows part of the module stacking configuration of the ResNet34 model. The parameters in each module configuration include the convolution kernel size and the number of channels (usually the more channels, the smaller the generated feature map). In addition, the number of modules indicates the number of module repetitions. Layers that do not contain parameters, such as activation layers and pooling layers, are not counted in the layer statistics.

[0039] Table 1 ResNet34 module stacking information

[0040]

[0041] As shown in Table 1, in the ResNet34 model (or network), a layer is a stage in the network, in which the network is built by stacking residual blocks of the same type. Each layer is responsible for extracting features at a specific level, and as the network goes deeper, the layer usually gradually reduces the resolution of the feature map (by downsampling) and increases the number of channels (to capture richer semantic information). In the ResNet34 model, there are usually 4-5 such layers, excluding the initial 7x7 convolution layer and 3x3 maxpooling layer.

[0042] block_config refers to the configuration information of the convolutional layer in the ResNet34 model, including the size of the convolution kernel, the number of output channels, and parameters such as stride or padding. These parameters define how the convolutional layer processes the input data to extract features. For example, (7x7,64, / 2) describes a convolutional layer that uses a 7x7 convolution kernel, outputs 64 feature maps, and has a stride of 2; while ([3x3,64]) represents a sequence of convolutional layers consisting of 64 3x3 convolution kernels.

[0043] block_count refers to the configuration or number of residual blocks in each layer. For the ResNet34 model, its block_count is usually expressed as [3,4,6,3], which means: the first layer (conv2_x or layer1) contains 3 residual blocks; the second layer (conv3_x or layer2) contains 4 residual blocks; the third layer (conv4_x or layer3) contains 6 residual blocks; the fourth layer (conv5_x or layer4) contains 3 residual blocks. This configuration helps the ResNet34 model capture rich feature information at different levels.

[0044] In step 104, deep learning may be performed on the feature map to obtain physiological indicators.

[0045] In a non-limiting example, performing deep learning on the feature graph to obtain the physiological index may further include performing deep learning on the feature graph using a sequence-to-sequence model. The sequence-to-sequence model may include an encoder component and a decoder component, wherein the encoder component is used to input a plurality of time series signals constituted by the feature graph, and the decoder component is used to output the physiological index, and the sequence lengths of the plurality of time series signals and the physiological index are varied.

[0046] Figure 3 A schematic diagram of a sequence-to-sequence model according to an embodiment of the present invention is shown. The sequence-to-sequence model is an architecture for processing variable-length sequences of input and output, and is particularly suitable for extracting features from a series of input data and mapping them to another series of output data, where the lengths of the input and output may be different.

[0047] like Figure 3As shown, according to the input-output classification, in order to process sequences with variable input and output lengths, the sequence-to-sequence model includes an architecture encoder-decoder with two main components, namely, encoder 302 and decoder 306. Among them, the first component encoder 302 accepts a sequence of variable length as input and converts it into a coded state with fixed dimensions (i.e., hidden state 304), which can be a compressed representation of the input sequence and contains all key information in the input sequence. In the present invention, the input of the encoder 302 can be realized, for example, by a series of convolutional layers, pooling layers, and other neural network layers, which work together to extract features from the input image (or video frame). The second component decoder 306 maps the fixed-dimensional state encoding (i.e., hidden state 304) to a sequence of variable length (i.e., output 308). This output sequence can be another sequence of different lengths from the input sequence. For example, in the present invention, the output can be a numerical sequence representing a physiological indicator. That is, this scenario corresponds to a many-to-one model, inputting a time series signal consisting of a feature map (feature map) and outputting a physiological indicator.

[0048] It can be appreciated that physiological indicators can also be obtained from feature maps in other ways, for example, Fourier transform can be used to process the time series composed of feature maps, convert it into a frequency domain signal, and select the peak value to obtain the target value.

[0049] In another non-limiting example, the physiological indicator may include at least one of heart rate, heart rate variability, respiratory rate, blood oxygen saturation, body temperature, systolic blood pressure, and diastolic blood pressure. Combined with the examples described above, it can be seen that the sequence-to-sequence model based on the encoder-decoder dual component can be applied to process optical signals extracted from face videos. These optical signals reflect slight color changes on the skin surface, which are related to physiological indicators (such as heart rate, blood pressure, etc.), and are further associated with fatigue status. It can be understood that the above indicators are only exemplary, as long as they can reflect the user's fatigue.

[0050] Figure 4a-4g The timing diagram of the actual measured physiological indicators according to an embodiment of the present invention is shown, wherein the horizontal axis is time (min), i.e. time (minutes), and the vertical axis is heart rate (HR), heart rate variability (HRV), respiratory rate (RR), blood oxygen saturation (SpO2), body temperature (T), systolic blood pressure (SBP) and diastolic blood pressure (DBP). Figure 4a-4g The actual measured physiological indicators are compared with the physiological indicators calculated based on the face video above, and it is found that the physiological indicators calculated by the present invention are within a reasonable error range. In other words, the experiment shows that the above-mentioned image recognition and feature extraction operations of the present invention are feasible.

[0051] In step 106, the fatigue index may be integrated with the physiological index through the fatigue index analysis model to obtain a fatigue index. It can be appreciated that the fatigue index may be updated over time to reflect the fatigue state of the user corresponding to the face video.

[0052] In a non-limiting example, the fatigue index analysis model can use contrastive learning to train physiological indicators, determine the fatigue level of the user corresponding to the face video, and convert the fatigue level into a fatigue index. Contrastive Learning is a machine learning method that is mainly used to train models to learn representations in data. The basic idea is to learn meaningful representations by comparing the similarities and differences between samples and other samples. The goal of contrastive learning is to make similar samples closer in the representation space, while dissimilar samples are more dispersed. For example, the present invention can adopt a contrastive learning strategy to compare the changes in physiological indicators under relative fatigue and relative non-fatigue states. When predicting, the current physiological indicators are compared with the user's baseline value to determine the user's fatigue level relative to his or her historical situation, and then convert it into a fatigue index.

[0053] One of the advantages of contrastive learning is that it does not require label information, that is, it can be trained without explicit labels. This makes contrastive learning an effective tool for processing large-scale unlabeled data. Contrastive learning has achieved remarkable results in the fields of computer vision, natural language processing, and other fields, especially in pre-training deep learning models.

[0054] From this, it can be understood that fatigue index analysis models with other structures can also be used. For example, if fatigue labels can be provided, an ordinal regression strategy can be used, or ordinal regression and contrastive learning can be used simultaneously to combine the two.

[0055] When contrastive learning is used to train physiological indicators, it may further include: constructing contrast pairs for physiological indicators; and designing a loss function, and optimizing the loss function using gradient descent or a variant of gradient descent. Further, a stochastic gradient descent (SGD) optimization algorithm may be used to update the parameters of the fatigue index analysis model.

[0056] Figure 5A schematic diagram of a fatigue analysis system based on face recognition according to an embodiment of the present invention is shown. The system shows a general hardware environment in which the present invention can be applied according to exemplary embodiments of the present invention. The system can be any machine configured to perform processing and / or calculations, and can be but not limited to a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal digital assistant (PDA), a smart phone, or any combination thereof. The above system can be implemented in whole or in part by the device or a similar device or system.

[0057] The system may include components connected to or in communication with a bus 520. For example, the system may include a bus 520, a processor 505, and one or more memories 510, among other things.

[0058] The processor(s) 505 may be any type of processor and may include, but are not limited to, general purpose processors and / or dedicated processors (e.g., special processing chips), intelligent hardware devices (e.g., general purpose processors, DSPs, CPUs, microcontrollers, ASICs, FPGAs, programmable logic devices, discrete gate or transistor logic components, discrete hardware components, or any combination thereof). In some cases, the processor 505 may be configured to operate a memory array using a memory controller. In other cases, a memory controller (not shown) may be integrated into the processor(s) 505. The processor(s) 505 may be responsible for managing the bus 520 and general processing, including executing software 515 stored on the memory 510. The processor(s) 505 may also be configured to perform various functions described herein related to the fatigue analysis method based on face recognition. For example, the processor(s) 505 may be configured to: obtain a face video, intercept and identify the face video, thereby extracting image features for stacking to form a feature map; perform deep learning on the feature map to obtain physiological indicators; and fuse the physiological indicators through a fatigue index analysis model to obtain a fatigue index.

[0059] The memory 510 may be any storage device that can implement data storage. The memory 510 may include, but is not limited to, a disk drive, an optical storage device, a solid-state memory, a floppy disk, a floppy disk, a hard disk, a tape or any other magnetic medium, an optical disk or any other optical medium, a ROM (read-only memory), a RAM (random access memory), a cache memory and / or any other memory chip or box, and / or any other medium from which a computer can read data, instructions and / or codes. The memory 510 may store computer executable software 515 including computer readable instructions that, when executed, cause the processor to perform the various functions described herein. The memory 510 may have various data / instructions / codes for implementing the various functions described herein related to the design of a fatigue analysis system based on face recognition.

[0060] Software 515 may be stored in memory 510, including but not limited to an operating system, one or more applications, drivers and / or other data and codes. Instructions for performing the various functions described herein may be included in one or more applications, and the components of the system may be implemented by the processor 505 reading and executing the instructions of one or more applications. In some cases, the software 515 may not be directly executable by the processor, but may (for example, when compiled and executed) enable the computer to perform the various functions described herein related to the fatigue analysis method based on face recognition.

[0061] Figure 6 FIG. 1 is a schematic diagram of a fatigue analysis architecture based on face recognition according to an embodiment of the present invention. Figure 6 As shown, the fatigue analysis architecture based on face recognition may include a collection end 610, a service end 620 and a management end 630. The collection end 610 is configured with a mobile phone APP 602, the service end 620 is configured with a fatigue analysis service platform 629, and the management end 630 is configured with a PC end analysis and monitoring platform 632. Further, the fatigue analysis service platform 629 may have a collection service module 621, a personnel management module 622, an account management module 623, a log and audit module 624, a backup and recovery module 625, a model service module 626, a data management service module 627 and a monitoring and analysis service system management service module 628.

[0062] It should be noted that Figure 6 The fatigue analysis architecture shown is only illustrative and not restrictive. The fatigue analysis architecture based on face recognition of the present invention may include: Figure 6 More or fewer components of the face recognition based fatigue analysis architecture shown in .

[0063] Specifically, key personnel of the test flight can download the mobile phone APP 602 for the fatigue analysis method based on face recognition, and bind key personnel information through the mobile phone APP 602. These account information are synchronized on the service end 620 and the management end 630. It can be understood that the system administrator can add relevant personnel information on the management end 630 in advance for key personnel to bind.

[0064] refer to Figure 1 In step 102, Figure 6At the collection terminal 610, employees (i.e. key personnel) can turn on the front camera of the mobile phone to collect facial videos for more than 20 seconds, thereby completing the data collection. As a non-limiting example, it is also possible to require the recording of facial videos with a resolution of not less than 480x480 and a video frame rate of not less than 15fps. If the effective duration of the recorded video is 20 seconds, the recording timing is paused when the recorded picture does not meet the requirements, such as the face moves out of the picture, the distance is too far, etc. After the progress is paused, wait for the picture to recover to meet the requirements, then continue recording and timing until the effective picture duration is enough for 20 seconds. Optionally, during recording, if the cumulative number of times the picture does not meet the requirements exceeds 3 times or the cumulative duration exceeds 3 seconds, the recording is terminated and needs to be re-recorded.

[0065] Enter the server 620 to perform image recognition analysis. First, the acquired face video is intercepted frame by frame for image recognition, optical signals are extracted, and tiny color changes on the skin surface in the image are captured. For the acquired video data, each frame in the video can be input into the video image processing model ResNet34 for image feature extraction, including convolution and pooling operations, using ReLu as the activation function, completing optical signal processing, obtaining the corresponding feature map, and forming a series of time series signals.

[0066] Then, refer to Figure 1 In step 104, Figure 6 The server 620 then uses periodic feature extraction technology to output physiological index values, for example, through a sequence-to-sequence model of an encoder-decoder architecture including two main components, inputs a time series signal consisting of a feature map, and outputs physiological indicators. The value of the physiological index data obtained here will be used as the data input of the fatigue index analysis model below.

[0067] In addition, if Figure 6 As shown, the server 620 may also have functions such as personnel management, account management, log and audit, backup and recovery, etc.

[0068] refer to Figure 1 In step 106, Figure 6 The server 620 enters the fatigue index analysis model and integrates the physiological indicators to obtain the fatigue index. This operation can be performed by Figure 6 The monitoring and analysis service and system management service module 628 in the system is executed. As an example, the fatigue index analysis model can be trained and generated by comparative learning, comparing the changes in physiological indicators under relatively fatigued and relatively non-fatigued states. When predicting, the current physiological indicators are compared with the user's baseline value to determine the user's fatigue level relative to his or her historical situation and convert it into a fatigue index. The specific operations are as follows:

[0069] (1) For the physiological indicators obtained above, construct a comparison pair: For a given sample, construct a comparison pair, that is, a positive sample pair (similar samples) and a negative sample pair (dissimilar samples). This is usually done by selecting a pair of samples from the dataset and determining whether they belong to the same category or different categories based on some criteria.

[0070] x=concat(x1,x2)

[0071] f(x)=Re LU(dense(x))=max(0,W T *x)

[0072] Logit=deep_learning_model(x)=f(f...f(x))

[0073]

[0074] Among them, x1, x2 are the characteristic values ​​of the sample, including physiological indicators and feature maps, which are concatenated into x in the channel dimension through the concat operation; W is a parameter obtained by training; ReLu is an activation function; and the Logit function maps probability values ​​to real numbers; Score the prediction, i.e., the prediction index of the fatigue index analysis model.

[0075] (2) Design loss function: Introduce a loss function whose goal is to make positive sample pairs closer in the representation space and negative sample pairs more dispersed, so that the final fatigue index is more accurate. Common loss functions include contrast loss functions, such as ternary contrast loss and cross entropy loss.

[0076] The cross entropy loss function is used here:

[0077]

[0078] Among them, y is the label, that is, the label of the fatigue index, which can be obtained by contrastive learning; Score the prediction, i.e., the prediction index of the fatigue index analysis model.

[0079] As a non-limiting embodiment, the fatigue index label can also be obtained by combining the user's answer questionnaire and test time (working time), for example, fatigue labels of 1-7 levels can be set. Accordingly, if labels can be provided, ordered regression and contrastive learning can be used at the same time to integrate the two; if there are no labels, only contrastive learning can be used, and the test data of different users are not comparable. Therefore, it can be expected that supervised learning with labels will have better results.

[0080] (3) Training the model: By optimizing the loss function, the model is trained to learn meaningful representations of samples, which can also improve the computational efficiency of the model. For example, gradient descent or its variants can be used to update the parameters of the model.

[0081] The stochastic gradient descent (SGD) optimization algorithm is used here:

[0082]

[0083] Among them, w is a parameter obtained by training; η is the learning rate, which can be pre-set based on experience; is the gradient of the loss function f with respect to the parameter w.

[0084] As a non-limiting embodiment, the predicted index obtained by the fatigue index analysis model can be further corrected according to the situation of each user to obtain the final fatigue index. Specifically, for different users, their historical data can be used to establish a personalized benchmark distribution. This can help the fatigue analysis system better adapt to the user's unique health status and changing trends. The user's real-time indicators are compared with their personalized benchmark distribution, so as to adjust the fatigue index to more accurately reflect the user's status.

[0085] like Figure 6 As shown, further, the physiological index data (obtained through the sequence-to-sequence model) and fatigue index (obtained through the fatigue index analysis model) generated above can be transmitted back to the management terminal 630 and displayed on the PC terminal analysis and monitoring platform 632, so that managers can understand the fatigue status of key position employees. Managers can specify the website page through the computer terminal to view the fatigue index and historical data of employees in each key position, so that they can adjust the plan and task volume in time according to the data trend, prevent employees from working in fatigue, and reduce the risk of human factors.

[0086] From this we can understand that Figure 6 The schematic diagram shown is only exemplary and each step may be adjusted, modified, added and / or deleted according to actual needs as long as the reference Figure 1 The fatigue analysis method based on face recognition can be used.

[0087] The above describes the fatigue analysis method and system based on face recognition of the present invention. Compared with the solutions in the prior art, the data acquisition terminal of the present invention can be a mobile phone APP, which is low-cost, convenient to carry and measure at any time, and easy to use and popularize. In addition, the existing algorithms for calculating fatigue values ​​based on physiological indicators mainly focus on correlation calculations through individual experiments, and the fatigue detection results vary greatly with individual changes. On the contrary, the present invention uses a machine learning method, so that the accuracy of the model can be improved as the amount of unlabeled sample data increases.

[0088] What has been described above includes examples of various aspects of the claimed subject matter. Of course, it is not possible to describe every conceivable combination of components or methods for the purpose of describing the claimed subject matter, but one of ordinary skill in the art will recognize that many further combinations and permutations of the claimed subject matter are possible. Thus, the disclosed subject matter is intended to encompass all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.

Claims

1. A fatigue analysis method based on face recognition, characterized in that: include: Acquire a face video, intercept and identify the face video, thereby extracting image features and stacking them to form a feature map; Performing deep learning on the feature map to obtain physiological indicators; as well as The fatigue index is obtained by integrating the physiological indicators through a fatigue index analysis model.

2. The fatigue analysis method according to claim 1, characterized in that: Acquiring the face video further includes: Setting threshold times; and The face video is captured by a camera for a time period greater than the threshold time period.

3. The fatigue analysis method according to claim 1, characterized in that: Extracting the image features and stacking them to form the feature map further includes: A 34-layer ResNet model is used to process the face video to generate the feature map.

4. The fatigue analysis method according to claim 1, characterized in that: Performing deep learning on the feature graph to obtain the physiological index further includes: A sequence-to-sequence model is used to perform deep learning on the feature graph, wherein the sequence-to-sequence model includes an encoder component and a decoder component, the encoder component is used to input a plurality of time series signals constituted by the feature graph, and the decoder component is used to output the physiological indicator, and the sequence lengths of both the plurality of time series signals and the physiological indicator are variable.

5. The fatigue analysis method according to claim 1, characterized in that: The physiological index includes at least one of heart rate, heart rate variability, respiratory rate, blood oxygen saturation, body temperature, systolic blood pressure, and diastolic blood pressure.

6. The fatigue analysis method according to claim 1, characterized in that: The fatigue index analysis model uses contrastive learning to train the physiological index, determines the fatigue level of the user corresponding to the face video, and converts the fatigue level into the fatigue index.

7. The fatigue analysis method according to claim 6, characterized in that: Using contrastive learning to train the physiological indicators further includes: Constructing comparison pairs for the physiological indicators; and A loss function is designed and optimized using gradient descent or a variant of the gradient descent.

8. The fatigue analysis method according to claim 7, characterized in that: Optimizing the loss function using gradient descent or a variant of the gradient descent further comprises: The parameters of the fatigue index analysis model are updated using a stochastic gradient descent optimization algorithm.

9. A fatigue analysis system based on face recognition, characterized in that: include: Memory; Communication interface; as well as At least one controller communicatively coupled to the memory and the communication interface, the at least one controller configured to perform the method of any one of claims 1-8.

10. A non-transitory computer-readable storage medium storing computer-executable instructions, characterized in that: The computer executable instructions, when executed by a processor, cause the processor to perform the method of any one of claims 1-8.

Images