Method for constructing mode signal set for representing human body fatigue state by facial expression

Abstract

The invention provides a mode signal set construction method for representing a human body fatigue state through facial expressions. The method comprises the steps of obtaining fatigue state data of an operator, determining a feature vector of the fatigue state data, determining an emotional state value of the operator according to the feature vector, performing label classification processing on the fatigue state data of the operator according to the emotional state value to obtain sample data, and constructing training data based on the sample data; constructing a neural network model, training the neural network model by adopting the sample data to generate a feature enhancement model, performing feature enhancement processing on the sample data and the training data by adopting the feature enhancement model to generate an enhanced data set, and performing expansion processing on the enhanced data set by adopting a generative adversarial network to generate a mode signal set; wherein the mode signal set is used for constructing a human body fatigue state recognition model. The problem of incomplete data for fatigue recognition model training in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the technical field of constructing a pattern signal set for facial expressions to represent the fatigue state of a human body, and specifically to a method, device, computer-readable storage medium and electronic device for constructing a pattern signal set for facial expressions to represent the fatigue state of a human body. Background Art

[0002] In the field of human fatigue monitoring, related technologies are developing rapidly, focusing primarily on how to accurately and real-timely identify and assess worker fatigue to improve work efficiency and ensure safety. Traditional fatigue monitoring methods rely on subjective assessments and physiological parameter measurements, such as heart rate and electrooculogram (EOG). However, these methods often suffer from poor real-time performance, large individual differences, and complex operations. In recent years, with the development of computer vision and deep learning technologies, facial expression-based fatigue recognition has become a research hotspot. This method infers fatigue levels by analyzing facial muscle activity and changes, and is non-invasive and highly real-time.

[0003] However, existing facial expression-based fatigue monitoring technologies still have numerous shortcomings. First, existing technologies often neglect the integrated application of multi-source data, such as eye movement data, physiological data, and body movement data, which are crucial for accurately assessing fatigue status. Second, existing methods are relatively simple in data acquisition and preprocessing, lacking adaptability to complex operating environments, resulting in insufficient model generalization capabilities, which, to a certain extent, limits the recognition accuracy and robustness of the models. Summary of the Invention

[0004] The main purpose of this application is to provide a method, device, computer-readable storage medium and electronic device for constructing a pattern signal set of facial expressions representing the fatigue state of the human body, so as to at least solve the problem of incomplete data used in fatigue recognition model training in the existing technology.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for constructing a pattern signal set for facial expression characterization of human fatigue state is provided, comprising: obtaining fatigue state data of an operator, wherein the fatigue state data comprises facial expression data, body movement data and physiological data; determining a feature vector of the fatigue state data, and determining the emotional state value of the operator based on the feature vector, and performing label classification processing on the fatigue state data of the operator based on the emotional state value to obtain sample data, and constructing training data based on the sample data, wherein the training data comprises facial expression feature vectors, physiological feature vectors and corresponding emotional state labels; constructing a neural network model, using the sample data to train the neural network model to generate a feature enhancement model, and using the feature enhancement model to perform feature enhancement processing on the sample data and training data to generate an enhanced data set, and using a generative adversarial network to expand the enhanced data set to generate a pattern signal set, wherein the pattern signal set is used to construct a human fatigue state recognition model.

[0006] Optionally, in the process of training the neural network model using the sample data, the method also includes: determining the loss function and F1 score of the neural network model, wherein the F1 score represents the degree of match between the predicted emotional state label of the neural network model and the actual emotional state label; and adjusting the model parameters of the neural network model according to the loss function and the F1 score.

[0007] Optionally, determining the loss function and F1 score of the neural network model includes: according to a first formula: Determine the loss function of the neural network model, where L is the loss function, N is the number of samples in the data set, and y i is the actual emotional state label of the i-th sample data, P(y i ) is the predicted emotional state label of the i-th sample data; according to the second formula: Determine the F1 score of the neural network model, where TP is the number of positive examples correctly predicted by the model, FP is the number of negative examples incorrectly predicted as positive by the model, TN is the number of negative examples correctly predicted by the model, FN is the number of positive examples incorrectly predicted as negative by the model, and α is a weighting factor.

[0008] Optionally, adjusting the model parameters of the neural network model according to the loss function and the F1 score includes: according to the third formula: L final =L+ζ·R(Y), determine the target loss function L of the neural network model final, where L is the loss function, ζ is the regularization strength, and R(Y) is the regularization term; according to the fourth formula: Determine the dynamic learning rate η of the neural network model t , where η0 is the initial learning rate, λ c is the decay constant, and fre is the number of iterations in the model training process.

[0009] Optionally, after performing feature enhancement processing on the sample data and the training data using the feature enhancement model to generate an enhanced data set, the method further includes: using the fifth formula: B i,j =Z combined +ε·H(σ)+γ·Morph(F i,j ), add noise processing to the features in the enhanced data set to generate a target enhanced data set, where B i,j is the jth feature of the i-th sample after adding noise, ε is the weight set according to the fatigue level, H(σ) is the Gaussian noise added, γ is the parameter used to control the deformation intensity, Morph(F i,j ) is a morphological change function of facial features, which is used to simulate the impact of fatigue on facial features.

[0010] Optionally, determining the characteristic vector of the fatigue state data includes: according to a sixth formula: Determine the characteristic vector Z of the fatigue state data, where F i is the feature vector of the i-th facial expression data, Phy i is the feature vector of the i-th physiological data, A i is the feature vector of the i-th body motion data, a F is the weight coefficient of the feature vector of the i-th facial expression data, a Phy is the weight coefficient of the eigenvector of the i-th physiological data, a A is the weight coefficient of the eigenvector of the i-th body motion data, and n, m, and k respectively represent the number of eigenvectors of the facial expression data, the eigenvectors of the physiological data, and the eigenvectors of the body motion data.

[0011] Optionally, label classification processing is performed on the fatigue status data of the operator according to the emotional state value, including: according to the seventh formula: The emotional state value is adjusted to generate a target emotional state value E(S), where S is the emotional state value, f1 is the adjustment parameter affecting the exponential decay part, f2 is the adjustment parameter affecting the emotional state on the emotional score, M represents the order of the polynomial, C iRepresents the coefficient of the i-th high-order term; and performs label classification processing on the fatigue state data of the operator according to the target emotional state value.

[0012] According to another aspect of the present application, a device for constructing a pattern signal set for characterizing a human fatigue state by using facial expressions is provided, comprising: an acquisition unit for acquiring fatigue state data of an operator, wherein the fatigue state data comprises facial expression data, body movement data and physiological data; a determination unit for determining a feature vector of the fatigue state data, and determining an emotional state value of the operator based on the feature vector, and performing label classification processing on the fatigue state data of the operator based on the emotional state value to obtain sample data, and constructing training data based on the sample data, wherein the training data comprises a facial expression feature vector, a physiological feature vector and a corresponding emotional state label; a processing unit for constructing a neural network model, training the neural network model using the sample data to generate a feature enhancement model, and performing feature enhancement processing on the sample data and training data using the feature enhancement model to generate an enhanced data set, and expanding the enhanced data set using a generative adversarial network to generate a pattern signal set, wherein the pattern signal set is used for constructing a human fatigue state recognition model.

[0013] According to another aspect of the present application, a computer-readable storage medium is provided, which includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute any one of the methods for constructing a pattern signal set for facial expressions representing a human fatigue state.

[0014] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for constructing a pattern signal set for executing any one of the facial expressions used to represent the fatigue state of the human body.

[0015] The technical solution of the present application is used to obtain fatigue status data of a worker, wherein the fatigue status data includes facial expression data, body movement data, and physiological data; determine the feature vectors of the fatigue status data, and determine the worker's emotional state value based on the feature vectors. The worker's fatigue status data is then labeled and classified based on the emotional state value to obtain sample data, and training data is constructed based on the sample data, wherein the training data includes facial expression feature vectors, physiological feature vectors, and corresponding emotional state labels; construct a neural network model, use the sample data to train the neural network model, generate a feature enhancement model, and use the feature enhancement model to perform feature enhancement on the sample data and training data to generate an enhanced dataset, and use a generative adversarial network to expand the enhanced dataset to generate a pattern signal set, wherein the pattern signal set is used to construct a human fatigue status recognition model. By extracting feature points and calculating feature vectors, facial expressions and physiological data can be more accurately quantified, so that the model can more accurately capture subtle changes in fatigue status, providing rich training resources for iterative training and optimization of the model, thereby improving the model's generalization ability and performance in practical applications. This solves the problem of incomplete data used for fatigue recognition model training in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for executing a method for constructing a pattern signal set for representing a human fatigue state through facial expression provided in an embodiment of the present application is shown;

[0018] Figure 2 A schematic flow chart of a method for constructing a pattern signal set for representing a human fatigue state using facial expressions provided in an embodiment of the present application is shown;

[0019] Figure 3 A flowchart of a method for constructing a pattern signal set for representing a human fatigue state using a specific facial expression provided in an embodiment of the present application is shown;

[0020] Figure 4 A module diagram of a system for constructing a pattern signal set for facial expression representation of human fatigue state provided in accordance with an embodiment of the present application is shown;

[0021] Figure 5 A structural block diagram of a device for constructing a pattern signal set for representing a human fatigue state through facial expressions provided in an embodiment of the present application is shown.

[0022] The above drawings include the following reference numerals:

[0023] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. DETAILED DESCRIPTION

[0024] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] As introduced in the background technology, the data used in the existing technology for training fatigue recognition models is incomplete. In order to solve the problem of incomplete data used in the existing technology for training fatigue recognition models, the embodiments of the present application provide a method, device, computer-readable storage medium and electronic device for constructing a pattern signal set for facial expressions to represent the fatigue state of the human body.

[0028] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0029] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a method of constructing a pattern signal set for facial expression representation of human fatigue state according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0030] The memory 104 can be used to store computer programs, such as software programs and modules of application software, such as the computer program corresponding to the method for constructing a pattern signal set for facial expression representation of human fatigue status in an embodiment of the present invention. The processor 102 executes the computer program stored in the memory 104 to execute various functional applications and data processing, thereby implementing the aforementioned method. The memory 104 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. The transmission device 106 is used to receive or transmit data via a network. Specific examples of such networks may include a wireless network provided by the mobile terminal's telecommunications provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] In this embodiment, a method for constructing a pattern signal set of facial expressions representing the fatigue state of the human body running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0032] Figure 2This is a flow chart of a method for constructing a pattern signal set for facial expression representation of human fatigue state according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0033] Step S201, obtaining fatigue status data of an operator, wherein the fatigue status data includes facial expression data, body movement data, and physiological data;

[0034] Specifically, fatigue status data includes setting a miniature camera in the brim of the operator's hat to capture the employee's facial expression at intervals of 2 minutes and record the timestamp. The eye movement data collected by the miniature camera in the brim includes eye movement data and eyelid closure frequency data. The wearable detector records the operator's physiological data including heart rate data, skin temperature data, and electrical skin response data. The motion capture sensor is used to record the operator's body posture and movement changes. Each time a recording is made, the outdoor working environment, including ambient light and noise level, is captured and attached to the data. At the same time, the facial expressions collected by the miniature camera and the body posture data of the motion capture sensor are simultaneously input into the emotional state model, and different emotional states are combined to prevent the occurrence of negative emotions.

[0035] Step S202: determining a feature vector of the fatigue state data, determining an emotional state value of the operator based on the feature vector, performing label classification processing on the fatigue state data of the operator based on the emotional state value to obtain sample data, and constructing training data based on the sample data, wherein the training data includes a facial expression feature vector, a physiological feature vector, and a corresponding emotional state label;

[0036] Specifically, before determining the characteristic vector of the above fatigue status data, the fatigue status data needs to be preprocessed, including preliminary cleaning of facial expression images and physiological data signals, eliminating images with obscured faces, using facial expression recognition technology to extract facial feature points including 68 key points, calculating the characteristic vector of the facial expression image including the eye closure rate, the degree of mouth corner upturn, and the degree of eyebrow upturn, and calculating the average value and variation range of the characteristic vector of the physiological data signal, the heart rate variability and the skin electrode response.

[0037] Step S203: construct a neural network model, use the above-mentioned sample data to train the above-mentioned neural network model, generate a feature enhancement model, and use the above-mentioned feature enhancement model to perform feature enhancement processing on the above-mentioned sample data and training data to generate an enhanced data set, and use a generative adversarial network to expand the above-mentioned enhanced data set to generate a pattern signal set, wherein the above-mentioned pattern signal set is used to construct a human fatigue state recognition model.

[0038] Specifically, data expansion includes using a generative adversarial network to enhance fatigue feature labels and generate fatigue state images with similar features to real samples based on sample data. This solves the problem of insufficient training data for human fatigue state recognition models in existing technologies.

[0039] This embodiment obtains fatigue status data of a worker, wherein the fatigue status data includes facial expression data, body movement data, and physiological data; determines feature vectors of the fatigue status data, determines the worker's emotional state value based on the feature vectors, and performs label classification processing on the worker's fatigue status data based on the emotional state value to obtain sample data. Training data is then constructed based on the sample data, wherein the training data includes facial expression feature vectors, physiological feature vectors, and corresponding emotional state labels; a neural network model is constructed, trained using the sample data to generate a feature enhancement model, and feature enhancement processing is performed on the sample data and training data using the feature enhancement model to generate an enhanced dataset. The enhanced dataset is then augmented using a generative adversarial network to generate a pattern signal set, wherein the pattern signal set is used to construct a human fatigue status recognition model. Feature point extraction and feature vector calculation enable more precise quantification of facial expression and physiological data, enabling the model to more accurately capture subtle changes in fatigue status. This provides rich training resources for iterative training and optimization of the model, thereby improving the model's generalization ability and performance in practical applications. This solves the problem of incomplete data used for fatigue recognition model training in existing technologies.

[0040] In a specific implementation, while training the neural network model using the sample data, the method further includes: determining a loss function and an F1 score for the neural network model, wherein the F1 score represents the degree of match between the predicted emotional state label of the neural network model and the actual emotional state label; and adjusting the model parameters of the neural network model based on the loss function and the F1 score. After each training session, the method uses the F1 score to evaluate the degree of match between the model performance and the actual label, and fine-tunes the neural network model based on the F1 score and the loss function to prevent overfitting of the neural network model.

[0041] Specifically, determining the loss function and F1 score of the neural network model includes: according to the first formula: Determine the above loss function of the above neural network model, where L is the above loss function, N is the number of samples in the data set, and y i is the actual emotional state label of the i-th sample data, P(y i ) is the predicted emotional state label of the i-th sample data; according to the second formula Determine the above F1 score for the above neural network model, where TP is the number of positive examples correctly predicted by the model, FP is the number of negative examples incorrectly predicted as positive by the model, TN is the number of negative examples correctly predicted by the model, FN is the number of positive examples incorrectly predicted as negative by the model, and α is a weighting factor.

[0042] More specifically, adjusting the model parameters of the neural network model according to the loss function and the F1 score includes: according to the third formula: L final =L+ζ·R(Y), determine the target loss function L of the above neural network model final , where L is the above loss function, ζ is the regularization strength, and R(Y) is the regularization term; according to the fourth formula: Determine the dynamic learning rate η of the above neural network model t , where η0 is the initial learning rate, λ c is the decay constant, and fre is the number of iterations during model training. The decay constant in this method is used to control the decay rate of the model learning rate. By determining the dynamic learning rate of the model, the convergence speed of model training is adjusted, thereby improving the generalization performance of the model.

[0043] Furthermore, after the feature enhancement model is used to perform feature enhancement processing on the sample data and the training data to generate an enhanced data set, the method further includes: using the fifth formula: B i,j =Z combined +ε·H(σ)+γ·Morph(F i,j ), add noise processing to the features in the above enhanced dataset to generate the target enhanced dataset, where B i,j is the jth feature of the i-th sample after adding noise, ε is the weight set according to the fatigue level, H(σ) is the Gaussian noise added, γ is the parameter used to control the deformation intensity, Morph(F i,j ) is a morphological variation function of facial features, which is used to simulate the effects of fatigue on facial features. This method adds noise to the enhanced facial feature F′ and physiological feature Phy′ samples after model training to reduce low-quality images and uses the morphological variation function to simulate the effects of fatigue on facial features.

[0044] Furthermore, determining the characteristic vector of the fatigue state data includes: according to the sixth formula: Determine the characteristic vector Z of the fatigue state data, where F i is the feature vector of the i-th facial expression data, Phy i is the eigenvector of the above physiological data, A i is the feature vector of the i-th body motion data, aF is the weight coefficient of the feature vector of the i-th facial expression data, a Phy is the weight coefficient of the eigenvector of the above physiological data, a A is the weight coefficient of the ith feature vector of the above-mentioned body motion data, and n, m, and k respectively represent the number of feature vectors of the above-mentioned facial expression data, feature vectors of physiological data, and feature vectors of body motion data.

[0045] Specifically, label classification processing is performed on the fatigue status data of the operator according to the emotional state value, including: according to the seventh formula: The above emotional state value is adjusted to generate the target emotional state value E(S), where S is the above emotional state value, f1 is the adjustment parameter affecting the exponential decay part, f2 is the adjustment parameter affecting the emotional state on the emotional score, M represents the order of the polynomial, C i represents the coefficient of the i-th high-order term; and performing label classification processing on the fatigue state data of the above-mentioned operator according to the above-mentioned target emotional state value.

[0046] in, S is the emotional state value at time t, S(t-1) is the emotional state at the previous time point t-1, η is the learning rate, set ω = 0.1. ω is the rate adjustment coefficient, λ is the state attenuation factor, is the influence coefficient of external environmental factors, θ is the frequency of the periodic influence factor that simulates the periodic fluctuation of emotions, E(S) represents the emotional state value after adjustment of the current emotional state, f1 affects the adjustment parameter of the exponential decay part, f2 affects the adjustment parameter of the emotional state on the emotional score, M represents the order of the polynomial, C i represents the coefficient of the i-th higher-order term, defining the equilibrium point of the center of gravity of the emotional state.

[0047] For example: when E(S) < 0.4, the operator's emotional value is low and is judged to be emotionally fatigued; when 0.4 ≤ E(S) < 0.7, the operator's emotional value is in the normal range and is judged to be emotionally normal; when E(S) ≥ 0.7, the operator's emotional value is high and is judged to be anti-emotionally excessively fatigued. The classification results are saved in their respective generated data sets and the data sets are marked.

[0048] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the method for constructing a pattern signal set for facial expression representation of human fatigue state of the present application will be described in detail below in combination with specific embodiments.

[0049] This embodiment relates to a specific method for constructing a pattern signal set of facial expressions representing human fatigue state, such as Figure 3 As shown, the following steps are included:

[0050] Step S1: Collect data on the fatigue level of personnel in outdoor work scenarios to construct basic sample data.

[0051] Furthermore, the sample data includes a miniature camera set in the brim of the operator's hat, which captures the employee's facial expressions at intervals of 2 minutes and records the timestamp. The eye movement data collected by the miniature camera set in the brim includes eye movement data and eyelid closure frequency data.

[0052] Wearable detectors are used to record the worker's physiological data, including heart rate data, skin temperature data, and electrical skin response data.

[0053] Use motion capture sensors to record the operator's body posture and movement changes.

[0054] Each time a record is made, the outdoor working environment including ambient lighting and noise level is captured and appended to the data.

[0055] At the same time, the facial expressions collected by the micro camera and the body posture data of the motion capture sensor are input into the emotional state model at the same time, and different emotional states are combined to prevent the occurrence of anti-emotions.

[0056] Preprocessing includes preliminary cleaning of facial expression images and physiological data signals, removing images with occluded faces, extracting facial feature points including 68 key points using facial expression recognition technology, and calculating the feature vector F = [F1, F2, ..., F n ] including eye closure rate, mouth corner uplift degree, eyebrow uplift degree, calculate the characteristic vector Phy of physiological data signal Phy=[Phy1,Phy2,…,Phy m ] Mean values ​​and ranges of heart rate variability and galvanic skin response.

[0057] It should be noted that the facial expression data collected by the micro camera and the body movement data of the motion capture sensor A=[A1, A2, ..., A k ] is input into the emotional state model and the comprehensive feature vector is defined as:

[0058]

[0059] Among them, Z is a comprehensive index describing the emotional state, F is the facial feature vector, and F i is the i-th facial feature, Phy is the physiological signal vector, which contains features related to physiological state, Phy i is the i-th physiological feature, A is the body movement data, A i is the i-th action feature, a Fis the weight coefficient of the i-th facial feature, a Phy is the weight coefficient of the i-th physiological feature, a A is the weight coefficient of the i-th action feature, n, m, k represent the number of facial features, physiological features and action features respectively, i is the variable index, which only represents the index number.

[0060] Observe the variability of emotional states and adjust emotional values:

[0061]

[0062] Where S is the emotional state value at time t, S(t-1) is the emotional state at the previous time point t-1, η is the learning rate, set ω = 0.1. ω is the rate adjustment coefficient, λ is the state attenuation factor, is the influence coefficient of external environmental factors, θ is the frequency of the periodic influence factor that simulates the periodic fluctuation of emotions, E(S) represents the emotional state value after adjustment of the current emotional state, f1 affects the adjustment parameter of the exponential decay part, f2 affects the adjustment parameter of the emotional state on the emotional score, M represents the order of the polynomial, C i represents the coefficient of the i-th higher-order term, defining the equilibrium point of the center of gravity of the emotional state.

[0063] Different classification results of comprehensive emotional states are obtained: when E(S)<0.4, the operator's emotional value is low and is judged as emotional fatigue; when 0.4≤E(S)<0.7, the operator's emotional value is in the normal range and is judged as normal; when E(S)≥0.7, the operator's emotional value is high and is judged as anti-emotional excessive fatigue. The classification results are saved in their respective generated data sets and the data sets are marked.

[0064] Step S2: Construct matching training data through deep learning and perform tuning.

[0065] Furthermore, datasets are constructed for the feature vector F of facial expression images and the feature Phy of physiological data signals, and model training is performed on the datasets through deep learning.

[0066] The feature vector F of the facial expression image and the feature Phy of the physiological data signal are respectively enhanced to F′=CNN(Augment(F)) and Phy′=RNN(Phy). The trained facial features F′ and physiological features Phy′ are combined to form a multimodal dataset Z combined = [F′, Phy′], select the deep learning combination, and combine the feature Z combined enter:

[0067] P(y|Z combined )=ηt Softmax(W·Z combined +b);

[0068] Modify the number of neurons in the output layer to 3, corresponding to different fatigue levels. When there is no fatigue, y = 0. When there is mild fatigue, y = 1. When there is severe fatigue, y = 2.

[0069] Among them, P(y|Z combined ) represents a given eigenvector Z combined When , the model predicts the probability distribution of label y. Softmax represents the activation function, which converts the input into a probability distribution. W represents the weight matrix, which is used to map the feature vector to the prediction space. b represents the bias term, η t is the learning rate of deep learning at time step t.

[0070] Calculate the loss function:

[0071] Among them, L represents the loss function, N represents the number of samples in the dataset, and y i Represents the true label of the i-th sample data.

[0072] After each training session, the F1-score is used to evaluate the matching degree between the model performance and the actual label:

[0073]

[0074] Among them, TP is the number of positive samples correctly predicted by the model, FP is the number of negative samples incorrectly predicted as positive by the model, TN is the number of negative samples correctly predicted by the model, FN is the number of positive samples incorrectly predicted as negative by the model, and α is the weight factor.

[0075] Tuning includes preventing overfitting based on F1_Score and L adjustment, where ζ is the regularization strength and R(Y) is the regularization term:

[0076] Integrate dynamic learning rate:

[0077] Among them, η0 is the initial learning rate, λ c The decay constant is used to control the decay rate of the learning rate. fre is the number of iterations during the training process.

[0078] The optimized results are fed back into the training model, and data enhancement is performed on both the training data and the sample data. The data is then expanded by comparing the enhancement results.

[0079] It should be noted that after model training, the enhanced facial feature F′ and physiological feature Phy′ samples are added with noise to reduce the low-quality image, and the morphological change function is used to simulate the effect of fatigue on facial features:

[0080] B i,j =Z combined +ε·H(σ)+γ·Morph(F i,j );

[0081] Among them, B i,j is the jth feature of the i-th sample after enhancement, i and j only represent variable indexes, ε is the weight set according to the fatigue level, which affects the intensity of the noise. H(σ) is the Gaussian noise added, γ is the parameter used to control the deformation intensity, Morph(F i,j ) is the morphological variation function of facial features.

[0082] Data augmentation involves using a generative adversarial network to generate fatigue state images with similar characteristics to real samples based on the enhanced fatigue feature labels:

[0083] G(z)=D(x)+h1·GAN(z)+δ·Noise(z)·R;

[0084] Among them, G(z) is the generated sample, D(x) is the real sample feature including F and Phy, h1 is the parameter that controls the diversity of generated samples, GAN(z) is the output of the generative adversarial network, z is random noise, δ is the noise effect behind the adjustment of the generated sample, and R is the adversarial loss.

[0085] The generated samples are combined with the augmented real samples to form a new dataset.

[0086] Step S3: Perform data expansion based on data enhancement and generative adversarial network to construct a pattern signal set.

[0087] Furthermore, the expanded facial feature data, physiological feature data, and body movement data are comprehensively aggregated to obtain a comprehensive expanded dataset U, and a multi-index evaluation is performed to comprehensively obtain the fatigue state value:

[0088]

[0089] Where Fat represents the fatigue state value, T is the entire time period, U(t) represents the extended data set at time t, μ is the weight of the adjustment integral term, U(t′) represents the extended data set at time t′, μ2 represents the constant of the characteristic of the adjustment signal, U(τ) represents the extended data set at time u, and w is an arbitrary time period.

[0090] Combine the fatigue state value obtained with the calculated emotional state value to determine the fatigue state range Ω:

[0091]

[0092] Where r(t) is a parameter that adjusts the time dependence of the emotional state value, is the time-dependent standard deviation.

[0093] Different ranges are grouped into different facial expressions to represent the pattern signal sets of human fatigue status. When detecting fatigue values, different fatigue warnings are issued when the range values ​​of different pattern signal sets are met.

[0094] This embodiment can accurately capture facial micro-expressions through a micro-camera, while wearable detectors and motion capture sensors can monitor physiological states and movement changes in real time, thereby ensuring that the sample data can accurately reflect fatigue status; the application of preprocessing technology significantly improves data quality and reduces the impact of noise and invalid data. Through feature point extraction and feature vector calculation, facial expressions and physiological data can be quantified more accurately, providing a strong data foundation for the establishment of emotional state models; by observing the variability of emotional states, the model can more accurately capture subtle changes in fatigue status, and the adjustment of emotional values ​​improves the adaptability of the model. The preservation of classification results and the labeling of data sets provide rich training resources for iterative training and optimization of the model, thereby improving the model's generalization ability and performance in practical applications.

[0095] This embodiment also provides a system for constructing a pattern signal set of facial expressions representing human fatigue status, such as Figure 4 As shown, it includes a data acquisition module, an emotion judgment module, a training module, a data expansion module and a pattern signal set construction module.

[0096] The data acquisition module, including a data acquisition unit and a data processing unit, collects facial expressions, physiological data, body movement data and environmental data of outdoor workers at different fatigue levels and performs preliminary processing.

[0097] The system boots up, and each module completes initialization, preparing for data collection. The data acquisition unit activates its micro-camera to periodically capture the operator's facial expressions and record timestamps. The wearable detector begins recording the operator's heart rate, skin temperature, and galvanic skin response data. The motion capture sensor records the operator's body posture and movements. The environmental monitoring device records the current ambient light and noise level.

[0098] The data processing unit performs preliminary cleaning on the collected facial expression images, removes occluded or low-quality images, extracts facial feature points, calculates the feature vectors of facial expression images and physiological data signals, and passes the processed data to the emotion judgment module.

[0099] The emotion judgment module combines facial expressions, physiological data and body movement data to build an emotional state model and evaluate the emotional state of the operator.

[0100] The emotion judgment module receives the data transmitted by the data acquisition module, inputs the facial expression and body movement data into the emotion state model, defines the comprehensive feature vector, calculates the emotion state value, observes the variability, adjusts the emotion state value, obtains the emotion state results of different categories, and passes the emotion state results and data set labels to the training data module.

[0101] The training module, including the model training unit and the tuning unit, performs deep learning training on facial expressions and physiological data features, constructs a multimodal dataset, and tunes the model.

[0102] The training data module receives the dataset and labels passed by the emotion judgment module. The training unit constructs feature datasets for facial expression images and physiological data signals, respectively. Deep learning models are trained on these datasets, and data enhancement is performed on the trained features to form a multimodal dataset. The tuning unit adjusts model parameters based on the evaluation results to prevent overfitting. The F1-score is used to evaluate model performance and provide feedback adjustments. The tuned model and dataset are then passed to the data augmentation module.

[0103] The data augmentation module, including the enhancement unit and the expansion unit, performs data augmentation and expansion on the trained samples to improve the accuracy of the model in identifying different fatigue states.

[0104] The data augmentation module receives the model and dataset passed by the training data module, adds noise to the enhancement unit, reduces low-quality images, uses the morphological change function to simulate the impact of fatigue on facial features, inputs the augmentation unit, and uses the generative adversarial network to generate fatigue state images with similar features. The generated samples are combined with the enhanced real samples to form a new dataset and passed to the pattern signal set construction module.

[0105] The pattern signal set construction module synthesizes the expanded data set to evaluate the fatigue status of the operators and integrates different ranges of fatigue status into the pattern signal set.

[0106] The pattern signal set construction module receives the new data set and performs a multi-metric assessment on the expanded data set to obtain a fatigue state value. This is combined with the emotional state value to determine the fatigue state range. Fatigue states within different ranges are grouped into different pattern signal sets. When detecting fatigue values, fatigue warnings are issued based on the range values ​​of different pattern signal sets. The system provides feedback based on the warning results and actual fatigue state to optimize the parameters of each module.

[0107] The embodiment of the present application also provides a device for constructing a pattern signal set for facial expression representation of human fatigue state. It should be noted that the device for constructing a pattern signal set for facial expression representation of human fatigue state in the embodiment of the present application can be used to execute the method for constructing a pattern signal set for facial expression representation of human fatigue state provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation methods, and the details that have been explained will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0108] The following introduces the device for constructing a pattern signal set for facial expressions representing human fatigue status provided in an embodiment of the present application.

[0109] Figure 5 Schematic diagram of a device for constructing a pattern signal set for facial expression representation of human fatigue state according to an embodiment of the present application. Figure 5 As shown, the device includes:

[0110] An acquisition unit 51 is used to acquire fatigue status data of an operator, wherein the fatigue status data includes facial expression data, body movement data, and physiological data;

[0111] a first determining unit 52 for determining a feature vector of the fatigue state data, determining an emotional state value of the operator based on the feature vector, performing label classification processing on the fatigue state data of the operator based on the emotional state value to obtain sample data, and constructing training data based on the sample data, wherein the training data includes a facial expression feature vector, a physiological feature vector, and a corresponding emotional state label;

[0112] The processing unit 53 is used to construct a neural network model, use the above-mentioned sample data to train the above-mentioned neural network model, generate a feature enhancement model, and use the above-mentioned feature enhancement model to perform feature enhancement processing on the above-mentioned sample data and training data to generate an enhanced data set, and use a generative adversarial network to expand the above-mentioned enhanced data set to generate a pattern signal set, wherein the above-mentioned pattern signal set is used to construct a human fatigue state recognition model.

[0113] In this embodiment, an acquisition unit is used to acquire fatigue state data of an operator, wherein the fatigue state data includes facial expression data, body movement data, and physiological data; a first determination unit is used to determine a feature vector of the fatigue state data, determine the operator's emotional state value based on the feature vector, and perform label classification processing on the operator's fatigue state data based on the emotional state value to obtain sample data, and construct training data based on the sample data, wherein the training data includes facial expression feature vectors, physiological feature vectors, and corresponding emotional state labels; a processing unit is used to construct a neural network model, train the neural network model using the sample data to generate a feature enhancement model, perform feature enhancement processing on the sample data and training data using the feature enhancement model to generate an enhanced data set, and expand the enhanced data set using a generative adversarial network to generate a pattern signal set, wherein the pattern signal set is used to construct a human fatigue state recognition model. By extracting feature points and calculating feature vectors, facial expressions and physiological data can be more accurately quantified, so that the model can more accurately capture subtle changes in fatigue state, providing rich training resources for iterative training and optimization of the model, thereby improving the model's generalization ability and performance in practical applications. This solves the problem of incomplete data used in fatigue recognition model training in existing technologies.

[0114] As an optional solution, the device also includes a second determination unit and an adjustment unit; the second determination unit is used to determine the loss function and F1 score of the above-mentioned neural network model in the process of training the above-mentioned neural network model using the above-mentioned sample data, wherein the above-mentioned F1 score represents the degree of matching between the predicted emotional state label of the above-mentioned neural network model and the actual emotional state label; the adjustment unit adjusts the model parameters of the above-mentioned neural network model according to the above-mentioned loss function and the above-mentioned F1 score.

[0115] In an optional solution, the second determining unit further includes a first determining module and a second determining module; the second determining module is configured to determine, according to the first formula: Determine the above loss function of the above neural network model, where L is the above loss function, N is the number of samples in the data set, and y i is the actual emotional state label of the i-th sample data, P(y i ) is the predicted emotional state label of the i-th sample data; the second determination module is used to determine the emotional state of the i-th sample data according to the second formula: Determine the above F1 score for the above neural network model, where TP is the number of positive examples correctly predicted by the model, FP is the number of negative examples incorrectly predicted as positive by the model, TN is the number of negative examples correctly predicted by the model, FN is the number of positive examples incorrectly predicted as negative by the model, and α is a weighting factor.

[0116] In an optional solution, the second determining unit further includes a third determining module and a fourth determining module; the third determining module is configured to determine the value of L according to the third formula: final =L+ζ·R(Y), determine the target loss function L of the above neural network model final , where L is the above loss function, ζ is the regularization strength, and R(Y) is the regularization term; the fourth determination module is used according to the fourth formula: Determine the dynamic learning rate η of the above neural network model t , where η0 is the initial learning rate, λ c is the decay constant, and fre is the number of iterations in the model training process.

[0117] In an optional solution, the device further includes a noise processing unit for performing feature enhancement processing on the sample data and the training data using the feature enhancement model to generate an enhanced data set, and then using the fifth formula: B i,j =Z combined +ε·H(σ)+γ·Morph(F i,j ), add noise processing to the features in the above enhanced dataset to generate the target enhanced dataset, where B i,j is the jth feature of the i-th sample after adding noise, ε is the weight set according to the fatigue level, H(σ) is the Gaussian noise added, γ is the parameter used to control the deformation intensity, Morph(F i,j ) is a morphological change function of facial features, which is used to simulate the impact of fatigue on facial features.

[0118] In an optional solution, the first determining unit further includes a fifth determining module, configured to determine the value of the first determining unit according to a sixth formula: Determine the characteristic vector Z of the fatigue state data, where F i is the feature vector of the i-th facial expression data, Phy i is the eigenvector of the above physiological data, A i is the feature vector of the i-th body motion data, a F is the weight coefficient of the feature vector of the i-th facial expression data, a Phy is the weight coefficient of the eigenvector of the above physiological data, a A is the weight coefficient of the ith feature vector of the above-mentioned body motion data, and n, m, and k respectively represent the number of feature vectors of the above-mentioned facial expression data, feature vectors of physiological data, and feature vectors of body motion data.

[0119] In an optional solution, the first determination unit further includes an adjustment processing module and a label classification processing module; the adjustment processing module is configured to determine the label classification according to the seventh formula: The above emotional state value is adjusted to generate the target emotional state value E(S), where S is the above emotional state value, f1 is the adjustment parameter affecting the exponential decay part, f2 is the adjustment parameter affecting the emotional state on the emotional score, M represents the order of the polynomial, C i Represents the coefficient of the i-th high-order term; the label classification processing module is used to perform label classification processing on the above-mentioned fatigue state data of the above-mentioned operator according to the above-mentioned target emotional state value.

[0120] The apparatus for constructing a pattern signal set for facial expressions representing fatigue includes a processor and memory. The acquisition unit, first determination unit, and processing unit are all stored as program units in the memory, and the processor executes the program units stored in the memory to implement the corresponding functions. The modules are all located in the same processor; alternatively, the modules can be located in different processors in any combination.

[0121] The processor includes a core, which retrieves the corresponding program unit from the memory. One or more cores can be set, and the problem of incomplete data used for fatigue recognition model training in existing technologies can be solved by adjusting the core parameters.

[0122] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0123] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program is run, the device where the computer-readable storage medium is located is controlled to execute the method for constructing a pattern signal set for facial expressions to represent the fatigue state of the human body.

[0124] Specifically, the method for constructing a pattern signal set of facial expressions representing the fatigue state of a human body includes:

[0125] Step S201, obtaining fatigue status data of an operator, wherein the fatigue status data includes facial expression data, body movement data, and physiological data;

[0126] Step S202: determining a feature vector of the fatigue state data, determining an emotional state value of the operator based on the feature vector, performing label classification processing on the fatigue state data of the operator based on the emotional state value to obtain sample data, and constructing training data based on the sample data, wherein the training data includes a facial expression feature vector, a physiological feature vector, and a corresponding emotional state label;

[0127] Step S203: construct a neural network model, use the above-mentioned sample data to train the above-mentioned neural network model, generate a feature enhancement model, and use the above-mentioned feature enhancement model to perform feature enhancement processing on the above-mentioned sample data and training data to generate an enhanced data set, and use a generative adversarial network to expand the above-mentioned enhanced data set to generate a pattern signal set, wherein the above-mentioned pattern signal set is used to construct a human fatigue state recognition model.

[0128] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program, when running, executes the method for constructing a pattern signal set of facial expressions representing the fatigue state of the human body.

[0129] Specifically, the method for constructing a pattern signal set of facial expressions representing the fatigue state of a human body includes:

[0130] Step S201, obtaining fatigue status data of an operator, wherein the fatigue status data includes facial expression data, body movement data, and physiological data;

[0131] Step S202: determining a feature vector of the fatigue state data, determining an emotional state value of the operator based on the feature vector, performing label classification processing on the fatigue state data of the operator based on the emotional state value to obtain sample data, and constructing training data based on the sample data, wherein the training data includes a facial expression feature vector, a physiological feature vector, and a corresponding emotional state label;

[0132] Step S203: construct a neural network model, use the above-mentioned sample data to train the above-mentioned neural network model, generate a feature enhancement model, and use the above-mentioned feature enhancement model to perform feature enhancement processing on the above-mentioned sample data and training data to generate an enhanced data set, and use a generative adversarial network to expand the above-mentioned enhanced data set to generate a pattern signal set, wherein the above-mentioned pattern signal set is used to construct a human fatigue state recognition model.

[0133] An embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are performed:

[0134] Step S201, obtaining fatigue status data of an operator, wherein the fatigue status data includes facial expression data, body movement data, and physiological data;

[0135] Step S202: determining a feature vector of the fatigue state data, determining an emotional state value of the operator based on the feature vector, performing label classification processing on the fatigue state data of the operator based on the emotional state value to obtain sample data, and constructing training data based on the sample data, wherein the training data includes a facial expression feature vector, a physiological feature vector, and a corresponding emotional state label;

[0136] Step S203: construct a neural network model, use the above-mentioned sample data to train the above-mentioned neural network model, generate a feature enhancement model, and use the above-mentioned feature enhancement model to perform feature enhancement processing on the above-mentioned sample data and training data to generate an enhanced data set, and use a generative adversarial network to expand the above-mentioned enhanced data set to generate a pattern signal set, wherein the above-mentioned pattern signal set is used to construct a human fatigue state recognition model.

[0137] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0138] The present application also provides a computer program product, which, when executed on a data processing device, is adapted to execute a program for initializing at least the following method steps:

[0139] Step S201, obtaining fatigue status data of an operator, wherein the fatigue status data includes facial expression data, body movement data, and physiological data;

[0140] Step S202: determining a feature vector of the fatigue state data, determining an emotional state value of the operator based on the feature vector, performing label classification processing on the fatigue state data of the operator based on the emotional state value to obtain sample data, and constructing training data based on the sample data, wherein the training data includes a facial expression feature vector, a physiological feature vector, and a corresponding emotional state label;

[0141] Step S203: construct a neural network model, use the above-mentioned sample data to train the above-mentioned neural network model, generate a feature enhancement model, and use the above-mentioned feature enhancement model to perform feature enhancement processing on the above-mentioned sample data and training data to generate an enhanced data set, and use a generative adversarial network to expand the above-mentioned enhanced data set to generate a pattern signal set, wherein the above-mentioned pattern signal set is used to construct a human fatigue state recognition model.

[0142] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, can be centralized on a single computing device, or can be distributed across a network of multiple computing devices. They can be implemented using program code executable by the computing device, and thus, can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described herein can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0143] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0144] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0145] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0147] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0148] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0149] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0150] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0151] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0152] 1) The present invention discloses a method for constructing a pattern signal set for facial expression representation of human fatigue status, comprising: obtaining fatigue status data of an operator, wherein the fatigue status data includes facial expression data, body movement data, and physiological data; determining a feature vector of the fatigue status data, determining the operator's emotional state value based on the feature vector, and performing label classification processing on the operator's fatigue status data based on the emotional state value to obtain sample data, and constructing training data based on the sample data, wherein the training data includes a facial expression feature vector, a physiological feature vector, and a corresponding emotional state label; constructing a neural network model, training the neural network model using the sample data to generate a feature enhancement model, and performing feature enhancement processing on the sample data and training data using the feature enhancement model to generate an enhanced data set, and performing feature point extraction and feature vector calculation on the enhanced data set using a generative adversarial network to more accurately quantify facial expressions and physiological data. This allows the model to more accurately capture subtle changes in fatigue status, providing rich training resources for iterative training and optimization of the model, thereby improving the model's generalization ability and performance in practical applications. This solves the problem of incomplete data used for fatigue recognition model training in the prior art. The expansion process generates a pattern signal set, wherein the pattern signal set is used to construct a human fatigue state recognition model.

[0153] 2) The present application provides a device for constructing a pattern signal set for facial expression representation of human fatigue state, comprising: an acquisition unit for acquiring fatigue state data of an operator, wherein the fatigue state data comprises facial expression data, body movement data and physiological data; a first determination unit for determining a feature vector of the fatigue state data, and determining the emotional state value of the operator based on the feature vector, and performing label classification processing on the fatigue state data of the operator based on the emotional state value to obtain sample data, and constructing training data based on the sample data, wherein the training data comprises facial expression feature vectors, physiological feature vectors and corresponding emotional state labels; a processing unit for constructing a neural network model, using sample data to train the neural network model, generating a feature enhancement model, and using the feature enhancement model to perform feature enhancement processing on the sample data and the training data to generate an enhanced data set, and using a generative adversarial network to expand the enhanced data set to generate a pattern signal set, wherein the pattern signal set is used to construct a human fatigue state recognition model. By extracting feature points and calculating eigenvectors, facial expressions and physiological data can be more accurately quantified. This allows the model to more accurately capture subtle changes in fatigue status, providing rich training resources for iterative training and optimization of the model, thereby improving the model's generalization ability and performance in practical applications. This addresses the problem of incomplete data used in existing fatigue recognition model training.

[0154] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for constructing a pattern signal set for facial expression characterization of human fatigue state, characterized in that: include: Acquiring fatigue status data of the operator, wherein the fatigue status data includes facial expression data, body movement data, and physiological data; Determine a feature vector of the fatigue state data, determine the emotional state value of the operator according to the feature vector, perform label classification processing on the fatigue state data of the operator according to the emotional state value to obtain sample data, and construct training data based on the sample data, wherein the training data includes a facial expression feature vector, a physiological feature vector and a corresponding emotional state label; A neural network model is constructed, and the sample data is used to train the neural network model to generate a feature enhancement model, and the feature enhancement model is used to perform feature enhancement processing on the sample data and training data to generate an enhanced data set, and a generative adversarial network is used to expand the enhanced data set to generate a pattern signal set, wherein the pattern signal set is used to construct a human fatigue state recognition model.

2. The method according to claim 1, characterized in that In the process of training the neural network model using the sample data, the method further includes: Determining a loss function and an F1 score of the neural network model, wherein the F1 score represents a degree of match between a predicted emotional state label of the neural network model and an actual emotional state label; The model parameters of the neural network model are adjusted according to the loss function and the F1 score.

3. The method according to claim 2, characterized in that Determine the loss function and F1 score of the neural network model, including: According to the first formula: Determine the loss function of the neural network model, where L is the loss function, N represents the number of samples in the data set, and y i is the actual emotional state label of the i-th sample data, P(y i ) is the predicted emotional state label of the i-th sample data; According to the second formula: Determine the F1 score of the neural network model, where TP is the number of positive samples correctly predicted by the model, FP is the number of negative samples incorrectly predicted as positive by the model, TN is the number of negative samples correctly predicted by the model, FN is the number of positive samples incorrectly predicted as negative by the model, and α is a weight factor.

4. The method according to claim 2, characterized in that: Adjusting model parameters of the neural network model according to the loss function and the F1 score includes: According to the third formula: L final =L+ζ·R(Y), determine the target loss function L of the neural network model final , where L is the loss function, ζ is the regularization strength, and R(Y) is the regularization term; According to the fourth formula: Determine the dynamic learning rate η of the neural network model t , where η0 is the initial learning rate, λ c is the decay constant, and fre is the number of iterations in the model training process.

5. The method according to claim 1, characterized in that After the feature enhancement model is used to perform feature enhancement processing on the sample data and the training data to generate an enhanced data set, the method further includes: Use the fifth formula: B i,j =Z combined +ε·H(σ)+γ·Morph(F i,j ), adding noise to the features in the enhanced data set to generate a target enhanced data set, where B i,j is the jth feature of the i-th sample after adding noise, ε is the weight set according to the fatigue degree, H(σ) is the added Gaussian noise, γ is the parameter used to control the deformation intensity, Morph(F i,j ) is a morphological change function of facial features, and the morphological change function is used to simulate the influence of fatigue on facial features.

6. The method according to claim 1, characterized in that Determining a characteristic vector of the fatigue state data includes: According to the sixth formula: Determine the characteristic vector Z of the fatigue state data, where F i is the feature vector of the i-th facial expression data, Phy i is the feature vector of the i-th physiological data, A i is the feature vector of the i-th body motion data, a F is the weight coefficient of the feature vector of the i-th facial expression data, a Phy is the weight coefficient of the eigenvector of the i-th physiological data, a A is the weight coefficient of the ith feature vector of the body motion data, and n, m, and k respectively represent the number of feature vectors of the facial expression data, the feature vectors of the physiological data, and the number of feature vectors of the body motion data.

7. The method according to claim 1, characterized in that The fatigue state data of the operator is subjected to label classification processing according to the emotional state value, including: According to the seventh formula: The emotional state value is adjusted to generate a target emotional state value E(S), wherein S is the emotional state value, f1 is an adjustment parameter affecting the exponential decay part, f2 is an adjustment parameter affecting the emotional state on the emotional score, and M represents the order of the polynomial. C i represents the coefficient of the i-th higher-order term; The fatigue state data of the operator is subjected to label classification processing according to the target emotional state value.

8. A device for constructing a pattern signal set for facial expression characterization of human fatigue state, characterized in that: include: An acquisition unit, used for acquiring fatigue status data of an operator, wherein the fatigue status data includes facial expression data, body movement data and physiological data; A first determination unit is used to determine a feature vector of the fatigue state data, determine an emotional state value of the operator according to the feature vector, perform label classification processing on the fatigue state data of the operator according to the emotional state value to obtain sample data, and construct training data based on the sample data, wherein the training data includes a facial expression feature vector, a physiological feature vector, and a corresponding emotional state label; A processing unit is used to construct a neural network model, use the sample data to train the neural network model to generate a feature enhancement model, and use the feature enhancement model to perform feature enhancement processing on the sample data and training data to generate an enhanced data set, and use a generative adversarial network to expand the enhanced data set to generate a pattern signal set, wherein the pattern signal set is used to construct a human fatigue state recognition model.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for constructing a pattern signal set for representing a human fatigue state through facial expressions as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for constructing a pattern signal set for executing facial expressions to represent human fatigue status as described in any one of claims 1 to 7.

Images