Emotion recognition method and device for road and bridge engineering scenes

Abstract

The present invention provides an emotion recognition method and device for road and bridge engineering scenarios, relating to the field of emotion recognition technology. The present invention pre-processes the acquired user's facial video and physiological information, then performs feature extraction to obtain the user's facial motion units and physiological features; then selects and fuses the optimal feature subset of the user's facial motion units and physiological features; and finally, based on the multi-channel feature inverse reasoning of a dynamic Bayesian network and an interpretable emotion recognition model, obtains the mapping relationship between the fused optimal feature subset and the emotion component, thereby obtaining the user's emotional state recognition result. Compared with the existing technology, the emotion recognition system of the present invention is more reliable and robust, and can achieve multi-level, all-round, and stable emotion recognition of users in the complex environment of road and bridge engineering.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition technology, and in particular to an emotion recognition method and device for road and bridge engineering scenarios. Background Art

[0002] Emotions not only have a significant impact on a person's physical health, but also significantly influence their lives and work. Due to the high workload and emotional stability required of workers in road and bridge construction projects, emotional control is particularly important for workers in these scenarios. Emotional instability can impact their physical and mental health at best, and even hinder the continued stable progress of the project. Losing control of emotions can easily lead to major accidents, necessitating an accurate and effective emotion recognition technology for workers in these scenarios.

[0003] Most traditional emotion recognition methods only focus on single body signals such as facial expressions or voice signals to identify the emotions of the subject, and most of them focus on identifying non-contact body signals such as facial expressions and voice.

[0004] However, road and bridge construction scenarios are often complex, characterized by high noise, high temperature, and high humidity. Facial expressions and voice signals from road and bridge workers cannot be accurately and stably acquired. Without other reliable information to supplement this, emotion recognition of the subject will be subject to significant errors or even mistakes. Furthermore, road and bridge workers, for various reasons, may control and conceal their true emotions. Without other stable contact information (such as their physiological information) for verification and integration, relying solely on single-modal, non-contact information for emotion recognition will also result in significant errors. Therefore, traditional emotion recognition methods are not suitable for emotion recognition of road and bridge workers in road and bridge construction scenarios. Summary of the Invention

[0005] (1) Technical problems solved

[0006] In response to the shortcomings of the existing technology, the present invention provides an emotion recognition method and device for road and bridge engineering scenarios, which solves the problem of low accuracy in emotion recognition of road and bridge workers in road and bridge engineering scenarios in the existing technology.

[0007] (2) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] In a first aspect, the present invention first proposes an emotion recognition method for a road and bridge engineering scene, the method comprising:

[0010] Obtaining and preprocessing the user's facial video and physiological information, and extracting the user's facial motion units based on the preprocessed facial video using a deep facial segmentation network based on U-NET, and extracting the user's physiological features based on the preprocessed physiological information;

[0011] Obtaining an optimal feature subset of the facial motion unit and the physiological feature, and fusing the optimal feature subset based on a feature verification and fusion mechanism of a deep neural network and transfer learning;

[0012] The multi-channel feature reverse reasoning and explainable emotion recognition model based on the dynamic Bayesian network obtains the mapping relationship between the fused optimal feature subset and the emotion component, and obtains the user's emotional state recognition result based on the mapping relationship.

[0013] Preferably, the acquiring of the user's facial video and physiological information and pre-processing includes:

[0014] S11. Capturing a user's facial video using a camera and recording the user's physiological signals using a wristband; the physiological signals include: heart rate variability signals, respiratory variability signals, and skin electrical signals;

[0015] S12. Detecting faces from the video sequence using facial feature point positioning and face detection algorithms, and then cropping faces from the video using the Dlib facial feature point positioning model and a face edge detection mechanism based on a Gaussian difference filter to obtain a high-definition face image set.

[0016] S13. Perform a detrending operation and a fast Fourier transform on the heart rate variability signal to obtain time domain and frequency domain indicators of the heart rate variability signal related to emotion recognition; use a Butterworth low-pass filter to remove interference signals from the respiratory variability signal and the skin electrical signal, respectively, to obtain respiratory variability indicators and skin electrical indicators related to emotion recognition.

[0017] Preferably, the obtaining of the optimal feature subset of the facial motion unit and the physiological feature includes: obtaining the optimal feature subset of the facial motion unit and the physiological feature using a random forest model.

[0018] Preferably, the multi-channel feature reverse reasoning and explainable emotion recognition model of the dynamic Bayesian network adds a multi-channel feature reverse reasoning mechanism to the dynamic Bayesian network.

[0019] In a second aspect, the present invention further proposes an emotion recognition device for road and bridge engineering scenarios, the device comprising:

[0020] A signal acquisition and feature extraction module is used to obtain and preprocess the user's facial video and physiological information, extract the user's facial motion units based on the preprocessed facial video using a deep facial segmentation network based on U-NET, and extract the user's physiological features based on the preprocessed physiological information;

[0021] An optimal feature subset acquisition and fusion module, configured to acquire the optimal feature subsets of the facial motion units and the physiological features, and fuse the optimal feature subsets based on a feature verification and fusion mechanism of a deep neural network and transfer learning;

[0022] The emotional state recognition module is used to obtain the mapping relationship between the fused optimal feature subset and the emotional component based on the multi-channel feature inverse reasoning of the dynamic Bayesian network and the explainable emotion recognition model, and obtain the user's emotional state recognition result based on the mapping relationship.

[0023] Preferably, the signal acquisition and feature extraction module acquires the user's facial video and physiological information and performs preprocessing, including:

[0024] S11. Capturing a user's facial video using a camera and recording the user's physiological signals using a wristband; the physiological signals include: heart rate variability signals, respiratory variability signals, and skin electrical signals;

[0025] S12. Detecting faces from the video sequence using facial feature point positioning and face detection algorithms, and then cropping faces from the video using the Dlib facial feature point positioning model and a face edge detection mechanism based on a Gaussian difference filter to obtain a high-definition face image set.

[0026] S13. Perform a detrending operation and a fast Fourier transform on the heart rate variability signal to obtain time domain and frequency domain indicators of the heart rate variability signal related to emotion recognition; use a Butterworth low-pass filter to remove interference signals from the respiratory variability signal and the skin electrical signal, respectively, to obtain respiratory variability indicators and skin electrical indicators related to emotion recognition.

[0027] Preferably, the optimal feature subset acquisition and fusion module acquires the optimal feature subset of the facial motion unit and the physiological feature, including: acquiring the optimal feature subset of the facial motion unit and the physiological feature using a random forest model.

[0028] Preferably, the multi-channel feature reverse reasoning and explainable emotion recognition model of the dynamic Bayesian network in the emotional state recognition module adds a multi-channel feature reverse reasoning mechanism to the dynamic Bayesian network.

[0029] In a third aspect, the present invention further proposes a computer-readable storage medium storing a computer program for emotion recognition for road and bridge engineering scenes, wherein the computer program enables a computer to execute the emotion recognition method for road and bridge engineering scenes as described above.

[0030] In a fourth aspect, the present invention further provides an electronic device, comprising:

[0031] one or more processors;

[0032] Memory; and

[0033] One or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the emotion recognition method for the road and bridge engineering scene as described above.

[0034] (3) Beneficial effects

[0035] The present invention provides an emotion recognition method and device for road and bridge engineering scenarios. Compared with the existing technology, it has the following advantages:

[0036] 1. The present invention pre-processes the acquired facial video and physiological information of the user and then extracts features to obtain the user's facial motion units and physiological features; then selects the optimal feature subset of the user's facial motion units and physiological features and fuses them; finally, based on the multi-channel feature inverse reasoning of the dynamic Bayesian network and the interpretable emotion recognition model, the mapping relationship between the fused optimal feature subset and the emotion component is obtained to obtain the user's emotional state recognition result. The present invention fuses multi-dimensional features such as the user's facial video and contact physiological information to realize the user's emotion recognition in the complex scene of road and bridge engineering. Compared with the existing technology, the emotion recognition system of the present invention is more reliable and robust, and can realize multi-level, all-round and stable emotion recognition of users in the complex environment of road and bridge engineering.

[0037] 2. The present invention makes full use of the multimodal, heterogeneous, and multidimensional body and signals such as the user's facial motion unit, heart rate variability, respiratory variability, and skin electricity for emotion recognition, ensuring the accuracy of user emotion recognition at multiple levels and in all directions.

[0038] 3. The present invention constructs an emotion recognition architecture that mainly uses non-contact data from visible light videos of users' faces and uses contact physiological signal data as verification and compensation, as well as a multi-channel feature reverse reasoning and interpretable emotion recognition model based on a dynamic Bayesian network. This enhances the stability and robustness of the emotion recognition model and can realize the emotion recognition and monitoring of construction workers in complex scenarios of road and bridge projects. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 This is a flow chart of an emotion recognition method for road and bridge engineering scenes according to the present invention;

[0041] Figure 2 This is a diagram illustrating an embodiment of the emotion recognition method for road and bridge engineering scenarios in the present invention;

[0042] Figure 3 Schematic diagram of facial motion unit extraction in an embodiment of the present invention;

[0043] Figure 4 This is a diagram of the u-net deep neural network architecture in an embodiment of the present invention;

[0044] Figure 5 Schematic diagram of a random forest model in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by ordinary technicians in this field based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0046] The embodiments of the present application provide an emotion recognition method and device for road and bridge engineering scenarios, thereby solving the problem of low accuracy in emotion recognition of road and bridge workers in road and bridge engineering scenarios in the existing technology, and achieving highly reliable, multi-level and comprehensive emotion recognition of construction workers in complex scenarios.

[0047] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0048] To enable multi-level, comprehensive, and stable emotion recognition for construction workers in complex road and bridge construction scenarios, the present invention first obtains and preprocesses a user's facial video and physiological information, extracting the user's facial motion units and physiological features. It then selects an optimal feature subset of these facial motion units and physiological features, and fuses this optimal feature subset using a feature validation and fusion mechanism based on deep neural networks and transfer learning. Finally, a mapping relationship between the optimal feature subset and emotional components is derived using a dynamic Bayesian network-based multi-channel feature inverse reasoning and an interpretable emotion recognition model. Based on this mapping relationship, the user's emotional state recognition result is obtained. This present invention enables accurate and highly reliable emotion recognition for construction workers in complex scenarios such as road and bridge construction.

[0049] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0050] The emotion recognition method for road and bridge engineering scenes proposed in the present invention is not limited to the emotion recognition of road and bridge workers in complex road and bridge engineering scenes, but can also be used for emotion recognition of workers in other complex environments such as high noise, high temperature, and high humidity.

[0051] Example 1:

[0052] In the first aspect, the present invention first proposes an emotion recognition method for road and bridge engineering scenes, see Figure 1-2 , the method comprising:

[0053] S1. Obtain and preprocess the user's facial video and physiological information, extract the user's facial motion units based on the preprocessed facial video using a deep facial segmentation network based on U-NET, and extract the user's physiological features based on the preprocessed physiological information;

[0054] S2. Obtain an optimal feature subset of the facial motion unit and the physiological feature, and fuse the optimal feature subset based on a feature verification and fusion mechanism of a deep neural network and transfer learning;

[0055] S3. A multi-channel feature reverse reasoning and explainable emotion recognition model based on a dynamic Bayesian network obtains a mapping relationship between the fused optimal feature subset and the emotion component, and obtains a user's emotional state recognition result based on the mapping relationship.

[0056] It can be seen that this embodiment pre-processes the acquired facial video and physiological information of the user and then extracts features to obtain the user's facial motion units and physiological features; then selects the optimal feature subset of the user's facial motion units and physiological features and fuses them; finally, based on the multi-channel feature inverse reasoning of the dynamic Bayesian network and the interpretable emotion recognition model, the mapping relationship between the fused optimal feature subset and the emotion component is obtained to obtain the user's emotional state recognition result. The present invention fuses multi-dimensional features such as the user's facial video and contact physiological information to realize the user's emotion recognition in the complex scene of the road and bridge engineering. Compared with the existing technology, the emotion recognition system of the present invention is more reliable and robust, and can realize multi-level, all-round and stable emotion recognition of users in the complex environment of the road and bridge engineering.

[0057] The following takes the emotion monitoring of construction workers in complex road and bridge engineering as an example, combined with the Figure 1-5 The implementation process of an embodiment of the present invention is described in detail with explanations of the specific steps S1-S3.

[0058] S1. Obtain the user's facial video and physiological information and preprocess them. Based on the preprocessed facial video, use the U-NET-based deep facial segmentation network to extract the user's facial motion units, and based on the preprocessed physiological information, extract the user's physiological features.

[0059] S11. Capture the user's facial video using a camera, and record the user's physiological signals using a wristband, where the user's physiological signals include: heart rate variability signals, respiratory variability signals, and skin electrical signals.

[0060] When performing emotion recognition on users in road and bridge engineering scenarios, a camera is used to capture the user's facial video, while a wristband is used to record the user's physiological signals. The original facial video and physiological signals of the user are then preprocessed, and feature extraction is performed based on the preprocessed facial video and physiological signals. Specifically:

[0061] S12. Detect faces from video sequences using facial feature point positioning and face detection algorithms. Then, use the Dlib facial feature point positioning model and the face edge detection mechanism based on the Gaussian difference filter to crop faces from the video to obtain a set of high-definition face images.

[0062] Facial video preprocessing.

[0063] In the preprocessing process of facial videos, we first need to preprocess the facial video clips, use facial feature point positioning and face detection algorithms to detect faces from video sequences, use the Dlib facial feature point positioning model, and apply the facial edge detection mechanism proposed based on the Gaussian difference filter to crop the faces in the video, separate the faces from the background image, and complete a series of operations such as face calibration, alignment, and fill lighting to obtain a high-definition face image set.

[0064] Extraction of facial motion units (AUs). Research on data enhancement algorithms based on adversarial learning mechanisms, innovative facial feature analysis methods based on LBP-TOP, such as eye movement trajectory, head posture, micro-expressions, and visual feature calibration technology, give full play to the advantages of multi-task learning, and then design a deep two-stream neural network model for mining AU motion unit convolution feature map flow and optical flow information, such as Figure 3 At the same time, in order to solve the problem that the facial motion unit area is difficult to locate in traditional algorithms, we innovatively developed a deep facial segmentation network based on U-NET, see Figure 4 The segmentation network consists of a series of Aures-block attention residuals. This design allows input information to be directly transferred to subsequent layers, avoiding the vanishing gradient problem caused by stacking network layers. Because AUs are extracted for specific facial regions (such as the eyes and mouth), a deep face segmentation network based on U-Net can segment these regions. A deep two-stream neural model is then used to extract AU information from these segmented facial regions. In this process, U-Net primarily addresses the difficulty of AU localization. After segmentation using U-Net, the deep two-stream network model eliminates the need for additional AU localization when extracting AUs. Furthermore, a weighted fusion method is designed for input and output information within the blocks, and an attention mechanism is used to enhance the self-learning of facial spatial features in the feature map, enabling the accurate extraction of information for 16 facial AUs. After obtaining the AU values ​​using the above method, they are first normalized using min-max normalization to prevent feature values ​​from being too large or too small. Finally, in order to reduce the interference of other irrelevant features and make the corresponding features more prominent, the normalized results are squared to achieve more accurate classification. The specific process can be seen in the following formula:

[0065]

[0066] Since each emotional state corresponds to multiple AU motion units, in order to more accurately recognize emotions, we analyze the relationship between different emotions and AU motion units and combine the 16 AUs into 6 combinations. We can obtain the eigenvalue of the combination through the following formula:

[0067]

[0068] Where m represents the number of AU motion units in each AU motion unit combination. Where k represents the kth AU motion combination in the six AU motion combinations. i ' represents the value of AU motion unit in each combination.

[0069] S13. Perform a detrending operation and a fast Fourier transform on the heart rate variability signal to obtain time domain and frequency domain indicators of the heart rate variability signal related to emotion recognition; use a Butterworth low-pass filter to remove interference signals from the respiratory variability signal and the skin electrical signal, respectively, to obtain respiratory variability indicators and skin electrical indicators related to emotion recognition.

[0070] Extraction of contact features. Emotions are the experience and perception of changes in the circulatory, digestive, endocrine, and sympathetic nervous systems, all triggered by stimuli. Therefore, emotions can be detected by analyzing the patterns of physiological changes associated with each emotion. Furthermore, because the physiological changes associated with emotions are autonomous and not controlled by subjective will, these physiological signals can serve as an auxiliary tool for emotion recognition.

[0071] Extracting heart rate variability. Extracting PPG signals from contactless devices. To better extract emotion-related features, the collected raw PPG signals require preprocessing. This preprocessing primarily involves eliminating trend terms and removing low- and high-frequency interference.

[0072] Zero-point drift in the signal, caused by factors such as acquisition equipment, often causes the signal to deviate from the baseline, and the magnitude of the deviation can even vary over time. This entire process of baseline deviation over time is called the signal's trend term. This trend term can affect signal quality and accuracy, so detrending is necessary to eliminate it from PPG signals.

[0073] Furthermore, the normal human heart rate is 45-200 bpm, corresponding to a frequency band of 0.75-3.33 Hz. Outside this frequency band, there are some low- and high-frequency interference signals that can significantly interfere with the extraction of heart rate variability. We use a fast Fourier transform (FFT) to transform the time domain signal into the frequency domain. We then use a bandpass filter to retain data with frequencies between 0.75 and 3.33 Hz and reset values ​​outside of this range to zero. This method not only eliminates noise frequencies, but also transforms the time domain signal into the frequency domain, extracting frequency-domain information relevant to emotion recognition.

[0074] After preprocessing the signal, time domain and frequency domain indicators related to emotion recognition are extracted from the obtained PPG signal for subsequent state recognition.

[0075] Extraction of respiratory variability. The frequency of a normal person's respiratory signal is 0-0.35Hz. During the acquisition process, the respiratory signal will be affected by high-frequency noise such as power frequency and myoelectric interference. Therefore, based on the frequency range of the respiratory signal, a Butterworth low-pass filter is selected to remove interference. Its cutoff frequency is 0.4Hz. After filtering, a relatively pure signal can be obtained. From this, respiratory variability indicators related to emotion recognition can be extracted for subsequent state recognition. The Butterworth filter is defined as:

[0076]

[0077] Among them, ω p and ω are the cutoff frequencies of the upper and lower limits of the passband, and N is the order of the Butterworth filter. Typically, the maximum attenuation allowed by the passband is selected to be 3dB, in which case ε = 1.

[0078] Processing of electrical skin signals. Electrical skin signals are weak bioelectrical signals from the human body. During acquisition, they are susceptible to interference from electromagnetic signals, noise from electrode-skin contact, power frequency artifacts during experiments, and motion artifacts. These factors can overwhelm the electrical skin signals, adversely affecting subsequent feature extraction.

[0079] The effective frequency range of human skin electrodermal signals is 0.02 to 0.20 Hz, which is lower than the frequency of most interfering signals and does not overlap with the spectrum of other physiological signals, such as noise. Therefore, the frequency bands of noise and SCR response signals are separated. Therefore, a Butterworth low-pass filter can be used to remove noise outside the skin electrodermal signal band and eliminate high-frequency interference. Removing high-frequency components is also part of signal smoothing. Using a Butterworth low-pass filter, useless noise signals are removed to the greatest extent possible, ultimately resulting in a pure skin electrodermal signal. Skin electrodermal indicators relevant to emotion recognition can then be extracted from this pure skin electrodermal signal for subsequent state recognition.

[0080] S2. Obtain an optimal feature subset of the facial motion unit and the physiological feature, and fuse the optimal feature subset based on a feature verification and fusion mechanism of a deep neural network and transfer learning.

[0081] Since a data set often has hundreds or even thousands of features in reality, selecting the features that have the greatest impact on the results can reduce the number of features when building the model to simplify the model building and eliminate the complicated calculations caused by redundant data. In this embodiment, we use random forest to screen the above features. The random forest model is as follows: Figure 5 shown.

[0082] 1) Calculate the Gini index. Assume there are c features X1, X2, X3, ..., X c Now we need to calculate each feature X j The Gini index is calculated as follows:

[0083]

[0084] Here, M represents the number of categories, and pcm represents the proportion of category m in node c. Intuitively, it is the probability that two samples randomly drawn from node c will have inconsistent category labels.

[0085] 2) Calculate the variable importance score. Feature X j The importance of node c, that is, the change in the Gini index before and after the node c score is:

[0086]

[0087] Among them, G l and G r Represents the Gini index of the two new nodes before and after the score. If the feature X j The nodes that appear in the decision tree are set C, then X j The importance of the i-th tree is:

[0088]

[0089] Assume that there are n trees in the random forest, then

[0090]

[0091] Finally, all the obtained importance scores are normalized.

[0092]

[0093] 3) Optimal feature subset selection: By setting an appropriate threshold, all features with a value greater than the threshold are selected, which is the optimal feature subset and applied to emotion recognition.

[0094] 4) Feature verification and fusion. After obtaining facial motion units and contact physiological features (heart rate variability, respiratory variability, skin electricity, etc.), an emotion recognition system is constructed that uses non-contact data of facial visible light video as the main data and contact physiological signal data as verification compensation. Through diversified combination modeling methods such as model fusion and unified modeling, a data layer and semantic layer feature verification and fusion mechanism based on deep neural network and transfer learning is established, and the optimal feature subset selected above is verified and fused through this feature verification and fusion mechanism.

[0095] S3. Construct a multi-channel feature reverse reasoning and explainable emotion recognition model based on a dynamic Bayesian network, obtain the mapping relationship between the fused optimal feature subset and the POMS emotion component based on the model, and obtain the user's emotional state recognition result based on the mapping relationship.

[0096] After extracting and fusing features from the user's facial motion units and contact physiological parameters, a multi-channel feature inverse reasoning process was added to the dynamic Bayesian network to overcome the low robustness and high computational cost of existing emotion recognition models, as well as to improve the accuracy and interpretability of emotion recognition. This led to the construction of a multi-channel feature inverse reasoning and interpretable emotion recognition architecture based on the dynamic Bayesian network. This architecture aims to explore the correspondence between the POMS emotion component and the features of both contactless and contact data, thereby obtaining a more robust model and continuously improving the accuracy and interpretability of emotion recognition. Specifically, the fused contactless and contact data are fed into the dynamic Bayesian network as input, with the POMS emotion component being used as the result of emotion recognition. The addition of a reverse reasoning process uses the POMS emotion component as input to identify the correspondence between the POMS emotion component and the features of both contactless and non-contact data.

[0097] Finally, after training the multi-channel feature reverse reasoning and interpretable emotion recognition architecture model based on the dynamic Bayesian network, we use the POMS emotion components corresponding to the contactless and contact data features input into the model as the user's emotional state recognition results.

[0098] At this point, the entire process of the emotion recognition method for road and bridge engineering scenes of the present invention is completed.

[0099] In addition, the technical solution of the present invention is not only applicable to the emotion recognition of construction workers in complex scenes of road and bridge projects, but also to the emotion recognition of construction workers in other complex environments with high noise, high temperature, and high humidity for a long time. The facial video and contact physiological information of the construction workers are collected under weak interference conditions. The facial video of the subject is preprocessed and the facial motion unit information of the subject is extracted using a deep facial segmentation network. After preprocessing the contact physiological information, multi-dimensional information such as heart rate variability, respiratory variability, and skin electricity is extracted. After dimensionality reduction and fusion of the multi-dimensional information, the emotional state of the subject is identified, and a highly reliable, multi-level, and comprehensive emotion recognition system is constructed, which can achieve stable and robust emotion recognition of the subject in complex environments.

[0100] Example 2:

[0101] In a second aspect, the present invention further provides an emotion recognition device for road and bridge engineering scenes, the device comprising:

[0102] A signal acquisition and feature extraction module is used to obtain and preprocess the user's facial video and physiological information, extract the user's facial motion units based on the preprocessed facial video using a deep facial segmentation network based on U-NET, and extract the user's physiological features based on the preprocessed physiological information;

[0103] An optimal feature subset acquisition and fusion module, configured to acquire the optimal feature subsets of the facial motion units and the physiological features, and fuse the optimal feature subsets based on a feature verification and fusion mechanism of a deep neural network and transfer learning;

[0104] The emotional state recognition module is used to obtain the mapping relationship between the fused optimal feature subset and the emotional component based on the multi-channel feature inverse reasoning of the dynamic Bayesian network and the explainable emotion recognition model, and obtain the user's emotional state recognition result based on the mapping relationship.

[0105] Optionally, the signal acquisition and feature extraction module acquires the user's facial video and physiological information and performs preprocessing, including:

[0106] S11. Capturing a user's facial video using a camera and recording the user's physiological signals using a wristband; the physiological signals include: heart rate variability signals, respiratory variability signals, and skin electrical signals;

[0107] S12. Detecting faces from the video sequence using facial feature point positioning and face detection algorithms, and then cropping faces from the video using the Dlib facial feature point positioning model and a face edge detection mechanism based on a Gaussian difference filter to obtain a high-definition face image set.

[0108] S13. Perform a detrending operation and a fast Fourier transform on the heart rate variability signal to obtain time domain and frequency domain indicators of the heart rate variability signal related to emotion recognition; use a Butterworth low-pass filter to remove interference signals from the respiratory variability signal and the skin electrical signal, respectively, to obtain respiratory variability indicators and skin electrical indicators related to emotion recognition.

[0109] Optionally, the optimal feature subset acquisition and fusion module acquires the optimal feature subset of the facial motion unit and the physiological feature, including: acquiring the optimal feature subset of the facial motion unit and the physiological feature using a random forest model.

[0110] Optionally, the multi-channel feature reverse reasoning and explainable emotion recognition model of the dynamic Bayesian network in the emotional state recognition module adds a multi-channel feature reverse reasoning mechanism to the dynamic Bayesian network.

[0111] It can be understood that the emotion recognition device for road and bridge engineering scenarios provided in the embodiment of the present invention corresponds to the above-mentioned emotion recognition method for road and bridge engineering scenarios. The explanations, examples, beneficial effects, etc. of the relevant contents can refer to the corresponding contents in the emotion recognition method for road and bridge engineering scenarios, and will not be repeated here.

[0112] Example 3:

[0113] In a third aspect, the present invention further proposes a computer-readable storage medium storing a computer program for emotion recognition for road and bridge engineering scenes, wherein the computer program enables a computer to execute the emotion recognition method for road and bridge engineering scenes as described above.

[0114] Example 4:

[0115] In a fourth aspect, the present invention further provides an electronic device, comprising:

[0116] one or more processors;

[0117] Memory; and

[0118] One or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for executing the emotion recognition method for the road and bridge engineering scene as described above.

[0119] In summary, compared with the existing technology, the present invention has the following beneficial effects:

[0120] 1. The present invention pre-processes the acquired facial video and physiological information of the user and then extracts features to obtain the user's facial motion units and physiological features; then selects the optimal feature subset of the user's facial motion units and physiological features and fuses them; finally, based on the multi-channel feature inverse reasoning of the dynamic Bayesian network and the interpretable emotion recognition model, the mapping relationship between the fused optimal feature subset and the emotion component is obtained to obtain the user's emotional state recognition result. The present invention fuses multi-dimensional features such as the user's facial video and contact physiological information to realize the user's emotion recognition in the complex scene of road and bridge engineering. Compared with the existing technology, the emotion recognition system of the present invention is more reliable and robust, and can realize multi-level, all-round and stable emotion recognition of users in the complex environment of road and bridge engineering.

[0121] 2. The present invention makes full use of the multimodal, heterogeneous, and multidimensional body and signals such as the user's facial motion unit, heart rate variability, respiratory variability, and skin electricity for emotion recognition, ensuring the accuracy of user emotion recognition at multiple levels and in all directions.

[0122] 3. The present invention constructs an emotion recognition architecture that mainly uses non-contact data from visible light videos of users' faces and uses contact physiological signal data as verification and compensation, as well as a multi-channel feature reverse reasoning and interpretable emotion recognition model based on a dynamic Bayesian network. This enhances the stability and robustness of the emotion recognition model and can realize the emotion recognition and monitoring of construction workers in complex scenarios of road and bridge projects.

[0123] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0124] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An emotion recognition method for road and bridge engineering scenes, characterized by: The method comprises: Obtaining and preprocessing the user's facial video and physiological information, and extracting the user's facial motion units based on the preprocessed facial video using a deep facial segmentation network based on U-NET, and extracting the user's physiological features based on the preprocessed physiological information; Obtaining an optimal feature subset of the facial motion unit and the physiological feature, and fusing the optimal feature subset based on a feature verification and fusion mechanism of a deep neural network and transfer learning; The multi-channel feature reverse reasoning and explainable emotion recognition model based on the dynamic Bayesian network obtains the mapping relationship between the fused optimal feature subset and the emotion component, and obtains the user's emotional state recognition result based on the mapping relationship.

2. The method according to claim 1, wherein The acquiring of the user's facial video and physiological information and preprocessing thereof includes: S11. Capturing a user's facial video using a camera and recording the user's physiological signals using a wristband; the physiological signals include: heart rate variability signals, respiratory variability signals, and skin electrical signals; S12. Detecting faces from the video sequence using facial feature point positioning and face detection algorithms, and then cropping faces from the video using the Dlib facial feature point positioning model and a face edge detection mechanism based on a Gaussian difference filter to obtain a high-definition face image set. S13. Perform a detrending operation and a fast Fourier transform on the heart rate variability signal to obtain time domain and frequency domain indicators of the heart rate variability signal related to emotion recognition; use a Butterworth low-pass filter to remove interference signals from the respiratory variability signal and the skin electrical signal, respectively, to obtain respiratory variability indicators and skin electrical indicators related to emotion recognition.

3. The method according to claim 1, wherein The obtaining of the optimal feature subset of the facial motion unit and the physiological feature includes: obtaining the optimal feature subset of the facial motion unit and the physiological feature using a random forest model.

4. The method according to claim 1, wherein The multi-channel feature reverse reasoning and explainable emotion recognition model of the dynamic Bayesian network adds a multi-channel feature reverse reasoning mechanism to the dynamic Bayesian network.

5. An emotion recognition device for road and bridge engineering scenes, characterized in that: The device comprises: A signal acquisition and feature extraction module is used to obtain and preprocess the user's facial video and physiological information, extract the user's facial motion units based on the preprocessed facial video using a deep facial segmentation network based on U-NET, and extract the user's physiological features based on the preprocessed physiological information; An optimal feature subset acquisition and fusion module, configured to acquire the optimal feature subsets of the facial motion units and the physiological features, and fuse the optimal feature subsets based on a feature verification and fusion mechanism of a deep neural network and transfer learning; The emotional state recognition module is used to obtain the mapping relationship between the fused optimal feature subset and the emotional component based on the multi-channel feature inverse reasoning of the dynamic Bayesian network and the explainable emotion recognition model, and obtain the user's emotional state recognition result based on the mapping relationship.

6. The device according to claim 5, characterized in that The signal acquisition and feature extraction module acquires the user's facial video and physiological information and performs preprocessing, including: S11. Capturing a user's facial video using a camera and recording the user's physiological signals using a wristband; the physiological signals include: heart rate variability signals, respiratory variability signals, and skin electrical signals; S12. Detecting faces from the video sequence using facial feature point positioning and face detection algorithms, and then cropping faces from the video using the Dlib facial feature point positioning model and a face edge detection mechanism based on a Gaussian difference filter to obtain a high-definition face image set. S13. Perform a detrending operation and a fast Fourier transform on the heart rate variability signal to obtain time domain and frequency domain indicators of the heart rate variability signal related to emotion recognition; use a Butterworth low-pass filter to remove interference signals from the respiratory variability signal and the skin electrical signal, respectively, to obtain respiratory variability indicators and skin electrical indicators related to emotion recognition.

7. The device according to claim 5, characterized in that The optimal feature subset acquisition and fusion module acquires the optimal feature subset of the facial motion unit and the physiological feature, including: using a random forest model to acquire the optimal feature subset of the facial motion unit and the physiological feature.

8. The device according to claim 5, wherein The multi-channel feature reverse reasoning and explainable emotion recognition model of the dynamic Bayesian network in the emotional state recognition module adds a multi-channel feature reverse reasoning mechanism to the dynamic Bayesian network.

9. A computer-readable storage medium, characterized in that It stores a computer program for emotion recognition for road and bridge engineering scenes, wherein the computer program enables a computer to execute the emotion recognition method for road and bridge engineering scenes as described in any one of claims 1-4.

10. An electronic device, characterized in that: include: one or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the emotion recognition method for road and bridge engineering scenarios as described in any one of claims 1-4.

Images