An office worker emotion recognition method and system based on visible light and voice signals

Abstract

The application discloses an office staff emotion recognition method and system based on visible light and voice signals, relates to the technical field of computer vision and voice signal processing, and the method is characterized in that: face images and voice signals are synchronously collected, first, it is judged whether a staff is in a silent state or a speaking state, if the staff is in the silent state, emotion is recognized by analyzing facial motion unit features of the whole face, if the staff is in the speaking state, the voice source is confirmed by lip reading verification, and then emotion is recognized by fusing facial motion features based on only eyebrow and eye regions and voice acoustic features. The application effectively solves the interference problem of mouth movement on expression recognition when speaking, and improves the accuracy and robustness of emotion state monitoring.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and speech signal processing technology, and in particular to a method and system for recognizing the emotions of office workers based on visible light and speech signals. Background Technology

[0002] In the fields of human factors engineering and industrial safety monitoring, real-time and objective quantitative assessment of the emotional state of personnel in special positions (such as drivers and operators) is a crucial link in preventing human-caused accidents and ensuring production safety. With the development of computer vision and signal processing technologies, using non-contact sensors to collect facial images and voice signals, and then identifying human emotions through deep learning algorithms, has become the mainstream technical approach. Existing technical solutions typically focus on improving the robustness of recognition through multimodal data fusion, attempting to combine visual modality micro-expression analysis with auditory modality acoustic features to achieve a comprehensive perception of human psychological states.

[0003] However, despite the progress made in existing multimodal emotion recognition technologies, significant technical bottlenecks remain in practical applications, particularly in handling feature interference from different states of "speaking" and "silence." On one hand, some existing technologies rely on a single data modality. For example, the patent application "A Method and Device for Academic Emotion Recognition Based on Facial Expressions and Contextual Situation" (202510426941X) only relies on image information to analyze facial expressions and environmental features, neglecting the crucial role of voice information in emotion expression, resulting in insufficient recognition accuracy in complex scenarios. On the other hand, while some technologies incorporate multimodal data, the acquisition methods are cumbersome. For instance, the "Emotion Recognition and Intervention System Based on the Fusion of Facial Micro-expressions and Physiological Signals" (CN121456675B) requires the collection of physiological signals, and the contact-wearing of sensors increases the burden on subjects, hindering its widespread application in routine monitoring in specific positions.

[0004] More critically, existing multimodal fusion methods often lack targeted processing for visual feature extraction during "speaking states." For example, while patent application CN120873762B, titled "An Asynchronous Multimodal Emotion Recognition Method, Apparatus, Device, and Medium" utilizes multidimensional data such as facial, hand, and speech data, the mouth area undergoes significant muscle movements (e.g., mouth opening and lip shape changes) during speech. These language-driven motor units (AUs) severely interfere with facial expression recognition, causing the system to misinterpret "speaking actions" as "emotional expressions." Existing technologies typically simply superimpose visual and auditory features, failing to effectively distinguish between "mouth movements caused by speaking" and "mouth expressions caused by emotions." Furthermore, they fail to dynamically adjust the visual feature extraction area based on the subject's vocalization state, thus introducing redundant or even erroneous interference information and reducing the accuracy and reliability of emotion recognition. Summary of the Invention

[0005] One of the objectives of this invention is to address the shortcomings of the aforementioned background technologies in handling speech interference, modal fusion strategies, and the convenience of non-contact monitoring by providing an emotion recognition method for office workers based on visible light and speech signals. This method aims to solve the problem of speech actions interfering with facial expression recognition and improve the robustness and accuracy of emotion monitoring for personnel in special positions during their on-duty hours.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: a method for recognizing the emotions of office workers based on visible light and voice signals, comprising the following steps:

[0007] S1. Simultaneously acquire facial image sequences and voice signals of the tested person in real time through deployed visible light cameras and microphones;

[0008] S2. Perform voice activity detection on the voice signal, extract the effective voice segments, and determine whether the person being tested is currently in a silent state or a speaking state based on the effective voice segments.

[0009] S3. If the state is determined to be silent, proceed to step S4; if the state is determined to be speaking, proceed to step S5.

[0010] S4. Based on the facial image sequence, extract facial motion unit features of the entire face region, identify the first emotional state based on the facial motion unit features of the entire face region, and output it.

[0011] S5. Processing the facial image sequence and speech signal in the speaking state, including:

[0012] S51. Perform lip reading recognition on the facial image sequence to generate a lip reading sequence, and compare the lip reading sequence with the speech signal to verify whether the speech signal originates from the person being tested.

[0013] S52. If the verification passes, then the following steps will be executed simultaneously:

[0014] S521. Based on the speech signal, extract its acoustic features, and identify the second emotional state according to the acoustic features;

[0015] S522. Based on the facial image sequence, extract facial motion unit features of the eyebrow and eye sub-region, and identify the third emotional state based on the facial motion unit features of the eyebrow and eye sub-region.

[0016] S53. Perform multimodal fusion on the second and third emotional states to obtain the final emotional state and output it.

[0017] Preferably, in step S2, the voice activity detection includes: calculating the short-time energy and zero-crossing rate of the voice signal; when the short-time energy exceeds a first threshold and the zero-crossing rate exceeds a second threshold, it is determined that the current signal contains valid voice information.

[0018] More preferably, for a discrete speech signal x(n), the short-time energy E of the m-th frame m The calculation method is as follows:

[0019] ;

[0020] Where x m (n) represents the nth sampling point in the mth frame, where N is the frame length;

[0021] The zero-crossing rate Z of the m-th frame m The calculation method is as follows:

[0022] ;

[0023] Where sgn() is the sign function, which is 1 when the input is greater than or equal to 0, and -1 otherwise;

[0024] Define a decision function D(m). If D(m) = 1, it indicates that the speech contains valid speech. The calculation method is as follows:

[0025] ;

[0026] Among them, Threshold E Threshold is the threshold for short-time energy. Z The threshold for the zero-crossing rate.

[0027] More preferably, in step S51, comparing the lip-reading sequence with the speech signal specifically involves:

[0028] Extract the image sequence of the lip region from the facial image sequence and analyze its geometric change characteristics;

[0029] Generate lip-reading sequences based on the aforementioned geometric transformation features;

[0030] Determine whether the content of the speech signal is synchronized with the lip-reading sequence in time and whether the content is consistent;

[0031] If so, the verification is successful, and the voice is determined to originate from the person being tested.

[0032] More preferably, in step S4 or S522, the extraction of the facial motion unit features includes:

[0033] Locate and crop out the facial region from each frame of the facial image sequence;

[0034] Extract facial key points from the facial region and perform facial alignment based on the facial key points;

[0035] Based on the aligned facial region, features of at least one facial motion unit are extracted from the facial or eyebrow sub-region.

[0036] More preferably, in step S521, obtaining the second emotional state based on speech signal recognition includes:

[0037] The speech signal is preprocessed, including pre-emphasis, framing, and windowing.

[0038] Extract prosodic features, spectral features, and phonological features from the preprocessed speech signal;

[0039] The extracted features are input into the emotion classification model to obtain the second emotion state.

[0040] More preferably, in step S522, the eyebrow and eye sub-region includes the eyebrow region and the eye region; extracting facial motion unit features of the eyebrow and eye sub-region specifically involves extracting motion unit features related to eyebrow movement and eye movement.

[0041] More preferably, in step S53, the multimodal fusion adopts a weighted average method; let the prediction confidence of the second emotional state be P. speech The prediction confidence level for the third emotional state is P. visual The assigned weight coefficients are w speech and w visual And satisfy w speech +w visual =1; then the overall confidence level P after fusion is 1; final for:

[0042] ;

[0043] Based on the comprehensive confidence level P final Determine the final emotional state.

[0044] More preferably, the method further includes step S6: dynamically modeling the output emotional state within a continuous time window, analyzing the trajectory of emotional characteristics changing over time, in order to assess the stability of the emotional state.

[0045] In addition, the present invention also provides an office worker emotion recognition system based on visible light and voice signals, which includes:

[0046] The data acquisition module is used to simultaneously acquire facial image sequences and voice signals of the test subjects in real time through deployed visible light cameras and microphones;

[0047] The state detection module is used to detect voice activity in the voice signal, extract valid voice segments, and determine whether the person being tested is currently in a silent state or a speaking state based on the valid voice segments.

[0048] The silent state processing module is invoked when the state detection module determines that the state is silent. It extracts facial motion unit features of the entire face region based on the facial image sequence and performs emotion recognition, and outputs the first emotion state.

[0049] The speaking status processing module, which is invoked when the status detection module determines that a speaking status has been reached, includes:

[0050] The verification unit is used to perform lip reading recognition on the facial image sequence to generate a lip reading sequence, and to verify whether the voice signal originates from the person being tested by comparing the lip reading sequence with the voice signal.

[0051] The speech analysis unit is used to extract acoustic features and perform emotion recognition based on the speech signal when the verification is passed, so as to obtain a second emotional state.

[0052] The visual analysis unit is used to extract facial motion unit features of the eyebrow and eye sub-regions and perform emotion recognition based on the facial image sequence when the verification is passed, so as to obtain a third emotional state.

[0053] The multimodal fusion module is used to fuse the second emotional state and the third emotional state to obtain and output the final emotional state.

[0054] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0055] 1. Effectively distinguishes and eliminates interference from speech movements in emotion recognition, significantly improving monitoring accuracy. Existing technologies, when processing speech, often misinterpret the large movements of the mouth area, dominated by pronunciation, as emotional expressions, leading to recognition errors. This invention fundamentally distinguishes between the two states by determining whether the subject is currently silent or speaking and executing different paths accordingly. Specifically, in the speaking state, by extracting facial motion unit features from the eyebrow and eye sub-regions, it actively avoids the mouth area, which moves violently due to speaking, extracting emotional features only from the less affected eyebrow and eye areas. This eliminates the noise of speech movements, ensuring the purity of visual emotional feature extraction and directly solving the key defects pointed out in the background technology.

[0056] 2. This invention achieves efficient and reliable fusion of multimodal information, enhancing the system's robustness. Instead of simply superimposing speech and visual signals, it uses a lip-reading verification process to ensure that the fused speech and visual information originates from the same subject (the test subject), preventing misjudgments caused by ambient speech. Based on this, it fuses verified homologous speech emotions (second emotional state) and eyebrow / eyelid visual emotions (third emotional state), allowing the two modalities to corroborate and complement each other. This fusion strategy, based on state judgment and source verification, is more targeted than simple superposition, effectively improving the stability and reliability of emotion recognition in complex office or industrial environments (where there may be conversations with others and environmental noise).

[0057] 3. The entire solution relies entirely on non-contact visible light cameras and microphone sensors, meeting the needs of routine, non-intrusive monitoring in specialized positions. The process defined by this invention integrates state detection, identity verification, feature extraction, and decision-making. The system can automatically switch to the optimal analysis path (full-face feature analysis or eyebrow and voice analysis) based on whether the subject is silent or speaking. This adaptability allows the method to complete continuous and accurate emotional state assessments in a natural state without human intervention or subject-specific behavioral changes, enhancing the method's practicality and convenience. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the method flow in the embodiment;

[0059] Figure 2 This is a schematic diagram showing the location of key facial feature points in the embodiment;

[0060] Figure 3 This is a schematic diagram illustrating how adjusting the face to eliminate interference from large head movements in micro-expression capture, as shown in the embodiment.

[0061] Figure 4 This is a schematic diagram illustrating how the face is divided into several motion units in an embodiment.

[0062] Figure 5 This is a schematic diagram of the lip area in the embodiment;

[0063] Figure 6 This is a schematic diagram of the detection process for determining whether a state is speaking or silent, as shown in the embodiment.

[0064] Figure 7 This is a schematic diagram illustrating different emotions in the eyebrow and eye sub-regions in the embodiment;

[0065] Figure 8 This is a schematic diagram illustrating the multimodal fusion of the second and third emotional states in an embodiment. Detailed Implementation

[0066] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.

[0067] Figure 1 This is a flowchart illustrating a method for recognizing office workers' emotions based on visible light and voice signals, according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S1 to S6.

[0068] S1. By deploying visible light cameras and microphones, the facial image sequence and voice signal of the person being tested are acquired synchronously and in real time.

[0069] In this step, a visible light camera is deployed directly in front of the pre-designed detection area (such as an office workstation or pre-job checkpoint). Its optical axis must be coaxially aligned with the normal to the subject's face to ensure the frontal and completeness of the captured facial image. Simultaneously, a high-sensitivity directional microphone is installed beside or integrated into the camera, with its pickup axis parallel or coincident with the camera's optical axis to achieve a high degree of consistency between the audio and video capture field of view. The system continuously acquires facial video streams of the subject through the camera and continuously acquires ambient sound signals through the microphone. A high-precision analog-to-digital converter transforms the analog audio stream into a discrete digital speech signal sequence, providing the raw data foundation for subsequent speech signal processing and analysis.

[0070] S2. Perform voice activity detection on the voice signal, extract the effective voice segments, and determine whether the person being tested is currently in a silent state or a speaking state based on the effective voice segments.

[0071] State detection aims to determine whether the subject is currently in a silent or speaking state, and different emotion detection strategies will be used for different states. Specifically, it includes the following sub-steps:

[0072] First, voice activity detection is performed on the digital speech signal captured by the microphone to determine whether the current sound signal contains speech information. Specific methods are not limited to calculating features such as short-time energy and zero-crossing rate of the audio signal. When the feature value exceeds a threshold, the current signal is determined to contain "valid speech information"; otherwise, it is considered background noise or silence. Short-time energy is the most intuitive physical quantity for determining the presence or absence of speech; therefore, this embodiment calculates the short-time energy and zero-crossing rate features of the speech signal. For a discrete speech signal x(n), the short-time energy E of the m-th frame... m It is usually defined as the sum of squares of the signal amplitudes within the frame, and the calculation method is as follows:

[0073] ;

[0074] Where x m (n) represents the nth sampling point of the mth frame, where N is the frame length (usually the number of sampling points corresponding to 20ms~30ms).

[0075] Zero-crossing rate reflects the frequency at which a signal crosses zero, and is often used to distinguish high-frequency noise or unclear sounds from low-frequency muddy sounds. The zero-crossing rate Z of the m-th frame... m The calculation is as follows:

[0076] ;

[0077] Where sgn() is the sign function, which is 1 when the input is greater than or equal to 0, and -1 otherwise;

[0078] Define a decision function D(m). A result of 1 indicates the presence of valid speech, while a result of 0 indicates silence or noise. The calculation method is as follows:

[0079] ;

[0080] Among them, Threshold E Threshold is the threshold for short-time energy. Z The threshold for the zero-crossing rate.

[0081] When D(m)=1, it is determined that the current time period contains valid voice information and the person being tested is in a potential speaking state; otherwise, it is determined to be in a silent state.

[0082] S3. If the state is determined to be silent, proceed to step S4; if the state is determined to be speaking, proceed to step S5.

[0083] S4. Based on the facial image sequence, extract the facial motion unit features of the entire face region, identify the first emotional state based on the facial motion unit features of the entire face region, and output it.

[0084] When the system determines that the subject is in a silent state, emotion recognition is based solely on the visual modality. A camera is used to capture facial video stream data of the subject, and the corresponding expression / micro-expression types are analyzed. The camera captures facial expression features, especially micro-expression changes. A deep learning model analyzes facial motor units (AUs) to identify emotional tendencies such as anxiety, depression, and fatigue. Collecting speech samples and extracting tone, speech rate, pause frequency, and acoustic features can also help determine the subject's emotional state. The specific process is as follows:

[0085] 1. Data Acquisition and Preprocessing: The facial video stream acquired by the camera is used to locate the face position in each frame, crop out irrelevant background, and retain only the facial area image.

[0086] 2. Feature Point Extraction and Facial Alignment: Key facial feature points (such as corners of the eyes, corners of the mouth, eyebrows, etc., typically 68 or 468 points) are located using pre-trained models (e.g., Dlib or MediaPipe). Figure 2 As shown, calculate the distance and angle changes between these points. For example, the distance the corners of the mouth turn up, the angle the eyebrows furrow, etc.

[0087] Meanwhile, since subjects may move their heads, algorithms (such as those based on the tip of the nose or key points around the eyes) are used to "straighten" the face to eliminate interference from large head movements on micro-expression capture. Figure 3 (As shown). Let (x, y) be the coordinates of a feature point in the original image, (x', y') be the coordinates of the aligned feature point, and T be the transformation matrix, then we have:

[0088] ;

[0089] Here, matrix T is the affine transformation matrix calculated using the original feature points and the standard reference point. The affine transformation matrix describes the change of a geometric object under linear transformations such as translation, rotation, scaling, and shearing. For facial alignment, common affine transformation matrices can be calculated using the following steps:

[0090] ① Calculate the centroid (or other geometric center) based on the feature points or feature regions in the original image;

[0091] ② Calculate the reference point corresponding to the target position or shape (e.g., the standard eye position or mouth position);

[0092] ③Calculate the affine transformation matrix T based on these corresponding points.

[0093] 3. Motion Feature Extraction: Based on the anatomical characteristics of the human face, it is divided into several independent yet interconnected motion units. The motion characteristics of these units, their main controlled areas, and related facial expressions are analyzed. Figure 4 Here are some schematic diagrams of the main AUs.

[0094] 4. Facial Expression / Micro-expression Classification: Deep learning models (such as CNNs) are used to perform fine-grained analysis of detected facial AU regions. Regression or classification algorithms quantify the movement intensity and activation state of specific muscle groups (e.g., the degree of upward movement of the corners of the mouth in AU12), thereby capturing minute textures and geometric changes that are difficult to detect with the naked eye. Subsequently, the system combines and infers these independent AU signals temporally according to the logical rules of the Facial Action Coding System (FACS). The model not only analyzes the characteristics of individual action units but also focuses on the synergistic effects between multiple AUs and their dynamic evolution over time. Ultimately, these combinations of muscle movements are mapped to specific emotion categories (such as anger, surprise, and repression), achieving a precise interpretation from "muscle twitching" to "psychological emotion," outputting the first emotional state.

[0095] S5. Process the facial image sequence and speech signal in the speaking state.

[0096] When the system determines that the subject is speaking, it must first verify the source of the speech before initiating multimodal sentiment analysis. This step includes steps S51 to S53.

[0097] S51. Perform lip reading recognition on the facial image sequence to generate a lip reading sequence, and compare the lip reading sequence with the speech signal to verify whether the speech signal originates from the person being tested.

[0098] To prevent interference from other people's voices in the environment, voice source verification is required:

[0099] 1. From the facial video stream obtained from S1, key points are located in the lip area, and the image sequence of this area is extracted.

[0100] 2. Analyze the geometric changes of the lips over time (such as mouth opening, lip width changes, and corner displacement of the mouth, etc.). Figure 5 (As shown). Using a lip-reading model, the dynamic changes in the lips are mapped to corresponding "phoneme sequences" or "visual text" to generate lip-reading sequences.

[0101] 3. Simultaneously, for the valid speech segments detected by S2, automatic speech recognition (ASR) is performed using acoustic and language models based on deep neural networks (such as the Conformer architecture), converting them into text sequences. During the decoding process, the next character y is predicted. t The probability distribution can be approximated as:

[0102] ;

[0103] Among them, P AMIt is the probability output by the acoustic model, P LM λ is the probability output by the language model, used to correct homophone errors and predict punctuation. λ is the fusion weight coefficient, used to balance the contributions of acoustic features and semantic information.

[0104] 4. Compare the lip-reading sequence with the speech recognition text sequence. If they are synchronized in time and the phoneme / semantic content is consistent, the speech is determined to be spoken by the subject, the verification passes, and S52 is executed. If speech is detected, but there is no lip movement, the movement is out of sync, or the content does not match, the verification fails. In this case, the system can ignore the speech segment, process it as a silent state (S4), or discard the data segment. The entire state monitoring process is as follows: Figure 6 As shown.

[0105] S52. If the verification passes, then execute the following steps S521 and S522 simultaneously.

[0106] S521. Based on the speech signal, extract its acoustic features, and identify the second emotional state according to the acoustic features.

[0107] 1. Speech signal preprocessing: The verified speech signal is preprocessed, including pre-emphasis, framing and windowing, to eliminate environmental noise and enhance high-frequency components.

[0108] 2. Speech Feature Extraction: Since different emotions trigger physiological changes in vocal cord vibration and vocal tract shape, three key features need to be extracted: ① Prosodic features, primarily capturing changes in fundamental frequency, energy, and speech rate. For example, when angry or excited, the fundamental frequency is usually high and fluctuates dramatically, with strong energy; while when sad or depressed, the fundamental frequency is low, the speech rate is slow, and there are more pauses. ② Spectral features, using techniques such as Mel-frequency cepstral coefficients (MFCC), simulate the human ear's perception of sound frequencies to capture the timbre and texture of the sound. This is one of the most robust features for distinguishing different emotional states. ③ Phonological features, including vibrato and shimmer, are used to measure the periodicity and stability of the sound, reflecting the tension of the speaker's vocal cords, and are often used to identify fear or extreme anxiety. These features collectively characterize the speech characteristics under different emotional states.

[0109] 3. Emotion Classification: After acquiring high-dimensional acoustic features, deep learning models are needed to learn the complex nonlinear relationships between these features and emotional labels. Convolutional Neural Networks (CNNs) are used to extract local spatial features from the spectrogram, and Long Short-Term Memory (LSTM) networks are used to capture the dynamic trends of sound changes over time, thus understanding the fluctuations in emotion. Alternatively, Transformers or pre-trained models can be used to directly process the raw waveform or spectrum, automatically learning global dependencies and deep semantic information in speech, and even pre-training on massive amounts of unlabeled data, thereby significantly improving the model's generalization ability on specific emotional datasets. The model learns the complex relationships between features and emotional labels, ultimately classifying speech into basic emotional categories such as happiness, anger, sadness, fear, surprise, or neutrality, outputting a secondary emotional state.

[0110] S522. Based on the facial image sequence, extract facial motion unit features of the eyebrow and eye sub-region, and identify the third emotional state based on the facial motion unit features of the eyebrow and eye sub-region.

[0111] To avoid interference from mouth movements during speech on facial expression recognition, this step only analyzes the eyebrow and eye area:

[0112] 1. Data initialization and alignment: Similar to step S4, perform face localization, cropping and alignment on the facial image in the speaking state.

[0113] 2. AU feature extraction of eyebrow and eye region: such as Figure 7 As shown, on the aligned facial images, motion features related to the eyebrow and eye sub-regions are specifically selected and extracted based on the facial motion unit (AU) model, such as eyebrow raising (AU1 / AU2), eyebrow lowering and converging (AU4), and eye widening (AU5). These features constitute a visual sequence representing emotions.

[0114] 3. Emotion Classification: The AU feature sequence of the eyebrow and eye area is input into the emotion classification model. The model judges the emotion based on the typical motion features related to the eyebrow and eye area shown in the table below, and outputs the third emotion state.

[0115] surprise Her eyebrows were raised high and arched, and there were horizontal lines on her forehead. Lift the upper eyelid and open the eyes wide. fear Eyebrows are raised and gathered together, forming a straight line. Lift the upper eyelid and pull the lower eyelid up. anger Eyebrows that are pressed down and clustered together (frown lines) Stare, eyelids taut, gaze focused. sad Upward slant at the inner corner of the eyebrow (droopy eyebrows) The eyes may be dull, and the lower eyelids may be relaxed. disgust Eyebrows are usually relaxed or slightly lowered Slightly squint (accompanied by wrinkling the nose) hapiness Eyebrows are usually relaxed or slightly lowered Squinting, crow's feet appear at the corners of the eyes

[0116] S53. Perform multimodal fusion on the second and third emotional states to obtain the final emotional state and output it (the entire process ultimately results in...). Figure 8 (As shown).

[0117] When using both speech signals and eyebrow / eye image information to analyze emotions, the results of single-modal analysis often have limitations. Therefore, combining the two for multimodal fusion is key to improving the accuracy and robustness of emotion recognition.

[0118] First, the speech analysis model and the eyebrow and eye image analysis model are run independently, each outputting a predicted probability distribution for the emotion category. Then, a second emotion state based on speech (confidence P) is obtained. speech ) and the third emotional state based on eyebrows and eyes (confidence level P) visual Afterwards, a weighted average method is used for decision-level fusion. Weighting coefficients w are assigned to the two modes. speech and w visual And satisfy w speech +w visual =1. The final overall confidence level P final for:

[0119] ;

[0120] Ultimately, the system selects P. final The emotion category with the highest probability value is taken as the final emotional state of the person being tested at the current moment and output.

[0121] S6. Results summary and continuous monitoring.

[0122] The two modes are switched by detecting voice signals. When no voice signal is detected, only the facial expression / micro-expression detection subsystem is invoked for analysis. When a voice signal is detected, it is determined whether the source of the voice signal is the subject. If it is the subject, the voice signal analysis subsystem is invoked simultaneously, working in conjunction with the micro-expression monitoring system based on eyebrow and brow sub-motor units to obtain the subject's emotional information. The system supports continuous monitoring of the same subject within a short time window, and assesses the stability of the results by analyzing the trajectory of emotional characteristics changing over time (dynamic emotion modeling), thus eliminating random interference factors.

[0123] In some optional embodiments, speech activity detection may also employ an end-to-end detection method based on deep neural networks. The facial keypoint localization model can be a 68-point model or a 468-point model. The deep learning model for emotion classification in the eyebrow / eye region or the entire face region can be one or more combinations of convolutional neural networks (CNN), recurrent neural networks (RNN), long short-term memory networks (LSTM), or Transformer architectures. The weighting coefficients w for multimodal fusion... speech and w visual The signal-to-noise ratio (SNR) of the current speech signal or the image quality can be dynamically adjusted. For example, in noisy environments, the weight of the speech modality can be appropriately reduced. speech .

[0124] To facilitate understanding by those skilled in the art of the improvements of this invention over the prior art, some of the accompanying drawings and descriptions have been simplified. The above embodiments are preferred implementations of this invention. In addition, this invention can be implemented in other ways. Any obvious substitutions without departing from the concept of this technical solution are within the protection scope of this invention.

Claims

1. A method for recognizing office workers' emotions based on visible light and speech signals, characterized in that, Includes the following steps: S1. Simultaneously acquire facial image sequences and voice signals of the tested person in real time through deployed visible light cameras and microphones; S2. Perform voice activity detection on the voice signal, extract the effective voice segments, and determine whether the person being tested is currently in a silent state or a speaking state based on the effective voice segments. S3. If the state is determined to be silent, proceed to step S4; if the state is determined to be speaking, proceed to step S5. S4. Based on the facial image sequence, extract facial motion unit features of the entire face region, identify the first emotional state based on the facial motion unit features of the entire face region, and output it. S5. Processing the facial image sequence and speech signal in the speaking state, including: S51. Perform lip reading recognition on the facial image sequence to generate a lip reading sequence, and compare the lip reading sequence with the speech signal to verify whether the speech signal originates from the person being tested. S52. If the verification passes, then the following steps will be executed simultaneously: S521. Based on the speech signal, extract its acoustic features. The second emotional state is then identified based on the acoustic features. S522. Based on the facial image sequence, extract facial motion unit features of the eyebrow and eye sub-region, and identify the third emotional state based on the facial motion unit features of the eyebrow and eye sub-region. S53. Perform multimodal fusion on the second and third emotional states to obtain the final emotional state and output it.

2. The method for office worker emotion recognition based on visible light and speech signals according to claim 1, characterized in that, In step S2, the voice activity detection includes: calculating the short-time energy and zero-crossing rate of the voice signal; when the short-time energy exceeds a first threshold and the zero-crossing rate exceeds a second threshold, it is determined that the current signal contains valid voice information.

3. The method for office worker emotion recognition based on visible light and speech signals according to claim 2, characterized in that: For a discrete speech signal x(n), the short-time energy E of the m-th frame is... m The calculation method is as follows: ； Where x m (n) represents the nth sampling point in the mth frame, where N is the frame length; The zero-crossing rate Z of the m-th frame m The calculation method is as follows: ； Where sgn() is the sign function, which is 1 when the input is greater than or equal to 0, and -1 otherwise; Define a decision function D(m). If D(m) = 1, it indicates that the speech contains valid speech. The calculation method is as follows: ； Among them, Threshold E Threshold is the threshold for short-time energy. Z The threshold for the zero-crossing rate.

4. The method for office worker emotion recognition based on visible light and speech signals according to claim 1, characterized in that, In step S51, comparing the lip-reading sequence with the speech signal specifically involves: Extract the image sequence of the lip region from the facial image sequence and analyze its geometric change characteristics; Generate lip-reading sequences based on the aforementioned geometric transformation features; Determine whether the content of the speech signal is synchronized with the lip-reading sequence in time and whether the content is consistent; If so, the verification is successful, and the voice is determined to originate from the person being tested.

5. The method for office worker emotion recognition based on visible light and speech signals according to claim 1, characterized in that, In step S4 or S522, the extraction of facial motion unit features includes: Locate and crop out the facial region from each frame of the facial image sequence; Extract facial key points from the facial region and perform facial alignment based on the facial key points; Based on the aligned facial region, features of at least one facial motion unit are extracted from the facial or eyebrow sub-region.

6. The method for office worker emotion recognition based on visible light and speech signals according to claim 1, characterized in that, In step S521, obtaining the second emotional state based on speech signal recognition includes: The speech signal is preprocessed, including pre-emphasis, framing, and windowing. Extract prosodic features, spectral features, and phonological features from the preprocessed speech signal; The extracted features are input into the emotion classification model to obtain the second emotion state.

7. The method for office worker emotion recognition based on visible light and speech signals according to claim 1, characterized in that: In step S522, the eyebrow and eye sub-region includes the eyebrow region and the eye region; extract the facial motion unit features of the eyebrow and eye sub-region, specifically extracting motion unit features related to eyebrow movement and eye movement.

8. The method for office worker emotion recognition based on visible light and speech signals according to claim 1, characterized in that: In step S53, the multimodal fusion employs a weighted average method; let P be the prediction confidence level of the second emotional state. speech The prediction confidence level for the third emotional state is P. visual The assigned weight coefficients are w speech and w visual And satisfy w speech +w visual =1; then the overall confidence level P after fusion is 1; final for: ； Based on the comprehensive confidence level P final Determine the final emotional state.

9. The method for office worker emotion recognition based on visible light and speech signals according to any one of claims 1-8, characterized in that, It also includes step S6: Within a continuous time window, the output emotional state is dynamically modeled, and the trajectory of emotional characteristics changing over time is analyzed to assess the stability of the emotional state.

10. An office worker emotion recognition system based on visible light and voice signals, characterized in that, The system for implementing the method of any one of claims 1-9, the system comprising: The data acquisition module is used to simultaneously acquire facial image sequences and voice signals of the test subjects in real time through deployed visible light cameras and microphones; The state detection module is used to detect voice activity in the voice signal, extract valid voice segments, and determine whether the person being tested is currently in a silent state or a speaking state based on the valid voice segments. The silent state processing module is invoked when the state detection module determines that the state is silent. It extracts facial motion unit features of the entire face region based on the facial image sequence and performs emotion recognition, and outputs the first emotion state. The speaking status processing module, which is invoked when the status detection module determines that a speaking status has been reached, includes: The verification unit is used to perform lip reading recognition on the facial image sequence to generate a lip reading sequence, and to verify whether the voice signal originates from the person being tested by comparing the lip reading sequence with the voice signal. The speech analysis unit is used to extract acoustic features and perform emotion recognition based on the speech signal when the verification is passed, so as to obtain a second emotional state. The visual analysis unit is used to extract facial motion unit features of the eyebrow and eye sub-regions and perform emotion recognition based on the facial image sequence when the verification is passed, so as to obtain a third emotional state. The multimodal fusion module is used to fuse the second emotional state and the third emotional state to obtain and output the final emotional state.

Images