Pilot emotion recognition method

Abstract

The present application relates to the technical field of emotion recognition, in particular to a pilot emotion recognition method. A target user's corresponding target face image sequence and target physiological signal data are obtained. The target face image sequence is recognized to determine the corresponding basic face confidence of the target face image sequence. The target physiological signal data is recognized to determine the corresponding basic physiological signal confidence of the target physiological signal data. The target face image sequence, target physiological signal data, basic face confidence and basic physiological signal confidence are input into a preset emotion recognition model to output the target emotion corresponding to the target user. Efficient integration of multi-modal information is realized, and the target emotion is finally accurately output, providing a reliable basis for flight safety monitoring (such as recognizing the pilot's anxiety, fear and other negative emotions).

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of emotion recognition, in particular to a pilot emotion recognition method. BACKGROUND

[0002] In modern aviation tasks, the emotional state of the pilot is a key human factor affecting flight safety and decision-making efficiency. Negative emotions such as excessive load, anxiety or fear can lead to operational errors, and traditional emotion assessment methods rely on subjective reports, lacking objectivity and real-time performance. Therefore, developing an intelligent model that can accurately and unobtrusively recognize the emotional state of the pilot in real time has great theoretical and application value.

[0003] Facial expressions and physiological signals (such as electrocardiogram, ECG) are two important objective manifestations of emotional state. Facial expressions are rich in external visual cues of emotions, while ECG signals can reflect changes in autonomic nervous system activity caused by emotional fluctuations.

[0004] In the prior art, single modal recognition methods are susceptible to noise interference and have limited information dimensions. Although multi-modal fusion methods have been proposed, how to effectively process high-frequency time-series image data and long sequence physiological signals and achieve deep fusion of the two remains a technical problem to be solved in the field.

[0005] Therefore, there is an urgent need to develop a pilot emotion recognition method that fuses high-frequency time-series image data and long sequence physiological signals to improve the accuracy of emotion recognition. SUMMARY

[0006] The present application provides a pilot emotion recognition method to fuse high-frequency time-series image data and long sequence physiological signals to improve the accuracy of emotion recognition.

[0007] In a first aspect, the present application provides a pilot emotion recognition method, which comprises: acquiring a target facial image sequence and target physiological signal data corresponding to a target user; recognizing the target facial image sequence to determine a basic facial confidence corresponding to the target facial image sequence; recognizing the target physiological signal data to determine a basic physiological signal confidence corresponding to the target physiological signal data; inputting the target facial image sequence, the target physiological signal data, the basic facial confidence and the basic physiological signal confidence into a preset emotion recognition model to output a target emotion corresponding to the target user.

[0008] In an optional implementation, the target face image sequence, the target physiological signal data, the base face confidence, and the base physiological signal confidence are input into a preset emotion recognition model, and a target emotion corresponding to the target user is output, including: inputting the target face image sequence and the base face confidence into a first branch in the preset emotion recognition model, outputting a target face feature and a target face confidence; inputting the target physiological signal data and the base physiological signal confidence into a second branch in the preset emotion recognition model, outputting a target physiological signal feature and a target physiological signal confidence; and outputting the target emotion corresponding to the target user based on the target face feature, the target face confidence, the target physiological signal feature, and the target physiological signal confidence.

[0009] In an optional implementation, the target face image sequence and the base face confidence are input into the first branch in the preset emotion recognition model, and the target face feature and the target face confidence are output, including: performing feature extraction on the target face image sequence based on a front preset layer network in a first sub-feature extraction network in the first branch, outputting a local face feature; calibrating the local face feature, outputting a calibrated face feature; performing feature extraction on the calibrated face feature based on a rear preset layer network after the front preset layer network in the first sub-feature extraction network, outputting a current face feature; determining the target face confidence based on the current face feature; and performing feature extraction on the calibrated face feature based on a second sub-feature extraction network in the first branch from two time dimensions of a forward direction and a reverse direction, generating the target face feature.

[0010] In an optional implementation, the local face feature includes emotion-sensitive local regions corresponding to a plurality of parts of a face; the calibrated face feature is output by calibrating the local face feature, including: obtaining a local physiological signal waveform feature output by performing feature extraction on the target physiological signal data by a one-dimensional convolution layer in the second branch; screening a key dimension directly related to an emotion from the local physiological signal waveform feature as a guide signal; calculating a first similarity between each emotion-sensitive local region and the guide signal for each emotion-sensitive local region; comparing the first similarity between each emotion-sensitive local region and the guide signal with a first preset similarity threshold; and calibrating each emotion-sensitive local region according to a comparison result, and outputting the calibrated face feature.

[0011] In an optional implementation, the target face confidence is determined based on the current face feature, including: calculating the Euclidean distance between the current face feature and each type of emotion feature center; if each calculated Euclidean distance is greater than a first preset distance threshold or less than a second preset distance threshold, the base face confidence is reduced by a first preset value to obtain a candidate face confidence; the first preset distance threshold is greater than the second preset distance threshold; if only one Euclidean distance is less than the second preset distance threshold, the base face confidence is added by a second preset value to obtain the candidate face confidence; obtaining the acquisition scene parameters corresponding to the target face image sequence; adjusting the candidate face confidence according to the acquisition scene parameters to obtain the target face confidence.

[0012] In an optional implementation, the target face feature is generated by performing feature extraction on the calibration face feature from the two time dimensions of the forward direction and the reverse direction based on the second sub-feature extraction network in the first branch, including: stacking the calibration face feature in time sequence to generate a calibration face feature sequence; performing feature processing on the calibration face feature sequence according to the input requirements of the second sub-feature extraction network to generate a face time sequence feature; the second sub-feature extraction network performs forward feature extraction on the face time sequence feature in the forward time sequence, and outputs a forward hidden state at each time step; the second sub-feature extraction network performs reverse feature extraction on the face time sequence feature in the reverse time sequence, and outputs a reverse hidden state at each time step; selecting a target forward hidden state and a target reverse hidden state corresponding to the last time step;

[0013] The target forward hidden state and the target reverse hidden state are spliced to generate the target face feature.

[0014] In an optional implementation, the target physiological signal data and the base physiological signal confidence are input into the second branch in the preset emotion recognition model to output the target physiological signal feature and the target physiological signal confidence, including: performing feature extraction on the target physiological signal data based on the one-dimensional convolution layer in the second branch to output a local physiological signal waveform feature; calibrating the local physiological signal waveform feature to obtain a calibrated physiological signal feature; extracting an emotion-sensitive feature from the calibrated physiological signal feature; comparing the emotion-sensitive feature with an emotion arousal threshold; adjusting the base physiological signal confidence according to the comparison result to obtain the target physiological signal confidence; performing feature extraction on the calibrated physiological signal feature to generate the target physiological signal feature.

[0015] In an optional implementation, the local physiological signal waveform feature is calibrated to obtain a calibrated physiological signal feature, including: obtaining a local facial feature output by a first preset layer network in a first sub-feature extraction network for feature extraction on a target facial image sequence; the first sub-feature extraction network belongs to a first branch; the local facial feature includes emotion-sensitive local regions corresponding to multiple parts of a face; a mouth opening and closing related feature is selected from the local facial feature; the local physiological signal waveform feature is identified, and the local physiological signal waveform feature is divided into at least one target segment according to an identification result; the target segment is a core segment or an interference segment; a second similarity between the mouth opening and closing related feature and each target segment is calculated;

[0016] The second similarity between the mouth opening and closing related feature and each target segment is compared with a second preset similarity threshold; according to the comparison result, each target segment is calibrated, and a calibrated physiological signal feature is output.

[0017] In an optional implementation, the calibrated physiological signal feature is feature extracted to generate a target physiological signal feature, including: calculating a signal value variance for the calibrated physiological signal feature; determining a current emotional fluctuation stage according to the signal value variance; the current emotional fluctuation stage includes a stable period, a fluctuation period, and a severe period; determining a target sampling factor value according to the current emotional fluctuation stage; calculating a target value based on the target sampling factor value; adding position encoding to the calibrated physiological signal feature to obtain a position encoded physiological signal feature; inputting the target value and the position encoded physiological signal feature into a first layer encoder in a multi-layer encoder; the first layer encoder splits the position encoded physiological signal feature into query features, key features, and value features; for each query feature, the most significant target key feature is selected from all key features, and each target key feature and the query feature are calculated to output attention features; performing nonlinear transformation on the attention features to output first encoding features; performing down-sampling processing on the first encoding features to generate first down-sampling features; inputting the first down-sampling features into a second layer encoder in the multi-layer encoder to output second encoding features; performing down-sampling processing on the second encoding features to generate second down-sampling features; repeating the above steps until a top layer encoder in the multi-layer encoder outputs top layer encoding features; performing global average pooling operation on the top layer encoding features to generate a target physiological signal feature.

[0018] In an optional implementation, based on the target facial feature, the target facial confidence, the target physiological signal feature, and the target physiological signal confidence, a target emotion corresponding to a target user is output, including:

[0019] a third similarity between the target facial feature and the target physiological signal feature is calculated; if the third similarity is greater than or equal to a third similarity threshold, the target facial feature and the target physiological signal feature are fused by weighting based on the target facial confidence and the target physiological signal confidence to generate a target fusion feature; and the target emotion corresponding to the target user is output based on the target fusion feature; or if the third similarity is less than the third similarity threshold, the target facial confidence and the target physiological signal confidence are compared, and a maximum target confidence is determined from the two; and the target emotion corresponding to the target user is output based on the target feature corresponding to the maximum target confidence.

[0020] The pilot emotion recognition method provided by the embodiments of the present application acquires a target facial image sequence and target physiological signal data, ensures that the facial expression and the physiological signal are strictly aligned in the time dimension, provides a high-quality and time-consistent data source for subsequent multi-modal fusion, and avoids emotion recognition deviation caused by asynchronous data. The basic facial confidence is determined, low-quality facial data can be quickly screened out, a basis is provided for subsequent model weight allocation, and the interference of invalid data on the emotion recognition result is reduced. The basic physiological signal confidence is determined, which can effectively identify high-noise and incomplete waveform ECG data, avoid such data misleading emotion judgment, and improve the accuracy of physiological signal feature extraction. The dual-modal data and the corresponding confidence are jointly input into the model, the model can dynamically adjust the dual-modal feature weight combined with the confidence, realize efficient integration of multi-modal information, and finally accurately output the target emotion, providing a reliable basis for flight safety monitoring (such as identifying negative emotions such as anxiety and fear of pilots). BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0022] Figure 1 FIG. 1 is a first flowchart of a pilot emotion recognition method according to an embodiment of the present application;

[0023] Figure 2 FIG. 2 is a second flowchart of a pilot emotion recognition method according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0025] In the present embodiment, a pilot emotion recognition method is provided, which can be used for electronic devices in target aircraft or target planes, Figure 1 is a flowchart of the pilot emotion recognition method according to the embodiments of the present application, as shown in Figure 1 , which includes the following steps:

[0026] In step S101, a target user corresponding target face image sequence and target physiological signal data are acquired.

[0027] Specifically, the electronic device can continuously shoot the original pilot face video stream at a sampling frequency of 120 Hz based on the high-speed camera fixedly installed in the cockpit. The camera position needs to face the pilot's face directly to ensure no obstruction, and the lens resolution needs to support clear capture of facial details. 120 frames of face images are generated per second, directly reflecting the dynamic changes of the pilot's emotions (such as the continuous process of eye instantaneously widening when surprised, and eyebrow arch lifting when angry). Then, the electronic device can use the MTCNN algorithm to detect faces in each frame of the original video image, and crop the image containing only the face region according to the detection bounding box, and normalize the size to 224x224 pixels.

[0028] Then, random rotation (±10°), horizontal flip, brightness and contrast adjustment (±15%) are performed on the cropped image to improve the model generalization ability, the image pixel value is linearly scaled from the [0, 255] interval to the [0, 1] interval, and finally the continuous face image frames after preprocessing are constructed into a 240-frame feature sequence according to a 2-second time window to obtain the target face image sequence.

[0029] The electronic device can collect the original physiological signal through the wireless wearable physiological sensor worn by the pilot at a sampling frequency of 120 Hz. The physiological sensor uses a medical-grade electrode sheet to ensure good contact with the skin and reduce motion artifacts. The original physiological signal can be an ECG (Electrocardiogram) signal. The ECG signal reflects the electrical activity of the heart, and emotional fluctuations will cause changes in the autonomic nervous system, which in turn leads to changes in ECG waveform characteristics (such as RR interval, QRS wave amplitude) (such as shortening of RR interval when anxious, and increase of QRS wave amplitude when fearful).

[0030] The electronic device can employ a Butterworth band-pass filter (passband frequency 0.5-40 Hz) to remove power frequency interference, baseline drift and electromyographic noise in the original physiological signal. Then, the processed original physiological signal is segmented according to a 2-second time window, and Z-score standardization (mean 0, standard deviation 1) is performed on the signal in each segment to generate target physiological signal data.

[0031] Step S102, the target face image sequence is recognized to determine the basic face confidence corresponding to the target face image sequence.

[0032] Specifically, the electronic device can calculate the area ratio of the occluded face in the target face image sequence, and determine the completeness of the target face image sequence according to the area ratio of the occluded face.

[0033] For example, the area ratio of the occluded face ≤30% (such as only a few hairs slightly occluded) is high completeness (1.0 points); the area ratio of the occluded face >30% and ≤50% (such as part of the cheek is occluded by the helmet strap) is medium (0.7 points); the area ratio of the occluded face >50% (such as the mouth is occluded by the mask) is low (0.4 points).

[0034] In addition, the electronic device can calculate the image pixel contrast (gray value difference) corresponding to the target face image sequence, and then determine the sharpness of the target face image sequence according to the calculated image pixel contrast. For example, the image pixel contrast ≥0.2 (such as a clear face under normal light) is high definition (1.0 points); the image pixel contrast <0.2 and ≥0.1 (such as a blurred face under weak light) is medium (0.6 points); the image pixel contrast <0.1 (such as an overexposed face caused by direct sunlight) is low (0.3 points).

[0035] The electronic device can determine the basic face confidence corresponding to the target face image sequence according to the completeness and sharpness of the target face image sequence.

[0036] For example, the basic face confidence = completeness score × 0.5 + sharpness score × 0.5 (both dimensions are equally important, each accounting for 50% weight).

[0037] Step S103, the target physiological signal data is recognized to determine the basic physiological signal confidence corresponding to the target physiological signal data.

[0038] Specifically, the electronic device can identify the target physiological signal data, and determine a signal-to-noise ratio. For example, a signal-to-noise ratio ≥ 15 dB (low noise, clear effective waveform) is 1.0 points; 10 dB ≤ signal-to-noise ratio < 15 dB (moderate noise, part of the waveform is disturbed) is 0.7 points; and a signal-to-noise ratio < 10 dB (high noise, effective waveform is blurred) is 0.4 points.

[0039] Then, the electronic device can identify the target physiological signal data, and count the detection rate of QRS complex (the core waveform of ventricular contraction, which changes most significantly under emotional stress). For example, a detection rate ≥ 95% (almost all QRS waves can be identified) is 1.0 points; 90% ≤ detection rate < 95% (a small amount of QRS waves are missed) is 0.7 points; and a detection rate < 90% (a large number of QRS waves cannot be identified) is 0.4 points.

[0040] Finally, the electronic device can determine the basic physiological signal confidence corresponding to the target physiological signal data according to the signal-to-noise ratio and the detection rate of QRS complex corresponding to the target physiological signal data.

[0041] For example, the basic physiological signal confidence = SNR score × 0.5 + waveform integrity score × 0.5 (noise level and waveform availability are equally important).

[0042] Step S104, input the target face image sequence, the target physiological signal data, the basic face confidence and the basic physiological signal confidence into a preset emotion recognition model, and output a target emotion corresponding to the target user.

[0043] Specifically, the electronic device can input the target face image sequence, the target physiological signal data, the basic face confidence and the basic physiological signal confidence into the preset emotion recognition model, and then perform feature fusion on the target face image sequence and the target physiological signal data based on the basic face confidence and the basic physiological signal confidence, and output a target emotion corresponding to the target user.

[0044] This step will be described in detail below.

[0045] The pilot emotion recognition method provided in the embodiments of the present application acquires a target face image sequence and target physiological signal data, ensures that the facial expression and the physiological signal are strictly aligned in the time dimension, provides a high-quality and time-consistent data source for subsequent multi-modal fusion, and avoids emotion recognition deviation caused by asynchronous data. The basic face confidence is determined, low-quality face data can be quickly screened out, a basis is provided for subsequent model weight distribution, and the interference of invalid data on the emotion recognition result is reduced. The basic physiological signal confidence is determined, which can effectively identify high-noise and incomplete waveform ECG data, avoid misleading emotion judgment of such data, and improve the accuracy of physiological signal feature extraction. The dual-modal data and the corresponding confidence are jointly input into the model, the model can dynamically adjust the dual-modal feature weight combined with the confidence, realize efficient integration of multi-modal information, and finally accurately output the target emotion, providing a reliable basis for flight safety monitoring (such as identifying negative emotions such as anxiety and fear of pilots).

[0046] A pilot emotion recognition method is provided in the embodiments, Figure 2 The flowchart of the pilot emotion recognition method according to the embodiments of the present application is shown in Figure 2 The flowchart includes the following steps:

[0047] In step S201, a target face image sequence corresponding to a target user and target physiological signal data are acquired.

[0048] For this step, please refer to the introduction of step S101 above, which will not be repeated here.

[0049] In step S202, the target face image sequence is recognized to determine the basic face confidence corresponding to the target face image sequence.

[0050] For this step, please refer to the introduction of step S102 above, which will not be repeated here.

[0051] In step S203, the target physiological signal data is recognized to determine the basic physiological signal confidence corresponding to the target physiological signal data.

[0052] For this step, please refer to the introduction of step S103 above, which will not be repeated here.

[0053] In step S204, the target face image sequence, the target physiological signal data, the basic face confidence and the basic physiological signal confidence are input into a preset emotion recognition model, and a target emotion corresponding to the target user is output.

[0054] Specifically, the above step S204 can include the following steps:

[0055] Step S2041, input the target face image sequence and the base face confidence into a first branch in a preset emotion recognition model, and output target face features and a target face confidence.

[0056] Specifically, the step S2041 can include the following steps.

[0057] Step a1, performing feature extraction on the target face image sequence based on a front preset layer network in a first sub-feature extraction network in the first branch, and outputting local face features.

[0058] The first sub-feature extraction network can be an EfficientNet-B0 network. The EfficientNet-B0 network is obtained by training based on preset face-labeled training data. The front preset layer network can be a front 5-layer network.

[0059] Specifically, the electronic device can output the 5th layer features from the adapted EfficientNet-B0 convolution base for each frame of the target face image sequence (224x224x3 pixels, having completed face cropping and normalization) according to the following hierarchy. The first to fourth layer shallow convolution (3x3) performs down-sampling (2x2) processing on each frame of the target face image, reduces the dimension of each frame of the target face image from 224x224 to 32x32, outputs a 32x32 feature map, and extracts low-level features such as face edges, skin color, and basic contours, thereby laying a foundation for middle-level emotion feature extraction. The 5th layer network includes 5 groups of MBConv (Mobile Inverted Bottleneck) modules and 19 convolution kernels, extracts features from the feature map output by the 4th layer network, and outputs a feature map with a feature dimension of 28x28x192, thereby completing the conversion of middle-level emotion semantic features and outputting emotion-sensitive features of local regions such as eyes, corners of the mouth, and eyebrow arches. The MBConv module in the 5th layer network uses "dilated convolution + depthwise separable convolution" to expand the receptive field (covering about 11x11 pixel regions) while reducing the amount of calculation, so that the features of emotion-related local regions such as "eyes + eyebrow arches" or "corners of the mouth + cheeks" can be completely captured, thereby avoiding the fragmentation of local information caused by a too small receptive field.

[0060] Then, the electronic device stacks the 5th layer output (28x28x192 for each frame) of each frame of the target face image (such as 240 frames of images) in the "frame order" to form a three-dimensional feature sequence, and finally outputs local face features with a dimension of "240x28x28x192" (time stepxheightxwidthxchannel number).

[0061] Eye region: [5:15, 8:18] pixel range of the feature map (covering both eyes and lower eyebrow, 10x10 pixels in total); Mouth region: [18:25, 10:20] pixel range of the feature map (covering both upper and lower lips and both cheeks, 7x10 pixels in total); Eyebrow region: [2:8, 6:22] pixel range of the feature map (covering both eyebrows and lower forehead, 6x16 pixels in total); Irrelevant region: pixels in the feature map other than the above three regions (such as the region where the edge of the face, moles, and scars may exist).

[0062] Step a2, calibrate the local facial features, and output the calibrated facial features.

[0063] Specifically, the local facial features include emotion-sensitive local regions corresponding to multiple parts of the face. The above step a2 can include the following steps:

[0064] Step a21, obtain the local physiological signal waveform features output by the one-dimensional convolution layer in the second branch for feature extraction on the target physiological signal data.

[0065] Specifically, the electronic device can extract the local physiological signal waveform features output by the one-dimensional convolution layer in the second branch for feature extraction on the target physiological signal data.

[0066] The process of generating the local physiological signal waveform features will be described in detail below.

[0067] Step a22, select the key dimensions directly related to emotions from the local physiological signal waveform features as the guide signals.

[0068] Specifically, the electronic device can retain the dimensions directly related to emotions based on the “physiological signal feature-emotion” correlation (such as Pearson correlation coefficient) calculated during the preset emotion recognition model training phase:

[0069] Key dimension examples: QRS wave amplitude (20%-30% higher than in a calm state when angry), P-R interval (5%-10% shorter than in a calm state when afraid). Screening threshold: retain 20 channel features with a correlation of >0.5 (exclude irrelevant channels with low correlation, such as low-frequency noise channels with no emotional correlation).

[0070] Finally, the guide signals with a dimension of 240x20 (time step x number of key channels) are formed, ensuring that the guide signals aggregate emotional information and avoiding redundant dimensions from interfering with subsequent similarity calculations.

[0071] Step a23, for each emotion-sensitive local region, calculate the first similarity between the emotion-sensitive local region and the guide signals.

[0072] Specifically, the electronic device respectively performs "global average pooling" on each emotion-sensitive local area (such as the eye area, the corner of the mouth area, and the eyebrow arch area) in each of the 240 frames, compresses the 28x28x192 features in the emotion-sensitive local area into a one-dimensional vector of 1x192 (eliminates the spatial dimension, facilitates similarity calculation with the ECG guide signal), and forms a feature sequence of three types of areas (each 240x192).

[0073] The guide signal has a current dimension of 1x20 (single time step), which is mapped to 1x192 through a lightweight fully connected layer (input dimension 20, output dimension 192), which is consistent with the feature vector dimension of each emotion-sensitive local area.

[0074] Next, the electronic device calculates the first similarity between the guide feature vector (Et) and the emotion-sensitive local area feature vector (Ft) in "time steps" (each frame corresponds to a time step), and the formula is as follows: .

[0075] According to the calculated first similarity, a correlation matrix with a dimension of 240x3 (time step x emotion-sensitive area number) is generated, and each element represents the emotion matching degree of the corresponding frame and the corresponding area (such as the eyebrow arch area similarity of the 100th frame is 0.75).

[0076] Step a24, compare the first similarity between each emotion-sensitive local area and the guide signal with the first preset similarity threshold.

[0077] Specifically, the electronic device can compare the first similarity between each emotion-sensitive local area and the guide signal with the first preset similarity threshold. For example, the high correlation threshold is sim≥0.6, which determines that the emotion-sensitive local area is highly matched with the emotion reflected by the ECG guide signal, and feature enhancement is required. The low correlation threshold is sim≤0.2, which determines that the region has low ECG emotion matching degree, or is an irrelevant region, and feature suppression is required. The medium correlation is 0.2

[0078] Step a25, according to the comparison result, calibrate each emotion-sensitive local area, and output the calibrated face feature.

[0079] Specifically, for different emotion types, the weight of the emotion-sensitive local area feature with sim≥0.6 is dynamically improved in combination with the correlation matrix, and the emotion key morphology is strengthened.

[0080] For the angry emotional scene, the guided signal feature shows an increase in QRS wave amplitude (consistent with the physiological characteristics of anger), and the eyebrow arch region has a correlation degree sim=0.75 and the mouth corner region has a correlation degree sim=0.72 (both ≥0.6).

[0081] For the eyebrow arch region, the original weight 1.0 is increased to 1.2-1.3 (an increase of 20%-30%), strengthening the morphological feature of "eyebrow arch lifting" (a typical representation of an angry emotion). For the mouth corner region, the original weight 1.0 is increased to 1.2-1.3 (an increase of 20%-30%), strengthening the morphological feature of "mouth corner pulling down".

[0082] The correlation degree is dynamically calculated by "correlation degree x improvement coefficient" (such as improvement coefficient = 1.2 + 0.1 x (sim-0.6)), the higher the correlation degree, the greater the improvement (such as sim=0.8, improvement coefficient = 1.4, weight improvement 40%), avoiding over-strengthening or insufficient strengthening caused by fixed weight.

[0083] For the fear emotional scene, the guided signal feature shows a shortening of the P-R interval (consistent with the physiological characteristics of fear), and the eye region has a correlation degree sim=0.8 and the forehead region (eyebrow arch upper extension) has a correlation degree sim=0.78 (both ≥0.6).

[0084] For the eye region, the original weight 1.0 is increased to 1.25-1.35 (an increase of 25%-35%), strengthening the morphological feature of "pupil dilation". For the forehead region, the original weight 1.0 is increased to 1.25-1.35 (an increase of 25%-35%), strengthening the morphological feature of "forehead wrinkle deepening".

[0085] For the irrelevant region, the feature map region corresponding to facial nevus, scar, and hair obstruction (such as feature map [12:14, 22:24] is the position of nevus) is determined, and regardless of the correlation degree, the feature weight is down-regulated from 1.0 to 0.4-0.5 (down-regulated by 50%-60%).

[0086] Low correlation region, for sim≤0.2 emotional sensitive region (such as sim=0.15 for the eyebrow arch region in a calm emotional state), the feature weight is down-regulated from 1.0 to 0.4-0.5, weakening the interference of features with no emotional correlation.

[0087] Example: In a certain frame, the face edge is obstructed by hair (corresponding to feature map [25:28, 2:5]), the weight of this region is reduced from 1.0 to 0.4, avoiding the misjudgment of hair texture as "forehead wrinkle" and other emotional features.

[0088] Finally, the electronic device recombines the "enhanced emotional region features" and the "inhibited irrelevant region features" of each frame in the 240 frames according to the original spatial positions to form a single-frame calibrated facial feature (the dimension is still 28x28x192). The 240 single-frame calibrated facial features are stacked in sequence, and a calibrated facial feature sequence with a dimension of 240x28x28x192 is output.

[0089] In step a3, the calibrated facial feature is subjected to feature extraction based on a preset back layer network after a preset front layer network in the first sub-feature extraction network, and a current facial feature is output.

[0090] Specifically, the first sub-feature extraction network can be an EfficientNet-B0 network. The preset front layer network can be the first 5 layers, and the preset back layer network after the first 5 layers can be the 6th and 7th layers in the EfficientNet-B0 network, wherein the 6th and 7th layers belong to deep convolution modules, and are designed to realize "abstract semantic extraction + feature dimension promotion" through "dilated convolution + deep separable convolution + residual connection", and complement the "mid-layer local feature extraction" function of the 5th layer.

[0091] Specifically, the 6th layer in the EfficientNet-B0 network is composed of 6 groups of MBConv (Mobile Inverted Bottleneck) modules, each group of MBConv includes four steps of "1x1 convolution dimension lifting -> 3x3 deep separable convolution -> 1x1 convolution dimension reduction -> residual connection", and the specific parameter adaptation is as follows:

[0092] 1x1 dimension lifting convolution: lifting the input 192-channel feature to 672 channels (through a dimension expansion ratio of 192x3.5, which is the standard configuration of the EfficientNet series), the purpose is to increase the feature expression dimension, and provide more abundant local semantic combination space for the subsequent deep separable convolution.

[0093] 3x3 deep separable convolution: dilated convolution with an expansion rate of 2 (expanding the receptive field to 15x15 pixels) can cover the combined features of the emotion-related regions such as "eyebrows + eyebrow arch" and "mouth corner + cheek" in the calibrated feature map (such as the cooperative features of "pupil dilation + forehead wrinkle deepening" when frightened), and at the same time, the deep separable convolution reduces the calculation amount (about 80% lower than the traditional convolution).

[0094] 1x1 dimension reduction convolution: compressing the 672-channel feature back to 192 channels to keep the channel number consistent with the input, which is convenient for subsequent residual connection.

[0095] Residual connection: only when the input and output feature map size is consistent (28x28), the 6th layer input feature is directly added to the output feature element by element, the shallow calibrated emotion sensitive local information is retained, and the feature loss caused by deep convolution is avoided.

[0096] Output feature form: for each frame in the 240-frame calibrated facial feature sequence, the 6th layer outputs a single-frame feature with a dimension of 28x28x192. After stacking 240 frames, a feature sequence with a dimension of 240x28x28x192 is formed. At this time, the feature has been upgraded from "mid-layer local feature" to "deep-layer feature containing emotion-related regional combined semantics" (e.g., abstracting the combined feature of "eyebrow arch lifting + mouth corner pulling down" as "local semantic feature of anger emotion").

[0097] The 7th layer is the top convolutional module of the EfficientNet-B0 convolutional base, and its core function is to further abstract the emotion semantic feature and compress the spatial resolution through downsampling to reduce the computational complexity for subsequent temporal information aggregation (Bi-LSTM):

[0098] The 7th layer is composed of 4 groups of MBConv modules, and "downsampling operation" is added based on the 6th layer. The specific design is as follows. 1x1 upsizing convolution: similarly, the 192-channel feature is raised to 672 channels, continuing the high-dimensional expression of deep features. 3x3 depth separable convolution (with downsampling): a 3x3 depth separable convolution with a step of 2 is used to extract high-level semantic features while compressing the feature map resolution from 28x28 to 14x14 (half of the spatial size), which can not only reduce the computational load of subsequent processing, but also integrate larger range of emotion semantics through downsampling (e.g., a 14x14 resolution feature map can cover the overall emotion pattern of the entire face).

[0099] 1x1 downsizing convolution: compressing 672 channels to 320 channels (standard output channel number of EfficientNet-B0 pre-trained model), further enriching the expression dimension of emotion semantics through channel number (320 channels can cover high-level emotion features of multiple regional combinations such as "eye + eyebrow arch + mouth corner").

[0100] Residual connection: due to the existence of downsampling (resolution from 28x28→14x14), the input feature map size and channel number need to be adjusted through 1x1 convolution (from 192 channels→320 channels), and then the residual connection is performed with the output feature to ensure the effective fusion of deep and shallow features.

[0101] Output feature form: after the 7th layer processing, the single-frame feature dimension becomes 14x14x320, and after stacking the 240-frame calibrated facial feature sequence, the final output dimension is 240x14x14x320 of "current facial feature sequence".

[0102] Step a4, determine the target face confidence based on the current facial features.

[0103] Specifically, the above step a4 can include the following steps:

[0104] Step a41, calculate the Euclidean distance between the current facial features and each type of emotion feature center.

[0105] Specifically, the electronic device can calculate the "facial feature center" of each type of emotion, i.e. each type of emotion feature center, in advance based on the 6 types of emotion (calm, surprise, joy, anger, fear, anxiety) samples in the model training phase.

[0106] The electronic device can use the Euclidean distance formula to calculate the distance between the current facial feature vector (F current ) and each type of emotion feature center (C1-C6), which is as follows:

[0107]

[0108] Where k is the feature dimension index (1-320) and i is the emotion category index (1-6). Six Euclidean distance values (d1-d6) are generated, corresponding to the similarity (the smaller the distance, the higher the similarity, and the stronger the emotion differentiation) between the current facial features and the 6 types of emotion centers.

[0109] Step a42, if each Euclidean distance calculated is greater than a first preset distance threshold or less than a second preset distance threshold, then subtract a first preset value from the basic face confidence to obtain a candidate face confidence.

[0110] Where the first preset distance threshold is greater than the second preset distance threshold.

[0111] Specifically, the electronic device can set two types of distance thresholds based on the model training set statistics. The first preset distance threshold (T1): the mean of all "same type emotion feature and emotion feature center distance" in the training set × 1.2 (e.g. if the training set mean is 0.8, then T1=0.96), representing the "feature and all emotion centers have large differences, low differentiation" threshold. The second preset distance threshold (T2): the mean of all "same type emotion feature and emotion feature center distance" in the training set × 0.8 (e.g. if the training set mean is 0.8, then T2=0.64), representing the "feature and certain type of emotion center have high similarity, strong differentiation" threshold. T1>T2, to ensure that the two types of judgments do not overlap.

[0112] If one of the following two conditions is met, it is determined that the current facial features have low emotion differentiation, and the basic face confidence needs to be adjusted downward:

[0113] All Euclidean distances d1-d6 > T1: the current facial features are all different from the 6 emotional feature centers, and cannot be matched with any emotion (such as feature confusion caused by a blurred face).

[0114] All Euclidean distances d1-d6 < T2: the current facial features are all similar to the 6 emotional feature centers, and the feature distinction is blurred (such as a neutral facial feature close to the centers of multiple emotions).

[0115] Then, the electronic device can calculate the candidate facial confidence based on the following formula.

[0116] Candidate facial confidence = basic facial confidence - first preset value (the first preset value is 0.2-0.3, or 20% or 30% of the basic facial confidence).

[0117] Step a43, if there is only one Euclidean distance less than the second preset distance threshold, the basic facial confidence is added to the second preset value to obtain the candidate facial confidence.

[0118] Specifically, if there is only one Euclidean distance (such as d3, corresponding to the happy emotion) < T2, and the remaining 5 distances > T2, it is determined that the current facial feature has a high emotional distinction. It is highly similar to only one type of emotional center, and the feature is reliable.

[0119] Candidate facial confidence = basic facial confidence + second preset value (the second preset value can be 0.1-0.15, or 10% or 15% of the basic facial confidence).

[0120] Step a44, obtaining the acquisition scene parameters corresponding to the target facial image sequence.

[0121] Specifically, the electronic device can identify the target facial image sequence to determine the acquisition scene parameters corresponding to the target facial image sequence.

[0122] Among them, the acquisition scene parameters include night cruising scene (< 50 lux), daytime cruising scene (50-500 lux), and strong light direct incidence scene (> 500 lux).

[0123] Step a45, adjusting the candidate facial confidence according to the acquisition scene parameters to obtain the target facial confidence.

[0124] Specifically, the electronic device can adjust the candidate facial confidence according to the acquisition scene parameters to obtain the target facial confidence.

[0125] For example, in the night cruising scenario (<50 lux), low light causes loss of facial details (e.g., eye and mouth corner features are blurred), and the candidate face confidence is reduced by 15-25% (e.g., from 0.8 to 0.6-0.68). In the daytime cruising scenario (50-500 lux), the light is normal, and the facial features are clear, so the candidate face confidence remains unchanged. In the strong light direct hitting scenario (>500 lux), overexposure causes feature distortion, and the candidate face confidence is reduced by 20-30% (e.g., from 0.8 to 0.56-0.64).

[0126] The output result is the adjusted "target face confidence" (range 0-1), which integrates the "basic quality (integrity, clarity)" "feature distinguishability (distance from emotion center)" and "scene reliability (light)" three dimensions, providing a core reliability basis for subsequent dual-modal fusion and conflict resolution.

[0127] Step a5, based on the second sub-feature extraction network in the first branch, the calibration face features are extracted from the forward and reverse time dimensions to generate target face features.

[0128] Specifically, the above step a5 can include the following steps:

[0129] Step a51, stack the calibration face features in chronological order to generate a calibration face feature sequence.

[0130] Specifically, the electronic device can strictly stack the 240 single-frame calibration face features into a 240x28x28x192 calibration face feature sequence (time step x height x width x channel number) according to the chronological order of face image acquisition (from the 1st frame to the 240th frame).

[0131] Step a52, according to the input requirements of the second sub-feature extraction network, the calibration face feature sequence is processed to generate a face time sequence feature.

[0132] Specifically, the electronic device can perform a "global average pooling" operation on the 28x28x192 calibration face features of each frame in the 240 frames. The 28x28 spatial pixels of each frame feature map are averaged, and the 28x28x192 features are compressed into a 1x192 one-dimensional vector. A 240x192 face time sequence feature (240 is the time step, and 192 is the feature vector dimension) is generated, which meets the input format requirements of the second sub-feature extraction network.

[0133] Step a53, the second sub-feature extraction network performs forward feature extraction on the face time sequence feature in the forward time order, and outputs a forward hidden state for each time step.

[0134] Specifically, the second sub-feature extraction network (e.g., Bi-LSTM) contains 256 hidden units, and the forward network processes the facial time series features in the time order of "1st frame -> 240th frame".

[0135] The initial hidden state (hforward,0) of the forward network is set as an all-0 vector (dimension 1x256). For each time step t (1-240), the 1x192 feature vector of this step is inputted, combined with the forward hidden state h forward,t-1 of the previous time step, the hidden state is updated through the gating mechanism (input gate, forget gate, output gate) of LSTM, and the forward hidden state h forward,t of the current time step (dimension 1x256) is outputted. Thus, the dynamic dependence of emotion (e.g., the association of the 50th frame "slightly raised eyebrow arch" with the 100th frame "obviously raised eyebrow arch", reflecting the accumulation process of angry emotion) from "early frame -> late frame" is captured.

[0136] Step a54, the second sub-feature extraction network performs reverse feature extraction on the facial time series features in the reverse time order, and outputs the reverse hidden state for each time step.

[0137] Specifically, the structure is completely consistent with the forward network (256 hidden units), but processes the facial time series features in the reverse time order of "240th frame -> 1st frame".

[0138] The initial hidden state (h backward,240 ) of the reverse network is set as an all-0 vector (dimension 1x256). For each time step t (240-1), the 1x192 feature vector of this step is inputted, combined with the reverse hidden state h backward,t+1 of the next time step, the hidden state is updated, and the reverse hidden state h backward,t of the current time step (dimension 1x256) is outputted. The dynamic dependence of emotion (e.g., the association of the 200th frame "obviously pulled down corner of the mouth" with the 150th frame "slightly pulled down corner of the mouth", verifying the retroactive causal relationship of angry emotion from stable to initial state) from "late frame -> early frame" is captured.

[0139] Step a55, the target forward hidden state and the target reverse hidden state corresponding to the last time step are selected.

[0140] Specifically, the hidden state of the last time step (240th frame) of the second sub-feature extraction network has integrated all the previous time step emotional dynamic information, therefore, the hidden state h forward,240 of the forward network at the 240th frame (dimension 1x256) containing the forward time series dependence information of the 1st-240th frame is selected as the target forward hidden state. The hidden state h backward,1(1x256), containing the reverse time-dependent information of the 240th frame, as the target reverse hidden state.

[0141] Step a56, splicing the target forward hidden state and the target reverse hidden state to generate the target facial feature.

[0142] Specifically, the electronic device directly splices the target forward hidden state (1x256) and the target reverse hidden state (1x256) in the feature dimension, without changing the feature values, only expanding the dimension, to generate a target facial feature vector of 1x512 (256+256=512 dimensions), which covers the "forward + reverse" time dynamic information of the facial emotion within the 2-second window, and is the final output of the facial global emotion feature of the first branch.

[0143] Step S2042, inputting the target physiological signal data and the basic physiological signal confidence into the second branch in the preset emotion recognition model to output the target physiological signal feature and the target physiological signal confidence.

[0144] Specifically, the above step S2042 can include the following steps:

[0145] Step b1, performing feature extraction on the target physiological signal data based on the one-dimensional convolution layer in the second branch to output the local physiological signal waveform feature.

[0146] Specifically, the electronic device can call the one-dimensional convolution layer (convolution kernel size 3x1, step 1, channel number 128) in the second branch to perform local feature extraction on the target physiological signal data, and convert the target physiological signal data of 1x240 into a local physiological signal waveform feature of 240x128 (time step x channel number), each channel corresponding to an ECG local waveform mode (such as channels 1-20 corresponding to P waveform mode, and channels 80-100 corresponding to QRS wave amplitude).

[0147] Step b2, calibrating the local physiological signal waveform feature to obtain a calibrated physiological signal feature.

[0148] Specifically, the above step b2 can include the following steps:

[0149] Step b21, obtaining the local facial feature output by the first preset layer network in the first sub-feature extraction network for feature extraction on the target facial image sequence.

[0150] Wherein, the first sub-feature extraction network belongs to the first branch, and the local facial feature includes emotion-sensitive local regions corresponding to multiple parts of the face.

[0151] Specifically, the electronic device can call the local facial features output by the "preset layer network" (layer convolution in the 5th layer) of EfficientNet-B0 in the first branch, with a single-frame feature dimension of 28x28x192 (HxWxC), and a sequence of 240x28x28x192 after stacking 240 frames, containing features of sensitive emotional areas such as eyes, corners of the mouth, and eyebrow arches.

[0152] Step b22, screening the mouth opening and closing related features from the local facial features.

[0153] Specifically, according to the mapping relationship between the facial anatomy and the 5th layer feature map of EfficientNet-B0, the [18:25, 10:20] pixel range (7x10 pixels) of the feature map corresponding to the mouth region covers the upper and lower lips and the sides of the cheeks, excluding other irrelevant areas of the face (such as the nose and ears). Through the "channel-semantic" correlation (such as the Pearson correlation coefficient of the feature response of channel 80-100 and the mouth opening and closing action >0.6) statistical in the model training stage, 32 channels strongly related to the degree of mouth opening and closing are selected from the 192 channels. The feature values of these channels dynamically change with the amplitude of mouth opening and closing (the channel value increases when the mouth is open, and decreases when the mouth is closed).

[0154] Then, for the mouth region (7x10 pixels) and the 32 target channels of each frame in the 240 frames, "spatial global average pooling" is performed to compress the local features of 7x10x32 into a one-dimensional vector of 1x32 (eliminate the spatial dimension and retain the semantic information of "the degree of mouth opening and closing"). Through a lightweight fully connected layer (input 32 dimensions, output 128 dimensions, ReLU activation function), the 1x32 vector is mapped to a 1x128 vector, which is consistent with the channel number (128) of the ECG local feature, ensuring that the subsequent correlation calculation dimension matches. Finally, the electronic device stacks the 1x128 single-frame guide vectors of the 240 frames in time sequence to form a 240x128 mouth opening and closing related feature, which serves as a "guide signal" for ECG calibration.

[0155] Step b23, identifying the local physiological signal waveform features, and dividing the local physiological signal waveform features into at least one target segment according to the identification result.

[0156] Among them, the target segment is a core segment or an interference segment.

[0157] Specifically, based on the ECG physiological signal structure, the local physiological signal waveform features of 240x128 are divided into two types of target segments according to "functional attributes", and the emotion-related and noise regions are clearly defined.

[0158] Core segment: subtle fluctuations within the R-R interval (corresponding to the autonomic nervous activity during atrial and ventricular diastole), which is the main manifestation area of respiratory sinus arrhythmia (e.g., R-R interval lengthens during inhalation and shortens during exhalation), corresponding to channels 1-20 of ECG local features - such segments directly reflect the respiratory changes caused by emotions (e.g., rapid breathing due to anxiety leads to increased R-R interval fluctuations).

[0159] Interference segment: waveform segment that may contain motion artifacts (e.g., pilot adjusting the control stick causes electrode displacement, resulting in sharp, irregular pulse signals), no fixed channel, need to be judged by subsequent correlation degree with mouth opening and closing guide features (e.g., low correlation degree is determined as interference).

[0160] Step b24, calculate the second similarity between the mouth opening and closing related features and each target segment.

[0161] Specifically, the electronic device can quantify the emotional correlation degree of the dual-modal features in "time steps" (each step corresponds to 1 frame of face guide features and 1 ECG data point feature):

[0162] Face mouth opening and closing guide vector F of the tth step t (1x128), ECG local feature vector E t (1x128).

[0163] Cosine similarity formula: ; where, ∥F t ∥, ∥E t ∥ is the vector norm.

[0164] Finally, generate a 240x1 correlation degree sequence (each time step corresponds to a similarity value), and mark "high correlation segments (sim≥0.5)" and "low correlation segments (sim≤0.2)" in ECG local features.

[0165] Step b25, compare the second similarity between the mouth opening and closing related features and each target segment with the second preset similarity threshold.

[0166] Specifically, the electronic device can set two thresholds to distinguish "emotion-related fluctuations" and "noise" based on the validation set results in the model training phase. Reinforcement threshold (second preset similarity high threshold): sim≥0.5 - ECG local waveform and mouth opening and closing guide features have high correlation, which is determined as respiratory-related emotional fluctuations and needs to be reinforced. Inhibition threshold (second preset similarity low threshold): sim≤0.2 - ECG local waveform and guide features have low correlation, which is determined as motion artifacts and other noise and needs to be inhibited. Intermediate correlation (0.2

[0167] Step b26, according to the comparison result, calibrate each target segment, and output the calibrated physiological signal feature.

[0168] Specifically, the electronic device can locate the respiratory-related fluctuations within the R-R interval through physiological signal feature detection such as R-wave peak detection. The R-R interval segment corresponding to the physiological signal feature during inspiration is the t-10th to tth step, and the corresponding physiological signal feature during expiration is the tth to t+10th step.

[0169] The correlation degree is positively correlated, and the formula is: W enhanced =W original ×(1+0.1×(sim-0.5)). Example: sim=0.5, increase by 15% (weight 1.0→1.15), sim=0.7, increase by 25% (weight 1.0→1.25), ensure that the higher the correlation degree, the more significant the emotional fluctuation (such as the feature of the lengthening of the R-R interval during inspiration).

[0170] Combined with "waveform morphology + correlation degree" dual judgment. Motion artifacts are characterized by "sharp, irregular pulse waveforms" (such as sudden rise and fall of ECG feature values, slope > 5ms / step), and sim≤0.2.

[0171] The weight is adjusted by the formula: W suppressed =W original ×(0.5+0.1×(sim-0.2)). Example: sim=0.1, down-regulate by 50% (weight 1.0→0.5), sim=0.2, down-regulate by 40% (weight 1.0→0.6), significantly weaken the interference of noise on subsequent feature extraction.

[0172] Finally, stack the "enhanced core segment features" of 240 time steps and the "suppressed interference segment features" according to the original time sequence to generate the calibrated physiological signal feature, which still has a dimension of 240×128 (consistent with the calibrated one). In the calibrated physiological signal feature, the respiratory-related emotional fluctuation signal is clearer (such as the increased R-R interval fluctuation during anxiety), and the proportion of motion artifact noise is reduced by 40%-50%, providing high-quality input for deep time sequence feature extraction.

[0173] Step b3, extract the emotion-sensitive feature from the calibrated physiological signal feature.

[0174] Specifically, the electronic device locates the R-wave peak by a physiological signal waveform detection algorithm (such as Pan-Tompkins algorithm), calculates the time interval (RR interval) of adjacent R waves, and then calculates the values of SDNN (standard deviation of RR interval) and RMSSD (root mean square of adjacent RR interval difference) based on the RR interval sequence of 240 time steps. Among them, SDNN reflects the overall fluctuation degree of RR interval within 240 time steps, the more excited (such as angry, fearful) the emotion, the stronger the activity of the autonomic nervous system, and the smaller the SDNN (SDNN≈50ms when calm, SDNN<20ms when anxious). RMSSD reflects the difference degree of adjacent RR intervals, and RMSSD drops sharply (RMSSD≈30ms when calm, RMSSD<15ms when fearful) when the emotion is stressed (such as surprise).

[0175] Step b4, comparing the emotion-sensitive feature with the emotion arousal threshold.

[0176] Specifically, the electronic device can set two types of emotion arousal thresholds based on the "emotion-feature" corresponding relationship calculated by the model training set; calm emotion arousal threshold: SDNN≈50ms, RMSSD≈30ms (reflecting the autonomic nervous relaxation state). Negative emotion arousal threshold such as anxiety / fear: SDNN<20ms, RMSSD<15ms (reflecting the autonomic nervous tension state). If the extracted SDNN>50ms, RMSSD>30ms: the current calibration physiological signal feature matches the physiological pattern of calm emotion, and the emotion discrimination degree is high. If SDNN<20ms, RMSSD<15ms: the current calibration physiological signal feature matches the physiological pattern of negative emotion, and the emotion discrimination degree is high. If the feature fluctuation is between the thresholds (such as SDNN=30-50ms): the current calibration physiological signal feature cannot be clearly matched to a certain type of emotion, and the discrimination degree is low.

[0177] Step b5, adjusting the confidence of the basic physiological signal according to the comparison result to obtain the target physiological signal confidence.

[0178] Specifically, if the feature fluctuation≥the corresponding emotion threshold (such as SDNN=55ms≥calm threshold 50ms): the emotion discrimination degree is high, and the basic physiological signal confidence remains unchanged. If the feature fluctuation<the corresponding threshold (such as SDNN=18ms<anxiety threshold 20ms, but there is no obvious emotion direction): the emotion discrimination degree is low, and the basic physiological signal confidence is reduced by 15%-25% (such as the basic confidence 0.8, and the reduced confidence is 0.6-0.68).

[0179] Through the discrimination degree judgment, the weight of the "feature ambiguous" physiological signal data is weakened (such as the physiological signal feature that cannot be judged to be calm or slightly anxious).

[0180] Step b6, feature extraction is performed on the calibration physiological signal feature to generate a target physiological signal feature.

[0181] Specifically, the above step b6 can include the following steps:

[0182] Step b61, the signal value variance is calculated for the calibration physiological signal feature.

[0183] Specifically, the electronic device can calculate the numerical variance of the 240 time step feature values for each of the 128 channels, and the formula is: where x i is the feature value of the i-th time step, is the feature mean of the 240 time steps of the channel, and n=240 is the time step length.

[0184] Then, the electronic device calculates the arithmetic mean of the variances of the 128 channels to obtain the signal value variance. Avoiding the misjudgment caused by the abnormally high variance of a single channel due to noise, and ensuring that the variance can truly reflect the overall emotional fluctuation intensity.

[0185] Step b62, determining the current emotional fluctuation stage according to the signal value variance.

[0186] Wherein, the current emotional fluctuation stage includes a stable period, a fluctuation period, and a severe period.

[0187] Specifically, the electronic device can set three-stage threshold values based on the "emotion-variance" corresponding relationship (such as calm emotion, signal value variance <5ms², anxiety sample signal value variance in 5-15ms², and fear / anger sample signal value variance >15ms²) statistically obtained in the model training stage.

[0188] Stable period: signal value variance <5ms², corresponding to the calm emotion in the cruising stage, physiological signal stable (such as small R-R interval fluctuation, stable QRS wave amplitude), and no obvious change in emotion. Fluctuation period: signal value variance 5-15ms², corresponding to slight anxiety emotion when encountering air flow, physiological signal appears moderate amplitude fluctuation (such as small R-R interval shortening, QRS wave amplitude slightly rising). Severe period: signal value variance >15ms², corresponding to fear / anger emotion when sudden failure occurs, physiological signal appears large amplitude fluctuation due to stress reaction (such as sudden rise of QRS wave amplitude, severe change of R-R interval).

[0189] Step b63, determining the target sampling factor value according to the current emotional fluctuation stage.

[0190] Specifically, the electronic device can determine the preset fluctuation data corresponding to the current emotional fluctuation stage. For example, the preset fluctuation data is 8 when the current emotional fluctuation stage is a stable period, the preset fluctuation data is 4 when the current emotional fluctuation stage is a fluctuation period, and the preset fluctuation data is 2 when the current emotional fluctuation stage is a severe period.

[0191] Then, the basic sampling factor value is obtained, and the basic sampling factor value is (L=240 is the time step). The electronic device divides the basic sampling factor value by the preset fluctuation data corresponding to the current emotional fluctuation stage to obtain the target sampling factor value.

[0192] For example, the stable period (variance <5ms²): the target sampling factor value is set to , and only 2 key keys are sampled, which improves the calculation efficiency by 40% (reduces the amount of key calculation and avoids redundant calculation). Fluctuation period (variance 5-15ms²): the target sampling factor value is set to , and 4 key keys are sampled. Severe period (variance >15ms²): the target sampling factor value is set to , and 8 key keys are sampled (increased from the default value to ensure that the details of stress emotions are not lost).

[0193] Step b64, calculating the target value based on the target sampling factor value.

[0194] Specifically, the electronic device can calculate the target value by multiplying the sequence length by the target sampling factor value. For example, q=L×c (L=240 is the sequence length, and c is the target sampling factor value).

[0195] Step b65, adding position coding to the calibrated physiological signal feature to obtain a position coding physiological signal feature.

[0196] Specifically, for the 240x128 ECG calibration feature sequence, a position coding matrix (240x128) with completely matched dimensions is generated to ensure that each time step of the ECG feature can superimpose the corresponding position information.

[0197] The electronic device generates a 1x128 position coding vector for the pos time step (pos range 1-240), adopts sine-cosine coding, and the even dimension formula is: , and the odd dimension formula is: , where d model =128 is the feature channel number, and i is the channel index (0-63).

[0198] Then, the electronic device stacks the 1x128 encoded vectors of 240 time steps in chronological order to form a 240x128 position encoding matrix. Then, the position encoding matrix is element-wise superimposed with the ECG calibration feature sequence using "element-wise addition" instead of splicing. The formula is: (pos is the time step, ch is the channel), generating a "position encoded physiological signal feature" with a dimension of 240x128, providing "temporal + semantic" dual-dimensional information for the input of the subsequent multi-layer encoder.

[0199] Step b66, input the target value and the position encoded physiological signal feature into the first layer encoder in the multi-layer encoder.

[0200] Specifically, the electronic device can input the target value and the position encoded physiological signal feature into the first layer encoder in the multi-layer encoder.

[0201] Step b67, the first layer encoder splits the position encoded physiological signal feature into query features, key features, and value features.

[0202] Specifically, the electronic device splits the 240x128 position encoded physiological signal feature into three feature matrices of the same dimension according to the requirements of the Informer self-attention mechanism.

[0203] Query feature (Query, Q): dimension 240x128, representing the feature that needs to "query associated information" at each time step (e.g., QRS wave feature at step 50 needs to query other time step features related to it).

[0204] Key feature (Key, K): dimension 240x128, representing the feature that can be "queried" at each time step (e.g., P wave feature at step 100 can be used as the associated Key of other time steps).

[0205] Value feature (Value, V): dimension 240x128, representing the specific feature value corresponding to each Key, used for subsequent attention weighted summation.

[0206] Step b68, for each query feature, filter the most significant target value from all key features Target key features, and perform attention calculation on each target key feature and the query feature to output an attention feature.

[0207] Specifically, for each query feature Query (a total of 240), the top q Key (target value) is selected by "dot product score of Q and K" ranking, and the remaining 240-q Key is ignored. Among them, the higher the dot product score, the higher the similarity of Q and K features (for example, the QRS wave feature in step 50 and the QRS wave feature in step 100 have a high score, indicating that both reflect the increase of QRS wave amplitude reflecting anger emotion). Taking q=8 as an example, the computational complexity is reduced from O(240²) of traditional self-attention to O(240x8), which significantly improves efficiency and adapts to long sequence ECG processing.

[0208] Then, the electronic device calculates the dot product score of Q and K for the q Key after screening, and normalizes the attention weight by the Softmax function (the weight sum is 1, highlighting the contribution of high correlation Key). The normalized weight and the corresponding value feature are weighted and summed to obtain the attention feature of each Query, generating a "attention feature" of 240x128, preserving the time sequence length while strengthening the long sequence dependence of emotion-related (such as the association between the increase of P wave amplitude in step 50 and the increase of QRS wave amplitude in step 100, which corresponds to the accumulation process of anxiety emotion).

[0209] Step b69, performing nonlinear transformation on the attention feature to output the first encoding feature.

[0210] Specifically, the electronic device can input the attention feature into the feedforward neural network (FFN) to perform nonlinear transformation on the attention feature and output the first encoding feature.

[0211] Wherein, the feedforward neural network (FFN) is configured to include two fully connected layers, and the intermediate activation function is ReLU, and the specific structure is:

[0212] The first layer of full connection: the input dimension is 128, and the output dimension is 512 (128x4, Informer standard configuration), which aims to increase the feature expression dimension and strengthen the nonlinear combination of emotion semantics (such as combining "R-R interval fluctuation + QRS wave amplitude" into "anxiety emotion semantic feature").

[0213] The second layer of full connection: the input dimension is 512, and the output dimension is 128, which is the regression input channel number, ensuring that the dimension matches the subsequent residual connection.

[0214] Through FFN, the "time sequence dependent feature" captured by attention is converted into more abstract "emotion semantic feature", which improves the emotion discriminability of the feature, and finally generates a "first encoding feature" of 240x128, completing the core processing of the first layer encoder.

[0215] Step b610, performing down-sampling processing on the first encoding feature to generate a first down-sampling feature.

[0216] Specifically, the electronic device can adopt a one-dimensional convolutional layer with a step size of 2, where the convolution kernel size is 3: covering 3 consecutive time steps of features, filtering local high-information pieces (such as time pieces with dense emotional fluctuations); step size 2: sliding window moves 2 time steps each time, making the output sequence length halved.

[0217] The electronic device performs down-sampling on the 240x128 first encoded features based on the one-dimensional convolutional layer, and outputs 120x128 "first down-sampled features" (length halved, core emotional timing features retained).

[0218] Step b611, input the first down-sampled features into the second layer encoder in the multi-layer encoder, and output the second encoded features.

[0219] Specifically, the second layer encoder processes: input: first down-sampled features (120x128), target value q (adapted by stage, such as 8 in the intense period). Core operation: repeat steps b67-b610 (split Q / K / V -> filter Key -> attention calculation -> FFN nonlinear transformation -> step 2 down-sampling). Output: 60x128 "second down-sampled features" (length halved again).

[0220] Step b612, down-sampling processing is performed on the second encoded features to generate second down-sampled features.

[0221] Specifically, the electronic device can adopt a one-dimensional convolutional layer with a step size of 2 to down-sample the second encoded features to generate the second down-sampled features.

[0222] Step b613, the cycle continues until the top layer encoder in the multi-layer encoder outputs the top layer encoded features.

[0223] Specifically, the top layer encoder (third layer) processes:

[0224] Input: second down-sampled features (60x128), target value q.

[0225] Core operation: repeat steps b67-b69 (split Q / K / V -> filter Key -> attention calculation -> FFN nonlinear transformation), without performing down-sampling (the top layer needs to retain the complete global emotional features, without further compression).

[0226] Output: 60x128 "top layer encoded features". This feature is the core emotional representation of ECG timing signal, covering long sequence dependence within 2-second window (such as the whole process of ECG fluctuation from calm to anger) and emotional semantic information (such as the angry feature pattern of QRS wave amplitude rising and R-R interval shortening).

[0227] Step b614, for the top layer encoding features, a global average pooling operation is performed to generate the target physiological signal feature.

[0228] Specifically, the electronic device performs a global average pooling on the top layer encoding features (60x128) in the "time step dimension", calculates the arithmetic mean of 60 time step feature values of each channel (a total of 128), and compresses the 60x128 time sequence features into a 1x128 vector. Then, through a lightweight fully connected layer (input 128 dimensions, output 512 dimensions, ReLU activation function), the 1x128 pooling result is mapped to a 1x512 target physiological signal feature, which is consistent in dimension with the "target face feature" (512 dimensions) output by the first branch, providing a dimension matching basis for subsequent multi-modal vector splicing.

[0229] Step S2043, based on the target face feature, the target face confidence, the target physiological signal feature, and the target physiological signal confidence, outputting a target emotion corresponding to the target user.

[0230] Specifically, the above step S2043 can include the following steps:

[0231] Step c1, calculating a third similarity between the target face feature and the target physiological signal feature.

[0232] Specifically, the electronic device can use the cosine similarity formula to quantify the correlation of the double-modal features, and the third similarity between the target face feature and the target physiological signal feature is calculated as follows:

[0233]

[0234] Where F face is the target face feature vector, F ecg is the target physiological signal feature vector, the numerator is the vector dot product (reflecting the feature semantic overlap), and the denominator is the vector length product (standardizing the calculation range).

[0235] Step c2, if the third similarity is greater than or equal to a third similarity threshold, then based on the target face confidence and the target physiological signal confidence, the target face feature and the target physiological signal feature are weighted and fused to generate a target fusion feature.

[0236] Specifically, the electronic device can set the third similarity threshold (such as 0.5) based on the "similarity-emotion consistency" statistical results of the validation set in the model training phase.

[0237] Then, the electronic device compares the calculated third similarity with the third similarity threshold. If the third similarity is greater than or equal to the third similarity threshold (third similarity ≥ 0.5): the double-modal feature emotion semantics are consistent, and can be weighted and fused.

[0238] Next, the electronic device can calculate a facial feature weight and a physiological signal feature weight based on the target facial confidence and the target physiological signal confidence.

[0239] Let the target facial confidence be C face (range 0-1), and the target physiological signal confidence be C ecg (range 0-1), then: the facial feature weight: The physiological signal feature weight: The sum of the weights is 1, ensuring that the contribution of the bimodal features in the fusion process is positively correlated with their own reliability.

[0240] Then, the electronic device weights the 512-dimensional facial features and the 512-dimensional ECG features according to the facial feature weight and the physiological signal feature weight, and then splices them in the feature dimension, with the formula being: F fusion,raw =[W face ×F face ;W ecg ×F ecg ], generating a 1024-dimensional original fusion feature vector (“;” represents dimension splicing, the first 512 dimensions are weighted facial features, and the last 512 dimensions are weighted ECG features).

[0241] Then, the electronic device inputs the 1024-dimensional original fusion features into a preset fully connected layer. The fully connected layer is configured with 512 neurons and uses a ReLU activation function to learn the non-linear relationship between the bimodal features (such as the interaction feature “facial mouth up + ECG R-R interval stable” corresponding to “pleasure”). The fully connected layer is followed by a Dropout layer with a dropout rate of 0.5, which randomly masks 50% of the neurons to prevent the model from relying too much on a certain type of modal feature.

[0242] Finally, the electronic device generates a 512-dimensional “target fusion feature”, which not only retains the bimodal emotional information but also simplifies the subsequent classification calculation through dimension reduction.

[0243] Step c3, based on the target fusion feature, output the target emotion corresponding to the target user.

[0244] Specifically, the electronic device can input the 512-dimensional target fusion feature into a preset “final classification layer”.

[0245] The final classification layer is configured with 6 neurons, each corresponding to “calm, surprised, happy, angry, fearful, and anxious” six core emotions. A Softmax activation function is used to convert the classification layer output into a 6-dimensional probability distribution vector (each element takes a value in the range [0, 1], and the sum is 1), with each element representing the probability that the model predicts the pilot's current emotion as the corresponding category (such as a “pleasure” probability of 0.85 and a “calm” probability of 0.12).

[0246] The electronic device selects the emotion category corresponding to the "index with the highest probability value" in the probability distribution vector as the "target emotion" of the target user. Example: if the probability vector is [0.05, 0.03, 0.85, 0.02, 0.04, 0.01] (corresponding to the above six emotions), "joy" is selected as the target emotion.

[0247] Step c4, if the third similarity is less than the third similarity threshold, the target facial confidence and the target physiological signal confidence are compared, and the maximum target confidence is determined from the two.

[0248] Specifically, when the third similarity < the third similarity threshold (bimodal emotion semantic conflict, such as the face showing "calm" but the physiological signal showing "anxiety"), by comparing the "target facial confidence" and the "target physiological signal confidence", the modality feature with higher reliability is selected as the basis for emotion recognition, and the specific logic is as follows: if C face >C ecg : the target facial feature has higher reliability (such as target facial confidence 0.8, target physiological signal confidence 0.5), the "maximum target confidence" is C face , corresponding to the "target facial feature".

[0249] If CECG > CFACE: the target physiological signal feature has higher reliability (such as target physiological signal confidence 0.9, target facial confidence 0.6), the "maximum target confidence" is C ecg , corresponding to the "target physiological signal feature". If C face =C ecg : the target physiological signal feature is selected by default (because the physiological signal is less affected by subjective disguise, the emotion representation is more objective).

[0250] Step c5, based on the target feature corresponding to the maximum target confidence, the target emotion corresponding to the target user is output.

[0251] Specifically, if the maximum target confidence corresponds to the "target facial feature", the 512-dimensional target facial feature is directly input into the "final classification layer". If the maximum target confidence corresponds to the "target physiological signal feature": the 512-dimensional target physiological signal feature is directly input into the "final classification layer". The emotion classification and output classification process is consistent with step c3: the classification layer generates a 6-dimensional probability distribution vector through the Softmax activation function. The class with the highest probability is selected as the target emotion and output to the cockpit or ground monitoring system. Example: if the maximum confidence is ECG confidence 0.9, the classification layer outputs "anxiety" probability 0.88, and the final target emotion is "anxiety".

[0252] The pilot emotion recognition method provided by the embodiments of the present application focuses on the facial emotion sensitive area to extract local features through the first sub-feature extraction network, provides a fine-grained basis for subsequent calibration, and avoids the loss of local details caused by direct processing of deep networks. Combined with physiological signal guided calibration, emotion related region features are strengthened, irrelevant noise is suppressed, facial feature emotion discriminability is improved, and the influence of single modal noise on subsequent recognition is reduced. Then, the deep network after the pre-set layer is used to further abstract the calibrated local features, generate more emotion semantic high-level features, and provide high-quality input for confidence determination and time sequence extraction. Next, the confidence is adjusted through the emotion discrimination degree of the current facial feature and the collection scene, the reliability of the facial feature is quantified, and the weight basis is provided for subsequent fusion. Finally, the second sub-feature extraction network captures the dynamic dependence of the calibrated facial feature from the positive and negative time dimensions, splices the bidirectional hidden state to form the target facial feature, completely retains the facial emotion time sequence information within 2 seconds, and improves the emotion representation integrity.

[0253] The second branch accurately captures the physiological signal local waveform of the target physiological signal data through a one-dimensional convolution layer with fixed parameters, provides physiological emotion clues for subsequent calibration, and adapts to the long sequence processing requirement. Combined with the mouth opening and closing features, the local physiological signal waveform features are calibrated to obtain calibrated physiological signal features, thereby realizing the screening of breathing related emotion fluctuations and the suppression of motion artifacts, improving the emotion relevance of physiological signal features, and reducing the interference of cockpit environmental noise. The emotion sensitive features are extracted from the calibrated physiological signal features and compared with the emotion arousal threshold, thereby focusing on the features in the calibrated physiological signal features that are strongly related to emotions, quantifying the emotion fluctuation intensity through the threshold, and providing an objective physiological basis for confidence adjustment. Adjust the confidence of the basic physiological signal to obtain the confidence of the target physiological signal, adjust the confidence combined with the emotion feature discrimination degree and the scene, and ensure the accuracy of the quantification of the reliability of the ECG feature, avoiding the misguidance of low-quality physiological data on emotion judgment. Next, the calibrated physiological signal features are position encoded, and then the long sequence dependence of the calibrated physiological signal features is extracted through a deep extraction of the Informer multi-layer encoder to generate target physiological signal features, which are matched in dimension with the target facial features, laying a foundation for multi-modal fusion.

[0254] Finally, the third similarity between the target face and the physiological features is calculated to quantify the dual-modal emotional semantic correlation degree, to provide a judgment standard for "fusion or conflict resolution", and to avoid result deviation caused by blind integration. When the third similarity meets the standard, the target fusion feature is generated by weighted fusion, the weights are dynamically allocated based on the dual-modal confidence, the advantages of dual-modal are integrated, the emotional recognition accuracy is improved, and the reliable recognition demand in multiple scenes is adapted. The target emotion is output based on the target fusion feature, the fusion feature covers dual-modal emotional information, and the accurate emotional result is output after the classification layer processing, to provide a core basis for flight safety monitoring. If the third similarity does not meet the standard, the maximum target confidence is determined, the more reliable modal feature is selected through confidence comparison, the dual-modal conflict problem is solved, and even if the single-modal quality is poor, the reliable result can still be output. The target emotion is output based on the feature corresponding to the maximum confidence, the high reliability modal information is preferentially used, the low quality modal interference is avoided, and the stability and usability of the emotional recognition result are ensured.

[0255] Although the embodiments of the present application are described in conjunction with the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A method for pilot emotion recognition, characterized in that, The method includes: Acquire the target user's facial image sequence and target physiological signal data; The target facial image sequence is identified to determine the basic facial confidence level corresponding to the target facial image sequence; The target physiological signal data is identified, and the confidence level of the basic physiological signal corresponding to the target physiological signal data is determined. The target facial image sequence, the target physiological signal data, the basic facial confidence score, and the basic physiological signal confidence score are input into a preset emotion recognition model, and the target emotion corresponding to the target user is output. The step of inputting the target facial image sequence, the target physiological signal data, the basic facial confidence score, and the basic physiological signal confidence score into a preset emotion recognition model, and outputting the target emotion corresponding to the target user, includes: The target facial image sequence and the basic facial confidence score are input into the first branch of the preset emotion recognition model, and the target facial features and target facial confidence score are output. The target physiological signal data and the confidence level of the basic physiological signal are input into the second branch of the preset emotion recognition model, and the target physiological signal features and the confidence level of the target physiological signal are output. Based on the target facial features, the target facial confidence level, the target physiological signal features, and the target physiological signal confidence level, the target emotion corresponding to the target user is output; The step of inputting the target facial image sequence and the basic facial confidence score into the first branch of the preset emotion recognition model, and outputting the target facial features and the target facial confidence score, includes: Based on the pre-preset layer network in the first sub-feature extraction network of the first branch, the target facial image sequence is extracted to output local facial features; The local facial features are calibrated, and the calibrated facial features are output. Based on the pre-preset layer network after the pre-preset layer network in the first sub-feature extraction network, the calibrated facial features are extracted, and the current facial features are output. Based on the current facial features, determine the confidence level of the target face; Based on the second sub-feature extraction network in the first branch, the calibrated facial features are extracted from both forward and reverse time dimensions to generate the target facial features.

2. The method according to claim 1, characterized in that, The local facial features include emotionally sensitive local areas corresponding to multiple parts of the face; the calibration of the local facial features and the output of calibrated facial features include: Obtain the local physiological signal waveform features output by the one-dimensional convolutional layer in the second branch for feature extraction of the target physiological signal data; Key dimensions directly related to emotions are selected from the local physiological signal waveform features as guiding signals; For each of the aforementioned emotionally sensitive local regions, a first similarity is calculated between the emotionally sensitive local region and the guidance signal; The first similarity between each of the emotionally sensitive local regions and the guidance signal is compared with a first preset similarity threshold. Based on the comparison results, each of the emotionally sensitive local areas is calibrated, and the calibrated facial features are output.

3. The method according to claim 1, characterized in that, The step of determining the confidence level of the target face based on the current facial features includes: Calculate the Euclidean distance between the current facial features and the centers of various emotional features; If all the calculated Euclidean distances are greater than the first preset distance threshold, or are all less than the second preset distance threshold, then the basic face confidence score is subtracted from the first preset value to obtain the candidate face confidence score; the first preset distance threshold is greater than the second preset distance threshold. If there is only one Euclidean distance less than the second preset distance threshold, then the basic face confidence is added to the second preset value to obtain the candidate face confidence score. Obtain the acquisition scene parameters corresponding to the target facial image sequence; Based on the collected scene parameters, the confidence scores of the candidate faces are adjusted to obtain the confidence scores of the target faces.

4. The method according to claim 1, characterized in that, The second sub-feature extraction network in the first branch extracts features from the calibrated facial features from both forward and reverse time dimensions to generate the target facial features, including: The calibrated facial features are stacked in chronological order to generate a calibrated facial feature sequence; Based on the input requirements corresponding to the second sub-feature extraction network, the calibrated facial feature sequence is processed to generate facial time series features; The second sub-feature extraction network performs positive feature extraction on the facial time series features in a forward temporal order, and outputs a positive hidden state at each time step; The second sub-feature extraction network performs reverse feature extraction on the facial time series features in reverse time order, and outputs the reverse hidden state at each time step; Select the target's forward hidden state and target's reverse hidden state corresponding to the last time step; The target's forward hidden state and the target's reverse hidden state are concatenated to generate the target's facial features.

5. The method according to claim 1, characterized in that, The step of inputting the target physiological signal data and the basic physiological signal confidence level into the second branch of the preset emotion recognition model, and outputting the target physiological signal features and the target physiological signal confidence level, includes: Based on the one-dimensional convolutional layer in the second branch, feature extraction is performed on the target physiological signal data, and local physiological signal waveform features are output. The local physiological signal waveform features are calibrated to obtain calibrated physiological signal features; Extract emotion-sensitive features from the calibrated physiological signal features; The emotional sensitivity characteristics are compared with the emotional arousal threshold; The confidence level of the basic physiological signal is adjusted based on the comparison results to obtain the confidence level of the target physiological signal; Feature extraction is performed on the calibration physiological signal features to generate the target physiological signal features.

6. The method according to claim 5, characterized in that, The calibration of the local physiological signal waveform features to obtain calibrated physiological signal features includes: The local facial features are obtained from the feature extraction output of the target facial image sequence by the pre-preset layer network in the first sub-feature extraction network; the first sub-feature extraction network belongs to the first branch; the local facial features include emotionally sensitive local regions corresponding to multiple parts of the face; Features related to mouth opening and closing are selected from the aforementioned local facial features; The local physiological signal waveform features are identified, and based on the identification results, the local physiological signal waveform features are divided into at least one target segment; the target segment is a core segment or an interference segment; Calculate the second similarity between the mouth opening and closing related features and each of the target segments; The second similarity between the mouth opening and closing related features and each of the target segments is compared with a second preset similarity threshold; Based on the comparison results, each target segment is calibrated, and the calibrated physiological signal characteristics are output.

7. The method according to claim 5, characterized in that, The step of extracting features from the calibration physiological signal features to generate the target physiological signal features includes: For the calibrated physiological signal characteristics, calculate the numerical variance of the signal; Based on the variance of the signal values, the current emotional fluctuation stage is determined; the current emotional fluctuation stage includes a stable period, a fluctuating period, and a period of intense emotion. Determine the target sampling factor value based on the current emotional fluctuation stage; Calculate the target value based on the target sampling factor value; The calibration physiological signal features are then encoded with location codes to obtain location-coded physiological signal features. The target value and the location-encoded physiological signal features are input into the first layer encoder of the multilayer encoder; The first layer encoder decomposes the location-encoded physiological signal features into query features, key features, and value features; For each query feature, the most significant target value target key features are selected from all the key features, and attention is calculated between each target key feature and the query feature to output the attention feature; The attention features are subjected to a nonlinear transformation to output the first encoded features; The first encoded feature is downsampled to generate a first downsampled feature; The first downsampling feature is input into the second layer encoder of the multilayer encoder, and the second encoded feature is output. The second encoded feature is downsampled to generate a second downsampled feature; This process is repeated until the top encoder in the multi-layer encoder outputs the top-level encoded feature; The target physiological signal features are generated by performing a global average pooling operation on the top-level encoded features.

8. The method according to claim 1, characterized in that, The step of outputting the target emotion corresponding to the target user based on the target facial features, the target facial confidence level, the target physiological signal features, and the target physiological signal confidence level includes: Calculate the third similarity between the target facial features and the target physiological signal features; If the third similarity is greater than or equal to the third similarity threshold, then based on the target face confidence and the target physiological signal confidence, the target face features and the target physiological signal features are weighted and fused to generate target fusion features; Based on the target fusion features, the target emotion corresponding to the target user is output; If the third similarity is less than the third similarity threshold, then the confidence score of the target face and the confidence score of the target physiological signal are compared, and the maximum target confidence score is determined from the two. Based on the target features corresponding to the maximum target confidence, the target emotion corresponding to the target user is output.

Images