Method and device for emotion recognition based on facial tissue oxygen saturation data

Abstract

The present application provides a method and device for emotion recognition based on facial tissue oxygen saturation data, which relates to the field of emotion measurement technology. The method includes: preprocessing the collected initial data to obtain standard physiological signal data; wherein the standard physiological signal data is standard facial tissue oxygen saturation data; extracting features from each standard physiological signal data to obtain target coding features; inputting the target coding features into a deep learning model for emotion recognition to obtain emotion results. It is capable of collecting the physiological signals of the real reaction of the person under test as the data basis for emotion recognition, preprocessing the collected data and then extracting features to obtain coding features input into the deep learning model for recognition, and being able to recognize multiple emotions of the person under test, obtain accurate and true emotion results, effectively reduce the adverse effects of factors such as subjective control of the person on emotion recognition, and improve the accuracy and authenticity of emotion recognition.

Description

Technical Field

[0001] The present application relates to the field of emotion measurement technology, and in particular, to a method and device for emotion recognition based on facial tissue oxygen saturation data. Background Art

[0002] In terms of emotion recognition, there are many ways to identify a person's emotions. For example, the person's emotional state can be inferred based on their external appearance, such as facial expressions, voice intonation, and body movements. However, since external expressions can be controlled by individuals, this emotion recognition method cannot accurately determine the person's true emotions and cannot meet current emotion recognition needs. Summary of the invention

[0003] In view of this, the purpose of the embodiments of the present application is to provide a method and device for emotion recognition based on facial tissue oxygen saturation data, so as to improve the problem of low emotion recognition accuracy in the prior art.

[0004] In order to solve the above problems, in a first aspect, an embodiment of the present application provides a method for emotion recognition based on facial tissue oxygen saturation data, the method comprising:

[0005] Preprocessing the collected initial data to obtain standard physiological signal data; wherein the standard physiological signal data is standard facial tissue oxygen saturation data;

[0006] Performing feature extraction on each of the standard physiological signal data to obtain target coding features;

[0007] The target encoding features are input into a deep learning model for emotion recognition to obtain an emotion result.

[0008] In the above implementation process, physiological signals with real reactions of the person under test, namely, facial tissue oxygen saturation data, can be collected as the data basis for emotion recognition, and the collected initial data is preprocessed to obtain standard physiological signal data of the person's emotions that can be used for emotion recognition, and then feature extraction is performed on the standard physiological signal data to obtain the target coding features input into the deep learning model for recognition, so that multiple emotions of the person under test can be recognized and accurate and true emotion results can be obtained. Compared with emotion recognition based on external features such as facial expressions, voice intonation, and body movements, the present application uses the physiological signals of the person for emotion recognition, which can effectively reduce the adverse effects of factors such as the subjective control of the person on emotion recognition, thereby improving the accuracy and authenticity of emotion recognition.

[0009] Optionally, the extracting features of each of the standard physiological signal data to obtain target coding features includes:

[0010] Acquiring the standard physiological signal data including baseline data before the emotion is generated and emotion data after the emotion is generated;

[0011] Cluster analysis is performed based on the standard physiological signal data to obtain the target coding features.

[0012] In the above implementation process, a corresponding method is designed according to the characteristics of the standard physiological signal data to extract the target coding features therein. The standard physiological signal data containing baseline data and emotional data can be obtained, and cluster analysis is performed on the standard physiological signal data to quantify it and obtain the corresponding target coding features. The physiological signals of various emotional states can be processed accordingly, which effectively improves the effectiveness of the target coding features.

[0013] Optionally, the target coding feature includes: an intensity difference coding feature;

[0014] The cluster analysis based on the standard physiological signal data to obtain the target coding feature includes:

[0015] Cluster analysis and subtraction processing are performed based on the baseline data and the emotion data to obtain the intensity difference coding feature.

[0016] In the above implementation process, the intensity difference coding feature can be designed as the target coding feature for emotion recognition in the input deep learning model. Cluster analysis is performed on the data of the baseline data before the emotion is generated and the emotional data after the emotion is generated in the standard physiological signal data, and then the baseline data after clustering is subtracted from the clustered emotional data to obtain the intensity difference coding feature after quantifying the standard physiological signal data.

[0017] Optionally, the target coding feature includes: a histogram difference coding feature;

[0018] The cluster analysis is performed based on the standard physiological signal data to obtain the target coding feature, and further includes:

[0019] Histogram processing is performed based on the intensity difference coding feature to obtain the histogram difference coding feature.

[0020] In the above implementation process, based on the intensity difference coding feature, the histogram difference coding feature can also be designed as the target coding feature for emotion recognition in the input machine learning model. By performing histogram processing on the intensity difference coding feature to obtain the corresponding histogram difference coding feature, the emotion can be recognized from the perspective of the histogram, which further improves the effectiveness of the target coding feature.

[0021] Optionally, the target coding feature includes: a mask difference coding feature;

[0022] The cluster analysis is performed based on the standard physiological signal data to obtain the target coding feature, and further includes:

[0023] Calculating facial difference value data based on the baseline data and the emotion data;

[0024] Performing scaling processing according to the intensity difference coding feature to obtain mask data;

[0025] The mask difference encoding feature is obtained by performing calculation based on the mask data and the facial difference value data.

[0026] In the above implementation process, based on the intensity difference coding feature, a mask difference coding feature can also be designed as a target coding feature for emotion recognition in the input deep learning model. The facial difference value data is calculated using the initial baseline data and the emotion data, and the intensity difference coding feature is scaled to obtain the corresponding mask data, and then the mask data and the facial difference value data are combined for calculation to obtain the mask difference coding feature after mask processing, which further improves the effectiveness of the target coding feature.

[0027] Optionally, the preprocessing of the collected initial data to obtain standard physiological signal data includes:

[0028] Acquire original facial images of the person under test before and after viewing the emotional material, which are collected based on spectral imaging technology; wherein the original facial images include first facial data before viewing the emotional material and second facial data after viewing the emotional material;

[0029] Obtaining scoring data determined by the tested person based on the emotional material;

[0030] Screening the original facial image based on a preset screening rule and the scoring data to obtain valid initial data;

[0031] Extracting the initial data to obtain initial physiological signal data;

[0032] The initial physiological signal data is registered to obtain the standard physiological signal data.

[0033] In the above implementation process, in order to extract physiological signals, the original facial images of the person under test before and after the test based on the emotional material can be collected based on spectral imaging technology, and the scoring data determined by the person under test based on the emotional material can be obtained, so as to screen the original facial images according to the screening rules and the scoring data, and obtain the initial data that the person under test considers to be valid. The physiological signals can be extracted from the initial data and facial registration can be performed to obtain valid and uniform-sized standard physiological signal data for subsequent processing. The validity and size of the data can be screened and adjusted in the preprocessing stage of the data, thereby improving the validity and adaptability of the standard physiological signal data.

[0034] Optionally, the method further comprises:

[0035] Determining an emotion label corresponding to the standard physiological signal data;

[0036] The emotion result is verified according to the emotion label.

[0037] In the above implementation process, when obtaining standard physiological signal data, the emotional state of the standard physiological signal data can also be classified to obtain the corresponding type of emotion label, so as to verify the final emotion result in the model according to the emotion label, so as to optimize the recognition effect of the model, and effectively improve the accuracy of subsequent models when performing emotion recognition based on physiological signals.

[0038] In a second aspect, an embodiment of the present application further provides a device for emotion recognition based on facial tissue oxygen saturation data, the device comprising: a preprocessing module, a feature extraction module and a recognition module;

[0039] The preprocessing module is used to preprocess the collected initial data to obtain standard physiological signal data; wherein the standard physiological signal data is standard facial tissue oxygen saturation data;

[0040] The feature extraction module is used to extract features from each of the standard physiological signal data to obtain target coding features;

[0041] The recognition module is used to input the target coding features into a deep learning model to perform emotion recognition and obtain an emotion result.

[0042] In the above implementation, the collected initial data is preprocessed by a preprocessing module to obtain standard physiological signal data of emotions produced by a person capable of emotion recognition, and feature extraction is performed on the standard physiological signal data by a feature extraction module to obtain target coding features. The target coding features are input into a deep learning model by a recognition module to perform emotion recognition, which can recognize multiple emotions of the person being tested and obtain accurate and true emotion results.

[0043] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores program instructions, and when the processor reads and runs the program instructions, it executes the steps in any implementation of the above-mentioned method for emotion recognition based on facial tissue oxygen saturation data.

[0044] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions. When the computer program instructions are read and executed by a processor, the steps in any implementation of the above-mentioned method for emotion recognition based on facial tissue oxygen saturation data are executed.

[0045] In summary, the embodiments of the present application provide a method and device for emotion recognition based on facial tissue oxygen saturation data, which can collect physiological signals of the real reaction of the person under test as the data basis for emotion recognition, and perform feature extraction after preprocessing the collected data to obtain coding features for input into the deep learning model for recognition. It can recognize multiple emotions of the person under test and obtain accurate and true emotion results, effectively reduce the adverse effects of factors such as subjective control of the person on emotion recognition, and improve the accuracy and authenticity of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A block diagram of an electronic device provided in an embodiment of the present application;

[0048] Figure 2 A flowchart of a method for emotion recognition based on facial tissue oxygen saturation data provided in an embodiment of the present application;

[0049] Figure 3 A detailed flowchart of step S300 provided in an embodiment of the present application;

[0050] Figure 4 A detailed flow chart of step S320 provided in an embodiment of the present application;

[0051] Figure 5 A detailed flow chart of step S200 provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the structure of a device for performing emotion recognition based on facial tissue oxygen saturation data provided in an embodiment of the present application.

[0053] Icons: 100-electronic device; 111-memory; 112-storage controller; 113-processor; 114-peripheral interface; 115-input and output unit; 116-display unit; 500-device for emotion recognition based on facial tissue oxygen saturation data; 510-preprocessing module; 520-feature extraction module; 530-recognition module. DETAILED DESCRIPTION

[0054] The technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments of the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the embodiments of the present application.

[0055] The six basic emotions are universally shared by humans, including anger, disgust, fear, happiness, sadness, and surprise. In terms of emotion recognition, the emotions of a person can be recognized in many ways. For example, the emotional state of the person can be inferred based on the person's external appearance, such as facial expressions, voice intonation, and body movements. However, since external expressions can be controlled by individuals, this emotion recognition method cannot accurately determine the person's true emotions and cannot meet the current emotion recognition needs.

[0056] In order to solve the above problems, an embodiment of the present application provides a method for emotion recognition based on facial tissue oxygen saturation data, which is applied to electronic devices. The electronic devices may include servers, personal computers (PCs), tablet computers, smart phones, personal digital assistants (PDAs), and other electronic devices with logical computing functions. The method can recognize the emotions of a person based on the physiological signals of the person and obtain accurate and true emotional results.

[0057] Optionally, see Figure 1 , Figure 1The block diagram of an electronic device provided in an embodiment of the present application. The electronic device 100 may include a memory 111, a storage controller 112, a processor 113, a peripheral interface 114, an input and output unit 115, and a display unit 116. A person skilled in the art may understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the electronic device 100. For example, the electronic device 100 may further include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0058] The memory 111, storage controller 112, processor 113, peripheral interface 114, input / output unit 115 and display unit 116 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The processor 113 is used to execute the executable module stored in the memory.

[0059] The memory 111 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 111 is used to store programs, and the processor 113 executes the program after receiving the execution instruction. The method executed by the electronic device 100 defined by the process disclosed in any embodiment of the present application can be applied to the processor 113, or implemented by the processor 113.

[0060] The processor 113 may be an integrated circuit chip with signal processing capability. The processor 113 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor.

[0061] The peripheral interface 114 couples various input / output devices to the processor 113 and the memory 111. In some embodiments, the peripheral interface 114, the processor 113 and the memory controller 112 can be implemented in a single chip. In other embodiments, they can be implemented by separate chips.

[0062] The input / output unit 115 is used to provide input data to the user. The input / output unit 115 may be, but is not limited to, a mouse and a keyboard.

[0063] The above-mentioned display unit 116 provides an interactive interface (such as a user operation interface) between the electronic device 100 and the user or is used to display image data for the user's reference. In the present embodiment, the display unit may be a liquid crystal display or a touch display. If it is a touch display, it may be a capacitive touch screen or a resistive touch screen that supports single-point and multi-point touch operations. Supporting single-point and multi-point touch operations means that the touch display can sense touch operations generated simultaneously from one or more positions on the touch display, and the sensed touch operations are handed over to the processor for calculation and processing. In an embodiment of the present application, the display unit 116 can display data such as the processed analysis chart data, target coding features, and recognized emotional results.

[0064] The electronic device in this embodiment can be used to execute each step in the method for emotion recognition based on facial tissue oxygen saturation data provided in the embodiments of the present application. The implementation process of the method for emotion recognition based on facial tissue oxygen saturation data is described in detail below through several embodiments.

[0065] See also Figure 2 , Figure 2A flowchart of a method for emotion recognition based on facial tissue oxygen saturation data is provided in an embodiment of the present application. The method may include steps S200-S400.

[0066] Step S200, preprocessing the collected initial data to obtain standard physiological signal data.

[0067] Among them, the initial data of physiological signals with real reactions of the person being tested is collected as the data basis for emotion recognition, and the initial data is preprocessed to extract standard physiological signal data therein.

[0068] Optionally, the standard physiological signal data may include standard facial tissue oxygen saturation data. Facial tissue oxygen saturation (Tissue Oxygen Saturation) data, namely StO2, is a two-dimensional physiological signal related to blood, which reflects the degree of binding of hemoglobin to oxygen molecules at each pixel position. The higher the oxygenation degree, the larger the StO2 value. Therefore, StO2 is a physiological indicator, which fluctuates with changes in hormone levels. StO2 can be obtained through spectral imaging technology, such as hyperspectral imaging technology HSI (Hyperspectral Imaging) and other spectral imaging technologies that meet the band range requirements. For example, in actual application, in order to improve accuracy, 33 narrow band data between 516 and 580nm are used to perform non-contact extraction based on different bands, and it has the advantages of non-contact collection convenience and physiological signal authenticity. For example, it can be extracted from the 516 to 580nm band range in the hyperspectral cube through the MBL3C model, and can intuitively present different changes in facial StO2 under various basic emotional states through graphical analysis. Therefore, StO2 has the ability to distinguish emotional states.

[0069] Step S300, extracting features from each standard physiological signal data to obtain target coding features.

[0070] Among them, since the changes in physiological signals are obviously different when in different emotional states, the standard physiological signal data can be feature extracted to obtain corresponding target coding features. The target coding features can be various types of data such as pixel image data of facial images.

[0071] Step S400: input the target coding features into the deep learning model for emotion recognition to obtain the emotion results.

[0072] The extracted target coding features are input as input data into the deep learning model for emotion recognition, and the corresponding emotion results can be obtained. The emotion results can be corresponding emotional states, such as anger, disgust, fear, happiness, sadness, surprise or more emotional states.

[0073] Optionally, the deep learning model may include a variety of deep learning network models, such as neural networks, random forests, etc., which can use the target encoding features as input to deep learning frameworks such as convolutional neural networks CNN, residual networks ResNet, etc., and automatically extract deep features through network learning for state classification.

[0074] It should be noted that other methods can also be used to perform emotion recognition on the target coding features. For example, based on the target coding features, focus on specific facial areas, such as the forehead, nose, cheeks, etc., to extract the difference between the StO2 mean of the emotional state of the area and the StO2 mean of the corresponding area when no emotion is generated, that is, the StO2 mean difference feature, and then determine the corresponding emotional result based on the StO2 mean difference feature.

[0075] Optionally, in order to verify the final emotional result, the method may further include: determining the emotional label corresponding to the standard physiological signal data; and verifying the emotional result according to the emotional label. When obtaining the standard physiological signal data, the emotional state of the standard physiological signal data may also be classified to obtain the corresponding type of emotional label, so as to verify the final emotional result in the model according to the emotional label. For example, when the emotional label is angry and the emotional result is also angry, the verification is successful, which can optimize the recognition effect of the model and effectively improve the accuracy of the subsequent model in emotion recognition based on physiological signals.

[0076] exist Figure 2 In the embodiment shown, emotion recognition based on a person's physiological signals can effectively reduce the adverse effects of factors such as the person's subjective control on emotion recognition, compared to emotion recognition based on external features such as facial expressions, voice intonation, and body movements, thereby improving the accuracy and authenticity of emotion recognition.

[0077] Optionally, see Figure 3 , Figure 3 A detailed flowchart of step S300 is provided for an embodiment of the present application. Step S300 may include steps S310-S320.

[0078] Step S310, obtaining standard physiological signal data including baseline data before the emotion is generated and emotion data after the emotion is generated.

[0079] Among them, the database can store multiple standard physiological signal data obtained after preprocessing. When performing feature extraction, standard physiological signal data including baseline data before the generation of emotion and emotion data after the generation of emotion can be obtained from the database according to needs.

[0080] Step S320: performing cluster analysis based on standard physiological signal data to obtain target coding features.

[0081] Among them, since the deep model directly processes the standard physiological signal data, it cannot pay attention to the StO2 changes in the key areas before and after emotional stimulation. Therefore, it is necessary to perform cluster analysis on the standard physiological signal data to quantify the StO2 changes. The standard physiological signal data before and after emotional stimulation is used as the data basis for encoding. The cluster analysis method is used to divide the standard physiological signal data into a certain number of levels. The corresponding target coding features are calculated using the divided standard physiological signal data. The target coding features have the advantages of simple values ​​and clear areas with the same StO2 change level. They can be directly used as input data for processing by the deep learning model, which improves the robustness of the coding features in different tasks.

[0082] Optionally, in the standard physiological signal data, the baseline data before emotional stimulation, that is, before the emotion is generated, can be recorded as B, and the emotional data after emotional stimulation, that is, after the emotion is generated, can be recorded as D. The feasibility of facial StO2 being used for emotion recognition can be proved based on the preset analysis algorithm. Among them, the preset analysis algorithm can be a variety of algorithms such as the difference mean map algorithm or the t-value map analysis algorithm. The difference mean map algorithm and the t-value map analysis algorithm can perform calculations and analysis on the baseline data and emotional data of each emotional state to obtain corresponding analysis map data.

[0083] Optionally, the calculation method of the difference mean graph algorithm may include using all D data under a certain emotional state minus the corresponding B data, and averaging the obtained difference value data pixel by pixel, so as to obtain the difference mean graph under the emotional state as the corresponding analysis graph data. The calculation method of the t-value graph analysis algorithm may include: performing a paired sample t test on the D data and B data under a certain emotional state pixel by pixel, and obtaining the t graph under the emotional state as the corresponding analysis graph data. Among them, the specific calculation methods of the difference mean graph and the t graph are as follows:

[0084]

[0085] t_(i,j)=ttest[ s (i,j),B s (,j)]

[0086] Among them, s represents a standard physiological signal data, N represents the total number of valid data under each emotion category, and (i, j) represents the StO2 pixel of the face. According to the number and situation of samples, the paired sample t test can be used to calculate the t graph, and the p value can be corrected using FDR.

[0087] Since the combination of the difference mean map and the t map can effectively and accurately analyze the changes in facial StO2 caused by emotional stimulation, the significance of the analysis map can be analyzed to observe the differences in the changes in physiological signals under different emotional states in the analysis map data.

[0088] For example, since the calculated difference mean graph shows the overall changes of the tested person under a certain emotional stimulus, it is necessary to analyze the significance of the analysis graph in combination with statistical analysis methods. The difference mean graph represents the average changes of the existing data set. When angry, the whole face rises most obviously, accompanied by the warmest tones under all emotions, but a small area such as the lip area has decreased. Compared with the changes in the nose area under other emotions, the changes in the nose area of ​​disgust are smaller, but the changes around the nose and forehead are larger. The facial StO2 under fear emotion decreases significantly with the nose area as the center, including the forehead area, but the cheek and chin areas rise slightly. When happy, the whole face rises more evenly, and the lips also drop a little. When sad, the distribution around the eyes is even, and the rise around the mouth is obvious. When surprised, the mouth and chin areas rise significantly, and the tones of this area are warmer than the rest of the area.

[0089] The calculated t-map shows the t-value after the paired sample test, which contains information on the significance level. Therefore, the positive and negative values ​​of the pixel values ​​in the t-map respectively indicate that D is higher or lower than B, and the p-value is a value representing significance. In statistical analysis, if the means of the two samples are different, the statistical values ​​will be positive or negative, and this positive and negative represents the difference between the original two sets of data. The p-value can show significance in the difference pattern map (the difference pattern map shows the t-value, and the t-value and the p-value appear in pairs. If the p of a certain pixel t in the difference pattern is greater than 0.05, then this pixel is set to 0, indicating that the positive and negative changes at this position are not significant). Therefore, if the corresponding p-value is less than 0.05, it means that the high and low values ​​are significant at this position, indicating that the changes in facial StO2 under different emotional states are different. Otherwise, it is not significant and has no statistical significance, indicating that the data for calculation is invalid and is deleted. The quantified t-map can also intuitively show the differences in StO2 changes under different emotional states.

[0090] Optionally, in order to more directly characterize the StO2 change pattern of different emotional states, the significant t value in the t-chart can also be quantitatively displayed in 5 levels. For example, statistical significance means p value <0.05, and the corresponding critical t value can be obtained by looking up the table. The critical value is related to the significance level and the data sample size, that is, the smaller the sample size, the larger the critical t value to reach the significance level. When processing a certain emotional database, anger has the least amount of data. At this time, the critical |t| value corresponding to p<0.05 is 2.020. For ease of representation, the value of all emotional categories can be 2.1, that is, |t|<2.1 is not significant and is displayed in white. The five quantitative levels are named level1 to level5, and the level division rules are as follows: -8 to -5 is level 1, displayed in dark blue; -5 to -2.1 is level 2, displayed in light blue; -2.1 to 2.1 is an insignificant area, displayed in white; 2.1 to 5 is level 3, displayed in yellow; 5 to 8 is level 4, displayed in red; 8 to 10 is level 5, displayed in dark red. Among them, level 1 to 2 indicates low significance, and the lower the level, the more significant it is; level 3 to 5 indicates high significance, and the higher the level, the more significant it is; the white area in the middle indicates that the change is not significant.

[0091] Correspondingly, in all emotional states, the increase in facial StO2 is mostly at level 3 and level 4, with a smaller increase at level 5. When fearful, the decrease in facial StO2 is basically at level 2. It should be noted that the left and right sides of the face are mirror images. The specific descriptions of each emotion category are as follows:

[0092] (1) anger: There was no significant change around the eyes; except for a slight decrease in the lips, most areas of the face rose; the left cheek rose more significantly than the right cheek.

[0093] (2) Disgust: There is no significant change in the tip of the nose, but the forehead and nose area rise more significantly; the left side of the nose and mouth rises more than the right side.

[0094] (3) Fear: The pattern of change most easily distinguished from the other emotions; a significant drop in the nose area; a small, discrete drop in the forehead; a small, discrete rise below the left cheek.

[0095] (4) happiness: The increase in StO2 on the face is most widespread and significant, especially on the forehead, indicating that after being stimulated by happy emotions, the change in StO2 on the face of the person being tested is most obvious.

[0096] (5) Sadness: Most of the facial StO2 increases were level 3, with a small number of level 4 increases that were not concentrated. However, the difference from the pattern under emotion was that there was a more significant increase in StO2 around the eyes. During the experiment, when watching the sadness stimulus material, some of the subjects could not restrain themselves and shed tears. Although the data of these expression control failures have been eliminated, the increase in StO2 around the eyes under the sadness emotion may be a manifestation of the subjects wanting to shed tears after being stimulated by the stimulus material.

[0097] (6) Surprise: The facial StO2 around the mouth, especially in the philtrum, increases significantly, and the increase in level 4 is concentrated in this area.

[0098] exist Figure 3 In the illustrated embodiment, physiological signals of various emotional states can be processed accordingly, thereby effectively improving the effectiveness of target coding features.

[0099] Optionally, the target coding feature may include a plurality of different types of manual coding features, for example, one or more features such as intensity difference coding feature, histogram difference coding feature and mask difference coding feature.

[0100] See also Figure 4 , Figure 4 A detailed flow chart of step S320 is provided for an embodiment of the present application. Step S320 may include steps S321-S325.

[0101] Step S321, cluster analysis and subtraction processing are performed based on the baseline data and the sentiment data to obtain intensity difference coding features.

[0102] Among them, the intensity difference coding feature, namely IDP (intensity difference pattern), can be designed as the target coding feature for emotion recognition in the input deep learning model. Cluster analysis is performed on the data of the baseline data B before the emotion is generated in the standard physiological signal data and the emotional data D after the emotion is generated. Then, the clustered emotional data is subtracted from the clustered baseline data to obtain the intensity difference coding feature after quantizing the standard physiological signal data.

[0103] It should be noted that the intensity pattern IP is IP kIt is a general term for StO2 data after quantification, where k represents the number of StO2 levels. When performing cluster analysis, basic clustering algorithms such as k-means, mean shift clustering, density-based clustering, and manual level division can be used to classify StO2. It can fully consider the overall data situation, perform k-cluster clustering (k≥2) on the original paired B and D data, and retain the background area without operation.

[0104] Optionally, the cluster analysis process may include operations such as concatenation, reordering, and separation. Since the StO2 data has a uniform size, the numerical regions of the paired B and D data may be converted into vector forms V of the same dimension. B and V D The object of k-means is V B and V D The concatenated vector V. In order to ensure that the original StO2 values ​​correspond to the numbers after clustering, a reordering operation is required when using k-means to cluster the vector V. The specific reordering operation may include: the StO2 values ​​after k-means clustering are quantized to {1,2,…c,…,k}, and the corresponding cluster centroids are {μ1,μ2,…,μ c ,…,μ k}. Sort the centroids μ in ascending order and reassign category numbers 1 to k according to the sorted positions. For example, μ c The original expression is c, but μ c is the minimum value among all the centroids. After the reordering operation, the rank of the original expression c will be changed to 1. Therefore, the reordering operation can be expressed as: r = index(rank(μ)). Where μ represents the centroid of k-means; rank(*) represents the sorting from small to large (ascending order), index(*) represents the position index of μ in the sorting; r represents the new clustered expression that can represent the StO2 intensity after the reordering operation. Get the clustered StO2 intensity vector V k , decompose V by separation operation kB and V kD Finally, the StO2 moment IP is regenerated by combining the unprocessed background area (value is 0). The calculation of IDP is the subtraction of the paired IP of D and B. Considering that it needs to be fed into the neural network later, the overall data after IP subtraction plus the cluster number k can be adjusted to a positive value, that is: IDP = IP D -IP B +. Since IDP can have the same size as the original StO2 data, the background is 0 and the background area is not considered, IP k The value range is [1, k], (k ≥ 2, k ∈ N + ), a total of k levels. Corresponding to IDPK The value range is [1,2k-1], (k≥2,k∈N + ), are all positive integers. The background area in IDP is all 0. A value less than k indicates that the intensity of point D is lower than that of point B. Conversely, it indicates that the intensity of point D is greater than that of point B.

[0105] Optionally, in terms of the selection of the number of clusters for k-means, multiple numbers of clusters can be set for calculation and comparison, so as to determine the highest preferred number of clusters such as Acc (accuracy), UF1 (F1 score), and UAR (unweighted average recall). For example, 6 is taken as the optimal number of clusters, and the corresponding intensity difference coding feature is recorded as IDP6.

[0106] Or, step S322, performing histogram processing based on the intensity difference coding feature to obtain a histogram difference coding feature.

[0107] Among them, on the basis of the intensity difference coding feature, the histogram difference coding feature can be designed, that is, the intensity difference histogram pattern HIDP (histogram of intensity difference pattern) is used as the target coding feature for emotion recognition in the input machine learning model. By performing histogram processing on the intensity difference coding feature to obtain the corresponding histogram difference coding feature, the emotion can be recognized from the perspective of the histogram, which further improves the effectiveness of the target coding feature.

[0108] Optionally, corresponding HIDP6 features may be extracted based on IDP6.

[0109] Or, step S323, calculate facial difference value data based on the baseline data and the emotion data.

[0110] The facial difference value data is calculated with the initial baseline data and the emotional data, and is denoted as OD. The calculation method may be OD=D-B+100, where 100 is the value range of the original StO2, and other values ​​may also be used.

[0111] Step S324, performing scaling processing according to the intensity difference coding feature to obtain mask data.

[0112] The intensity difference encoding feature is scaled to obtain corresponding mask data. The specific scaling process used may include:

[0113]

[0114] Among them, mask(,j) is the calculated mask data, n is a positive number greater than or equal to 1, indicating the scaling ratio, and the scaling operation includes zooming in and out. When IDP is equal to k, it means that the StO2 level of D after stimulation is the same as that of B before stimulation, and there is no change, so the corresponding mask value point is 1; IDP less than k means that the StO2 level decreases after stimulation, and the corresponding mask value is also less than 1; IDP greater than k means that the StO2 level increases after stimulation, and the corresponding mask value is greater than 1.

[0115] Step S325, performing calculation based on the mask data and the facial difference value data to obtain a mask difference coding feature.

[0116] Among them, based on the intensity difference coding feature, a mask difference coding feature, that is, IMF (IDP-masked feature), can be designed as the target coding feature for emotion recognition in the input deep learning model. Therefore, the mask data and the facial difference value data are combined for calculation, and the mask data and the facial difference value data are multiplied, which can enlarge the high and low distribution of the values ​​in the high and low distribution of the values ​​in OD, and obtain the mask difference coding feature after mask processing, which further improves the effectiveness of the target coding feature.

[0117] Optionally, different scaling ratios and scaling methods may be selected to perform scaling processing on the intensity difference coding feature, and an addition method may be selected to calculate the intensity difference coding feature and the facial difference value data.

[0118] exist Figure 4 In the illustrated embodiment, a variety of different target coding feature extraction methods are provided, which can be applied to a variety of different usage scenarios.

[0119] Optionally, see Figure 5 , Figure 5 A detailed flow chart of step S200 is provided for an embodiment of the present application. Step S200 may include steps S210-S250.

[0120] Step S210, obtaining original facial images of the person under test before and after viewing the emotional material, which are collected based on spectral imaging technology.

[0121] Among them, in order to extract physiological signals, the original facial images of the person being tested before and after the test based on emotional materials can be collected based on spectral imaging technology. The emotional materials can include various types of emotional stimulation videos, pictures, etc. The original facial images include the first facial data before viewing the emotional material and the second facial data after viewing the emotional material.

[0122] Optionally, the emotional material may be an emotional video material, and the emotion may be triggered by watching the emotional video. In order to eliminate the influence of facial muscle movement on StO2 when the expression is generated, the person being tested may be required to strictly control facial expression during the video watching process, and multiple original facial images may be collected using the hyperspectral imaging technology HSI before and after watching the video. The face of the person being tested may be collected before and after watching the emotional material, respectively, to obtain first facial data representing the baseline state and second facial data representing the emotional state.

[0123] Step S220, obtaining the scoring data determined by the person being tested based on the emotional material.

[0124] In order to determine whether the image generates emotion, the person being tested can self-evaluate based on the emotional material after watching the video. The scoring data can include the emotional state and arousal score. By self-scoring the emotional state and arousal level of the video, various situations in the original facial image can be determined, for example, whether an expression is generated, the type of expression, and the arousal score.

[0125] Step S230, screening the original facial image based on preset screening rules and scoring data to obtain valid initial data.

[0126] Among them, multiple original facial images can be screened according to preset screening rules and scoring data. The screening rules can be: controlling facial expressions when watching videos, emotional states corresponding to the target emotions of the video, awakening scores (total score of 5 points) greater than 3 points, etc., in order to screen out valid initial data.

[0127] Step S240: extracting the initial data to obtain initial physiological signal data.

[0128] The initial physiological signal data may be facial tissue oxygen saturation data, namely StO2 data. For determining the effective initial data, the MBL3C model may be used to extract the original StO2 data from 33 bands of data between 516 and 580 nm with a step length of 2 nm as the initial physiological signal data.

[0129] Step S250: registering the initial physiological signal data to obtain standard physiological signal data.

[0130] The initial physiological signal data may also be subjected to facial registration to obtain standard physiological signal data of uniform size.

[0131] Optionally, since the original facial image includes a first facial data before viewing the emotional material and a second facial data after viewing the emotional material, the standard physiological signal data obtained after screening, extracting and aligning the original facial image includes baseline data before the emotion is generated and emotional data after the emotion is generated. The size of the standard physiological signal data can be a two-dimensional matrix of 481*411, with a numerical range of 0 to 100, and can be displayed in the form of a picture using pseudo-color.

[0132] It should be noted that, in order to facilitate subsequent processing, the standard physiological signal data can be integrated with the emotional expression to obtain an emotional database.

[0133] exist Figure 5 In the illustrated embodiment, the validity and size of the data can be screened and adjusted in the data preprocessing stage, thereby improving the validity and adaptability of the standard physiological signal data.

[0134] See also Figure 6 , Figure 6 A schematic diagram of the structure of a device for performing emotion recognition based on facial tissue oxygen saturation data provided in an embodiment of the present application, wherein the device 500 for performing emotion recognition based on facial tissue oxygen saturation data may include: a preprocessing module 510, a feature extraction module 520 and a recognition module 530;

[0135] The preprocessing module 510 is used to preprocess the collected initial data to obtain standard physiological signal data; wherein the standard physiological signal data is standard facial tissue oxygen saturation data;

[0136] The feature extraction module 520 is used to extract features from each standard physiological signal data to obtain target coding features;

[0137] The recognition module 530 is used to input the target coding features into the deep learning model for emotion recognition to obtain the emotion results.

[0138] In an optional implementation, the preprocessing module 510 may further include an acquisition submodule and a clustering submodule;

[0139] An acquisition submodule, used to acquire standard physiological signal data including baseline data before the emotion is generated and emotional data after the emotion is generated;

[0140] The clustering submodule is used to perform clustering analysis based on standard physiological signal data to obtain target coding features.

[0141] In an optional embodiment, the target coding feature includes: intensity difference coding feature; the clustering submodule is specifically used to: perform clustering analysis and subtraction processing based on baseline data and sentiment data to obtain intensity difference coding features.

[0142] In an optional implementation, the target coding feature includes: a histogram difference coding feature; the clustering submodule is further specifically used for: performing histogram processing based on the intensity difference coding feature to obtain the histogram difference coding feature.

[0143] In an optional embodiment, the target coding feature includes: mask difference coding feature; the clustering submodule is also specifically used to: calculate facial difference value data based on baseline data and emotion data; perform scaling processing according to the intensity difference coding feature to obtain mask data; perform calculation based on the mask data and facial difference value data to obtain mask difference coding feature.

[0144] In an optional embodiment, the preprocessing module 510 is specifically used to: obtain original facial images of the person under test before and after viewing the emotional material, which are collected based on spectral imaging technology; wherein the original facial images include first facial data before viewing the emotional material and second facial data after viewing the emotional material; obtain scoring data determined by the person under test based on the emotional material; filter the original facial images based on preset filtering rules and scoring data to obtain valid initial data; extract the initial data to obtain initial physiological signal data; and align the initial physiological signal data to obtain standard physiological signal data.

[0145] In an optional embodiment, the device 500 for emotion recognition based on facial tissue oxygen saturation data may also include a verification module for determining the emotion label corresponding to the standard physiological signal data; and verifying the emotion result according to the emotion label.

[0146] Since the principle of solving the problem by the device 500 for emotion recognition based on facial tissue oxygen saturation data in the embodiment of the present application is similar to that of the aforementioned embodiment of the method for emotion recognition based on facial tissue oxygen saturation data, the implementation of the device 500 for emotion recognition based on facial tissue oxygen saturation data in this embodiment can refer to the description in the embodiment of the method for emotion recognition based on facial tissue oxygen saturation data, and the repeated parts will not be repeated.

[0147] An embodiment of the present application also provides a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions are read and executed by a processor, the steps of any one of the methods for emotion recognition based on facial tissue oxygen saturation data provided in the present embodiment are executed.

[0148] In several embodiments provided in the present application, it should be understood that the disclosed device can also be implemented in other ways. The device embodiments described above are merely schematic, for example, the block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices according to the multiple embodiments of the present application. In this regard, each box in the block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram, and the combination of the block diagrams, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0149] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0150] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0151] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0152] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

[0153] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for emotion recognition based on facial tissue oxygen saturation data, characterized in that: The method comprises: Preprocessing the collected initial data to obtain standard physiological signal data; wherein the standard physiological signal data is standard facial tissue oxygen saturation data; Performing feature extraction on each of the standard physiological signal data to obtain target coding features; Inputting the target encoding features into a deep learning model for emotion recognition to obtain an emotion result; The feature extraction of each of the standard physiological signal data to obtain the target coding feature includes: obtaining the standard physiological signal data including baseline data before the emotion is generated and emotion data after the emotion is generated; performing cluster analysis based on the standard physiological signal data to obtain the target coding feature; Among them, the target coding features include: intensity difference coding features; the clustering analysis based on the standard physiological signal data to obtain the target coding features includes: performing clustering analysis based on the baseline data and the emotional data respectively, subtracting the clustered baseline data from the clustered emotional data to obtain the intensity difference coding features.

2. The method according to claim 1, characterized in that in, The target coding features include: histogram difference coding features; The cluster analysis is performed based on the standard physiological signal data to obtain the target coding feature, and further includes: Histogram processing is performed based on the intensity difference coding feature to obtain the histogram difference coding feature.

3. The method according to claim 1, characterized in that in, The target coding features include: mask difference coding features; The cluster analysis is performed based on the standard physiological signal data to obtain the target coding feature, and further includes: Based on the baseline data and the emotion data, facial difference value data is calculated, and the calculation method includes: OD = D - B + 100; Wherein, OD is the facial difference value data, D is the emotion data after the emotion is generated, B is the baseline data before the emotion is generated, and 100 is the numerical variation range of the original facial tissue oxygen saturation data; Scaling processing is performed according to the intensity difference coding feature to obtain mask data, and the scaling processing includes: Wherein, mask(i,j) is the mask data, n is a positive number greater than or equal to 1, indicating the scaling ratio, the scaling operation includes zooming in and out, and k indicates the number of levels of facial tissue oxygen saturation data; when the intensity difference coding feature is equal to k, it indicates that the level of facial tissue oxygen saturation data of the emotional data is the same as that of the baseline data, and there is no change, and the corresponding mask value point is 1; the intensity difference coding feature less than k indicates that the level of facial tissue oxygen saturation data of the emotional data decreases, and the corresponding mask value is less than 1; the intensity difference coding feature greater than k indicates that the level of facial tissue oxygen saturation data of the emotional data increases, and the corresponding mask value is greater than 1; Calculating based on the mask data and the facial difference value data to obtain the mask difference coding feature includes: multiplying the mask data and the facial difference value data to obtain the mask difference coding feature.

4. The method according to claim 1, characterized in that: The preprocessing of the collected initial data to obtain standard physiological signal data includes: Acquire original facial images of the person under test before and after viewing the emotional material, which are collected based on spectral imaging technology; wherein the original facial images include first facial data before viewing the emotional material and second facial data after viewing the emotional material; Obtaining scoring data determined by the tested person based on the emotional material; Screening the original facial image based on a preset screening rule and the scoring data to obtain valid initial data; Extracting the initial data to obtain initial physiological signal data; The initial physiological signal data is registered to obtain the standard physiological signal data.

5. The method according to claim 1, characterized in that The method further comprises: Determining an emotion label corresponding to the standard physiological signal data; The emotion result is verified according to the emotion label.

6. A device for emotion recognition based on facial tissue oxygen saturation data, characterized in that: The device comprises: a preprocessing module, a feature extraction module and a recognition module; The preprocessing module is used to preprocess the collected initial data to obtain standard physiological signal data; wherein the standard physiological signal data is standard facial tissue oxygen saturation data; The feature extraction module is used to extract features from each of the standard physiological signal data to obtain target coding features; The recognition module is used to input the target coding features into a deep learning model to perform emotion recognition and obtain an emotion result; The feature extraction module is specifically used to: obtain the standard physiological signal data including baseline data before the emotion is generated and emotional data after the emotion is generated; perform cluster analysis based on the standard physiological signal data to obtain the target coding feature; Among them, the target coding features include: intensity difference coding features; the feature extraction module is specifically used to: perform cluster analysis based on the baseline data and the emotional data respectively, subtract the clustered baseline data from the clustered emotional data, and obtain the intensity difference coding features.

7. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores program instructions, and when the processor runs the program instructions, the steps in the method according to any one of claims 1 to 5 are executed.

8. A computer-readable storage medium, characterized in that: The readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the steps in the method according to any one of claims 1 to 5 are executed.

Images