Emotion recognition method, device, storage medium, system and program product

Through the feature extraction and fusion of near-infrared brain imaging data and multiple state data, the problem of low accuracy of emotion recognition in the prior art is solved, and higher accuracy of emotion recognition is achieved.

CN118861785BActive Publication Date: 2024-12-03BEIJING YOUGUO TECH CO LTD

Patent Information

Application Number
CN202411328316.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-23
Publication Date
2024-12-03
Estimated Expiration
2044-09-23

AI Technical Summary

Technical Problem

The prior art is difficult to accurately identify complex human emotions in emotion recognition, and relying solely on a single data such as facial expressions, the accuracy rate is limited.

Method used

By obtaining the target user's near-infrared brain imaging data and status data, the brain imaging data of multiple channels is extracted, and the near-infrared brain imaging map is generated, and combined with speech, facial video and physiological signal data, feature extraction and fusion are performed, and inputted to the recognition model Perform emotion recognition.

Benefits of technology

It improves the accuracy of emotional recognition, uses brain hemooxygen dynamic characteristics and various state data to provide more essential and rich identification basis, and enhances the accuracy of recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118861785B_ABST
    Figure CN118861785B_ABST
Patent Text Reader

Abstract

The present disclosure provides an emotion recognition method, device, storage medium, system and program product, the emotion recognition method comprising: obtaining near-infrared brain imaging data and state data of a target user in a target period, wherein the state data is data other than the near-infrared brain imaging data used to reflect the current state of the corresponding user; using a near-infrared feature extraction model, performing data processing on the near-infrared brain imaging data to obtain near-infrared features; performing feature extraction processing on the state data to obtain state features; fusing the near-infrared features and the state features into comprehensive features; inputting the comprehensive features into a recognition model to obtain the emotion recognition result of the target user. The method can comprehensively improve the accuracy of emotion recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure generally relates to the field of data processing, and more specifically, to an emotion recognition method, device, storage medium, system and program product. Background Art

[0002] With the development of artificial intelligence technology, recognition technologies such as image recognition and semantic recognition are becoming more mature, but it is still difficult to accurately recognize the complex content of human emotions. Existing technologies usually use single data such as facial expressions to recognize emotions, which can only realize the recognition of basic emotions such as joy, anger, sadness, and happiness. In reality, human emotions are a complex concept. Therefore, relying solely on single data to recognize people's actual complex emotions has very limited accuracy. Summary of the invention

[0003] The present disclosure provides an emotion recognition method, device, storage medium, system and program product to solve at least one of the above problems.

[0004] According to one aspect of the present disclosure, there is provided an emotion recognition method, the emotion recognition method comprising: acquiring near-infrared brain imaging data and state data of a target user in a target time period, wherein the near-infrared brain imaging data comprises brain imaging data of multiple channels, and the state data is data other than the near-infrared brain imaging data for reflecting the current state of the corresponding user; grouping the brain imaging data of the multiple channels according to the corresponding brain regions to obtain brain imaging data of each brain region; generating a near-infrared brain imaging map based on the brain imaging data of the multiple brain regions, wherein the near-infrared brain imaging map is used to display the brain imaging data of the multiple brain regions in partitioned manner; inputting the near-infrared brain imaging map into a near-infrared feature extraction model to obtain near-infrared features; performing feature extraction processing on the state data to obtain state features; fusing the near-infrared features and the state features into comprehensive features; and inputting the comprehensive features into a recognition model to obtain an emotion recognition result of the target user.

[0005] Optionally, the brain imaging data is time domain data, wherein the generating of a near-infrared brain imaging atlas based on the brain imaging data of multiple brain regions includes: for each brain region, performing fitting processing on the brain imaging data of each channel of the brain region according to the channel dimension to obtain a brain imaging time domain curve of the brain region; performing Fourier transform on the brain imaging data of each channel of the brain region respectively to obtain brain imaging frequency domain data of each channel of the brain region; performing fitting processing on the brain imaging frequency domain data of each channel of the brain region according to the channel dimension to obtain a brain imaging frequency domain curve of the brain region; generating the near-infrared brain imaging atlas based on the brain imaging time domain curves and brain imaging frequency domain curves of multiple brain regions.

[0006] Optionally, generating the near-infrared brain imaging atlas based on brain imaging time domain curves and brain imaging frequency domain curves of multiple brain regions includes: splicing the brain imaging time domain curves and brain imaging frequency domain curves of multiple brain regions according to preset rules to obtain a brain imaging splicing map; preprocessing and visualizing the brain imaging splicing map to obtain a near-infrared brain imaging atlas.

[0007] Optionally, the status data includes different types of status data, wherein performing feature extraction processing on the status data to obtain status features includes: performing corresponding feature extraction processing on each type of status data to obtain features of each type of status data; and taking the features of various types of status data together as the status features.

[0008] Optionally, the state data includes at least one of the following: behavioral data and physiological signal data; the behavioral data includes at least one of the following: voice data and facial video data; the physiological signal data includes at least one of the following: heart rate, respiratory rate, and blood pressure; wherein the corresponding feature extraction processing is performed on each type of state data to obtain the features of each type of state data, including at least one of the following steps: for the voice data, extracting at least one voice feature value of the voice data as a voice feature, wherein the at least one voice feature value includes at least one of the following: kurtosis, spectral contrast, approximate open entropy, center of gravity frequency, amplitude micro-fluctuation, and harmonic noise ratio; for the facial video data, inputting the facial video data into a facial expression recognition model to obtain a plurality of basic emotion indicators as facial features, wherein the basic emotion indicators are used to reflect the possibility that the target user has the corresponding basic emotions, and the basic emotions include at least one of the following: happiness, sadness, anger, disgust, fear, surprise, and neutrality; for the physiological signal data, determining a stability index of the physiological signal data as a physiological feature, wherein the stability index is used to indicate the degree to which the corresponding physiological signal data deviates from a stable state.

[0009] Optionally, the near-infrared feature extraction model and the recognition model are trained by the following steps: obtaining sample near-infrared brain imaging data, sample state data and sample emotion information of a sample user in a sample time period; grouping brain imaging data of multiple channels in the sample near-infrared brain imaging data according to the corresponding brain regions to obtain sample brain imaging data of each brain region; generating a sample near-infrared brain imaging atlas based on the sample brain imaging data of multiple brain regions; inputting the sample near-infrared brain imaging atlas into the near-infrared feature extraction model to be trained to obtain sample near-infrared features; performing the feature extraction processing on the sample state data to obtain sample state features; fusing the sample near-infrared features and the sample state features into sample comprehensive features; inputting the sample comprehensive features into the recognition model to be trained to obtain the emotion recognition result of the sample user; determining the value of the loss function based on the emotion recognition result and the sample emotion information; adjusting the parameters of the near-infrared feature extraction model to be trained and the recognition model to be trained based on the value of the loss function to obtain the near-infrared feature extraction model and the recognition model.

[0010] According to another aspect of the present disclosure, an emotion recognition device is provided, the emotion recognition device comprising: an acquisition unit, configured to acquire near-infrared brain imaging data and state data of a target user in a target time period, wherein the near-infrared brain imaging data comprises brain imaging data of multiple channels, and the state data is data other than the near-infrared brain imaging data for reflecting the current state of the corresponding user; a grouping unit, configured to group the brain imaging data of the multiple channels according to the corresponding brain regions to obtain brain imaging data of each brain region; a generation unit, configured to generate a near-infrared brain imaging map based on the brain imaging data of multiple brain regions, wherein the near-infrared brain imaging map is used to display the brain imaging data of the multiple brain regions in partitioned manner; a first extraction unit, configured to input the near-infrared brain imaging map into a near-infrared feature extraction model to obtain near-infrared features; a second extraction unit, configured to perform feature extraction processing on the state data to obtain state features; a fusion unit, configured to fuse the near-infrared features and the state features into comprehensive features; and a recognition unit, configured to input the comprehensive features into a recognition model to obtain an emotion recognition result of the target user.

[0011] Optionally, the brain imaging data is time domain data, and the generating unit is further configured to: for each brain region, perform fitting processing on the brain imaging data of each channel of the brain region according to the channel dimension to obtain the brain imaging time domain curve of the brain region; perform Fourier transform on the brain imaging data of each channel of the brain region respectively to obtain the brain imaging frequency domain data of each channel of the brain region; perform fitting processing on the brain imaging frequency domain data of each channel of the brain region according to the channel dimension to obtain the brain imaging frequency domain curve of the brain region; generate the near-infrared brain imaging atlas according to the brain imaging time domain curves and brain imaging frequency domain curves of multiple brain regions.

[0012] Optionally, the generating unit is further configured to: splice the brain imaging time domain curves and the brain imaging frequency domain curves of multiple brain regions according to preset rules to obtain a brain imaging splicing map; preprocess and visualize the brain imaging splicing map to obtain a near-infrared brain imaging atlas.

[0013] Optionally, the status data includes different types of status data, and the second extraction unit is further configured to: perform corresponding feature extraction processing on each type of status data to obtain features of each type of status data; and use the features of various types of status data together as the status features.

[0014] Optionally, the state data includes at least one of the following: behavioral data and physiological signal data, the behavioral data includes at least one of the following: voice data and facial video data, the physiological signal data includes at least one of the following: heart rate, respiratory rate, blood pressure, and the second extraction unit is further configured to perform at least one of the following steps: for the voice data, extracting at least one voice feature value of the voice data as a voice feature, wherein the at least one voice feature value includes at least one of the following: kurtosis, spectral contrast, approximate open entropy, center of gravity frequency, amplitude micro-fluctuation, harmonic noise ratio; for the facial video data, inputting the facial video data into a facial expression recognition model to obtain a plurality of basic emotion indicators as facial features, wherein the basic emotion indicators are used to reflect the possibility that the target user has the corresponding basic emotions, and the basic emotions include at least one of the following: happiness, sadness, anger, disgust, fear, surprise, and neutrality; for the physiological signal data, determining a stability index of the physiological signal data as a physiological feature, wherein the stability index is used to indicate the degree to which the corresponding physiological signal data deviates from a stable state.

[0015] Optionally, the near-infrared feature extraction model and the recognition model are trained by the following steps: obtaining sample near-infrared brain imaging data, sample state data and sample emotion information of a sample user in a sample time period; grouping brain imaging data of multiple channels in the sample near-infrared brain imaging data according to the corresponding brain regions to obtain sample brain imaging data of each brain region; generating a sample near-infrared brain imaging atlas based on the sample brain imaging data of multiple brain regions; inputting the sample near-infrared brain imaging atlas into the near-infrared feature extraction model to be trained to obtain sample near-infrared features; performing the feature extraction processing on the sample state data to obtain sample state features; fusing the sample near-infrared features and the sample state features into sample comprehensive features; inputting the sample comprehensive features into the recognition model to be trained to obtain the emotion recognition result of the sample user; determining the value of the loss function based on the emotion recognition result and the sample emotion information; adjusting the parameters of the near-infrared feature extraction model to be trained and the recognition model to be trained based on the value of the loss function to obtain the near-infrared feature extraction model and the recognition model.

[0016] According to another aspect of the present disclosure, a computer-readable storage medium storing instructions is provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the emotion recognition method as described above.

[0017] According to another aspect of the present disclosure, a system is provided, including at least one computing device and at least one storage device storing instructions, wherein when the instructions are executed by the at least one computing device, the at least one computing device is prompted to perform the emotion recognition method as described above.

[0018] According to another aspect of the present disclosure, a computer program product is provided, comprising instructions, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the emotion recognition method as described above.

[0019] According to the emotion recognition method, device, storage medium, system and program product of the exemplary embodiment of the present disclosure, by simultaneously using near-infrared brain imaging data and state data to extract features respectively, and fusing them to obtain comprehensive features, and then using the recognition model to process the comprehensive features, the target user's emotions can be recognized, and rich information can be used to improve the accuracy of emotion recognition. Specifically, emotions are the product of the operation of the human brain. By using near-infrared brain imaging data that can reflect the characteristics of brain oxygenation dynamics, it is possible to see through superficial information such as facial expressions and use more essential brain operation information as the basis for recognition, which helps to ensure recognition accuracy. At the same time, with the help of other state data, the accuracy of emotion recognition can be further improved.

[0020] Additional aspects and / or advantages of the present general inventive concept will be set forth in part in the following description and in part will be apparent from the description or may be learned through practice of the present general inventive concept. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] These and / or other aspects and advantages of the present disclosure will become clear and more easily understood from the following description of the embodiments in conjunction with the accompanying drawings.

[0022] Figure 1 is a flowchart illustrating an emotion recognition method according to an exemplary embodiment of the present disclosure.

[0023] Figure 2 is a time domain graph showing oxygenated hemoglobin and deoxygenated hemoglobin in nine brain regions according to an exemplary embodiment of the present disclosure.

[0024] Figure 3 is a frequency domain graph showing oxygenated hemoglobin and deoxygenated hemoglobin in nine brain regions according to an exemplary embodiment of the present disclosure.

[0025] Figure 4 is a diagram illustrating a brain imaging mosaic according to an exemplary embodiment of the present disclosure.

[0026] Figure 5 is a diagram showing a near-infrared brain imaging atlas according to an exemplary embodiment of the present disclosure.

[0027] Figure 6 It is a flowchart showing an emotion recognition method according to a specific embodiment of the present disclosure.

[0028] Figure 7 is a block diagram illustrating an emotion recognition apparatus according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the embodiments of the present invention as defined by the claims and their equivalents. Various specific details are included to assist in understanding, but these details are considered to be exemplary only. Therefore, one of ordinary skill in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. In addition, descriptions of well-known functions and structures are omitted for clarity and brevity.

[0030] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three types of parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three types of parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0031] Refer to the following Figures 1 to 7 The emotion recognition method, apparatus, storage medium, system, and program product according to exemplary embodiments of the present disclosure are described in detail.

[0032] Figure 1 is a flow chart showing an emotion recognition method according to an exemplary embodiment of the present disclosure. The emotion recognition method according to an exemplary embodiment of the present disclosure can be implemented in a computing device with sufficient computing power.

[0033] Reference Figure 1 In step S101, near-infrared brain imaging data and state data of a target user in a target period of time are obtained.

[0034] The target user is the object of emotion recognition, that is, the user whose emotions need to be recognized. The target period is the period targeted by emotion recognition, and accordingly, the near-infrared brain imaging data and state data are the data generated by the target user during the target period, so as to obtain clear data.

[0035] Near-infrared brain imaging data is obtained through near-infrared brain imaging technology (Functional near-infrared spectroscopy, referred to as fNIRS), which is a spectral imaging technology that continuously irradiates the human brain scalp with near-infrared light of 650nm-850nm wavelength, so that the near-infrared light passes through the scalp and skull, enters the cerebral cortex at a depth of 2cm-3cm, forms a propagation arc, and is detected by another detector on the scalp. During the propagation of near-infrared light in human tissue, the absorption of photons by tissue has a certain regularity, especially the oxygenated hemoglobin and deoxygenated hemoglobin in the capillaries in the cerebral cortex. Due to this particularity, the concentration change of oxygenated hemoglobin and deoxygenated hemoglobin in the irradiation interval can be inferred from the absorption rate obtained on the detector. When the human brain is working, nerve cells discharge continuously, which consumes a lot of energy. And all this energy is provided by the oxygen carried by the capillaries around the nerve cells. Therefore, when the brain is activated, its supply of oxygenated hemoglobin will increase, thereby accelerating the oxygen supply and reducing the concentration of deoxygenated hemoglobin. After activation, energy is converted, the concentration of oxygenated hemoglobin decreases, the concentration of deoxygenated hemoglobin increases, and the cerebral cortex is in an oxygen-demanding state again, waiting for the next oxygen supply. This cycle repeats itself, and this is the neurovascular coupling mechanism in neuroscience.

[0036] The prefrontal lobe of the human brain is responsible for various high-level emotional activities and regulation. Current studies have shown that emotions originate from the limbic system deep in the brain, and the regulation of emotions is a very complex process, which is organically regulated by various parts of the prefrontal lobe. Therefore, studying the hemodynamic feedback mechanism of the prefrontal lobe can provide insights into an individual's response and regulation ability to different emotions. Using near-infrared brain imaging technology to act on the prefrontal lobe, the cerebral cortex can be activated through task states (including emotional stimulation tasks, language fluency tasks, resting state tasks, etc., where the resting state task refers to the subject remaining still and relaxed without performing specific cognitive tasks). By observing the changes in oxygenated hemoglobin and deoxygenated hemoglobin, the individual's emotional characteristics can be obtained, thereby establishing an individual emotion regulation model.

[0037] It should be understood that near-infrared brain imaging data needs to be collected with corresponding equipment, such as a frontal headgear and a detector for collecting data. It should also be understood that in order to ensure that the near-infrared brain imaging data of the target period is fully acquired, data collection can be started before the target period begins, and collection can be stopped a period of time after the target period ends, and then the data corresponding to the target period can be intercepted. In other words, it is limited here that the near-infrared brain imaging data generated by the target user during the target period is used in the emotion recognition process, but there is no restriction on how to collect and obtain the data.

[0038] The state data is data other than the near-infrared brain imaging data that is used to reflect the current state of the corresponding user, so that richer data can be used as the data basis for emotion recognition. The state content only needs to reflect the state of the user. The present disclosure does not limit its specific form. More exemplary explanations will be given later, which will not be expanded here. It should be understood that the rules regarding whether the corresponding user needs to complete the specified task and whether the data collection time needs to be extended are the same as the collection rules of near-infrared brain imaging data, and will not be repeated.

[0039] In step S102, a near-infrared feature extraction model is used to perform data processing on the near-infrared brain imaging data to obtain near-infrared features.

[0040] As an example, the near-infrared feature extraction model includes a convolutional neural network and a machine learning model.

[0041] In step S103, feature extraction processing is performed on the state data to obtain state features.

[0042] Here, according to the specific form of the status data, a suitable feature extraction processing method can be adopted, which will be introduced later when the status data is exemplified and will not be expanded here.

[0043] In step S104, the near infrared feature and the state feature are fused into a comprehensive feature.

[0044] In step S105, the comprehensive features are input into the recognition model to obtain the emotion recognition result of the target user.

[0045] It should be understood that different forms of emotion recognition results can be configured according to the different purposes of emotion recognition. As an example, a comprehensive indicator can be used to reflect the positivity of emotions. The higher the value of the comprehensive indicator, the more positive the target user's emotions are; multiple emotion indicators can also be used to respectively represent the possibility of the target user being in different emotions; a large language model can also be combined to generate a text that interprets the user's emotions; multiple indicators of different dimensions can also be used to interpret the target user's emotions from different dimensions, such as instantaneous emotions, long-term emotions (such as the overall comprehensive emotions within the target period), and emotional self-regulation ability. Accordingly, depending on the emotion recognition results, a reasonable method can be used to output the emotion recognition results. For example, for the interpretation of text, the corresponding text or voice can be output; for multiple indicators, a radar chart can be output. The present disclosure does not list them one by one here.

[0046] According to the emotion recognition method, device, storage medium, system and program product of the exemplary embodiment of the present disclosure, by simultaneously using near-infrared brain imaging data and state data to extract features respectively, and fusing them to obtain comprehensive features, and then using the recognition model to process the comprehensive features, the target user's emotions can be recognized, and rich information can be used to improve the accuracy of emotion recognition. Specifically, emotions are the product of the operation of the human brain. By using near-infrared brain imaging data that can reflect the characteristics of brain oxygenation dynamics, it is possible to see through superficial information such as facial expressions and use more essential brain operation information as the basis for recognition, which helps to ensure recognition accuracy. At the same time, with the help of other state data, the accuracy of emotion recognition can be further improved.

[0047] It should be understood that, in order to facilitate the description of the content of each step, each step is marked with a number, but the number is not used to limit the execution order of each step. Under reasonable circumstances, the execution order of the above steps can be adjusted. For example, step S102 and step S103 can be executed simultaneously or sequentially, and when executing sequentially, step S102 can be executed first, or step S103 can be executed first. This disclosure does not limit this.

[0048] Next, we will further introduce near-infrared brain imaging data and its feature extraction.

[0049] Optionally, the near-infrared brain imaging data includes brain imaging data of multiple channels, and step S102 includes: grouping the brain imaging data of multiple channels according to the corresponding brain regions to obtain brain imaging data of each brain region; generating a near-infrared brain imaging atlas based on the brain imaging data of the multiple brain regions, wherein the near-infrared brain imaging atlas is used to display the brain imaging data of the multiple brain regions in partitions; and inputting the near-infrared brain imaging atlas into a near-infrared feature extraction model to obtain near-infrared features.

[0050] By generating a near-infrared brain imaging atlas that centrally displays brain imaging data of multiple brain regions as the basis for extracting near-infrared features, it is possible to include rich and comprehensive brain information in one atlas, which helps the extracted near-infrared features express the operational correlation between different brain regions, and provides a richer and more reliable basis for emotion recognition. In addition, by establishing a task-state near-infrared brain imaging atlas, a unified, unambiguous, comprehensive two-dimensional result can be established under different devices, different near-infrared instruments, different numbers of channels, and different tasks. The result can reflect the blood oxygen dynamics feedback pattern of the irradiated area of ​​the individual's cerebral cortex, and is suitable for processing, identifying, and classifying the atlas using tools such as convolutional neural networks. As an example, brain imaging data of multiple brain regions can be arranged according to specific arrangement rules, so that the position of each brain imaging data in the atlas can represent its corresponding brain region, so that the operational correlation between each brain region can be learned when training the near-infrared feature extraction model.

[0051] Further optionally, the brain imaging data is time domain data. In this case, the step of generating a near-infrared brain imaging atlas based on the brain imaging data of multiple brain regions includes the following steps.

[0052] Step 1: For each brain region, the brain imaging data of each channel of the brain region are fitted according to the channel dimension to obtain the brain imaging time domain curve of the brain region. This step is to fit the data of different channels belonging to the same brain region together to obtain the brain imaging time domain curve of the brain region, thereby reasonably compressing the data and helping to reduce the amount of data.

[0053] As an example, during the fitting process, the average value of the data of each channel at each sampling time can be calculated by averaging, and the average value of multiple sampling times can be obtained, and then a curve of the average value changing over time can be drawn as a brain imaging time domain curve. It should be understood that when the brain imaging data includes at least two items of data, at least two time domain curves can be obtained accordingly. For example Figure 2 The figure shows the time domain curves of oxygenated hemoglobin (oxyHb) and deoxyhemoglobin (deoxyHb) in nine brain regions of a certain user. The nine brain regions are Left Dorsolateral Prefrontal Cortex (lDLPFC), Left Superior Frontal Cortex (lSFC), Left Superior Temporal Cortex (lSTC), Left Ventrolateral Prefrontal Cortex (lVLPFC), Medial Prefrontal Cortex (mPFC), Right Dorsolateral Prefrontal Cortex (rDLPFC), Right Superior Frontal Cortex (rSFC), Right Superior Temporal Cortex (lSTC), Left Ventrolateral Prefrontal Cortex (lVLPFC), Medial Prefrontal Cortex (mPFC), Right Dorsolateral Prefrontal Cortex (rDLPFC), Right Superior Frontal Cortex (rSFC), Right Superior Temporal Cortex (rSTC), and Left Ventrolateral Prefrontal Cortex (lVLPFC). It should be understood that the above nine brain regions are only examples and are not limitations on the multiple brain regions of the near-infrared brain imaging data of this application. The multiple brain regions of the exemplary embodiments of the present disclosure can be selected according to actual needs.

[0054] Step 2: Perform Fourier transform on the brain imaging data of each channel of the brain region to obtain the brain imaging frequency domain data of each channel of the brain region. By performing Fourier transform (for example, using the Pwelch method) to achieve data conversion from the time domain to the frequency domain, the brain imaging frequency domain data can be obtained, thereby extracting more information.

[0055] Step three, fit the brain imaging frequency domain data of each channel of the brain region according to the channel dimension to obtain the brain imaging frequency domain curve of the brain region. This step is also for data fitting in the same brain region, but the fitted data is the converted brain imaging frequency domain data. The fitting method is the same as the step, the difference is that the fitted value of the data of each channel at each sampling frequency within a given frequency range (for example, 0.009Hz to 1Hz) is calculated. For example Figure 3 Shown are the frequency domain curves of oxygenated hemoglobin (oxyHb) and deoxyhemoglobin (deoxyHb) in the above nine brain regions of a certain user.

[0056] Step 4: Generate a near-infrared brain imaging atlas based on the brain imaging time domain curves and brain imaging frequency domain curves of multiple brain regions. This step realizes the integration of the time and frequency domain curves of each brain region.

[0057] It should be understood that Figure 1 The execution order of each step can be adjusted under reasonable circumstances. Here, for the convenience of describing each step separately, names such as step one and step two are used, but it does not limit the execution order. Specifically, step one realizes the fitting of time domain data, steps two and three realize the fitting of frequency domain data, and step four generates a spectrum based on the fitted time domain curve and frequency domain curve. Therefore, step four needs to be executed after steps one to three, and step three needs to be executed after step two. However, step one has no execution order requirement relative to steps two and three.

[0058] Regarding step 4, in some embodiments, optionally, the above step 4 includes: splicing brain imaging time domain curves and brain imaging frequency domain curves of multiple brain regions according to preset rules to obtain a brain imaging splicing map, which has realized the centralized display of brain imaging data of multiple brain regions, and the brain imaging splicing map can be used as a near-infrared brain imaging atlas. The use of preset rules can ensure that the obtained brain imaging splicing map has a unified standard. As an example, the preset rule may be that the vertical axis is nine brain regions arranged in a specified order from top to bottom in the screen coordinate system, and the curve of oxygenated hemoglobin (oxyHb) in each brain region is on the top and the curve of deoxyhemoglobin (deoxyHb) is on the bottom, with a total of 18 different regions; the size of the vertical axis is equal to the size of the horizontal axis, and the horizontal axis is divided into two equal regions, the left half region is the time domain region, and the right half region is the frequency domain region; each region of the vertical axis is initially 32 pixels, totaling 18×32=576 pixels, and each region of the horizontal axis is 288 pixels, totaling 2×288=576 pixels, so the size of the overall brain imaging mosaic is 576×576, for example Figure 4 shown.

[0059] Regarding step 4, in some other embodiments, optionally, the above step 4 includes: splicing brain imaging time domain curves and brain imaging frequency domain curves of multiple brain regions according to preset rules to obtain, for example Figure 4 The brain imaging mosaic is shown in the figure; the brain imaging mosaic is preprocessed and visualized to obtain a near-infrared brain imaging atlas. Considering that the brain imaging mosaic shows various curves, it reflects more the shape characteristics of the curves. By further preprocessing and visualizing the brain imaging mosaic to color the corresponding points according to the values ​​of each point in the curve, the numerical differences of each point can be better reflected, which helps to extract more features reflecting brain operation information, thereby improving the accuracy of emotion recognition.

[0060] Specifically, preprocessing is to ensure the effect of visualization processing. As an example, preprocessing may include normalizing and interpolating each curve in the brain imaging mosaic, normalizing each curve to a uniform length (e.g., 288). The time domain curve can be obtained by dividing the total time length by a uniform length (e.g., 288) to obtain a sampling period and a number of corresponding sampling moments, and interpolating at each sampling moment to obtain 18 sets of sampled blood oxygen concentration data with a length of 288 (unit: mmol / mm·Hg), and taking its first differential (the error ratio between the latter sampling and the previous sampling). The above normalization, interpolation, and first differential of the two hemoglobins can be expressed by calculation formulas as S ample (T(t) oxy )' and S ample (T(t) deoxy)'. The frequency domain curve can be directly interpolated to obtain data (unit power dB) of uniform length (for example, 288 sampling frequencies). The above interpolation processing of the two hemoglobins can be expressed by the calculation formula P ample (F(w) oxy ) and P ample (F(w) deoxy ).

[0061] Regarding visualization processing, as an example, for each sampling point in the preprocessed brain imaging mosaic, the ordinate value of the sampling point is used as the value of the point, and the value of the point without a sampling point is 0, thereby converting the entire brain imaging mosaic into a digital matrix, and then using the topographic map generation method to draw the digital matrix, and using the Jet palette to color each point in the digital matrix to achieve visualization processing, as shown below Figure 5 Near-infrared brain imaging atlas shown.

[0062] It should be noted that, regarding the step of generating a near-infrared brain imaging atlas based on brain imaging data of multiple brain regions, in addition to adopting the embodiment including the above steps one to four, only steps one and four can be performed. In this case, step four specifically generates a near-infrared brain imaging atlas based on the brain imaging time domain curves of multiple brain regions, and similar to the embodiment including the above steps one to four, step four at this time can splice the brain imaging time domain curves of multiple brain regions into a brain imaging splicing map, directly as a near-infrared brain imaging atlas, or obtain a near-infrared brain imaging atlas after preprocessing and visualization. These are all implementations of the present disclosure and fall within the protection scope of the present disclosure.

[0063] In general, by constructing a near-infrared brain imaging atlas, we can establish a unified, unambiguous, two-dimensional presentation of hemoglobin concentration data collected in multiple brain regions by any near-infrared brain imaging instrument, which can comprehensively reflect the patterns and characteristics of blood oxygen dynamics feedback in multiple brain regions of an individual, making it easier to use convolutional neural networks for unified recognition.

[0064] Next, the state data and its feature extraction are introduced.

[0065] In some embodiments, optionally, the state data includes different types of state data. Accordingly, step S103 includes: performing corresponding feature extraction processing on each type of state data to obtain features of each type of state data; and taking the features of various types of state data together as state features. By using a variety of different types of state data, the amount of information can be fully improved, which helps to improve the accuracy of emotion recognition. Of course, by reasonably selecting state data, it is possible to take into account both the accuracy of emotion recognition and the consumption of computing resources, thereby improving recognition efficiency. It should be understood that the features of different types of state data are taken together as state features, which means that in step S104, the near-infrared features and the features of different types of state data will be fused. At this time, the features should be fused according to the specified rules. For example, in the case where each feature is a one-dimensional feature, the features can be directly spliced ​​into a comprehensive feature in the specified order.

[0066] Further optionally, the state data includes at least one of the following: behavioral data, physiological signal data, the behavioral data includes at least one of the following: voice data, facial video data, and the physiological signal data includes at least one of the following: heart rate, respiratory rate, blood pressure. By using at least one of the above behavioral data, information can be obtained from the user's speaking status, facial expressions and other behaviors, and by using at least one of the above physiological signal data, information can be obtained from other physiological reactions outside the brain, both of which are helpful in identifying the user's current emotions. In addition, the changes in hemoglobin concentration reflected in the near-infrared brain imaging data can reflect the user's long-term emotions and emotional self-regulation ability, and the behavioral data and physiological signal data can reflect the user's instantaneous emotions at the moment each data is generated. Combining these two aspects of data, the user's emotions can be identified and interpreted from different perspectives.

[0067] Correspondingly, the above-mentioned step of performing corresponding feature extraction processing on each type of state data to obtain the features of each type of state data includes at least one of the following steps: for voice data, extracting at least one voice feature value of the voice data as a voice feature, wherein the at least one voice feature value includes at least one of the following: kurtosis, spectral contrast, approximate open entropy, center of gravity frequency, amplitude micro-fluctuation, harmonic noise ratio, directly calculating the above voice feature value, can understand the fluency, voice, speaking speed, intonation and other characteristics of the user during the speech process, thereby reflecting the user's emotions; for facial video data, inputting the facial video data into a facial expression recognition model to obtain multiple basic emotion indicators as facial features, wherein the basic emotion indicators are used to reflect the possibility that the target user has the corresponding basic emotions, and the basic emotions include at least one of the following: happiness, sadness, anger, disgust, fear, surprise, neutrality, and by interpreting the basic emotions contained in the facial expressions, a more reliable basis can be provided for the recognition of complex emotions; for physiological signal data, determining the stability index of the physiological signal data as a physiological feature, wherein the stability index is used to indicate the degree to which the corresponding physiological signal data deviates from the stable state, thereby reflecting the intensity of the user's emotions, and can also provide an effective reference for emotion recognition.

[0068] In other embodiments, optionally, the status data is a single type of data, such as any of the above-mentioned behavioral data, or any of the above-mentioned physiological signal data. Of course, it can also be other types of data, as long as it is related to the user's emotions and can help identify the user's emotions. The present disclosure does not impose any restrictions on this.

[0069] Next, we will introduce the training method of the model used in the emotion recognition process.

[0070] Optionally, the near-infrared feature extraction model and the recognition model are trained by the following steps: obtaining sample near-infrared brain imaging data, sample state data and sample emotion information of the sample user in the sample time period; grouping the brain imaging data of multiple channels in the sample near-infrared brain imaging data according to the corresponding brain regions to obtain sample brain imaging data of each brain region; generating a sample near-infrared brain imaging atlas based on the sample brain imaging data of multiple brain regions; inputting the sample near-infrared brain imaging atlas into the near-infrared feature extraction model to be trained to obtain sample near-infrared features; performing feature extraction processing on the sample state data to obtain sample state features; fusing the sample near-infrared features and the sample state features into sample comprehensive features; inputting the sample comprehensive features into the recognition model to be trained to obtain the emotion recognition results of the sample user; determining the value of the loss function based on the emotion recognition results and the sample emotion information; adjusting the parameters of the near-infrared feature extraction model to be trained and the recognition model to be trained based on the value of the loss function to obtain the near-infrared feature extraction model and the recognition model. By obtaining sample near-infrared brain imaging data and sample state data as input, and sample emotional information as output reference, the near-infrared feature extraction model and the recognition model are uniformly and centrally trained, which can reduce the difficulty of sample labeling in training, that is, there is no need to label samples for the output data of the near-infrared feature extraction model, and it helps to improve the correlation between the trained near-infrared feature extraction model and the recognition model. It should be understood that for the specific implementation embodiments of each step introduced above, the same processing is also performed in the stage of training the model to ensure that the training process of the model corresponds to the reasoning process of the model. As an example, the sample emotional information can be manually labeled by personnel with professional knowledge.

[0071] Furthermore, in the case where a model is also used when performing feature extraction processing in step S103, such as the case where the above-mentioned facial expression recognition model is used, since the state data is often simpler in data volume and more mature in processing methods than near-infrared brain imaging data, such models can be pre-trained in advance, and then the pre-trained model can be directly used to perform feature extraction processing in the process of training the near-infrared feature extraction model and the recognition model, which can effectively simplify the scale of the centralized training model and reduce the difficulty of training while meeting feasibility. When pre-training it, image frames with different expressions can be used as samples, and the basic emotions corresponding to each image frame can be manually marked. Accordingly, in the process of using the pre-trained facial expression recognition model for reasoning and generating multiple basic emotion indicators, the score of each basic emotion can be generated for each image frame, and then based on the score of each basic emotion in all image frames, the score of each basic emotion of the entire video can be obtained as each basic emotion indicator.

[0072] Next, the emotion recognition method of the present disclosure is introduced in conjunction with a specific embodiment.

[0073] Reference Figure 6 This specific embodiment collects and processes three modalities (near infrared brain imaging, facial video, and voice) data at the same time. It is mainly divided into three steps: 1. Data collection and processing; 2. Feature extraction; 3. Multimodal feature fusion and classification.

[0074] 1. Data collection and processing.

[0075] Near-infrared brain imaging data: The collected full-channel hemoglobin concentration data is fitted and Fourier transformed, and the time domain data and frequency domain data of different channels corresponding to each brain region are superimposed, spliced, preprocessed, and visualized, and finally converted into a near-infrared brain imaging atlas. Half of the horizontal axis of the atlas is time, corresponding to the target period (such as the task state time period, and the resting state cutting segment), and the other half is frequency. The vertical axis is a uniformly defined standard ROI (Region of Interest) brain area, which can unambiguously reflect the cerebral cortex blood oxygen dynamics feedback pattern.

[0076] Facial video data: The rasterized data matrix of the facial video of the entire target period is processed by frame division. Through a sampling rate of less than 10Hz, a face capture algorithm (such as the Haar operator) is used for each frame, and the face ROI is determined by convolution calculation.

[0077] Voice data: Use microphones and other sound pickup devices to collect voice data, and perform basic preprocessing such as filtering and noise reduction.

[0078] 2. Feature extraction.

[0079] Near-infrared features: Use multiple convolutional layers in the near-infrared feature extraction model to extract local features of the near-infrared imaging map, capture the temporal / spatial features of the near-infrared imaging map through supervised training, and then use the pooling layer (such as maximum pooling or average pooling) in the near-infrared feature extraction model to reduce the dimension of the features, reduce the amount of calculation and retain important features, and obtain a 1×n near-infrared feature vector.

[0080] Facial features: Use multiple convolutional layers in the facial expression recognition model to extract facial features from facial video data. Use the pooling layer in the facial expression recognition model to reduce the dimension of the features and obtain seven basic emotion indexes (all in %), including happiness index, sadness index, anger index, disgust index, fear index, surprise index, and neutral index, as a 1×7 facial feature vector.

[0081] Speech features: For the speech data after basic preprocessing, a set of eigenvalues ​​in the speech data are extracted by calculation or using a comprehensive audio feature extraction model. They are kurtosis (KUR), spectral contrast (Constrast), approximate open entropy (QOQ), center of gravity frequency (COG), amplitude slight fluctuation (Shimmer), and harmonic noise ratio (hnr-0), as a 1×6 speech feature vector.

[0082] 3. Multimodal feature fusion.

[0083] The data feature vectors of different modes obtained previously are concatenated together to form a comprehensive feature vector.

[0084] 4. Classification or regression.

[0085] The comprehensive feature vector is processed using a recognition model. For example, the concatenated comprehensive feature vector can be further machine-learned using a fully connected layer, and then input into a comprehensive linear classifier of the fully connected layer for emotion classification. Finally, the Softmax or Sigmoid function is used to output the emotion category or emotion intensity as the emotion recognition result.

[0086] Figure 7 is a block diagram illustrating an emotion recognition apparatus according to an exemplary embodiment of the present disclosure.

[0087] Reference Figure 7 The emotion recognition device 700 includes an acquisition unit 701, a grouping unit 702, a generation unit 703, a first extraction unit 704, a second extraction unit 705, a fusion unit 706, and a recognition unit 707.

[0088] The acquisition unit 701 is configured to acquire near-infrared brain imaging data and status data of the target user in a target time period, wherein the near-infrared brain imaging data includes brain imaging data of multiple channels, and the status data is data other than the near-infrared brain imaging data used to reflect the current status of the corresponding user.

[0089] The grouping unit 702 is configured to group the brain imaging data of the multiple channels according to the corresponding brain regions to obtain brain imaging data of each brain region.

[0090] The generating unit 703 is configured to generate a near-infrared brain imaging atlas according to the brain imaging data of the multiple brain regions, wherein the near-infrared brain imaging atlas is used to display the brain imaging data of the multiple brain regions in partitions.

[0091] The first extraction unit 704 is configured to input the near-infrared brain imaging atlas into the near-infrared feature extraction model to obtain the near-infrared features.

[0092] The second extraction unit 705 is configured to perform feature extraction processing on the state data to obtain state features.

[0093] The fusion unit 706 is configured to fuse the near infrared feature and the state feature into a comprehensive feature.

[0094] The recognition unit 707 is configured to input the comprehensive features into the recognition model to obtain the emotion recognition result of the target user.

[0095] Optionally, the brain imaging data is time domain data, and the generating unit 703 is further configured to: for each brain region, perform fitting processing on the brain imaging data of each channel of the brain region according to the channel dimension to obtain the brain imaging time domain curve of the brain region; perform Fourier transform on the brain imaging data of each channel of the brain region respectively to obtain the brain imaging frequency domain data of each channel of the brain region; perform fitting processing on the brain imaging frequency domain data of each channel of the brain region according to the channel dimension to obtain the brain imaging frequency domain curve of the brain region; generate a near-infrared brain imaging atlas based on the brain imaging time domain curves and brain imaging frequency domain curves of multiple brain regions.

[0096] Optionally, the generating unit 703 is further configured to: splice the brain imaging time domain curves and the brain imaging frequency domain curves of multiple brain regions according to preset rules to obtain a brain imaging splice map; preprocess and visualize the brain imaging splice map to obtain a near-infrared brain imaging atlas.

[0097] Optionally, the state data includes different types of state data, and the second extraction unit 705 is further configured to: perform corresponding feature extraction processing on each type of state data to obtain features of each type of state data; and use the features of various types of state data together as state features.

[0098] Optionally, the state data includes at least one of the following: behavioral data and physiological signal data, the behavioral data includes at least one of the following: voice data and facial video data, the physiological signal data includes at least one of the following: heart rate, respiratory rate, and blood pressure, and the second extraction unit 705 is further configured to perform at least one of the following steps: for voice data, extract at least one voice feature value of the voice data as a voice feature, wherein the at least one voice feature value includes at least one of the following: kurtosis, spectral contrast, approximate open entropy, center of gravity frequency, small amplitude fluctuations, and harmonic noise ratio; for facial video data, input the facial video data into a facial expression recognition model to obtain multiple basic emotion indicators as facial features, wherein the basic emotion indicators are used to reflect the possibility that the target user has corresponding basic emotions, and the basic emotions include at least one of the following: happiness, sadness, anger, disgust, fear, surprise, and neutrality; for physiological signal data, determine a stability index of the physiological signal data as a physiological feature, wherein the stability index is used to indicate the degree to which the corresponding physiological signal data deviates from a stable state.

[0099] Optionally, the near-infrared feature extraction model and the recognition model are trained by the following steps: obtaining sample near-infrared brain imaging data, sample state data and sample emotion information of the sample user in the sample time period; grouping the brain imaging data of multiple channels in the sample near-infrared brain imaging data according to the corresponding brain regions to obtain sample brain imaging data of each brain region; generating a sample near-infrared brain imaging atlas based on the sample brain imaging data of multiple brain regions; inputting the sample near-infrared brain imaging atlas into the near-infrared feature extraction model to be trained to obtain sample near-infrared features; performing feature extraction processing on the sample state data to obtain sample state features; fusing the sample near-infrared features and the sample state features into sample comprehensive features; inputting the sample comprehensive features into the recognition model to be trained to obtain the emotion recognition results of the sample user; determining the value of the loss function based on the emotion recognition results and the sample emotion information; adjusting the parameters of the near-infrared feature extraction model to be trained and the recognition model to be trained based on the value of the loss function to obtain the near-infrared feature extraction model and the recognition model.

[0100] The above has been referred to Figures 1 to 7 An emotion recognition method and apparatus according to exemplary embodiments of the present disclosure are described.

[0101] Figure 7 Each unit in the illustrated emotion recognition device may be configured as software, hardware, firmware, or any combination of the above items to perform a specific function. For example, each unit may correspond to a dedicated integrated circuit, or may correspond to a pure software code, or may correspond to a module that combines software and hardware. In addition, one or more functions implemented by each unit may also be uniformly executed by components in a physical entity device (e.g., a processor, a client, or a server, etc.).

[0102] In addition, refer to Figure 1 The described emotion recognition method can be implemented by a program (or instruction) recorded on a computer-readable storage medium. For example, according to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions can be provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the emotion recognition method according to the present disclosure.

[0103] The computer program in the computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. It should be noted that the computer program can also be used to perform additional steps in addition to the above steps or perform more specific processing when performing the above steps. The contents of these additional steps and further processing have been described in reference to Figure 1 It is mentioned in the description of the related method, so it will not be repeated here to avoid repetition.

[0104] It should be noted that the various units in the emotion recognition device according to the exemplary embodiment of the present disclosure can completely rely on the operation of the computer program to realize the corresponding functions, that is, each unit corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (for example, lib library) to realize the corresponding function.

[0105] on the other hand, Figure 7 The various units shown may also be implemented by hardware, software, firmware, middleware, microcode, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segment for performing the corresponding operation may be stored in a computer-readable medium such as a storage medium, so that the processor may perform the corresponding operation by reading and running the corresponding program code or code segment.

[0106] For example, an exemplary embodiment of the present disclosure may also be implemented as a computing device, which includes a storage component and a processor, wherein a set of computer executable instructions is stored in the storage component, and when the set of computer executable instructions is executed by the processor, the emotion recognition method according to the exemplary embodiment of the present disclosure is executed.

[0107] Specifically, the computing device can be deployed in a server or client, or can be deployed on a node device in a distributed network environment. In addition, the computing device can be a PC, a tablet device, a personal digital assistant, a smart phone, a web emotion recognition, or other device capable of executing the above instruction set.

[0108] Here, the computing device is not necessarily a single computing device, but may also be any device or circuit collection that can execute the above instructions (or instruction sets) individually or jointly. The computing device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0109] In a computing device, a processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, a processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0110] According to the exemplary embodiment of the present disclosure, some operations described in the emotion recognition method can be implemented by software, some operations can be implemented by hardware, and in addition, these operations can also be implemented by a combination of software and hardware.

[0111] The processor may execute instructions or codes stored in one of the storage components, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0112] The storage component may be integrated with the processor, for example, RAM or flash memory is arranged within an integrated circuit microprocessor, etc. In addition, the storage component may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The storage component and the processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the storage component.

[0113] In addition, the computing device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the computing device may be connected to each other via a bus and / or a network.

[0114] The emotion recognition method according to the exemplary embodiment of the present disclosure can be described as various interconnected or coupled functional blocks or functional diagrams. However, these functional blocks or functional diagrams can be equally integrated into a single logical device or operate according to non-precise boundaries.

[0115] Therefore, refer to Figure 1 The described emotion recognition method may be implemented by a system including at least one computing device and at least one storage device storing instructions.

[0116] According to an exemplary embodiment of the present disclosure, at least one computing device is a computing device for executing an emotion recognition method according to an exemplary embodiment of the present disclosure, and a computer executable instruction set is stored in the storage device. When the computer executable instruction set is executed by the at least one computing device, the execution reference Figure 1 The described emotion recognition method.

[0117] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, comprising instructions, which, when executed by at least one computing device, prompt the at least one computing device to perform the emotion recognition method according to the present disclosure.

[0118] The above describes various exemplary embodiments of the present disclosure, and it should be understood that the above description is only exemplary and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present disclosure, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present disclosure should be based on the scope of the claims.

Claims

1. A method for emotion recognition, It is characterized in that The emotion recognition method comprises: Acquire near-infrared brain imaging data and state data of a target user in a target period of time, wherein the near-infrared brain imaging data includes brain imaging data of multiple channels, and the state data is data other than the near-infrared brain imaging data used to reflect the current state of the corresponding user; Grouping the brain imaging data of the multiple channels according to the corresponding brain regions to obtain brain imaging data of each brain region; Generate a near-infrared brain imaging atlas according to the brain imaging data of the multiple brain regions, wherein the near-infrared brain imaging atlas is used to display the brain imaging data of the multiple brain regions in a partitioned manner; Inputting the near-infrared brain imaging atlas into a near-infrared feature extraction model to obtain near-infrared features; Performing feature extraction processing on the state data to obtain state features; fusing the near infrared feature and the state feature into a comprehensive feature; Inputting the comprehensive features into a recognition model to obtain an emotion recognition result of the target user; The brain imaging data is time domain data, wherein generating a near-infrared brain imaging atlas based on the brain imaging data of multiple brain regions includes: For each brain region, performing fitting processing on the brain imaging data of each channel of the brain region according to the channel dimension to obtain a brain imaging time domain curve of the brain region; Performing Fourier transform on the brain imaging data of each channel of the brain region respectively to obtain brain imaging frequency domain data of each channel of the brain region; Performing fitting processing on the brain imaging frequency domain data of each channel of the brain region according to the channel dimension to obtain a brain imaging frequency domain curve of the brain region; Splicing brain imaging time domain curves and brain imaging frequency domain curves of multiple brain regions according to preset rules to obtain a brain imaging splicing image; The brain imaging mosaic is preprocessed and visualized to obtain a near-infrared brain imaging atlas.

2. The emotion recognition method according to claim 1, It is characterized in that The state data includes different types of state data, wherein the performing feature extraction processing on the state data to obtain state features includes: Performing corresponding feature extraction processing on each type of state data to obtain features of each type of state data; The features of various types of status data are collectively regarded as the status feature.

3. The emotion recognition method according to claim 2, It is characterized in that The state data includes at least one of the following: behavior data and physiological signal data, the behavior data includes at least one of the following: voice data and facial video data, and the physiological signal data includes at least one of the following: heart rate, respiratory rate, and blood pressure, wherein performing corresponding feature extraction processing on each type of state data to obtain the features of each type of state data includes at least one of the following steps: Extracting at least one speech feature value of the speech data as a speech feature from the speech data, wherein the at least one speech feature value includes at least one of the following: kurtosis, spectrum contrast, approximate open entropy, centroid frequency, amplitude micro-fluctuation, and harmonic-to-noise ratio; Inputting the facial video data into a facial expression recognition model to obtain a plurality of basic emotion indicators as facial features, wherein the basic emotion indicators are used to reflect the possibility that the target user has a corresponding basic emotion, and the basic emotion includes at least one of the following: happiness, sadness, anger, disgust, fear, surprise, and neutrality; For the physiological signal data, a stability index of the physiological signal data is determined as a physiological feature, wherein the stability index is used to indicate the degree to which the corresponding physiological signal data deviates from a stable state.

4. The emotion recognition method according to any one of claims 1 to 3, It is characterized in that The near infrared feature extraction model and the recognition model are trained by the following steps: Obtaining sample near-infrared brain imaging data, sample state data, and sample emotion information of a sample user during a sample period; Grouping brain imaging data of multiple channels in the sample near-infrared brain imaging data according to corresponding brain regions to obtain sample brain imaging data of each brain region; Generate a sample near-infrared brain imaging atlas based on sample brain imaging data of multiple brain regions; Inputting the sample near-infrared brain imaging atlas into the near-infrared feature extraction model to be trained to obtain the sample near-infrared features; Performing feature extraction processing on the sample state data to obtain sample state features; fusing the sample near-infrared feature and the sample state feature into a sample comprehensive feature; Inputting the sample comprehensive features into the recognition model to be trained to obtain the emotion recognition result of the sample user; Determining a value of a loss function according to the emotion recognition result and the sample emotion information; Based on the value of the loss function, the parameters of the near-infrared feature extraction model to be trained and the recognition model to be trained are adjusted to obtain the near-infrared feature extraction model and the recognition model.

5. An emotion recognition device, It is characterized in that The emotion recognition device comprises: an acquisition unit configured to acquire near-infrared brain imaging data and state data of a target user in a target period of time, wherein the near-infrared brain imaging data includes brain imaging data of multiple channels, and the state data is data other than the near-infrared brain imaging data for reflecting a current state of the corresponding user; a grouping unit configured to group the brain imaging data of the multiple channels according to the corresponding brain regions to obtain brain imaging data of each brain region; A generating unit is configured to generate a near-infrared brain imaging atlas according to the brain imaging data of the plurality of brain regions, wherein the near-infrared brain imaging atlas is used to display the brain imaging data of the plurality of brain regions in a partitioned manner; A first extraction unit is configured to input the near-infrared brain imaging atlas into a near-infrared feature extraction model to obtain near-infrared features; A second extraction unit is configured to perform feature extraction processing on the state data to obtain state features; a fusion unit configured to fuse the near infrared feature and the state feature into a comprehensive feature; A recognition unit configured to input the comprehensive features into a recognition model to obtain an emotion recognition result of the target user; The brain imaging data is time domain data, and the generating unit is further configured to: For each brain region, performing fitting processing on the brain imaging data of each channel of the brain region according to the channel dimension to obtain a brain imaging time domain curve of the brain region; Performing Fourier transform on the brain imaging data of each channel of the brain region respectively to obtain brain imaging frequency domain data of each channel of the brain region; Performing fitting processing on the brain imaging frequency domain data of each channel of the brain region according to the channel dimension to obtain a brain imaging frequency domain curve of the brain region; Splicing brain imaging time domain curves and brain imaging frequency domain curves of multiple brain regions according to preset rules to obtain a brain imaging splicing image; The brain imaging mosaic is preprocessed and visualized to obtain a near-infrared brain imaging atlas.

6. A computer-readable storage medium storing instructions, It is characterized in that When the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the emotion recognition method according to any one of claims 1 to 4.

7. A system comprising at least one computing device and at least one storage device storing instructions, It is characterized in that When the instructions are executed by the at least one computing device, the at least one computing device is prompted to perform the emotion recognition method according to any one of claims 1 to 4.

8. A computer program product comprising instructions, It is characterized in that When the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the emotion recognition method according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • Electroencephalogram time frequency information visualizing method

    CN110251117A

  • Motor imagery electroencephalogram decoding method based on space-frequency image feature extraction and classification

    CN115828059A

  • FNIRS emotion recognition method and system based on probability distribution labels and brain region features

    CN117582227A

  • Emotion recognition method and device, chip, electronic equipment and medium

    CN117770816A

  • Use of EEG to measure cerebral changes during computer-based motion sickness-inducing tasks

    US20060241373A1

Cited By

  • Method, device, storage medium, system and program product for generating psychoanalysis report

    CN119785965A