Emotion recognition method, emotion recognition system, electronic equipment and storage medium
By collecting brainwave signals in an immersive virtual reality environment and using deep learning models for feature extraction and classification, the complex problem of brainwave signal preprocessing is solved, and the reliability and accuracy of emotion recognition are improved.
Patent Information
- Application Number
- CN202310538462.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-12
AI Technical Summary
In the prior art, the preprocessing process of brain wave signals is complicated and cumbersome, resulting in low reliability and accuracy of emotion recognition and requires professional knowledge to perform feature extraction.
The virtual reality headset is used to collect brain wave signals in an immersive virtual reality environment, and through the feature extraction module and emotion classification module in the deep learning model, the complex preprocessing process is avoided, and the spatiotemporal feature vectors are directly extracted for emotion classification.
It improves the reliability and accuracy of emotion classification, simplifies the brain wave signal processing process, and reduces the dependence on professional knowledge.
Smart Images

Figure CN118940099A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to an emotion recognition method, an emotion recognition system, an electronic device, and a storage medium. Background Art
[0002] With the development of human-computer interaction technology and affective computing in neuromarketing research, more and more research works are devoted to studying methods to measure customers' emotional responses.
[0003] Affective computing technology in related technologies uses non-physiological signals such as facial expressions, voice, and gestures to detect and classify emotions. However, since subjects may suppress or fake their true emotions, it is impossible to reliably detect true emotional reactions from non-physiological signals.
[0004] Electrophysiological signals such as brain waves are difficult to manipulate and can provide reliable information to accurately detect and classify customers' emotional responses. However, due to the contamination of various non-EEG signals, which are often called "false signals", EEG signals have a low signal-to-noise ratio. Before using EEG signals for specific applications, they need to be pre-processed such as filtering to reduce the impact of artificial interference and noise; and feature extraction of brain wave signals often requires expertise in specific fields.
[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0006] The disclosed embodiments provide an emotion recognition system, an emotion recognition system, an electronic device and a storage medium. The method avoids the complicated and tedious preprocessing process of brain wave signals and improves the reliability and accuracy of emotion classification.
[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.
[0008] The disclosed embodiment provides an emotion recognition method, comprising: obtaining brain wave signals generated by a target object when viewing product information using a virtual reality head mounted device in an immersive virtual reality environment; performing feature extraction on the brain wave signals based on a feature extraction module in an emotion recognition model to obtain a spatiotemporal feature vector of the brain wave signals; and classifying the spatiotemporal feature vector based on an emotion classification module in the emotion recognition model to obtain the emotional state of the target object with respect to the product information.
[0009] In some exemplary embodiments of the present disclosure, the feature extraction module includes a first layer of convolutional network, a second layer of convolutional network and a third layer of convolutional network; wherein, based on the feature extraction module in the emotion recognition model, feature extraction is performed on the brain wave signal to obtain the spatiotemporal feature vector of the brain wave signal, including: inputting the brain wave signal into the first layer of convolutional network for time domain feature extraction to obtain the time domain feature vector of the brain wave signal; inputting the time domain feature vector of the brain wave signal into the second layer of convolutional network for spatial feature extraction to obtain the spatial feature vector of the brain wave signal; inputting the time domain feature vector and the spatial feature vector of the brain wave signal into the third layer of convolutional network for fusion processing to obtain the spatiotemporal feature vector of the brain wave signal.
[0010] In some exemplary embodiments of the present disclosure, the brain wave signal includes multiple time series signals collected respectively under multiple channels; wherein, the brain wave signal is input into the first layer of convolutional network for time domain feature extraction to obtain the time domain feature vector of the brain wave signal, including: inputting the multiple time series signals into the first layer of convolutional network for time domain feature extraction to obtain the time domain feature vector corresponding to each channel; wherein, the time domain feature vector of the brain wave signal is input into the second layer of convolutional network for spatial feature extraction to obtain the spatial feature vector of the brain wave signal, including: inputting the time domain feature vectors corresponding to all channels into the second layer of convolutional network for cross-channel feature extraction to obtain the spatial feature vector of the brain wave signal.
[0011] In some exemplary embodiments of the present disclosure, the emotion classification module is a deep learning model based on a gated recurrent unit (GRU).
[0012] In some exemplary embodiments of the present disclosure, obtaining brain wave signals generated by a target object when viewing product information in an immersive virtual reality environment using a virtual reality head-mounted device includes: displaying product information to the target object in the immersive virtual reality environment through the virtual reality head-mounted device; and collecting brain wave signals generated by the target object when viewing the product information in the immersive virtual reality environment through the virtual reality head-mounted device.
[0013] In some exemplary embodiments of the present disclosure, the emotion recognition model is trained in the following manner: obtaining brain wave signals generated by the test subjects when viewing product information in an immersive virtual reality environment, and obtaining the emotional state labels of the test subjects for the product information; performing feature extraction on the brain wave signals based on the feature extraction module in the emotion recognition model to be trained, and obtaining the spatiotemporal feature vectors of the brain wave signals; classifying the spatiotemporal feature vectors based on the emotion classification module in the emotion recognition model to be trained, and obtaining the predicted emotional state of the test subjects for the product information; training the emotion recognition model to be trained according to the emotional state labels and the predicted emotional state, and obtaining a trained emotion recognition model.
[0014] In some exemplary embodiments of the present disclosure, obtaining the emotional state label of the test subject with respect to the product information includes: obtaining the valence score and the arousal score of the test subject with respect to the product information after viewing the product information; and determining the emotional state label of the test subject with respect to the product information based on the valence score and the arousal score.
[0015] The disclosed embodiment provides an emotion recognition system, comprising: a virtual reality head mounted device and an emotion recognition model; wherein the virtual reality head mounted device is used to obtain brain wave signals generated by a target object when viewing product information in an immersive virtual reality environment; the emotion recognition model is used to extract features of the brain wave signals based on a feature extraction module in the emotion recognition model to obtain a spatiotemporal feature vector of the brain wave signals; and classify the spatiotemporal feature vector based on an emotion classification module in the emotion recognition model to obtain an emotional state of the target object with respect to the product information.
[0016] An embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a storage device for storing at least one program, wherein when the at least one program is executed by the at least one processor, the at least one processor implements any one of the above-mentioned emotion recognition methods.
[0017] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above-mentioned emotion recognition methods.
[0018] The emotion recognition method provided by the embodiment of the present disclosure obtains the brain wave signal generated by the target object when viewing product information in an immersive virtual reality environment. The immersive virtual reality environment provides an immersive user experience, making the collected brain wave signal more accurate; based on the feature extraction module in the emotion recognition model, the feature of the brain wave signal is extracted to obtain the spatiotemporal feature vector of the brain wave signal; based on the emotion classification module in the emotion recognition model, the spatiotemporal feature vector is classified to obtain the emotional state of the target object for the product information; the method adopts a deep learning model, avoids the complex and tedious preprocessing process of the brain wave signal, and improves the reliability and accuracy of emotion classification.
[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0021] Figure 1 is a flow chart of an emotion recognition method in an exemplary embodiment of the present disclosure.
[0022] Figure 2 is a schematic diagram of the structure of an emotion recognition model according to an example.
[0023] Figure 3 is a schematic diagram of a set of brain waves in an exemplary embodiment of the present disclosure.
[0024] Figure 4 4 is a flowchart of emotion recognition model training in an exemplary embodiment of the present disclosure.
[0025] Figure 5 is a schematic diagram showing a comparison of emotion classification accuracy according to an example.
[0026] Figure 6 is a block diagram of an emotion recognition system in an exemplary embodiment of the present disclosure.
[0027] Figure 7 is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION
[0028] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.
[0029] The features, structures or characteristics described in the present disclosure may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.
[0030] The accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some block diagrams shown in the accompanying drawings do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in at least one hardware module or integrated circuit, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0031] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and steps, nor must they be executed in the order described. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may change according to actual conditions.
[0032] In addition, in the description of the present disclosure, the terms "a", "an", "the", "said" and "at least one" are used to indicate the presence of at least one element or component; the terms "comprising", "including" and "having" are used to express an open-ended inclusion and mean that additional elements or components may exist in addition to the listed elements or components; the terms "first", "second" and "third" etc. are used only as labels and are not intended to limit the quantity of their objects.
[0033] Below, each step of the emotion recognition method in the exemplary embodiment of the present disclosure will be described in more detail with reference to the accompanying drawings and embodiments.
[0034] Figure 1is a flow chart of an emotion recognition method in an exemplary embodiment of the present disclosure. The method provided in the embodiment of the present disclosure can be executed by any electronic device with computing and processing capabilities, but the present disclosure is not limited thereto.
[0035] like Figure 1 As shown, the emotion recognition method provided by the embodiment of the present disclosure may include the following steps.
[0036] In step S102, brain wave signals generated when the target object uses a virtual reality head mounted device to view product information in an immersive virtual reality environment are obtained.
[0037] In the disclosed embodiments, the electroencephalogram (EEG) signal generated by the target object when facing specific stimulation in an immersive virtual reality (VR) environment using a virtual reality head mounted device may be collected, wherein the specific stimulation may be, for example, specific product information.
[0038] In the disclosed embodiment, the target object may be a user who wants to purchase a product, and the product information may include but is not limited to the product's 3D model, brand and price information, etc.; the user views the product information in a virtual reality environment, which allows the user to view and interact with the product information from different angles.
[0039] In the disclosed embodiment, the brain wave signals generated by the user when viewing product information can be collected to identify the user's emotional state.
[0040] In an exemplary embodiment, obtaining brain wave signals generated by a target object when viewing product information in an immersive virtual reality environment using a virtual reality head-mounted device includes: displaying product information to the target object in the immersive virtual reality environment through the virtual reality head-mounted device; and collecting brain wave signals generated by the target object when viewing product information in the immersive virtual reality environment through the virtual reality head-mounted device.
[0041] In the embodiments of the present disclosure, a virtual reality head mounted device refers to a device that can be worn on the head of a target object and display information in a virtual reality environment. The virtual reality head mounted device may be, for example, a VR headset (which may be referred to as a VR head display); the target object may wear the virtual reality head mounted device, and the virtual reality head mounted device displays product information to the target object in an immersive virtual reality environment, and the virtual reality head mounted device collects brain wave signals generated by the target object when the target object views the product information.
[0042] In the disclosed embodiment, the brain wave signal collected by the virtual reality head mounted device may also be referred to as a raw brain wave signal, which is a time series signal (S) collected at a preset sampling frequency (T) and a preset number of channels (C). For example, the sampling frequency may be 128 Hz and the number of channels may be 14.
[0043] In step S104, a feature extraction module in an emotion recognition model based on a deep learning framework performs feature extraction on the brain wave signal to obtain a spatiotemporal feature vector of the brain wave signal.
[0044] In the embodiment of the present disclosure, the emotion recognition model based on the deep learning framework may be a trained emotion recognition model. The training process of the emotion recognition model may refer to Figure 4 The present disclosure is not limited to the embodiments shown.
[0045] In the embodiments of the present disclosure, reference Figure 2 The emotion recognition model may include a feature extraction module 201 and an emotion classification module 202; the original brain wave signal (T×C×S) collected by the virtual reality head mounted device may be input into the feature extraction module 201 to extract time domain features and spatial features to obtain the spatiotemporal feature vector of the brain wave signal.
[0046] In the exemplary embodiment, continue to refer to Figure 2 , the feature extraction module 201 may include a first layer of convolutional network, a second layer of convolutional network and a third layer of convolutional network; wherein, the feature extraction module in the emotion recognition model based on the deep learning framework performs feature extraction on the brain wave signal to obtain the spatiotemporal feature vector of the brain wave signal, which may include: inputting the brain wave signal into the first layer of convolutional network for time domain feature extraction to obtain the time domain feature vector of the brain wave signal; inputting the time domain feature vector of the brain wave signal into the second layer of convolutional network for spatial feature extraction to obtain the spatial feature vector of the brain wave signal; inputting the time domain feature vector and the spatial feature vector of the brain wave signal into the third layer of convolutional network for fusion processing to obtain the spatiotemporal feature vector of the brain wave signal.
[0047] Specifically, the first layer of convolutional network can be composed of a two-dimensional convolutional neural network (2D-CNN); in the first layer of convolutional network, a one-dimensional filter (1D filter, also called a time filter) can be used to extract time domain features from the original brain wave signal, and the spatial features of the original brain wave signal are retained. The time domain feature vector extracted by the first layer of convolutional network is used as the input of the second layer of convolutional network; compared with other network structures, the two-dimensional convolutional neural network with a one-dimensional filter used in the embodiment of the present disclosure can effectively extract the time domain features of the original brain wave signal.
[0048] Specifically, the second layer of convolutional network can be composed of a two-dimensional Depthconv2D (deep 2D convolution) filter, which is used to learn the spatial features of the EEG signals of each channel; the time domain feature vector extracted by the first layer of convolutional network (2D-CNN) is used as the input of the second convolutional layer (DepthConv2D), and a deep filter (also called a spatial filter) with a kernel size of (1, C) is used to learn the deep features (also called spatial features) of each channel; compared with other network structures, the Depthconv2D used in the embodiment of the present disclosure can better capture spatial or channel-related features.
[0049] Specifically, the third convolutional network can be composed of a one-dimensional CNN, whose main task is to integrate the time domain feature vector and the spatial feature vector to reduce the dimension of the generated spatiotemporal feature vector.
[0050] In an exemplary embodiment, the brain wave signal may include a time series signal of each channel collected separately under multiple channels; wherein, the brain wave signal is input into the first layer of convolutional network for time domain feature extraction to obtain the time domain feature vector of the brain wave signal, including: inputting multiple time series signals into the first layer of convolutional network for time domain feature extraction to obtain the time domain feature vector corresponding to each channel; wherein, the time domain feature vector of the brain wave signal is input into the second layer of convolutional network for spatial feature extraction to obtain the spatial feature vector of the brain wave signal, including: inputting the time domain feature vectors corresponding to all channels into the second layer of convolutional network for cross-channel feature extraction to obtain the spatial feature vector of the brain wave signal.
[0051] Specifically, refer to Figure 2 and Figure 3 The first layer of convolutional network uses temporal filters to extract the features of EEG signals in the temporal dimension, and the second layer of convolutional network uses spatial (depth) filters to extract the features of EEG signals in the spatial (channel) dimension.
[0052] In step S106, the spatiotemporal feature vector is classified based on the emotion classification module in the emotion recognition model to obtain the target object's emotional state with respect to the product information.
[0053] In the disclosed embodiments, emotional state refers to the emotion caused by stimulation (such as viewing product information), and the emotional state can be determined from two dimensions: valence and arousal, that is, valence value and arousal value can be used as the basis for emotion classification; the emotional state of the target object can be, for example, the first emotion category, the second emotion category, etc., and the emotional state can include but is not limited to "happiness", "sadness", "disgust", "surprise", etc., and each emotional state can correspond to a set of valence value ranges and / or arousal value ranges.
[0054] In an exemplary embodiment, reference Figure 2 , the emotion classification module 202 can be a deep learning model based on a gated recurrent unit (GRU).
[0055] Specifically, the spatiotemporal feature vectors obtained through the above steps can be input into multiple GRU units for processing to obtain the emotional state of the target object with respect to the product information.
[0056] The emotion recognition method provided by the embodiment of the present disclosure obtains the brain wave signal generated by the target object when viewing product information in an immersive virtual reality environment. The immersive virtual reality environment provides an immersive user experience, making the collected brain wave signal more accurate; based on the feature extraction module in the emotion recognition model, the feature of the brain wave signal is extracted to obtain the spatiotemporal feature vector of the brain wave signal; based on the emotion classification module in the emotion recognition model, the spatiotemporal feature vector is classified to obtain the emotional state of the target object for the product information; the method adopts a deep learning model, avoids the complex and tedious preprocessing process of the brain wave signal, and improves the reliability and accuracy of emotion classification.
[0057] In addition, the emotion recognition method provided by the embodiment of the present disclosure can be applied to the field of shopping; when a user browses a product, the emotion recognition method provided by the embodiment of the present disclosure is used to obtain the user's emotional state, and then the user's purchasing preference is predicted by combining the user's historical purchasing behavior, product rating data and the user's emotional state.
[0058] In addition, the emotion recognition method provided by the embodiment of the present disclosure can be applied to the real estate field; a user can browse real estate information in a virtual reality environment, use the emotion recognition method provided by the embodiment of the present disclosure to obtain the user's emotional state, and predict the user's preferences based on the user's emotional state.
[0059] In addition, the emotion recognition method provided by the embodiments of the present disclosure can also be applied to emotion perception recommendation systems, such as implemented in furniture showrooms and museums.
[0060] Figure 4 4 is a flowchart of emotion recognition model training in an exemplary embodiment of the present disclosure.
[0061] like Figure 4 As shown, the emotion recognition model provided by the embodiment of the present disclosure can be obtained through the following steps: Training.
[0062] In step S402, the brain wave signals generated by the test subject when viewing the product information in the immersive virtual reality environment are obtained, and the emotional state label of the test subject in response to the product information is obtained.
[0063] In the disclosed embodiment, the test subject refers to a user who participates in collecting training data of the model. The test subject may be one or more. The test subject and the target subject may be the same user or different users.
[0064] In the disclosed embodiment, during the process of collecting training data for the model, the test subject can wear a virtual reality head-mounted device, product information can be displayed to the test subject in an immersive virtual reality environment, and the brain wave signals generated by the test subject when viewing the product information can be collected through the virtual reality head-mounted device.
[0065] In the disclosed embodiment, the product information displayed to the test subjects in the process of collecting training data for the model and the product information displayed to the test subjects in the process of applying data for the model may be information about the same product or information about different products.
[0066] In an exemplary embodiment, obtaining the test subject's emotional state label for product information includes: obtaining the test subject's valence score and arousal score for the product information after viewing the product information; and determining the test subject's emotional state label for the product information based on the valence score and the arousal score.
[0067] Among them, valence refers to the degree of pleasure (positive) or unpleasantness (negative) for a specific stimulus, and arousal refers to an emotional, physiological or cognitive activation state, usually caused by external stimuli, such as noise, light, smell, etc. In psychology, arousal is usually related to psychological processes such as emotions, attention and memory. Arousal can make individuals more likely to notice specific stimuli, increase the speed of response to the stimulus, and improve the strength of memory; the specific stimulus in the embodiment of the present disclosure refers to specific product information.
[0068] The training labels of the emotion recognition model in the embodiment of the present disclosure are obtained from each user's feedback on the product information. After viewing the product information, each user scores the emotional state caused by each stimulus (i.e., each product information) on the valence score table and / or the arousal score table. The scores provided by the users on the valence score table and / or the arousal score table can be used as training labels for training the model.
[0069] In the disclosed embodiments, the scores provided by the user on the valence score table and / or the arousal score table can be directly used as training labels (i.e., emotional state labels), or the user's emotional state in response to the product information can be determined based on the scores provided by the user on the valence score table and / or the arousal score table as training labels (i.e., emotional state labels).
[0070] In the disclosed embodiment, each emotional state label may correspond to a set of valence value ranges and / or arousal value ranges.
[0071] In step S404, feature extraction is performed on the brain wave signal based on the feature extraction module in the emotion recognition model to be trained to obtain the spatiotemporal feature vector of the brain wave signal.
[0072] In the disclosed embodiment, the emotion recognition model to be trained may include a feature extraction module and an emotion classification module, wherein the feature extraction module may include a first-layer convolutional network, a second-layer convolutional network and a third-layer convolutional network; the brain wave signal may be input into the first-layer convolutional network for time domain feature extraction to obtain the time domain feature vector of the brain wave signal; the time domain feature vector of the brain wave signal may be input into the second-layer convolutional network for spatial feature extraction to obtain the spatial feature vector of the brain wave signal; the time domain feature vector and the spatial feature vector of the brain wave signal may be input into the third-layer convolutional network for fusion processing to obtain the time-space feature vector of the brain wave signal.
[0073] In step S406, the spatiotemporal feature vectors are classified based on the emotion classification module in the emotion recognition model to be trained to obtain the predicted emotional state of the test subject with respect to the product information.
[0074] In the disclosed embodiment, the spatiotemporal feature vector of the brain wave signal may be input into an emotion classification module to predict the emotional state of the test subject and obtain the predicted emotional state of the test subject in response to the product information.
[0075] In step S408, the emotion recognition model to be trained is trained according to the emotion state label and the predicted emotion state to obtain a trained emotion recognition model.
[0076] In the disclosed embodiment, a loss value can be determined based on the emotional state label and the predicted emotional state, and model parameters in the emotion recognition model to be trained can be adjusted based on the loss value until the loss value reaches a preset condition, thereby completing the training of the emotion recognition model.
[0077] In the disclosed embodiment, the Keras tuner may be used to adjust the model parameters in the emotion recognition model to obtain target parameters. The target parameters may be, for example, as shown in Table 1.
[0078] Table 1 Model hyperparameter settings
[0079]
[0080]
[0081] Among them, No.of temporal filters refers to the parameters of the temporal filter, No.of depth filters refers to the number of depth filter layers, Dropout rate refers to the packet loss rate, No.of 1D Conv.Filters refers to the number of filter layers of the one-dimensional convolution layer, The activation function of 1D Conv.layers refers to the activation function of the filter of the one-dimensional convolution layer, Type of padding for 1D Conv.layers refers to the padding type of the filter of the one-dimensional convolution layer, and No.of filters for GRU layers refers to the number of filter layers of the GRU layer.
[0082] In the disclosed embodiment, after the emotion recognition model is trained, the Dreamer dataset (a public EEG dataset) can be used to test the emotion recognition model to detect and classify emotions in the valence dimension and the arousal dimension.
[0083] The emotion recognition model trained in the disclosed embodiment, when tested using the Dreamer dataset, achieved an average accuracy of 81.6% and 73.6% in emotion classification in the valence dimension and the arousal dimension, respectively. Compared with the emotion recognition models in the related technologies (such as ChronoNet, ShallowConvNet, DeepConvnet and EEGNet), the classification accuracy has been significantly improved.
[0084] refer to Figure 5 , which shows the accuracy of emotion classification in the arousal dimension of the emotion recognition model provided by the embodiment of the present disclosure and the DeepConv model in the related art for the EEG records of 23 subjects in the Dreamer dataset, through Figure 5 It can be seen that the emotion recognition model provided by the embodiment of the present disclosure has significantly improved classification accuracy compared with the DeepConv model in the related art.
[0085] It should also be understood that the above is only to help those skilled in the art better understand the embodiments of the present disclosure, and is not intended to limit the scope of the embodiments of the present disclosure. Based on the above examples given, those skilled in the art can obviously make various equivalent modifications or changes. For example, some steps in the above method may not be necessary, or some new steps may be added. Or a combination of any two or any multiple embodiments of the above. Such modifications, changes or combined solutions also fall within the scope of the embodiments of the present disclosure.
[0086] It should also be understood that the above description of the embodiments of the present disclosure focuses on emphasizing the differences between the various embodiments, and the same or similar points that are not mentioned can be referenced to each other. For the sake of brevity, they will not be repeated here.
[0087] It should also be understood that the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present disclosure.
[0088] It should also be understood that in the various embodiments of the present disclosure, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.
[0089] The above describes in detail an example of the emotion recognition method provided by the present disclosure. It is understandable that in order to implement the above functions, the computer device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.
[0090] The following are system embodiments of the present disclosure, which can be used to execute the method embodiments of the present disclosure. For details not disclosed in the system embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.
[0091] Figure 6 is a block diagram of an emotion recognition system in an exemplary embodiment of the present disclosure.
[0092] like Figure 6 As shown, the emotion recognition system 600 may include: a virtual reality head mounted device 602 and an emotion recognition model 604 based on a deep learning framework.
[0093] The emotion recognition system (also referred to as the EmoDLNet system) provided in the embodiment of the present disclosure can be used to measure the emotional impact of a specific stimulus presented to a user in a VR environment, and the system can include a virtual reality head mounted device 602 and an emotion recognition model 604 (i.e., a deep learning module). The deep learning module can extract EEG features from the original EEG signal recorded when the subject faces a specific stimulus in the VR environment, and automatically detect and classify the user's emotional state based on the extracted EEG features.
[0094] In the disclosed embodiment, the virtual reality head mounted device 602 and the emotion recognition model 604 may be two independent devices / modules, or may be integrated together. For example, the emotion recognition model 604 may be embedded in the virtual reality head mounted device to determine the emotional state of the subject.
[0095] Among them, the virtual reality head-mounted device 602 is used to obtain the brain wave signal generated by the target object when viewing product information in an immersive virtual reality environment; the emotion recognition model 604 is used to extract features of the brain wave signal based on the feature extraction module in the emotion recognition model to obtain the spatiotemporal feature vector of the brain wave signal; and classify the spatiotemporal feature vector based on the emotion classification module in the emotion recognition model to obtain the emotional state of the target object for the product information.
[0096] In some exemplary embodiments of the present disclosure, the feature extraction module includes a first layer of convolutional network, a second layer of convolutional network and a third layer of convolutional network; wherein, based on the feature extraction module in the emotion recognition model, feature extraction is performed on the brain wave signal to obtain the spatiotemporal feature vector of the brain wave signal, including: inputting the brain wave signal into the first layer of convolutional network for time domain feature extraction to obtain the time domain feature vector of the brain wave signal; inputting the time domain feature vector of the brain wave signal into the second layer of convolutional network for spatial feature extraction to obtain the spatial feature vector of the brain wave signal; inputting the time domain feature vector and the spatial feature vector of the brain wave signal into the third layer of convolutional network for fusion processing to obtain the spatiotemporal feature vector of the brain wave signal.
[0097] In some exemplary embodiments of the present disclosure, the brain wave signal includes multiple time series signals collected respectively under multiple channels; wherein, the brain wave signal is input into the first layer of convolutional network for time domain feature extraction to obtain the time domain feature vector of the brain wave signal, including: inputting the multiple time series signals into the first layer of convolutional network for time domain feature extraction to obtain the time domain feature vector corresponding to each channel; wherein, the time domain feature vector of the brain wave signal is input into the second layer of convolutional network for spatial feature extraction to obtain the spatial feature vector of the brain wave signal, including: inputting the time domain feature vectors corresponding to all channels into the second layer of convolutional network for cross-channel feature extraction to obtain the spatial feature vector of the brain wave signal.
[0098] In some exemplary embodiments of the present disclosure, the emotion classification module is a deep learning model based on a gated recurrent unit (GRU).
[0099] In some exemplary embodiments of the present disclosure, obtaining brain wave signals generated by a target object when viewing product information in an immersive virtual reality environment includes: displaying the product information to the target object in the immersive virtual reality environment through a virtual reality head-mounted device; and collecting brain wave signals generated by the target object when viewing the product information in the immersive virtual reality environment through the virtual reality head-mounted device.
[0100] In some exemplary embodiments of the present disclosure, the emotion recognition model is trained in the following manner: obtaining brain wave signals generated by test subjects when viewing product information in a virtual reality environment, and obtaining emotional state labels of the test subjects for the product information; performing feature extraction on the brain wave signals based on a feature extraction module in the emotion recognition model to be trained, and obtaining spatiotemporal feature vectors of the brain wave signals; classifying the spatiotemporal feature vectors based on an emotion classification module in the emotion recognition model to be trained, and obtaining predicted emotional states of the test subjects for the product information; training the emotion recognition model to be trained according to the emotional state labels and the predicted emotional states, and obtaining a trained emotion recognition model.
[0101] In some exemplary embodiments of the present disclosure, obtaining the emotional state label of the test subject with respect to the product information includes: obtaining the valence score and the arousal score of the test subject with respect to the product information after viewing the product information; and determining the emotional state label of the test subject with respect to the product information based on the valence score and the arousal score.
[0102] It should be noted that the block diagrams shown in the above figures are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor terminal devices and / or microcontroller terminal devices.
[0103] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0104] Figure 7 is a schematic diagram showing a structure of an electronic device suitable for implementing an exemplary embodiment of the present disclosure according to an exemplary embodiment. It should be noted that: Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0105] like Figure 7 As shown, the electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0106] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.
[0107] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 709, and / or installed from a removable medium 711. When the computer program is executed by a central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present disclosure are executed.
[0108] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0109] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0110] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor includes a sending unit, an acquisition unit, a determination unit, and a first processing unit. Among them, the names of these units do not constitute a limitation on the units themselves in certain cases, for example, the sending unit may also be described as "a unit that sends a picture acquisition request to the connected server."
[0111] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment. For example, the electronic device may implement the method described in the above embodiment. Figure 1 The steps shown.
[0112] According to one aspect of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various optional implementations of the above-mentioned embodiments.
[0113] It should be understood that any number of elements in the drawings of the present disclosure is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.
[0114] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0115] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. An emotion recognition method, characterized in that: include: Obtaining brain wave signals generated by the target object when viewing product information in an immersive virtual reality environment using a virtual reality headset; A feature extraction module in an emotion recognition model based on a deep learning framework performs feature extraction on the brain wave signal to obtain a spatiotemporal feature vector of the brain wave signal; The spatiotemporal feature vector is classified based on the emotion classification module in the emotion recognition model to obtain the emotional state of the target object with respect to the product information.
2. The method according to claim 1, characterized in that The feature extraction module includes a first convolutional network layer, a second convolutional network layer and a third convolutional network layer; The feature extraction module in the emotion recognition model based on the deep learning framework performs feature extraction on the brain wave signal to obtain the spatiotemporal feature vector of the brain wave signal, including: Inputting the brain wave signal into the first convolutional network layer to extract time domain features, and obtaining a time domain feature vector of the brain wave signal; Inputting the time domain feature vector of the brain wave signal into the second convolutional network to extract spatial features, thereby obtaining the spatial feature vector of the brain wave signal; The time domain feature vector and the space feature vector of the brain wave signal are input into the third convolutional network for fusion processing to obtain the time and space feature vector of the brain wave signal.
3. The method according to claim 2, characterized in that The brain wave signal includes a plurality of time series signals collected respectively in a plurality of channels; The step of inputting the brain wave signal into the first convolutional network layer to extract time domain features and obtaining the time domain feature vector of the brain wave signal includes: Inputting the multiple time series signals into the first convolutional network layer to extract time domain features, and obtaining a time domain feature vector corresponding to each channel; The step of inputting the time domain feature vector of the brain wave signal into the second convolutional network layer to extract the spatial feature and obtaining the spatial feature vector of the brain wave signal comprises: The time domain feature vectors corresponding to all channels are input into the second convolutional network to perform cross-channel feature extraction to obtain the spatial feature vectors of the brain wave signal.
4. The method according to any one of claims 1 to 3, characterized in that: The emotion classification module is a deep learning model based on the gated recurrent unit (GRU).
5. The method according to claim 1, characterized in that Obtain brain wave signals generated by the target object when viewing product information in an immersive virtual reality environment using a virtual reality headset, including: Displaying product information to the target object in the immersive virtual reality environment through a virtual reality head mounted device; The brain wave signals generated by the target object when viewing the product information in the immersive virtual reality environment are collected through a virtual reality head mounted device.
6. The method according to claim 1, characterized in that The emotion recognition model is trained in the following way: Obtaining brain wave signals generated by the test subject when viewing product information in an immersive virtual reality environment, and obtaining an emotional state label of the test subject in response to the product information; Extracting features of the brain wave signal based on a feature extraction module in the emotion recognition model to be trained to obtain a spatiotemporal feature vector of the brain wave signal; Classifying the spatiotemporal feature vector based on the emotion classification module in the emotion recognition model to be trained to obtain the predicted emotional state of the test subject with respect to the product information; The emotion recognition model to be trained is trained according to the emotion state label and the predicted emotion state to obtain a trained emotion recognition model.
7. The method according to claim 6, characterized in that Obtaining the emotional state label of the test subject with respect to the product information includes: obtaining the valence score and arousal score of the test subject for the product information after viewing the product information; The emotional state label of the test subject with respect to the product information is determined according to the valence score and the arousal score.
8. An emotion recognition system, characterized in that: include: Virtual reality headsets and emotion recognition models based on deep learning frameworks; Wherein, the virtual reality head mounted device is used to obtain brain wave signals generated by the target object when viewing product information in an immersive virtual reality environment; The emotion recognition model is used to extract features from the brain wave signal based on the feature extraction module in the emotion recognition model to obtain the spatiotemporal feature vector of the brain wave signal; and to classify the spatiotemporal feature vector based on the emotion classification module in the emotion recognition model to obtain the emotional state of the target object with respect to the product information.
9. An electronic device, characterized in that: include: at least one processor; A storage device, used to store at least one program, when the at least one program is executed by the at least one processor, enables the at least one processor to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that: When the executable instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Anesthesia depth monitoring method and system
CN119344680A
