A method and apparatus for emotion detection

Abstract

The application discloses an emotion detection method in the field of artificial intelligence. The method comprises the following steps: acquiring a PPG signal of a pulse of a user through a PPG sensor; sending the PPG signal to a cloud server; receiving a user emotion state value sent by the cloud server, wherein the user emotion state value is obtained by processing a physiological feature vector according to a neural network model after the cloud server determines the physiological feature vector according to the PPG signal, and the physiological feature vector comprises blood oxygen saturation, heart rate and respiratory rate; and outputting a user emotion state according to the user emotion state value. The emotion detection method can improve emotion detection accuracy. The application also discloses a device capable of implementing the above emotion detection method.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to an emotion detection method and apparatus. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0003] People in modern society face significant work and life pressures, leading to varying degrees of emotional problems such as anxiety and depression. Therefore, effectively detecting and recording mood fluctuations has become a matter of concern.

[0004] One current method for eliciting emotional responses is roughly as follows: the user's heart rate, respiratory rate, body impedance, and body temperature changes are measured. If the results exceed the normal range of these parameters, the user is judged to have negative emotions.

[0005] In practical applications, the normal range of parameters is an estimated value set based on experience, and such a judgment method is not very accurate. Summary of the Invention

[0006] In view of this, this application provides an emotion detection method and apparatus that can improve the accuracy of emotion detection.

[0007] The first aspect provides an emotion detection method. In this method, a PPG sensor is used to acquire the user's pulse PPG signal; the PPG signal is sent to a cloud server; the cloud server determines a physiological feature vector based on the PPG signal, the physiological feature vector including at least two of blood oxygen saturation, heart rate, and respiratory rate; the cloud server inputs the physiological feature vector into a neural network model and outputs the user's emotional state value; after receiving the user's emotional state value sent by the cloud server, the cloud server outputs the user's emotional state based on the user's emotional state value. In this implementation, after acquiring the PPG signal, the cloud server obtains the physiological feature vector based on the PPG signal, processes the physiological feature vector through a neural network model, and obtains the user's emotional state value. Because the neural network model can better fit actual user emotional data, the detection of user emotional state using this neural network model is more accurate.

[0008] In another possible implementation, the aforementioned emotion detection method further includes: generating an emotion state curve based on the user's emotion state values ​​at different times; and outputting a reminder signal to alert the user to negative emotions when the duration of negative emotions in the emotion state curve exceeds a preset duration. This can remind the user of negative emotions and help the user control those emotions.

[0009] In another possible implementation, an alert signal is output when an emotional state change amplitude exceeding a preset value is detected within a set time period. This alerts the user to significant emotional fluctuations, helping them to manage their emotions.

[0010] The second aspect provides an emotion detection method. In this method, target model parameters are acquired; a PPG signal sent by a wearable device is received; a physiological feature vector is determined based on the PPG signal; the physiological feature vector is input into a neural network model with the target model parameters; and the user's emotional state value output by the neural network model is sent to the wearable device. The physiological feature vector includes at least two of blood oxygen saturation, heart rate, and respiratory rate. Accordingly, after acquiring the PPG signal, a physiological feature vector is obtained based on the PPG signal, and the physiological feature vector is processed by the neural network model to obtain the user's emotional state value. Because the neural network model can better fit actual user emotional data, the detection of user emotional state using this neural network model is more accurate.

[0011] In one possible implementation, obtaining the target model parameters includes: acquiring multiple user emotion data sets; acquiring initial model parameters for the neural network model; and processing the multiple user emotion data sets and the initial model parameters using a neural network algorithm to obtain the target model parameters. Each user emotion data set includes a physiological feature vector and a user emotion state value. Optionally, the neural network algorithm can be, but is not limited to, a backpropagation neural network algorithm, an adaptive resonance theory network algorithm, a learned vector quantization network algorithm, a Kohonen network algorithm, or a Hopfield network algorithm. This allows for the acquisition of multiple target model parameters. Since each target model parameter corresponds to a different neural network model, multiple neural network models can be generated for emotion state detection.

[0012] In another possible implementation, determining the physiological feature vector based on the PPG signal includes: decomposing the PPG signal into multiple IMF components and a residual component using an empirical mode decomposition method, whereby the multiple IMF components include a noise-dominant component and a signal-dominant component; denoising each noise-dominant component; performing morphological filtering on each signal-dominant component; constructing the target PPG signal from the denoised noise-dominant component, the morphologically filtered signal-dominant component, and the residual component; and determining the physiological feature vector based on the target PPG signal. This denoising process improves the accuracy of emotion detection by using the denoised PPG signal as input data for a neural network model.

[0013] In another possible implementation, decomposing the PPG signal into multiple IMF components and a residual component using the Empirical Mode Decomposition (EMD) method includes: treating the PPG signal as the signal to be processed; acquiring the upper and lower envelopes of the signal to be processed; determining the target signal based on the signal to be processed, its upper and lower envelopes; when the target signal is not an IMF component, treating it as the signal to be processed and triggering the step of acquiring its upper and lower envelopes; when the target signal is an IMF component, recording it; determining the difference signal based on the signal to be processed and the target signal; when the frequency of the difference signal is greater than a preset frequency, updating the signal to be processed to the difference signal and triggering the step of acquiring its upper and lower envelopes; when the frequency of the difference signal is less than or equal to the preset frequency, treating the difference signal as the residual component. This provides a feasible scheme for signal decomposition, facilitating implementation.

[0014] The third aspect provides an emotion detection device, which includes a PPG sensor, a transmitting module, a receiving module, and an output module. The PPG sensor is used to acquire the PPG signal of the user's pulse. The transmitting module is used to send the PPG signal to a cloud server. The receiving module is used to receive the user's emotional state value sent by the cloud server. The user's emotional state value is obtained by processing the physiological feature vector using a neural network model after the cloud server determines the physiological feature vector based on the PPG signal. The physiological feature vector includes blood oxygen saturation, heart rate, and respiratory rate. The output module is used to output the user's emotional state based on the user's emotional state value.

[0015] In another possible implementation, the emotion detection device also includes a generation module, which generates an emotion state curve based on the user's emotion state values ​​at different times; the output module is also used to output a reminder signal to remind the user of the negative emotion when the duration of the negative emotion in the emotion state curve exceeds a preset duration.

[0016] The steps performed by each module in the third aspect of the emotion detection device and its beneficial effects can be found in the relevant description in the first aspect.

[0017] The fourth aspect provides an emotion detection device, which includes: an acquisition module, a receiving module, a determination module, a neural network processing module, and a transmission module; the acquisition module is used to acquire target model parameters; the receiving module is used to receive PPG signals transmitted by a wearable device; the determination module is used to determine physiological feature vectors based on the PPG signals, the physiological feature vectors including blood oxygen saturation, heart rate, and respiratory rate; the neural network processing module is used to input the physiological feature vectors into a neural network model with target model parameters; and the transmission module is used to send the user's emotional state value output by the neural network model to the wearable device.

[0018] In another possible implementation, the acquisition module is specifically used to acquire multiple user emotion data, each of which includes a physiological feature vector and a user emotion state value; acquire the initial model parameters of the neural network model; and process the multiple user emotion data and the initial model parameters through a neural network algorithm to obtain the target model parameters.

[0019] In another possible implementation, the neural network algorithm is the backpropagation neural network algorithm, the adaptive resonance theory network algorithm, the learning vector quantization network algorithm, the Kohonen network algorithm, or the Hopfield network algorithm.

[0020] In another possible implementation, the determination module includes a decomposition unit, a denoising unit, a filtering unit, a construction unit, and a determination unit. The decomposition unit is used to decompose the PPG signal into multiple IMF components and a residual component using the empirical mode decomposition method. The multiple IMF components include a noise-dominant component and a signal-dominant component. The denoising unit is used to denoise each noise-dominant component. The filtering unit is used to perform morphological filtering on each signal-dominant component. The construction unit is used to construct the target PPG signal from the denoised noise-dominant component, the morphologically filtered signal-dominant component, and the residual component. The determination unit is used to determine the physiological feature vector based on the target PPG signal.

[0021] In another possible implementation, the decomposition unit is specifically used to: use the PPG signal as the signal to be processed; acquire the upper and lower envelopes of the signal to be processed; determine the target signal based on the signal to be processed, its upper and lower envelopes; when the target signal is not an IMF component, use the target signal as the signal to be processed, triggering the decomposition unit to acquire the upper and lower envelopes of the signal to be processed; when the target signal is an IMF component, record the target signal, and determine the difference signal based on the signal to be processed and the target signal; when the frequency of the difference signal is greater than a preset frequency, update the signal to be processed to the difference signal, triggering the step of acquiring the upper and lower envelopes of the signal to be processed; when the frequency of the difference signal is less than or equal to a preset frequency, use the difference signal as a residual component.

[0022] For details on the steps performed by each module or unit in the emotion detection device in the fourth aspect, as well as the beneficial effects, please refer to the relevant description in the second aspect.

[0023] The fifth aspect provides a wearable device including a PPG sensor, a processor, and a memory; the PPG sensor is used to acquire PPG signals; the memory is used to store programs and data; and the processor executes programs to implement the emotion detection method of the first aspect.

[0024] The sixth aspect provides a cloud server including a processor and a memory, the memory being used to store programs and data; the processor executing programs to implement the emotion detection method of the second aspect.

[0025] The seventh aspect provides an emotion detection system, which includes the wearable device of the fifth aspect and the cloud server of the sixth aspect.

[0026] The eighth aspect provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods of the above aspects.

[0027] The ninth aspect provides a computer program product containing instructions that, when run on a computer, causes the computer to perform the methods of the above aspects. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A schematic diagram of an artificial intelligence main framework provided in an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of an application environment provided for an embodiment of this application;

[0031] Figure 3 This is a schematic diagram of another application environment provided for an embodiment of this application;

[0032] Figure 4 A signaling interaction diagram for the emotion detection method provided in the embodiments of this application;

[0033] Figure 5 A structural diagram of the emotion detection device provided in the embodiments of this application;

[0034] Figure 6 A structural diagram of the emotion detection device provided in the embodiments of this application;

[0035] Figure 7 A structural diagram of a wearable device provided in an embodiment of this application;

[0036] Figure 8 A structural diagram of a cloud server provided in an embodiment of this application. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Figure 1 A schematic diagram of an artificial intelligence framework is shown, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence domain needs.

[0039] The above-mentioned artificial intelligence framework will be elaborated from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0040] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0041] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0042] (1) Infrastructure:

[0043] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0044] (2) Data

[0045] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0046] (3) Data processing

[0047] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0048] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0049] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0050] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0051] (4) General ability

[0052] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0053] (5) Smart Products and Industry Applications

[0054] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, and intelligent terminals.

[0055] See appendix Figure 2 This invention provides a system architecture 200. A data acquisition device 260 is used to acquire PPG signals. Photoplethysmography (PPG) is a non-invasive detection method that uses photoelectric means to detect changes in blood volume in living tissue. When a light beam of a certain wavelength shines on the skin surface of the fingertip, the beam is transmitted to the photodetector through transmission or reflection. During this process, due to absorption and attenuation by the skin, muscles, and blood at the detection end, the light intensity detected by the detector will decrease. The absorption of light by the skin, muscles, and tissues remains constant throughout the blood circulation, while the blood volume within the skin changes pulsatilely under the action of the heart. When the heart contracts, the peripheral blood volume is at its maximum, and the light absorption is also at its maximum, resulting in the minimum detected light intensity. Conversely, during diastole, the detected light intensity is at its maximum. Therefore, the light intensity received by the photodetector changes pulsatilely. By converting this light intensity change signal into an electrical signal, the change in pulsatile blood flow can be obtained. Volumetric pulse flow contains important physiological information about blood flow, including but not limited to blood oxygen saturation, heart rate, and respiratory rate.

[0056] Data acquisition device 260 determines emotion-related physiological characteristics, such as blood oxygen saturation, heart rate, and respiratory rate, based on PPG signals. It then sets corresponding user emotional state values ​​for these physiological characteristics, thus obtaining user emotion data that includes both physiological characteristics and user emotional state values. This user emotion data is stored in database 230, and training device 220 generates a target model / rule 201 based on the user emotion data maintained in database 230. The following will describe in more detail how training device 220 obtains the target model / rule 201 based on user emotion data. The target model / rule 201 can obtain the corresponding user emotional state value based on the input physiological feature vector.

[0057] The target model / rules obtained from training device 220 can be applied to different systems or devices. (See attached...) Figure 2 In this process, the execution device 210 is equipped with an I / O interface 212 to interact with external devices. Users can input data to the I / O interface 212 through the client device 240. The I / O interface is an input / output interface.

[0058] The execution device 210 can call data, code, etc. in the data storage system 250, and can also store data, instructions, etc. in the data storage system 250.

[0059] The calculation module 211 processes the input data using the target model / rule 201. Specifically, it determines blood oxygen saturation, heart rate, and respiratory rate based on the PPG signal obtained from the I / O interface 212, and uses the physiological feature vector composed of blood oxygen saturation, heart rate, and respiratory rate as input data for the target model / rule 201. After processing by the target model / rule 201, the user's emotional state value is obtained. The calculation module 211 can also perform noise reduction processing on the PPG signal.

[0060] The method for calculating blood oxygen saturation is as follows: Oxyhemoglobin (HbO2) and deoxyhemoglobin (Hb) are detected using red light (600–800 nm) and near-infrared light (800–1000 nm), respectively, to obtain the PPG signals of HbO2 and Hb. The corresponding ratios are then calculated using Beer-Lambert's law, thus yielding blood oxygen saturation. Notably, Hb has a higher absorption coefficient in the 600–800 nm range, while HbO2 has a higher absorption coefficient in the 800–1000 nm range.

[0061] The general method for calculating heart rate and respiratory rate is as follows: The PPG signal is filtered to obtain the number of peaks within a certain time period. The heart rate and respiratory rate are then determined based on the number of peaks. For example, if the sampling time is 5 seconds, and the number of peaks within 5 seconds is N, then the heart rate is N*12, and the respiratory rate is N*6.

[0062] Finally, I / O interface 212 returns the output of computing module 211 to client device 240 for use by the user.

[0063] At a deeper level, the training device 220 can generate corresponding target models / rules 201 based on different data for different objectives, in order to provide users with better results.

[0064] In the appendix Figure 2 In the illustrated scenario, the user can manually specify the data input to execution device 210, for example, by operating through the interface provided by I / O interface 212. Alternatively, client device 240 can automatically input data to I / O interface 212 and obtain results. If automatic data input by client device 240 requires user authorization, the user can set appropriate permissions within client device 240. The user can view the output results of execution device 210 on client device 240, which can be presented through display, sound, or animation. Client device 240 can also act as a data acquisition terminal, storing the acquired PPG signals into database 230.

[0065] It is worth noting that, attached Figure 2 This is merely a schematic diagram of a system architecture according to an embodiment of the present invention. The positional relationships between the devices, components, modules, etc. shown in the diagram do not constitute any limitation. For example, in the appendix... Figure 2 The data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may be placed in the execution device 210.

[0066] See appendix Figure 3 This invention provides a system architecture 300. The execution device 210 is implemented by one or more servers, optionally in conjunction with other computing devices, such as data storage, routers, load balancers, etc. The execution device 210 can be deployed on a single physical site or distributed across multiple physical sites. The execution device 210 can use data from the data storage system 250 or call program code from the data storage system 250. Figure 4 The steps in the illustrated embodiment are performed by the cloud server.

[0067] Users can interact with the execution device 210 by operating the wearable device 301. The wearable device 301 can be a smartwatch, smart bracelet, smart earphone, smart helmet, smart ankle bracelet, etc. Each user's wearable device 301 can interact with the execution device 210 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0068] In another implementation, one or more aspects of the execution device 210 may be implemented by each wearable device 301, for example, the wearable device 301 may provide local data or feedback calculation results to the execution device 210.

[0069] It should be noted that all the functions of the execution device 210 can also be implemented by the wearable device 301. For example, the wearable device 301 implements the functions of the execution device 210 and provides services to its own users.

[0070] Artificial neural networks (ANNs), also known as neural networks or neural network-like structures, can be understood as a computational model composed of numerous interconnected nodes (or neurons). Each node represents a specific output function, called the activation function. Each connection between two nodes represents a weighted value for the signal passing through that connection, called a weight, which is analogous to the memory of the artificial neural network. When the connections, weights, and activation functions of the neural network change, the network's output also changes. A neural network can approximate a certain algorithm or function found in nature, or it can be an expression of a logical strategy.

[0071] The function of each layer in a neural network can be expressed mathematically. To describe it: From a physical perspective, the work of each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are... The operation 4 is completed using +b, and the operation 5 is implemented using a(). The term "space" is used here because the objects being classified are not individual things, but a class of things; space refers to the set of all individuals within this class of things. Here, W is the weight vector, where each value represents the weight of a neuron in that layer of the neural network. This vector W determines the spatial transformation from the input space to the output space, as described above; that is, the weights W of each layer control how the space is transformed. The purpose of training the neural network is to ultimately obtain the weight matrix of all layers of the trained neural network (a weight matrix formed by the vectors W from many layers). Therefore, the training process of a neural network is essentially learning how to control the spatial transformation, more specifically, learning the weight matrix.

[0072] Because we want the output of the neural network to be as close as possible to the actual predicted value, we can compare the current network's prediction with the desired target value, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring the parameters of each layer in the neural network). For example, if the network's prediction is too high, the weight vector is adjusted to make it predict lower, and this adjustment is continued until the neural network can predict the actual target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value," which is the loss function or objective function. These are important equations used to measure the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so training the neural network becomes the process of minimizing this loss as much as possible.

[0073] The working process of a neural network consists of two phases: a learning phase and a working phase. During the learning phase, the connection weights of the neural network are modified to obtain the target model parameters. During the working phase, the neural network model with the target model parameters is used to calculate the input data and obtain the recognition result.

[0074] The following description uses wearable devices and cloud servers as examples to illustrate the emotion detection method of this application. (See also...) Figure 4 An embodiment of the emotion detection method provided in this application includes:

[0075] Step 401: Obtain the target model parameters from the cloud server.

[0076] In this embodiment, the target model parameters are the model parameters of the trained neural network model.

[0077] Step 402: The wearable device acquires the PPG signal of the user's pulse through the PPG sensor.

[0078] In this embodiment, the PPG signal of the user's pulse can be measured by the PPG sensor on the user's body parts (such as wrist, arm, ankle, ear, etc.).

[0079] Step 403: The wearable device sends the PPG signal to the cloud server.

[0080] Step 404: The cloud server determines the physiological feature vector based on the PPG signal. The physiological feature vector includes blood oxygen saturation, heart rate, and respiratory rate.

[0081] Step 405: The cloud server inputs the physiological feature vector into the neural network model with the target model parameters.

[0082] The cloud server uses physiological feature vectors as input data for a neural network model. After processing by the neural network model, the user's emotional state value is obtained. Emotional states include, but are not limited to, neutral, angry, contemptuous, disgusted, fearful, happy, sad, and surprised. User emotional state values ​​can be numbers, for example, 1 represents neutral, 2 represents happy, and 3 represents sad. User emotional state values ​​can also be binary values, for example, 001 represents neutral, 010 represents happy, and 011 represents sad.

[0083] Specifically, a neural network model includes an input layer, hidden layers, and an output layer, with one or more hidden layers. The input layer receives the physiological feature vector to be identified, performs a weighted calculation on the feature vector, and then passes the result to the nodes in the hidden layer. Each node in the hidden layer performs a weighted calculation on the output of the node in the previous layer, and the result of the calculation of the last hidden layer node is passed to the node in the output layer. Since each node in the output layer represents a type of emotional state, the calculation result is the possible probability of the emotional state. When the value of a node in the output layer is significantly greater than the values ​​of other nodes, the node with the largest value wins, and the emotional state represented by that node is taken as the emotional state corresponding to the physiological feature vector to be identified.

[0084] Step 406: The cloud server sends the user's emotional state value output by the neural network model to the wearable device.

[0085] Optionally, the cloud server determines the user's emotional state based on the user's emotional state value and sends the user's emotional state to the wearable device.

[0086] Step 407: The wearable device outputs the user's emotional state based on the user's emotional state value.

[0087] After receiving the user's emotional state value from the cloud server, the wearable device can remind the user of the current emotional state through text, vibration, audio signals (such as ringing and music), and video as reminder signals.

[0088] In this embodiment, since the neural network model can fit the actual user emotion data well, the detection of user emotion state based on the neural network model is more accurate.

[0089] Secondly, the PPG sensor of this application can acquire PPG signals and then calculate physiological characteristics such as blood oxygen saturation, heart rate and respiratory rate, requiring fewer sensors.

[0090] Furthermore, this application does not require storing large amounts of data, thus reducing the storage space requirements of wearable devices and offering the advantage of low cost.

[0091] In another optional embodiment, the above emotion detection method further includes: generating an emotion state curve based on the user's emotion state values ​​at different times; and outputting a reminder signal to remind the user of the negative emotion when the duration of the negative emotion in the emotion state curve exceeds a preset duration.

[0092] In this embodiment, the duration of negative emotions refers to the length of time a user is continuously in a negative emotional state. Negative emotions include, but are not limited to, anger, contempt, disgust, fear, sadness, and surprise. After plotting an emotional state curve based on user emotional state values ​​over a statistical period (e.g., one hour, one day, one week, or one month), the emotional state curve is monitored. When the duration of negative emotions in the emotional state curve exceeds a preset duration, a reminder signal is output to help the user adjust their emotions. The duration of the statistical period and the preset duration can be set according to actual conditions and are not limited in this application.

[0093] It should be understood that if an emotional state change amplitude exceeding a preset value is detected during a certain period, an alert signal will be output to remind the user of a drastic emotional fluctuation. This can serve as a reminder for illnesses affected by drastic emotional fluctuations (such as heart disease or hypertension), allowing for timely intervention and aiding in disease analysis, thereby promoting prevention. The duration of the detection period can be set according to actual circumstances; this application does not limit its value.

[0094] In another optional embodiment, multiple user emotion data are acquired, each user emotion data including a physiological feature vector and a user emotion state value, the physiological feature vector including blood oxygen saturation, heart rate and respiratory rate; initial model parameters of the neural network model are acquired; the multiple user emotion data and the initial model parameters are processed by the neural network algorithm to obtain the target model parameters.

[0095] Optionally, multiple user sentiment datasets can be divided into training, validation, and test sets. The training set is used to modify the initial model parameters to the first candidate model parameters. The user sentiment data in the validation set is used to determine the goodness of fit of the neural network model with the first candidate model parameters; if the goodness of fit is less than a preset goodness of fit, the initial model parameters are adjusted, and the training set is used to modify the adjusted initial model parameters to the second candidate model parameters. If the goodness of fit of the neural network model with the second candidate model parameters reaches the preset goodness of fit, the second candidate model parameters are determined as the target model parameters. Following this method, model parameters with a goodness of fit greater than the preset goodness of fit can be obtained, thereby preventing overfitting. The test set is used to test the error of the neural network model with the target model parameters.

[0096] The model parameters of a neural network model include the weights from input layer nodes to hidden layer nodes, the weights from hidden layer nodes to hidden layer nodes, the weights from hidden layer nodes to output layer nodes, and the number of hidden layers. The initial model parameters are the initial values ​​of these parameters. Neural network algorithms can be, but are not limited to, back propagation (BP) neural network algorithms, adaptive resonance theory network algorithms, learned vector quantization network algorithms, Kohonen network algorithms, or Hopfield network algorithms.

[0097] In an optional embodiment, the above-mentioned determination of physiological feature vectors based on PPG signals includes: decomposing the PPG signal into multiple intrinsic mode function (IMF) components and a residual component using an empirical mode decomposition method, wherein the multiple IMF components include noise-dominant components and signal-dominant components; denoising each noise-dominant component; performing morphological filtering on each signal-dominant component; constructing a target PPG signal from the denoised noise-dominant component, the morphologically filtered signal-dominant component, and the residual component; and determining physiological feature vectors based on the target PPG signal.

[0098] In this embodiment, Empirical Mode Decomposition (EMD) is a signal processing method suitable for handling nonlinear or non-stationary signals. It overcomes some shortcomings of traditional time-domain analysis methods and has strong adaptability. Decomposing the PPG signal into multiple IMF components and a residual component using EMD can be expressed as:

[0099]

[0100] x(t) represents the PPG signal, c i Let r be the i-th IMF component, n be the total number of IMF components, and r be the i-th component. n This represents the residual component.

[0101] Optionally, denoising for each dominant noise component includes: denoising each dominant noise component using a wavelet-like thresholding method.

[0102] In this embodiment, the cloud server can denoise the PPG signal using the Empirical Mode Decomposition (EMD) method, and then determine blood oxygen saturation, heart rate, and respiratory rate based on the denoised PPG signal. This provides more accurate blood oxygen saturation, heart rate, and respiratory rate, thereby increasing the accuracy of emotion detection. It should be understood that, in addition to cloud servers, wearable devices can also denoise PPG signals.

[0103] The empirical mode decomposition method is described in detail below. In another optional embodiment, using the empirical mode decomposition method to decompose the PPG signal into multiple IMF components and a residual component includes:

[0104] Step A1: Treat the PPG signal as the signal to be processed, x(t).

[0105] Step A2: Obtain the upper envelope xu(t) and lower envelope xl(t) of the signal to be processed.

[0106] Find the local maxima and minima of x(t), and use cubic spline interpolation to calculate the upper envelope xu(t) and the lower envelope xl(t).

[0107] Step A3: Determine the target signal d1(t) based on the signal to be processed x(t), the upper envelope xu(t) and the lower envelope xl(t) of the signal to be processed.

[0108] Optionally, d1(t) = x(t) - (xu(t) + xl(t)) / 2.

[0109] Step A4: When the target signal d1(t) is not an IMF component, the target signal d1(t) is used as the signal to be processed x(t), triggering steps A2 and A3.

[0110] Step A5: When the target signal d1(t) is an IMF component, record the target signal d1(t) and determine the difference signal r1(t) based on the signal to be processed x(t) and the target signal d1(t).

[0111] IMF component c1(t) = d1(t);

[0112] r1(t) = x(t) - c1(t).

[0113] Step A6: When the frequency of the differential signal r1(t) is greater than the preset frequency, update the signal to be processed x(t) to the differential signal r1(t) and trigger steps A2 to A5.

[0114] Step A7: When the frequency of the differential signal r1(t) is less than or equal to the preset frequency, the differential signal is used as the residual component.

[0115] When the frequency of the differential signal r1(t) is less than or equal to the preset frequency, it indicates that the differential signal is a stable signal. At this point, the iteration stops and the EMD decomposition is completed.

[0116] In this embodiment, multiple IMF components and a residual component can be obtained through iterative methods described above, thus providing a feasible solution.

[0117] The emotion detection method of this application will be introduced below using a specific application scenario:

[0118] In this specific application scenario, assume there are 5000 user emotion data points. Emotional states include neutral, happy, and sad, represented by 1, 2, and 3 respectively.

[0119]

[0120] Taking the backpropagation (BP) neural network algorithm as an example, the initial number of hidden layers in the BP neural network model is set to 10. Physiological feature vectors are used as input data to the BP neural network model, and user emotional state values ​​are used as the expected values. The BP neural network algorithm is used for computation. When the average error between the BP neural network output value and the expected value is greater than or equal to a preset error, iterative training is performed; when the average error is less than the preset error, the first candidate model parameters are obtained, and training stops. The goodness of fit of the BP neural network model with the first candidate model parameters is calculated. When the goodness of fit of this BP neural network model is less than a preset goodness of fit, the number of hidden layers in the BP neural network model is adjusted to 12, and then the second candidate model parameters are obtained according to the BP neural network algorithm. When the goodness of fit of the BP neural network model with the second candidate model parameters reaches or exceeds the preset goodness of fit, the second candidate model parameters are determined as the target model parameters.

[0121] Taking a smartwatch as an example, when a wearable device acquires the current PPG signal, it sends it to a cloud server. The cloud server determines blood oxygen saturation, heart rate, and respiratory rate based on this PPG signal. This physiological data is then used as the basis for a neural network model with target model parameters. The neural network model processes this data to obtain a user emotional state value (e.g., 3), which is then sent to the smartwatch. The smartwatch determines the user's emotional state as "sad." The data collection period is typically one hour, with the user's emotional state value sent every minute. If the user's emotional state value is 3 for five consecutive minutes, a reminder voice is played to help the user adjust their mood.

[0122] In this specific application scenario, the number of user emotional state values, the number of implicit layers, the collection duration, the preset duration for measuring the duration of negative emotions, and the period for sending user emotional state values ​​are all illustrative examples. Their values ​​can be set according to the actual situation, and this application does not impose any limitations.

[0123] The above embodiments describe the process of representing a user's emotional state numerically, obtaining target model parameters based on physiological feature vectors, user emotional state values, and initial model parameters, thereby obtaining a neural network model with target model parameters. This application can also represent emotional intensity using numerical or binary values, and establish a neural network model based on physiological feature vectors and emotional intensity. For example, 21 represents slight happiness, 22 represents moderate happiness, and 13 represents very happy. Emotional intensity is not limited to the examples above; other emotional intensity can be deduced similarly. This application can also obtain target model parameters based on physiological feature vectors, user emotional intensity, and initial model parameters, thereby obtaining a neural network model with target model parameters. It should be understood that the process of obtaining target model parameters based on physiological feature vectors, user emotional intensity, and initial model parameters is similar to the process of obtaining target model parameters based on physiological feature vectors, user emotional state values, and initial model parameters, and will not be repeated here.

[0124] The emotion detection method of this application has been introduced above. The emotion detection device of this application will be introduced below. (See reference...) Figure 5 This application provides an emotion detection device 500 comprising:

[0125] PPG sensor 501 is used to acquire the PPG signal of the user's pulse;

[0126] The transmitting module 502 is used to transmit PPG signals to the cloud server;

[0127] The receiving module 503 is used to receive the user's emotional state value sent by the cloud server. The user's emotional state value is obtained by processing the physiological feature vector using a neural network model after the cloud server determines the physiological feature vector based on the PPG signal. The physiological feature vector includes blood oxygen saturation, heart rate and respiratory rate.

[0128] Output module 504 is used to output the user's emotional state based on the user's emotional state value.

[0129] The emotion detection device 500 in this embodiment can achieve... Figure 4 The functions performed by the wearable device in the illustrated embodiment. Figure 5 The information interaction and execution process between the modules / units in the emotion detection device 500 shown are based on the same concept as the method embodiment of this application, and the resulting technical effects are the same as those in the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, and it will not be repeated here.

[0130] In an optional embodiment, the emotion detection device 500 further includes:

[0131] The generation module is used to generate emotional state curves based on the user's emotional state values ​​at different times.

[0132] The output module is also used to output a reminder signal to remind people of negative emotions when the duration of negative emotions in the emotional state curve exceeds a preset duration.

[0133] See Figure 6 This application provides an emotion detection device 600 comprising:

[0134] Module 601 is used to obtain the target model parameters;

[0135] Receiver module 602 is used to receive photoplethysmography (PPG) signals sent by the wearable device;

[0136] The determination module 603 is used to determine the physiological feature vector based on the PPG signal. The physiological feature vector includes blood oxygen saturation, heart rate, and respiratory rate.

[0137] The neural network processing module 604 is used to input physiological feature vectors into a neural network model with target model parameters;

[0138] The sending module 605 is used to send the user's emotional state value output by the neural network model to the wearable device.

[0139] The emotion detection device 600 in this embodiment can achieve... Figure 4 The steps performed by the cloud server in the illustrated embodiment. Figure 6 The information interaction and execution process between the modules / units in the emotion detection device 600 shown are based on the same concept as the method embodiment of this application, and the resulting technical effects are the same as those in the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, and it will not be repeated here.

[0140] In one alternative embodiment,

[0141] The acquisition module 601 is specifically used to acquire multiple user emotion data, each user emotion data including a physiological feature vector and a user emotion state value; acquire the initial model parameters of the neural network model; and process the multiple user emotion data and the initial model parameters through a neural network algorithm to obtain the target model parameters.

[0142] In another alternative embodiment, the neural network algorithm is a BP neural network algorithm, an adaptive resonance theory network algorithm, a learned vector quantization network algorithm, a Kohonen network algorithm, or a Hopfield network algorithm.

[0143] In another alternative embodiment, the determining module 602 includes:

[0144] The decomposition unit is used to decompose the PPG signal into multiple intrinsic mode function (IMF) components and a residual component using the empirical mode decomposition method. The multiple IMF components include noise-dominant components and signal-dominant components.

[0145] A noise reduction unit is used to denoise each dominant noise component.

[0146] The filtering unit is used to perform morphological filtering on each dominant component of the signal.

[0147] The construction unit is used to combine the denoised noise-dominant component, the morphologically filtered signal-dominant component, and the residual component to form the target PPG signal.

[0148] The determination unit is used to determine the physiological feature vector based on the target PPG signal.

[0149] Optionally, the decomposition unit is specifically used to: use the PPG signal as the signal to be processed; acquire the upper and lower envelopes of the signal to be processed; determine the target signal based on the signal to be processed and its upper and lower envelopes; when the target signal is not an IMF component, use the target signal as the signal to be processed, triggering the decomposition unit to acquire the upper and lower envelopes of the signal to be processed; when the target signal is an IMF component, record the target signal; determine the difference signal based on the signal to be processed and the target signal; when the frequency of the difference signal is greater than a preset frequency, update the signal to be processed to the difference signal, triggering the step of acquiring the upper and lower envelopes of the signal to be processed; when the frequency of the difference signal is less than or equal to a preset frequency, use the difference signal as a residual component.

[0150] The wearable device and cloud server of this application are described below from a hardware perspective:

[0151] See Figure 7 One embodiment of the wearable device 700 of this application includes:

[0152] The PPG sensor 701, processor 702 and memory 703, receiver 704, transmitter 705 and display 706 are connected via bus 707.

[0153] PPG sensor 701 is used to acquire the target PPG signal of the user's pulse;

[0154] Transmitter 705 is used to send the target PPG signal to the cloud server under the control of processor 702;

[0155] Receiver 704 is used to receive user emotional state values ​​sent by cloud server under the control of processor 702. The user emotional state values ​​are obtained by processing the physiological feature vectors using a neural network model after the cloud server determines the physiological feature vectors based on the target PPG signal. The physiological feature vectors include blood oxygen saturation, heart rate and respiratory rate.

[0156] Display 706 is used to display the user's emotional state based on the user's emotional state value under the control of processor 702.

[0157] Memory 703 is used to store programs and data;

[0158] In an optional embodiment, the processor 702 is used to generate an emotion state curve based on the user's emotion state values ​​at different times; when the duration of negative emotions in the emotion state curve exceeds a preset duration, the processor controls the display 706 to output a reminder signal to remind the user of the negative emotions.

[0159] See Figure 8 One embodiment of the cloud server 800 of this application includes:

[0160] Processor 801, memory 802, receiver 803 and transmitter 804;

[0161] Memory 802 is used to store programs and data;

[0162] Receiver 803 is used to receive data;

[0163] Transmitter 804 is used to send data;

[0164] Processor 801 executes the following methods by calling the program stored in memory 802:

[0165] Obtain the target model parameters;

[0166] Receive PPG signals sent by wearable devices;

[0167] Physiological feature vectors are determined based on PPG signals, which include blood oxygen saturation, heart rate, and respiratory rate.

[0168] The physiological feature vectors are input into a neural network model with the parameters of the target model;

[0169] The user's emotional state value output by the neural network model is sent to the wearable device.

[0170] In an optional embodiment, the processor 801 is also configured to perform the following methods:

[0171] Acquire multiple user emotion data, each including a physiological feature vector and a user emotion state value; obtain the initial model parameters for the neural network model; process the multiple user emotion data and the initial model parameters using a neural network algorithm to obtain the target model parameters.

[0172] In another alternative embodiment, the neural network algorithm is a BP neural network algorithm, an adaptive resonance theory network algorithm, a learned vector quantization network algorithm, a Kohonen network algorithm, or a Hopfield network algorithm.

[0173] In another alternative embodiment, processor 801 is specifically configured to perform the following methods:

[0174] The PPG signal is decomposed into multiple intrinsic mode function (IMF) components and a residual component using the empirical mode decomposition method. The multiple IMF components include noise-dominant components and signal-dominant components.

[0175] Denoise each dominant noise component;

[0176] Morphological filtering is performed on each dominant signal component;

[0177] The noise-dominant component after denoising, the signal-dominant component after morphological filtering, and the residual component constitute the target PPG signal.

[0178] The physiological feature vector is determined based on the target PPG signal.

[0179] In another alternative embodiment, processor 801 is specifically configured to perform the following methods:

[0180] Use the PPG signal as the signal to be processed;

[0181] Obtain the upper and lower envelopes of the signal to be processed;

[0182] The target signal is determined based on the signal to be processed, its upper and lower envelopes.

[0183] When the target signal is not an IMF component, the target signal is treated as the signal to be processed, triggering the steps of obtaining the upper and lower envelopes of the signal to be processed.

[0184] When the target signal is an IMF component, record the target signal;

[0185] Determine the difference signal based on the signal to be processed and the target signal;

[0186] When the frequency of the differential signal is greater than the preset frequency, the signal to be processed is updated to the differential signal, triggering the step of obtaining the upper and lower envelopes of the signal to be processed.

[0187] When the frequency of the differential signal is less than or equal to the preset frequency, the differential signal is treated as a residual component.

[0188] This application provides an emotion detection system, the emotion detection system including... Figure 7 The wearable device 700 and in the illustrated embodiment Figure 8 The cloud server 800 in the illustrated embodiment.

[0189] This application provides a computer storage medium including instructions that, when run on a computer, cause the computer to execute the emotion detection method in any of the above embodiments or optional embodiments.

[0190] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0191] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0192] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An emotion detection method, characterized in that, include: Obtain the target model parameters; Receive photoplethysmography (PPG) signals sent by wearable devices; The PPG signal is used as the signal to be processed. Obtain the upper and lower envelopes of the signal to be processed; The target signal is determined based on the signal to be processed, the upper envelope of the signal to be processed, and the lower envelope. When the target signal is not an Intrinsic Mode Function (IMF) component, the target signal is used as the signal to be processed, triggering the step of obtaining the upper and lower envelopes of the signal to be processed; When the target signal is an IMF component, the target signal is recorded, and the difference signal is determined based on the signal to be processed and the target signal. When the frequency of the differential signal is greater than the preset frequency, the signal to be processed is updated to the differential signal, triggering the step of obtaining the upper and lower envelopes of the signal to be processed; When the frequency of the differential signal is less than or equal to the preset frequency, the differential signal is used as a residual component, and the multiple IMF components include noise-dominant components and signal-dominant components. Denoise is applied to each of the dominant noise components using a wavelet-like thresholding method. Morphological filtering is performed on each dominant component of the signal; The noise-dominant component after denoising, the signal-dominant component after morphological filtering, and the residual component constitute the target PPG signal. A physiological feature vector is determined based on the target PPG signal, and the physiological feature vector includes blood oxygen saturation, heart rate, and respiratory rate; The physiological feature vector is input into a neural network model with the target model parameters; The user's emotional state value output by the neural network model is sent to the wearable device.

2. The method according to claim 1, characterized in that, The acquisition of target model parameters includes: Acquire multiple user emotion data, each of which includes a physiological feature vector and a user emotion state value; Obtain the initial model parameters of the neural network model; The multiple user emotion data and the initial model parameters are processed using a neural network algorithm to obtain the target model parameters.

3. The method according to claim 2, characterized in that, The neural network algorithm is either the backpropagation neural network algorithm, the adaptive resonance theory network algorithm, the learning vector quantization network algorithm, the Kohonen network algorithm, or the Hopfield network algorithm.

4. An emotion detection device, characterized in that, include: The acquisition module is used to obtain the parameters of the target model; The receiving module is used to receive the photoplethysmography (PPG) signal sent by the wearable device; The determination module is used to determine physiological feature vectors based on PPG signals, wherein the physiological feature vectors include blood oxygen saturation, heart rate, and respiratory rate; A neural network processing module is used to input the physiological feature vector into a neural network model having the target model parameters; A sending module is used to send the user's emotional state value output by the neural network model to the wearable device; The determining module includes: The decomposition unit is configured to: use the PPG signal as the signal to be processed; acquire the upper and lower envelopes of the signal to be processed; determine a target signal based on the signal to be processed, its upper and lower envelopes; when the target signal is not an Intrinsic Mode Function (IMF) component, use the target signal as the signal to be processed, triggering the step of acquiring the upper and lower envelopes of the signal to be processed; when the target signal is an IMF component, record the target signal and determine a difference signal based on the signal to be processed and the target signal; when the frequency of the difference signal is greater than a preset frequency, update the signal to be processed to the difference signal, triggering the step of acquiring the upper and lower envelopes of the signal to be processed; when the frequency of the difference signal is less than or equal to a preset frequency, use the difference signal as a residual component, wherein the plurality of IMF components include noise-dominant components and signal-dominant components; A denoising unit is used to denoise each dominant noise component using a wavelet-like thresholding method. The filtering unit is used to perform morphological filtering on each dominant component of the signal. The construction unit is used to combine the denoised noise-dominant component, the morphologically filtered signal-dominant component, and the residual component to form the target PPG signal. The determining unit is used to determine the physiological feature vector based on the target PPG signal.

5. The emotion detection device according to claim 4, characterized in that, The acquisition module is specifically used to acquire multiple user emotion data, each of which includes a physiological feature vector and a user emotion state value; acquire initial model parameters of a neural network model; and process the multiple user emotion data and the initial model parameters through a neural network algorithm to obtain target model parameters.

6. The emotion detection device according to claim 5, characterized in that, The neural network algorithm is either the backpropagation neural network algorithm, the adaptive resonance theory network algorithm, the learning vector quantization network algorithm, the Kohonen network algorithm, or the Hopfield network algorithm.

7. A computer storage medium comprising instructions, characterized in that, When it is run on a computer, it causes the computer to perform the emotion detection method according to any one of claims 1 to 3.

Images