Emotion recognition method and device, electronic equipment and readable storage medium

Abstract

The application discloses an emotion recognition method and device, electronic equipment and a readable storage medium, and belongs to the technical field of artificial intelligence. The emotion recognition method comprises the following steps: collecting state information of a user, wherein the state information comprises photoelectric plethysmography (PPG) information and acceleration (ACC) information; performing feature calculation on the PPG information and the ACC information to obtain target feature information; and determining that the user is in an anxious state when the difference between the target feature information and a reference feature value is greater than a preset threshold, wherein the reference feature value is calculated through sample state information, and the sample state information is information obtained when the user is in a non-anxious state.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to an emotion recognition method, device, electronic device, and readable storage medium. Background Technology

[0002] In related technologies, certain groups may require real-time monitoring of their emotional state in daily life. For example, individuals with mental illnesses may struggle to accurately express their needs, necessitating monitoring of their emotional state by caregivers who provide appropriate feedback. However, existing emotion recognition methods suffer from poor accuracy in identifying user emotions. Summary of the Invention

[0003] This application provides an emotion recognition method, apparatus, electronic device, and readable storage medium that can improve the accuracy of recognizing a user's anxiety emotions.

[0004] In a first aspect, embodiments of this application provide an emotion recognition method, the method comprising: Collect user status information, which includes photoplethysmography (PPG) information and acceleration (ACC) information; Feature calculations are performed on the PPG information and the ACC information to obtain target feature information; If the difference between the target feature information and the reference feature value is greater than a preset threshold, it is determined that the user is in an anxious state. The reference feature value is calculated through sample state information, which is information obtained when the user is in a non-anxious state.

[0005] Secondly, embodiments of this application provide an emotion recognition device, the device comprising: The acquisition module is used to acquire the user's status information, which includes photoplethysmography (PPG) information and acceleration (ACC) information. The calculation module is used to perform feature calculations on the PPG information and the ACC information to obtain target feature information; The determination module is used to determine that the user is in an anxious state when the difference between the target feature information and the reference feature value is greater than a preset threshold. The reference feature value is calculated through sample state information, which is information obtained when the user is in a non-anxious state.

[0006] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method as described in the first aspect.

[0007] Fourthly, embodiments of this application provide a readable storage medium, characterized in that a program or instructions are stored on the readable storage medium, and when the program or instructions are executed by a processor, they implement the steps of the method as described in the first aspect.

[0008] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the method as described in the first aspect.

[0009] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the steps of the method as described in the first aspect.

[0010] In this embodiment of the application, during the process of user emotion recognition, target feature information is obtained by performing feature calculation based on PPG information and ACC information, and the user is judged to be in an anxious state based on the target feature information. Since the target feature information integrates the user's PPG information and ACC information, it is beneficial to improve the accuracy of user anxiety emotion recognition. Attached Figure Description

[0011] Figure 1 This is one of the flowcharts illustrating the emotion recognition method provided in the embodiments of this application; Figure 2 This is a second schematic flowchart of the emotion recognition method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the emotion recognition device provided in the embodiments of this application; Figure 4 This application provides a schematic diagram of the structure of an electronic device; Figure 5 This application provides a schematic diagram of the hardware structure of an electronic device. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0013] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0014] The following description, in conjunction with the accompanying drawings, details an emotion recognition method, apparatus, electronic device, and readable storage medium provided in this application through specific embodiments and application scenarios.

[0015] Please see Figure 1 , Figure 1 This is a flowchart illustrating an emotion recognition method provided in an embodiment of this application. The method includes: Step 101: Collect user status information, wherein the status information includes photoplethysmographic (PPG) information and acceleration (ACC) information; Step 102: Perform feature calculations on the PPG information and the ACC information to obtain target feature information; Step 103: If the difference between the target feature information and the reference feature value is greater than a preset threshold, it is determined that the user is in an anxious state. The reference feature value is calculated through sample state information, which is information obtained when the user is in a non-anxious state.

[0016] The aforementioned emotion recognition method can be applied to various emotion recognition devices. In some embodiments of this application, the emotion recognition method can be applied to smart wearable devices, whereby the user is the wearer of the smart wearable device. Thus, by wearing the smart wearable device, the user's emotions can be monitored in real time. The following explanation uses the application of the emotion recognition method to a smart wearable device as an example to further illustrate the emotion recognition method in the embodiments of this application.

[0017] The aforementioned smart wearable devices can be of various types, such as smartwatches, smart bracelets, etc.

[0018] It is understood that the smart wearable device is equipped with detection elements for detecting the wearer's PPG and ACC information. For example, the smart wearable device may include an optical sensor for detecting PPG information and a triaxial accelerometer for detecting ACC information.

[0019] The PPG information described above can characterize the user's heart rate changes. Since different emotional states usually correspond to different heart rate values, the user's state of anxiety can be predicted based on the PPG information.

[0020] The ACC information described above can characterize a user's activity level. Since a user's heart rate typically increases during exercise, their activity level directly affects the PPG information. Therefore, directly identifying a user's anxiety level based on PPG information without considering their activity level may lead to poor accuracy in anxiety recognition due to the influence of their activity level.

[0021] Based on this, in this embodiment of the application, when identifying a user's emotions, feature calculation is performed by fusing PPG information and ACC information to obtain target feature information. The user's anxiety is then identified based on this target feature information to obtain the identification result. This eliminates the interference of the user's movement state on the anxiety identification process.

[0022] The aforementioned status information can be the user's status information within a preset time window. This preset time window can be a short period of time prior to the current time, such as status information within 5 minutes or 3 minutes prior to the current time. Since the emotional state identified based on this status information can be considered the user's current emotional state, the user's emotional state can be predicted in real-time based on the status information within the preset time window prior to the current time, or predicted every preset time window, thereby enabling real-time monitoring of the user's emotional state.

[0023] The aforementioned users can be of various types, specifically individuals with emotional care needs. The following explanation uses children with ASD as an example to further illustrate the method provided in this application embodiment. Thus, the method based on this application embodiment can monitor in real time whether children with ASD are in an anxious state, allowing for more care and attention, and better support. Furthermore, the method of this application embodiment only requires conventional wearable products to monitor whether children with ASD are in an anxious state, making monitoring more seamless and aesthetically pleasing.

[0024] The aforementioned reference feature value can be generated based on historical state information of the user in a non-anxiety state. Specifically, feature calculations are performed on PPG and ACC information from the historical state information to obtain feature information indicating the user's non-anxiety state. That is, the reference feature value can serve as an ideal value for the feature information indicating the user's non-anxiety state. Thus, when the target feature information is near the reference feature value, it can be determined that the user is in a non-anxiety state. Conversely, when the target feature information deviates significantly from the reference feature value, it can be determined that the user is in an anxious state.

[0025] The aforementioned preset threshold can be a threshold determined based on practical experience. Specifically, the user's anxiety state can be judged based on the following formula:

[0026] Among them, the This represents target feature information, the This represents the reference feature value, the This represents the preset threshold.

[0027] In this embodiment, during the process of user emotion recognition, target feature information is obtained by performing feature calculation based on PPG information and ACC information, and the user is judged to be in an anxious state based on the target feature information. Since the target feature information integrates the user's PPG information and ACC information, it is beneficial to improve the accuracy of user anxiety emotion recognition.

[0028] Optionally, the step of performing feature calculations on the PPG information and the ACC information to obtain target feature information includes: Feature extraction is performed on the PPG information to obtain the peak-to-peak interval (PPI) feature, and the ACC information is processed to obtain the user's triaxial acceleration feature; The target feature information is obtained by performing feature calculations on the PPI feature and the triaxial acceleration feature.

[0029] The PPI mentioned above refers to the PP interval, which represents the duration of one cardiac cycle and is generally expressed as the RR interval.

[0030] Specifically, to ensure that the signal input to the algorithm is subject to as little noise interference as possible, feature extraction of the PPG information and processing of the ACC information can be performed before... The PPG and ACC information are preprocessed, and then features are extracted from the preprocessed PPG and ACC information to obtain the PPI features and triaxial acceleration features. The preprocessing can involve denoising the continuously acquired PPG and ACC information using a filtering algorithm.

[0031] In this embodiment, the target feature information is obtained by performing feature calculations on the PPI feature and the triaxial acceleration feature.

[0032] Optionally, the PPG information includes the user's heart rate data within a preset time window; The step of extracting features from the PPG information to obtain PPI features during the peak-to-peak period includes: The P-waves in the PPG information are identified to obtain the number of PPIs in the PPG information; The PPI feature is generated based on the number of PPIs and the length of the preset time window, wherein the PPI feature is the ratio of the number of PPIs to the length of the preset time window.

[0033] The preset time window can be a short period of time before the current time point, such as within 5 minutes or 3 minutes before the current time point. Identifying the P waves in the PPG information means identifying all P waves in the PPG information. Specifically, this can be done by locating the P waves using various P wave localization algorithms, and then calculating the number of PPIs based on the number of P waves. Specifically, since one PPI includes one P wave, the number of identified P waves can be determined as the number of PPIs. The P wave is the initially generated deviating wave, reflecting the potential change during atrial depolarization and representing the depolarization of both atria.

[0034] Specifically, the aforementioned PPI feature can be a feature that reflects the overall trend of PPI. Since the PP interval, which reflects changes in the sympathetic and parasympathetic nervous systems, is relatively slow, a slowly changing trend of the PP interval is required. Based on this, the embodiments of this application use the average value of the PP interval within a sliding window to reflect this trend. Specifically, the PPI feature can be calculated using the following formula:

[0035] in, It is the set of PP intervals extracted from PPG information, where T represents the... The number of PP periods in the middle, The sliding window length refers to the duration of the preset time window mentioned above. The PPI feature is calculated from the k-th sliding window. .

[0036] In some embodiments of this application, the changing trend of the number of PP intervals for users under various emotional and movement states can be predetermined. During actual monitoring, the PPI characteristics within each sliding window in a continuous time period can be continuously calculated, thereby determining the changing trend of the number of real-time PP intervals based on the PPI characteristics. Thus, the user's current emotional state can be determined based on the determined changing trend of the number of real-time PP intervals and the current movement state.

[0037] In this embodiment, the number of PPIs in the PPG information is obtained by identifying the P waves, and the PPI features are generated based on the number of PPIs and the time length of the preset time window, thereby realizing the extraction process of the PPI features.

[0038] Optionally, the ACC information includes the user's motion acceleration within the preset time window; The process of processing the ACC information to obtain the user's three-axis acceleration characteristics includes: Based on the ACC information, a first acceleration parameter, a second acceleration parameter, and a third acceleration parameter are calculated. The first acceleration parameter is the standard deviation of the user's acceleration along the X-axis of the three-axis coordinate system within the preset time window; the second acceleration parameter is the standard deviation of the user's acceleration along the Y-axis of the three-axis coordinate system within the preset time window; and the third acceleration parameter is the standard deviation of the user's acceleration along the Z-axis of the three-axis coordinate system within the preset time window. Based on the first weight matrix, the first acceleration parameter, the second acceleration parameter and the third acceleration parameter are weighted and summed to obtain the triaxial acceleration feature; The step of performing feature calculations on the PPI features and the triaxial acceleration features to obtain the target feature information includes: The product of the PPI feature and the triaxial acceleration feature is determined as the target feature information.

[0039] It is understood that the ACC information may include the user's three-axis acceleration at various times within the preset time window.

[0040] The first acceleration parameter, the second acceleration parameter, and the third acceleration parameter are calculated based on the aforementioned ACC information. Specifically, they can be calculated using the following formula:

[0041]

[0042]

[0043] in, , , They are the first The average acceleration of the X, Y, and Z axes in each sliding window. Indicates the first The first acceleration parameter in the sliding window, Indicates the first The second acceleration parameter in the sliding window, Indicates the first The third acceleration parameter in the sliding window. W represents the third... The length of the sliding window.

[0044] The first weight matrix mentioned above can be a weight matrix determined in advance based on experience, and the first weight matrix can include three weight values ​​that correspond one-to-one with the three coordinate axes in the three-axis coordinate system. In this way, by using the first weight matrix, the first acceleration parameter, the second acceleration parameter and the third acceleration parameter are weighted and summed to obtain the three-axis acceleration feature. Since the three-axis acceleration feature calculates the acceleration features of the three coordinate axes, it can reflect the overall motion state of the user.

[0045] Specifically, the first weight matrix mentioned above can be expressed as: In the process of weighted summation of the first acceleration parameter, the second acceleration parameter, and the third acceleration parameter, the following can be constructed based on the aforementioned first acceleration parameter, second acceleration parameter, and third acceleration parameter: The matrix:

[0046] Thus, based on the first weight matrix, the first acceleration parameter, the second acceleration parameter, and the third acceleration parameter are weighted and summed to obtain the triaxial acceleration feature, which can be specifically calculated using the following formula:

[0047] Among them, the This indicates the triaxial acceleration characteristics.

[0048] Given the unique characteristics of children with autism, it is difficult to collect large amounts of data, making it impossible to use supervised learning methods that require a large amount of data. However, anxiety recognition is extremely important for them. Therefore, this application adopts a multimodal unsupervised learning algorithm, which continuously inputs features related to PPI trends and ACC-related features related to whether the wearer is moving into the unsupervised learning algorithm. The algorithm continuously updates the current state. When anxiety arousal is detected, the algorithm outputs the anxiety arousal state; otherwise, it continues to detect.

[0049] Specifically, the multimodal unsupervised learning algorithm can achieve the fusion of PPI features and triaxial acceleration features to obtain target feature information through the following formula:

[0050] Among them, the This represents target feature information, the This represents the PPI feature, and the... It can be calculated using the PPI feature calculation formula in the above embodiments.

[0051] It is understood that after obtaining the target feature information, the multimodal unsupervised learning algorithm can determine the user's emotional state based on the target feature information. For the specific implementation process, please refer to the relevant embodiments of determining the user's emotional state in the following examples.

[0052] In this embodiment, the first acceleration parameter, the second acceleration parameter, and the third acceleration parameter are weighted and summed based on the first weight matrix to obtain the triaxial acceleration features. Furthermore, the product of the PPI features and the triaxial acceleration features is determined as the target feature information, thereby realizing the process of feature calculation of the PPG information and ACC information.

[0053] Optionally, if the difference between the target feature information and the reference feature value is less than or equal to the preset threshold, the user is determined to be in a non-anxiety state.

[0054] In this embodiment, if the difference between the target feature information and the reference feature value is less than or equal to a preset threshold, it is determined that the user is in a non-anxious state. In this way, the process of judging the user's emotional state based on the target feature information can be realized.

[0055] Optionally, after obtaining the target feature information, the method further includes: If the difference between the target feature information and the reference feature value is less than or equal to the preset threshold, the reference feature value is updated based on the target feature information.

[0056] Specifically, for real users, emotions typically change slowly under normal circumstances. Therefore, in a normal, non-anxious state, the target feature information detected between two consecutive emotion detections will show only minor fluctuations. However, when the number of cumulative detections is large, even if the user remains in a non-anxious state, the accumulated emotional fluctuations will result in significant differences between the last detected target feature information and the first detected target feature information. Therefore, consistently using the same reference feature value may lead to inaccurate emotion recognition in subsequent detections due to the continuous accumulation of the user's normal emotional fluctuations.

[0057] Based on this, in this embodiment of the application, the reference feature value can be updated after each emotion recognition is completed to avoid the accumulation of normal emotional fluctuations of the user, which could lead to inaccurate subsequent emotion recognition results. Specifically, the reference feature value can be updated using the following formula:

[0058] Among them, the The updated reference feature value is the value obtained after the k-th emotion detection. The reference feature value in the k-th emotion detection process is described as follows. Let the target feature value be the value used in the k-th emotion detection process. This is a preset threshold.

[0059] In this embodiment, when the difference between the target feature information and the reference feature value is greater than the preset threshold, that is, when the emotion recognition result indicates that the user is in a non-anxious state, the reference feature value is updated based on the target feature information. In this way, the problem of inaccurate emotion recognition results caused by normal fluctuations in the user's emotions can be avoided.

[0060] Optionally, after determining that the user is in an anxious state when the difference between the target feature information and the reference feature value is greater than a preset threshold, the method further includes: If the difference between the target feature information and the reference feature value is greater than a preset threshold, an alert message is output.

[0061] The reminder information may be a flashing light, an audio alarm, or a text message sent to the terminal bound to the relevant person.

[0062] In this embodiment, when the difference between the target feature information and the reference feature value is greater than a preset threshold, that is, when the emotion recognition result indicates that the user is in an anxious state, a reminder message is output. This makes it convenient to notify relevant personnel in time that the user is in an anxious state, so that relevant personnel can comfort the user in time and thus better take care of the user.

[0063] Optionally, before determining that the user is in an anxious state when the difference between the target feature information and the reference feature value is greater than a preset threshold, the method further includes: Collect N sample state information, wherein the sample state information is sample PPG information and sample ACC information collected when the user is in a non-anxious state, and N is an integer greater than 1; Feature calculations are performed on the PPG information and ACC information in the N sample state information to obtain N target feature information that correspond one-to-one with the N sample state information; The average value of the N target feature information is determined as the reference feature value.

[0064] The N sample state information can be state information collected by N sliding windows when the user is in a non-anxiety state. For each sample state information, the sample PPG information and sample ACC information are fused using the fusion method described in the above embodiment to obtain N target feature information. That is, these N target feature information are all target feature information when the user is in a non-anxiety state. Then, the average value of the N target feature information is determined as the reference feature value, thereby improving the accuracy of the determined reference feature value.

[0065] The process of determining the reference eigenvalue described above can be achieved using the following formula: Based on the above PPI feature calculation formula, the PPI features corresponding to the PPG information in the N sample state information are calculated, resulting in the following N PPI features: ; Based on the above formula for calculating triaxial acceleration features, the triaxial acceleration features corresponding to the PPG information in the N sample state information are calculated, resulting in the following N triaxial acceleration features: , , ; Combining the triaxial acceleration characteristics into one The matrix:

[0066] Using an empirical weight matrix To fuse the triaxial acceleration features, the eigenvalue matrix B after fusion is calculated as follows:

[0067] As can be seen from the above formula, after calculation, B is a... The matrix, and the N PPI features can also be formed as follows The matrix:

[0068] To ensure that the final calculated state value reflects both the trend of PPI and the changes in ACC, the fused ACC feature matrix B needs to be fused with the PPI eigenvalue matrix to obtain the baseline state. It can be calculated using the following formula:

[0069] Among them, the These can be used as the reference feature values ​​mentioned above.

[0070] In this embodiment, the reference feature value is determined based on the state information of N samples, which helps to improve the accuracy of the determined reference feature value.

[0071] Please see Figure 2 The following is a flowchart illustrating an emotion recognition method provided in this application embodiment, taking a smartwatch as an example of a smart wearable device. The method includes the following steps: When the user is wearing a smartwatch, the user's PPG and ACC information are collected in real time. Preprocess the PPG and ACC information to obtain preprocessed PPG and ACC information; Feature extraction was performed on the preprocessed PPG and ACC information to obtain PPI features and triaxial acceleration features; Based on a multimodal unsupervised learning algorithm, the user's emotional state is predicted using the PPI features and triaxial acceleration features. When a user is in a state of anxiety, output a reminder message indicating that the user is in a state of anxiety; When a user is in a state of anxiety, the user's emotional state can be predicted based on the subsequently collected state information. At the same time, the reference feature values ​​can be updated based on the target feature information obtained this time.

[0072] The specific implementation process of this embodiment is similar to that of the above embodiments, and will not be repeated here to avoid repetition.

[0073] Please see Figure 3 , Figure 3This is a schematic diagram of the structure of an emotion recognition device 300 provided in an embodiment of this application. The device includes: The acquisition module 301 is used to acquire the user's status information, wherein the status information includes photoplethysmography (PPG) information and acceleration (ACC) information. Calculation module 302 is used to perform feature calculation on the PPG information and the ACC information to obtain target feature information; The determination module 303 is used to determine that the user is in an anxious state when the difference between the target feature information and the reference feature value is greater than a preset threshold. The reference feature value is calculated through sample state information, which is information obtained when the user is in a non-anxious state.

[0074] Optionally, the computing module 302 includes: The feature extraction submodule is used to extract features from the PPG information to obtain the PPI features during the peak-to-peak period, and to process the ACC information to obtain the user's triaxial acceleration features. The calculation submodule is used to perform feature calculations on the PPI features and the triaxial acceleration features to obtain the target feature information.

[0075] Optionally, the PPG information includes the user's heart rate data within a preset time window; the feature extraction submodule includes: The identification unit is used to identify the P wave in the PPG information to obtain the number of PPIs in the PPG information; A generation unit is configured to generate the PPI feature based on the number of PPIs and the length of the preset time window, wherein the PPI feature is the ratio of the number of PPIs to the length of the preset time window.

[0076] Optionally, the ACC information includes the user's motion acceleration within the preset time window; the feature extraction submodule further includes: The calculation unit is used to calculate a first acceleration parameter, a second acceleration parameter, and a third acceleration parameter based on the ACC information. The first acceleration parameter is the standard deviation of the user's acceleration along the X-axis of the three-axis coordinate system within the preset time window; the second acceleration parameter is the standard deviation of the user's acceleration along the Y-axis of the three-axis coordinate system within the preset time window; and the third acceleration parameter is the standard deviation of the user's acceleration along the Z-axis of the three-axis coordinate system within the preset time window. The calculation unit is also used to perform a weighted summation of the first acceleration parameter, the second acceleration parameter, and the third acceleration parameter based on the first weight matrix to obtain the triaxial acceleration feature; The calculation submodule is specifically used to determine the target feature information by multiplying the PPI feature and the triaxial acceleration feature.

[0077] Optionally, if the difference between the target feature information and the reference feature value is less than or equal to the preset threshold, the user is determined to be in a non-anxiety state.

[0078] Optionally, the device further includes: An update module is used to update the reference feature value based on the target feature information when the difference between the target feature information and the reference feature value is less than or equal to the preset threshold.

[0079] Optionally, the acquisition module 301 is further configured to acquire N sample state information, wherein the sample state information is sample PPG information and sample ACC information acquired when the user is in a non-anxiety state, and N is an integer greater than 1; Feature calculations are performed on the PPG information and ACC information in the N sample state information to obtain N target feature information that correspond one-to-one with the N sample state information; The average value of the N target feature information is determined as the reference feature value.

[0080] In this embodiment, during the process of user emotion recognition, target feature information is obtained by performing feature calculation based on PPG information and ACC information, and the user is judged to be in an anxious state based on the target feature information. Since the target feature information integrates the user's PPG information and ACC information, it is beneficial to improve the accuracy of user anxiety emotion recognition.

[0081] The emotion recognition device 300 in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a smartwatch, smart bracelet, augmented reality (AR) / virtual reality (VR) device, robot, other wearable devices, etc., and this application embodiment does not specifically limit the scope of the device.

[0082] The emotion recognition device 300 in this embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment does not specifically limit its use.

[0083] The emotion recognition device 300 provided in this application embodiment can achieve... Figure 1 and Figure 2 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0084] Optionally, such as Figure 4 As shown, this application embodiment also provides an electronic device 400, including a processor 401, a memory 402, and a program or instructions stored in the memory 402 and executable on the processor 401. When the program or instructions are executed by the processor 401, they implement the various processes of the above-described emotion recognition method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0085] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0086] Figure 5 A schematic diagram of the hardware structure of an electronic device according to an embodiment of this application.

[0087] The electronic device 500 includes, but is not limited to, components such as: radio frequency unit 501, network module 502, audio output unit 503, input unit 504, sensor 505, display unit 506, user input unit 507, interface unit 508, memory 509, and processor 510.

[0088] The sensor 505 is used to collect the user's status information, which includes photoplethysmography (PPG) information and acceleration (ACC) information. The processor 510 is used to perform feature calculations on the PPG information and the ACC information to obtain target feature information; The processor 510 is further configured to determine that the user is in an anxious state when the difference between the target feature information and the reference feature value is greater than a preset threshold, wherein the reference feature value is calculated by sample state information, and the sample state information is information obtained when the user is in a non-anxious state.

[0089] Optionally, the processor 510 is further configured to extract features from the PPG information to obtain PPI features during the peak-to-peak period, and to process the ACC information to obtain the user's triaxial acceleration features. The processor 510 is further configured to perform feature calculations on the PPI features and the triaxial acceleration features to obtain the target feature information.

[0090] Optionally, the processor 510 is further configured to identify the P wave in the PPG information to obtain the number of PPIs in the PPG information; The processor 510 is further configured to generate the PPI feature based on the number of PPIs and the length of the preset time window, wherein the PPI feature is the ratio of the number of PPIs to the length of the preset time window.

[0091] Optionally, the processor 510 is further configured to calculate a first acceleration parameter, a second acceleration parameter, and a third acceleration parameter based on the ACC information, wherein the first acceleration parameter is the standard deviation of the user's acceleration along the X-axis direction in the three-axis coordinate system within the preset time window; the second acceleration parameter is the standard deviation of the user's acceleration along the Y-axis direction in the three-axis coordinate system within the preset time window; and the third acceleration parameter is the standard deviation of the user's acceleration along the Z-axis direction in the three-axis coordinate system within the preset time window. The processor 510 is further configured to perform a weighted summation of the first acceleration parameter, the second acceleration parameter, and the third acceleration parameter based on the first weight matrix to obtain the triaxial acceleration feature; The processor 510 is further configured to determine the product of the PPI feature and the triaxial acceleration feature as the target feature information.

[0092] Optionally, the processor 510 is further configured to update the reference feature value based on the target feature information when the difference between the target feature information and the reference feature value is less than or equal to the preset threshold.

[0093] Optionally, the sensor 505 is further configured to collect N sample state information, wherein the sample state information is sample PPG information and sample ACC information collected when the user is in a non-anxiety state, and N is an integer greater than 1; The processor 510 is further configured to perform feature calculations on the PPG information and ACC information in the N sample state information to obtain N target feature information that correspond one-to-one with the N sample state information; The processor 510 is further configured to determine the average value of the N target feature information as the reference feature value.

[0094] Those skilled in the art will understand that the electronic device 500 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 510 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0095] It should be understood that, in this embodiment, the input unit 504 may include a graphics processing unit (GPU) 5041 and a microphone 5042. The GPU 5041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 506 may include a display panel 5061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 507 includes a touch panel 5071 and other input devices 5072. The touch panel 5071 is also called a touch screen. The touch panel 5071 may include a touch detection device and a touch controller. Other input devices 5072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0096] The memory 509 can be used to store software programs and various data. The memory 509 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 509 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 509 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0097] Processor 510 may include one or more processing units; optionally, processor 510 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 510.

[0098] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described emotion recognition method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0099] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0100] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described emotion recognition method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0101] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0102] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0104] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method of emotion recognition, characterized by, The method, applied to emotion recognition devices, includes: Collect user status information, which includes photoplethysmography (PPG) information and acceleration (ACC) information, wherein the ACC information includes the user's motion acceleration within a preset time window; Feature extraction is performed on the PPG information to obtain the PPI feature during the peak-to-peak period. Based on the ACC information, a first acceleration parameter, a second acceleration parameter, and a third acceleration parameter are calculated. The first acceleration parameter is the standard deviation of the user's acceleration along the X-axis in the three-axis coordinate system within the preset time window; the second acceleration parameter is the standard deviation of the user's acceleration along the Y-axis in the three-axis coordinate system within the preset time window; and the third acceleration parameter is the standard deviation of the user's acceleration along the Z-axis in the three-axis coordinate system within the preset time window. Based on a first weight matrix, the first acceleration parameter, the second acceleration parameter, and the third acceleration parameter are weighted and summed to obtain the three-axis acceleration feature. The product of the PPI feature and the triaxial acceleration feature is determined as the target feature information; If the difference between the target feature information and the reference feature value is greater than a preset threshold, it is determined that the user is in an anxious state. The reference feature value is calculated through sample state information, which is information obtained when the user is in a non-anxious state.

2. The method of claim 1, wherein, The PPG information includes the user's heart rate data within the preset time window; The step of extracting features from the PPG information to obtain PPI features during the peak-to-peak period includes: The P-waves in the PPG information are identified to obtain the number of PPIs in the PPG information; The PPI feature is generated based on the number of PPIs and the length of the preset time window, wherein the PPI feature is the ratio of the number of PPIs to the length of the preset time window.

3. The method according to claim 1, characterized in that, After obtaining the target feature information, the method further includes: If the difference between the target feature information and the reference feature value is less than or equal to the preset threshold, the reference feature value is updated based on the target feature information.

4. The method according to claim 1, characterized in that, Before determining that the user is in an anxious state when the difference between the target feature information and the reference feature value is greater than a preset threshold, the method further includes: Collect N sample state information, wherein the sample state information is sample PPG information and sample ACC information collected when the user is in a non-anxious state, and N is an integer greater than 1; Feature calculations are performed on the PPG information and ACC information in the N sample state information to obtain N target feature information that correspond one-to-one with the N sample state information; The average value of the N target feature information is determined as the reference feature value.

5. An emotion recognition device, characterized in that, The device includes: The acquisition module is used to acquire the user's status information, which includes photoplethysmography (PPG) information and acceleration (ACC) information. The ACC information includes the user's motion acceleration within a preset time window. The calculation module includes a feature extraction submodule and a calculation submodule. The feature extraction submodule is used to extract features from the PPG information to obtain the PPI feature during the peak-to-peak period, and to calculate a first acceleration parameter, a second acceleration parameter, and a third acceleration parameter based on the ACC information. The first acceleration parameter is the standard deviation of the user's acceleration along the X-axis in the three-axis coordinate system within the preset time window; the second acceleration parameter is the standard deviation of the user's acceleration along the Y-axis in the three-axis coordinate system within the preset time window; and the third acceleration parameter is the standard deviation of the user's acceleration along the Z-axis in the three-axis coordinate system within the preset time window. Based on a first weight matrix, the first acceleration parameter, the second acceleration parameter, and the third acceleration parameter are weighted and summed to obtain the three-axis acceleration feature. The calculation submodule is used to determine the product of the PPI feature and the triaxial acceleration feature as the target feature information; The determination module is used to determine that the user is in an anxious state when the difference between the target feature information and the reference feature value is greater than a preset threshold. The reference feature value is calculated through sample state information, which is information obtained when the user is in a non-anxious state.

6. The apparatus according to claim 5, characterized in that, The PPG information includes the user's heart rate data within the preset time window; The feature extraction submodule includes: The identification unit is used to identify the P wave in the PPG information to obtain the number of PPIs in the PPG information; A generation unit is configured to generate the PPI feature based on the number of PPIs and the length of the preset time window, wherein the PPI feature is the ratio of the number of PPIs to the length of the preset time window.

7. The apparatus according to claim 5, characterized in that, The device further includes: An update module is used to update the reference feature value based on the target feature information when the difference between the target feature information and the reference feature value is less than or equal to the preset threshold.

8. The apparatus according to claim 5, characterized in that, The acquisition module is also used to acquire N sample state information, wherein the sample state information is sample PPG information and sample ACC information acquired when the user is in a non-anxiety state, and N is an integer greater than 1; Feature calculations are performed on the PPG information and ACC information in the N sample state information to obtain N target feature information that correspond one-to-one with the N sample state information; The average value of the N target feature information is determined as the reference feature value.

Images