Emotion recognition method, system, device and storage medium

Abstract

The embodiment of the present invention provides an emotion recognition method, system, device and storage medium, wherein the method includes: obtaining terminal motion data, terminal environment data and terminal usage data of a target user within a preset time period, using a preset feature extraction algorithm to extract features from the terminal motion data, terminal environment data and terminal usage data to obtain a user behavior feature set, using a preset normalization algorithm to normalize the user behavior feature set to obtain a user behavior data set, using a preset weight feature fusion algorithm to fuse various types of data in the user behavior data set in the vector dimension to obtain a user behavior feature vector, inputting the user behavior feature vector into a preset annular emotion recognition model for processing, and determining the emotion identification of the target user within the preset time period according to the processing result of the preset annular emotion recognition model. The present invention improves the recognition accuracy of user emotions.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition, and in particular to an emotion recognition method, system, device and storage medium. Background Art

[0002] Existing emotion recognition technology mainly collects users' daily behavior data through wearable devices or smart mobile terminals, extracts features from the daily behavior data, establishes a feature library, and uses discrete emotion models as a benchmark for recognition.

[0003] The existing discrete emotion model can only describe a limited range of emotions. When the user's actual emotional state is not in the enumerated discrete emotional state, the existing discrete emotion model cannot add emotion labels to the current data, which reduces the recognition accuracy. At the same time, the existing collected behavior data only reflects the user's objective behavior, but emotions are mostly changes in the user's subjective behavior. Therefore, the behavior data collected by the existing technology cannot accurately reflect the user's emotional changes. It can be seen that the existing technology cannot achieve accurate recognition of emotions. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide an emotion recognition method, system, device and storage medium to improve the recognition accuracy of user emotions. The specific technical solution is as follows:

[0005] An emotion recognition method, the method comprising:

[0006] Obtain the target user's terminal motion data, terminal environment data, and terminal usage data within a preset time period.

[0007] Using a preset feature extraction algorithm, feature extraction is performed on the terminal motion data, the terminal environment data, and the terminal usage data to obtain a user behavior feature set.

[0008] The user behavior feature set is normalized using a preset normalization algorithm to obtain a user behavior data set.

[0009] Using a preset weight feature fusion algorithm, various types of data in the user behavior data set are fused in the vector dimension to obtain a user behavior feature vector.

[0010] The user behavior feature vector is input into a preset cyclic emotion recognition model for processing, and the emotion identification of the target user within the preset time period is determined according to the processing result of the preset cyclic emotion recognition model.

[0011] Optionally, the training process of the preset cyclic emotion recognition model includes:

[0012] The sample terminal motion data, the sample terminal environment data, the sample terminal usage data and the emotion quantization coordinate values ​​in each of the time periods are obtained.

[0013] The preset feature extraction algorithm is used to extract features from the sample terminal motion data, the sample terminal environment data and the sample terminal usage data of each time period to obtain a sample user behavior feature set; the preset normalization algorithm is used to perform the normalization processing on the sample user behavior feature set to obtain a sample user behavior data set of each time period; the preset equal-weight feature fusion algorithm is used to fuse various types of data in the sample user behavior data set in the vector dimension to obtain the first sample user behavior feature vector of each time period; the first sample user behavior feature vector of each time period and the emotion quantization coordinate value in the corresponding time period are used as initial training samples to train the initial annular emotion recognition model to obtain an equal-weight annular emotion recognition model.

[0014] Using the preset weight feature fusion algorithm, the various types of data in the sample user behavior data set of each time period are fused in the vector dimension to obtain the second sample user behavior feature vector of each time period, and the second sample user behavior feature vector of each time period and the emotion quantization coordinate value in the corresponding time period are used as the final training samples to train the equal-weighted circular emotion recognition model to obtain the preset circular emotion recognition model.

[0015] The input of the preset annular emotion recognition model is the terminal motion data, the terminal environment data and the terminal usage data of the target user within the preset time period, and the output is the emotion identification of the target user within the preset time period.

[0016] Optionally, the preset feature extraction algorithm is composed of a preset sliding window algorithm, a preset time proportion algorithm, a preset clustering algorithm, and a preset data number statistical algorithm. The preset feature extraction algorithm is used to extract features from the terminal motion data, the terminal environment data, and the terminal usage data to obtain a user behavior feature set, including:

[0017] For the terminal motion data: using the preset sliding window algorithm, feature extraction is performed on the motion sensor data in the terminal motion data to obtain a motion sensor data feature group, wherein the motion sensor data feature group includes multiple types of motion sensor data feature values. Using the preset time proportion algorithm, feature extraction is performed on the motion state duration data in the terminal motion data to obtain a motion state duration feature group, wherein the motion state duration data is the proportion of the duration of each type of motion state data in the terminal motion data to the total duration of the preset time period, and the motion state duration feature group includes multiple types of motion state duration feature values.

[0018] For the terminal environment data: using the preset time proportion algorithm, extract features from the terminal working environment duration data in the terminal environment data to obtain a working environment duration data feature group, wherein the working environment duration data feature group includes terminal working duration data feature values ​​under multiple working environment types. Using the preset clustering algorithm, extract features from the terminal geographic data in the terminal environment data to obtain a terminal location data feature group, wherein the terminal location data feature group includes multiple types of terminal location data feature values.

[0019] For the terminal usage data: using the preset data number counting algorithm, extract features from the social data in the terminal usage data to obtain a social data feature group, wherein the social data feature group includes multiple types of social data feature values. Using the preset data number counting algorithm and the preset duration ratio algorithm, extract features from the terminal usage data to obtain a terminal software usage status data feature group, wherein the terminal software usage status data feature group includes multiple types of terminal software usage status data feature values.

[0020] The motion sensor data feature group, the motion state duration feature group, the working environment duration data feature group, the terminal location data feature group, the social data feature group and the terminal software usage status data feature group are merged to obtain the user behavior feature set.

[0021] Optionally, the using of a preset weight feature fusion algorithm to fuse various types of data in the user behavior data set in a vector dimension to obtain a user behavior feature vector includes:

[0022] Using the preset weight feature fusion algorithm, a first weight is assigned to the first type of user behavior data in the user behavior data set, and a second weight is assigned to the second type of user behavior data in the user behavior data combination, and the first type of user behavior data and the second type of user behavior data after the weights are assigned are fused in the vector dimension to obtain the user behavior feature vector, wherein the first weight is less than the second weight, the first type of user behavior data includes the data in the motion sensor data feature group, the motion state duration feature group, the working environment duration data feature group and the terminal location data feature group after the normalization processing, and the second type of user behavior data includes the data in the social data feature group and the terminal software usage status data feature group after the normalization processing.

[0023] Optionally, determining the emotion identifier of the target user within the preset time period according to the processing result of the preset cyclic emotion recognition model includes:

[0024] The area identifier corresponding to the processing result in the preset annular emotion model is determined as the emotion identifier, wherein the preset annular emotion model is a two-dimensional coordinate system, and the emotion identifier is a coordinate value in the two-dimensional coordinate system.

[0025] An emotion recognition system, the system comprising:

[0026] The initial data acquisition module is used to acquire the terminal motion data, terminal environment data and terminal usage data of the target user within a preset time period.

[0027] The behavior feature extraction module uses a preset feature extraction algorithm to extract features from the terminal motion data, the terminal environment data, and the terminal usage data to obtain a user behavior feature set.

[0028] The behavior data acquisition module uses a preset normalization algorithm to normalize the user behavior feature set to obtain a user behavior data set.

[0029] The feature vector acquisition module uses a preset weight feature fusion algorithm to fuse various types of data in the user behavior data set in the vector dimension to obtain a user behavior feature vector.

[0030] The emotion recognition module is used to input the user behavior feature vector into a preset cyclic emotion recognition model for processing, and determine the emotion identification of the target user within the preset time period according to the processing result of the preset cyclic emotion recognition model.

[0031] Optionally, the emotion recognition system further includes:

[0032] The model training module is used to obtain sample terminal motion data, sample terminal environment data, sample terminal usage data and emotion quantization coordinate values ​​in each of the time periods.

[0033] The preset feature extraction algorithm is used to extract features from the sample terminal motion data, the sample terminal environment data and the sample terminal usage data of each time period to obtain a sample user behavior feature set; the preset normalization algorithm is used to perform the normalization processing on the sample user behavior feature set to obtain a sample user behavior data set of each time period; the preset equal-weight feature fusion algorithm is used to fuse various types of data in the sample user behavior data set in the vector dimension to obtain the first sample user behavior feature vector of each time period; the first sample user behavior feature vector of each time period and the emotion quantization coordinate value in the corresponding time period are used as initial training samples to train the initial annular emotion recognition model to obtain an equal-weight annular emotion recognition model.

[0034] Using the preset weight feature fusion algorithm, the various types of data in the sample user behavior data set of each time period are fused in the vector dimension to obtain the second sample user behavior feature vector of each time period, and the second sample user behavior feature vector of each time period and the emotion quantization coordinate value in the corresponding time period are used as the final training samples to train the equal-weighted circular emotion recognition model to obtain the preset circular emotion recognition model.

[0035] The input of the preset annular emotion recognition model is the terminal motion data, the terminal environment data and the terminal usage data of the target user within the preset time period, and the output is the emotion identification of the target user within the preset time period.

[0036] Optionally, the behavior feature extraction module is configured to:

[0037] For the terminal motion data: using the preset sliding window algorithm, feature extraction is performed on the motion sensor data in the terminal motion data to obtain a motion sensor data feature group, wherein the motion sensor data feature group includes multiple types of motion sensor data feature values. Using the preset time proportion algorithm, feature extraction is performed on the motion state duration data in the terminal motion data to obtain a motion state duration feature group, wherein the motion state duration data is the proportion of the duration of each type of motion state data in the terminal motion data to the total duration of the preset time period, and the motion state duration feature group includes multiple types of motion state duration feature values.

[0038] For the terminal environment data: using the preset time proportion algorithm, extract features from the terminal working environment duration data in the terminal environment data to obtain a working environment duration data feature group, wherein the working environment duration data feature group includes terminal working duration data feature values ​​under multiple working environment types. Using the preset clustering algorithm, extract features from the terminal geographic data in the terminal environment data to obtain a terminal location data feature group, wherein the terminal location data feature group includes multiple types of terminal location data feature values.

[0039] For the terminal usage data: using the preset data number counting algorithm, extract features from the social data in the terminal usage data to obtain a social data feature group, wherein the social data feature group includes multiple types of social data feature values. Using the preset data number counting algorithm and the preset duration ratio algorithm, extract features from the terminal usage data to obtain a terminal software usage status data feature group, wherein the terminal software usage status data feature group includes multiple types of terminal software usage status data feature values.

[0040] The motion sensor data feature group, the motion state duration feature group, the working environment duration data feature group, the terminal location data feature group, the social data feature group and the terminal software usage status data feature group are merged to obtain the user behavior feature set.

[0041] Optionally, the feature vector acquisition module is configured to:

[0042] Using the preset weight feature fusion algorithm, a first weight is assigned to the first type of user behavior data in the user behavior data set, and a second weight is assigned to the second type of user behavior data in the user behavior data combination, and the first type of user behavior data and the second type of user behavior data after the weights are assigned are fused in the vector dimension to obtain the user behavior feature vector, wherein the first weight is less than the second weight, the first type of user behavior data includes the data in the motion sensor data feature group, the motion state duration feature group, the working environment duration data feature group and the terminal location data feature group after the normalization processing, and the second type of user behavior data includes the data in the social data feature group and the terminal software usage status data feature group after the normalization processing.

[0043] Optionally, the emotion recognition module is configured to:

[0044] The area identifier corresponding to the processing result in the preset annular emotion model is determined as the emotion identifier, wherein the preset annular emotion model is a two-dimensional coordinate system, and the emotion identifier is a coordinate value in the two-dimensional coordinate system.

[0045] An emotion recognition device, comprising:

[0046] processor;

[0047] A memory for storing instructions executable by the processor.

[0048] The processor is configured to execute the instructions to implement any of the emotion recognition methods described above.

[0049] A computer storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the device to execute any of the emotion recognition methods described above.

[0050] An emotion recognition method, system, device and storage medium provided by the embodiments of the present invention characterize the subjective behavior of the user by introducing terminal usage data, thereby improving the accuracy of the user's emotion recognition. At the same time, by using a preset weight feature fusion algorithm, the weight value of the feature that matches the terminal environment data in the user behavior feature vector is increased, thereby improving the final recognition accuracy of the user's emotions. Finally, by constructing a preset annular emotion model based on the annular emotion model, the present invention avoids the problem of reduced accuracy of emotion recognition due to incomplete enumeration of basic emotional states in the prior art. And because the annular emotion model can characterize the continuous change and degree of change of emotions, the preset annular emotion recognition model in this solution can reflect the intensity of the user's emotional state. It can be seen that the present invention improves the recognition accuracy of user emotions.

[0051] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A flowchart of an emotion recognition method provided by an embodiment of the present invention;

[0054] Figure 2A schematic diagram of emotion regions of a preset annular emotion model provided for an optional embodiment of the present invention;

[0055] Figure 3 A block diagram of an emotion recognition system provided by an embodiment of the present invention;

[0056] Figure 4 A block diagram of an emotion recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The embodiment of the present invention provides an emotion recognition method, such as Figure 1 As shown, the emotion recognition method includes:

[0059] S101. Acquire terminal motion data, terminal environment data, and terminal usage data of a target user within a preset period of time.

[0060] Optionally, in an optional embodiment of the present invention, the terminal motion data, terminal environment data and terminal usage data can be collected through sensors in smart mobile terminal devices. The data type of the terminal motion data can be: acceleration, rotation angle, magnetic field strength, etc. of the terminal. The data type of the environmental data can be: temperature, light intensity and geographic location information of the environment in which the terminal is located. The data type of the terminal usage data can be: call records, number of communications of social software, software usage, screen unlocking frequency, camera startup frequency.

[0061] The prior art usually uses the above-mentioned terminal motion data and terminal environment data as the basis for identifying user emotions. However, the motion data and environment data reflect the objective environment or state of the user. The user's emotions are a subjective behavior. Therefore, the present invention introduces terminal usage data to characterize the user's subjective behavior, thereby improving the accuracy of user emotion recognition. For example, if the user frequently unlocks the screen of a smart mobile terminal device in a short period of time, it reflects that the user's emotions are active during this period. If the number of communications of the user's social software within a week is lower than the preset number threshold, it reflects that the user's emotional response is not active during this week.

[0062] S102: Using a preset feature extraction algorithm, extract features from the terminal motion data, the terminal environment data, and the terminal usage data to obtain a user behavior feature set.

[0063] Optionally, in an optional embodiment of the present invention, since there is a lot of redundant information in the terminal motion data, terminal environment data and terminal usage data, the redundant information is likely to interfere with the final emotion recognition result. Therefore, the redundant information is removed by the preset feature extraction algorithm to improve the accuracy of user emotion recognition.

[0064] S103: using a preset normalization algorithm to normalize the user behavior feature set to obtain a user behavior data set.

[0065] Optionally, in an optional embodiment of the present invention, since the various types of data in the above user behavior feature set are collected by different sensors, and the different collection standards of different sensors are different, the data units in the user behavior feature set are different, and subsequent feature fusion and recognition cannot be performed. Therefore, the above preset normalization algorithm is used to normalize the various types of data in the above user behavior feature set to achieve the unification of data dimensions.

[0066] The specific types of the above-mentioned preset normalization algorithm include, but are not limited to: deviation normalization algorithm, extreme value normalization algorithm, function conversion algorithm, etc. The present invention does not impose too many limitations on this.

[0067] S104: Using a preset weight feature fusion algorithm, various types of data in the user behavior data set are fused in the vector dimension to obtain a user behavior feature vector.

[0068] Optionally, in an optional embodiment of the present invention, the terminal usage data reflects the user's emotions more accurately than the terminal motion data and the terminal environment data than the other two types of data. Therefore, the present invention improves the weight value of the feature that matches the terminal environment data in the user behavior feature vector by using a preset weight feature fusion algorithm, thereby improving the final recognition accuracy of the user's emotions. Among them, the types of the above-mentioned preset weight feature fusion algorithms include but are not limited to: feature fusion algorithms based on Bayesian theory, feature fusion algorithms based on linear feature dependency models, feature fusion algorithms based on sparse representation theory, etc. The present invention does not place too many restrictions on the types of the above-mentioned preset weight feature fusion algorithms.

[0069] It should be noted that the above weights can be set according to actual application scenarios, and the present invention does not make too many limitations here.

[0070] S105: Input the user behavior feature vector into a preset cyclic emotion recognition model for processing, and determine the emotion identification of the target user within a preset time period according to the processing result of the preset cyclic emotion recognition model.

[0071] Optionally, in an optional embodiment of the present invention, the preset cyclic emotion recognition model can be constructed based on the cyclic emotion model. The prior art uses a discrete emotion model for emotion recognition, and the discrete emotion model processes the collected data and compares it with the basic emotional state preset in the model to achieve emotion recognition of the user. However, since the selection range of the above basic emotional state is limited, when the collected data cannot match any of the basic emotional states preset in the model after processing, emotion recognition will fail.

[0072] The above-mentioned annular emotion model is composed of two parts: the pleasure dimension and the active dimension. Among them, the pleasure dimension can characterize whether the user's current emotional state is positive or negative. The active dimension can characterize whether the user's behavior in the current emotional state is active or passive. Since the emotional state is continuous and gradual, the user's emotional state can be determined by decomposing the emotional state into a two-dimensional space constructed by the above two dimensions and determining the position of the collected data in the above two-dimensional space. By constructing a preset annular emotion model based on the annular emotion model, the present invention avoids the problem of reduced accuracy of emotion recognition due to incomplete enumeration of basic emotional states in the prior art. At the same time, since the annular emotion model can characterize the continuous change and degree of change of emotions, the preset annular emotion recognition model in this scheme can reflect the intensity of the user's emotional state.

[0073] The present invention characterizes the user's subjective behavior by introducing terminal usage data, thereby improving the accuracy of user emotion recognition. At the same time, through the preset weight feature fusion algorithm, the weight value of the feature matching the terminal environment data in the user behavior feature vector is increased, thereby improving the final recognition accuracy of the user's emotions. Finally, by constructing a preset annular emotion model based on the annular emotion model, the present invention avoids the problem of reduced accuracy of emotion recognition due to incomplete enumeration of basic emotional states in the prior art. And because the annular emotion model can characterize the continuous change and degree of change of emotions, the preset annular emotion recognition model in this solution can reflect the intensity of the user's emotional state. It can be seen that the present invention improves the recognition accuracy of user emotions.

[0074] Optionally, the training process of the preset circumplex emotion recognition model includes:

[0075] Obtain sample terminal motion data, sample terminal environment data, sample terminal usage data and emotion quantification coordinate values ​​in each time period.

[0076] A preset feature extraction algorithm is used to extract features from the sample terminal motion data, sample terminal environment data and sample terminal usage data of each time period to obtain a sample user behavior feature set. A preset normalization algorithm is used to normalize the sample user behavior feature set to obtain a sample user behavior data set of each time period. A preset equal-weight feature fusion algorithm is used to fuse various types of data in the sample user behavior data set in the vector dimension to obtain the first sample user behavior feature vector of each time period. The first sample user behavior feature vector of each time period and the emotion quantization coordinate value in the corresponding time period are used as initial training samples to train the initial annular emotion recognition model to obtain an equal-weight annular emotion recognition model.

[0077] Using the preset weight feature fusion algorithm, the various types of data in the sample user behavior data set of each time period are fused in the vector dimension to obtain the second sample user behavior feature vector of each time period. The second sample user behavior feature vector of each time period and the emotion quantization coordinate value in the corresponding time period are used as the final training samples to train the equal-weighted circular emotion recognition model to obtain the preset circular emotion recognition model.

[0078] The input of the preset annular emotion recognition model is the terminal motion data, terminal environment data and terminal usage data of the target user within a preset period of time, and the output is the emotion identification of the target user within the preset period of time.

[0079] Optionally, in an optional embodiment of the present invention, the above-mentioned emotion quantification coordinate value can be obtained by quantifying the user's emotion during the period of collecting the above-mentioned sample terminal motion data, sample terminal environment data and sample terminal usage data. For example, the user's emotional pleasure interval is set to 1 to 10, and the active interval is set to 1 to 10. Assuming that the user is in an extremely happy state during the preset period, after collecting the sample terminal motion data, sample terminal environment data and sample terminal usage data during the preset period, the above-mentioned emotion quantification coordinate value collected is (10, 10).

[0080] Optionally, in another optional embodiment of the present invention, the preset equal-weight feature fusion algorithm can be used to improve the accuracy of emotion recognition based on objective data. Specifically, although the sample terminal motion data and sample terminal environment data reflect the user's objective behavior data, the accuracy of characterizing the user's emotions is lower than that of the sample terminal usage data, which is subjective behavior data, but the objective environment or state can still reflect the user's emotions. Therefore, by using the preset equal-weight feature fusion algorithm, the sample data is subjected to indifferent feature fusion, which can ensure the accuracy of emotion recognition based on objective behavior data by the equal-weighted cyclic emotion recognition model. Then, by using the preset weight feature fusion algorithm, after improving the sample terminal usage data in the user behavior feature vector, the equal-weighted cyclic emotion recognition model is trained again. Thereby improving the accuracy of the preset cyclic emotion recognition model finally obtained in recognizing user emotions based on objective behavior data and subjective behavior data.

[0081] Optionally, the preset feature extraction algorithm is composed of a preset sliding window algorithm, a preset time proportion algorithm, a preset clustering algorithm, and a preset data number statistical algorithm. The preset feature extraction algorithm is used to extract features of the terminal motion data, the terminal environment data, and the terminal usage data to obtain a user behavior feature set, including:

[0082] For terminal motion data: using a preset sliding window algorithm, extract features from the motion sensor data in the terminal motion data to obtain a motion sensor data feature group, wherein the motion sensor data feature group includes multiple types of motion sensor data feature values. Using a preset time proportion algorithm, extract features from the motion state duration data in the terminal motion data to obtain a motion state duration feature group, wherein the motion state duration data is the proportion of the duration of each type of motion state data in the terminal motion data to the total duration of a preset time period, and the motion state duration feature group includes multiple types of motion state duration feature values.

[0083] Optionally, in an optional embodiment of the present invention, the preset sliding window algorithm can segment the data according to a preset overlap ratio. Since the terminal motion data is a continuous time variable, segmenting the terminal motion data using the preset sliding window algorithm and performing feature extraction in each time series data segment after segmentation can increase the proportion of valid data in the extracted features, thereby improving the final emotion recognition accuracy.

[0084] Optionally, in another optional embodiment of the present invention, the data types in the above-mentioned motion sensor data feature group include but are not limited to: maximum value, minimum value, mean, standard deviation, number of waveforms, peak mean, trough mean, maximum value of the difference between peaks and troughs, minimum value of peak and trough interpolation, etc.

[0085] Optionally, in another optional embodiment of the present invention, the data types in the above-mentioned motion duration feature group include but are not limited to: still, slow, fast, etc.

[0086] For terminal environment data: using a preset time proportion algorithm, extract features from the terminal working environment duration data in the terminal environment data to obtain a working environment duration data feature group, wherein the working environment duration data feature group includes terminal working duration data feature values ​​under multiple working environment types. Using a preset clustering algorithm, extract features from the terminal geographic data in the terminal environment data to obtain a terminal location data feature group, wherein the terminal location data feature group includes multiple types of terminal location data feature values.

[0087] Optionally, in another optional embodiment of the present invention, the above-mentioned preset time proportion algorithm can be used to calculate the proportion of each divided time duration to the total time duration of the preset time period.

[0088] Optionally, in another optional embodiment of the present invention, the data types in the above-mentioned working environment time data feature group include but are not limited to: the proportion of time under no light conditions, the proportion of time under weak light conditions, the proportion of time under lighting conditions, the proportion of time under strong light conditions, etc.

[0089] Optionally, in another optional embodiment of the present invention, the above-mentioned preset clustering algorithm can be used to determine the locations that the terminal has visited, and the stay time of the terminal at each location point.

[0090] For terminal usage data: using a preset data number statistical algorithm, extract features from the social data in the terminal usage data to obtain a social data feature group, wherein the social data feature group includes multiple types of social data feature values. Using a preset data number statistical algorithm and a preset duration ratio algorithm, extract features from the terminal usage data to obtain a terminal software usage status data feature group, wherein the terminal software usage status data feature group includes multiple types of terminal software usage status data feature values.

[0091] Optionally, in another optional embodiment of the present invention, the data types in the above-mentioned social data feature group include but are not limited to: the number of call records, the number of text messages, the number of messages from social software, the number of Bluetooth device connections, the number of network hotspot device connections, etc.

[0092] Optionally, in another optional embodiment of the present invention, the data types in the above-mentioned terminal software usage status data feature group include but are not limited to: software usage status, screen unlocking frequency, camera usage frequency, ringtone mode duration, operating mode duration, charging mode duration, etc.

[0093] The motion sensor data feature group, the motion state duration feature group, the work environment duration data feature group, the terminal location data feature group, the social data feature group and the terminal software usage status data feature group are merged to obtain a user behavior feature set.

[0094] Optionally, a preset weight feature fusion algorithm is used to fuse various types of data in the user behavior data set in the vector dimension to obtain a user behavior feature vector, including:

[0095] Using a preset weight feature fusion algorithm, a first weight is assigned to the first type of user behavior data in the user behavior data set, and a second weight is assigned to the second type of user behavior data in the user behavior data combination, and the weighted first type of user behavior data and the second type of user behavior data are fused in the vector dimension to obtain a user behavior feature vector, wherein the first weight is less than the second weight, the first type of user behavior data includes data in a normalized motion sensor data feature group, a motion state duration feature group, a working environment duration data feature group, and a terminal location data feature group, and the second type of user behavior data includes data in a normalized social data feature group and a terminal software usage status data feature group.

[0096] Optionally, determining the emotion identifier of the target user within a preset time period according to the processing result of the preset cyclic emotion recognition model includes:

[0097] The region mark corresponding to the processing result in the preset annular emotion model is determined as the emotion mark, wherein the preset annular emotion model is a two-dimensional coordinate system, and the emotion mark is the coordinate value in the two-dimensional coordinate system.

[0098] In order to facilitate the understanding of the above processing results based on the preset cyclic emotion recognition model to determine the target user's emotion identification process within the preset time period, hereby combined with the following Figure 2 An alternative embodiment of the present invention is shown to illustrate:

[0099] like Figure 2 The figure shows a schematic diagram of the emotion area of ​​the above-mentioned preset annular emotion model.

[0100] It is assumed that the terminal motion data, terminal environment data and terminal usage data of the target user in four preset time periods are respectively input into the above preset annular emotion model for processing. According to the order of the preset time periods, the four processing results obtained are: (1, -10), (8, 9), (-10, 5) and (-11, -2).

[0101] According to Figure 2It can be seen that the area corresponding to the processing result of the first preset time period is marked as "calm", the area corresponding to the processing result of the second preset time period is marked as "excited", the area corresponding to the processing result of the third preset time period is marked as "nervous", and the area corresponding to the processing result of the fourth preset time period is marked as "sad".

[0102] Corresponding to the above method embodiment, the present invention also provides an emotion recognition system, such as Figure 3 As shown, the emotion recognition system includes:

[0103] The initial data acquisition module 301 is used to acquire the terminal motion data, terminal environment data and terminal usage data of the target user within a preset time period.

[0104] The behavior feature extraction module 302 uses a preset feature extraction algorithm to extract features from the terminal motion data, the terminal environment data, and the terminal usage data to obtain a user behavior feature set.

[0105] The behavior data acquisition module 303 uses a preset normalization algorithm to normalize the user behavior feature set to obtain a user behavior data set.

[0106] The feature vector acquisition module 304 uses a preset weight feature fusion algorithm to fuse various types of data in the user behavior data set in the vector dimension to obtain a user behavior feature vector.

[0107] The emotion recognition module 305 is used to input the user behavior feature vector into a preset cyclic emotion recognition model for processing, and determine the emotion identification of the target user within a preset time period according to the processing result of the preset cyclic emotion recognition model.

[0108] Optional, as above Figure 3 The emotion recognition system shown also includes:

[0109] A model training module is used to obtain sample terminal motion data, sample terminal environment data, sample terminal usage data and emotion quantification coordinate values ​​in each time period;

[0110] Using a preset feature extraction algorithm to extract features from sample terminal motion data, sample terminal environment data, and sample terminal usage data for each time period, obtaining a sample user behavior feature set, using a preset normalization algorithm to normalize the sample user behavior feature set, obtaining a sample user behavior data set for each time period, using a preset equal-weight feature fusion algorithm to fuse various types of data in the sample user behavior data set in a vector dimension, obtaining a first sample user behavior feature vector for each time period, using the first sample user behavior feature vector for each time period and the emotion quantization coordinate value in the corresponding time period as an initial training sample, training an initial annular emotion recognition model, and obtaining an equal-weight annular emotion recognition model;

[0111] Using a preset weight feature fusion algorithm, various types of data in the sample user behavior data set of each time period are fused in the vector dimension to obtain the second sample user behavior feature vector of each time period, and the second sample user behavior feature vector of each time period and the emotion quantization coordinate value in the corresponding time period are used as the final training sample to train the equal weighted annular emotion recognition model to obtain the preset annular emotion recognition model;

[0112] The input of the preset annular emotion recognition model is the terminal motion data, terminal environment data and terminal usage data of the target user within a preset period of time, and the output is the emotion identification of the target user within the preset period of time.

[0113] Optionally, the behavior feature extraction module 302 is configured as follows:

[0114] For terminal motion data: using a preset sliding window algorithm, extract features from the motion sensor data in the terminal motion data to obtain a motion sensor data feature group, wherein the motion sensor data feature group includes multiple types of motion sensor data feature values. Using a preset time proportion algorithm, extract features from the motion state duration data in the terminal motion data to obtain a motion state duration feature group, wherein the motion state duration data is the proportion of the duration of each type of motion state data in the terminal motion data to the total duration of a preset time period, and the motion state duration feature group includes multiple types of motion state duration feature values.

[0115] For terminal environment data: using a preset time proportion algorithm, extract features from the terminal working environment duration data in the terminal environment data to obtain a working environment duration data feature group, wherein the working environment duration data feature group includes terminal working duration data feature values ​​under multiple working environment types. Using a preset clustering algorithm, extract features from the terminal geographic data in the terminal environment data to obtain a terminal location data feature group, wherein the terminal location data feature group includes multiple types of terminal location data feature values.

[0116] For terminal usage data: using a preset data number statistical algorithm, extract features from the social data in the terminal usage data to obtain a social data feature group, wherein the social data feature group includes multiple types of social data feature values. Using a preset data number statistical algorithm and a preset duration ratio algorithm, extract features from the terminal usage data to obtain a terminal software usage status data feature group, wherein the terminal software usage status data feature group includes multiple types of terminal software usage status data feature values.

[0117] The motion sensor data feature group, the motion state duration feature group, the work environment duration data feature group, the terminal location data feature group, the social data feature group and the terminal software usage status data feature group are merged to obtain a user behavior feature set.

[0118] Optionally, the feature vector acquisition module 304 is configured as follows:

[0119] Using a preset weight feature fusion algorithm, a first weight is assigned to the first type of user behavior data in the user behavior data set, and a second weight is assigned to the second type of user behavior data in the user behavior data combination, and the weighted first type of user behavior data and the second type of user behavior data are fused in the vector dimension to obtain a user behavior feature vector, wherein the first weight is less than the second weight, the first type of user behavior data includes data in a normalized motion sensor data feature group, a motion state duration feature group, a working environment duration data feature group, and a terminal location data feature group, and the second type of user behavior data includes data in a normalized social data feature group and a terminal software usage status data feature group.

[0120] Optionally, the emotion recognition module 305 is configured as follows:

[0121] The region mark corresponding to the processing result in the preset annular emotion model is determined as the emotion mark, wherein the preset annular emotion model is a two-dimensional coordinate system, and the emotion mark is the coordinate value in the two-dimensional coordinate system.

[0122] The embodiment of the present invention provides an emotion recognition device, such as Figure 4 As shown, the emotion recognition device comprises:

[0123] Processor 401;

[0124] The memory 402 is used to store instructions executable by the processor 401 .

[0125] The processor 402 is configured to execute instructions to implement any of the above-mentioned emotion recognition methods.

[0126] An embodiment of the present invention provides a computer storage medium. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the device is enabled to execute any of the above-mentioned emotion recognition methods.

[0127] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.

[0128] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0129] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0130] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0131] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0132] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. An emotion recognition method, characterized in that: The method comprises: Obtain terminal motion data, terminal environment data, and terminal usage data of the target user within a preset period of time; Using a preset feature extraction algorithm, feature extraction is performed on the terminal motion data, the terminal environment data, and the terminal usage data to obtain a user behavior feature set; Using a preset normalization algorithm, normalizing the user behavior feature set to obtain a user behavior data set; Using a preset weight feature fusion algorithm, various types of data in the user behavior data set are fused in the vector dimension to obtain a user behavior feature vector; Inputting the user behavior feature vector into a preset cyclic emotion recognition model for processing, and determining the emotion identification of the target user within the preset time period according to the processing result of the preset cyclic emotion recognition model; The training process of the preset cyclic emotion recognition model includes: Obtaining sample terminal motion data, sample terminal environment data, sample terminal usage data and emotion quantification coordinate values ​​in each of the time periods; Using the preset feature extraction algorithm to extract features from the sample terminal motion data, the sample terminal environment data, and the sample terminal usage data of each time period, to obtain a sample user behavior feature set, using the preset normalization algorithm to perform the normalization processing on the sample user behavior feature set to obtain a sample user behavior data set of each time period, using a preset equal-weight feature fusion algorithm to fuse various types of data in the sample user behavior data set in a vector dimension to obtain a first sample user behavior feature vector of each time period, using the first sample user behavior feature vector of each time period and the emotion quantization coordinate value in the corresponding time period as initial training samples, to train an initial annular emotion recognition model to obtain an equal-weight annular emotion recognition model; Using the preset weight feature fusion algorithm, each type of data in the sample user behavior data set of each time period is fused in the vector dimension to obtain a second sample user behavior feature vector of each time period, and using the second sample user behavior feature vector of each time period and the emotion quantization coordinate value in the corresponding time period as the final training sample, the equal-weighted annular emotion recognition model is trained to obtain the preset annular emotion recognition model; The input of the preset annular emotion recognition model is the terminal motion data, the terminal environment data and the terminal usage data of the target user within the preset period, and the output is the emotion identification of the target user within the preset period; The method of using a preset weight feature fusion algorithm to fuse various types of data in the user behavior data set in the vector dimension to obtain a user behavior feature vector includes: Using the preset weight feature fusion algorithm, a first weight is assigned to the first type of user behavior data in the user behavior data set, and a second weight is assigned to the second type of user behavior data in the user behavior data set. The first type of user behavior data and the second type of user behavior data after the weights are assigned are fused in the vector dimension to obtain the user behavior feature vector, wherein the first weight is less than the second weight, the first type of user behavior data includes the data in the normalized motion sensor data feature group, motion state duration feature group, work environment duration data feature group and terminal location data feature group, and the second type of user behavior data includes the data in the normalized social data feature group and terminal software usage status data feature group.

2. The method according to claim 1, characterized in that The preset feature extraction algorithm is composed of a preset sliding window algorithm, a preset time proportion algorithm, a preset clustering algorithm and a preset data number statistical algorithm. The preset feature extraction algorithm is used to extract features from the terminal motion data, the terminal environment data and the terminal usage data to obtain a user behavior feature set, including: For the terminal motion data: using the preset sliding window algorithm, feature extraction is performed on the motion sensor data in the terminal motion data to obtain a motion sensor data feature group, wherein the motion sensor data feature group includes multiple types of motion sensor data feature values; using the preset time proportion algorithm, feature extraction is performed on the motion state duration data in the terminal motion data to obtain a motion state duration feature group, wherein the motion state duration data is the proportion of the duration of each type of motion state data in the terminal motion data to the total duration of the preset time period, and the motion state duration feature group includes multiple types of motion state duration feature values; For the terminal environment data: using the preset time proportion algorithm, feature extraction is performed on the terminal working environment duration data in the terminal environment data to obtain a working environment duration data feature group, wherein the working environment duration data feature group includes terminal working duration data feature values ​​under multiple working environment types; using the preset clustering algorithm, feature extraction is performed on the terminal geographic data in the terminal environment data to obtain a terminal location data feature group, wherein the terminal location data feature group includes multiple types of terminal location data feature values; For the terminal usage data: using the preset data number statistical algorithm, feature extraction is performed on the social data in the terminal usage data to obtain a social data feature group, wherein the social data feature group includes multiple types of social data feature values; using the preset data number statistical algorithm and the preset duration proportion algorithm, feature extraction is performed on the terminal usage data to obtain a terminal software usage status data feature group, wherein the terminal software usage status data feature group includes multiple types of terminal software usage status data feature values; The motion sensor data feature group, the motion state duration feature group, the working environment duration data feature group, the terminal location data feature group, the social data feature group and the terminal software usage status data feature group are merged to obtain the user behavior feature set.

3. The method according to claim 1, characterized in that The step of determining the emotion identification of the target user within the preset time period according to the processing result of the preset annular emotion recognition model includes: The area identifier corresponding to the processing result in the preset annular emotion model is determined as the emotion identifier, wherein the preset annular emotion model is a two-dimensional coordinate system, and the emotion identifier is a coordinate value in the two-dimensional coordinate system.

4. An emotion recognition system, characterized in that: The system comprises: An initial data acquisition module is used to acquire terminal motion data, terminal environment data, and terminal usage data of a target user within a preset period of time; A behavior feature extraction module, using a preset feature extraction algorithm, performs feature extraction on the terminal motion data, the terminal environment data and the terminal usage data to obtain a user behavior feature set; A behavior data acquisition module, using a preset normalization algorithm, performs normalization processing on the user behavior feature set to obtain a user behavior data set; A feature vector acquisition module, using a preset weight feature fusion algorithm, fuses various types of data in the user behavior data set in the vector dimension to obtain a user behavior feature vector; An emotion recognition module, used to input the user behavior feature vector into a preset cyclic emotion recognition model for processing, and determine the emotion identification of the target user within the preset time period according to the processing result of the preset cyclic emotion recognition model; The training process of the preset cyclic emotion recognition model includes: Obtaining sample terminal motion data, sample terminal environment data, sample terminal usage data and emotion quantification coordinate values ​​in each of the time periods; Using the preset feature extraction algorithm to extract features from the sample terminal motion data, the sample terminal environment data, and the sample terminal usage data of each time period, to obtain a sample user behavior feature set, using the preset normalization algorithm to perform the normalization processing on the sample user behavior feature set to obtain a sample user behavior data set of each time period, using a preset equal-weight feature fusion algorithm to fuse various types of data in the sample user behavior data set in a vector dimension to obtain a first sample user behavior feature vector of each time period, using the first sample user behavior feature vector of each time period and the emotion quantization coordinate value in the corresponding time period as initial training samples, to train an initial annular emotion recognition model to obtain an equal-weight annular emotion recognition model; Using the preset weight feature fusion algorithm, each type of data in the sample user behavior data set of each time period is fused in the vector dimension to obtain a second sample user behavior feature vector of each time period, and using the second sample user behavior feature vector of each time period and the emotion quantization coordinate value in the corresponding time period as the final training sample, the equal-weighted annular emotion recognition model is trained to obtain the preset annular emotion recognition model; The input of the preset annular emotion recognition model is the terminal motion data, the terminal environment data and the terminal usage data of the target user within the preset period, and the output is the emotion identification of the target user within the preset period; The feature vector acquisition module is configured as follows: Using the preset weight feature fusion algorithm, a first weight is assigned to the first type of user behavior data in the user behavior data set, and a second weight is assigned to the second type of user behavior data in the user behavior data combination, and the first type of user behavior data and the second type of user behavior data after the weights are assigned are fused in the vector dimension to obtain the user behavior feature vector, wherein the first weight is less than the second weight, the first type of user behavior data includes the data in the normalized motion sensor data feature group, the motion state duration feature group, the working environment duration data feature group and the terminal location data feature group, and the second type of user behavior data includes the data in the normalized social data feature group and the terminal software usage status data feature group.

5. The system according to claim 4, characterized in that The behavior feature extraction module is configured as follows: For the terminal motion data: using a preset sliding window algorithm, feature extraction is performed on the motion sensor data in the terminal motion data to obtain a motion sensor data feature group, wherein the motion sensor data feature group includes multiple types of motion sensor data feature values; using a preset time proportion algorithm, feature extraction is performed on the motion state duration data in the terminal motion data to obtain a motion state duration feature group, wherein the motion state duration data is the proportion of the duration of each type of motion state data in the terminal motion data to the total duration of the preset time period, and the motion state duration feature group includes multiple types of motion state duration feature values; For the terminal environment data: using the preset time proportion algorithm, feature extraction is performed on the terminal working environment duration data in the terminal environment data to obtain a working environment duration data feature group, wherein the working environment duration data feature group includes terminal working duration data feature values ​​under multiple working environment types; using a preset clustering algorithm, feature extraction is performed on the terminal geographic data in the terminal environment data to obtain a terminal location data feature group, wherein the terminal location data feature group includes multiple types of terminal location data feature values; For the terminal usage data: using a preset data number statistical algorithm, feature extraction is performed on the social data in the terminal usage data to obtain a social data feature group, wherein the social data feature group includes multiple types of social data feature values; using the preset data number statistical algorithm and the preset duration proportion algorithm, feature extraction is performed on the terminal usage data to obtain a terminal software usage status data feature group, wherein the terminal software usage status data feature group includes multiple types of terminal software usage status data feature values; The motion sensor data feature group, the motion state duration feature group, the working environment duration data feature group, the terminal location data feature group, the social data feature group and the terminal software usage status data feature group are merged to obtain the user behavior feature set.

6. An emotion recognition device, characterized in that: The device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the emotion recognition method as claimed in any one of claims 1 to 3.

7. A computer storage medium, characterized in that: When the instructions in the computer storage medium are executed by a processor of an electronic device, the device is enabled to perform the emotion recognition method according to any one of claims 1 to 3.

Images