Voice user emotion recognition method and system based on AI intelligent neural network

Abstract

The present application provides a method and system for voice user emotion recognition based on AI intelligent neural network, which relates to the technical field of voice recognition. The method includes steps such as data collection, first modeling, first training and emotion recognition; the system includes modules such as data collection module, first modeling module, first training module and emotion recognition module. The present application trains the emotion recognition model based on historical voice data and historical driving data, which can improve the accuracy of identifying the driver's emotions.

Description

Technical Field

[0001] The present application relates to the technical field of speech recognition, and in particular to a method and system for speech user emotion recognition based on an AI intelligent neural network. Background Art

[0002] With the rapid development of science and technology, intelligent driving assistance systems have gradually become an important part of modern cars. These systems not only improve driving safety and comfort, but also provide users with a more intelligent driving experience by integrating various advanced technologies such as autonomous driving, voice recognition, gesture control, etc. However, while pursuing technological advancement, how to better understand and respond to the driver's emotional state has become an urgent problem to be solved.

[0003] The driver's emotional state has a significant impact on driving behavior. For example, when drivers are in an angry or anxious emotional state, they may make more aggressive or dangerous driving decisions, thereby increasing the risk of traffic accidents. On the contrary, when drivers are in a pleasant or relaxed emotional state, they may be more cautious and calm in dealing with various driving scenarios. Therefore, accurately identifying and understanding the driver's emotional state is crucial to improving the safety and user experience of intelligent driving assistance systems. Summary of the invention

[0004] In order to improve the accuracy of identifying the driver's emotions, the present application provides a voice user emotion recognition method and system based on AI intelligent neural network.

[0005] In the first aspect, the present application provides a method for voice user emotion recognition based on AI intelligent neural network, which adopts the following technical solutions:

[0006] The method for voice user emotion recognition based on AI intelligent neural network includes the following steps:

[0007] Data collection: including the first collection and the second collection;

[0008] First collection: collect historical voice data, emotion labels and real-time voice data;

[0009] Second collection: collect historical driving data and real-time driving data;

[0010] First modeling: establishing an emotion recognition model, wherein the emotion recognition model includes: a BI-LSTM sub-model and a naive Bayes sub-model;

[0011] First training: including preliminary training and retraining to obtain the trained emotion recognition model;

[0012] Initial training: Use historical driving data, historical speech data, and emotion labels to train the BI-LSTM sub-model and output emotion features;

[0013] Retraining: Use the emotion features and emotion labels output by the BI-LSTM sub-model to train the Naive Bayes sub-model to obtain the trained Naive Bayes sub-model;

[0014] Emotion recognition: Input real-time speech data and real-time driving data into the trained emotion recognition model to obtain emotion recognition results.

[0015] By adopting the above technical solution, the present application trains the BI-LSTM sub-model through historical driving data, historical voice data and emotion labels, so that the emotion recognition model can learn the association between driving behavior and emotion categories, as well as the mapping relationship between voice features and emotion categories, in order to predict emotional features. In addition, the BI-LSTM sub-model is good at processing data with a time sequence, and the BI-LSTM sub-model can also capture the driver's emotional changes. The present application also uses historical voice data and emotion labels to train the Naive Bayes sub-model, so that the emotion recognition model has the ability to recognize emotions for emotional features, which helps the emotion recognition model to more accurately identify the driver's emotions. The present application uses voice data and driving data to train the emotion recognition model, so that the emotion recognition model can more comprehensively understand the user's emotional changes and improve the accuracy of emotion recognition.

[0016] Optionally, after executing the step of data collection and before executing the step of first modeling, the method further includes:

[0017] Second modeling: establishing SVM model;

[0018] Data elimination: including feature extraction, second training, weight acquisition and feature elimination;

[0019] Feature extraction: extract the features of historical driving data, recorded as driving features; extract the features of historical voice data, recorded as voice features, and integrate the driving features and voice features into a feature set;

[0020] Second training: The emotion labels are used as labels for driving features and speech features in the feature set, and the feature set is used to train the SVM model;

[0021] Obtain weights: Based on the trained SVM model, obtain the weight of each feature;

[0022] Feature elimination: Delete the features whose weight is less than the preset weight threshold, obtain the remaining features, use the remaining features as the new feature set, and perform the second training step until the stopping condition is met;

[0023] Acquire data: acquire the remaining features in the data elimination step, acquire the historical driving data and / or historical voice data corresponding to the remaining features, and perform the first training step as new historical driving data and / or historical voice data.

[0024] By adopting the above technical solution, the present application obtains the weight of each feature based on the trained SVM model, can identify which features are more critical for distinguishing different emotion categories, delete features whose weights are less than the preset weight threshold, and use the historical driving data and / or historical voice data corresponding to the remaining features to train the emotion recognition model, so that the emotion recognition model is more focused on important historical driving data and / or historical voice data, further improving the accuracy of emotion recognition. By deleting some historical driving data and / or historical voice data, the training data is reduced and the training speed of the emotion recognition model is improved.

[0025] Optionally, after performing the feature extraction step and before performing the second training step, the method further includes:

[0026] The third collection: collecting historical traffic data and real-time traffic data, and extracting the features of the historical traffic data, which are recorded as traffic features;

[0027] Update feature set: Add traffic features to the feature set.

[0028] By adopting the above-mentioned technical solution, the driver's emotions are easily affected by road conditions. When traffic is congested, the probability of the driver experiencing negative emotions will increase. This application collects and integrates road condition data and extracts its features to enhance the integrity and diversity of the feature set, which helps to improve the integrity of emotion recognition model training, improve the accuracy and adaptability of emotion recognition, and also enhance the robustness of the emotion recognition model, so that it can better cope with emotion recognition needs under different road conditions.

[0029] Optionally, after executing the step of data elimination and before executing the step of acquiring data, the method further includes:

[0030] Calculate correlation: Calculate the correlation between the road condition features and the eliminated features;

[0031] Correlation judgment: judging whether the correlation is greater than a preset correlation threshold, if so, executing the step of filling data; if not, executing the step of acquiring data;

[0032] Data filling: adding the driving features corresponding to the correlation degree back into the feature set.

[0033] By adopting the above technical solution, after eliminating the features whose weights are less than the preset weight threshold, the present application recalculates the correlation between the eliminated features and the road condition features, and determines whether the eliminated features need to be added back to the feature set based on the correlation, so that the features that are highly correlated with the road condition features will not be completely ignored due to the data elimination step. Re-adding the eliminated features can enrich the content of the feature set, reduce information loss, and improve the diversity of the feature set, thereby enhancing the correlation between the feature set and the emotion recognition task, thereby helping the emotion recognition model to more comprehensively capture the driver's emotional changes under different road conditions.

[0034] Optionally, in the emotion recognition step, real-time traffic data, real-time voice data and real-time driving data are used as inputs of the SVM model to obtain the remaining features, and the real-time driving data and real-time voice data corresponding to the remaining features are used as inputs of the emotion recognition model to obtain new emotion recognition results.

[0035] By adopting the above technical solution and inputting real-time road condition data into the SVM model, it is possible to further screen the real-time driving data and real-time voice data through the road condition data, thereby reducing the probability of accidental deletion, making the data more complete, and enhancing the generalization of the emotion recognition model.

[0036] Optionally, after executing the step of data collection and before executing the step of first modeling, the method further includes:

[0037] Fourth collection: collecting key data in the historical voice data and / or historical driving data, and recording it as the first data;

[0038] Data mining: Based on the FP-Growth algorithm, frequent items in the first data are mined.

[0039] By adopting the above technical solution, the present application sets key data in historical speech data and / or historical driving data to obtain the most relevant information for a specific emotion category, which helps to reduce noise data and improve the training efficiency and accuracy of the emotion recognition model. Before the first modeling step, by adding the fourth acquisition and data mining step, the data processing work is greatly enhanced, which not only improves the quality of the training data, but also provides a more effective feature set for the first training step. The present application also provides a more effective and concise data representation for the emotion recognition model by mining frequent items, further reducing the amount of input data and improving the training speed of the emotion recognition model.

[0040] Optionally, after performing the first modeling step and before performing the first training step, the method further includes:

[0041] Calculate the ratio: Get the infrequent items in the first data, and calculate the ratio of frequent items to infrequent items. The calculation model is as follows:

[0042] ;

[0043] ;

[0044] in, is the proportion of frequent items; A is the number of infrequent items in the first data; B is the number of frequent items in the first data; is the proportion of infrequent items; is a constant;

[0045] Update training data: according to the proportion of frequent items Adjust the proportion of key data in historical driving data and / or historical voice data; according to the proportion of infrequent items Adjust the proportion of non-critical data in the historical driving data and / or historical voice data.

[0046] By adopting the above technical solution, by calculating the ratio of frequent items and infrequent items, their distribution in the first data (i.e., key data) can be quantified, and the proportion of the data with a small proportion can be increased. The proportion of key data in historical driving data and / or historical voice data is adjusted according to the proportion of frequent items, so that the emotion recognition model pays more attention to those features that have an important impact on tasks such as emotion recognition during training. Adjusting the proportion of non-critical data according to the proportion of infrequent items helps reduce the risk of overfitting the emotion recognition model to frequent items and improve the generalization ability of the model. This application optimizes the distribution of training data by setting a step of updating training, increases the proportion of smaller amounts of data in historical data, and reduces data bias.

[0047] Optionally, the emotion recognition model further includes an attention submodule, and after performing the first modeling step and before performing the first training step, further includes:

[0048] Calculate relevance score: The attention submodule calculates the relevance score between the input data and the current task;

[0049] Weighted summation: the attention submodule performs weighted summation on each data in the input data according to the relevance score to obtain the contribution value of the input data to the current task, wherein the input data includes: historical driving data and / or historical speech data;

[0050] Sorting: Sorting the input data according to the contribution value in descending order to obtain new historical driving data and / or new historical voice data.

[0051] By adopting the above technical solution, the attention submodule can calculate the relevance score between the input data and the current task (emotion recognition), which enables the emotion recognition model to focus on the data part that is most relevant to the task. In this way, the emotion recognition model can ignore those data that are irrelevant to the task or have low relevance, thereby making more efficient use of limited computing resources. The attention mechanism weights each feature in the input data according to the relevance score by weighted summation, which helps to highlight those features that contribute more to the task, so that the emotion recognition model can learn more accurate and effective feature representations, thereby improving the prediction performance of the model. The relevance score is dynamically calculated based on the relevance between the input data and the current task, which means that the emotion recognition model can dynamically adjust its weight according to different input data. This dynamic adjustment mechanism enables the model to respond to different input situations more flexibly and improve the adaptability and robustness of the model. The weighted summation process is actually the process of combining the features in the input data. Through weighted summation, the emotion recognition model can learn the interactions and relationships between different features, which helps the model capture more complex and subtle data features, thereby improving the prediction accuracy of the model. The input data is sorted according to the contribution value, and the emotion recognition model is trained with the input data with high contribution value first. This allows the emotion recognition model to quickly learn the relationship between the input data and the prediction results, thereby improving the training efficiency.

[0052] Optionally, after performing the weighted summing step and before performing the sorting step, the method further includes:

[0053] Deletion: Delete input data whose contribution value is less than the preset contribution value threshold and obtain new input data.

[0054] By adopting the above technical solution, input data with smaller contribution values ​​contain less information related to the current task, and may even contain noise or interference information. By deleting these data with smaller contribution values, the impact of noise on model training can be reduced and the robustness of the model can be improved. The deletion step enables the model to focus more on those data parts that contribute more to the task. These data parts contain more information related to the current task, which helps the emotion recognition model learn more accurate and effective feature representations. By performing the deletion step, the amount of input data is reduced, which helps to speed up the sorting process. After deleting the data with smaller contribution values, the remaining data is more focused on key features, and the distribution of the sorted data is more in line with the training requirements of the emotion recognition model, which helps the model better learn the potential laws and patterns in the data.

[0055] In a second aspect, the present invention provides a voice user emotion recognition system based on AI intelligent neural network, which adopts the following technical solutions:

[0056] The voice user emotion recognition system based on AI intelligent neural network includes:

[0057] The data acquisition module includes a first acquisition unit and a second acquisition unit;

[0058] The first collection unit is used to collect historical voice data, emotion tags and real-time voice data;

[0059] A second collection unit, used for collecting historical driving data and real-time driving data;

[0060] A first modeling module is used to establish an emotion recognition model, wherein the emotion recognition model includes: a BI-LSTM sub-model and a naive Bayes sub-model;

[0061] A first training module, including a preliminary training unit and a retraining unit, wherein the first training module is in communication with the data acquisition module and the first modeling module to obtain a trained emotion recognition model;

[0062] A preliminary training unit, which is used to train the BI-LSTM sub-model using historical driving data, historical speech data and emotion labels, and output emotion features;

[0063] A retraining unit, used for training the Naive Bayes sub-model using the emotion features and emotion labels output by the BI-LSTM sub-model to obtain a trained Naive Bayes sub-model;

[0064] The emotion recognition module is connected to the data acquisition module and the first training module for inputting the real-time voice data and the real-time driving data into the trained emotion recognition model to obtain the emotion recognition result.

[0065] By adopting the above technical solution, the system realizes an efficient and accurate emotion recognition system through the collaborative work of the data acquisition module, the first modeling module, the first training module and the emotion recognition module. The system can make full use of diverse data resources and quickly and accurately identify the user's emotional state through advanced model structure and training strategies.

[0066] In summary, the present application includes at least one of the following beneficial technical effects:

[0067] 1. This application trains the BI-LSTM sub-model through historical driving data, historical voice data and emotion labels, so that the emotion recognition model can learn the relationship between driving behavior and emotion categories, as well as the mapping relationship between voice features and emotion categories, in order to predict emotional features. In addition, the BI-LSTM sub-model is good at processing data with a time sequence, and the BI-LSTM sub-model can also capture the driver's emotional changes. This application also uses historical voice data and emotion labels to train the Naive Bayes sub-model, so that the emotion recognition model has the ability to recognize emotion features, which helps the emotion recognition model to more accurately identify the driver's emotions. This application uses voice data and driving data to train the emotion recognition model, so that the emotion recognition model can more comprehensively understand the user's emotional changes and improve the accuracy of emotion recognition.

[0068] 2. The present application sets key data in historical speech data and / or historical driving data to obtain the most relevant information for a specific emotion category, which helps to reduce noise data and improve the training efficiency and accuracy of the emotion recognition model. Before the first modeling step, by adding the fourth acquisition and data mining step, the data processing work is greatly enhanced, which not only improves the quality of the training data, but also provides a more effective feature set for the first training step. The present application also provides a more effective and concise data representation for the emotion recognition model by mining frequent items, further reducing the amount of input data and improving the training speed of the emotion recognition model.

[0069] 3. The attention submodule can calculate the relevance score between the input data and the current task (emotion recognition), which enables the emotion recognition model to focus on the data that is most relevant to the task. In this way, the emotion recognition model can ignore data that is irrelevant to the task or has low relevance, thereby making more efficient use of limited computing resources. The attention mechanism weights each feature in the input data according to the relevance score by weighted summation, which helps to highlight the features that contribute more to the task, so that the emotion recognition model can learn more accurate and effective feature representations, thereby improving the prediction performance of the model. The relevance score is dynamically calculated based on the relevance between the input data and the current task, which means that the emotion recognition model can dynamically adjust its weight according to different input data. This dynamic adjustment mechanism enables the model to respond to different input situations more flexibly and improve the adaptability and robustness of the model. The weighted summation process is actually the process of combining the features in the input data. Through weighted summation, the emotion recognition model can learn the interactions and relationships between different features, which helps the model capture more complex and subtle data features, thereby improving the prediction accuracy of the model. The input data is sorted according to the contribution value, and the emotion recognition model is trained with the input data with high contribution value first. This allows the emotion recognition model to quickly learn the relationship between the input data and the prediction results, thereby improving the training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is a flow chart of Example 1 of the present application;

[0071] Figure 2 This is a flow chart from S2 first modeling to S3 first training in Example 2 of the present application;

[0072] Figure 3 This is a flow chart from S1 data collection to S3 first training in Example 2 of the present application;

[0073] Figure 4 is a flow chart of Example 3 of the present application;

[0074] Figure 5 This is a flow chart of Example 4 of the present application. DETAILED DESCRIPTION

[0075] The following combination Figures 1 to 5 This application is described in further detail.

[0076] Embodiment 1: This embodiment discloses a method for voice user emotion recognition based on AI intelligent neural network, referring to Figure 1The recognition method includes: S1 data collection, S2 first modeling, S3 first training and S4 emotion recognition. First, relevant historical data and real-time data are collected, then an emotion recognition model is established, then the emotion recognition model is trained using historical data, and then the real-time data is input into the trained emotion recognition model to obtain an emotion recognition result. This embodiment includes the following steps:

[0077] S1 data collection, including S11 first collection and S12 second collection.

[0078] S11 is the first step of collection, which collects historical voice data, emotion tags, and real-time voice data. Historical voice data refers to voice information recorded in the past, which is used to train the emotion recognition model to identify various emotion features. Emotion tags are associated with historical voice data, and they identify the emotions expressed by each voice segment (such as happiness, sadness, anger, etc.). Real-time voice data refers to the voice information currently received, which is used to identify real-time emotions.

[0079] S12 collects historical driving data and real-time driving data. Historical driving data records driving behaviors under different emotional conditions in the past, such as vehicle speed, acceleration, braking frequency, etc. These driving data are used to analyze the relationship between driving behaviors and different emotional categories, for example, acceleration and braking are more frequent when angry. Real-time driving data is the currently monitored driving behavior information, which will be used to help identify the driver's emotional state.

[0080] S2 first models the emotion recognition model, wherein the emotion recognition model includes a BI-LSTM sub-model and a naive Bayes sub-model, wherein the output of the BI-LSTM sub-model is connected to the input of the naive Bayes sub-model.

[0081] The BI-LSTM (Bidirectional Long Short-Term Memory Network) sub-model is good at processing sequence data and can capture long-term dependencies in speech signals, thereby extracting emotion-related emotional features. The Naive Bayes sub-model is a simple probabilistic classifier that receives the output of the BI-LSTM sub-model as input and classifies emotions based on these emotional features. This combination enables the emotion recognition model to learn complex feature representations while maintaining a fast classification speed.

[0082] S3 first training, including S31 preliminary training and S32 retraining, this step is to obtain the trained emotion recognition model.

[0083] S31 preliminary training uses historical driving data, historical voice data and emotion labels to train the BI-LSTM sub-model. The trained BI-LSTM sub-model can learn the emotion-related features in the voice data and driving data to predict the emotion features corresponding to various data. The process is as follows:

[0084] Collect and organize historical driving data, historical voice data and corresponding emotion labels, use the historical voice data and historical driving data as the input of the BI-LSTM sub-model, and adjust the parameters of the BI-LSTM sub-model through the back-propagation algorithm so that it can extract emotion-related features from the input data from a time series perspective. The output of the BI-LSTM sub-model is a high-dimensional vector representing emotion features.

[0085] S32 retraining uses the emotion features and emotion labels output by the BI-LSTM sub-model to train the Naive Bayes sub-model to obtain the trained Naive Bayes sub-model. The process is as follows:

[0086] Use the emotion features and corresponding emotion labels extracted by the trained BI-LSTM sub-model as the training data for the Naive Bayes sub-model (use the emotion feature vectors extracted from the BI-LSTM model as the input features of the Naive Bayes sub-model, and use the emotion labels corresponding to these features as the output labels), calculate the conditional probability and prior probability of each category, and evaluate the performance of the Naive Bayes sub-model using methods such as cross-validation or holdout method.

[0087] S4 emotion recognition, inputs real-time voice data and real-time driving data into the trained emotion recognition model to obtain emotion recognition results.

[0088] This embodiment uses historical data to train the BI-LSTM sub-model of the emotion recognition model to extract emotion features in time series, and then trains the Naive Bayes sub-model of the emotion recognition model based on the emotion features and emotion labels to achieve the training of the emotion recognition model. Then, the current data is input into the trained emotion recognition model to obtain the emotion recognition result. This embodiment takes into account the temporal connection of historical data to extract emotion features, and uses the Naive Bayes sub-model to classify the emotion features, which can perform emotion recognition more accurately and quickly.

[0089] Example 2: Reference Figure 2 The difference between this embodiment and embodiment 1 is that the emotion recognition model further includes an attention submodule, the output of the attention submodel is connected to the input of the BI-LSTM submodel, and after executing S2 first modeling and before executing S3 first training, it also includes:

[0090] S21 calculates the relevance score. The attention submodule evaluates the importance of each element in the input data to the current task through the attention mechanism. The process is as follows: the attention submodule uses a neural network to calculate the relevance score of each element in the input data (such as vehicle speed, acceleration, etc. in historical driving data, or volume, speaking rate, etc. in historical voice data) with the current task.

[0091] S22 weighted summation, the attention submodule performs weighted summation on each element in the input data according to the relevance score (data that is more relevant to the current task will obtain a higher weight and thus occupy a larger proportion in the final summation result), and obtains the contribution value of the input data to the current task, wherein the input data includes: historical driving data and / or historical voice data.

[0092] S23 deletes, deleting input data whose contribution value is less than a preset contribution value threshold, and obtaining new input data.

[0093] S24 sorts, sorts the new input data in S23 according to the contribution value in descending order, and the sorted input data (i.e., new historical driving data and / or new historical voice data) will be used as the input of the BI-LSTM sub-model for further feature extraction and emotion recognition.

[0094] This embodiment introduces the attention submodule, so that the emotion recognition model can more accurately capture the key information related to emotions in the input data, thereby improving the accuracy and efficiency of recognition. In addition, the attention submodule can also be dynamically adjusted according to the importance of the input data, so that the emotion recognition model can maintain the best performance when processing data in different situations.

[0095] Example 3: Reference Figure 3 The difference between this embodiment and embodiment 1 is that after executing S1 data collection and before executing S2 first modeling step, it also includes:

[0096] S51 is the second modeling, which is to establish an SVM model. In this embodiment, the output of the SVM model is connected to the input of the emotion recognition model to evaluate the importance of each data.

[0097] S52 data elimination includes S521 feature extraction, S522 third collection, S523 updating feature set, S524 second training, S525 obtaining weights and S526 feature elimination.

[0098] S521 feature extraction, extracting features of historical driving data, recorded as driving features; extracting features of historical voice data, recorded as voice features, and integrating driving features and voice features into a feature set.

[0099] The driving characteristics include vehicle speed, acceleration and braking frequency; the voice characteristics include volume, speaking speed and tone.

[0100] S522 is the third collection, collecting historical road condition data and real-time road condition data, and extracting features of the historical road condition data, which are recorded as road condition features. The road condition features include: road conditions, other vehicle conditions, weather conditions, etc.

[0101] S523 updates the feature set, and adds the road condition feature to the feature set. The road condition feature does not carry an emotion label, so the road condition feature is not used for the second training in S524 to train the SVM model.

[0102] S524 The second training uses the features with emotion labels in the feature set (i.e., driving features and speech features) to train the SVM model.

[0103] S525 obtains weights. Based on the trained SVM model, obtains the weight of each feature (limited to driving features and voice features). The weight represents the importance of the feature in the SVM model decision-making process.

[0104] S526 feature elimination, delete the features whose weights are less than the preset weight threshold (limited to driving features and voice features), obtain the remaining features, use the remaining features as the new feature set, execute S524 second training, until the stop condition is met and execute S53 to obtain data.

[0105] The stopping condition includes: S524 the number of executions of the second training reaches a preset execution number threshold or the performance of the SVM model is no longer significantly improved.

[0106] S53 acquires data, acquires features remaining in the data elimination in S52, acquires historical driving data and / or historical voice data corresponding to the remaining features, and performs S3 first training as new historical driving data and / or historical voice data.

[0107] This embodiment performs effective preprocessing and feature screening on the input data before the first modeling in S2, thereby eliminating historical data, thereby improving the performance and accuracy of the emotion recognition model, reducing the training resources for the subsequent S3 first training, and improving the training speed.

[0108] Reference Figure 4 In other embodiments, after executing the step of S52 data elimination and before executing the step of S53 data acquisition, the method further includes:

[0109] S54 calculates the correlation, and calculates the correlation between the road condition feature and the eliminated feature. The correlation between the road condition feature and the eliminated driving feature and / or voice feature helps to identify those features that have a low weight when used alone but are closely related to the road condition feature. These features have a significant impact on the driver's emotions or behaviors under specific road conditions, and therefore need to be reconsidered.

[0110] S55 correlation judgment, judging whether the correlation degree is greater than a preset correlation degree threshold, if so, executing S56 data filling; if not, executing S53 data acquisition.

[0111] S56 data filling, adding the driving features corresponding to the correlation degree back into the feature set, and in the step of acquiring data in S53, not only the remaining features but also the features re-added into the feature set are acquired, and the historical driving data and / or historical voice data corresponding to the above features are used as new historical driving data and / or historical voice data.

[0112] In the emotion recognition step, real-time traffic data, real-time voice data and real-time driving data are used as inputs of the SVM model to obtain features (including the features remaining after S52 data is eliminated and the features re-added after S56 data filling), and the real-time driving data and real-time voice data corresponding to the features are used as inputs of the emotion recognition model to obtain new emotion recognition results.

[0113] For example, the driver often travels during the day and only travels at night once. The road condition features include darkness, unclear roads, and the high beams of vehicles in the opposite lane are too bright. It is necessary to calculate the correlation between the road condition features and the removed features (assuming that the removed driving features are a speed of 60 km / h, 10 m / s2, and a braking frequency of 0.7, and the removed speech features are a volume of 59, a speaking speed of 264 words / minute, and a falling tone). In this example, the emotional label of the removed feature is anxiety.

[0114] S54 calculates the correlation, and respectively calculates the correlation between the road condition feature, the driving feature, and the voice feature. The process is as follows:

[0115] All features are encoded: the road condition feature is encoded as [0.1, 0.5, 0.3]; the driving feature is encoded as [0.9, 0.7, 0.4]; and the speech feature is encoded as [0.2, 0.4, 0.3].

[0116] The process of calculating the correlation between the road condition feature, the driving feature, and the speech feature using cosine similarity is as follows:

[0117] The correlation between road condition characteristics and driving characteristics is:

[0118] ;

[0119] ;

[0120] ;

[0121] .

[0122] Similarly, the correlation between road condition features and speech features is:

[0123] ;

[0124] ;

[0125] ;

[0126] .

[0127] It is determined whether 0.79 and 0.97 exceed the preset correlation threshold. If so, the above-mentioned driving features and speech features are added to the feature set again, and S53 is continued to be executed to obtain data.

[0128] This embodiment can improve the performance of the emotion recognition model while minimizing information loss and improving the generalization ability and accuracy of the emotion recognition model.

[0129] Example 4: Reference Figure 5 The difference between this embodiment and embodiment 1 is that after executing S1 data collection and before executing S2 first modeling, it also includes:

[0130] S61 is a fourth collection, in which key data in the historical voice data and / or historical driving data are collected and recorded as first data, wherein the key data is the historical voice data and / or historical driving data under negative emotions.

[0131] S62 data mining, based on the FP-Growth algorithm, mines frequent items in the first data, where frequent items refer to items that frequently appear in historical voice data and / or historical driving data, and they represent common behavior patterns or characteristics of the driver under negative emotions.

[0132] As in Example 1, S2 first modeling is performed, but before S3 first training is performed, the method further includes:

[0133] S71 calculates the ratio, obtains the infrequent items in the first data, calculates the number of the frequent items and the infrequent items, and further calculates the ratio of the frequent items and the infrequent items. The calculation model is as follows:

[0134] ;

[0135] ;

[0136] in, is the proportion of frequent items; A is the number of infrequent items in the first data; B is the number of frequent items in the first data; is the proportion of non-frequent items; the proportion of frequent items and the proportion of infrequent items During calculations, only two significant figures need to be retained.

[0137] is a constant, a preset value, whose value is less than 1×10 -3 , the purpose is to avoid the problem of calculation errors caused by the absence of frequent items or infrequent items in the first data, and will not have a significant impact on the proportion of frequent items and infrequent items in the original data. In this embodiment, its value is 1×10 -5 In other embodiments, It can also be set to other values.

[0138] S72 updates the training data according to the proportion of frequent items Adjust the proportion of key data in historical driving data and / or historical voice data; according to the proportion of infrequent items Adjust the proportion of non-critical data in the historical driving data and / or historical voice data.

[0139] In the process of executing S72 to update the training data, the proportion of the key data and / or the non-key data in the historical driving data and / or the historical voice data is adjusted in the following manner:

[0140] The proportion of fixed key data in historical driving data and / or historical speech data is equal to the proportion of frequent items , and then based on the proportion of infrequent items Calculate the amount of non-critical data in the historical driving data and / or historical voice data, and record it as the second data. If the amount of non-critical data is less than the second data, use a generative adversarial network to generate a portion of non-critical data and add it to the historical driving data and / or historical voice data to ensure that the proportion of non-critical data in the historical driving data and / or historical voice data is equal to the proportion of non-frequent items. Otherwise, undersampling is used to filter the infrequent items until the proportion of non-critical data in the historical driving data and / or historical voice data is equal to the proportion of infrequent items. .

[0141] In other embodiments, the proportion of non-critical data in the historical driving data and / or historical voice data may be fixed to be equal to the proportion of infrequent items. , adjust the quantity of key data.

[0142] This embodiment adjusts the proportion of key data in historical driving data and / or historical voice data by using the number of frequent items and infrequent items, which helps to adjust the proportion of key data and non-key data in historical driving data and / or historical voice data, increase the proportion of the small amount of data, and optimize the structure of the training set.

[0143] Embodiment 5: This embodiment discloses a voice user emotion recognition system based on AI intelligent neural network, the system comprising:

[0144] The data acquisition module includes a first acquisition unit and a second acquisition unit.

[0145] The first collection unit is used to collect historical voice data, emotion tags and real-time voice data, wherein the historical voice data is marked with emotion tags, and the emotion tags include: happiness, sadness, anger, anxiety, etc., and the voice data includes: volume, speaking speed, intonation, etc.

[0146] The second collection unit is used to collect historical driving data and real-time driving data, wherein the historical driving data includes vehicle speed, acceleration, braking frequency, etc.

[0147] The first modeling module is used to establish an emotion recognition model, which includes a BI-LSTM sub-model and a naive Bayes sub-model. The BI-LSTM sub-model uses historical driving data and / or historical voice data to extract emotion features by capturing long-term dependencies in the data. The naive Bayes sub-model uses the emotion features and emotion labels output by the BI-LSTM sub-model for further training to improve the accuracy of emotion recognition.

[0148] The first training module includes a preliminary training unit and a retraining unit. The first training module is connected to the data acquisition module and the first modeling module in communication to obtain a trained emotion recognition model.

[0149] The preliminary training unit is used to train the BI-LSTM sub-model using historical driving data, historical speech data, and emotion labels to output emotion features. This unit also adjusts the BI-LSTM sub-model parameters through the back-propagation algorithm until the BI-LSTM sub-model can accurately extract emotion features.

[0150] The retraining unit is used to take the emotion features output by the BI-LSTM sub-model as input and the emotion label as the target output to realize the training of the Naive Bayes sub-model.

[0151] The emotion recognition module is connected to the data acquisition module and the first training module for inputting the real-time voice data and the real-time driving data into the trained emotion recognition model to obtain the emotion recognition result.

[0152] The system collects historical and real-time voice and driving data through the data acquisition module, and uses the first modeling module to build an emotion recognition model including a BI-LSTM sub-model and a naive Bayes sub-model. The historical data is trained through the preliminary training unit and the retraining unit in the first training module. Finally, the emotion recognition module inputs the real-time data into the trained model to quickly output the emotion recognition results, thereby providing support for emotion monitoring during intelligent driving.

[0153] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for voice user emotion recognition based on AI intelligent neural network, characterized in that: include: Data collection: including the first collection and the second collection; First collection: collect historical voice data, emotion labels and real-time voice data; Second collection: collect historical driving data and real-time driving data; First modeling: establishing an emotion recognition model, wherein the emotion recognition model includes: a BI-LSTM sub-model and a naive Bayes sub-model; First training: including preliminary training and retraining to obtain the trained emotion recognition model; Initial training: Use historical driving data, historical speech data, and emotion labels to train the BI-LSTM sub-model and output emotion features; Retraining: Use the emotion features and emotion labels output by the BI-LSTM sub-model to train the Naive Bayes sub-model to obtain the trained Naive Bayes sub-model; Emotion recognition: Input real-time speech data and real-time driving data into the trained emotion recognition model to obtain emotion recognition results.

2. The method for voice user emotion recognition based on AI intelligent neural network according to claim 1 is characterized in that: After executing the step of data collection and before executing the step of first modeling, the method further includes: Second modeling: establishing SVM model; Data elimination: including feature extraction, second training, weight acquisition and feature elimination; Feature extraction: extract the features of historical driving data and record them as driving features; extract the features of historical voice data and record them as voice features, and integrate the driving features and voice features into a feature set; Second training: The emotion labels are used as labels for driving features and speech features in the feature set, and the feature set is used to train the SVM model; Obtain weights: Based on the trained SVM model, obtain the weight of each feature; Feature elimination: Delete the features whose weight is less than the preset weight threshold, obtain the remaining features, use the remaining features as the new feature set, and perform the second training step until the stopping condition is met; Acquire data: acquire the remaining features in the data elimination step, acquire the historical driving data and / or historical voice data corresponding to the remaining features, and perform the first training step as new historical driving data and / or historical voice data.

3. The method for voice user emotion recognition based on AI intelligent neural network according to claim 2 is characterized in that: After performing the feature extraction step and before performing the second training step, the method further includes: The third collection: collecting historical traffic data and real-time traffic data, and extracting the features of the historical traffic data, which are recorded as traffic features; Update feature set: Add traffic features to the feature set.

4. The method for voice user emotion recognition based on AI intelligent neural network according to claim 3 is characterized in that: After executing the step of data elimination and before executing the step of obtaining data, the method further includes: Calculate correlation: Calculate the correlation between the road condition features and the eliminated features; Correlation judgment: judging whether the correlation is greater than a preset correlation threshold, if so, executing the step of filling data; if not, executing the step of acquiring data; Data filling: adding the driving features corresponding to the correlation degree back into the feature set.

5. The method for voice user emotion recognition based on AI intelligent neural network according to claim 4 is characterized in that: In the emotion recognition step, real-time traffic data, real-time speech data and real-time driving data are used as inputs of the SVM model to obtain the remaining features, and the real-time driving data and real-time speech data corresponding to the remaining features are used as inputs of the emotion recognition model to obtain new emotion recognition results.

6. The method for voice user emotion recognition based on AI intelligent neural network according to any one of claims 1 to 5, characterized in that: After executing the step of data collection and before executing the step of first modeling, the method further includes: Fourth collection: collecting key data in the historical voice data and / or historical driving data, and recording it as the first data; Data mining: Based on the FP-Growth algorithm, frequent items in the first data are mined.

7. The method for voice user emotion recognition based on AI intelligent neural network according to claim 6 is characterized in that: After executing the first modeling step and before executing the first training step, the method further includes: Calculate the ratio: Get the infrequent items in the first data, and calculate the ratio of frequent items to infrequent items. The calculation model is as follows: ； ； in, is the proportion of frequent items; A is the number of infrequent items in the first data; B is the number of frequent items in the first data; is the proportion of infrequent items; is a constant; Update training data: according to the proportion of frequent items Adjust the proportion of key data in historical driving data and / or historical voice data; according to the proportion of infrequent items Adjust the proportion of non-critical data in the historical driving data and / or historical voice data.

8. The method for voice user emotion recognition based on AI intelligent neural network according to any one of claims 1 to 5, characterized in that: The emotion recognition model further includes an attention submodule, and after performing the first modeling step and before performing the first training step, further includes: Calculate relevance score: The attention submodule calculates the relevance score between the input data and the current task; Weighted summation: the attention submodule performs weighted summation on each data in the input data according to the relevance score to obtain the contribution value of the input data to the current task, wherein the input data includes: historical driving data and / or historical speech data; Sorting: Sorting the input data according to the contribution value in descending order to obtain new historical driving data and / or new historical voice data.

9. The method for voice user emotion recognition based on AI intelligent neural network according to claim 8 is characterized in that: After performing the weighted summation step and before performing the sorting step, the method further includes: Deletion: Delete input data whose contribution value is less than the preset contribution value threshold and obtain new input data.

10. A voice user emotion recognition system based on AI intelligent neural network, the system is applicable to the method according to any one of claims 1 to 9, characterized in that: include: The data acquisition module includes a first acquisition unit and a second acquisition unit; The first collection unit is used to collect historical voice data, emotion tags and real-time voice data; A second collection unit, used for collecting historical driving data and real-time driving data; A first modeling module is used to establish an emotion recognition model, wherein the emotion recognition model includes: a BI-LSTM sub-model and a naive Bayes sub-model; A first training module, including a preliminary training unit and a retraining unit, wherein the first training module is in communication with the data acquisition module and the first modeling module to obtain a trained emotion recognition model; A preliminary training unit, which is used to train the BI-LSTM sub-model using historical driving data, historical speech data and emotion labels, and output emotion features; A retraining unit, used for training the Naive Bayes sub-model using the emotion features and emotion labels output by the BI-LSTM sub-model to obtain a trained Naive Bayes sub-model; The emotion recognition module is connected to the data acquisition module and the first training module for inputting the real-time voice data and the real-time driving data into the trained emotion recognition model to obtain the emotion recognition result.

Images