Emotion analysis and state early warning system based on deep learning

Abstract

The invention discloses an emotion analysis and state early warning system based on deep learning, and solves the problem that an SVM classifier cannot accurately judge the emotion state of a person and cannot guarantee accurate recognition of complex states such as emotion due to the fact that only continuous face images are collected and only local information can be provided in an existing analysis method. The system comprises a user side, a model construction module, an emotion analysis module, a state evaluation module, a result judgment module and a psychological archive database. In the embodiment of the invention, the user side for collecting the multi-modal emotional information in real time is arranged, the multi-modal emotional information can be normalized through the user side, and the normalized information set after noise reduction and filtering is output, so that the emotional normalization model can effectively fuse and identify the multi-modal emotional information; and the emotion analysis and state recognition precision of the emotion normalization model and the state early warning model is ensured.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a sentiment analysis and status warning system based on deep learning. Background Art

[0002] Facial emotion monitoring is a biometric recognition technology that recognizes emotions based on facial images. Facial emotions are a state that integrates a person's feelings, thoughts, and behaviors, and to a certain extent, reflects a person's response to external or self-stimuli. Facial expression recognition technology has a wide range of applications in human-computer interaction, public safety monitoring, mental health diagnosis, student emotion analysis, virtual reality, and other fields. Therefore, it is very meaningful to recognize the emotional characteristics of human faces.

[0003] Chinese patent CN110765838A discloses a real-time dynamic analysis method of facial feature areas for emotional state monitoring, wherein, based on the Dlib face monitoring library and the Surrey face model, the method extracts spatial feature information of multiple preset facial feature points from a continuous facial image sequence of a monitored target individual, and obtains the color change characteristics of a specific facial skin area of ​​interest to discriminate the heart rate value of the target individual, and builds an SVM classifier based on the feature information of multiple preset feature points and the heart rate value of the target individual to determine the emotional distribution of the monitored target individual. However, the existing analysis method only collects continuous facial images and can only provide local information, so that the SVM classifier cannot accurately judge a person's emotional state, and cannot guarantee accurate recognition of complex states such as emotions. To address the above problems, we propose an emotional analysis and state warning system based on deep learning. Summary of the invention

[0004] The purpose of the present invention is to address the shortcomings of the prior art and provide an emotion analysis and status warning system based on deep learning, which solves the problem that the existing analysis method only collects continuous facial images and can only provide local information, making it impossible for the SVM classifier to accurately judge a person's emotional state, and unable to guarantee accurate recognition of complex states such as emotions.

[0005] The present invention is implemented in this way: a deep learning-based emotion analysis and status warning system, the deep learning-based emotion analysis and status warning system comprising:

[0006] The user end is used to obtain multimodal emotion information, normalize the multimodal emotion information, and output a normalized information set;

[0007] The model building module pre-builds the emotion normalization model based on deep learning combined with the cascaded posture regression model, uses hybrid enhancement technology to enhance the normalized information set, and divides the normalized information set into a training set and a test set. The training set and the test set are used to iteratively train the emotion normalization model and output a converged emotion normalization model.

[0008] The emotion analysis module collects multimodal data in real time, takes the multimodal data as input, executes the emotion normalization model, the emotion normalization model convolutionally fuses the multimodal data, outputs the emotion analysis result, and uploads the emotion analysis result to the psychological archive database.

[0009] The state evaluation module is used to traverse the psychological archive database, which issues a time-series sequence of emotion analysis results, analyze and judge the emotion analysis result sequence based on the pre-built state warning model, and output the state evaluation result;

[0010] The result judgment module presets a risk status threshold, judges whether the status assessment result exceeds the risk status threshold, and triggers a status warning instruction if it exceeds the risk status threshold. If it does not exceed the risk status threshold, it automatically generates an analysis result report and sends the analysis result report to the psychological archive database and the user end;

[0011] The psychological profile database is used to store multimodal emotion information, emotion analysis results, state assessment results and analysis result reports. The psychological profile database is respectively connected to the user end, the model building module, the emotion analysis module, the state assessment module and the result judgment module.

[0012] Preferably, the user terminal specifically includes:

[0013] A data collection unit for contactless and real-time collection of distributed multimodal emotion information;

[0014] A normalization unit, which loads multimodal emotion information and normalizes the multimodal emotion information;

[0015] The data sending unit is used to obtain a normalized information set and send the normalized information set to the model building module.

[0016] Preferably, the method for normalizing multimodal emotion information specifically includes:

[0017] Loading multimodal emotion information, extracting image and video information from the multimodal emotion information, and obtaining a face information set and a multimodal information set;

[0018] Obtain a face information set, identify the face information set using a Haar feature classifier, and output a face feature set;

[0019] Load the facial feature set, and perform face cropping and face enhancement on the facial feature set;

[0020] Load the multimodal information set and process missing values ​​and outliers on the multimodal information set;

[0021] Integrate the face feature set and the multimodal information set into an information integration set, perform iterative filtering and denoising on the information integration set based on a Gaussian denoising function, and output Gaussian denoising information;

[0022] Among them, Gaussian noise reduction information is expressed as:

[0023]

[0024] Among them, x(t) represents the Gaussian noise reduction information output value, f s represents the information sampling frequency, A represents the number of iterative filtering times, x(n) represents the input representation of the information integration set, σ represents the variance of the information integration set, B s represents the filter angle of the Gaussian function, Δω A represents the sampling angle difference of the information integration set, θ A is the signal phase representation;

[0025] Gaussian denoising information is obtained, and the deep features of the Gaussian denoising information are normalized and reduced in dimension using skip connections to obtain a normalized information set.

[0026] Preferably, the model building module specifically includes:

[0027] Model building unit: The model building unit pre-builds an emotion normalization model based on deep learning combined with a cascaded posture regression model. The emotion normalization model uses a cascaded posture regression model as an initial model, and introduces a multi-layer perceptron and an EfficientNet-B5 extraction structure into the initial model. The EfficientNet-B5 extraction structure is used for facial expression feature extraction, and the multi-layer perceptron is used for facial expression and speech feature fitting. The activation function of the initial model is a Gaussian error linear unit function.

[0028] An information set processing unit, which uses a hybrid enhancement technique to enhance the normalized information set and divides the normalized information set into a training set and a test set;

[0029] The model training unit uses the training set and the test set to iteratively train the emotion normalization model and outputs a converged emotion normalization model.

[0030] Preferably, the method of iteratively training the emotion normalization model using a training set and a test set specifically includes:

[0031] Obtain the training set, set the number of iterative training times and hyperparameters for the emotion normalization model, and iteratively train the emotion normalization model based on deep learning using the training set;

[0032] The emotion normalization model outputs the first probability value of the seven-category expression recognition results of the training set and the second probability value of the language recognition results combining the emotional expression, pleasure, activation, and dominance;

[0033] Among them, the seven-category expression recognition results are neutral, fear, surprise, happiness, anger, disgust, and sadness;

[0034] A multi-layer perceptron is used to fit and score the first probability value and the second probability value, and output a comprehensive probability value of emotion;

[0035] Obtaining a comprehensive probability value of emotion, calculating a loss value of an emotion normalization model, and performing error back propagation until the model converges, thereby obtaining at least one set of emotion normalization models;

[0036] A test set is obtained, and the model accuracy of at least one group of emotion normalization models is tested using the test set. The emotion normalization model with the highest accuracy is selected based on a preset accuracy threshold, and a converged emotion normalization model is output.

[0037] Preferably, the method of performing convolution fusion of multimodal data by the emotion normalization model and outputting emotion analysis results specifically includes:

[0038] Acquire multimodal data and normalize the multimodal data;

[0039] Taking the normalized multimodal data as input, the emotion normalization model is executed, and the EfficientNet-B5 extraction structure in the emotion normalization model extracts features to obtain a feature extraction set;

[0040] Load the feature extraction set, input the feature extraction set into the multi-layer perceptron to perform convolution fusion of expression features and speech features, and output the feature fusion set;

[0041] Obtain a feature fusion set, and calculate the probability values ​​of the expression features and speech features in the feature fusion set based on the attention mechanism, to obtain a first probability value of the expression recognition result and a second probability value of the language recognition result combining the emotional expression, pleasure, activation, and dominance;

[0042] The multi-layer perceptron performs weighted fitting on the first probability value and the second probability value to calculate the comprehensive probability value of the emotion;

[0043] Among them, the comprehensive probability value of emotion is calculated by the following formula:

[0044]

[0045] Among them, Act is the comprehensive probability value of emotion, λ1 and λ2 represent the weight coefficient of expression feature and voice feature respectively, and G V , G F Represent the second probability value and the first probability value respectively, M is the number of expression categories, N is the number of voice categories, and W V ,h V Respectively represent the speech feature attention weight matrix and speech feature attention vector, b V is the speech feature bias term, p V is the speech recognition label coefficient, t F , They represent the input value of expression features and the mean value of expression features, respectively, and mask F Represents the expression recognition mask coefficient.

[0046] Preferably, the status assessment module specifically includes:

[0047] An analysis sequence generation unit, the analysis sequence generation unit is used to traverse the psychological archive database, and the psychological archive database issues a time-series emotion analysis result sequence;

[0048] The early warning model building unit combines fuzzy clustering algorithm, genetic algorithm and regression scorer to build a status early warning model;

[0049] The state assessment unit analyzes and judges the emotion analysis result sequence based on the pre-built state warning model and outputs the state assessment result.

[0050] Preferably, the method for analyzing and judging the sequence of emotion analysis results based on the pre-built status warning model specifically includes:

[0051] Load the sentiment analysis result sequence, initialize at least one set of state cluster populations based on the genetic algorithm, and calculate the volatility of the state cluster populations;

[0052] Based on the state cluster population constraint condition, the fitness of at least one group of state cluster populations is sorted, the optimal state cluster population is selected according to the fitness, and the state cluster population is updated;

[0053] Combined with fuzzy clustering algorithm, the membership degree between the emotion analysis result sequence and the state clustering population is calculated;

[0054] Assigning the sentiment analysis result sequence to the nearest state cluster population based on the membership of the sentiment analysis result sequence and the state cluster population;

[0055] The regression scorer analyzes and judges the sentiment analysis result sequence based on the state classification scenario corresponding to the state clustering population, and outputs the state evaluation result.

[0056] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0057] In an embodiment of the present invention, a user terminal is provided for collecting multimodal emotional information in real time. The multimodal emotional information can be normalized through the user terminal, and a normalized information set after noise reduction filtering is output, so that the emotional normalization model can effectively integrate and identify multimodal emotional information, thereby ensuring the accuracy of emotional normalization model and state warning model emotional analysis and state recognition; and overcoming the problem that the existing analysis method only collects continuous facial images and can only provide local information, so that the SVM classifier cannot accurately judge a person's emotional state, and cannot guarantee accurate recognition of complex states such as emotions.

[0058] In an embodiment of the present invention, when normalizing multimodal emotional information, iterative filtering and denoising processing is performed on the information integration set based on the Gaussian denoising function, thereby effectively reducing the interference of noise in the multimodal emotional information, making the data purer and more accurate, and providing higher quality input data for the deep learning model, thereby improving the training effect and prediction performance of the model.

[0059] In an embodiment of the present invention, a model building module is provided, and the model building module is composed of a model building unit, an information set processing unit, and a model training unit. Through the coordinated cooperation of the model building unit, the information set processing unit, and the model training unit, an emotion normalization model based on deep learning combined with a cascade posture regression model can be built. The emotion normalization model extracts emotion and behavior characteristics through comprehensive analysis of multidimensional data to achieve accurate assessment of the individual's psychological state, and can cooperate with the status warning model to provide preliminary emotion clues for the status warning model. The status warning model can further integrate other information to make a more accurate judgment on the overall state of the person, thereby reducing the misjudgment that may exist in a single model. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a structural diagram of the deep learning-based emotion analysis and status warning system provided by the present invention.

[0061] Figure 2 A schematic diagram of the implementation process of the normalization processing method for multimodal emotion information is shown.

[0062] Figure 3 The figure shows a schematic diagram of the implementation process of the iterative training method of the emotion normalization model using a training set and a test set.

[0063] Figure 4 The figure shows a schematic diagram of the implementation process of the emotion normalization model for convolutional fusion of multimodal data and output of emotion analysis results.

[0064] Figure 5 The figure shows a schematic diagram of the implementation process of the method for analyzing and judging the sequence of sentiment analysis results based on the pre-built status warning model.

[0065] Figure 6 A schematic diagram of the implementation process of the sentiment analysis and status warning method based on deep learning is shown.

[0066] Figure 7 A schematic diagram of the operation interface of the psychological archive database is shown.

[0067] Figure 8 A schematic diagram of the analysis result report visualization is shown.

[0068] Fig. 9 A schematic diagram of the user terminal operation interface is shown.

[0069] Fig.10 A schematic diagram of the operating interface of the status assessment module is shown.

[0070] In the figure: 100-user end, 110-data acquisition unit, 120-normalization unit, 130-data sending unit, 200-model construction module, 210-model building unit, 220-information set processing unit, 230-model training unit, 300-emotion analysis module, 400-state evaluation module, 410-analysis sequence generation unit, 420-early warning model building unit, 430-state evaluation unit, 500-result judgment module, 600-psychological archive database. DETAILED DESCRIPTION

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of this application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0072] The existing analysis method only collects continuous facial images and can only provide local information, which makes it impossible for the SVM classifier to accurately judge a person's emotional state, and it is also impossible to guarantee accurate recognition of complex states such as emotions. To address the above problems, we propose an emotion analysis and state warning system based on deep learning. In short, the system consists of a user terminal 100, a model building module 200, an emotion analysis module 300, a state evaluation module 400, a result judgment module 500 and a psychological archive database 600. When the system is working, the user terminal 100 first obtains multimodal emotion information, normalizes the multimodal emotion information, and outputs a normalized information set. At the same time, the model building module 200 uses a training set and a test set to normalize the emotion. The model is iteratively trained to output a converged emotion normalization model. The emotion analysis module 300 collects multimodal data in real time, takes the multimodal data as input, executes the emotion normalization model, and the emotion normalization model convolutionally fuses the multimodal data to output the emotion analysis result. The state assessment module 400 traverses the psychological archive database 600, and the psychological archive database 600 issues a time-series emotion analysis result sequence. The emotion analysis result sequence is analyzed and judged based on the pre-built state warning model, and the state assessment result is output. The final result judgment module 500 judges whether the state assessment result exceeds the risk state threshold. If it exceeds the risk state threshold, the state warning instruction is triggered. If it does not exceed the risk state threshold, an analysis result report is automatically generated. In an embodiment of the present invention, a user terminal 100 is provided for collecting multimodal emotional information in real time. The multimodal emotional information can be normalized through the user terminal 100, and a normalized information set after noise reduction filtering can be output, so that the emotional normalization model can effectively integrate and identify multimodal emotional information, thereby ensuring the accuracy of emotional analysis and state recognition of the emotional normalization model and the state warning model; and overcoming the problem that the existing analysis method only collects continuous facial images and can only provide local information, so that the SVM classifier cannot accurately judge a person's emotional state, and cannot guarantee the accurate recognition of complex states such as emotions.

[0073] The embodiment of the present invention provides a deep learning-based emotion analysis and status warning system. Figure 1 The structure diagram of the emotion analysis and status warning system based on deep learning is shown, and the emotion analysis and status warning system based on deep learning specifically includes:

[0074] The user terminal 100 is used to obtain multimodal emotion information, normalize the multimodal emotion information, and output a normalized information set;

[0075] The model building module 200 pre-builds an emotion normalization model based on deep learning combined with a cascaded posture regression model, uses a hybrid enhancement technique to enhance the normalized information set, divides the normalized information set into a training set and a test set, uses the training set and the test set to iteratively train the emotion normalization model, and outputs a converged emotion normalization model;

[0076] The emotion analysis module 300 collects multimodal data in real time, takes the multimodal data as input, executes the emotion normalization model, the emotion normalization model convolutionally fuses the multimodal data, outputs the emotion analysis result, and uploads the emotion analysis result to the psychological profile database 600.

[0077] The state evaluation module 400 is used to traverse the psychological archive database 600, the psychological archive database 600 issues a time-series emotion analysis result sequence, analyze and judge the emotion analysis result sequence based on a pre-built state warning model, and output a state evaluation result;

[0078] The result judgment module 500 presets a risk state threshold, judges whether the state assessment result exceeds the risk state threshold, triggers a state warning instruction if it exceeds the risk state threshold, and automatically generates an analysis result report if it does not exceed the risk state threshold, and sends the analysis result report to the psychological profile database 600 and the user terminal 100;

[0079] The psychological profile database 600 is used to store multimodal emotion information, emotion analysis results, state assessment results and analysis result reports. The psychological profile database 600 is respectively communicated with the user terminal 100, the model building module 200, the emotion analysis module 300, the state assessment module 400, and the result judgment module 500.

[0080] It should be noted that the psychological profile database 600 can store multimodal emotion information, emotion analysis results, state assessment results and analysis result reports, and the psychological profile database 600 can establish and maintain personnel psychological profile big data, support long-term psychological state tracking and analysis, Figure 7 FIG. 6 shows a schematic diagram of the operation interface of the psychological profile database 600. Figure 8 A visual schematic diagram of the analysis result report is shown. The analysis result report displays the indicator scores in the form of bar charts and line charts. The analysis result report supports report export and online printing functions.

[0081] In an embodiment of the present invention, a user terminal 100 is provided for collecting multimodal emotional information in real time. The multimodal emotional information can be normalized through the user terminal 100, and a normalized information set after noise reduction filtering can be output, so that the emotional normalization model can effectively integrate and identify multimodal emotional information, thereby ensuring the accuracy of emotional analysis and state recognition of the emotional normalization model and the state warning model; and overcoming the problem that the existing analysis method only collects continuous facial images and can only provide local information, so that the SVM classifier cannot accurately judge a person's emotional state, and cannot guarantee the accurate recognition of complex states such as emotions.

[0082] In this embodiment, the user terminal 100 specifically includes:

[0083] A data collection unit 110, for contactless and real-time collection of distributed multimodal emotion information;

[0084] A normalization unit 120, which loads the multimodal emotion information and performs normalization processing on the multimodal emotion information;

[0085] The data sending unit 130 is used to obtain a normalized information set and send the normalized information set to the model building module 200 .

[0086] It should be noted that the data acquisition unit 110, the normalization unit 120, and the data sending unit 130 realize data interaction through Bluetooth, DTU or 5G communication connection. Fig. 9 The schematic diagram of the operation interface of the user terminal 100 is shown. Through the user terminal 100, the collection location, time, place and associated equipment of the distributed multimodal emotion information can be visualized. In this embodiment, the data collection unit 110 collects information in real time without contact through the expression collection camera and audio recording equipment. The expression collection camera: pixels ≥ 4 million, resolution 1080P; the expression collection camera supports software setting of focal length and focus. Its installation height is ≤ 2.5 meters; the overlooking angle is about 10%; the optimal shooting range is about 4 meters (Note: taking the focal length of 6mm as an example, the specific details are subject to the on-site environment and camera parameters).

[0087] The embodiment of the present invention provides a method for normalizing multimodal emotion information. Figure 2 The figure shows a schematic diagram of the implementation process of the method for normalizing multimodal emotion information, wherein the method for normalizing multimodal emotion information specifically includes:

[0088] Step S101, loading multimodal emotion information, extracting image and video information from the multimodal emotion information, and obtaining a face information set and a multimodal information set;

[0089] Step S102, obtaining a face information set, identifying the face information set in combination with a Haar feature classifier, and outputting a face feature set;

[0090] Step S103, loading a facial feature set, and performing face cropping and face enhancement processing on the facial feature set;

[0091] Step S104, loading the multimodal information set, and processing missing values ​​and outliers on the multimodal information set;

[0092] Step S105, integrating the facial feature set and the multimodal information set into an information integration set, iteratively filtering and denoising the information integration set based on a Gaussian denoising function, and outputting Gaussian denoising information;

[0093] Among them, Gaussian noise reduction information is expressed as:

[0094]

[0095] Among them, x(t) represents the Gaussian noise reduction information output value, f s represents the information sampling frequency, A represents the number of iterative filtering times, x(n) represents the input representation of the information integration set, σ represents the variance of the information integration set, B s represents the filter angle of the Gaussian function, Δω A represents the sampling angle difference of the information integration set, θ A is the signal phase representation;

[0096] Step S106, obtaining Gaussian noise reduction information, and using skip connections to normalize and reduce the dimension of deep features of the Gaussian noise reduction information to obtain a normalized information set.

[0097] In an embodiment of the present invention, when normalizing multimodal emotional information, iterative filtering and denoising processing is performed on the information integration set based on the Gaussian denoising function, thereby effectively reducing the interference of noise in the multimodal emotional information, making the data purer and more accurate, and providing higher quality input data for the deep learning model, thereby improving the training effect and prediction performance of the model.

[0098] The embodiment of the present invention provides a model building module 200, such as Figure 1 As shown, the model building module 200 specifically includes:

[0099] A model building unit 210, the model building unit 210 pre-builds an emotion normalization model based on deep learning combined with a cascaded posture regression model, wherein the emotion normalization model uses the cascaded posture regression model as an initial model, introduces a multi-layer perceptron and an EfficientNet-B5 extraction structure into the initial model, the EfficientNet-B5 extraction structure is used for facial expression feature extraction, the multi-layer perceptron is used for facial expression and speech feature fitting, and the activation function of the initial model is a Gaussian error linear unit function;

[0100] An information set processing unit 220, which uses a hybrid enhancement technique to enhance the normalized information set and divides the normalized information set into a training set and a test set;

[0101] The model training unit 230 uses the training set and the test set to iteratively train the emotion normalization model and output a converged emotion normalization model.

[0102] In an embodiment of the present invention, a model building module 200 is provided. The model building module 200 is composed of a model building unit 210, an information set processing unit 220, and a model training unit 230. Through the coordinated cooperation of the model building unit 210, the information set processing unit 220, and the model training unit 230, an emotion normalization model based on deep learning combined with a cascade posture regression model can be built. The emotion normalization model extracts emotion and behavior characteristics through comprehensive analysis of multidimensional data to achieve accurate assessment of the individual's psychological state, and can cooperate with the state warning model to provide preliminary emotion clues for the state warning model. The state warning model can further integrate other information to make a more accurate judgment on the overall state of the person, thereby reducing the possible misjudgment of a single model.

[0103] In this embodiment, the state assessment module 400 specifically includes:

[0104] An analysis sequence generating unit 410, the analysis sequence generating unit 410 is used to traverse the psychological profile database 600, and the psychological profile database 600 issues a time-series emotion analysis result sequence;

[0105] Early warning model building unit 420210, combining fuzzy clustering algorithm, genetic algorithm, and regression scorer to build a status early warning model;

[0106] The state evaluation unit 430 analyzes and judges the emotion analysis result sequence based on the pre-built state warning model, and outputs the state evaluation result.

[0107] It should be noted that the analysis sequence generation unit 410, the early warning model building unit 420210, and the state evaluation unit 430 in the state evaluation module 400 can be connected by Bluetooth or 5G communication. Fig.10A schematic diagram of the operation interface of the state evaluation module 400 is shown. Through the coordinated cooperation of the analysis sequence generation unit 410, the early warning model building unit 420210, and the state evaluation unit 430, real-time monitoring and alarming of key abnormal emotion indicators can be performed.

[0108] The embodiment of the present invention provides a method for iteratively training an emotion normalization model using a training set and a test set. Figure 3 The following is a schematic diagram of the implementation process of the iterative training method for the emotion normalization model using the training set and the test set. The method of iterative training the emotion normalization model using the training set and the test set specifically includes:

[0109] Step S201, obtaining a training set, setting the number of iterative training times and hyperparameters of the emotion normalization model, and iteratively training the emotion normalization model based on deep learning through the training set;

[0110] Step S202, the emotion normalization model outputs a first probability value of the seven-category expression recognition result of the training set and a second probability value of the language recognition result combining the emotion expression degree, pleasure degree, activation degree, and dominance degree;

[0111] Among them, the seven-category expression recognition results are neutral, fear, surprise, happiness, anger, disgust, and sadness;

[0112] Step S203, a multi-layer perceptron is used to fit and score the first probability value and the second probability value, and an emotional comprehensive probability value is output; in the embodiment of the present invention, 18 muscle movements of the human face are considered when fitting and scoring by a multi-layer perceptron, and micro-expression analysis (eyebrows raised, eye corners raised, mouth corners pulled down, chin contracted, etc.) is performed, and the emotional normalization model also supports dynamic and static detection, and native dynamic emotion recognition. 10 indicators such as aggressiveness, tension, pressure, suspicion, physical and mental balance, charm, and vitality are collected.

[0113] Step S204, obtaining a comprehensive emotion probability value, calculating a loss value of an emotion normalization model, and performing error back propagation until the model converges, thereby obtaining at least one set of emotion normalization models;

[0114] Step S205, obtaining a test set, using the test set to test the model accuracy of at least one group of emotion normalization models, selecting the emotion normalization model with the highest accuracy based on a preset accuracy threshold, and outputting a converged emotion normalization model. In this embodiment, the accuracy threshold is set to 0.95-0.98.

[0115] The embodiment of the present invention provides a method for convolutionally fusing multimodal data using an emotion normalization model to output emotion analysis results. Figure 4The figure shows a schematic diagram of the implementation process of the method for outputting the emotion analysis result by convolution fusion of the emotion normalization model on multimodal data. The method for outputting the emotion analysis result by convolution fusion of the emotion normalization model on multimodal data specifically includes:

[0116] Step S301, acquiring multimodal data and normalizing the multimodal data;

[0117] In this embodiment, the method for normalizing the multimodal data is the same as step S101 to step S106, and the specific steps may refer to step S101 to step S106.

[0118] Step S302, taking the normalized multimodal data as input, executing the emotion normalization model, extracting features using the EfficientNet-B5 extraction structure in the emotion normalization model, and obtaining a feature extraction set;

[0119] Step S303, loading a feature extraction set, inputting the feature extraction set into a multi-layer perceptron to perform convolution fusion of expression features and speech features, and outputting a feature fusion set.

[0120] Step S304, obtaining a feature fusion set, and calculating the probability values ​​of the expression features and speech features in the feature fusion set based on the attention mechanism, respectively, to obtain a first probability value of the expression recognition result, and a second probability value of the language recognition result combining the emotional expression, pleasure, activation, and dominance;

[0121] Step S305, a multilayer perceptron performs weighted fitting on the first probability value and the second probability value to calculate a comprehensive probability value of emotion;

[0122] Among them, the comprehensive probability value of emotion is calculated by the following formula:

[0123]

[0124] Among them, Act is the comprehensive probability value of emotion, λ1 and λ2 represent the weight coefficient of expression feature and voice feature respectively, and G V , G F Represent the second probability value and the first probability value respectively, M is the number of expression categories, N is the number of voice categories, and W V ,h V Respectively represent the speech feature attention weight matrix and speech feature attention vector, b V is the speech feature bias term, p V is the speech recognition label coefficient, t F , They represent the input value of expression features and the mean value of expression features, respectively, and mask F Represents the expression recognition mask coefficient.

[0125] In the embodiment of the present invention, the first probability value and the second probability value are weightedly fitted by a multi-layer perceptron to calculate the comprehensive probability value of emotions, and the comprehensive probability value of emotions is used to measure the quantitative statistical data of the emotional fluctuation of personnel. It reflects the relative value of emotions in the negative and positive ranges, and helps to grasp the direction and trend changes of emotions in time and space.

[0126] The embodiment of the present invention provides a method for analyzing and judging a sequence of emotion analysis results based on a pre-built status warning model. Figure 5 The schematic diagram of the implementation process of the method for analyzing and judging the sequence of emotion analysis results based on the pre-built state warning model is shown. The method for analyzing and judging the sequence of emotion analysis results based on the pre-built state warning model specifically includes:

[0127] Step S401, loading a sequence of sentiment analysis results, initializing at least one group of state cluster populations based on a genetic algorithm, and calculating the volatility of the state cluster populations;

[0128] Step S402, sorting the fitness of at least one group of state cluster populations based on the state cluster population constraint conditions, selecting the optimal state cluster population according to the fitness, and updating the state cluster population;

[0129] Step S403, calculating the membership degree of the emotion analysis result sequence and the state cluster population in combination with the fuzzy clustering algorithm;

[0130] Step S404, assigning the emotion analysis result sequence to the nearest state cluster population based on the membership degree between the emotion analysis result sequence and the state cluster population;

[0131] Step S405 , the regression scorer analyzes and judges the emotion analysis result sequence based on the state classification scenario corresponding to the state cluster population, and outputs the state evaluation result.

[0132] The embodiment of the present invention also provides a method for emotion analysis and status warning based on deep learning. Figure 6 The schematic diagram of the implementation process of the emotion analysis and status warning method based on deep learning is shown. The emotion analysis and status warning system based on deep learning specifically includes:

[0133] Step S10, acquiring multimodal emotion information, normalizing the multimodal emotion information, and outputting a normalized information set;

[0134] Step S20, pre-constructing an emotion normalization model based on deep learning combined with a cascaded posture regression model, using a hybrid enhancement technique to enhance the normalized information set, and dividing the normalized information set into a training set and a test set, using the training set and the test set to iteratively train the emotion normalization model, and outputting a converged emotion normalization model;

[0135] Step S30, real-time multimodal data is collected, and the emotion normalization model is executed with the multimodal data as input. The emotion normalization model convolutionally fuses the multimodal data, outputs the emotion analysis result, and uploads the emotion analysis result to the psychological profile database 600.

[0136] Step S40, traversing the psychological profile database 600, the psychological profile database 600 issues a time-series emotion analysis result sequence, analyzing and judging the emotion analysis result sequence based on a pre-built state warning model, and outputting a state evaluation result;

[0137] Step S50, determining whether the status assessment result exceeds the risk status threshold;

[0138] Step S60, if the risk status threshold is exceeded, triggering a status warning instruction;

[0139] Step S70 , if the risk status threshold is not exceeded, an analysis result report is automatically generated and sent to the psychological profile database 600 and the user terminal 100 .

[0140] On the other hand, an embodiment of the present invention further provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the computer program implementing any one of the above-mentioned methods when executed by the processor.

[0141] In this embodiment, processor CPU: Core i9 or domestic processor with equivalent performance or above; memory: 128GB or above; hard disk: solid state drive with 8TB or above; graphics card: RTX4000 or domestic GPU with equivalent performance or above.

[0142] Another aspect of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions, and the computer program instructions can be executed by a processor. When the computer program instructions are executed, the method of any one of the above embodiments is implemented.

[0143] As a non-volatile computer-readable storage medium, the memory can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the program instructions / modules corresponding to the deep learning-based emotion analysis and status warning method in the embodiment of the present application. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of the deep learning-based emotion analysis and status warning method, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the local module via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] The computer-readable storage medium (e.g., memory) herein can be a volatile memory or a nonvolatile memory, or can include both a volatile memory and a nonvolatile memory. As an example and not by way of limitation, a nonvolatile memory can include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. A volatile memory can include a random access memory (RAM), which can act as an external cache memory. As an example and not by way of limitation, RAM can be obtained in a variety of forms, such as synchronous RAM (DRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The storage device of the disclosed aspects is intended to include, but is not limited to, these and other suitable types of memories.

[0145] In summary, the present invention provides an emotion analysis and status warning system based on deep learning. In an embodiment of the present invention, a user terminal 100 is provided for real-time collection of multimodal emotion information. The multimodal emotion information can be normalized through the user terminal 100, and a normalized information set after noise reduction filtering is output, so that the emotion normalization model can effectively integrate and identify multimodal emotional information, thereby ensuring the emotion normalization model and the emotion analysis and status recognition accuracy of the status warning model; it overcomes the problem that the existing analysis method only collects continuous facial images and can only provide local information, so that the SVM classifier cannot accurately judge a person's emotional state, and cannot guarantee the accurate recognition of complex states such as emotions.

[0146] It should be noted that, for the above-mentioned embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, some steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0147] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be in the form of telecommunication or other forms.

[0148] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the scope of protection of the invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on these embodiments, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in this field can still combine, add, delete or make other adjustments to the features in the various embodiments of the present invention according to the circumstances without conflict, without making creative work, so as to obtain different other technical solutions that do not deviate from the concept of the present invention in essence, and these technical solutions also belong to the scope of protection of the present invention.

Claims

1. The emotion analysis and status warning system based on deep learning is characterized by: The deep learning-based emotion analysis and status warning system includes: The user end is used to obtain multimodal emotion information, normalize the multimodal emotion information, and output a normalized information set; The model building module pre-builds the emotion normalization model based on deep learning combined with the cascaded posture regression model, uses hybrid enhancement technology to enhance the normalized information set, and divides the normalized information set into a training set and a test set. The training set and the test set are used to iteratively train the emotion normalization model and output a converged emotion normalization model. The emotion analysis module collects multimodal data in real time, takes the multimodal data as input, executes the emotion normalization model, the emotion normalization model convolutionally fuses the multimodal data, outputs the emotion analysis result, and uploads the emotion analysis result to the psychological archive database.

2. The deep learning-based emotion analysis and status warning system according to claim 1, characterized in that: The deep learning-based emotion analysis and status warning system also includes: The state evaluation module is used to traverse the psychological archive database, which issues a time-series sequence of emotion analysis results, analyze and judge the emotion analysis result sequence based on the pre-built state warning model, and output the state evaluation result; The result judgment module presets a risk status threshold, judges whether the status assessment result exceeds the risk status threshold, and triggers a status warning instruction if it exceeds the risk status threshold. If it does not exceed the risk status threshold, it automatically generates an analysis result report and sends the analysis result report to the psychological archive database and the user end; The psychological profile database is used to store multimodal emotion information, emotion analysis results, state assessment results and analysis result reports. The psychological profile database is respectively connected to the user end, the model building module, the emotion analysis module, the state assessment module and the result judgment module.

3. The deep learning-based emotion analysis and status warning system according to claim 2, characterized in that: The user terminal specifically includes: A data collection unit for contactless and real-time collection of distributed multimodal emotion information; A normalization unit, which loads multimodal emotion information and normalizes the multimodal emotion information; The data sending unit is used to obtain a normalized information set and send the normalized information set to the model building module.

4. The deep learning-based emotion analysis and status warning system according to claim 3, characterized in that: The method for normalizing multimodal emotion information specifically includes: Loading multimodal emotion information, extracting image and video information from the multimodal emotion information, and obtaining a face information set and a multimodal information set; Obtain a face information set, identify the face information set using a Haar feature classifier, and output a face feature set; Load the facial feature set, and perform face cropping and face enhancement on the facial feature set; Load the multimodal information set and process missing values ​​and outliers on the multimodal information set; Integrate the face feature set and the multimodal information set into an information integration set, perform iterative filtering and denoising on the information integration set based on a Gaussian denoising function, and output Gaussian denoising information; Among them, Gaussian noise reduction information is expressed as: Among them, x(t) represents the Gaussian noise reduction information output value, f s represents the information sampling frequency, A represents the number of iterative filtering times, x(n) represents the input representation of the information integration set, σ represents the variance of the information integration set, B s represents the filter angle of the Gaussian function, Δω A represents the sampling angle difference of the information integration set, θ A is the signal phase representation; Gaussian denoising information is obtained, and the deep features of the Gaussian denoising information are normalized and reduced in dimension using skip connections to obtain a normalized information set.

5. The deep learning-based emotion analysis and status warning system according to claim 1, characterized in that: The model building module specifically includes: Model building unit: The model building unit pre-builds an emotion normalization model based on deep learning combined with a cascaded posture regression model. The emotion normalization model uses a cascaded posture regression model as an initial model, and introduces a multi-layer perceptron and an EfficientNet-B5 extraction structure into the initial model. The EfficientNet-B5 extraction structure is used for facial expression feature extraction, and the multi-layer perceptron is used for facial expression and speech feature fitting. The activation function of the initial model is a Gaussian error linear unit function. An information set processing unit, which uses a hybrid enhancement technique to enhance the normalized information set and divides the normalized information set into a training set and a test set; The model training unit uses the training set and the test set to iteratively train the emotion normalization model and outputs a converged emotion normalization model.

6. The deep learning-based emotion analysis and status warning system according to claim 5, characterized in that: The method of iteratively training the emotion normalization model using the training set and the test set specifically includes: Obtain the training set, set the number of iterative training times and hyperparameters for the emotion normalization model, and iteratively train the emotion normalization model based on deep learning using the training set; The emotion normalization model outputs the first probability value of the seven-category expression recognition results of the training set and the second probability value of the language recognition results combining the emotional expression, pleasure, activation, and dominance; Among them, the seven-category expression recognition results are neutral, fear, surprise, happiness, anger, disgust, and sadness; A multi-layer perceptron is used to fit and score the first probability value and the second probability value, and output a comprehensive probability value of emotion; Obtaining a comprehensive probability value of emotion, calculating a loss value of an emotion normalization model, and performing error back propagation until the model converges, thereby obtaining at least one set of emotion normalization models; A test set is obtained, and the model accuracy of at least one group of emotion normalization models is tested using the test set. The emotion normalization model with the highest accuracy is selected based on a preset accuracy threshold, and a converged emotion normalization model is output.

7. The deep learning-based emotion analysis and status warning system according to claim 6, characterized in that: The method of the emotion normalization model convolutionally fusing multimodal data and outputting emotion analysis results specifically includes: Acquire multimodal data and normalize the multimodal data; Taking the normalized multimodal data as input, the emotion normalization model is executed, and the EfficientNet-B5 extraction structure in the emotion normalization model extracts features to obtain a feature extraction set; Load the feature extraction set, input the feature extraction set into the multi-layer perceptron for convolution fusion of expression features and speech features, and output the feature fusion set.

8. The deep learning-based emotion analysis and status warning system according to claim 7, characterized in that: The method of performing convolution fusion of multimodal data by the emotion normalization model and outputting emotion analysis results specifically includes: Obtain a feature fusion set, and calculate the probability values ​​of the expression features and speech features in the feature fusion set based on the attention mechanism, to obtain a first probability value of the expression recognition result and a second probability value of the language recognition result combining the emotional expression, pleasure, activation, and dominance; The multi-layer perceptron performs weighted fitting on the first probability value and the second probability value to calculate the comprehensive probability value of the emotion; Among them, the comprehensive probability value of emotion is calculated by the following formula: Among them, Act is the comprehensive probability value of emotion, λ1 and λ2 represent the weight coefficient of expression feature and voice feature respectively, and G V , G F Represent the second probability value and the first probability value respectively, M is the number of expression categories, N is the number of voice categories, and W V ,h V Respectively represent the speech feature attention weight matrix and speech feature attention vector, b V is the speech feature bias term, p V is the speech recognition label coefficient, t F , t F They represent the input value of expression features and the mean value of expression features, respectively, and mask F Represents the mask coefficient for expression recognition.

9. The deep learning-based emotion analysis and status warning system according to claim 1, characterized in that: The state assessment module specifically includes: An analysis sequence generation unit, the analysis sequence generation unit is used to traverse the psychological archive database, and the psychological archive database issues a time-series emotion analysis result sequence; The early warning model building unit combines fuzzy clustering algorithm, genetic algorithm and regression scorer to build a status early warning model; The state assessment unit analyzes and judges the emotion analysis result sequence based on the pre-built state warning model and outputs the state assessment result.

10. The deep learning-based emotion analysis and status warning system according to claim 9, characterized in that: The method for analyzing and judging the sequence of sentiment analysis results based on the pre-built status warning model specifically includes: Load the sentiment analysis result sequence, initialize at least one set of state cluster populations based on the genetic algorithm, and calculate the volatility of the state cluster populations; Based on the state cluster population constraint condition, the fitness of at least one group of state cluster populations is sorted, the optimal state cluster population is selected according to the fitness, and the state cluster population is updated; Combined with fuzzy clustering algorithm, the membership degree between the emotion analysis result sequence and the state clustering population is calculated; Assigning the sentiment analysis result sequence to the nearest state cluster population based on the membership of the sentiment analysis result sequence and the state cluster population; The regression scorer analyzes and judges the sentiment analysis result sequence based on the state classification scenario corresponding to the state clustering population, and outputs the state evaluation result.

Images