Patient emotion analysis device based on multiple modes

Abstract

The invention provides a patient emotion analysis device based on multiple modes. The device is suitable for emotion calculation, mental health monitoring and related application fields. According to the device, through a dynamic modal selection mechanism, a hierarchical fusion strategy, multi-modal interaction analysis, emotional evolution analysis and a real-time emotional feedback function, the accuracy, the real-time performance and the application range of emotional analysis are remarkably improved. Specifically, the device comprises a sample data extraction and preprocessing module, a dynamic modal selection module, a processing module, a hierarchical fusion module, a filtering and interaction analysis module, an emotion evolution analysis module, a real-time emotion feedback module and a classification result generation module. By dynamically selecting an optimal modal combination, hierarchically fusing multi-modal data, analyzing interaction among modals and monitoring emotion changes in real time, the problems of information redundancy, conflict, static analysis and the like in traditional multi-modal emotion analysis are solved, and more efficient and accurate emotion analysis is realized.

Description

Technical Field

[0001] The present invention relates to the field of medical equipment technology, and in particular to a multimodal patient emotion analysis device. Background Art

[0002] With the rapid development of artificial intelligence technology, sentiment analysis, as an important research direction, has been widely used in many fields. Sentiment analysis technology is mainly used to extract and identify people's emotional states from various forms of data such as text, voice, and images, and then classify emotions. This technology is not only crucial for business decision-making, marketing, product design, etc., but also plays an increasingly important role in mental health monitoring, medical diagnosis, social interaction and other fields.

[0003] Existing multimodal sentiment analysis techniques mainly improve the accuracy of sentiment recognition by fusing multiple types of data (such as text, audio, images, etc.). However, traditional multimodal sentiment analysis methods face a series of challenges in practical applications:

[0004] 1. Information redundancy and conflict: Since the information carried by different modalities may overlap or contradict each other, how to effectively integrate this information becomes a difficult problem. If the problem of information redundancy and conflict is not properly handled, it will directly affect the accuracy of sentiment classification.

[0005] 2. Limitations of static analysis: Most existing multimodal sentiment analysis techniques are trained and tested based on static data sets, which means they often fail to capture the characteristics of emotions changing over time. In actual application scenarios, people's emotions change over time, and this dynamic characteristic is difficult to cover with static analysis.

[0006] 3. Weight allocation between modalities: The data quality of different modalities varies. How to reasonably allocate the weight of each modality in sentiment analysis is also an important issue. If the data quality of a certain modality is poor, directly fusing it with other high-quality modal data may reduce the overall sentiment analysis performance.

[0007] 4. Lack of personalized analysis capabilities: Existing multimodal sentiment analysis technologies often ignore individual differences, that is, different people express emotions in very different ways. Failure to consider this factor may lead to weak generalization of analysis results, especially in cross-cultural or cross-population application scenarios.

[0008] 5. Real-time requirements: In some application scenarios, such as online customer service and intelligent assistants, the results of sentiment analysis need to be fed back to users or systems in real time, which requires sentiment analysis technology to have high real-time processing capabilities.

[0009] In view of the existence of the above problems, there is an urgent need to develop a new multimodal-based sentiment analysis device to effectively solve the problems of information redundancy and conflict, realize real-time sentiment monitoring and analysis, and at the same time have high accuracy and flexibility to adapt to different application scenarios and personal characteristics.

[0010] In view of this, the inventor specially designed a multimodal patient emotion analysis device, and this case was thus generated. Summary of the invention

[0011] In order to solve the above problems, the technical solution of the present invention is as follows:

[0012] A patient emotion analysis device based on multimodality, comprising:

[0013] A sample data extraction and preprocessing module, used to extract text data, audio data, and image data from multimodal samples and perform preprocessing, wherein the preprocessing includes but is not limited to data standardization, noise filtering, and feature enhancement;

[0014] The dynamic mode selection module is used to calculate the Score according to the quality of sample data and the complementarity between modes through the modal weight calculation formula m =α m ×Quality m +β m ×Complementarity m Select the optimal mode combination, where α m and β m is the adjustment coefficient, Quality m is the modal quality, Complementarity m score for modal complementarity;

[0015] A processing module, used for inputting the modal data output by the dynamic modality selection module into a multimodal conditional variational autoencoder for processing to extract emotional features;

[0016] The hierarchical fusion module is used to fuse single-modal local features first and then cross-modal global features during feature fusion. The fusion calculation formula is: and Fusion global =Attention(Fusion local ), where ω i is the local feature weight, and Attention is the cross-modal attention mechanism;

[0017] The filtering and interaction analysis module is used to filter redundant information and analyze the interaction relationship between modalities through cross-modal attention mechanism and self-attention mechanism;

[0018] The emotion evolution analysis module is used to analyze the dynamic changes of emotions over time through a time series model. The calculation formula of the time series model is h t =σ(W x x t +W h h t-1 +b), where h t is the current emotional state, x t is the input feature at the current moment, W x and W h are weight matrices, and b is the bias vector;

[0019] Real-time sentiment feedback module, used to provide instant feedback and visual display of sentiment classification results;

[0020] The classification result generation module is used to generate the final sentiment classification result based on the filtered fusion features, and to perform correction and optimization in combination with the evolutionary analysis results.

[0021] Preferably, the dynamic mode selection module realizes automatic selection through a mode weight calculation network, and the network calculates the mode weight according to the formula Score m =α m ×Quality m +β m ×Complementarity m Calculate the weight of each mode.

[0022] Preferably, the multimodal conditional variational autoencoder in the processing module includes a text autoencoder, an audio autoencoder and an image autoencoder, which encode and decode data of different modalities respectively.

[0023] Preferably, the hierarchical attention mechanism used by the hierarchical fusion module includes a local feature attention mechanism and a global feature attention mechanism, which perform weighted fusion on unimodal local features and cross-modal global features respectively.

[0024] Preferably, the weight ω of the local feature fusion i Calculated by the following formula:

[0025]

[0026] Among them, e i is the score of feature i, and exp is the exponential function.

[0027] Preferably, the filtering and interaction analysis module uses a cross-modal attention mechanism to calculate the attention score matrix A between different modalities, and the calculation formula is:

[0028]

[0029] Among them, S ij is the similarity score between modality i and modality j.

[0030] Preferably, the sentiment evolution analysis module uses a long short-term memory network LSTM as a time series model, and its calculation formula is:

[0031] h t =σ(W x x t +W h h t-1 +b)

[0032] Among them, h t is the hidden state at the current moment, x t is the current input feature, W x and W h is the weight matrix and b is the bias term.

[0033] Preferably, the emotion evolution result generated by the long short-term memory network LSTM is input into the classification result generation module to correct the final emotion classification result.

[0034] Preferably, the real-time emotion feedback module includes an API interface and a visualization tool for displaying emotion classification results and emotion change trends in real time.

[0035] Preferably, the classification result generation module generates the final sentiment classification result by the following formula:

[0036] y=softmax(W f ×h+b f )

[0037] Among them, W f is the fused feature matrix, h is the hidden layer output, b f is the bias term and softmax is the activation function, which is used to generate the final classification probability distribution.

[0038] The beneficial effects of the present invention are as follows:

[0039] The present invention improves processing efficiency and analysis accuracy, avoids interference from low-quality modal data, optimizes the fusion effect of multimodal data through a layered fusion strategy, significantly improves the accuracy of sentiment analysis, and further enhances the accuracy of sentiment classification by analyzing the interaction between modalities. It realizes dynamic analysis of sentiment over time, adapts to the needs of sentiment monitoring in dynamic scenes, and provides instant sentiment status feedback. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0041] in:

[0042] Figure 1 is an overview diagram of the system architecture of the present invention;

[0043] Figure 2 is a dynamic mode selection flow chart of the present invention;

[0044] Figure 3 It is a schematic diagram of hierarchical feature fusion of the present invention;

[0045] Figure 4 It is a multimodal information filtering and interactive analysis diagram of the present invention;

[0046] Figure 5 is the emotion evolution over time analysis diagram of the present invention;

[0047] Figure 6 It is a design diagram of the real-time emotional feedback user interface of the present invention;

[0048] Figure 7 It is a flow chart of generating sentiment classification results of the present invention. DETAILED DESCRIPTION

[0049] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0050] See also Figure 1-7 The present invention provides a multi-modal sentiment analysis device, which significantly improves the accuracy, real-time nature and application breadth of sentiment analysis through dynamic modality selection mechanism, hierarchical fusion strategy, multi-modal interaction analysis, sentiment evolution analysis and real-time sentiment feedback function. The specific implementation methods of each module are introduced in detail below:

[0051] 1. Sample data extraction and preprocessing module

[0052] This module is used to extract text, audio, and image data from multimodal samples and preprocess them. The specific steps are as follows:

[0053] (1) Text data extraction and preprocessing:

[0054] Use natural language processing (NLP) technology to perform word segmentation and word embedding on text data.

[0055] Convert text into vector representation to facilitate subsequent sentiment feature extraction.

[0056] The TF-IDF (Term Frequency-Inverse Document Frequency) algorithm is used to calculate the weight of the text to highlight the importance of keywords.

[0057] (2) Audio data extraction and preprocessing

[0058] Perform noise reduction on audio signals to remove background noise.

[0059] Extract acoustic features of audio, such as Mel-Frequency Cepstral Coefficients (MFCC), energy, pitch, etc.

[0060] Standardize the audio data to make it meet the requirements of subsequent processing.

[0061] (3) Image data extraction and preprocessing

[0062] Perform image enhancement processing, such as brightness adjustment, contrast enhancement, etc.

[0063] Use convolutional neural network (CNN) to extract features from images and obtain key features in the images.

[0064] Normalize the image data to ensure data consistency.

[0065] 2. Dynamic mode selection module

[0066] This module dynamically selects the most representative modal data for subsequent processing based on the quality of input samples, task requirements, and complementarity between modalities. The specific steps are as follows:

[0067] (1) Modal weight calculation

[0068] Each modality is scored using a pre-trained modality weight computation network.

[0069] The modal weight calculation formula is:

[0070] Score m =α m ×Quality m +β m ×Complementarity m

[0071] Among them, α m and β m is the adjustment coefficient, Quality m is the modal quality, Complementaritym Score for modal complementarity.

[0072] Based on the calculated scores, the modality with the highest score is selected for further processing.

[0073] 3. Processing module

[0074] The processing module receives the unimodal data output by the dynamic mode selection module and inputs it into the improved multimodal conditional variational autoencoder (VAE) for processing. The specific steps are as follows:

[0075] (1) Text Autoencoder

[0076] Use Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU) to encode text sequences.

[0077] Extract the sentiment features of the text and generate a text feature vector.

[0078] (2) Audio Autoencoder

[0079] A convolutional neural network (CNN) is used to encode audio features.

[0080] Extract the emotional features of the audio and generate an audio feature vector.

[0081] (3) Image Autoencoder

[0082] A convolutional neural network (CNN) is used to encode image features.

[0083] Extract the emotional features of the image and generate the image feature vector.

[0084] 4. Layered Fusion Module

[0085] This module adopts a hierarchical fusion strategy, first fusing the local features of each modality, and then fusing the global features across modalities. The specific steps are as follows:

[0086] (1) Local feature fusion

[0087] Use the local attention mechanism to fuse features within the same modality.

[0088] The local feature fusion formula is:

[0089]

[0090] Among them, ω i is the local feature weight, Feature i is the ith feature.

[0091] (2) Global feature fusion

[0092] The global attention mechanism is used to fuse cross-modal global features.

[0093] The global feature fusion formula is:

[0094] Fusion global =Attention(Fusion local )

[0095] Among them, Attention is a cross-modal attention mechanism.

[0096] 5. Filtering and interactive analysis module

[0097] This module uses a cross-modal attention mechanism to calculate the attention score matrix between different modalities to filter out redundant information.

[0098] The specific steps are as follows:

[0099] (1) Cross-modal attention mechanism

[0100] Calculate the attention score matrix A between different modalities, and its calculation formula is:

[0101]

[0102] Among them, S ij is the similarity score between modality i and modality j.

[0103] (2) Interaction Analysis

[0104] Use multimodal cross-attention mechanism to analyze and understand the interaction between modalities and further optimize the sentiment classification results.

[0105] 6. Emotional evolution analysis module

[0106] This module is used to analyze the dynamic changes of emotions over time, and is particularly suitable for long-term emotion monitoring scenarios. The specific steps are as follows:

[0107] (1) Timing Model

[0108] Use Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU) as the timing model.

[0109] The calculation formula of the timing model is:

[0110] h t =σ(W x x t +W h h t-1 +b)

[0111] Among them, h t is the hidden state at the current moment, x t is the current input feature, W x and W h is the weight matrix and b is the bias term.

[0112] (2) Emotional evolution analysis

[0113] The time series model is used to capture emotional changes and output an emotional fluctuation curve to help users or medical personnel understand the evolution of emotions.

[0114] 7. Real-time emotional feedback module

[0115] This module provides instant feedback on sentiment analysis and displays the analysis results in real time through the API interface or built-in visualization tools. The specific steps are as follows:

[0116] (1) Real-time feedback

[0117] The sentiment classification results are displayed in real time on the user interface, and alarms are issued for abnormal sentiment fluctuations.

[0118] The regional distribution of emotions is displayed through the emotion heat map.

[0119] 8. Classification result generation module

[0120] This module receives the output from the filtering and interaction analysis module, combines it with the sentiment evolution analysis results, and generates the final sentiment classification results through a fully connected neural network. The specific steps are as follows:

[0121] (1) Classification result generation

[0122] A fully connected neural network is used to generate the final sentiment classification results.

[0123] The classification result generation formula is:

[0124] y=softmax(W f ×h+b f )

[0125] Among them, W f is the fused feature matrix, h is the hidden layer output, b f is the bias term and softmax is the activation function, which is used to generate the final classification probability distribution.

[0126] The implementation steps are as follows:

[0127] Sample data extraction and preprocessing

[0128] The system extracts multimodal samples from the user's text, audio, and image data. Text data is vectorized through word segmentation and word embedding; audio data generates Mel-spectrogram coefficients through noise filtering and feature extraction; image data is enhanced and visual features are extracted through image enhancement technology.

[0129] Dynamic modal selection

[0130] The processing module calculates the scoring results of the network according to the modal weights and selects the modal combination that best suits the current sentiment analysis task. For example, when the audio data quality is high and the image is blurry, audio and text modalities are given priority for analysis.

[0131] Hierarchical fusion and multimodal interaction analysis

[0132] In the fusion process, local features of data of the same modality are first fused, and then global features are fused across modalities. The adopted hierarchical attention mechanism can better handle the complex relationships within the modality.

[0133] Emotional evolution analysis

[0134] In long conversations or video surveillance scenarios, the emotion evolution analysis module uses a time series model to capture emotion changes and outputs an emotion fluctuation curve to help users or medical personnel understand the evolution of emotions.

[0135] Real-time emotional feedback

[0136] The system's real-time emotion feedback module displays the emotion classification results in real time on the user interface, issues alarms for abnormal emotion fluctuations, and displays the regional distribution of emotions through emotion heat maps.

[0137] In summary, the present invention provides a sentiment analysis device based on multimodality, which significantly improves the accuracy, real-time performance and application breadth of sentiment analysis through a dynamic modal selection mechanism, a hierarchical fusion strategy, a multimodal interaction analysis, a sentiment evolution analysis and a real-time sentiment feedback function. It can effectively solve the problems of information redundancy, conflict and static analysis in traditional multimodal sentiment analysis, and achieve more efficient and accurate sentiment analysis.

[0138] The present invention is described above by way of example in conjunction with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned method. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention, or the concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A multimodal patient emotion analysis device, characterized in that: include: A sample data extraction and preprocessing module, used to extract text data, audio data, and image data from multimodal samples and perform preprocessing, wherein the preprocessing includes but is not limited to data standardization, noise filtering, and feature enhancement; The dynamic mode selection module is used to calculate the Score according to the quality of sample data and the complementarity between modes through the modal weight calculation formula m =α m ×Quality m +β m ×Complementarity m Select the optimal mode combination, where α m and β m is the adjustment coefficient, Quality m is the modal quality, Complementarity m score for modal complementarity; A processing module, used for inputting the modal data output by the dynamic modality selection module into a multimodal conditional variational autoencoder for processing to extract emotional features; The hierarchical fusion module is used to fuse single-modal local features first and then cross-modal global features during feature fusion. The fusion calculation formula is: and Fusion global =Attention(Fusion local ), where ω i is the local feature weight, and Attention is the cross-modal attention mechanism; The filtering and interaction analysis module is used to filter redundant information and analyze the interaction relationship between modalities through cross-modal attention mechanism and self-attention mechanism; The emotion evolution analysis module is used to analyze the dynamic changes of emotions over time through a time series model. The calculation formula of the time series model is h t =σ(W x x t +W h h t-1 +b), where h t is the current emotional state, x t is the input feature at the current moment, W x and W h are weight matrices, and b is the bias vector; Real-time sentiment feedback module, used to provide instant feedback and visual display of sentiment classification results; The classification result generation module is used to generate the final sentiment classification result based on the filtered fusion features, and to perform correction and optimization in combination with the evolutionary analysis results.

2. A multimodal patient emotion analysis device according to claim 1, characterized in that: The dynamic mode selection module realizes automatic selection through the mode weight calculation network, which calculates the mode weight according to the formula Score m =α m ×Quality m +β m ×Complementarity m Calculate the weight of each mode.

3. The multimodal patient emotion analysis device according to claim 1, characterized in that: The multimodal conditional variational autoencoder in the processing module includes a text autoencoder, an audio autoencoder and an image autoencoder, which encode and decode data of different modalities respectively.

4. The multimodal patient emotion analysis device according to claim 1, characterized in that: The hierarchical attention mechanism used by the hierarchical fusion module includes a local feature attention mechanism and a global feature attention mechanism, which respectively perform weighted fusion on unimodal local features and cross-modal global features.

5. The multimodal patient emotion analysis device according to claim 4, characterized in that: The weight ω of the local feature fusion i Calculated by the following formula: Among them, e i is the score of feature i, and exp is the exponential function.

6. The multimodal patient emotion analysis device according to claim 1, characterized in that: The filtering and interaction analysis module uses a cross-modal attention mechanism to calculate the attention score matrix A between different modalities, and its calculation formula is: Among them, S ij is the similarity score between modality i and modality j.

7. The multimodal patient emotion analysis device according to claim 1, characterized in that: The sentiment evolution analysis module uses the long short-term memory network LSTM as the time series model, and its calculation formula is: h t =σ(W x x t +W h h t-1 +b) Among them, h t is the hidden state at the current moment, x t is the current input feature, W x and W h is the weight matrix and b is the bias term.

8. The multimodal patient emotion analysis device according to claim 7, characterized in that: The emotion evolution result generated by the long short-term memory network LSTM is input into the classification result generation module to correct the final emotion classification result.

9. The multimodal patient emotion analysis device according to claim 1, characterized in that: The real-time emotion feedback module includes an API interface and a visualization tool for displaying emotion classification results and emotion change trends in real time.

10. The multimodal patient emotion analysis device according to claim 1, characterized in that: The classification result generation module generates the final sentiment classification result through the following formula: y =softmax(W f × h + b f ) Among them, W f is the fused feature matrix, h is the hidden layer output, b f is the bias term and softmax is the activation function, which is used to generate the final classification probability distribution.

Images