Multi-modal co-situation prediction method based on supervision text assistance

Abstract

The invention discloses a multi-modal co-situation prediction method based on supervision text assistance. The method comprises the following steps: 1, obtaining text, audio and video data and carrying out feature extraction; 2, calculating fused multi-modal features through an attention mechanism and a long-short-term memory network; 3, learning topic distribution of supervised texts by using a hidden Dirichlet topic model LDA; 4, network parameters are trained through a common situation level of a given training scene and theme distribution of a corresponding supervision text; and 5, calculating and predicting the common situation level of the multi-modal scene by using the trained network parameters. According to the method, the multi-modal data and the supervision text are comprehensively utilized as privilege information, so that the model prediction performance in a complex condition-sharing scene is enhanced, the condition-sharing level in a multi-modal scene can be predicted more meticulously, the accuracy and generalization of condition-sharing prediction are remarkably improved, and the mental health support effect is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of multimodal human-computer interaction and natural language processing technology, specifically to an empathy prediction method based on multimodal input. Background Technology

[0002] Empathy, also known as compassion, is an important research direction in the field of affective computing. Empathy is an emotional response triggered by the interaction of innate traits and environmental factors. This response is spontaneously triggered while being shaped by active cognitive regulation. The resulting emotion reflects both an individual's direct experience or imagined perception of the emotional stimulus and their cognitive understanding of it; that is, the origin of this emotion lies outside of oneself. As a core element of interpersonal communication, empathy builds a bridge for deep human understanding and emotional connection. Achieving empathy requires the ability to recognize and respond to the emotions and perspectives of others. Therefore, accurate prediction of empathy has significant application value in areas such as mental health intervention, social media optimization, and human-computer interaction. Taking mental health as an example, by predicting empathy levels, therapists can dynamically adjust intervention strategies to better align treatment plans with the client's emotional needs, thereby improving the personalization and effectiveness of treatment.

[0003] While current research has made significant progress in empathy prediction, it primarily utilizes unimodal input, especially textual information. However, most real-world scenarios requiring empathic understanding, such as psychological counseling, typically involve multimodal input. Over-reliance on unimodal information can lead to an incomplete understanding of empathy. Furthermore, little work on empathy prediction has utilized additional privileged information that could contain additional content helpful for empathy analysis. For example, supervision texts are descriptions of the entire counseling process conducted by a supervising expert. Figure 1 As shown, this description mainly includes the theme and background of the consultation, such as family, work, and marriage; the client's needs and emotional expressions; and the therapist's empathic behavior. In addition to these objective backgrounds and settings, supervision texts typically include the supervisor's subjective evaluation of the effectiveness of the empathic behavior. These supervisors possess extensive professional knowledge and can provide a clearer and more accurate analysis of empathy during the consultation process. Therefore, the subjective and objective information contained in these supervision texts helps to more comprehensively analyze empathic expressions in the consultation scenario. How to effectively integrate multimodal information with the aforementioned additional privileged information to improve the accuracy and versatility of empathy prediction in multimodal scenarios, thereby meeting the needs of the mental health field for refined and professional data analysis, remains a problem that urgently needs research and improvement. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention proposes a multimodal empathy prediction method based on supervised text assistance. This method aims to utilize supervised text as privileged information and integrate multimodal features from video, audio, and text to significantly improve the accuracy and generalization of empathy prediction, thereby enabling a more comprehensive understanding of the level of empathy in a given scenario.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: The present invention provides a multimodal empathy prediction method based on supervised text assistance, characterized by the following steps: Step 1: Obtain the multimodal dataset And the corresponding supervisory text, multimodal dataset Any training sample in the middle is denoted as ,in, , , They are respectively Any text, any audio, any visual data in the dataset, let The corresponding supervision text is denoted as The actual category label is ; Step 2: Construct an empathy level prediction network based on multimodal feature fusion, and then... , , The process is performed to obtain multimodal fusion features. ; Step 3: Based on the preset number of documents Using Hidden Dirichlet Theme Model (LDA) to analyze Extraction was performed to obtain the supervision text. Document topic distribution ,in, Indicating supervision text In the The probability distribution over each topic, and ; Step 4: Based on the empathy level of the given training scenario and the corresponding supervisory text. Main document topic distribution Construct the total loss The weight set used to train an empathy level prediction network based on multimodal feature fusion. and bias set ; Step 5: Iteratively train the empathy level prediction network based on multimodal feature fusion using the SGD optimizer, and calculate the total loss function. Adjust network parameters until total loss is reached. The training continues until convergence, resulting in a well-trained supervised text-assisted multimodal empathy prediction model. This model is then used to detect multimodal empathy scenarios and obtain predicted empathy labels.

[0006] The multimodal empathy prediction method based on supervised text assistance described in this invention is characterized in that step 2 is performed as follows: Step 2.1: Extract training samples using equation (1) Feature information: (1) In equation (1), express Feature information, express Feature information, express Feature information, Pre-trained networks representing text modalities Pre-trained networks representing audio modalities, Pre-trained networks representing visual modalities; Step 2.2, using equation (2) to... , Perform linear projection, for Perform nonlinear projection to obtain Training representations , Training representations ,as well as Training representations ; (2) In equation (2), This represents the hyperbolic tangent activation function. The training weights represent the text modality. The training weights represent the audio modalities. The training weights represent the visual modalities. The training bias represents the text modality. This represents the training bias for the audio modality. The training bias represents the visual modality; Step 2.3: Calculate the bimodal feature representation of the text modality and the visual modality using equation (3). : (3) In equation (3), This indicates the concatenation of vectors. This represents the visual attention weight to the text query. This indicates the attention weight of the text to the visual query. This represents the features after a text-based visual attention query. This represents the normalized exponential activation function; Indicates transpose; Step 2.4: Calculate the multimodal fusion features using equation (4). : (4) In equation (4), LSTM represents a Long Short-Term Memory network; This represents a bimodal feature representation of text and audio modalities.

[0007] Further steps are as follows: Step 4.1: Use equation (6) to obtain the predicted empathy labels. ; (6) In equation (6), This represents the training weights of the linear layer for multimodal feature prediction. This represents the training bias of the linear layer for multimodal feature prediction; Step 4.2: Use equation (7) to obtain the predicted distribution of dialogue topics. ; (7) In equation (6), This represents the training weights of the linear layer that predicts the topic distribution. This indicates the training bias of the linear layer for predicting topic distributions; Step 4.3: Construct the total loss ; Step 4.3.1: Construct empathy label loss using equation (8). : (8) In equation (8), express The total number of training samples in the dataset; Step 4.3.2: Construct the topic distribution loss using equation (9). : (9) Step 4.3.3: Construct the total loss using equation (10). : (10) In equation (10), The weighted parameter representing the loss of empathy labeling. The weighted parameter represents the topic distribution loss.

[0008] The present invention provides an electronic device, including a memory and a processor, characterized in that the memory is used to store a program that supports the processor in executing the supervised text-assisted multimodal empathy prediction method, and the processor is configured to execute the program stored in the memory.

[0009] The present invention discloses a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, performs the steps of the multimodal empathy prediction method based on supervised text assistance.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention designs a novel multimodal empathy prediction network, which enhances the empathy level of the model in predicting complex scenarios by aggregating multimodal information. Compared with current mainstream empathy prediction methods, this invention can collect more cross-modal information, thereby analyzing specific multimodal scenarios more comprehensively and accurately.

[0011] 2. This invention innovatively uses supervised text as privileged information and optimizes the representation of text information through a specialized training framework, which further enhances the versatility of empathy labels for predicting multimodal scenarios. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the supervisory text involved in the present invention; Figure 2 This is a flowchart illustrating the method of the present invention. Detailed Implementation

[0013] In this embodiment, a supervised text-assisted multimodal empathy prediction method utilizes multimodal inputs and additional privileged information to enhance the model's ability to predict complex empathy scenarios. This method integrates an empathy level prediction network based on multimodal feature fusion and an LDA model to integrate information from various modalities for overall training, such as... Figure 2 As shown, the specific steps are as follows: Step 1: Obtain the multimodal dataset And the corresponding supervisory text, multimodal dataset Any training sample in the middle is denoted as ,in, , , They are respectively Any text, any audio, any visual data in the dataset, let The corresponding supervision text is denoted as The actual category label is Supervision text As additional privileged information, this information is only used during the training phase of the invention. It is not needed in subsequent phases when detecting multimodal empathy scenarios, thus the invention has strong versatility.

[0014] Step 2: Construct an empathy level prediction network based on multimodal feature fusion, and then... , , The process is performed to obtain multimodal fusion features. ; Step 2.1: Extract training samples using equation (1) Feature information: (1) In equation (1), express Feature information, express Feature information, express Feature information, Pre-trained networks representing text Pre-trained networks representing audio, A pre-trained network representing vision.

[0015] Step 2.2, using equation (2) to... , Perform linear projection, for Perform nonlinear projection to obtain Training representations , Training representations ,as well as Training representations ; (2) In equation (2), This represents the hyperbolic tangent activation function. Represents the training weights for the text modality. Indicates the training weights for the audio modality. Represents the training weights for the visual modality. The training bias represents the text modality. This represents the audio modality training bias. This represents the visual modality training bias.

[0016] Step 2.3: Calculate the bimodal feature representation of text and visual modalities using equation (3). Through the covariance matrix Calculate the attention scores for the corresponding modality hidden state pairs. Then, assign attention weights to the generated visual features over the text temporal sequence. Using row direction The covariance matrix is ​​normalized using a function. Similarly, the attention weights of text words in visual features are obtained through column-direction normalization. Pay attention weights Text features Multiplication, calculating the attention context of text features. Ultimately, visual features With attention context After splicing, multiply by the attention weight. Obtaining dual-modal features : (3) In equation (3), This indicates the concatenation of vectors. This represents the visual attention weight to the text query. This indicates the attention weight of the text to the visual query. This represents the features after a text-based visual attention query. This represents the normalized exponential activation function; This indicates transpose.

[0017] Step 2.4: Calculate the multimodal fusion features using equation (4). Here, textual modality information is chosen as the main component of multimodal features primarily because previous research has shown that textual information plays a dominant role in most empathy prediction scenarios. (4) In equation (4), LSTM represents a Long Short-Term Memory network; This represents a bimodal feature representation of text and audio modalities, similar to that calculated by equation (3).

[0018] Step 3: Based on the preset number of documents The hidden Dirichlet (LDA) model in equation (5) is used to extract the document topic distribution of the supervised text. LDA is a statistical model used to mine the latent topic structure in a document set. Its basic assumption is that each document consists of a mixture of multiple topics, and each topic is represented by a specific vocabulary distribution. This model aims to learn the latent topic distribution of the document set and the vocabulary distribution of each topic, and can automatically discover hidden topics without manual annotation. (5) In equation (5), This indicates the document topic distribution corresponding to the supervisory text, where, Indicating supervision text In the The probability distribution over each topic, and .

[0019] Step 4: Based on the empathy level of the given training scenario and the topic distribution of the corresponding supervised text, construct the total loss. The weight set used to train an empathy level prediction network based on multimodal feature fusion. and bias set ; Step 4.1: Use equation (6) to obtain the predicted empathy labels. ; (6) In equation (6), This represents the training weights of the linear layer for multimodal feature prediction. This represents the training bias of the linear layer for multimodal feature prediction.

[0020] Step 4.2: Use equation (7) to obtain the predicted distribution of dialogue topics. ; (7) In equation (6), This represents the training weights of the linear layer that predicts the topic distribution. This represents the training bias of the linear layer predicting the topic distribution.

[0021] Step 4.3: Construct the total loss Given the true label distribution Calculate the cross-entropy loss function In a given real topic distribution Calculate the KL divergence loss Finally, by and With the corresponding weights and By performing a linear combination, the total loss function can be calculated. ,in and These represent the empathy prediction label distribution and the predicted topic distribution during the training process, respectively. Using KL divergence loss allows for a more precise comparison of the differences between the two distributions.

[0022] Step 4.3.1: Construct empathy label loss using equation (8). : (8) In equation (8), express The total number of training samples in the dataset; Step 4.3.2: Construct the topic distribution loss using equation (9). : (9) Step 4.3.3: Construct the total loss using equation (10). (10) In equation (10), The weighted parameter representing the loss of empathy labeling. The weighted parameter represents the topic distribution loss.

[0023] Step 5: Iteratively train the empathy level prediction network based on multimodal feature fusion using the SGD optimizer, and calculate the total loss function. Adjust network parameters until total loss is reached. The training continues until convergence, resulting in a well-trained supervised text-assisted multimodal empathy prediction model. This model is then used to detect multimodal empathy scenarios and obtain predicted empathy labels.

[0024] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the methods described above, and the processor is configured to execute the program stored in the memory.

[0025] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

Claims

1. A multi-modal empathy prediction method based on supervised text assistance, characterized in that, is performed by the following steps: Step 1: Obtain the multimodal dataset And the corresponding supervisory text, multimodal dataset Any training sample in the middle is denoted as ,in, , , They are respectively Any text, any audio, any visual data in the dataset, let The corresponding supervision text is denoted as The actual category label is ; Step 2, build an empathy level prediction network based on multi-modal feature fusion, and process 、 、 to obtain multi-modal fusion features ; Step 3, according to the pre-set number of documents , using the latent Dirichlet topic model LDA to extract, get the document topic distribution of the supervision text , wherein, represents the distribution probability of the supervision text on the first topic, and ;​ Step 4: Based on the empathy level of the given training scenario and the corresponding supervisory text. Main document topic distribution Construct the total loss The weight set used to train an empathy level prediction network based on multimodal feature fusion. and bias set ; Step 5, iteratively training the multi-modal feature fusion's empathy level prediction network by SGD optimizer and calculating total loss function to adjust network parameters until the total loss converges, thereby obtaining a trained supervised text-aided multi-modal empathy prediction model for detecting multi-modal empathy scenes and obtaining predicted empathy labels.

2. The method of claim 1, wherein the method is a multi-modal empathetic prediction method based on supervised text assistance. Step 2 is performed by the following steps: Step 2.1, extracting features from training samples using formula (1) characteristic information: (1) In formula (1), characteristic information of characteristic information of characteristic information of characteristic information of characteristic information of characteristic information of represent a pre-training network of a text modality, represent a pre-training network of an audio modality, represent a pre-training network of a visual modality; Step 2.

2. Linear projection of the training representation of , , , , , and ;​​​ (2) In formula (2), denotes a hyperbolic tangent activation function, denotes a training weight for the text modality, denotes a training weight for the audio modality, denotes a training weight for the visual modality, denotes a training bias for the text modality, denotes a training bias for the audio modality, denotes a training bias for the visual modality; Step 2.

3. Computing the bi-modal feature representation of the text and visual modalities using formula (3) : (3) In formula (3), denotes concatenation of vectors, denotes the attention query weight of vision to text, denotes the attention query weight of text to vision, denotes the feature after the attention query of text to vision, denotes the normalized exponential activation function; denotes the transpose; Step 2.

4. Compute multimodal fusion features by formula (4) : (4) In formula (4), LSTM represents a long short-term memory network. The bimodal feature representation represents the text modality and the audio modality.

3. The method of claim 2, wherein the method is a multi-modal empathetic prediction method based on supervised text assistance. is performed by the following steps: Step 4.1, obtaining predicted empathy labels with formula (6) ; (6) In formula (6), denotes the training weights of the multi-modal feature prediction linear layer, denotes the training bias of the multi-modal feature prediction linear layer. Step 4.2, Utilizing Formula (7) to obtain a predicted distribution of dialogue topics ; (7) In formula (6), denotes the training weights of the topic distribution prediction linear layer, denotes the training bias of the topic distribution prediction linear layer. Step 4.3, Building the total loss ; Step 4.3.1, constructing co-sympathy tag loss with formula (8) : (8) In formula (8), denotes the total number of training samples in the set Step 4.3.2, constructing the topic distribution loss with formula (9) : (9) Step 4.3.3, building the total loss with formula (10) : (10) In formula (10), denotes a weighting parameter for the co- empathy tag loss, denotes a weighting parameter for the topic distribution loss.

4. An electronic device comprising a memory and a processor, characterized in that The memory is configured to store a program supporting the processor to perform the method for multi-modal empathy prediction based on supervised text assistance according to any one of claims 1-3.

5. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program, when executed by the processor, performs the steps of the method for multi-modal empathy prediction based on supervised text assistance according to any one of claims 1-3.

Images