A multi-modal depression sentiment analysis system based on sensitive attribute label constraints

Abstract

The embodiment of the specification discloses a multi-modal depression sentiment analysis system based on sensitive attribute label constraint. The system comprises a feature extraction module configured to obtain sentiment features under each modality; a feature preprocessing module configured to output each modality feature from a multi-modal data processing model; a feature alignment module configured to construct a text modality-centered feature pair based on each modality feature, and calculate the contrastive loss of each feature pair to determine each target modality feature; a contrastive decoupling module configured to sequentially encode and decode each target modality feature to obtain sensitive attribute-independent features; a representation fusion module configured to perform cross-modal attention fusion on the sensitive attribute-independent features to obtain fusion features, and determine sentiment information according to the fusion features. In this embodiment, individualized and accurate emotion recognition and depression sentiment analysis can be performed according to the user's social media data while eliminating the generalization influence of demographic bias.

Description

Technical Field

[0001] The embodiments in this specification belong to the field of multimodal data processing, and in particular relate to a multimodal depressive sentiment analysis system based on sensitive attribute label constraints. Background Technology

[0002] Depression has become a prevalent mental illness worldwide, characterized by persistent and difficult-to-regulate mood disturbances accompanied by impairments in cognitive, physical, and social functions. Currently, the primary diagnostic method for depression is clinical psychiatric examination, including history taking, standardized scale assessments, and auxiliary physical examinations. However, clinical examination methods are complex and require prior medical knowledge. Furthermore, differences in region, gender, age, and life background significantly increase the risk of misdiagnosis. Therefore, there is an urgent need for a method that can screen and identify depression early on a large scale and with high accuracy. Summary of the Invention

[0003] Embodiments of this disclosure provide a multimodal depressive sentiment analysis system based on sensitive attribute label constraints, designed to address one or more of the above-mentioned problems and other potential problems.

[0004] According to a first aspect of this disclosure, a multimodal depressive sentiment analysis system based on sensitive attribute label constraints is provided. The system includes a feature extraction module configured to extract sentiment features of various modalities from a user's social media data, including text, voice, and video, based on preset feature extractors; a feature preprocessing module configured to output dimensionality-reduced and dimension-unified modal features from a multimodal data processing model based on the sentiment features, the multimodal data processing model including a feature mapping module based on a multilayer perceptron and a self-attention mechanism module; a feature alignment module configured to construct feature pairs centered on the text modality based on the modal features, and calculate the contrast loss of each feature pair under the constraint of sensitive attribute labels to determine the target modal features with the minimum contrast loss, the sensitive attribute labels being used to represent demographic differences; a contrast decoupling module configured to encode and decode the target modal features sequentially to obtain sensitive attribute-related features and sensitive attribute-independent features, the decoding process embedding sensitive attribute labels as constraints; and a representation fusion module configured to perform cross-modal attention fusion on the sensitive attribute-independent features of each modality to obtain fused features, and determine sentiment information based on the fused features.

[0005] The solution provided in the embodiments of this specification introduces demographic differences as a sensitive attribute constraint. Through information constraints, comparative calculations, and feature decoupling, the modal features extracted from social media data are decoupled into sentiment representations that are related or unrelated to the sensitive attribute. Then, sentiment information is determined through cross-modal fusion, thereby achieving individualized and accurate emotion recognition and depressive sentiment analysis based on users' social media data while eliminating the generalization effect of demographic bias. Attached Figure Description

[0006] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0007] Figure 1 The following is a schematic diagram of the architecture of a multimodal depressive sentiment analysis system based on sensitive attribute label constraints, according to some embodiments of this disclosure;

[0008] Figure 2 A schematic diagram illustrating the principle of a multimodal depressive sentiment analysis system based on sensitive attribute label constraints, according to some embodiments of this disclosure, is shown. Detailed Implementation

[0009] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0010] The terms “comprising” and “having”, and any variations thereof, in this specification, claims, and the foregoing drawings are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. Depending on the context, the word “if” as it applies herein may be interpreted as “when”, “in response to determination”, or “in response to detection”.

[0011] Figure 1The diagram illustrates the structure of a multimodal depressive sentiment analysis system 100 based on sensitive attribute label constraints, representing some embodiments of this disclosure. The various embodiments in this specification are described in a progressive manner; similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. Figure 1 As shown, the system 100 includes a feature extraction module 101, which is configured to extract emotional features of various modalities from the user's social media data based on preset feature extractors. The modalities include text, voice and video.

[0012] In this embodiment, the overall process flow principle can be as follows: Figure 2 As shown, the process begins by acquiring publicly available social media data from users who have granted them data access permissions. This data is presented in various modalities (such as text, voice, and video). Pre-defined feature extractors (e.g., Librosa for acoustics, OpenFace for vision, and BERT for text) are then used to extract sentiment features from these different modalities. These sentiment features from each modality are then integrated for subsequent processing.

[0013] System 100 also includes a feature preprocessing module 102, which is configured to output dimensionality-reduced and dimension-unified modal features based on each emotion feature by a multimodal data processing model. The multimodal data processing model includes a feature mapping module based on a multilayer perceptron and a self-attention mechanism module.

[0014] In this embodiment, a multimodal data processing model is set up. This model can be viewed as a data processing module that connects a multilayer perceptron and a self-attention mechanism (SAM) module. The acquired emotional features are input into the multimodal data processing model and processed by both MLP and SAM to obtain dimensionality-reduced and dimensionally uniform features for each modality. Specifically, the calculation formula for the feature mapping module based on the multilayer perceptron is as follows:

[0015]

[0016] in, This represents the selected modality type, including text modality. Speech modality and video modality . This represents the sentiment features extracted by multimodal feature tools. This represents the feature mapping matrix based on MLP. This indicates the output of the feature mapping module.

[0017] The calculation formula based on the SAM module is:

[0018]

[0019] in, This represents the feature transformation function based on multi-head SAM. This indicates the module output.

[0020] System 100 also includes a feature alignment module 103, which is configured to construct text modality-centered feature pairs based on each modality feature, and calculate the contrast loss of each feature pair under the constraint of sensitive attribute labels, so as to determine each target modality feature with the smallest contrast loss. The sensitive attribute labels are used to characterize demographic differences.

[0021] In this embodiment, the text modality will be the central focus, and the features of each modality will be constructed into feature pairs (e.g., text-acoustic modal paired features). Text-visual modality paired features Next, sensitive attribute labels are set as constraints, and the contrastive loss of feature pairs is calculated. By iteratively optimizing the contrastive loss, the target modal features corresponding to the minimum contrastive loss are determined, ensuring that the target modal features implicitly have semantic alignment relationships across modalities. As an example, the cosine similarity matrix of the feature pairs and the label similarity matrix generated based on the Gaussian similarity matrix can be calculated. The correlation between the cosine similarity matrix and the label similarity matrix is ​​calculated using matrix multiplication, thus constructing the contrastive loss. Sensitive attribute labels can include, for example, gender, age, language, region, and cultural background. Taking gender as an example, males can be labeled as 1 and females as 0. Specifically, a classification model can be pre-trained based on manually labeled training data, so that when performing sentiment analysis on specific users, the classification model can directly output the sensitive attribute labels corresponding to the modal features. This method of using sensitive attribute labels as constraints helps to subsequently determine sentiment representations unrelated to sensitive attribute labels, thus ensuring that subsequent prediction results are not affected by sensitive attributes and are more robust.

[0022] The above process can be directly calculated and processed, or the formulas can be integrated to construct a language-centered bimodal alignment model. For example, the MLP model can be used as a basis and the settings can be adjusted according to actual needs. When using the model for processing, sensitive attribute labels can be set on the training data during model training so that the model can recognize sensitive attribute labels at the same time.

[0023] System 100 also includes a comparison decoupling module 104, which is configured to encode and decode each target modal feature sequentially to obtain sensitive attribute-related features and sensitive attribute-independent features. The decoding process embeds sensitive attribute labels as constraints.

[0024] In this embodiment, after obtaining the target modal features, the target modal features are sequentially encoded and decoded to achieve decoupling of the target modal features' representation and data reconstruction. The encoding process is consistent with the conventional encoding process, but the decoding process incorporates sensitive attribute labels as constraints to guide data generation and reconstruction. Thus, regardless of whether the decoupling is related or unrelated to sensitive attributes, the identical encoding process ensures the generation of a consistent latent representation. Specifically, they are used to capture sensitive attribute-related features. and features that are independent of sensitive attributes and are used to capture sensitive attributes. Based on this, the decoding process simultaneously performs two steps: one is the decoding process of sensitive attribute-related features, that is, the decoding of sensitive attribute labels. Embedded into the decoding process, so as to be in the decoding process In this process, sensitive attribute tags are used as probabilistic conditions to form conditional probability decoding. To constrain the decoding function The quality of data reconstruction. Another is the decoupling process of features unrelated to sensitive attributes, i.e., maintaining the original decoding process. This process does not restrict whether the reconstructed data contains sensitive attributes. The specific process is as follows:

[0025]

[0026]

[0027] in, For the target modal features as input, For the decoded and reconstructed data, , .

[0028] In practical applications, the following formula can be used for calculation:

[0029]

[0030] in, and Indicates different modal features Encoded Generate the mean and variance of the latent representation vector. To learn a continuous, structured latent representation, a resampling technique can be used, as follows:

[0031]

[0032] In addition, to eliminate ordinal relationships between tags and ensure category equality, OneHot encoding is used to encode sensitive attribute tags, as follows:

[0033]

[0034] The final reconstructed data is as follows:

[0035]

[0036] Similarly, the above process can be directly calculated or implemented by constructing a contrast decoupling model based on the encoder-decoder structure.

[0037] System 100 also includes a representation fusion module 105, which is configured to perform cross-modal attention fusion on sensitive attribute-independent features under each modality to obtain fused features, and determine emotional information based on the fused features, including emotional intensity and emotional polarity.

[0038] In this embodiment, to eliminate the influence of demographic bias, only features unrelated to sensitive attributes are used as the multi-source input for multimodal feature fusion. Multimodal data fusion is performed based on a cross-modal attention fusion mechanism to finally obtain the fused features. The Query, Key, and Value values ​​in this fusion process all come from sensitive attribute-independent features of different modalities. The specific calculation process is as follows:

[0039]

[0040] in, These represent latent representations that are independent of sensitive attributes for text, audio, and video modalities, respectively, while CMAF represents the cross-modal attention fusion layer.

[0041] Finally, by calculating the matching degree between the fused features and the pre-set template feature vectors of each sentiment label in the high-dimensional feature space, the sentiment label with the highest matching degree can be used as the sentiment information determined after this sentiment analysis of the user. This serves as a benchmark for analyzing the potential correlation between depression, emotional state, and sensitivity attributes, thereby achieving early screening for whether a user is depressed. The sentiment information includes sentiment intensity and sentiment polarity. Sentiment intensity is a set of continuous values ​​used to indicate the degree of the current emotional state. Sentiment polarity is used to characterize the analysis results of sentiment intensity. For example, it can be roughly divided into negative, neutral, and positive sentiments based on the absolute value range of sentiment intensity or the rate of change of sentiment intensity per unit time. The specific division rules can be adjusted and set according to actual needs.

[0042] In one possible implementation, the feature extraction module is further configured to:

[0043] Adjust the tool parameters of each preset feature extractor to obtain multimodal features in different dimensions from general social media data;

[0044] The pre-trained model outputs the corresponding output results for each multimodal feature, and the optimal tool parameters for each feature extractor are determined based on the output results.

[0045] In this embodiment, the feature extractor can be configured with different tool parameters. Depending on the tool parameters, the feature extractor can extract features from different dimensions to obtain different types of sentiment features. Before using the feature extractor, the tool parameters can be adjusted to select different combinations of feature extraction dimensions (e.g., using the BERT model to extract 768-dimensional text features, using the Librosa tool to obtain 33-dimensional acoustic features, and using the OpenFace tool to extract 709-dimensional visual features). This allows for the acquisition of different multimodal features from publicly available social media data based on different feature extraction combinations. Next, the multimodal features from different combinations can be input into a pre-trained model, such as a multilayer perceptron (MLP). By comparing the output predicted by the pre-trained model with the true labels, the performance of different combinations can be evaluated based on the accuracy of the comparison. The optimal tool parameter settings for the feature extractor, corresponding to the combination with the best performance and highest accuracy, can be selected, allowing the feature extractor to operate with these optimal tool parameters during actual use.

[0046] In one implementation, the multimodal data processing model is also used to compress the output features obtained after processing by the feature mapping module and the self-attention mechanism module along the sequence dimension.

[0047] In this embodiment, a weighting parameter can be further used. A linear layer is constructed, and the ReLU activation function is applied to compress the output features along the sequence dimension to output modal features. The calculation formula is:

[0048]

[0049] In one possible implementation, the feature alignment module is further configured to:

[0050] Calculate the cosine similarity matrix for each feature pair;

[0051] Using sensitive attribute labels as constraints, the label similarity matrix of each feature pair is calculated based on the Gaussian similarity matrix;

[0052] Based on the comparison between the label similarity matrix and a predefined threshold, after determining the positive sample mask, a contrastive loss function is constructed according to cosine similarity, label similarity, and the positive sample mask, so as to calculate the contrastive loss of each feature pair.

[0053] In this embodiment, a cosine similarity matrix can be used. The feature similarity matrix for each feature pair And using sensitive attribute labels as constraints, the Gaussian similarity matrix is ​​used. Calculate the corresponding label similarity matrix Furthermore, to enhance the semantic consistency of bimodal features, the correlation between the feature similarity matrix and the label similarity matrix is ​​calculated using matrix multiplication. The calculation formula for the entire process is as follows:

[0054]

[0055]

[0056]

[0057]

[0058] in, This represents the text modal features in the feature pair. This represents the remaining modalities in the feature pair, namely the video modality or the audio modality. and These represent the sensitive attribute label vector and its transpose vector, respectively. For the defined scaling parameters, Indicates the first text modality Modal features of each sample, Indicates the first in the video modality or audio modality Modal features of each sample. Similarly, and They represent the first and the The sensitive attribute label vector of each sample. For ease of explanation, we will use... and They represent the first and the Cosine similarity and label similarity of each sample.

[0059] Next, the label similarity matrix was compared with a predefined threshold. By comparing, we obtain the positive sample mask, i.e.:

[0060]

[0061] in, Defined as an indicator function, it returns 1 if the internal condition is true, and 0 otherwise.

[0062] Based on the above matrix, the contrastive loss function for feature pairs is defined as follows:

[0063]

[0064] In this way, the semantic consistency of multimodal sentiment features under sensitive attribute constraints can be achieved by minimizing the contrastive loss function, while providing data support for subsequent contrastive decoupling.

[0065] In one possible implementation, the characterization fusion module is further configured to:

[0066] The fused features are input into a classification layer built on a multilayer perceptron to determine sentiment information based on the product of the fully connected layer weight matrix and the fused features.

[0067] In this embodiment, the fused features can be input into a classification layer built based on a multilayer perceptron to predict sentiment information. The specific process is as follows:

[0068]

[0069] in, This indicates the predicted sentiment information. Let b represent the weight matrix of the fully connected layer, where b is a constant.

[0070] In one possible implementation, the system further includes:

[0071] The function optimization module is configured to perform weighted calculations on the loss functions corresponding to the calculation processes of the target modality features, the sensitive attribute-related features, and the multimodal sentiment analysis, to obtain the total loss function, and then minimize the total loss function through iterative training.

[0072] In this embodiment, a contrastive loss function is set in the process of calculating the target modal features. Furthermore, by comparing the decoupling process and the multimodal sentiment analysis computation process, corresponding loss functions can be set to control and optimize the computation process. The loss function for the decoupling process includes the decoupling loss. Reconstruction loss And comparative loss The overall loss of contrast decoupling can be represented by the sum of the decoupling loss and the reconstruction loss, i.e. .

[0073] The loss function for the computational process of multimodal sentiment analysis is:

[0074]

[0075] in, For the true sentiment label of the i-th feature, Let i be the predicted sentiment information for the i-th feature.

[0076] Finally, the total loss function is:

[0077]

[0078] in, These represent the weight coefficients of the loss components, and their specific values ​​can be set according to actual needs. By iteratively training and minimizing the total loss function, the optimal parameter configuration for the overall calculation process can be obtained.

[0079] In one possible implementation, the system further includes:

[0080] The sentiment analysis module is configured to calculate the correlations between features unrelated to sensitive attributes and multimodal negative sentiment data, features unrelated to sensitive attributes and multimodal positive sentiment data, features related to sensitive attributes and multimodal negative sentiment data, and features related to sensitive attributes and multimodal positive sentiment data, respectively, and generate feature analysis results based on each correlation.

[0081] In this embodiment, based on the pre-determined multimodal negative sentiment data and multimodal positive sentiment data, the Pearson correlation between the sensitivity attribute-independent features and sensitivity attribute-related features and the multimodal negative sentiment data and the multimodal positive sentiment data can be calculated. The calculated correlation values ​​are used to generate feature analysis results, which more quantitatively and intuitively represent the potential association between different features and negative / positive sentiment. By analyzing the above potential associations, a personalized and customized multimodal depressive sentiment analysis system can be constructed to comprehensively assess the risk of depression under different sensitivity attribute conditions.

[0082] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this specification is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0083] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. Furthermore, although operations are depicted in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0084] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A multimodal depressive sentiment analysis system based on sensitive attribute label constraints, characterized in that, The system includes: The feature extraction module is configured to extract emotional features from the user's social media data in various modalities, including text, voice, and video, based on preset feature extractors. The feature preprocessing module is configured to output dimensionality-reduced and dimension-unified modal features based on each of the aforementioned sentiment features by a multimodal data processing model. The multimodal data processing model includes a feature mapping module based on a multilayer perceptron and a self-attention mechanism module. The feature alignment module is configured to construct text modality-centered feature pairs based on each of the modal features, and calculate the contrast loss of each feature pair under the constraint of sensitive attribute labels, so as to determine each target modal feature with the smallest contrast loss, wherein the sensitive attribute labels are used to characterize demographic differences; The decoupling module is configured to encode and decode each target modal feature sequentially to obtain sensitive attribute-related features and sensitive attribute-independent features. The decoding process embeds the sensitive attribute label as a constraint condition. The representation fusion module is configured to perform cross-modal attention fusion on the sensitive attribute-independent features under each modality to obtain fused features, and determine emotional information based on the fused features, wherein the emotional information includes emotional intensity and emotional polarity; The feature alignment module is also configured to: Calculate the cosine similarity matrix for each of the feature pairs; Using sensitive attribute labels as constraints, the label similarity matrix of each feature pair is calculated based on the Gaussian similarity matrix; Based on the comparison between the label similarity matrix and the predefined threshold, after determining the positive sample mask, a contrast loss function is constructed according to the cosine similarity, label similarity and positive sample mask, so as to calculate the contrast loss of each feature pair according to the contrast loss function; Among them, the and the The formula for calculating the cosine similarity of samples is: in, For the defined scaling parameters, For the text modality, the first Modal features of each sample, For the video modality or audio modality, the first Modal features of each sample, For the selected modal type, For speech modality, For video modality; No. and the The formula for calculating the label similarity of a sample is: in, and They represent the first and the Sensitive attribute label vectors for each sample; The formula for calculating the positive sample mask is: in, Defined as an indicator function, it returns 1 if the internal condition is true, otherwise it returns 0; For predefined thresholds; The formula for calculating the contrastive loss function is as follows: in, For positive sample masks, For the first and the Label similarity of each sample For the first and the Cosine similarity of samples, It is a preset positive number.

2. The multimodal depressive sentiment analysis system based on sensitive attribute label constraints according to claim 1, characterized in that, The feature extraction module is also configured to: Adjust the tool parameters of each preset feature extractor to obtain multimodal features in different dimensions from general social media data; The pre-trained model outputs the corresponding output results for each of the multimodal features, and the optimal tool parameters for each feature extractor are determined based on the output results.

3. The multimodal depressive sentiment analysis system based on sensitive attribute label constraints according to claim 1, characterized in that, The multimodal data processing model is also used to compress the output features obtained by the feature mapping module and the self-attention mechanism module along the sequence dimension.

4. The multimodal depressive sentiment analysis system based on sensitive attribute label constraints according to claim 1, characterized in that, The comparison decoupling module is also configured to: Encode each of the target modal features to obtain each potential representation; Using the sensitive attribute label as a probability condition, conditional probability decoding is performed on the latent representation to obtain sensitive attribute-related features and sensitive attribute-independent features.

5. A multimodal depressive sentiment analysis system based on sensitive attribute label constraints according to claim 1, characterized in that, The representation fusion module is also configured to: The fused features are input into a classification layer built on a multilayer perceptron to determine sentiment information based on the product of the fully connected layer weight matrix and the fused features.

6. The multimodal depressive sentiment analysis system based on sensitive attribute label constraints according to claim 1, characterized in that, The system also includes: The function optimization module is configured to perform weighted calculations on the loss functions corresponding to the calculation processes of the target modality features, the sensitive attribute-related features, and the multimodal sentiment analysis, to obtain the total loss function, and then minimize the total loss function through iterative training.

7. A multimodal depressive sentiment analysis system based on sensitive attribute label constraints according to claim 1, characterized in that, The system also includes: The sentiment analysis module is configured to calculate the correlations between features unrelated to sensitive attributes and multimodal negative sentiment data, features unrelated to sensitive attributes and multimodal positive sentiment data, features related to sensitive attributes and multimodal negative sentiment data, and features related to sensitive attributes and multimodal positive sentiment data, respectively, and generate feature analysis results based on each correlation.

Images