An interactive depression intelligent detection method assisted by social media analysis

Abstract

The present invention discloses an interactive depression intelligent detection method assisted by social media analysis, which relates to the technical field of automatic detection of depression, and solves the technical problem of low data comprehensiveness of depression detection methods in the prior art. The present invention includes four steps: user information acquisition, interactive behavior analysis, social media analysis and fusion recognition. First, the user is required to read and sign a privacy agreement, and input information such as occupation, gender, age, social media account, etc. Then, the user interacts with the intelligent assistant through audio and video, and a neural network model is constructed to characterize the data, and a depression characterization vector of the interactive multimodal behavior data is obtained. In addition, the text records of the user on the social platform are captured imperceptibly, the depression features in the data are extracted, and the depression characterization vector of the social media data is obtained. Finally, the two types of features are fused to identify whether the user is a depressed patient.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic detection of depression, and in particular to an interactive intelligent depression detection method assisted by social media analysis. Background Art

[0002] Early detection and intervention are crucial for the treatment of depression. Currently, the diagnosis of depression mainly relies on clinical interviews by psychiatrists and self-rating depression scales, which have significant limitations. On the one hand, both methods rely on the patient's self-description of their subjective feelings and are easily influenced by the patient's subjective wishes and social expectations, resulting in inaccurate diagnostic results. On the other hand, professional psychiatric medical resources are limited and cannot meet the needs of large-scale screening, resulting in many depression patients failing to receive timely diagnosis and treatment.

[0003] There are two main types of existing automatic depression detection technologies. One is to identify patients with depression by analyzing the interviewee's language content, voice, facial expressions and other behavioral characteristics during conversation, that is, automatic depression identification based on behavioral characteristic analysis; the other is to analyze users' network behavior and data in social media to identify patients with depression, that is, automatic depression identification based on social media analysis. However, due to the high heterogeneity of depression, a single data source often cannot fully capture the diversity and complexity of depressive symptoms. Existing depression detection methods still need to be improved in terms of data comprehensiveness, detection accuracy and practicality.

[0004] In view of the above technical defects, a solution is now proposed. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes an interactive depression intelligent detection method assisted by social media analysis. This method simultaneously collects the user's text records on the social platform and the multimodal behavior data when interacting with the intelligent assistant, and realizes depression detection by characterizing and effectively fusing the depression of the two data sources. This method faces challenges such as the scarcity of interactive multimodal behavior data, the sparse depression characteristics of social media data, and the lack of effective fusion strategies.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] An interactive depression intelligent detection method assisted by social media analysis, the intelligent detection method steps are as follows:

[0008] User information acquisition: users are required to collect information at the beginning, and they are guided to read and sign the privacy agreement when collecting user information;

[0009] Interactive behavior analysis, after completing the acquisition of user information, enters the interactive behavior analysis stage, and the interactive behavior analysis is divided into three sub-processes, namely interactive video interview, multimodal data preprocessing and multimodal data representation;

[0010] Interactive video interviews are used to collect facial video and audio behavioral data of users; multimodal data preprocessing is used to process audio data to obtain conversation text data, which together with facial video and audio data constitute multimodal behavioral data; multimodal data representation extracts depression features from interactive multimodal behavioral data;

[0011] Social media analysis is used to extract depression features from the text records posted by users on social platforms while analyzing interactive behaviors, and obtain depression representations of users' social media data. It is specifically divided into non-sensitive social media data capture, LLM-based depression understanding, and text representation;

[0012] Fusion recognition is used to effectively fuse the depression representations of interactive multimodal behavior data and social media data, and identify depressed / non-depressed users based on the fused vectors.

[0013] As a preferred implementation of the present invention, the user information includes occupation, gender, age and social media account.

[0014] As a preferred embodiment of the present invention, the interactive video interview process is as follows: the user conducts an interview with the intelligent assistant via video, the intelligent assistant asks questions to the user according to a set list of questions, and collects facial video and audio data of the user during the interaction.

[0015] As a preferred embodiment of the present invention, the multimodal data preprocessing process is: through speaker recognition technology, the user's voice fragments are extracted and spliced ​​together to form audio data; and through speaker recognition technology and speech recognition technology, the audio is transcribed into text to form conversation text data; and the facial video data and audio data are unified and summarized to form interactive multimodal behavior data.

[0016] As a preferred embodiment of the present invention, the multimodal data characterization process is: respectively using a video encoder, a text encoder and an audio encoder to extract features from corresponding modal data, the encoder uses a pre-trained model trained on a wide range of samples, and the pre-trained model is used to ensure robustness and generalization;

[0017] The three modal features are then concatenated to obtain the depression representation vector of the interactive multimodal behavioral data.

[0018] As a preferred embodiment of the present invention, the process of seamless social media data capture is: according to the social media account provided by the user, the text records posted by the user within the set time are captured from the social media platform, all user posts are spliced ​​together, and a prompt is added as the input of the large language model (LLM) to obtain the text description understood by the LLM; the depression understanding process based on LLM is: the large language model can adopt but is not limited to ChatGPT, Wenxinyiyan, and GLM models; the role of the prompt is to guide the large language model to summarize and summarize the depression symptoms of the user based on the user's posts.

[0019] As a preferred embodiment of the present invention, the depression understanding and text representation process based on LLM is as follows: a summary text description of the user's depressive symptoms is obtained based on the user's posts on the social platform, and the text description output by LLM will be subjected to feature extraction by a text encoder. The encoder adopts a pre-trained model trained on a wide range of samples, and the pre-trained model is used to ensure robustness and generalization; the vector output by the text encoder is used as the depression representation vector of social media data.

[0020] As a preferred embodiment of the present invention, the fusion recognition process is as follows:

[0021] When integrating the depression representation of interactive multimodal behavior data and social media data, let X1 and X2 represent the depression representation vectors of interactive multimodal behavior data and social media data respectively, and transform them into vectors of the same dimension h1 and h2 through different linear mappings. The formula is: , where W1 and W2 are linear mapping weight matrices;

[0022] ; In the above formula, It means concatenating the two vectors h1 and h2 together. W3 is the linear mapping weight matrix, which maps the concatenated vectors to the same dimension. represents the Sigmoid function, so each element in g is between 0 and 1; in the subsequent fusion, g will be used as the coefficient of h2, multiplied by h2, and added to h1 to obtain the fused feature vector h;

[0023] The formula is: ; In the above formula, since each element in g is between 0 and 1, and its value is determined by the two input features X1 and X2, g controls the degree of fusion of the amount of information in h2 with h1. g is the gating factor, and the above fusion network is called a gating network. Through this gating network, a fusion strategy with h1 as the main and h2 as the auxiliary is realized;

[0024] A multi-layer perceptron (MLP) is constructed to classify the fused feature vector h, and the output is two categories: depression / non-depression, as shown in the following formula: ; In the above formula, the Softmax function is used to normalize the output vector of MLP to obtain y; y represents the probability of predicting that the user is a depressed patient or a non-depressed person.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. The present invention integrates two data sources: social media data and interactive multimodal behavior data, which improves the comprehensiveness of data and the accuracy of detection;

[0027] 2. Proposal and implementation of fusion strategy: In the fusion process of the two data sources, considering that the topic of the interview is designed for depression, while the topic of social media data is open and less targeted, this method adopts the fusion strategy of "focusing on interactive behavior features and supplemented by social media features", and proposes a gated network to implement this strategy;

[0028] 3. Anti-sparseness of social media data: As social media data are open in subject matter, only a small amount of data is related to depressive symptoms, and the sparsity of depressive features is prominent. This method uses the powerful language understanding and induction capabilities of the large language model to summarize the depressive symptoms in social media data, and then represents the summarized text, which can effectively remove the interference of redundant information and noise, and realize the depressive representation of social media data;

[0029] 4. Solutions to the scarcity of interactive multimodal behavioral data: Considering the difficulty in acquiring interactive multimodal behavioral data and the limited size of the data set, the present invention widely uses pre-trained models to perform feature encoding on facial video, audio, and conversation text data in the depressive representation of multimodal data, and uses pre-trained models to ensure robustness and generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0031] Figure 1 It is the main flow chart of the method of the present invention;

[0032] Figure 2 is a flowchart of interactive behavior analysis in the present invention;

[0033] Figure 3 This is a flowchart of social media analysis in the present invention;

[0034] Figure 4 This is a fusion recognition flow chart of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it mutually exclusive, independent, or alternative to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] The present invention realizes intelligent detection of depressed users by collecting multimodal behavioral data such as text records of users on social platforms and audio and video when interacting with intelligent assistants, and integrating social media analysis and interactive behavior analysis.

[0038] See also Figure 1 As shown in FIG. 1 , an interactive depression intelligent detection method assisted by social media analysis is provided. The method is divided into four steps: user information acquisition, interactive behavior analysis, social media analysis, and fusion recognition. The specific steps are as follows:

[0039] User information acquisition

[0040] Since the causes of depression in different groups of people are obviously different, and vary greatly with age, occupation, and gender, therefore, at the beginning, users are required to enter information such as occupation, gender, age, and social media accounts. In addition, to protect user privacy, users are guided to read and sign the privacy agreement;

[0041] Interactive Behavior Analysis

[0042] After completing the acquisition of user information, the interactive behavior analysis phase begins; Figure 2 As shown, interactive behavior analysis includes three sub-processes: interactive video interview, multimodal data preprocessing and multimodal data representation.

[0043] Among them, the interactive video interview is designed to collect user behavior data;

[0044] The user conducts an interview with the intelligent assistant via video. The intelligent assistant asks the user questions based on a set list of questions and collects facial video and audio data during the user's interaction.

[0045] The list of questions mentioned above should be designed specifically for depression and can be drawn up by a psychiatrist or with reference to the diagnostic criteria for mental illness and other professional books.

[0046] The multimodal data preprocessing subprocess refers to the processing of audio data;

[0047] On the one hand, through speaker recognition technology, the user's voice fragments are extracted and spliced ​​together to form audio data;

[0048] On the other hand, through speaker recognition technology and speech recognition technology, the audio is transcribed into text to form conversation text data.

[0049] Finally, the user's facial video data, audio data, and conversation text data constitute interactive multimodal behavior data; among them, speaker recognition technology and speech recognition technology are both publicly known existing technologies;

[0050] Multimodal data representation is to extract depression features from interactive multimodal behavior data to obtain depression representation of interactive multimodal behavior data;

[0051] like Figure 2 As shown, firstly, a video encoder, a text encoder and an audio encoder are used to extract features of the corresponding modality data, and the encoder uses a pre-trained model trained on a wide range of samples;

[0052] Then, the three modal features are concatenated to obtain the depression representation vector of the interactive multimodal behavior data. In view of the challenge of scarce multimodal behavior data and to avoid overfitting of the model, the audio, text and video encoders in this scheme use pre-trained models trained on a wide range of samples. The speech encoder can use models such as Wav2Vec2.0 and HuBERT, the text encoder can use models such as BERT, RoBERTa or EmoBERTa, and the video encoder can use models such as CLIP. The pre-trained models are used to ensure robustness and generalization.

[0053] Social Media Analytics

[0054] While analyzing interactive behaviors, this method will conduct social media analysis to extract depression-related features from the text records posted by users on social platforms and obtain depression representations of users’ social media data. Figure 3 As shown in the figure, social media analysis includes three sub-processes: unconscious social media data capture, LLM-based depression understanding, and text representation.

[0055] Senseless social media data capture

[0056] Based on the social media account provided by the user, the text records posted by the user within a period of time are captured from the social media platform. Figure 3 The t1 and t2 shown in the figure represent posts made by the user at different time periods.

[0057] All user posts are concatenated together and a prompt is added as input to the large language model (LLM) to obtain the text description understood by the LLM.

[0058] The above-mentioned large language model can adopt but is not limited to ChatGPT, Wenxinyiyan, GLM and other models. The role of the prompt is to guide the large language model to summarize and summarize the user's depressive symptoms based on the user's post. Prompts such as "depressive symptoms include low mood, loss of interest, loss of pleasure, slow thinking, three no (uselessness, helplessness, hopelessness) symptoms, self-blame, lack of self-awareness, sleep disorders, eating and weight disorders, sexual dysfunction and other non-specific physical symptoms can be used. The above content is a post by the user on the social platform. Please summarize the user's depressive symptoms based on the post."

[0059] The text description output by LLM will be passed through a text encoder for feature extraction. The text encoder uses a pre-trained model trained on a wide range of samples, such as BERT, RoBERTa, or EmoBERTa, etc., with the help of the pre-trained model to ensure robustness and generalization.

[0060] The vector output by the text encoder is used as the depression representation vector of social media data;

[0061] Fusion recognition

[0062] When fusing the depressive representations of interactive multimodal behavioral data and social media data, considering that the interview topics are designed specifically for depression, while the topics of social media data are open and less targeted, this method adopts a fusion strategy of "focusing on interactive behavioral features and supplemented by social media features" and proposes a gating network to implement this strategy.

[0063] The above-mentioned fusion recognition process based on the gated network is as follows: Figure 4 As shown. Let X1 and X2 represent the depression representation vectors of interactive multimodal behavior data and social media data respectively, and transform them into vectors of the same dimension h1 and h2 through different linear mappings, as shown in the following formula, where W1 and W2 are linear mapping weight matrices: ; The calculation formula of g in the figure is as follows: ; In the above formula, It means concatenating the two vectors h1 and h2 together. W3 is the linear mapping weight matrix, which maps the concatenated vectors to the same dimension. Represents the Sigmoid function, so each element in g is between 0 and 1. In the subsequent fusion, g will be used as the coefficient of h2, multiplied by h2, and added to h1 to obtain the fused feature vector h;

[0064] The formula is: ; In the above formula, since each element in g is between 0 and 1, and its value is determined by the two input features X1 and X2, g controls the degree of fusion of the amount of information in h2 with h1. g is the gating factor, and the above fusion network is called a gating network. Through this gating network, a fusion strategy with h1 as the main and h2 as the auxiliary is realized.

[0065] Finally, the present invention constructs a multi-layer perceptron (followed by a Softmax activation function) to classify the fused feature h, and outputs two categories: depression / non-depression, as shown in the following formula: ; In the above formula, the output vector of the multi-layer perceptron (MLP) is normalized using the Softmax function to obtain y; y represents the probability of predicting that the user is a depressed patient or a non-depressed person.

[0066] When the present invention is used, the user is required to collect information at the beginning stage, and the user is guided to read and sign a privacy agreement when collecting user information; after completing the acquisition of user information, the interactive behavior analysis stage is entered, and the interactive behavior analysis is divided into three sub-processes, namely interactive video interview, multimodal data preprocessing and multimodal data characterization; the interactive video interview is used to collect two kinds of behavior data of the user's facial video and audio; the multimodal data preprocessing is used to process the audio data to obtain dialogue text data, which together with the facial video and audio data constitutes multimodal behavior data; the multimodal data characterization extracts depression features from the interactive multimodal behavior data; while the interactive behavior analysis is being performed, the depression features are extracted from the text records posted by the user on the social platform to obtain the depression representation of the user's social media data, which is specifically divided into non-sensory social media data capture, LLM-based depression understanding and text representation; finally, the depression representation of the interactive multimodal behavior data and the social media data are effectively integrated to realize the identification of depressed / non-depressed users.

[0067] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. An interactive depression intelligent detection method assisted by social media analysis, characterized in that: The steps of intelligent detection method are as follows: User information acquisition: users are required to collect information at the beginning, and they are guided to read and sign the privacy agreement when collecting user information; Interactive behavior analysis, after completing the acquisition of user information, enters the interactive behavior analysis stage, and the interactive behavior analysis is divided into three sub-processes, namely interactive video interview, multimodal data preprocessing and multimodal data representation; Interactive video interviews are used to collect facial video and audio behavioral data of users; multimodal data preprocessing is used to process audio data to obtain conversation text data, which together with facial video and audio data constitute multimodal behavioral data; multimodal data representation extracts depression features from interactive multimodal behavioral data; Social media analysis is used to extract depression features from the text records posted by users on social platforms while analyzing interactive behaviors, and obtain depression representations of users' social media data. It is specifically divided into non-sensitive social media data capture, LLM-based depression understanding, and text representation; Fusion recognition is used to fuse the depression representations of interactive multimodal behavior data and social media data, and identify depression based on the fused vectors; the fusion recognition process is as follows: When integrating the depression representation of interactive multimodal behavior data and social media data, let X1 and X2 represent the depression representation vectors of interactive multimodal behavior data and social media data respectively, and transform them into vectors of the same dimension h1 and h2 through different linear mappings. The formula is: , where W1 and W2 are linear mapping weight matrices; ; In the above formula, It means concatenating the two vectors h1 and h2. W3 is the linear mapping weight matrix, which maps the concatenated vectors to the same dimension. The gating factor g is multiplied by h2 and added to h1 to obtain the fused feature vector h. The formula is: ; In the above formula, since each element in g is between 0 and 1, and its value is determined by the two input features X1 and X2; A multi-layer perceptron is constructed to classify the fused feature vector h, and the output is the depression category, as shown in the following formula: ; y represents the probability of predicting that the user is a depressed patient or a non-depressed person.

2. The interactive depression intelligent detection method assisted by social media analysis according to claim 1, characterized in that: User information includes occupation, gender, age and social media accounts.

3. The interactive depression intelligent detection method assisted by social media analysis according to claim 1, characterized in that: The interactive video interview process is: The user conducts an interview with the intelligent assistant via video. The intelligent assistant asks the user questions based on a set list of questions and collects facial video and audio data during the user's interaction.

4. The interactive depression intelligent detection method assisted by social media analysis according to claim 3 is characterized in that: The multimodal data preprocessing process is: Through speaker recognition technology, the user's voice fragments are extracted and spliced ​​together to form audio data; and through speaker recognition technology and speech recognition technology, the audio is transcribed into text to form conversation text data; and it is unified and aggregated with facial video data and audio data to form interactive multimodal behavior data.

5. The interactive depression intelligent detection method assisted by social media analysis according to claim 4, characterized in that: The multimodal data characterization process is: The video encoder, text encoder and audio encoder are used to extract features from the corresponding modality data. The encoder uses a pre-trained model trained on a wide range of samples to ensure robustness and generalization. The three modal features are then concatenated to obtain the depression representation vector of the interactive multimodal behavioral data.

6. The interactive depression intelligent detection method assisted by social media analysis according to claim 1, characterized in that: The process of non-sensitive social media data capture is as follows: According to the social media account provided by the user, the text records posted by the user within the set time are captured from the social media platform, all the user posts are spliced ​​together, and a prompt is added as the input of the large language model to obtain the text description understood by the LLM; The depression understanding process based on LLM is as follows: the large language model adopts but is not limited to ChatGPT, Wenxinyiyan, and GLM models; the role of the prompt is to guide the large language model to summarize and generalize the user's depressive symptoms based on the user's posts.

7. The interactive depression intelligent detection method assisted by social media analysis according to claim 6, characterized in that: The process of depression understanding and text representation based on LLM is: Based on the user's posts on the social platform, a summary text description of the user's depressive symptoms is obtained. The text description output by LLM will be passed through a text encoder for feature extraction. The encoder uses a pre-trained model trained on a wide range of samples to ensure robustness and generalization. The vector output by the text encoder is used as the depression representation vector of social media data.

Images