Depression emotion recognition method and system based on emoticon perception and deep-wide model

Abstract

The present invention provides a method and system for identifying depression based on emoticon perception and deep-wide model, which belongs to the field of depression recognition, including: inputting the published content to be identified into a pre-trained deep module to obtain the deep features output by the deep module; inputting the published content to be identified and its attribute information into a pre-trained wide module to obtain the shallow features output by the wide module; inputting the deep features into a first classifier to obtain the emoticon prediction results output by the first classifier, and at the same time, splicing the deep features and the shallow features and inputting them into a second classifier to obtain the depression recognition results output by the second classifier as depression or non-depression. The present invention uses a multi-task learning method so that the deep features used for depression recognition simultaneously include the text and emoticon perception information of the published content to be identified, and improves the accuracy and generalization of depression recognition while introducing the emoticon modality.

Description

Technical Field

[0001] The present invention relates to the technical field of depression detection, and in particular to a depression recognition method and system based on emoticon perception and deep-wide model. Background Art

[0002] Depression has become a mental health disease that has received high attention and importance worldwide. However, due to the shortage of mental health resources, social discrimination and stigma, patients with depression generally face the dilemma of "being ill but not knowing it, and knowing it but not seeking medical treatment".

[0003] Traditional methods of depression detection rely on questionnaire assessments and physician expertise. However, the limited energy and number of mental health professionals make these methods inefficient and unable to bridge the substantial gap between the large number of depression patients.

[0004] Therefore, how to identify the depressive emotional state of the poster based on the massive social posts in social media has become an urgent problem that needs to be solved. Summary of the invention

[0005] The present invention provides a method and system for identifying depression based on emoticon perception and deep-wide model, so as to solve the defect of the prior art that there is a lack of a method for identifying the depressive state of a publisher based on the published content, and realize a method for identifying depression with high accuracy, versatility and robustness.

[0006] The present invention provides a method for identifying depression based on emoticon perception and deep-wide model, comprising:

[0007] Inputting the published content to be identified into a pre-trained deep module, and obtaining deep features output by the deep module, wherein the deep features at least represent text and emoticon perception information of the published content to be identified;

[0008] Inputting the published content to be identified and its attribute information into a pre-trained width module to obtain shallow features output by the width module;

[0009] The deep features are input into a first classifier to obtain an emoticon prediction result output by the first classifier. At the same time, the deep features and the shallow features are spliced ​​and input into a second classifier to obtain a depression emotion recognition result output by the second classifier, which is depression emotion or non-depression emotion.

[0010] According to a method for identifying depression based on emoticon perception and deep-wide model provided by the present invention, the step of inputting the published content to be identified into a pre-trained deep module and obtaining the deep features output by the deep module specifically includes:

[0011] Converting the image in the published content to be identified into descriptive text, and splicing the descriptive text to the text in the published content to be identified to obtain a spliced ​​text;

[0012] The deep features are obtained based on the concatenated text.

[0013] According to a method for identifying depression based on emoticon perception and deep-wide model provided by the present invention, the step of obtaining the deep features based on the concatenated text specifically includes:

[0014] The concatenated text is input into the BERT model in the deep module, and the output result of the BERT model is input into the BiLSTM model in the deep module. The output of the BiLSTM model is guided by the attention mechanism to obtain the deep features.

[0015] According to a method for identifying depressive emotions based on emoticon perception and a deep-wide model provided by the present invention, the shallow features include at least user-specific features, published content text features, published content image features, and published content emoticon features.

[0016] According to a method for identifying depression based on emoticon perception and deep-wide model provided by the present invention, before the step of inputting the text and image in the published content to be identified into the pre-trained deep module, it also includes:

[0017] Constructing a first data set based on multiple pieces of social platform postings containing emoticons;

[0018] A second data set is constructed based on the collected multiple posts related to depression;

[0019] The depth module, the width module, the first classifier, and the second classifier are trained using the first data set and the second data set.

[0020] According to a method for identifying depression based on emoticon perception and a deep-wide model provided by the present invention, the step of using the first data set and the second data set to train the depth module, the width module, the first classifier and the second classifier comprises:

[0021] In each training, a batch of data in the first data set and a batch of data in the second data set are taken as input together, wherein the batch of data in the first data set is used to iterate the loss of the first classifier.

[0022] The present invention also provides a depression emotion recognition system based on emoticon perception and deep-wide model, comprising:

[0023] A deep feature extraction module, used to input the published content to be identified into a pre-trained deep module, and obtain the deep features output by the deep module, wherein the deep features at least represent the text and emoticon perception information of the published content to be identified;

[0024] A shallow feature extraction module, used for inputting the published content to be identified into a pre-trained width module to obtain shallow features output by the width module;

[0025] The recognition module is used to input the deep features into the first classifier to obtain the emoticon prediction results output by the first classifier, and at the same time, splice the deep features and the shallow features and input them into the second classifier to obtain the depression recognition results output by the second classifier as depression or non-depression.

[0026] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for identifying depression emotions based on emoticon perception and deep-wide model as described above is implemented.

[0027] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for identifying depression based on emoticon perception and a deep-wide model.

[0028] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned methods for identifying depression based on emoticon perception and a deep-wide model.

[0029] The present invention provides a method and system for identifying depression based on emoticon perception and deep-wide model. Through multi-task learning, the deep features used for identifying depression include both the text and emoticon perception information of the published content to be identified. A new method for introducing emoticon modality into depression recognition is proposed, thereby realizing depression recognition based on multimodal information based on emoticon perception and deep-wide model, and improving the accuracy and generalization of depression recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1This is one of the flow charts of the depression emotion recognition method based on emoticon perception and deep-wide model provided by the present invention;

[0032] Figure 2 It is a structural schematic diagram of the EAWDDD framework in the depression emotion recognition method based on emoticon perception and deep-wide model provided by the present invention;

[0033] Figure 3 It is a schematic diagram of the BERT-BiLSTM-Attention model in the depression emotion recognition method based on emoticon perception and deep-wide model provided by the present invention;

[0034] Figure 4 (a) is one of the models used for comparative testing in the depressive emotion recognition method based on emoticon perception and deep-wide model provided by the present invention;

[0035] Figure 4 (b) is the second model used for comparative testing in the depressive emotion recognition method based on emoticon perception and deep-wide model provided by the present invention;

[0036] Figure 4 (c) is the EAWDDD framework used for comparative testing in the depressive emotion recognition method based on emoticon perception and deep-wide model provided by the present invention;

[0037] Figure 5 It is a structural schematic diagram of a depression emotion recognition system based on emoticon perception and deep-wide model provided by the present invention;

[0038] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Combine the following Figures 1 to 4 The present invention introduces a method for identifying depression based on emoticon perception and deep-wide model. Figure 1 As shown, the method includes:

[0041] Step 101, inputting the published content to be identified into a pre-trained deep module, and obtaining deep features output by the deep module, wherein the deep features at least represent the text and emoticon perception information of the published content to be identified;

[0042] Optionally, in this embodiment, a pre-built EAWDDD (Emoji-Aware Wide & Deep Learning Approach for Depression Detection, a depression emotion recognition that combines emoticon perception with a deep and wide model) framework is used to implement the depression emotion recognition method of the present invention. In other feasible implementations, other feasible frameworks can also be used to implement the depression emotion recognition method of the present invention.

[0043] like Figure 2 As shown, the EAWDDD framework includes a deep module (Multi-Modal Deep Component) for extracting deep features from the published content to be identified.

[0044] Among them, the published content to be identified is the content published by the publisher on the Internet platform.

[0045] Optionally, the published content to be identified may be in any form of text, image, video or a combination thereof.

[0046] Optionally, the published content to be identified may be published content on an Internet platform that is mainly text, picture or video.

[0047] Optionally, the text content in the published content to be identified may be pure text information, or a combination of text and emoticons, wherein the emoticons may be emoticons, emoji symbols, and / or symbols composed of various characters that express emotions and are customized by the platform to which the published content belongs.

[0048] Optionally, when the published content to be identified is an Internet platform mainly based on images, the publisher's description of the image can be used as the text content of the published content to be identified.

[0049] Optionally, when the published content to be identified is an Internet platform mainly based on videos, the title and introduction of the video can be used as the text content of the published content to be identified, or text information in the video can be extracted as the text content of the published content to be identified.

[0050] It can be understood that, when the published content to be identified includes information in multiple modalities, the deep module is used to extract the multimodal information, and finally obtain deep features that can simultaneously characterize the multimodal information.

[0051] Step 102, inputting the published content to be identified and its attribute information into a pre-trained width module to obtain shallow features output by the width module;

[0052] Continue to refer to Figure 2 ,The EAWDDD framework includes a multi-modal wide component for extracting shallow features from the published content to be identified.

[0053] Shallow feature representation is the feature simply extracted from the published content to be identified and its attribute information.

[0054] Optionally, the attribute information may include user attribute information used to characterize the user portrait, such as the number of followers, number of fans, type of following users and / or number of interactions of the publisher of the published content to be identified, which is specifically determined based on the information that can be obtained by the Internet platform to which the published content to be identified belongs.

[0055] Optionally, the attribute information may include information characterizing the degree of emotion of the published content, such as the number of emoticons and the number of each emotion word in the published content to be identified.

[0056] Optionally, the attribute information may include information characterizing the image, such as the number, hue, and average saturation of images in the published content to be identified.

[0057] In a possible implementation, Figure 2 As shown, in the width module, shallow manual features are first extracted based on the input published content to be identified and its attribute information, and a linear layer is used to connect each individual feature.

[0058] In order to effectively capture the complex interrelationships and interactions between different features, a dedicated feature interaction layer is introduced in the width module to extend the traditional width module.

[0059] In form, the published content to be identified The output of the multimodal width part, that is, the shallow feature The definition formula is as follows:

[0060] ;

[0061] In the formula, k represents the number of manual features, Indicates k Manual features, m and n is the index.

[0062] In other feasible implementations, the width module may also adopt other structures as long as it can achieve the extraction of shallow features.

[0063] Step 103, input the deep features into the first classifier to obtain the emoticon prediction results output by the first classifier, and at the same time, splice the deep features and the shallow features and input them into the second classifier to obtain the depression recognition results output by the second classifier as depression or non-depression.

[0064] For patients with depression, since their emotional expressions are diverse and their emotional states are complex, the use of multimodal data, especially the introduction of emoticons, can provide a unique advantage of more comprehensive emotional clues.

[0065] On this basis, in order to simultaneously identify the depressive emotions of posts that do not contain emoticons, the present invention defines two joint learning tasks through multi-task learning. Task one is the emoticon prediction task, which aims to predict the emoticons that represent the posts to be identified based on the deep features of the posts to be identified. Task two is the depressive emotion recognition task, which aims to jointly recognize the depressive emotions that represent the posts to be identified based on the deep and shallow features of the posts to be identified.

[0066] It can be understood that through joint learning, in the process of completing Task 1, the text of the published content to be identified is input and the semantic representation with emoticon perception is output as the deep feature, that is, the deep feature simultaneously represents the text of the published content to be identified and the emoticon perception information, so that the deep features used in the process of completing Task 2 include emoticon perception information, thereby introducing the emoticon modality information in the task of identifying depressive emotions, and can be applied to identifying depressive emotions in published content that lacks emoticons.

[0067] In a specific embodiment, Figure 2 As shown, the first classifier is used to complete task one, that is, the recognition of the emoji prediction task, so as to predict the emoji representing the published content to be identified based on the deep features. It can be understood that the emoji output by task one is specifically determined based on the emoji defined in the training process.

[0068] During the training process, the EAWDDD framework in this embodiment selects the 30 emoticons with the highest frequency of use in the publishing platform of the content to be identified for training, so the output of task one is one of the 30 emoticons.

[0069] Specifically, the deep features obtained by the deep module Input to the softmax layer (an activation function used to handle multi-classification problems) to complete the classification task of task 1, generate a probability vector of m dimensions, where each element represents the probability that the sample contains a specific emoji e in the set of m candidate emojis E, and the target emoji is finally output. , that is, the emoji prediction result of the first classifier is:

[0070] ;

[0071] In the formula, represents the model parameters, represents the input of the depth module, and m corresponds to 30 in this implementation.

[0072] At the same time, the second classifier is used to complete Task 2, which is the prediction of the depressive emotion recognition task.

[0073] Specifically, the extracted deep features and shallow features Input into the fully connected layer and the multi-layer perceptron.

[0074] The multilayer perceptron consists of a linear layer followed by a ReLU (Rectified Linear Unit) activation function. and shallow features The final representation vector obtained by concatenation It can effectively integrate the deep semantic information extracted from different modes of the published content to be identified, as well as the shallow, cross-product features derived from the published content and attribute information to be identified. Finally, a softmax layer is used as the second classifier to obtain the predicted probability distribution.

[0075] Task two in this implementation is defined as a binary classification task, that is, the recognition result of the published content to be identified is depressive emotion or non-depressive emotion. Therefore, the probabilities corresponding to depressive emotion and non-depressive emotion are obtained through the softmax layer, and the item with a larger probability is output as the prediction result of Task two, thereby realizing the recognition of the depressive emotion represented by the published content to be identified.

[0076] The present invention uses a multi-task learning approach so that the deep features used for depression recognition simultaneously include the text and emoticon perception information of the published content to be recognized, and proposes a new approach to introduce emoticon modality into depression recognition, thereby realizing depression recognition based on multimodal information based on emoticon perception and a deep-wide model, and can be applied to the prediction of published content without emoticons, thereby improving the accuracy and generalization of depression recognition.

[0077] In the depressive emotion recognition method based on emoticon perception and deep-wide model of the present invention, the step of inputting the published content to be recognized into a pre-trained deep module and obtaining the deep features output by the deep module specifically includes:

[0078] Converting the image in the published content to be identified into descriptive text, and splicing the descriptive text to the text in the published content to be identified to obtain a spliced ​​text;

[0079] The deep features are obtained based on the concatenated text.

[0080] like Figure 2 As shown in the figure, the image in the published content to be identified is converted into the image in the published content using the SOTA (State-of-the-arts, a model with good performance in this field) model BLIP (Bootstrapping Language-Image Pre-training, a pre-training model for unified visual language understanding and generation) in the field of image captioning. Convert to corresponding descriptive text .in, Indicates a published content to be identified N In the image i images; Indicates a piece of published content to be identified N In the image i Descriptive text converted from an image.

[0081] BLIP is based on a multimodal mixture of Encoder-Decoder (MED) architecture. It consists of a text encoder and an image encoder. The text encoder contains cross-attention layers to capture visual features, while the decoder uses causal self-attention to generate a text description of the image. Next, the generated Text content that is sequentially linked to the published content to be identified , and Update to get new , will update As concatenated text:

[0082] ;

[0083] Where L represents the maximum length of the published content to be identified.

[0084] It can be understood that, through the above method, the information of the image modality is integrated into the information of the text modality to obtain a spliced ​​text, and deep features are extracted based on the spliced ​​text features. Therefore, when the published content to be identified contains images, the extracted deep features can also contain information of the image modality, which is used to more accurately realize the prediction of Task 1 and Task 2.

[0085] It should be noted that the above method is still applicable when the published content to be identified does not contain images, so that the extracted deep features can at least include text information and emoticon perception information of the published content to be identified, thereby improving the overall generalization and robustness of the model.

[0086] In the depression emotion recognition method based on emoticon perception and deep-wide model of the present invention, the step of obtaining the deep features based on the spliced ​​text specifically includes:

[0087] The concatenated text is input into the BERT model in the deep module, and the output result of the BERT model is input into the BiLSTM model in the deep module. The output of the BiLSTM model is guided by the attention mechanism to obtain the deep features.

[0088] like Figure 2 As shown, in order to extract deep features based on concatenated text, DeEmoji2Vec (Depression Emoji-aware representation to Vector) is also defined in the EAWDDD framework in this embodiment, which is specifically a semantic representation layer that can map text, images and emoticons into a unified semantic space.

[0089] DeEmoji2Vec is based on Figure 3 The BERT-BiLSTM-Attention model shown in the figure extracts deep semantic representations. Compared with directly using the BERT (Bidirectional Encoder Representations from Transformers, a deep learning model) model, combining BERT with BiLSTM (Bidirectional Long Short-Term Memory, a recurrent neural network based on a long short-term memory network) can effectively capture the long-distance dependencies between input text sequence tokens, which can better solve the inherent limitations of the BERT architecture in modeling long-distance dependencies.

[0090] In addition, the self-attention mechanism is used to give the model the ability to dynamically focus on different parts of the input sequence. It enables the model to better focus on information that is critical to downstream tasks, thereby potentially improving the performance of downstream tasks.

[0091] Specifically, the concatenated text Input the pre-trained BERT model and get the embedding of Where L is the maximum length of the text, is the number of units in the last hidden layer of BERT, which is 768 in this implementation.

[0092] Then use BiLSTM to transform the representation of each tag Convert to contextual representation , where represents the dimension of each hidden state vector, represents the lth word of the ith input text, and L represents the length of the input text. The calculation formula is as follows:

[0093] ;

[0094] In the formula, For each input word BiLSTM representation of .

[0095] It can be understood that the spliced ​​text in this embodiment When the published content to be identified contains an image, the description information corresponding to the image is concatenated with the text content in the published content to be identified; when the published content to be identified does not contain an image, the description information is the same as the text content in the published content to be identified.

[0096] Furthermore, since not all words in the text content of the published content to be identified are equally helpful in identifying depression, the present invention introduces a multi-head attention mechanism to capture key information in the input sequence, thereby enhancing its representation ability. The multi-head attention mechanism allows the model to focus on different representations of the input sequence at the same time, capturing a variety of language attributes and dependencies. The calculation formula of the multi-head attention is as follows:

[0097] ;

[0098] In the formula, represents the number of heads of multi-head attention, For multi-head attention i The head is defined as follows:

[0099] ;

[0100] The scaled dot product attention formula is as follows:

[0101] ;

[0102] Where Q, K, and V represent the query, key, and value matrices derived from the concatenated BiLSTM hidden states. , and is a learnable projection matrix, is the learnable output projection matrix. , and Dimensions representing the number of queries, keys, values, and attention heads, respectively.

[0103] After being guided by the attention mechanism, the output is the extracted deep features .

[0104] The present invention integrates BERT, BiLSTM and multi-head self-attention and takes advantage of their complementary advantages, enabling them to effectively capture local and global contextual information and build complex language dependency models in the input text.

[0105] In the depressive emotion recognition method based on emoticon perception and deep-wide model of the present invention, the shallow features include at least user-specific features, published content text features, published content image features and published content emoticon features.

[0106] Since the attribute characteristics of the publisher of the published content to be identified and the published content itself are helpful for identifying depression, including user gender, socioeconomic status, posting time, negative vocabulary ratio, first-person pronoun ratio, etc. Therefore, the low-dimensional and sparse features are modeled through the linear layer through the width module.

[0107] In this embodiment, the width module is used to extract four different shallow features based on the input published content to be identified and its attribute information: user-specific features, published content text features, published content image features and published content emoticon features, as shown in Table 1:

[0108] Table 1

[0109]

[0110] In summary, the obtained shallow features integrate features from users, texts, images, and emoticons.

[0111] In the method for identifying depression based on emoticon perception and deep-wide model of the present invention, before the step of inputting the text and image in the published content to be identified into the pre-trained deep module, it also includes:

[0112] Constructing a first data set based on multiple pieces of social platform postings containing emoticons;

[0113] A second data set is constructed based on the collected multiple posts related to depression;

[0114] The depth module, the width module, the first classifier, and the second classifier are trained using the first data set and the second data set.

[0115] In order to train the model corresponding to the method for identifying depression provided by the present invention, that is, the EAWDDD model in this embodiment, two data sets are pre-constructed.

[0116] The first data set includes multiple published contents, each of which includes corresponding emoticons for the emoticon prediction task. In this implementation, a total of 72,283 published contents are collected as data in the first data set.

[0117] Furthermore, the distribution of emoji usage was analyzed, and the 30 most frequently used emojis were selected as candidate labels. For posts containing multiple emojis, they were split into several separate samples, each of which was labeled with a single emoji, and finally the first dataset for training was obtained.

[0118] The second data set includes multiple posts, each of which is associated with depression. In a feasible way, for different Internet platforms, posts under depression topics and depression tags are collected as posts associated with depression. In this implementation, a total of 37,218 posts are collected into the second data set for the depression recognition task.

[0119] Furthermore, to ensure the reliability of the labels, this study was based on the principle of cross-validation and manually annotated by five professionally trained annotators. If the user reported having depression or described being treated or taking antidepressants, the post was classified as a positive sample; otherwise, they were labeled as a negative sample. In the end, a total of 12,360 positive samples and 24,858 negative samples were obtained, which were then divided into training and test sets at a ratio of 0.8:0.2, eventually forming the second data set for training.

[0120] The depth module, the width module, the first classifier and the second classifier are pre-trained using the first data set and the second data set.

[0121] In the depression emotion recognition method based on emoticon perception and deep-wide model of the present invention, the step of using the first data set and the second data set to train the depth module, the width module, the first classifier and the second classifier comprises:

[0122] In each training, a batch of data in the first data set and a batch of data in the second data set are taken as input together, wherein the batch of data in the first data set is used to iterate the loss of the first classifier.

[0123] Specifically, in each training, a batch of data in the first data set and a batch of data in the second data set are taken as input together.

[0124] Since the invention is based on multi-task learning, model performance is enhanced by sharing common model structures and parameters among multiple tasks, while using task-specific output layers to promote parallel learning.

[0125] For Task 1 and Task 2 defined in the present invention, considering the serious sample imbalance in both tasks, FocalLoss is used as the loss function, which introduces a dynamic scaling factor to reduce the weight of samples that are easy to classify, thereby focusing the model's attention on samples that are difficult to classify. The definition formula of FocalLoss is as follows:

[0126] ;

[0127] In the formula, is the predicted probability, is the class imbalance weighting factor, is the focusing parameter, is the class balance parameter.

[0128] Furthermore, in the multi-task learning process, the present invention uses a simplified uncertainty weighted loss to optimize the impact of the two loss functions on parameter updates. This method can dynamically adjust the weight of the loss function according to the uncertainty of each task to ensure the balanced contribution of the two tasks during the training process. The loss function is expressed as:

[0129] ;

[0130] In the formula, and is the task-specific uncertainty parameter, and Individual loss functions for the emoji prediction task and the depression recognition task. Among them, a batch of data in the first dataset is used to calculate the loss of the first classifier Iterations are performed. Experimental verification shows that the above loss function and multi-task learning strategy can effectively combine the emoji prediction task and the depression emotion recognition task, and surpass the existing baseline method in the latter, achieving the best performance.

[0131] In this implementation, the batch size is 32, so a batch of 32 data is input for one training.

[0132] It should be particularly noted that, since the present invention introduces the modality of emoticons into the process of identifying depressive emotions, taking into account the overall generalization and robustness of the method, unlike the conventional method, feature extraction is performed on the information of each modality separately, and then prediction is achieved through attention guidance. In the application of image modality, it is converted into text description and spliced ​​with text content. In the emoticon modality, the emoticon prediction task is constructed through multi-task learning. This allows the present invention to achieve effective prediction of published content to be identified that does not contain images and emoticons.

[0133] In order to verify the performance of the depression recognition method provided by the present invention, Figure 4 As shown, during the test, two alternative multimodal information fusion strategies different from the present invention were designed for comparison.

[0134] in, Figure 4 The method shown in (a) adopts the traditional multimodal representation method BERT_VGG16_emoji2vec (BVe). Among them, BERT is used for text representation, VGG16 is used for image representation, and emoji2vec is used for emoji representation. These representations are then input into a deep neural network (DNN) to obtain the final prediction results.

[0135] Figure 4 The method shown in (b) in Figure 1 combines the emoji prediction task and the depression detection task while also adopting a sequential training strategy (pipeline). The model is first trained on the emoji prediction task to develop an emoji-aware text representation model, and then trained on the depression detection task based on this.

[0136] Figure 4 (c) is a depression recognition method based on emoji perception and deep-wide model provided by the present invention, which includes two tasks: emoji prediction task and depression recognition task. The two share the text representation layer DeEmoji2Vec to capture the semantic representation of expression perception.

[0137] In addition, comparative tests were conducted with an online social network depressed individual detection method (FusionNet) based on a multi-task heterogeneous modality fusion method, a multi-aspect depression detection with hierarchical attention network (MDHAN), and a multimodal hierarchical attention model for depression detection in social media (MHA).

[0138] The test results of the above six models are shown in Table 2 below:

[0139] Table 2

[0140]

[0141] As shown in Table 2, the EAWDDD model provided by the present invention has the highest weighted F1 score. Specifically, in the recognition of depressive emotions, the precision (Precision), recall rate (Recall) and F1 score are all the highest values; in the recognition of non-depressive emotions, the precision (Precision) and F1 score are also the highest values, and the recall rate (Recall) is the second highest value.

[0142] By comparing different multimodal fusion strategies, the proposed multi-task learning framework shows excellent performance in capturing the semantic representation of emoji perception. The proposed model achieves the best overall performance, especially in identifying depression samples, which highlights the importance of integrating multimodal information in mental health detection tasks.

[0143] The following describes the depression emotion recognition system based on expression symbol perception and deep and wide model provided by the present invention. The depression emotion recognition system based on expression symbol perception and deep and wide model described below and the depression emotion recognition method based on expression symbol perception and deep and wide model described above can be referenced to each other.

[0144] like Figure 5 As shown, the depression emotion recognition system based on emoticon perception and deep-wide model includes a deep feature extraction module 501, a shallow feature extraction module 502 and a recognition module 503:

[0145] A deep feature extraction module 501 is used to input the published content to be identified into a pre-trained deep module to obtain deep features output by the deep module, wherein the deep features at least represent the text and emoticon perception information of the published content to be identified;

[0146] Optionally, in this embodiment, a pre-built EAWDDD (Emoji-Aware Wide & Deep Learning Approach for Depression Detection, a depression emotion recognition that combines emoticon perception with a deep and wide model) framework is used to implement the depression emotion recognition method of the present invention. In other feasible implementations, other feasible frameworks can also be used to implement the depression emotion recognition method of the present invention.

[0147] like Figure 2 As shown, the EAWDDD framework includes a deep module (Multi-Modal Deep Component) for extracting deep features from the published content to be identified.

[0148] Among them, the published content to be identified is the content published by the publisher on the Internet platform.

[0149] Optionally, the published content to be identified may be in any form of text, image, video or a combination thereof.

[0150] Optionally, the published content to be identified may be published content on an Internet platform that is mainly text, picture or video.

[0151] Optionally, the text content in the published content to be identified may be pure text information, or a combination of text and emoticons, wherein the emoticons may be emoticons, emoji symbols, and / or symbols composed of various characters that express emotions and are customized by the platform to which the published content belongs.

[0152] Optionally, when the published content to be identified is an Internet platform mainly based on images, the publisher's description of the image can be used as the text content of the published content to be identified.

[0153] Optionally, when the published content to be identified is an Internet platform mainly based on videos, the title and introduction of the video can be used as the text content of the published content to be identified, or text information in the video can be extracted as the text content of the published content to be identified.

[0154] It can be understood that, when the published content to be identified includes information in multiple modalities, the deep module is used to extract the multimodal information, and finally obtain deep features that can simultaneously characterize the multimodal information.

[0155] A shallow feature extraction module 502 is used to input the published content to be identified into a pre-trained width module to obtain shallow features output by the width module;

[0156] Continue to refer to Figure 2 ,The EAWDDD framework includes a multi-modal wide component for extracting shallow features from the published content to be identified.

[0157] Shallow feature representation is the feature simply extracted from the published content to be identified and its attribute information.

[0158] Optionally, the attribute information may include user attribute information used to characterize the user portrait, such as the number of followers, number of fans, type of following users and / or number of interactions of the publisher of the published content to be identified, which is specifically determined based on the information that can be obtained by the Internet platform to which the published content to be identified belongs.

[0159] Optionally, the attribute information may include information characterizing the degree of emotion of the published content, such as the number of emoticons and the number of each emotion word in the published content to be identified.

[0160] Optionally, the attribute information may include information characterizing the image, such as the number, hue, and average saturation of images in the published content to be identified.

[0161] In a possible implementation, Figure 2 As shown, in the width module, shallow manual features are first extracted based on the input published content to be identified and its attribute information, and a linear layer is used to connect each individual feature.

[0162] In order to effectively capture the complex interrelationships and interactions between different features, a dedicated feature interaction layer is introduced in the width module to extend the traditional width module.

[0163] In form, the published content to be identified The output of the multimodal width part, that is, the shallow feature The definition formula is as follows:

[0164] ;

[0165] In the formula, k represents the number of manual features, Indicates k Manual features, m and n is the index.

[0166] In other feasible implementations, the width module may also adopt other structures as long as it can achieve the extraction of shallow features.

[0167] The recognition module 503 is used to input the deep features into the first classifier to obtain the emoticon prediction results output by the first classifier. At the same time, the deep features and the shallow features are spliced ​​and input into the second classifier to obtain the depression recognition results output by the second classifier as depression or non-depression.

[0168] For patients with depression, since their emotional expressions are diverse and their emotional states are complex, the use of multimodal data, especially the introduction of emoticons, can provide a unique advantage of more comprehensive emotional clues.

[0169] On this basis, in order to simultaneously identify the depressive emotions of posts that do not contain emoticons, the present invention defines two joint learning tasks through multi-task learning. Task one is the emoticon prediction task, which aims to predict the emoticons that represent the posts to be identified based on the deep features of the posts to be identified. Task two is the depressive emotion recognition task, which aims to jointly recognize the depressive emotions that represent the posts to be identified based on the deep and shallow features of the posts to be identified.

[0170] It can be understood that through joint learning, in the process of completing Task 1, the text of the published content to be identified is input and the semantic representation with emoticon perception is output as the deep feature, that is, the deep feature simultaneously represents the text of the published content to be identified and the emoticon perception information, so that the deep features used in the process of completing Task 2 include emoticon perception information, thereby introducing the emoticon modality information in the task of identifying depressive emotions, and can be applied to identifying depressive emotions in published content that lacks emoticons.

[0171] In a specific embodiment, Figure 2 As shown, the first classifier is used to complete task one, that is, the recognition of the emoji prediction task, so as to predict the emoji representing the published content to be identified based on the deep features. It can be understood that the emoji output by task one is specifically determined based on the emoji defined in the training process.

[0172] During the training process, the EAWDDD framework in this embodiment selects the 30 emoticons with the highest frequency of use in the publishing platform of the content to be identified for training, so the output of task one is one of the 30 emoticons.

[0173] Specifically, the deep features obtained by the deep module The input is sent to the softmax layer to complete the classification task of task 1, generating a probability vector of m dimensions, where each element represents the probability that the sample contains a specific emoji e in the set of m candidate emojis E, and the target emoji is finally output. , that is, the emoji prediction result of the first classifier is:

[0174] ;

[0175] In the formula, represents the model parameters, represents the input of the depth module, and m corresponds to 30 in this implementation.

[0176] At the same time, the second classifier is used to complete Task 2, which is the prediction of the depressive emotion recognition task.

[0177] Specifically, the extracted deep features and shallow features Input into the fully connected layer and the multi-layer perceptron.

[0178] Among them, the multi-layer perceptron consists of a linear layer followed by a ReLU activation function. and shallow features The final representation vector obtained by concatenation It can effectively integrate the deep semantic information extracted from different modalities of the published content to be identified, as well as the shallow, cross-product features derived from… Finally, a softmax layer is used as the second classifier to obtain the predicted probability distribution.

[0179] Task 2 in this implementation is defined as a binary classification task, that is, the recognition result of the published content to be identified is depressive emotion or non-depressive emotion. Therefore, the probabilities corresponding to depressive emotion and non-depressive emotion are obtained through the softmax layer, and the item with a larger probability is output as the prediction result of Task 2, thereby realizing the recognition of the depressive emotion represented by the published content to be identified.

[0180] The present invention uses a multi-task learning approach so that the deep features used for depression recognition simultaneously include the text and emoticon perception information of the published content to be recognized, and proposes a new approach to introduce emoticon modality into depression recognition, thereby realizing depression recognition based on multimodal information based on emoticon perception and deep-wide models, thereby improving the accuracy and generalization of depression recognition.

[0181] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a method for identifying depression based on emoticon perception and a deep-wide model, the method comprising: inputting the published content to be identified into a pre-trained deep module, obtaining deep features output by the deep module, wherein the deep features at least characterize the text and emoticon perception information of the published content to be identified; inputting the published content to be identified and its attribute information into a pre-trained wide module, obtaining shallow features output by the wide module; inputting the deep features into a first classifier, obtaining an emoticon prediction result output by the first classifier, and at the same time, splicing the deep features and the shallow features into a second classifier, obtaining a depression recognition result output by the second classifier as depression or non-depression.

[0182] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0183] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the depression emotion recognition method based on emoticon perception and deep-wide model provided by the above-mentioned methods, and the method includes: inputting the published content to be identified into a pre-trained deep module, and obtaining the deep features output by the deep module, wherein the deep features at least characterize the text and emoticon perception information of the published content to be identified; inputting the published content to be identified and its attribute information into a pre-trained wide module, and obtaining the shallow features output by the wide module; inputting the deep features into a first classifier, and obtaining the emoticon prediction result output by the first classifier, and at the same time, splicing the deep features and the shallow features and inputting them into a second classifier, and obtaining the depression emotion recognition result output by the second classifier as depression emotion or non-depression emotion.

[0184] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the depression emotion recognition method based on emoticon perception and deep-wide model provided by the above-mentioned methods, the method comprising: inputting the published content to be identified into a pre-trained deep module, obtaining the deep features output by the deep module, wherein the deep features at least characterize the text and emoticon perception information of the published content to be identified; inputting the published content to be identified and its attribute information into a pre-trained wide module, obtaining the shallow features output by the wide module; inputting the deep features into a first classifier, obtaining the emoticon prediction result output by the first classifier, and at the same time, splicing the deep features and the shallow features and inputting them into a second classifier, obtaining the depression emotion recognition result output by the second classifier as depression emotion or non-depression emotion.

[0185] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0186] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying depression based on emoticon perception and deep-wide model, characterized in that: include: Inputting the published content to be identified into a pre-trained deep module, and obtaining deep features extracted by a semantic representation layer in the deep module, wherein the published content to be identified includes at least text content, and the deep features at least represent the text of the published content to be identified; Inputting the published content to be identified and its attribute information into a pre-trained width module to obtain shallow features output by the width module, wherein the attribute information includes user attribute information representing a user portrait; A depression recognition task is performed in a multi-task learning-based manner, wherein task one is an emoji prediction task and task two is a depression recognition task; the deep features are input into a first classifier to obtain an emoji prediction result output by the first classifier to perform the emoji prediction task, and in the process of completing task one, a semantic representation with emoji perception is output as a new deep feature, and the new deep feature simultaneously represents the text and emoji perception information of the published content to be identified, so that the deep features used in the process of completing task two contain emoji perception information; the new deep features and the shallow features are spliced ​​and input into a second classifier to obtain a depression recognition result output by the second classifier as depression or non-depression.

2. The method for identifying depression based on emoticon perception and deep-wide model according to claim 1, characterized in that: The step of inputting the published content to be identified into a pre-trained deep module and obtaining deep features extracted and outputted by the semantic representation layer in the deep module specifically includes: Converting the image in the published content to be identified into descriptive text, and splicing the descriptive text to the text in the published content to be identified to obtain a spliced ​​text; The deep features are obtained based on the concatenated text.

3. The method for identifying depression based on emoticon perception and deep-wide model according to claim 2, characterized in that: The step of obtaining the deep features based on the concatenated text specifically includes: The concatenated text is input into the BERT model in the deep module, and the output result of the BERT model is input into the BiLSTM model in the deep module. The output of the BiLSTM model is guided by the attention mechanism to obtain the deep features.

4. The method for identifying depression based on emoticon perception and deep-wide model according to claim 1, characterized in that: The shallow features include at least user-specific features, published content text features, published content image features, and published content emoticon features.

5. The method for identifying depression based on emoticon perception and deep-wide model according to any one of claims 1 to 4, characterized in that: Before the step of inputting the published content to be identified into the pre-trained deep module, the method further includes: Constructing a first data set based on multiple pieces of social platform postings containing emoticons; A second data set is constructed based on the collected multiple posts related to depression; The depth module, the width module, the first classifier, and the second classifier are trained using the first data set and the second data set.

6. The method for identifying depression based on emoticon perception and deep-wide model according to claim 5, characterized in that: The step of using the first data set and the second data set to train the depth module, the width module, the first classifier and the second classifier comprises: In each training, a batch of data in the first data set and a batch of data in the second data set are taken as input together, wherein the batch of data in the first data set is used to iterate the loss of the first classifier.

7. A depression emotion recognition system based on emoticon perception and deep-wide model, characterized in that: include: A deep feature extraction module, used for inputting the published content to be identified into a pre-trained deep module, and obtaining deep features extracted by the semantic representation layer in the deep module, wherein the published content to be identified includes at least text content, and the deep features at least represent the text of the published content to be identified; A shallow feature extraction module, used for inputting the published content to be identified and its attribute information into a pre-trained width module to obtain shallow features output by the width module, wherein the attribute information includes user attribute information representing a user portrait; A recognition module is used to perform a depression recognition task in a multi-task learning-based manner, wherein Task 1 is an emoji prediction task and Task 2 is a depression recognition task; the deep features are input into a first classifier to obtain an emoji prediction result output by the first classifier to perform the emoji prediction task, and in the process of completing Task 1, a semantic representation with emoji perception is output as a new deep feature, and the new deep feature simultaneously represents the text and emoji perception information of the published content to be recognized, so that the deep features used in the process of completing Task 2 contain emoji perception information; the new deep features and the shallow features are spliced ​​and input into a second classifier to obtain a depression recognition result output by the second classifier as depression or non-depression.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for identifying depression emotions based on emoticon perception and deep-wide model is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for identifying depression emotions based on emoticon perception and a deep-wide model is implemented as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for identifying depression emotions based on emoticon perception and a deep-wide model is implemented as described in any one of claims 1 to 6.

Images