Anxiety tendency identification method and device based on social media data and electronic equipment

Abstract

The invention provides an anxiety tendency identification method and device of social media data and electronic equipment, and relates to the technical field of information identification. According to the specific implementation scheme, to-be-recognized social media data and auxiliary information associated with the to-be-recognized social media data are obtained, the to-be-recognized social media data comprise picture information and text information, and the auxiliary information comprises release time and interaction information of the to-be-recognized social media data; inputting the social media data to be identified and the auxiliary information into the trained anxiety tendency detection model for identification; and obtaining an identification result of the anxiety tendency detection model on the to-be-identified social media data. According to the technical scheme, the anxiety tendency detection of the social media data can be carried out by combining information of multiple dimensions such as pictures, texts, release time and interaction conditions, and the social media data valuable for anxiety emotion research can be efficiently and accurately screened out from a large amount of data on a social media platform; and therefore, the data validity of anxiety emotion research is improved.

Description

Technical Field

[0001] This application relates to the field of information recognition technology, and in particular, to a method, device, and electronic device for identifying anxiety tendencies in social media data. Background Art

[0002] With the development of the Internet, the scale of social media has been continuously expanding. On social media platforms, hundreds of millions of users share their hobbies, personal daily lives, and life experiences every day. The content in this social media data includes text, pictures, videos, audio, etc., and often contains relatively rich emotional information. Users with anxiety tendencies are more willing to express their true inner feelings on social media. These public and real data information provides a new data source for the study of anxiety emotions. Therefore, it is necessary to extract social media data with anxiety tendencies from a large amount of social media data for research. Summary of the Invention

[0003] Embodiments of this application provide a method, device, electronic device, and storage medium for identifying anxiety tendencies in social media data to identify social media data with anxiety tendencies.

[0004] In a first aspect, embodiments of this application provide a method for identifying anxiety tendencies in social media data, including: obtaining social media data to be identified and auxiliary information associated with the social media data to be identified, where the social media data to be identified includes picture information and text information, and the auxiliary information includes the release time and interaction information of the social media data to be identified; inputting the social media data to be identified and the auxiliary information into a trained anxiety tendency detection model for identification; obtaining an identification result of the anxiety tendency detection model for the social media data to be identified, where the identification result is used to indicate whether the social media data to be identified has anxiety tendencies;

[0005] Among them, inputting the social media data to be recognized and auxiliary information into the trained anxiety tendency detection model for recognition includes: generating picture description information of the picture information through a self-supervised language-image pre-training model; extracting feature vectors of the picture description information by using a text encoder to obtain picture description feature vectors; extracting feature vectors of the picture information by using an image encoder to obtain picture visual feature vectors; extracting feature vectors of the text information by using a text encoder to obtain text feature vectors; performing feature fusion on the picture description feature vectors and the picture visual feature vectors based on a multi-head cross-attention mechanism to obtain fused picture feature vectors; performing feature fusion on the picture description feature vectors and the text feature vectors based on a multi-head cross-attention mechanism to obtain fused text feature vectors; extracting feature vectors of the auxiliary information to obtain auxiliary feature vectors of the social media data to be recognized; performing feature fusion on the fused picture feature vectors, the fused text feature vectors and the auxiliary feature vectors to obtain final feature vectors; and determining the recognition result for the social media data to be recognized based on the final feature vectors.

[0006] In a possible implementation manner, generating picture description information of the picture information through a self-supervised language-image pre-training model includes: generating abstract semantic content of the picture information based on the self-supervised language-image pre-training model; extracting the original text content in the picture information by using character recognition technology; and generating the picture description information of the picture information according to the abstract semantic content and the original text content in the picture information.

[0007] In a possible implementation manner, performing feature fusion on the fused picture feature vectors, the fused text feature vectors and the auxiliary feature vectors to obtain final feature vectors includes: splicing the auxiliary feature vectors with the fused picture feature vectors and the fused text feature vectors respectively to obtain enhanced picture feature vectors and enhanced text feature vectors; and performing feature fusion on the enhanced picture feature vectors and the enhanced text feature vectors based on a cross-attention mechanism to obtain final feature vectors.

[0008] In a possible implementation manner, obtaining the social media data to be recognized includes: screening the original social media data and retaining the social media data that contains both picture information and text information; and performing data preprocessing on the screened social media data to obtain the social media data to be recognized, where the data preprocessing includes data cleaning of the text information and normalization processing of the picture information.

[0009] In a possible implementation manner, the training method of the anxiety tendency detection model includes: inputting the sample social media data and the auxiliary information associated with the sample social media data into the neural network model to be trained, and obtaining the recognition result of the neural network model for the sample social media data; adjusting the parameters of the neural network model according to the recognition result and the annotation information for the sample social media data until the error between the recognition result and the annotation information is less than a preset threshold, and using this neural network model as the anxiety tendency detection model, where the annotation information indicates that the sample social media data has an anxiety tendency or the sample social media data does not have an anxiety tendency.

[0010] In a second aspect, an embodiment of the present application provides an anxiety tendency recognition device for social media data, including: a data acquisition module, configured to acquire the social media data to be recognized and the auxiliary information associated with the social media data to be recognized, where the social media data to be recognized includes picture information and text information, and the auxiliary information includes the release time and interaction information of the social media data to be recognized; a recognition module, configured to input the social media data to be recognized and the auxiliary information into the trained anxiety tendency detection model for recognition; the recognition module is further configured to obtain the recognition result of the anxiety tendency detection model for the social media data to be recognized, and the recognition result is used to indicate whether the social media data to be recognized has an anxiety tendency;

[0011] The recognition module is further configured to: generate picture description information of the picture information through a self-supervised language-image pre-training model; extract features of the picture description information by using a text encoder to obtain a picture description feature vector; extract features of the picture information by using an image encoder to obtain a picture visual feature vector; extract features of the text information by using a text encoder to obtain a text feature vector; perform feature fusion on the picture description feature vector and the picture visual feature vector based on a multi-head cross-attention mechanism to obtain a fused picture feature vector; perform feature fusion on the picture description feature vector and the text feature vector based on a multi-head cross-attention mechanism to obtain a fused text feature vector; extract features of the auxiliary information to obtain an auxiliary feature vector of the social media data to be recognized; perform feature fusion on the fused picture feature vector, the fused text feature vector, and the auxiliary feature vector to obtain a final feature vector; and determine the recognition result for the social media data to be recognized based on the final feature vector.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory, where the processor implements the method according to any one of the embodiments of the present application when executing the computer program.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and the computer program implements the method according to any one of the embodiments of the present application when executed by a processor.

[0014] Based on the method of the embodiments of the present application, it is possible to detect the anxiety tendency of social media data by combining information from multiple dimensions such as pictures, text, release time, and interaction situations, so as to efficiently and accurately screen out valuable social media data for anxiety emotion research from a large amount of data on social media platforms, thereby improving the data effectiveness of anxiety emotion research.

[0015] The above description is only an overview of the technical solution of the present application. In order to be able to understand the technical means of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features, and advantages of the present application more obvious and understandable, the following specifically illustrates the specific embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings are used to better understand the solution and do not constitute a limitation to the present application. Among them:

[0017] Figure 1 is a flowchart of a method for identifying the anxiety tendency of social media data provided by an exemplary embodiment of the present application Figure 1 ;

[0018] Figure 2 is a flowchart of a method for identifying the anxiety tendency of social media data provided by an exemplary embodiment of the present application Figure 2 ;

[0019] Figure 3 is a schematic diagram of the functional structure of an anxiety tendency detection model provided by an exemplary embodiment of the present application;

[0020] Figure 4 is a schematic diagram of the processing flow of an anxiety tendency detection model for social media data to be identified provided by an exemplary embodiment of the present application;

[0021] Figure 5 is a schematic diagram of a device for identifying the anxiety tendency of social media data provided by an exemplary embodiment of the present application;

[0022] Figure 6 is a block diagram of an electronic device for implementing the embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] In the following, only some exemplary embodiments are simply described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the concept or scope of the present application. Therefore, the drawings and the description are considered to be exemplary in nature and not restrictive.

[0024] To facilitate the understanding of the technical solutions of the embodiments of the present application, the related technologies of the embodiments of the present application are described below. The following related technologies can be arbitrarily combined with the technical solutions of the embodiments of the present application as optional solutions, and they all fall within the protection scope of the embodiments of the present application.

[0025] Application scenario

[0026] Compared with methods such as questionnaire surveys for collecting research samples, using social media data on social media platforms as research samples to study anxiety tendency problems can reduce direct contact with subjects, achieve "non-intrusive" surveys, and greatly reduce the manpower and time required for data collection, thereby reducing the survey cost.

[0027] In addition, for some users, when they have anxiety emotions, they may not be willing to reveal their true inner feelings when being investigated. Moreover, some users may be difficult to objectively and clearly perceive their own anxiety emotions. However, in the era of the prevalence of social media, users may be more inclined to express their true feelings or describe their recent experiences on social media. Therefore, collecting publicly available social media data on social media platforms as the object of anxiety tendency research not only effectively expands the research samples, but also the data authenticity and research value are often superior to offline surveys.

[0028] Most of the anxiety tendency detection studies use the text in social media data as the classification basis to identify the emotional state of users, so as to determine the social media data with anxiety tendency for subsequent research. However, when anxiety disorder patients express their thoughts on social media, there may be a certain degree of obscurity and implicitness. Simply analyzing the text of users' posts does not always accurately detect the anxiety tendency of users.

[0029] Considering that the text content in social media data may not directly express the true thoughts of users. For example, a note published by a user on platform A has the content "I'm so happy today" in the text part of the note, while the picture part of the note describes the unhappy things that happened today. Another example is that the caption of a post published by a user on platform B is "This luck is unbeatable", while the accompanying picture of the post is many unwinning scratch-off tickets. It can be clearly seen that some users in social media data use rhetorical devices such as saying the opposite of what they mean. It is difficult to identify such social media data only by performing emotion recognition on the text content in social media data, which indicates that the picture content in social media data also plays an important role in anxiety tendency recognition.

[0030] In view of this, the present application provides a method for identifying anxiety tendencies in social media data, which can detect anxiety tendencies in social media data by combining the text content and image content of social media data. The anxiety tendency detection model used has a simple model structure and low training cost, and can improve the recognition rate of social media data containing anxious emotions or with anxiety tendencies, and quickly and efficiently extract social media data with anxiety tendencies from massive social media data for research.

[0031] Exemplary method

[0032] Figure 1 FIG is a flow chart of a method for identifying anxiety tendency in social media data provided by an exemplary embodiment of the present application. This embodiment can be applied to electronic devices such as Figure 1 As shown, the method includes the following steps:

[0033] Step S101: Acquire to-be-identified social media data and auxiliary information associated with the to-be-identified social media data. The to-be-identified social media data includes image information and text information. The auxiliary information includes the release time and interactive information of the to-be-identified social media data.

[0034] For example, the source of the social media data to be identified may be content publicly posted by users on various social media platforms, and the interactive information associated with the social media data includes the number of likes, reposts, and comments on the social media data by other users on the social media platform.

[0035] It's understandable that users experiencing anxiety are more likely to post longer, emotionally expressive social media posts than other users or those experiencing non-anxious states. These posts are often more engaging and likely to receive more likes and comments than shorter or less emotional posts. Research has shown that longer, emotional posts receive an average of 26% more likes and 75% more comments than regular posts.

[0036] Moreover, the posting time of social media data is also closely related to the current mood of the user to whom it belongs. Table 1 conducts a comparative analysis of social media data on a social media platform that contains anxiety and does not contain anxiety according to the posting time. As shown in Table 1, if a day is divided into four time periods: 00:00 - 06:00, 06:00 - 12:00, 12:00 - 18:00, and 18:00 - 24:00, and the proportion of the two types of social media data posted in each time period in the total amount of social media data posted on the same day is determined respectively, it can be found that in the time period of 00:00 - 06:00, the proportion difference between the two types of social media data is the largest. Thus, it can be reflected that in the time period of 00:00 - 06:00, the possibility that the social media data posted on the social media platform contains anxiety is higher than that without anxiety.

[0037] Table 1

[0038] Including anxiety Not including anxiety Proportion difference 00:00-06:00 13.41％ 10.15％ 3.26％ 06:00-12:00 18.95％ 18.55％ 0.40％ 12:00-18:00 27.09％ 30.30％ 2.81％ 18:00-24:00 40.55％ 41.00％ -0.45％

[0039] Therefore, when performing emotion recognition on social media data, in addition to obtaining the picture information and text information of the social media data itself, the posting time of the social media data and interactive information such as the number of likes, forwards, and comments can also be extracted as auxiliary information to assist in the emotion recognition of social media data.

[0040] Exemplarily, in step S101, obtaining the social media data to be recognized includes: screening the original social media data and retaining the social media data that contains both picture information and text information; performing data preprocessing on the screened social media data to obtain the social media data to be recognized, and the data preprocessing includes data cleaning of the text information and standardization processing of the picture information.

[0041] It can be understood that social media data may contain redundant features that are meaningless for emotion recognition, including but not limited to the Uniform Resource Locator (URL), special symbols, and platform topic characters unrelated to the main content carried in the posted content. These redundant features need to be removed through data cleaning. The standardization processing of picture information can be understood as adjusting the picture size in social media data to a unified standard to facilitate subsequent unified recognition and processing by the model and improve the effectiveness of the data input into the model.

[0042] It should be noted that the original social media data can be obtained from, for example, the anxiety-related sections of a certain platform. However, the non-text part of the original social media data obtained in this way may still carry "anxiety"-related character content that is not user input (such as the topic section identifiers automatically generated by the platform). To a large extent, these characters may affect the language model's recognition and judgment of whether there is anxiety in social media data. Therefore, after obtaining social media data from the corresponding section, it is necessary to clean this type of information.

[0043] In addition, the method for identifying the anxiety tendency of social media data provided in the embodiments of the present application combines image information and text information for emotion recognition to determine whether the social media data has an anxiety tendency or contains anxiety emotions. Specifically, it identifies social media data that combines both image information and text information. For social media data that only contains image information or text information, existing trained image recognition models or text recognition models in the prior art can be used for recognition, which will not be elaborated in this application.

[0044] Step S102: Input the social media data to be recognized and auxiliary information into a trained anxiety tendency detection model for recognition.

[0045] Step S103: Obtain the recognition result of the anxiety tendency detection model for the social media data to be recognized, and the recognition result is used to indicate whether the social media data to be recognized has an anxiety tendency.

[0046] Exemplarily, the training method of the anxiety tendency detection model includes: inputting sample social media data and auxiliary information associated with the sample social media data into a neural network model to be trained, and obtaining the recognition result of the neural network model for the sample social media data; adjusting the parameters of the neural network model according to the recognition result and the annotation information for the sample social media data until the error between the recognition result and the annotation recognition result is less than a preset threshold, and using this neural network model as the anxiety tendency detection model, where the annotation information indicates that the sample social media data has an anxiety tendency or the sample social media data does not have an anxiety tendency.

[0047] Among them, the training samples for training the anxiety tendency detection model include sample social media data, auxiliary information associated with the sample social media data, and annotation information for the sample social media data. The sample social media data are all social media data that contain both picture information and text information. And the neural network model to be trained may include the network structure of the self-supervised language-image pre-training model (Bootstrapping Language-Image Pre-training, BLIP). The BLIP model demonstrates strong generalization ability and generation quality in vision-language tasks through innovative self-supervised learning strategies and architecture designs, and can generate natural language descriptions about the image content based on the image content. Input the sample social media data and the auxiliary information associated with the sample social media data into the neural network model to be trained, and obtain the recognition result of the neural network model for the sample social media data, which specifically includes the following steps:

[0048] (1) Use the BLIP model to generate picture description information for the sample picture information, and extract features from the picture description information to obtain picture description features;

[0049] (2) Extract features from the picture information and text information in the sample social media data to obtain picture visual features and text features;

[0050] (3) Calculate the correlation between the picture description features and the picture visual features based on the multi-head cross-attention mechanism to generate fused picture features;

[0051] (4) Calculate the correlation between the picture description features and the text features based on the multi-head cross-attention mechanism to generate fused text features;

[0052] (5) Extract features from the sample auxiliary information to obtain auxiliary features;

[0053] (6) Concatenate the auxiliary features with the fused picture features and the fused text features respectively to obtain enhanced picture features and enhanced text features;

[0054] (7) Fuse the enhanced sample picture features and the enhanced sample text features to obtain the final features;

[0055] (8) Output the anxiety tendency probability of the sample social media data predicted based on the final features through the fully connected layer and the Softmax function of the neural network model as the recognition result of the neural network model for the sample social media data.

[0056] Among them, when training the anxiety tendency detection model, the set X of sample social media data = {(u1, v1, w1), (u2, v2, w2), ···, (un, vn, wn)}, where ui represents text information, vi represents image information, wi represents the corresponding auxiliary information, n represents the number of sample social media data, and i = 1, 2, ···, n; for the annotation information Y of each sample social media data = {0, 1}, 0 and 1 respectively represent having an anxiety tendency and not having an anxiety tendency. And the neural network model will output the prediction result y for the sample social media data based on the input sample social media data pred = Softmax(Wx + b), where W and b are the weight matrix and bias respectively, and y pred represents the predicted label. The loss function of the model uses cross-entropy loss.

[0057] Preferably, after obtaining the prediction result y pred it is possible to calculate the loss value based on the binary cross-entropy loss function (BinaryCross-Entropy Loss), calculate the gradients of the weight matrix W and the bias b according to the loss function, and backpropagate the calculated gradients to each trainable network structure of the neural network model, so as to update the parameters of the model until the loss value according to the neural network model is less than a preset threshold (such as 0.05) to stop training.

[0058] For example, for a certain sample social media data, its corresponding annotation recognition result is 1, while the recognition result output by the model is 70%, and the loss can be calculated according to the difference between 70% and 100% and the parameters of the neural network model can be updated until the loss value calculated according to the recognition result output by the neural network model and the annotation recognition result is less than 0.05, then the trained anxiety tendency detection model can be obtained.

[0059] In the embodiments of the present application, it is possible to detect the anxiety tendency of social media data by combining information in multiple dimensions such as pictures, texts, posting time, and interaction situations, and can efficiently and accurately screen out valuable social media data for anxiety emotion research from a large amount of data on social media platforms, thereby improving the data effectiveness of anxiety emotion research.

[0060] Such as Figure 2 shown, on the basis of the embodiment shown in Figure 1 the step S102 may include the following steps:

[0061] Step S1021, generate picture description information of the picture information through a self-supervised language-image pre-training model, and use a text encoder to extract features from the picture description information to obtain a picture description feature vector.

[0062] In step S1022, an image encoder is used to extract features from the picture information to obtain a picture visual feature vector, and a text encoder is used to extract features from the text information to obtain a text feature vector.

[0063] In step S1023, feature fusion is performed on the picture description feature vector and the picture visual feature vector based on the multi-head cross-attention mechanism to obtain a fused picture feature vector; feature fusion is performed on the picture description feature vector and the text feature vector based on the multi-head cross-attention mechanism to obtain a fused text feature vector.

[0064] Exemplarily, the multi-head cross-attention mechanism (MCA) is based on the cross-attention mechanism and introduces a multi-head mechanism. It allows interaction between feature sequences from multiple different sources, can decompose the query, key, and value into multiple independent subspaces (heads) for parallel computing of attention, and finally splices the results for output.

[0065] As Figure 3 shown, the anxiety tendency detection model 300 can be divided into a picture description generation module 301, a feature extraction module 302, a feature fusion module 303, and a detection module 304 based on its functions. Among them, the anxiety tendency detection model 300 may include the network structure of the self-supervised language-image pre-training model (Bootstrapping Language-Image Pre-training, BLIP). The BLIP model is a pre-trained text-image cross-modal model, including an image encoder, a text encoder, a visual text encoder, and a visual text decoder.

[0066] Exemplarily, after the social media data to be recognized is input into the anxiety tendency detection model 300, the picture description generation module 301 can use the BLIP model to generate picture description information about the picture information. Specifically, the BLIP model adopts an architecture of a text-image joint encoder - text decoder. When an image is input, the BLIP model first uses the text-image joint encoder to extract visual features, and then the text decoder generates words one by one in an autoregressive manner until the end-of-generation marker is generated. During the generation process, the BLIP model continuously refers to the visual features and dynamically adjusts the generated language content to ensure that the description highly matches the picture content, and finally outputs a complete, semantically smooth, and accurate natural language text describing the picture content as the picture description information.

[0067] It should be noted that while the BLIP model initially generates descriptive content information for an image based on image information, a filtering module (CapFilt) can also be used to screen the quality of the generated content, removing duplicate, low-relevance, or grammatically incorrect descriptions and retaining semantically accurate and informative content as the final image description information.

[0068] In one possible implementation, if the image information to be identified in the social media data also contains some text content, a self-supervised language-image pre-training model can be used to first generate abstract semantic content of the image information based on the visual features of the image information, and computer vision detection technologies such as optical character recognition can be used to detect text areas in the image, identify characters and convert them into editable text, thereby extracting the original text content in the image information, and then combining the original text content and abstract semantic content in the image information to generate image description information corresponding to the image information.

[0069] Specifically, an example of a method for generating image description information corresponding to the image information by combining the original text content and abstract semantic content in the image information is as follows:

[0070] Method 1: By designing a keyword list (such as "anxiety," "insomnia," and "stress") or using sentiment analysis tools to filter the original text content in the image information, key content related to emotions and psychological states can be screened out. For example, if the image contains a diary text, through keyword matching, sentences that may reflect anxiety tendencies, such as "I have been inexplicably irritable recently and have not been able to sleep well at night," can be screened out. The filtered key content is then spliced and integrated with the abstract semantic content to generate image description information corresponding to the image information.

[0071] Method 2: Perform semantic analysis on the original text content and abstract semantic content in the image information, extract the semantic features of the two, fuse the features through an attention mechanism or weighted fusion, and then use the text generation model to regenerate a coherent and complete description of the fused semantic features as the image description information corresponding to the image information.

[0072] Method three is to construct a knowledge graph related to anxiety tendencies, map the abstract semantic content generated by the BLIP model and the original text content in the image information to the nodes and relationships of the knowledge graph, and use the reasoning ability of the knowledge graph to explore the potential semantic associations between the two. Combined with the prior knowledge in the graph, a more comprehensive and in-depth description is generated as the image description information corresponding to the image information.

[0073] After generating the image description information about the image information, the feature extraction module 302 can use a text encoder to extract features from the image description information and the text information, obtaining an image description feature vector and a text feature vector, and use an image encoder to extract the visual features in the image information to generate an image visual feature vector.

[0074] Preferably, when generating the image description feature vector based on the description content information generated by the BLIP model, the RoBERTa (Robustly Optimized BERT Pretraining Approach) encoder can be used to extract the feature representation Hm of the description content information m, where Hm ∈ R d×q is the obtained image description feature vector, d is the embedding dimension of the word, and q is the length of the description content.

[0075] In the process of generating the text feature vector based on the text information in the social media data to be recognized, the RoBERTa encoder can still be used to extract the feature representation Hu of the text information u, where Hu ∈ R d× p is the obtained text semantic vector, d is the embedding dimension of the word, and p is the length of the text.

[0076] When extracting the visual feature representation of the image information, a pre-trained ResNet (Residual Network) model can be used as the image encoder to extract the visual feature representation. First, the image is adjusted to a size of 224×224 pixels, and the pre-trained ResNet model is used to extract the visual feature representation of the image information v. Using an average pooling layer with a size of 7×7×2048, the image visual feature representation is obtained: Hv = ResNet(v), where Hv ∈ R 2048 represents the obtained image visual feature vector, and 2048 represents the visual feature dimension of each block. Finally, the generated feature Hv is mapped to a d-dimensional space through a fully connected layer, which can be expressed as: Hv = Wv × Hv, where Wv ∈ R d×2048 is a learnable parameter matrix, and Hv ∈ R d is the extracted visual feature representation.

[0077] After obtaining the picture visual feature vector Hv, the picture description feature vector Hm, and the text feature vector Hu, the feature fusion module 303 calculates the correlation between the picture visual feature vector Hv and the picture description feature vector Hm through a multi-head cross-attention mechanism. The feature fusion module 303 performs feature fusion based on the correlation to obtain a fused picture feature vector Hmv, and the feature fusion module 303 calculates the correlation between the picture description feature vector Hm and the text feature vector Hu through multi-head cross-attention. The feature fusion module 303 obtains a fused text feature vector Hmu based on the correlation.

[0078] Preferably, in the multi-head cross-attention mechanism of the embodiment of the present application, the optimal effect is achieved when the number of multi-heads Num_heads = 64 obtained through experiments, the model recognition effect is optimal when Batch_size is taken as 32, the model classification effect is optimal when the learning rate is taken as 0.001, and the model classification effect is optimal when Dropout is taken as 0.5.

[0079] Based on this, the embodiment of the present application can generate a description content about a picture based on the text content and visual features of the picture information in the social media data to be recognized. Thus, based on the multi-head cross-attention mechanism, the picture description content is fused with the picture visual features to obtain fused picture features, and the picture description content is fused with the text information in the social media data to be recognized to obtain fused text features, so as to extract the features in the social media data to be recognized more fully. From the perspective of text feature recognition, it can avoid the problem of inaccurate recognition caused by only recognizing emotions based on text content. From the perspective of picture feature recognition, it can also more fully mine image features, ensure that the picture description generated by the model based on the picture is as consistent as possible with the original text content in the picture, and avoid the problem of inaccurate image semantic generation, greatly improving the recognition accuracy of social media data with both picture information and text information.

[0080] Step S1024: Extract features from the auxiliary information to obtain an auxiliary feature vector of the social media data to be recognized.

[0081] Exemplarily, the feature extraction module 302 can also extract features from the auxiliary information w associated with the social media data to be recognized, generate a 4D vector w = [w1, w2, w3, w4] based on four dimensions of the release time, the number of likes, the number of comments, and the number of forwards, process the value of each dimension based on the min-max normalization method, and then use a fully connected layer to map the processed data to a hidden space to obtain a feature representation Hw corresponding to the auxiliary information w.

[0082] Among them, the calculation formula for normalization is

[0083] w i represents the original dimension value, min(wi ) and max(w i ) are respectively the minimum and maximum values of the corresponding auxiliary information of the sample social media data in the training model for this dimension. w i ′ is the normalized value. For example, if the range of the number of likes in the corresponding auxiliary information of the sample social media data is [0, 1000], and the number of likes in the auxiliary information of a social media data to be recognized is 500, then after normalization, w2′ = (500 - 0) / (1000 - 0) = 0.5. The normalized data value can make the gradient descent more stable and avoid neuron saturation or gradient disappearance.

[0084] Furthermore, the normalized 4D vector is input into the fully connected layer, and through linear transformation and activation function, it is mapped to the hidden space to obtain the feature representation Hw = Ww × w′ + bw of the auxiliary information, where w′ is the normalized 4D vector, Ww ∈ R d×4 is a learnable weight matrix, bw ∈ R d is a bias vector, and Hw ∈ R d is the final auxiliary feature vector, and d represents the dimension of the hidden space.

[0085] Step S1025, fuse the fused picture feature vector, the fused text feature vector and the auxiliary feature vector to obtain the final feature vector.

[0086] Among them, in the process of generating the final feature vector, the feature fusion module 303 first concatenates the auxiliary feature vector Hw with the fused picture feature vector Hmv and the fused text feature vector Hmu respectively to obtain the enhanced picture feature vector Hmvw and the enhanced text feature vector Hmuw. Then, the feature fusion module 303 calculates the correlation between the enhanced picture feature vector Hmvw and the enhanced text feature vector Hmuw again by using the attention mechanism to fuse and obtain the final feature vector.

[0087] Step S1026, determine the recognition result for the social media data to be recognized based on the final feature vector.

[0088] Here, the detection module 304 can output the recognition result for the social media data to be recognized based on the final feature vector, thus converting the detection of the anxiety tendency of the social media data to be recognized into a binary classification problem from X to Y, and determining whether the social media data to be recognized has an anxiety tendency according to the value of Y finally calculated from the input social media data to be recognized X.

[0089] Figure 4It is a schematic diagram of the processing flow of an anxiety tendency detection model for social media data to be recognized according to an embodiment of the present application. Assume that the social media data to be recognized input into the anxiety tendency detection model includes picture information, text information, and auxiliary information associated with the social media data to be recognized. The processing flow of the anxiety tendency detection model for the social media data to be recognized is as follows:

[0090] 1) Input the picture information into the BLIP model for processing. This model deeply fuses the visual features of the picture and the latent semantic information through cross-modal alignment technology, so as to generate picture description information by combining the visual features in the picture information and the text in the picture format.

[0091] 2) Extract features from the picture information, picture description information, text information, and auxiliary information respectively to obtain a picture visual feature vector, a picture description feature vector, a text feature vector, and an auxiliary feature vector.

[0092] Visual feature extraction: Use a deep convolutional neural network such as ResNet-50 for the original picture, and extract a 2048-dimensional visual feature vector through multiple convolutional and residual modules to capture low-level visual information such as edges and textures, and map it to the target dimension through global average pooling and a fully connected layer.

[0093] Semantic feature extraction: Input the picture description information and the text information into the RoBERTa pre-trained language model respectively, and use the bidirectional Transformer encoder to capture the semantic context and output the corresponding semantic feature vectors.

[0094] Auxiliary information processing: Construct the auxiliary information (such as the release time, the number of likes, the number of comments, the number of forwards) associated with the social media information to be recognized into a 4-dimensional vector. After eliminating the dimension difference through min-max normalization, it is mapped into an auxiliary feature vector with the same dimension as other modalities through a fully connected layer. Among them, the release time can reflect the time law of anxiety mood fluctuations, and the interaction data helps to judge the content influence.

[0095] 3) Calculate the correlation between the picture visual feature vector and the picture description feature vector based on the multi-head cross-attention mechanism to generate a fused picture feature vector, and calculate the correlation between the text feature vector and the picture description feature vector based on the multi-head cross-attention mechanism to generate a fused text feature vector.

[0096] This step mines the semantic associations between modalities from multiple subspace perspectives based on the multi-head cross-attention mechanism. When calculating the correlation between the visual and semantic features of the image, the visual feature vector is used as the query (Query), and the semantic feature vector is split into the key (Key) and value (Value). Through multi-head parallel calculation, the corresponding relationships between image details and text descriptions are captured respectively. Finally, the outputs of each head are concatenated and projected to generate a fused image feature vector. Similarly, for the semantic features of text and image, the MCA is used to strengthen the interpretability of text semantics for visual information and generate a fused text feature vector.

[0097] 4) Concatenate the auxiliary feature vector with the fused image feature vector and the fused text feature vector respectively to obtain an enhanced image feature vector and an enhanced text feature vector.

[0098] This step concatenates the auxiliary feature vector with the fused image and fused text feature vectors along the feature dimension respectively to construct an enhanced feature representation. This process injects external scene information into the multi-modal features. For example, by combining the late-night posting time with negative semantic features, the basis for judging the anxiety tendency is strengthened; or the degree of resonance of emotional expression is verified by the high number of likes. The enhanced image feature vector and the enhanced text feature vector after concatenation not only retain the original modal interaction information but also integrate the data dissemination attributes and time features.

[0099] 5) Calculate the correlation between the enhanced image feature vector and the enhanced text feature vector based on the attention mechanism, and fuse the two to generate a final feature vector.

[0100] This step can use the self-attention mechanism to globally interact the enhanced image and enhanced text feature vectors. By calculating the attention weights of the two, the importance of different modal information is dynamically adjusted. For example, when anxiety keywords appear explicitly in the text, the model automatically increases the weight of the text features; if the visual features of the image contain strong emotional cues (such as crying, enclosed space), the influence of the visual features is enhanced. Finally, the weighted feature vectors are fused to generate a final feature vector containing all-modal information, comprehensively representing the anxiety-related clues in the data.

[0101] 6) Use the softmax function to predict the probability that the social media data to be recognized has an anxiety tendency based on the final feature vector and output the recognition result for the social media data to be recognized.

[0102] This step inputs the final feature vector into the Softmax function, converts it into a probability distribution in the range of 0-1 through exponential normalization operation, and outputs the predicted probabilities that the social media data to be recognized belongs to "anxiety tendency" and "non-anxiety tendency". For example, if the output probability is 0.85, it is determined that the probability that this piece of data has an anxiety tendency is 85%. The model finally outputs the recognition result in the form of a label, and can provide a confidence reference in combination with the probability value, providing a quantitative basis for subsequent data analysis of anxiety emotions.

[0103] The above describes the specific settings and implementation manners of the embodiments of the present application from different perspectives. Using the method provided by the above embodiments, it is possible to model by combining features in multiple dimensions such as pictures, texts, posting times, and interaction situations in social media data, and a multi-modal anxiety tendency recognition model is constructed. The model structure is simple and the training cost is low. It can fully integrate the data features of each modality based on the attention mechanism, improve the recognition rate of social media data containing anxiety emotions or having an anxiety tendency, and can efficiently and accurately screen out valuable social media data for anxiety emotion research from a large amount of data on the social media platform, thereby improving the data effectiveness of anxiety emotion research.

[0104] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0105] Exemplary device

[0106] As an implementation of the above various methods, as Figure 5 shown, the embodiments of the present application further provide an anxiety tendency recognition device for social media data. The device may include:

[0107] A data acquisition module 501, configured to acquire social media data to be recognized and auxiliary information associated with the social media data to be recognized. The social media data to be recognized includes picture information and text information, and the auxiliary information includes the posting time and interaction information of the social media data to be recognized;

[0108] A recognition module 502, configured to input the social media data to be recognized and the auxiliary information into a trained anxiety tendency detection model for recognition;

[0109] The recognition module is further configured to obtain the recognition result of the anxiety tendency detection model for the social media data to be recognized, and the recognition result is used to indicate whether the social media data to be recognized has an anxiety tendency.

[0110] In some embodiments, the recognition module 502 is further configured to: generate picture description information of the picture information through a self-supervised language-image pre-training model; extract feature vectors of the picture description information by using a text encoder to obtain picture description feature vectors; extract feature vectors of the picture information by using an image encoder to obtain picture visual feature vectors; extract feature vectors of the text information by using a text encoder to obtain text feature vectors; perform feature fusion on the picture description feature vectors and the picture visual feature vectors based on a multi-head cross-attention mechanism to obtain fused picture feature vectors; perform feature fusion on the picture description feature vectors and the text feature vectors based on a multi-head cross-attention mechanism to obtain fused text feature vectors; extract feature vectors of the auxiliary information to obtain auxiliary feature vectors of the social media data to be recognized; perform feature fusion on the fused picture feature vectors, the fused text feature vectors, and the auxiliary feature vectors to obtain final feature vectors; and determine the recognition result for the social media data to be recognized based on the final feature vectors.

[0111] In some embodiments, the recognition module 502 is further configured to: generate abstract semantic content of the picture information based on a self-supervised language-image pre-training model; extract the original text content in the picture information by using character recognition technology; and generate picture description information of the picture information according to the abstract semantic content and the original text content in the picture information.

[0112] In some embodiments, the recognition module 502 is further configured to: splice the auxiliary feature vectors with the fused picture feature vectors and the fused text feature vectors respectively to obtain enhanced picture feature vectors and enhanced text feature vectors; and perform feature fusion on the enhanced picture feature vectors and the enhanced text feature vectors based on a cross-attention mechanism to obtain final feature vectors.

[0113] In some embodiments, the data acquisition module 501 is further configured to: screen the original social media data and retain the social media data that contains both picture information and text information; and perform data preprocessing on the screened social media data to obtain the social media data to be recognized, where the data preprocessing includes data cleaning of the text information and standardization processing of the picture information.

[0114] In some embodiments, the training method of the anxiety tendency detection model in the recognition module 502 includes: inputting sample social media data and auxiliary information associated with the sample social media data into a neural network model to be trained, and obtaining the recognition result of the neural network model for the sample social media data; adjusting the parameters of the neural network model according to the recognition result and the annotation information for the sample social media data until the error between the recognition result and the annotation information is less than a preset threshold, and using the neural network model as the anxiety tendency detection model, where the annotation information indicates that the sample social media data has an anxiety tendency or the sample social media data does not have an anxiety tendency.

[0115] For the functions of each unit, module or sub-module in each device of the embodiments of the present application, reference may be made to the corresponding descriptions in the above method embodiments, and the corresponding beneficial effects are achieved, which will not be elaborated herein.

[0116] Exemplary electronic device and computer-readable storage medium

[0117] Figure 6 It is a block diagram of an electronic device for implementing the embodiments of the present application. As Figure 6 shown, the electronic device includes: a memory 601 and a processor 602, and a computer program that can run on the processor 602 is stored in the memory 601. When the processor 602 executes the computer program, the method in the above embodiments is implemented. The number of the memory 601 and the processor 602 can be one or more.

[0118] The electronic device further includes:

[0119] a communication interface 603, configured to communicate with external devices and perform data interaction and transmission.

[0120] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the memory 601, the processor 602 and the communication interface 603 can be interconnected through a bus and complete communication with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity of representation, Figure 6 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0121] Optionally, in a specific implementation, if the memory 601, the processor 602, and the communication interface 603 are integrated on a single chip, the memory 601, the processor 602, and the communication interface 603 can communicate with each other through an internal interface.

[0122] An embodiment of this application also provides a chip, which includes a processor for calling and running instructions stored in a memory, so that a communication device installed with the chip executes the method provided by the embodiment of this application.

[0123] An embodiment of this application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, the output interface, the processor, and the memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided by the embodiment of this application.

[0124] It should be understood that the above-mentioned processor may be a CPU, or other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the advanced reduced instruction set machine (ARM) architecture.

[0125] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), sync link dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM).

[0126] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0127] In addition to the above methods and devices, the embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which when run by a processor cause the processor to execute the steps in the image processing methods of various embodiments of the present disclosure described in the above "Exemplary Method" section.

[0128] A computer program product may be written in any combination of one or more programming languages for executing the program code of the operations of the embodiments of the present disclosure. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on a user computing device, partially on the user device, executed as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0129] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0130] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present application, "a plurality of" means two or more, unless otherwise specifically defined.

[0131] Any process or method described in the flowchart or otherwise described herein may be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a specific logical function or process. And the scope of the preferred embodiments of the present application includes additional implementations, where the functions may be executed in a substantially simultaneous manner or in a reverse order according to the functions involved, rather than in the order shown or discussed.

[0132] The logic and / or steps described in the flowchart or otherwise described herein, for example, may be considered as a sequenced list of executable instructions for implementing a logical function, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch instructions from the instruction execution system, apparatus, or device and execute the instructions), or in combination with these instruction execution systems, apparatus, or devices.

[0133] It should be understood that each part of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the method in the above embodiments can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0134] In addition, in each embodiment of the present application, each functional unit can be integrated in a processing module, or each unit can exist physically alone, or two or more units can be integrated in a module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk, an optical disc, etc.

[0135] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope recorded in the present application can easily think of various changes or substitutions, and these should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for identifying anxiety tendencies in social media data, characterized in that, Including: Obtain the social media data to be recognized and auxiliary information associated with the social media data to be recognized, where the social media data to be recognized includes picture information and text information, and the auxiliary information includes the release time and interaction information of the social media data to be recognized; Input the social media data to be recognized and the auxiliary information into a trained anxiety tendency detection model for recognition; Obtain the recognition result of the anxiety tendency detection model for the social media data to be recognized, where the recognition result is used to indicate whether the social media data to be recognized has an anxiety tendency; Among them, inputting the social media data to be recognized and the auxiliary information into a trained anxiety tendency detection model for recognition includes: Generate picture description information of the picture information through a self-supervised language-image pre-training model; Use a text encoder to extract features from the picture description information to obtain a picture description feature vector; Use an image encoder to extract features from the picture information to obtain a picture visual feature vector; Use a text encoder to extract features from the text information to obtain a text feature vector; Based on a multi-head cross-attention mechanism, perform feature fusion on the picture description feature vector and the picture visual feature vector to obtain a fused picture feature vector; Based on a multi-head cross-attention mechanism, perform feature fusion on the picture description feature vector and the text feature vector to obtain a fused text feature vector; Extract features from the auxiliary information to obtain an auxiliary feature vector of the social media data to be recognized; Perform feature fusion on the fused picture feature vector, the fused text feature vector, and the auxiliary feature vector to obtain a final feature vector; Determine the recognition result for the social media data to be recognized based on the final feature vector.

2. The method according to claim 1, wherein Generating picture description information of the picture information through a self-supervised language-image pre-training model includes: Generate the abstract semantic content of the picture information based on the self-supervised language-image pre-training model; Use character recognition technology to extract the original text content in the picture information; Generate the picture description information of the picture information according to the abstract semantic content and the original text content in the picture information.

3. The method according to claim 1, characterized in that, Performing feature fusion on the fused picture feature vector, the fused text feature vector, and the auxiliary feature vector to obtain a final feature vector includes: Concatenate the auxiliary feature vector with the fused picture feature vector and the fused text feature vector respectively to obtain an enhanced picture feature vector and an enhanced text feature vector; Based on a cross-attention mechanism, perform feature fusion on the enhanced picture feature vector and the enhanced text feature vector to obtain the final feature vector.

4. The method according to claim 1, wherein Obtaining the social media data to be recognized includes: Screen the original social media data and retain the social media data that contains both picture information and text information; Perform data preprocessing on the screened social media data to obtain the social media data to be recognized, where the data preprocessing includes data cleaning of the text information and normalization processing of the picture information.

5. The method according to claim 1, characterized in that The training method of the anxiety tendency detection model includes: Inputting sample social media data and auxiliary information associated with the sample social media data into a neural network model to be trained, and obtaining the recognition result of the neural network model for the sample social media data; Adjusting the parameters of the neural network model according to the recognition result and the annotation information for the sample social media data until the error between the recognition result and the annotation information is less than a preset threshold, and using this neural network model as the anxiety tendency detection model, where the annotation information indicates that the sample social media data has an anxiety tendency or the sample social media data does not have an anxiety tendency.

6. An anxiety tendency recognition device for social media data, characterized in that The device includes: A data acquisition module, configured to acquire social media data to be recognized and auxiliary information associated with the social media data to be recognized. The social media data to be recognized includes picture information and text information, and the auxiliary information includes the release time and interaction information of the social media data to be recognized; A recognition module, configured to input the social media data to be recognized and the auxiliary information into a trained anxiety tendency detection model for recognition; The recognition module is further configured to obtain the recognition result of the anxiety tendency detection model for the social media data to be recognized, and the recognition result is used to indicate whether the social media data to be recognized has an anxiety tendency; The recognition module is further configured to: generate picture description information of the picture information through a self-supervised language-image pre-training model; extract features from the picture description information using a text encoder to obtain a picture description feature vector; extract features from the picture information using an image encoder to obtain a picture visual feature vector; extract features from the text information using a text encoder to obtain a text feature vector; Based on a multi-head cross-attention mechanism, perform feature fusion on the picture description feature vector and the picture visual feature vector to obtain a fused picture feature vector; based on a multi-head cross-attention mechanism, perform feature fusion on the picture description feature vector and the text feature vector to obtain a fused text feature vector; extract features from the auxiliary information to obtain an auxiliary feature vector of the social media data to be recognized; perform feature fusion on the fused picture feature vector, the fused text feature vector, and the auxiliary feature vector to obtain a final feature vector; Determine the recognition result for the social media data to be recognized based on the final feature vector.

7. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory. When the processor executes the computer program, it implements the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the method according to any one of claims 1-5.

Images