Social robot detection method and system based on modal specific expert multi-gating mixing

Abstract

The invention relates to a social robot detection method and system based on modal specific expert multi-gating mixing, and belongs to the technical field of natural language processing. According to the method, three account features are fused, and a multi-gating hybrid expert mechanism is adopted for social robot detection. Firstly, metadata, relational graph data and tweet data of a social account are coded, and then coding information is sent to a multi-gating hybrid expert layer for learning in order to better cope with the community difference problem. Thirdly, after three user features (namely a relation feature, a tweet feature and an account feature) processed by the multi-gating hybrid expert layer are obtained, the three user features are sent to the multi-head attention layer, and the features are combined for account detection. According to the method, good accuracy is obtained in the field of social robots, and good performance is reflected in the aspects of generalization and robustness.

Description

Technical Field

[0001] The present invention relates to a social robot detection method and system based on modality-specific expert multi-gating hybrid, and belongs to the technical field of natural language processing. Background Art

[0002] Nowadays, social media platforms are expanding rapidly in terms of the number of users, data size, and applications. They are basically Internet-based applications that facilitate the exchange of user-generated content. The high usage of these platforms has created an all-time high in human communication. Social bots are automated programs used to participate in social media. These bots act in a partially or fully autonomous manner and are usually designed to mimic human behavior to conduct activities on social media. Although there are benign social media bots, many social media bots are still used to engage in fraudulent and nefarious behavior. Today's advanced social bots have learned to imitate the characteristics of real users to disguise themselves, and the distribution of social bots throughout the social media network is heterogeneous, and the behavior and characteristics of the bots vary greatly between different user groups, which makes the detection task difficult.

[0003] Most of the existing detection methods are based on single features of text or graphics for detection, and are detected by extracting text features and social relationship features of social media accounts. Early social robots had only very simple attributes, and they had almost no personal information and social relationships, so such robots were very easy to distinguish. Later, in order to increase their credibility, social robots added very rich personal account information, and they began to follow each other to form their own relationship network, which greatly increased the difficulty of detection. Today's social robots are carefully designed and are more like manually manipulated accounts. They have a large number of real friends and followers, and they will use stolen names and avatars and steal tweets from real users to intersperse their malicious information, which makes text-based detection less easy. This leads to certain problems in the detection method: currently advanced robots will disguise themselves as humans from multiple angles, such as using account information to steal real users' information, posting content copied from real users' tweets, and creating botnet relationships for themselves, so these multiple angles of disguise make it impossible for the model to use only a single feature for identification. Later, some researchers proposed to use a mixed feature method for detection. However, faced with the problem of community differences among users, the existing method only uses a single modality-specific expert fusion to model different combined communities, which has the problem of low fault tolerance and limits the accuracy, robustness and generalization of the model.

[0004] Therefore, the present invention provides a social robot detection method based on modality-specific expert multi-gated hybrid. Summary of the invention

[0005] The present invention provides a social robot detection method and system based on modality-specific expert multi-gated hybrid, which is used to solve the problems of low fault tolerance of social robot detection models and poor accuracy, robustness and generalization of the model.

[0006] The technical solution of the present invention is: in the first aspect, the present invention provides a social robot detection method based on modality-specific expert multi-gated hybrid, and the specific steps of the social robot detection method based on modality-specific expert multi-gated hybrid are as follows:

[0007] Step 1: In the feature coding layer, encode the account metadata, relationship graph data, and tweet data respectively;

[0008] Step 2: In the multi-gated hybrid expert layer, a multi-gated hybrid expert mechanism is used to learn the differences between different accounts in different communities;

[0009] Step 3: In the feature fusion layer, the three major user features processed by the multi-gated hybrid expert layer are sent to the multi-head attention layer to fuse these features; the three major user features include relationship features, tweet features, and account features;

[0010] Step 4: After the feature fusion layer, the features of the three modes of the user are obtained respectively. Then the fused features are concatenated with the features of the three modes, and linear transformation is applied to obtain the final detection result of the account.

[0011] As a further solution of the present invention, the specific steps of Step 1 are:

[0012] The specific steps of Step 1 are:

[0013] Step 1.1. Encode the account tweet data: Use the pre-trained language models RoBERTa and T5 to encode the tweet text content posted on the social media account; use the pre-trained RoBERTa and T5 as the encoder of the text module, defined as:

[0014]

[0015] Where LM represents the pre-trained language model encoder, t n i represents the nth tweet of user i, and then feeds the obtained account tweet encoding into a two-layer multi-layer perceptron to obtain a low-dimensional user text representation

[0016] Step 1.2, Encode the social account relationship graph data: There are many relationships between users on social media, such as (follow, fans, friends), etc. The present invention constructs a heterogeneous information network to obtain the relationship between users. The initial features of the user are composed of digital metadata, descriptions, and tweets published to obtain comprehensive user features. Then, graph neural networks (GNNs) are used to encode the user's graph modal information, and the message passing mechanism is used to model the interaction between users. Their message passing paradigm can be defined as:

[0017]

[0018] in, represents the hidden representation of user i in the kth layer, N r (i) represents the neighborhood of user i, represents the messages transmitted from the neighborhood of user i under relation r, AGGREGATE r and UPDATE represent the aggregation function and update function under the relation r, and RGCN is used as the aggregation and update function. Finally, after passing through the l-layer GNN encoder, we obtain the node features And represent them as the output of the graph encoder

[0019] Step 1.3, Encode account metadata: For the metadata in social user profiles, we will use a two-layer multi-layer perceptron to learn 5 numerical features (number of followers, number of followings, number of tweets, number of active days, screen name length) and 3 categorical metadata features (whether to protect the account, whether it has been verified, whether to use the default profile picture).

[0020] As a further solution of the present invention, the specific steps of Step 2 are:

[0021] Step 2.1. Accounts from different user communities have different characteristics, which poses a challenge to existing social robot detection systems. To this end, the present invention introduces a multi-gated hybrid expert mechanism. The multi-gated network can learn different Experts weights, thereby realizing the selective use of Experts. The gated networks corresponding to different community groups can learn different Experts combination modes. Specifically, user representations from metadata, text, and graphic modalities are first fed to different gate networks, one of which is a gate network G. h It can be expressed as:

[0022] G h (x) = softmax(KeepTopK(W g x,k))

[0023] Here, W g ∈R d×n represents a learnable parameter, n represents the number of experts, and KeepTopK represents a function that selects the top k highest gating values ​​based on the input feature x. The output generated by the gating network G assigns a user to k different communities and gives the corresponding assignment probabilities.

[0024] Step 2.2: Based on step 2.1, the user representation obtained from metadata, text and graph modalities is fed into its assigned expert network for community-aware robot detection:

[0025]

[0026] In the formula represents the probability that the h-th gating network assigns user i to the j-th community, represents the output of the j-th expert network, as a two-layer multi-layer perceptron, and mod∈{m,t,g} represents the hybrid expert module of metadata, text data and graph data respectively. is the weighted sum of the outputs of all experts selected by the h-th gating network.

[0027] Step 2.3: Based on step 2.3, merge all gated network outputs to get the final output It can be expressed as:

[0028]

[0029] Through a multi-gated hybrid expert mechanism, each user is assigned to its corresponding community for processing, so that the model can be adjusted for different user distributions in different communities, thereby solving the challenge of diverse user communities in Twitter robot detection.

[0030] As a further solution of the present invention, the specific steps of Step 3 are:

[0031] Step 3.1, after obtaining the three major user features processed by the multi-gated hybrid expert layer (respectively, relationship features, tweet features, and account features) and then we will send them into the multi-head transformer to combine these features:

[0032]

[0033] As a further solution of the present invention, the specific steps of Step 4 are:

[0034] Step 4.1. After the feature fusion layer, the present invention obtains the features of the three modes of the user. Then the fused features are spliced ​​with the three modal features, and a linear transformation is applied to obtain the final detection result y of the present invention on the account i :

[0035]

[0036] Where W o and b o is a learnable parameter and [,] is a connection operation.

[0037] Step 4.2: In order to prevent the model from overfitting, the cross entropy with L2 regularization term is used in the training of the present invention. In order to avoid the situation where some experts are too dominant and other experts are ignored, we introduce the balanced loss function BL(·). The final overall loss function can be expressed as:

[0038]

[0039] where y i is the predicted output of the present invention for the i-th user, t i is the true label corresponding to the i-th user, θ represents all trainable model parameters, and λ1 and λ2 are hyperparameters. For the balanced loss BL(·), it can be expressed as:

[0040] BL(x)=w imp CV(G(x)) 2 +w ld CV(P(x,i)) 2

[0041] Where CV represents the coefficient of variation, G(x) represents the output of the gating network, P(x,i) is the smoothing function defined by Shazeer et al., and w imp and w ld A hyperparameter to balance the importance and load of experts.

[0042] In a second aspect, the present invention provides a social robot detection system based on modality-specific expert multi-gated hybrid, including a module for executing the method described in the first aspect above.

[0043] The beneficial effects of the present invention are:

[0044] The present invention first encodes the metadata, tweet data, and relationship data of social accounts respectively, so that the model can obtain user information from multiple angles. Then, a multi-gated expert hybrid mechanism is introduced, so that the model can obtain unique features from users of different community groups, and improve the model's good performance when facing community group differences. Finally, the three user representations are fused and tested. This solves the problem of low fault tolerance of the social robot detection model, and the poor accuracy, robustness, and generalization of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is the overall flow chart of the present invention;

[0046] Figure 2 It is a multi-gated mixed expert layer diagram in the model of the present invention. DETAILED DESCRIPTION

[0047] Example 1: Figure 1-2 As shown in the figure, the social robot detection method based on modality-specific expert multi-gating hybrid has the following specific steps:

[0048] Step 1: In the feature coding layer, encode the account metadata, relationship graph data, and tweet data respectively;

[0049] Step 2: In the multi-gated hybrid expert layer, a multi-gated hybrid expert mechanism is used to learn the differences between different accounts in different communities;

[0050] Step 3: In the feature fusion layer, the three major user features processed by the multi-gated hybrid expert layer are sent to the multi-head attention layer to fuse these features; the three major user features include relationship features, tweet features, and account features;

[0051] Step 4: After the feature fusion layer, the features of the three modes of the user are obtained respectively. Then the fused features are concatenated with the features of the three modes, and linear transformation is applied to obtain the final detection result of the account.

[0052] As a further solution of the present invention, the specific steps of Step 1 are:

[0053] The specific steps of Step 1 are:

[0054] Step 1.1. Encode the account tweet data: Use the pre-trained language models RoBERTa and T5 to encode the tweet text content posted on the social media account; use the pre-trained RoBERTa and T5 as the encoder of the text module, defined as:

[0055]

[0056] Where LM represents the pre-trained language model encoder, t n i represents the nth tweet of user i, and then feeds the obtained account tweet encoding into a two-layer multi-layer perceptron to obtain a low-dimensional user text representation

[0057] Step 1.2, Encode the social account relationship graph data: There are many relationships between users on social media, such as (follow, fans, friends), etc. The present invention constructs a heterogeneous information network to obtain the relationship between users. The initial features of the user are composed of digital metadata, descriptions, and tweets published to obtain comprehensive user features. Then, graph neural networks (GNNs) are used to encode the user's graph modal information, and the message passing mechanism is used to model the interaction between users. Their message passing paradigm can be defined as:

[0058]

[0059] in, represents the hidden representation of user i in the kth layer, N r (i) represents the neighborhood of user i, represents the messages transmitted from the neighborhood of user i under relation r, AGGREGATE r and UPDATE represent the aggregation function and update function under the relation r, and RGCN is used as the aggregation and update function. Finally, after passing through the l-layer GNN encoder, we obtain the node features And represent them as the output of the graph encoder

[0060] Step 1.3, Encode account metadata: For the metadata in social user profiles, we will use a two-layer multi-layer perceptron to learn 5 numerical features (number of followers, number of followings, number of tweets, number of active days, screen name length) and 3 categorical metadata features (whether to protect the account, whether it has been verified, whether to use the default profile picture).

[0061] As a further solution of the present invention, the specific steps of Step 2 are:

[0062] Step 2.1. Accounts from different user communities have different characteristics, which poses a challenge to existing social robot detection systems. To this end, the present invention introduces a multi-gated hybrid expert mechanism. The multi-gated network can learn different Experts weights, thereby realizing the selective use of Experts. The gated networks corresponding to different community groups can learn different Experts combination modes. Specifically, user representations from metadata, text, and graphic modalities are first fed to different gate networks, one of which is a gate network G. h It can be expressed as:

[0063] G h (x) = softmax(KeepTopK(W g x,k))

[0064] Here, W g ∈R d×n represents a learnable parameter, n represents the number of experts, and KeepTopK represents a function that selects the top k highest gating values ​​based on the input feature x. The output generated by the gating network G assigns a user to k different communities and gives the corresponding assignment probabilities.

[0065] Step 2.2: Based on step 2.1, the user representation obtained from metadata, text and graph modalities is fed into its assigned expert network for community-aware robot detection:

[0066]

[0067] In the formula represents the probability that the h-th gating network assigns user i to the j-th community, represents the output of the j-th expert network, as a two-layer multi-layer perceptron, and mod∈{m,t,g} represents the hybrid expert module of metadata, text data and graph data respectively. is the weighted sum of the outputs of all experts selected by the h-th gating network.

[0068] Step 2.3: Based on step 2.3, merge all gated network outputs to get the final output It can be expressed as:

[0069]

[0070] Through a multi-gated hybrid expert mechanism, each user is assigned to its corresponding community for processing, so that the model can be adjusted for different user distributions in different communities, thereby solving the challenge of diverse user communities in Twitter robot detection.

[0071] As a further solution of the present invention, the specific steps of Step 3 are:

[0072] Step 3.1, after obtaining the three major user features processed by the multi-gated hybrid expert layer (respectively, relationship features, tweet features, and account features) and then we will send them into the multi-head transformer to combine these features:

[0073]

[0074] As a further solution of the present invention, the specific steps of Step 4 are:

[0075] Step 4.1. After the feature fusion layer, the present invention obtains the features of the three modes of the user. Then the fused features are spliced ​​with the three modal features, and a linear transformation is applied to obtain the final detection result y of the present invention on the account i :

[0076]

[0077] Where W o and b o is a learnable parameter and [,] is a connection operation.

[0078] Step 4.2: In order to prevent the model from overfitting, the cross entropy with L2 regularization term is used in the training of the present invention. In order to avoid the situation where some experts are too dominant and other experts are ignored, we introduce the balanced loss function BL(·). The final overall loss function can be expressed as:

[0079]

[0080] where y i is the predicted output of the present invention for the i-th user, t i is the true label corresponding to the i-th user, θ represents all trainable model parameters, and λ1 and λ2 are hyperparameters. For the balanced loss BL(·), it can be expressed as:

[0081] BL(x)=w imp CV(G(x)) 2 +w ld CV(P(x,i)) 2

[0082] Where CV represents the coefficient of variation, G(x) represents the output of the gating network, P(x,i) is the smoothing function defined by Shazeer et al., and w imp and w ld A hyperparameter to balance the importance and load of experts.

[0083] In a second aspect, the present invention provides a social robot detection system based on modality-specific expert multi-gating hybrid, comprising:

[0084] The encoding module is used to encode account metadata, relationship graph data, and tweet data respectively;

[0085] The multi-gated mixed expert module is used to learn the differences between different accounts in different communities using a multi-gated mixed expert mechanism;

[0086] The feature fusion module is used to send the three major user features processed by the multi-gated hybrid expert layer to the multi-head attention layer to fuse these features; the three major user features include relationship features, tweet features, and account features;

[0087] The acquisition module is used to obtain the features of the three modes of the user after the feature fusion layer, then concatenate the fused features with the three modal features, and apply linear transformation to obtain the final detection result of the account.

[0088] In order to illustrate the detection effect of the present invention, a baseline system is used to compare with the present invention on the public datasets Cresci-15, TwiBot-20 and TwiBot-22. Table 1 shows the performance comparison experimental results with the baseline model on the three datasets.

[0089] Table 1 Comparative experiment

[0090]

[0091] As can be seen from Table 1, first of all, among the selected baseline models, the models that use graph processing technology in the detection method, such as RGT, BotRGCN and BotMoE, are generally more accurate in detection than models that do not use graph networks. This once again confirms the importance of mining and utilizing the relationship between users. Secondly, the data sets from creci-15 to TwiBot-20 to TwiBot22 were proposed in 2015, 2020, and 2022 respectively. It can be seen from the experimental data that each baseline model shows a step-by-step decline on the more newly proposed data sets, which shows that social robots have been updating and iterating, becoming more and more difficult to identify, so the social robot detection method also needs to adapt to this change. Finally, BotBuster and BotMoE both use hybrid expert technology, but the method of the present invention is better than these two models because the model of the present invention introduces a multi-gating mechanism on the original expert model, so that the present invention can learn different Experts weight combinations for multiple gating networks for different community groups, and better adapt to the increasingly diverse community groups.

[0092] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A social robot detection method based on modality-specific expert multi-gating hybrid, characterized by: Specifically include: Step 1: In the feature coding layer, encode the account metadata, relationship graph data, and tweet data respectively; Step 2: In the multi-gated hybrid expert layer, a multi-gated hybrid expert mechanism is used to learn the differences between different accounts in different communities; Step 3: In the feature fusion layer, the three major user features processed by the multi-gated hybrid expert layer are sent to the multi-head attention layer to fuse these features; the three major user features include relationship features, tweet features, and account features; Step 4: After the feature fusion layer, the features of the three modes of the user are obtained respectively. Then the fused features are concatenated with the features of the three modes, and linear transformation is applied to obtain the final detection result of the account.

2. The social robot detection method based on modality-specific expert multi-gating hybrid according to claim 1 is characterized in that: The specific steps of Step 1 are: Step 1.

1. Encode the account tweet data: Use the pre-trained language models RoBERTa and T5 to encode the tweet text content posted on the social media account; use the pre-trained RoBERTa and T5 as the encoder of the text module, defined as: Where LM represents the pre-trained language model encoder, t n i represents the nth tweet of user i, and then feeds the obtained account tweet encoding into a two-layer multi-layer perceptron to obtain a low-dimensional user text representation Step 1.2: Encode the social account relationship graph data: There are multiple relationships between users on social media, so a heterogeneous information network is constructed to obtain the relationships between users. The initial features of users are composed of digital metadata, descriptions, and published tweets to obtain comprehensive user features. Then, a graph neural network is used to encode the user's graph modal information, and a message passing mechanism is used to model the interaction between users. Their message passing paradigm is defined as: in, represents the hidden representation of user i in the kth layer, N r (i) represents the neighborhood of user i, represents the messages transmitted from the neighborhood of user i under relation r, AGGREGATE r and UPDATE represent the aggregation function and update function under the relation r, and RGCN is used as the aggregation and update function; finally, after passing through the l-layer GNN encoder, the node features are obtained And represent them as the output of the graph encoder Step 1.3: Encode account metadata: For the metadata in the social user profile, a two-layer multi-layer perceptron is used to learn the numerical features and categorical metadata features, and finally the metadata encoding is obtained as 3. The social robot detection method based on modality-specific expert multi-gating hybrid according to claim 1 is characterized in that: The specific steps of Step 2 are: Step 2.1, introduce a multi-gated hybrid expert mechanism. The multi-gated network can learn different Experts weights, thereby realizing the selective use of Experts. The gated networks corresponding to different community groups can learn different Experts combination modes. User representations from metadata, text, and graphic modalities are first fed to different gate networks, one of which is a gate network G. h It is expressed as: G h (x)=softmax(KeepTopK(W g x,k)) Here, W g ∈R d×n represents a learnable parameter, n represents the number of experts, and KeepTopK represents a function that selects the top k highest gating values ​​according to the input feature x. The output generated by the gating network G assigns a user to k different communities and gives the corresponding assignment probabilities. Step 2.2: Based on step 2.1, the user representation obtained from metadata, text and graph modalities is fed into its assigned expert network for community-aware robot detection: In the formula represents the probability that the h-th gating network assigns user i to the j-th community, represents the output of the j-th expert network, as a two-layer multilayer perceptron, mod∈{m,t,g} represents the hybrid expert module of metadata, text data and graph data respectively; is the weighted sum of the outputs of all experts selected by the h-th gating network; Step 2.3: Based on step 2.3, merge all gated network outputs to get the final output It is expressed as: Through a multi-gated hybrid expert mechanism, each user is assigned to its corresponding community for processing.

4. The social robot detection method based on modality-specific expert multi-gating hybrid according to claim 3 is characterized by: The specific steps of Step 3 are: Step 3.1, after obtaining the three major user features processed by the multi-gated hybrid expert layer After that, we feed the features into the multi-head transformer to combine them:

5. The social robot detection method based on modality-specific expert multi-gating hybrid according to claim 1 is characterized in that: The specific steps of Step 4 are: Step 4.

1. After the feature fusion layer, the features of the three modes of the user are obtained respectively. Then the fused features are concatenated with the three modal features, and a linear transformation is applied to obtain the final detection result y for the account. i : Where W o and b o is a learnable parameter and [,] is a connection operation; Step 4.2, using the cross entropy with L2 regularization term, the balanced loss function BL(·) is introduced, and the final overall loss function is expressed as: where y i is the predicted output for the i-th user, t i is the true label corresponding to the i-th user, θ represents all trainable model parameters, λ1 and λ2 are hyperparameters; for the balance loss BL(·), it is expressed as: BL(x)=w imp ·CV(G(x)) 2 +w ld ·CV(P(x,i)) 2 Where CV represents the coefficient of variation, G(x) represents the output of the gating network, P(x,i) is the smoothing function defined by Shazeer et al., and w imp and w ld A hyperparameter to balance the importance and load of experts.

6. A social robot detection system based on modality-specific expert multi-gating hybrid, characterized by: The method comprises a module for executing any of the methods according to claims 1-5.

Images