Social robot detection method and system, computer equipment and storage medium

Abstract

The invention provides a social robot detection method and system, computer equipment and a storage medium, and belongs to the technical field of social robot detection.The method comprises the steps that user multi-source data on a social platform and the social relation between users are collected; a hybrid encoder is adopted to capture local dependence and global time sequence dynamic states of behaviors, and user behavior characteristics are generated; aggregating structure attention features between the initial node features of the target account and the initial node features of the social relation account to obtain multi-modal relation aggregated structure features; introducing a multi-modal adversarial training strategy, adaptively adjusting the disturbance intensity of each modal based on gradient sensitivity, and aligning the characterization of the clean sample and the adversarial sample in combination with a content discriminator and a behavior discriminator; and outputting a target user classification result through the multi-task target function optimization model. According to the method, the problems of insufficient modal fusion and insufficient adversarial robustness of an existing method can be effectively solved, and the accuracy and stability of social robot detection in a complex and adversarial scene are improved.

Description

Technical Field

[0001] This invention belongs to the field of social robot detection technology, specifically relating to a social robot detection method, system, computer equipment, and storage medium. Background Technology

[0002] In modern society, social media platforms have become core carriers of information dissemination, public opinion formation, and user interaction. However, these platforms also face a serious threat from social bots. Social bots are automated programs capable of mimicking human user behavior, participating in and even manipulating information flow within social networks by publishing content, forwarding information, and commenting. With the continuous evolution of technology, the disguise methods of social bots have become increasingly sophisticated, evolving from simple spam sending to highly anthropomorphic intelligent robots. These bots can not only generate natural language content but also dynamically adjust their interaction strategies based on topic popularity, user interests, and social profiles, exhibiting behavioral patterns highly similar to real users. They often integrate into real social networks by establishing false social relationships, participating in discussions of trending topics, and simulating emotional expression, forming hidden and synergistic social bot communities. This highly mimicry makes social bots more covert and deceptive online, greatly increasing the difficulty of detection. Social bots are not only experiencing explosive growth in number but also becoming more complex in their organizational forms, exhibiting characteristics of grouping and distribution. These bots can collaboratively create false public opinion storms, manipulate information dissemination paths, and even influence public issues through cross-platform collaboration, bringing profound social impacts. This phenomenon not only threatens the content ecosystem and user trust of social media platforms but may also trigger systemic risks in political, economic, and public safety fields. Therefore, effectively identifying and curbing the activities of social bots has become a key requirement for ensuring the healthy development of social media platforms.

[0003] Early social robot detection relied heavily on feature engineering, depending on manually extracted static features. While effective in simple scenarios, these methods were limited by the static nature of the features and struggled to adapt to evolving dynamic camouflage strategies. With the development of deep learning, multimodal social robot detection methods have emerged. For example, Zeng et al.'s MRLBot integrates text, temporal, and numerical signals through an ensemble network, and Hayawai et al.'s DeeProBot uses an autoencoder to mine abnormal patterns in user attributes and text content. However, these methods often employ simple modal concatenation, failing to address the modality imbalance problem—namely, the imbalance between behavioral and temporal characteristics. The importance of textual and structural modalities changes dynamically with robot strategies and application scenarios. A single fusion logic can easily dilute key modal information, affecting detection accuracy. The application of graph neural networks has further promoted the utilization of social structural information in detection. Feng et al. proposed BotRGCN to model the topology of social networks through relational graph convolutional networks, and Hu et al. designed HGT to introduce heterogeneous graph Transformers to capture multi-type user interaction relationships. However, most graph-based methods overemphasize the extraction and learning of structural features, ignoring the fine-grained characterization of textual semantic details and behavioral temporal dynamics, resulting in insufficient ability to identify disguised robots.

[0004] Based on the above analysis, existing technologies are insufficient in modeling the behavioral data of social robots, which limits the robot's recognition ability under complex social relationships and has limited generalization ability to emerging robot camouflage strategies. This results in a significant drop in detection performance when facing unknown camouflage strategies, and fails to meet the detection needs in complex scenarios. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a social robot detection method, system, computer device, and storage medium.

[0006] To achieve the above objectives, the present invention provides a social robot detection method, comprising: Obtain account attributes, text data, behavioral sequence data, and social relationship data of target users in social media software; obtain account attributes, text data, and behavioral sequence data of each social user in the target user's social relationships in social media software.

[0007] Add time-location encoding to the behavior sequence data of all users to obtain behavior embedding sequences containing behavior vectors and behavior occurrence times; extract local behavior features of each behavior embedding sequence; analyze the long-range interaction behavior dependencies of each local behavior feature to obtain global temporal features; process the global temporal features of each user into fixed-dimensional user behavior features.

[0008] Extract the categorical and numerical attribute features of the account attributes of each user; extract the text features of the text data of each user; concatenate the categorical attribute features, numerical attribute features, text features, and user behavior features to obtain multimodal fusion features; project the multimodal fusion features of each user into initial node features; aggregate the structural attention features between the initial node features of the target account and the initial node features of the social relationship account to obtain multimodal relationship aggregation structural features.

[0009] By utilizing the multimodal relationship aggregation structure features, it is possible to detect whether a target user is a social bot.

[0010] Preferably, the text data includes account profile and posted tweets; the social relationship data includes the follow and being followed relationships between the target user and other users; the account attributes include categorical attributes and numerical attributes; the categorical attributes include account type and geographic tags; and the numerical attributes include registration duration, number of followers, and number of followings.

[0011] Preferably, the process involves adding time-location encoding to the behavior sequence data of all users to obtain behavior embedding sequences containing behavior vectors and behavior occurrence times; extracting local behavior features from each behavior embedding sequence; analyzing the long-range interaction behavior dependencies of each local behavior feature to obtain global temporal features; and processing the global temporal features of each user into fixed-dimensional user behavior features. Specifically, this processing includes: converting all users' behavior sequence data into binary data using an embedding matrix, adding time-location encoding to the binary data to obtain behavior embedding sequences containing behavior vectors and behavior occurrence times; extracting local behavior features from each behavior embedding sequence using a bidirectional long short-term memory network; analyzing the long-range interaction behavior dependencies of each local behavior feature using a Transformer network to obtain global temporal features; filtering the global temporal features of each user using a mask, and performing mean pooling on the filtered sequences to obtain fixed-dimensional user behavior features for all users.

[0012] Preferably, the account attributes of each user are converted into encoded vectors using One-Hot encoding, and the categorical attribute features of the encoded vectors are extracted sequentially using a fully connected layer and the Leaky-ReLU activation function. The encoded vectors are then normalized and processed by a fully connected layer using the Z-score algorithm to obtain numerical attribute features. The text features of the text data are extracted using RoBERTa. Two layers of RGCN are used to aggregate the structural attention features between the initial node features of the target account and the initial node features of the social relationship account, respectively, to obtain multimodal relationship aggregation structural features.

[0013] The method also includes processing the multimodal relationship aggregation structure features into adversarial panorama features, specifically including: adding initial perturbations to the multimodal relationship aggregation structure features using Projected Gradient Descent (PGD) to generate initial adversarial panorama features; separating features of each modality from the initial adversarial panorama features to obtain initial adversarial samples for each modality; calculating the classification probability loss between the multimodal relationship aggregation structure features and the real label of the target account using a master classifier, calculating the gradient L2 norm of the classification probability loss with respect to each modality in the multimodal relationship aggregation structure features through gradient sensitivity, normalizing the gradient L2 norm of each modality to obtain the weights of each modality; cyclically executing PGD using each modality weight to specifically adjust the perturbation amplitude of each modality adversarial sample; stopping the loop when the classification probability loss is less than a set threshold to obtain optimized adversarial samples for each modality; and concatenating the optimized adversarial samples for each modality to obtain optimized adversarial panorama features.

[0014] Preferably, the method further includes training and optimizing the shared feature encoder using the multimodal relationship aggregation structural features and adversarial panopticon features, specifically including: Multimodal relationship aggregation structure features are mapped to clean fusion features, and clean behavior features of clean fusion features are extracted; adversarial panorama features are mapped to adversarial fusion features, and adversarial behavior features of adversarial fusion features are extracted. The main classifier is used to calculate the main classification loss between the multimodal relationship aggregation structure features of the target account and the real label; the content discriminator loss function is used to calculate the probability difference between the clean fusion feature and the adversarial fusion feature; the behavior discriminator loss function is used to calculate the probability difference between the clean behavior feature and the adversarial behavior feature; and the log loss function is used to convert the probability difference into the content discriminator loss and the behavior discriminator loss, respectively. A total objective function is constructed based on the main classification loss, content discriminator loss, and behavior discriminator loss. Based on the overall objective function, the gradients of all losses with respect to the weight matrix and biases of the shared feature encoder are calculated using the backpropagation algorithm. The weight matrix and biases are then optimized along the gradient descent direction to minimize the loss of the overall objective function, resulting in the optimized shared feature encoder.

[0015] Preferably, the method further includes detecting whether a target user is a social bot using the multimodal relationship aggregation structure features, specifically including: The optimized shared feature encoder is used to map the multimodal relationship aggregation structure features of the target user to generate low-dimensional fusion features; The low-dimensional fusion features are input into the main classifier to obtain the probability that the target user is a social robot; Based on a preset probability threshold, accounts with a probability greater than the threshold are determined to be social bots; those with a probability less than or equal to the threshold are determined to be human, and the target user classification result is output.

[0016] The present invention also provides a social robot detection system, comprising: The data acquisition module is used to acquire the account attributes, text data, behavioral sequence data, and social relationship data of the target user in the social software; and to acquire the account attributes, text data, and behavioral sequence data of each social user in the target user's social relationship in the social software.

[0017] The feature extraction module adds time-location encoding to the behavioral sequence data of all users to obtain behavioral embedding sequences containing behavioral vectors and behavioral occurrence times; extracts local behavioral features from each behavioral embedding sequence; analyzes the long-range interaction behavior dependencies of each local behavioral feature to obtain global temporal features; processes the global temporal features of each user into fixed-dimensional user behavioral features; extracts categorical and numerical attribute features of the account attributes of each user; extracts text features from the text data of each user; concatenates categorical attribute features, numerical attribute features, text features, and user behavioral features to obtain multimodal fusion features; projects the multimodal fusion features of each user into initial node features; and aggregates the structural attention features between the initial node features of the target account and the initial node features of the social relationship account to obtain multimodal relationship aggregation structural features.

[0018] The target detection module is used to detect whether a target user is a social bot by utilizing the multimodal relationship aggregation structure features.

[0019] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement any of the steps in the social robot detection method.

[0020] The present invention also provides a computer-readable storage medium storing a computer program that, when loaded by a processor, can execute any of the steps in the social robot detection method.

[0021] The social robot detection method provided by this invention has the following beneficial effects: This invention comprehensively analyzes user attribute, text, behavior, and social relationship data to construct a three-dimensional user profile; it effectively aggregates structural attention features between target accounts and social users, while strengthening the utilization of multimodal information such as account attributes, text, and behavior, thus improving the ability to identify bots in complex social relationships; it fully combines time-location encoding to mine the dynamic patterns of behavior, which is extremely effective in identifying advanced bots whose behavior appears normal but whose rhythm reveals patterns, accurately capturing the temporal evolution features of bot behavior patterns; the multimodal relationship aggregation structural features can keenly capture abnormal group features, even if some features of a single account are well disguised, the social network structure in which it belongs will expose them, improving the detection performance of social bots when facing unknown disguise strategies and meeting the detection needs in complex scenarios. Attached Figure Description

[0022] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a social robot detection method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall processing framework and a sensitivity analysis diagram of an embodiment of the present invention. Figure 3 This is a sensitivity analysis diagram for two training cycles in an embodiment of the present invention; Figure 4 This is a sensitivity analysis diagram for three training cycles in an embodiment of the present invention; Figure 5 Sensitivity analysis diagrams for 5 training cycles in an embodiment of the present invention; Figure 6 shows the accuracy and robustness evaluation of the detection method according to the embodiment of the present invention; Figure 6(a) shows the accuracy of the model at the FGSM attack intensity; Figure 6(b) shows the accuracy of the model at the PGD attack intensity; Figure 6(c) shows the robustness of the model at the FGSM attack intensity; Figure 6(d) shows the robustness of the model at the PGD attack intensity. Figure 7 This is an experimental diagram illustrating the data efficiency study of tag data in an embodiment of the present invention. Figure 8 This is an experimental diagram showing the data efficiency of edge features in an embodiment of the present invention. Figure 9 This is an experimental diagram illustrating the data efficiency study of node features in an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0025] This invention provides a method for detecting social robots, specifically as follows: Figure 1 As shown, it includes: S1. Obtain the account attributes, text data, behavioral sequence data, and social relationship data of the target user in the social software; obtain the account attributes, text data, and behavioral sequence data of each social user in the target user's social relationship in the social software.

[0026] Input multi-source user data from social media platforms; Represents the set of user nodes in a social platform. , Total number of users. Each user It includes attribute information and text content information. The attribute information includes categorical attributes. With numerical attributes ; Classification attributes This includes discrete attributes such as user account type and geographic tags, as well as numerical attributes. This includes attributes such as user registration duration, number of followers, and number of followers; text content information includes the user's personal description text. With the text of the tweet 'i' is the user index, and its value range is... The input data includes social connections such as "following" and "being followed" between users, used to construct the social network structure. A Transformer-BiLSTM hybrid encoder processes user behavior sequences, capturing local dependencies and global temporal dynamics of behavior to generate behavioral feature representations. A relational graph convolutional network is used to construct a heterogeneous social graph containing multiple relationship types and fuses multimodal relationship aggregation structural features. Social robot detection is achieved through modality importance adaptive adversarial training and multi-task optimization.

[0027] S2. Add time and location encoding to the behavior sequence data of all users to obtain behavior embedding sequences containing behavior vectors and behavior occurrence times; extract local behavior features of each behavior embedding sequence; analyze the long-range interaction behavior dependencies of each local behavior feature to obtain global temporal features; process the global temporal features of each user into fixed-dimensional user behavior features.

[0028] like Figure 2As shown, the personal profile feature extraction module in the upper left part of the social bot detection framework of this invention is used to extract static attribute information of users, including basic user information, number of followers, number of fans, and other statistical features, providing a user profile basis for subsequent analysis. The central part of the framework displays a relational graph convolutional network module, a relational modeling component based on graph neural networks. It constructs a social network topology by analyzing the follower and fan relationships between users, distinguishing the interaction patterns of human users (blue nodes) and potential bots (orange nodes). The user behavior sequence analysis module in the lower left uses a time-series modeling method to process dynamic user behavior data, including time-series analysis of multi-dimensional behavioral features such as forwarding, replying, posting, and mentioning. The user behavior sequence analysis module at the bottom of the framework is the core processing unit, implementing multi-layer feature abstraction and representation learning based on the Transformer encoding architecture, integrating time-series behavioral patterns and outputting high-dimensional feature representations. The multimodal adversarial learning module in the upper right is an innovative component, containing a multimodal importance perception module and a PGD (Projected Gradient Descent) adversarial example generator, enhancing the robustness and generalization ability of the model through adversarial training mechanisms. The classification decision module serves as the final output layer of the network, integrating multi-source feature information to achieve binary classification decisions for robot detection.

[0029] The user behavior sequence is processed by a Transformer-BiLSTM hybrid encoder to generate behavioral feature representations. The specific process is as follows: (1) For each target user Its behavioral sequence is represented as ,in The length of the behavior sequence. Selected from the fixed behavior vocabulary list ; through embedding matrix Each behavior Convert to initial embedding vector, To embed the dimension, learnable positional encoding is then superimposed. To preserve the temporal order, the embedded sequence is obtained. The calculation formula is as follows: (1) (2) Embedding sequence Input the bidirectional long short-term memory (BiLSTM) network and compute the forward hidden states respectively. With reverse hidden state The calculation formula is as follows: (2) The two are then concatenated to obtain the local behavioral features at each time step. The calculation formula is as follows: (3) Then, the BiLSTM output sequence is generated. The calculation formula is as follows: (4) (3) The input is a Transformer encoder, which first models the long-range interaction dependencies of the action sequence through a multi-head attention mechanism to obtain the attention output. The calculation formula is as follows: (5) The feedforward network then further processes the data to generate global temporal features. The calculation formula is as follows: (6) (4) Finally, on Perform mask mean pooling, using the mask By filtering the features corresponding to valid behaviors, fixed-dimensional user behavior features are calculated. This representation integrates local details of behavior with global temporal patterns, and the calculation formula is as follows: (7) S3. Extract the categorical attribute features and numerical attribute features of the account attributes of each user; extract the text features of the text data of each user; concatenate the categorical attribute features, numerical attribute features, text features and user behavior features to obtain multimodal fusion features; project the multimodal fusion features of each user into initial node features; aggregate the structural attention features between the initial node features of the target account and the initial node features of the social relationship account to obtain multimodal relationship aggregation structural features.

[0030] By leveraging multi-relationship type information in social networks, a social heterogeneous graph is constructed. Features of the heterogeneous graph are learned using a relation graph convolutional network, and user node representations are generated by fusing multimodal initial features of users. The specific steps are as follows: (1) Construct a social heterogeneous graph based on the user social relationship data collected by S1. This constructs a heterogeneous social graph, with users as nodes and the follow relationships between users as edges. ,in Represents the set of user nodes in a social network. , This represents the total number of users. This represents the set of social interaction edges between users, where each edge corresponds to one interaction between a pair of users; Correspondence type set .

[0031] (2) Integrate user multimodal data and behavioral features to generate user nodes. initial features The pre-trained language model RoBERTa was used to process user-described text. and the text of the tweet To obtain user description features and tweet features ;Classification attributes One-hot encoding is performed, followed by processing through a fully connected layer and a leaky-ReLU activation function to obtain the categorical attribute feature representation. ; to numerical attributes Numerical attribute features are obtained by Z-score normalization followed by transformation through a fully connected layer. .

[0032] (3) Through feature splicing and modal fusion transformation Projecting the above features and the behavioral features obtained in S2 onto the unified latent space yields the initial node features. The calculation formula is as follows: (8) in This indicates a feature concatenation operation. As input features of RGCN.

[0033] (4) A two-layer RGCN is used for structural feature extraction, and multiple relation types are used to model complex interactions between users. For the first... layer( , (for input layer features), nodes In the Features of the layer It is obtained by aggregating the features of its neighbors under different relationships with its own features, and the calculation formula is as follows:

[0034] (9) in For nodes In relationship The set of neighboring nodes; This is a normalization factor used to balance the differences in the number of neighbors under different relationships; For relationship In the The trainable weight matrix of the layer is used to adapt the neighbor feature transformation under this relationship; In relationship The neighboring nodes In the Features of the layer; This is the node self-loop weight matrix, used to retain the feature information of the node itself from the previous layer; The activation function is used to achieve nonlinear feature transformation.

[0035] (11) After propagation through two layers of RGCN, the output node Structural features This feature aggregates structural context information from multiple relation types in the social heterogeneous graph, and can characterize the positional attributes of user nodes in the social network topology and the interaction patterns between users, providing highly discriminative feature inputs for subsequent training tasks.

[0036] S4. Detect whether the target user is a social robot using the multimodal relationship aggregation structure features.

[0037] Social robot detection is achieved based on modality importance adaptive adversarial training and multi-task optimization. The specific process is as follows: (1) For the user multimodal features output by S3, Projective Gradient Descent (PGD) is used to generate adversarial examples for each modality, and the perturbation intensity is adaptively adjusted according to the modality importance. Modal features Adversarial examples are generated through PGD iteration. The calculation formula is: (10) in, For modality Disturbance budget. The iteration step size, The gradient of the main classification loss with respect to the modality feature. The projection function ensures that the perturbed features remain in the original feature space. Within the neighborhood.

[0038] The importance of a mode is quantified by gradient sensitivity, and the main classification loss is calculated with respect to the mode. The gradient L2 norm of a feature, used as a modality sensitivity index, is calculated using the following formula: (11) Based on this, the modal importance weights are obtained by normalizing the sensitivity index: (12) in, An index variable is used to iterate through all modalities, representing different modality types, through sensitivity indices for all modalities. Summation is used to normalize the sensitivity index of the current mode m; To prevent the use of tiny constants with a denominator of zero, a larger perturbation budget is allocated to modes of higher importance to enhance robustness. The calculation formula is as follows:

[0039] (13) in, This represents the global fundamental perturbation coefficient. , These are hyperparameters that can be adapted and adjusted.

[0040] To align the representation distributions of clean samples and adversarial samples, this paper introduces a dual discriminator architecture, where the content discriminator... Taking multimodal fusion features as input, specifically including clean fusion features Features of fusion with adversarial forces The probability that an output sample belongs to a clean sample is calculated using the following loss function: (14) in, Indicates clean fusion features Calculate the expected value of the probability distribution it follows; Indicates the characteristics of adversarial fusion The expected value of the probability distribution it follows is calculated; the goal is for the discriminator to distinguish cleanly from adversarial fusion features, while guiding the feature encoder to generate representations that are difficult for the discriminator to distinguish; behavior discriminator. Focusing on temporal behavioral modalities, inputting clean behavioral features and confrontational behavior characteristics The probability that an output sample belongs to a clean sample is calculated using the following loss function: (15) This is used to optimize the consistency of behavioral modal representations in order to address the imitation and perturbation of behavioral patterns by social robots.

[0041] (4) Construct a general objective function that integrates the main classification loss and the dual adversarial loss, and use minmax The strategy alternately optimizes the feature encoder and the dual discriminator. The overall objective function is calculated as follows: (16) in Primary classification loss, , The minmax optimization strategy formula is as follows: (This is a coefficient used to balance the adversarial losses and control the strength of adversarial regularization.) (17) in, For sharing feature encoders and classifiers; This indicates a shared feature encoder. Perform minimization optimization; Indicates the discriminator and The optimization is maximized; the model trained by this optimization strategy can output the classification results of human and robot users in social networks, thereby achieving accurate identification and classification of social robots in social networks.

[0042] Table 1 shows the scale of the dataset used in this embodiment. As shown in Table 1, the datasets used in this experiment consist of two real-world datasets: TwiBot-22 and Cresci-15, to evaluate the applicability and effectiveness of the proposed method. These datasets vary in size; TwiBot-22 is currently the largest publicly available Twitter bot dataset, offering a significant advantage in scale, covering approximately one million users. This dataset is rich in data dimensions, containing four entity types and fourteen relationship types, capable of simulating the complex relationship structures of real social platforms. The Cresci-15 dataset comprises verified human accounts and bot accounts from online marketplaces, containing 1950 human accounts and 3351 bot accounts. Its balanced sample distribution provides an ideal benchmark for evaluating model performance in a controlled and representative scenario. These two real-world datasets provide a comprehensive evaluation of the method's performance in different real-world scenarios.

[0043] Table 2 presents the results of seven methods on two real-world datasets, evaluated using four metrics: accuracy, precision, recall, and F1-score. The best results are highlighted in bold, while the next best are indicated by underline.

[0044] Table 2 Classification accuracy on real-world datasets As shown in Table 2, the proposed MIAAE method consistently demonstrates superior performance on both the Cresci-15 and TwiBot-22 datasets, outperforming all baseline methods in the four core metrics of accuracy, precision, recall, and F1 score. Its performance advantage is particularly significant on the TwiBot-22 dataset, a highly challenging benchmark set characterized by its large size, heterogeneous structure, and complex robot adversarial strategies, which places higher demands on the multimodal fusion capabilities and robustness of the detection model. MIAAE, however, maintains its leading edge.

[0045] Unlike existing methods, most baseline methods only perform well under specific metrics or limited scenarios. MIAAE maintains a strong balance between precision and recall, demonstrating its excellent generalization ability in both clean and adversarial environments. This robustness stems from MIAAE's unified architecture design: by combining multimodal fusion and adversarial training, it collaboratively models user behavior, content semantics, and social structure, effectively integrating multi-dimensional key information and avoiding the limitations of single-modal or static modeling.

[0046] Specifically, for traditional detection methods, while hand-designed feature-based methods can capture recurring patterns in user metadata or activity statistics in stable environments, thus demonstrating high accuracy, their static nature makes them difficult to adapt to novel or deceptive bot behaviors, especially in scenarios with high user diversity and strong adversarial noise, where their adaptability shortcomings are even more pronounced. MIAAE, on the other hand, learns discriminative feature representations through multimodal fusion, enabling it to capture subtle behavioral cues and social signals that traditional hand-designed features cannot cover, effectively overcoming this deficiency.

[0047] For deep learning methods: While these methods have driven the development of social robot detection by automatically extracting features from raw inputs, they often only model a limited range of modalities and lack robustness to data distribution shifts. In contrast, MIAAE integrates two core mechanisms—temporal modeling and adversarial training—within a unified multimodal framework, simultaneously improving the model's generalization ability and stability on complex real-world datasets.

[0048] For graph-based methods: While these methods can enhance detection performance by leveraging relational dependencies in social networks, they are often limited by performance issues when facing heterogeneity, data sparsity, or behavioral ambiguity. MIAAE effectively addresses these challenges by deeply fusing structural encoding with multimodal text, behavior, and adversarial signals, demonstrating stronger detection capabilities, especially in scenarios with information confusion.

[0049] To evaluate the effectiveness of each component in MIAAE, this embodiment conducted ablation experiments on the Cresci-15 and TwiBot-22 datasets, and the results are shown in Table 3. This embodiment systematically removed different components, covering single modalities (tweets, user descriptions, numerical and categorical user attributes), modeling modules (behavior extractors, adversarial training), and fusion strategies (importance weighting, adversarial integration).

[0050] Table 3 Ablation experiments of the method of this invention. Experimental results consistently show that model performance degrades when any component is removed, confirming the complementary effects between modules. On the Cresci-15 and TwiBot-22 datasets, semantic information from user descriptions has a greater impact, highlighting the importance of semantic context-aware modal fusion. The behavior encoder, composed of BiLSTM and Transformer, performs crucially in capturing temporal dynamics. Notably, adversarial training and importance weighting significantly improve model robustness in noisy and avoidance scenarios. These findings validate the ensemble design of MIAAE, achieving a balance between accuracy and robustness in social robot detection tasks.

[0051] Figure 3 This embodiment demonstrates sensitivity analysis performed on the TwiBot-22 dataset every two training epochs to study the impact of the frequency of adversarial training and the loss weights assigned to content and behavioral modalities on model performance. Here, λ1 and λ2 represent the loss weights for content and behavioral modalities, respectively, while the values ​​and color intensities in the heatmap reflect the model's accuracy. Specifically, when adversarial training is performed every two training epochs, the heatmap shows a large area of ​​dark blue, indicating that the model maintains a high accuracy of 82-85% even with a wide combination of content modal weights λ1 and behavioral modal weights λ2, demonstrating strong robustness in the overall parameter space. When adversarial training is performed at moderate intervals, such as... Figure 4 As shown in the sensitivity analysis every 3 training epochs on the TwiBot-22 dataset, the model maintains stable and excellent performance across a wide range of weight configurations. Although a slight decrease in accuracy occurs at certain λ1 and λ2 combinations compared to higher training frequencies, it still maintains a considerable high-performance stable region. With decreasing training frequency, as... Figure 5 The sensitivity analysis presented on the TwiBot-22 dataset every 5 training epochs shows that model performance becomes more sensitive to the selection of content and behavioral modality loss weights, and the effective parameter space is significantly reduced accordingly. The light-colored areas in the heatmap increase significantly, especially in areas with high behavioral modality weights (λ2), where accuracy drops noticeably, with some areas seeing accuracy fall to 76-78%. Furthermore, the results demonstrate that MIAAE exhibits strong robustness to training dynamics under appropriate regularization, ensuring its adaptability under different hyperparameter configurations, particularly achieving an optimal balance between accuracy and stability at moderate adversarial training frequencies. The experimental results of this embodiment show that even under moderate-intensity adversarial training, MIAAE is not prone to overfitting or sudden performance degradation; all three subplots show relatively smooth accuracy transitions, providing a safe operating margin for practical deployment.

[0052] Figure 6 illustrates the adversarial robustness evaluation of MIAAE. This embodiment was tested under two widely adopted adversarial attack strategies, FGSM and PGD, where accuracy represents the model's performance. As shown in Figure 6(a), under FGSM attacks, the accuracy of the baseline model without any adversarial defenses drops sharply from approximately 0.85 to nearly 0.05 as the perturbation strength increases from 0.00 to 0.10, indicating its high sensitivity to adversarial interference. In contrast, the MIAAE adversarial model maintains an accuracy level of approximately 0.85 to 0.35 under the same attack strength, demonstrating stronger stability. Figure 6(b) further verifies this advantage; under the more powerful adversarial method PGD attacks, the performance of the baseline model also deteriorates rapidly to near ineffectiveness, while MIAAE maintains a relatively high accuracy.

[0053] The robustness improvement analyses in Figures 6(c) and 6(d) more intuitively demonstrate the defensive effect of MIAAE. Under FGSM attack scenarios, as the attack intensity increases, the accuracy gain gradually increases from approximately 0.02 to over 0.35, showing a clear increasing trend. Under PGD attacks, although the overall gain is relatively small, it reaches a peak gain of approximately 0.15 at an attack intensity of 0.02. These results verify the effectiveness of the proposed adversarial training strategy in improving the model's ability to cope with evasive robot behavior in complex social network environments. As the attack intensity increases, the performance gap between MIAAE and the baseline model continues to widen, further highlighting the significant advantages of the method in this embodiment in terms of defensive capabilities.

[0054] Figure 7 , Figure 8 and Figure 9 This paper showcases data efficiency research experiments conducted by MIAAE on the Twibot-22 dataset. Current social robot detection methods typically rely on supervised learning, assuming access to large amounts of labeled data. However, in real-world scenarios, labeled samples are often scarce, noisy, or costly to obtain. To evaluate MIAAE's generalization ability under limited supervision, this embodiment tests its performance on progressively smaller subsets of labeled data. Figure 7 As shown, in terms of label features, the red accuracy curve demonstrates MIAAE’s excellent data efficiency, steadily improving from 82.47% when using only 10% of the training data to 84.45% when using the full amount of data. The pink F1 score histogram also remains at a high level of about 60%. Figure 8 The effective use of edge features was further verified. The yellow accuracy curve steadily increased from 83.71% to 84.45%, showing stronger initial performance and a stable improvement trajectory than the label features. The corresponding light yellow F1 histogram also maintained stable performance. Figure 9The blue accuracy curve of the mid-node features shows that the accuracy gradually increased from 82.79% to 84.45%, and the light blue F1 score histogram also showed a stable upward trend.

[0055] These experimental results demonstrate that MIAAE effectively utilizes additional training signals. Even in a low-resource environment with only 10% of the data, the accuracy curves for all three feature types maintain a competitive level above 82%. Furthermore, accuracy and robustness significantly improve with increasing label count. All three accuracy curves eventually converge to the performance ceiling of 84.45%, while the F1 score histogram remains stable in all cases, highlighting the framework's practicality for deployment in dynamic environments and its excellent data scalability.

[0056] Based on the same inventive concept, the present invention also provides a social robot detection system, comprising: The data acquisition module is used to acquire the account attributes, text data, behavioral sequence data, and social relationship data of the target user in the social software; and to acquire the account attributes, text data, and behavioral sequence data of each social user in the target user's social relationship in the social software.

[0057] The feature extraction module adds time-location encoding to the behavioral sequence data of all users to obtain behavioral embedding sequences containing behavioral vectors and behavioral occurrence times; extracts local behavioral features from each behavioral embedding sequence; analyzes the long-range interaction behavior dependencies of each local behavioral feature to obtain global temporal features; processes the global temporal features of each user into fixed-dimensional user behavioral features; extracts categorical and numerical attribute features of the account attributes of each user; extracts text features from the text data of each user; concatenates categorical attribute features, numerical attribute features, text features, and user behavioral features to obtain multimodal fusion features; projects the multimodal fusion features of each user into initial node features; and aggregates the structural attention features between the initial node features of the target account and the initial node features of the social relationship account to obtain multimodal relationship aggregation structural features.

[0058] The target detection module is used to detect whether a target user is a social bot by utilizing the multimodal relationship aggregation structure features.

[0059] This invention proposes a multimodal adaptive adversarial training module. This module combines modality importance gradient sensitivity analysis with a projected gradient descent perturbation generation mechanism. It dynamically adjusts the adversarial perturbation intensity based on the contribution of each modality to the detection results, strengthening the robustness of key modalities while avoiding ineffective perturbations from redundant modalities. This effectively solves the "modal imbalance" problem in multimodal detection and improves the discriminative power of feature representation. A Transformer-BiLSTM hybrid behavior encoder is introduced. BiLSTM captures the local temporal dependencies of user behavior sequences, while Transformer models global behavioral dynamics. Compared to traditional single temporal models, this more accurately extracts human-machine behavioral difference features, improving the detection model's ability to capture fine-grained behavioral cues. A multi-task objective function integrating main classification loss, content adversarial loss, and behavior adversarial loss is constructed. A minmax optimization strategy is used to alternately update the shared feature encoder and dual discriminator. While ensuring the classification accuracy between normal users and robots, the model learns features across clean and adversarial examples, further improving the stability and generalization ability of the detection results.

[0060] This invention also provides a computer device. At the hardware level, the computer device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the social robot detection method provided above.

[0061] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the social robot detection method provided above.

[0062] Specific limitations regarding the computational system for the social robot detection method can be found in the limitations of the social robot detection method described above, and will not be repeated here. Each module in the aforementioned social robot detection system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0063] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. Furthermore, the above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A social robot detection method, characterized by, The method comprises the following steps: Obtain the account attributes, text data, behavior sequence data, and social relationship data of the target user in the social software; Obtain the account attributes, text data, and behavior sequence data of each social user in the target user's social relationship in the social software; Add time position coding to the behavior sequence data of all users to obtain behavior embedding sequences containing behavior vectors and behavior occurrence times; extract local behavior features of each behavior embedding sequence; analyze the long-range interactive behavior dependency relationship of each local behavior feature to obtain global timing features; and process the global timing features of each user into fixed-dimensional user behavior features. Extract the classification attribute features and numerical attribute features of the account attributes of each user; extract the text features of the text data of each user; concatenate the classification attribute features, numerical attribute features, text features, and user behavior features to obtain multi-modal fusion features; and project the multi-modal fusion features of each user into initial node features. Respectively aggregate the structural attention features between the target account initial node features and the social relationship account initial node features to obtain multi-modal relationship aggregated structure features. Use the multi-modal relationship aggregated structure features to detect whether the target user is a social robot.

2. The social robot detection method of claim 1, wherein, The text data includes account profiles and published tweets; the social relationship data includes the attention and attention relationship between the target user and other users; the account attributes include classification attributes and numerical attributes; the classification attributes include account types and regional tags; and the numerical attributes include registration duration, number of fans, and number of attention.

3. The social robot detection method of claim 1, wherein, The behavior sequence data of all users is added with time position coding to obtain behavior embedding sequences containing behavior vectors and behavior occurrence times; local behavior features of each behavior embedding sequence are extracted; the long-range interactive behavior dependency relationship of each local behavior feature is analyzed to obtain global timing features; and the global timing features of each user are processed into fixed-dimensional user behavior features, which specifically include: converting the behavior sequence data of all users into binary data through an embedding matrix, adding time position coding to the binary data to obtain behavior embedding sequences containing behavior vectors and behavior occurrence times; using a bidirectional long short-term memory network to extract local behavior features of each behavior embedding sequence; using a Transformer network to analyze the long-range interactive behavior dependency relationship of each local behavior feature to obtain global timing features; using a mask to filter the global timing features of each user, and performing mean pooling on the filtered sequence to obtain fixed-dimensional user behavior features of all users.

4. The social robot detection method of claim 1, wherein, Each user's account attributes are converted into an encoding vector using One-Hot encoding, and classification attribute features of the encoding vector are extracted using a fully connected layer and a Leaky-ReLU activation function in sequence; the encoding vector is normalized and processed by a fully connected layer through a Z-score algorithm to obtain numerical attribute features; text features of the text data are extracted using RoBERTa; and two-layer RGCN is used to aggregate the structural attention features between the target account initial node features and the social relationship account initial node features to obtain multi-modal relationship aggregated structure features.

5. The social robot detection method of claim 1, wherein, The multi-modal relationship aggregation structure feature is also processed into an adversarial panoramic feature, specifically including: adding an initial disturbance to the multi-modal relationship aggregation structure feature by using a projection gradient descent (PGD) to generate an initial adversarial panoramic feature; splitting the features of each modality from the initial adversarial panoramic feature to obtain initial adversarial samples of each modality; calculating a classification probability loss of the multi-modal relationship aggregation structure feature of the target account and a real label by using a main classifier, calculating a gradient L2 norm of the classification probability loss with respect to each modality in the multi-modal relationship aggregation structure feature by using gradient sensitivity, normalizing the gradient L2 norm of each modality to obtain a modality weight; performing a cycle of PGD by using the modality weight to adjust the disturbance amplitude of the adversarial sample of each modality; when the classification probability loss is less than a set threshold, stopping the cycle to obtain an optimized adversarial sample of each modality; and splicing the optimized adversarial sample of each modality to obtain an optimized adversarial panoramic feature.

6. The social robot detection method of claim 5, wherein, The multi-modal relationship aggregation structure feature and the adversarial panoramic feature are also used to train and optimize a shared feature encoder, specifically including: The multi-modal relationship aggregation structure feature is mapped into a clean fusion feature, and a clean behavior feature of the clean fusion feature is extracted; the adversarial panoramic feature is mapped into an adversarial fusion feature, and an adversarial behavior feature of the adversarial fusion feature is extracted; A main classification loss of the multi-modal relationship aggregation structure feature of the target account and a real label is calculated by using a main classifier; a probability difference between the clean fusion feature and the adversarial fusion feature is calculated by using a content discriminator loss function; a probability difference between the clean behavior feature and the adversarial behavior feature is calculated by using a behavior discriminator loss function; and the probability differences are respectively converted into a content discriminator loss and a behavior discriminator loss by using a logarithmic loss function; Based on the main classification loss, the content discriminator loss and the behavior discriminator loss, a total target function is constructed; Based on the total target function, a gradient of all losses with respect to a weight matrix and a bias of the shared feature encoder is calculated by using a back propagation algorithm; the weight matrix and the bias are optimized in a gradient descent direction to minimize the loss of the total target function, and an optimized shared feature encoder is obtained.

7. The social robot detection method of claim 6, wherein, The multi-modal relationship aggregation structure feature is also used to detect whether a target user is a social robot, specifically including: The multi-modal relationship aggregation structure feature of the target user is mapped by using the optimized shared feature encoder to generate a low-dimensional fusion feature; The low-dimensional fusion feature is input into a main classifier to obtain a probability that the target user is a social robot; According to a preset probability threshold, an account with a probability greater than the threshold is determined to be a social robot, and an account with a probability less than or equal to the threshold is determined to be a human, and a target user classification result is output.

8. A social robot detection system, characterized in that, The method includes: a data acquisition module configured to acquire account attributes, text data, behavior sequence data and social relationship data of a target user in a social software; acquire account attributes, text data and behavior sequence data of each social user in a target user social relationship in a social software; The feature extraction module is configured to add time position coding to the behavior sequence data of all users to obtain behavior embedding sequences containing behavior vectors and behavior occurrence times; extract local behavior features of each behavior embedding sequence; analyze long-range interaction behavior dependency relationships of each local behavior feature to obtain global timing features; process the global timing features of each user into user behavior features of a fixed dimension; extract classification attribute features and numerical attribute features of the account attributes of each user; extract text features of the text data of each user; splice the classification attribute features, the numerical attribute features, the text features, and the user behavior features to obtain multi-modal fusion features; and project the multi-modal fusion features of each user into initial node features. The structure attention features between the initial node features of the target account and the initial node features of the social relationship account are aggregated respectively to obtain multi-modal relationship aggregation structure features. The target detection module is configured to detect whether the target user is a social robot by using the multi-modal relationship aggregation structure features.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1-8. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program can implement the steps of the method in any one of claims 1 to 7 when loaded by the processor. The computer program can implement the steps of the method in any one of claims 1 to 7 when loaded by the processor.

Images