Social robot discovery method and system based on traffic multi-modal feature fusion

Abstract

The application discloses a social robot discovery method and system based on traffic multi-modal feature fusion. z is the behavior sequence sample of the zth robot; an OBTT model is constructed, which comprises an original traffic-to-grayscale image module, an original sequence feature coding module, an encoder, a decoder, a time-LSTM module, a feature fusion module and a full-connection neural network; 2) the OBTT model is trained by using unmarked robot encrypted network flows; 3) the trained OBTT model is fine-tuned by using the robot encrypted traffic data set on the social platform with marks; 4) the encrypted network flow to be detected is input into the fine-tuned OBTT model, and a corresponding robot class is obtained. The application enhances the detection capability for different robot behavior modes.

Description

Technical Field

[0001] This invention belongs to the field of network security technology and relates to a method and system for discovering social robots based on traffic multimodal feature fusion. Background Technology

[0002] Online social platforms have developed rapidly, with a surge in users and ever-expanding functionality. From initial text-based communication to today's multimedia interaction, technological advancements and the widespread adoption of mobile internet have further propelled the rapid development of online social platforms, making them an important part of people's daily lives, connecting global users, and promoting information sharing and social interaction. However, this rapid development has also brought a proliferation of bots on social platforms, some of which are of questionable quality. These bots are software-controlled, automatically performing fixed tasks and mimicking the behavior of normal users. Some bots can provide users with more perspectives and knowledge, breaking down information silos; however, others are used by manipulators as tools for malicious online manipulation, engaging in cyberbullying—spreading false and harmful information, damaging social and even national interests, and possessing powerful destructive capabilities. These bots are collectively referred to as social media bots. In a data-driven, traffic-dominated environment, social media bots, as a primary means of cyberbullying, have gradually become an industry chain. The forwarding, liking, and promotion services on social media are openly priced, amplifying and spreading hashtags to guide and manipulate users.

[0003] In recent years, social media platforms have seen widespread user engagement and rapid growth in business volume, but this has also led to a surge in cyberbullying and other illegal activities, damaging the online ecosystem and causing serious harm. Bots on social media platforms may be manipulated for activities such as cyberbullying and financial crimes. Therefore, developing effective detection solutions for social media bots is urgently needed. Different types of bots exhibit different behavioral patterns compared to ordinary users due to their different purposes and behavioral preferences. Analyzing these online behavioral patterns can reveal the essential rules and anomalies of bots, providing strong support for understanding their operational status and for collecting evidence and monitoring their illegal and criminal activities.

[0004] Methods for detecting bots on social media platforms have strict requirements for data sources (user profiles, social relationships, activity history), focusing on mining users' private content data or social relationship structures, which easily infringes on user privacy. Furthermore, with the application of anti-malicious bot technologies on social media platforms, the manipulators' resistance capabilities have increased, allowing for the rapid generation of large-scale bots with highly realistic behavior, forming complex and dense social network structures that confuse the platform's bot detection. Traditional network traffic analysis methods typically identify different network services based on features such as communication protocols and port numbers, but these features are no longer applicable due to the complexity and variability of traffic on current social media platforms. To address these issues, some studies rely on expert experience to classify network traffic using statistical features and machine learning methods, which is inefficient. Other studies use more efficient deep learning methods to automatically extract network flow features, but with different encryption protocols, packet lengths can fluctuate significantly, leading to uneven information distribution. Key features in shorter packets are difficult to capture, and model training requires a large amount of labeled data, resulting in high costs.

[0005] Text-based detection methods aim to detect bot-related behaviors, accounts, or clusters of accounts using textual information on social media platforms. This textual information may include explicit account information, such as profile details, posted content, interaction information, and configuration files, as well as implicit information, such as content patterns and emotional language expressions. By analyzing semantic and sentiment features of this textual information, user behavior profiles are constructed, modeling the differences between bots and human users. Such methods, by capturing textual features, offer good accuracy and interpretability, but suffer from the following drawbacks:

[0006] (1) The complexity of natural language. Users on social platforms have different linguistic and cultural backgrounds and use different grammatical structures, dialects and slang.

[0007] (2) It involves user privacy-sensitive content data. In-depth analysis of user-generated content may raise legal and ethical issues and is vulnerable to adversarial attacks.

[0008] (3) It is not very adaptable to the obscurity and variability of text, and the model may need to be updated frequently to maintain its effectiveness.

[0009] Social graph-based detection methods typically treat bot accounts on online social platforms as nodes, establishing connections between themselves and human users. They identify bots by analyzing connection patterns between accounts in the social network graph and based on the structural differences between bot accounts and real users, rather than relying excessively on text content. Such methods are less susceptible to content-oriented adversarial attacks. However, they still have certain drawbacks:

[0010] (1) It can only be effectively detected when the social network structure changes. If the bot frequently posts or browses without interacting with other accounts, it cannot be detected.

[0011] (2) Account-level detection can only determine whether the account is a real user or a bot.

[0012] Network traffic detection methods are being developed. As network environments become increasingly complex, machine learning (ML) algorithms are being introduced to explore the high-dimensional statistical characteristics of traffic to analyze complex network traffic. Deep learning (DL), on the other hand, analyzes traffic based on raw data packets rather than manually designed features. However, some existing DL-based methods directly use the bytes of data packets in the stream to represent raw traffic, leading to long data packets overwhelming important information in other packets and requiring a large amount of labeled data. Furthermore, current methods for detecting fine-grained behavior on social platforms are typically based on network flow segmentation. However, in the HTTP / 2 protocol, all communication under the same domain is completed on a single connection, which can carry any number of bidirectional data streams. It introduces HTTP header field compression and supports multiple concurrent data exchanges on a single HTTP connection. New features such as reset messages allow clients to stop and restart ongoing operations without disconnecting and resetting the HTTP connection. Flow control and prioritization allow for multiplexing of data streams. Consequently, it is impossible to determine the true label of the data stream, making methods using data streams as labeled samples infeasible. Summary of the Invention

[0013] To address the problems existing in current technologies, this invention aims to provide a social robot discovery method and system based on multimodal feature fusion of traffic. This invention utilizes large-scale unlabeled data for pre-training, focusing on "different behavioral patterns," specifically a series of robot actions within a time period. A small amount of labeled data is used to fine-tune the model, achieving the goal of detecting different behavioral patterns. Traditional robot detection methods based on user profiles, content, or social relationships risk infringing on user privacy. This invention proposes the OBTT (OSN Bots Traffic Transformer) learning framework, which has significant advantages in identifying robot traffic on social platforms and mining different types of robot behavioral patterns. OBTT employs a multimodal approach, fusing graph embedding representations and sequence features of the original traffic, while also incorporating temporal information to explore the regularity of robot action sequences. Large-scale unlabeled data is used to pre-train the MAE, and a small amount of labeled data is used for fine-tuning, enhancing the model's ability to detect different robot behavioral patterns.

[0014] Appendix Figure 2 As shown, OBTT employs a raw flow to grayscale conversion module, a raw sequence feature encoding module, an encoder, a decoder, a time-LSTM module, a feature fusion module, and a fully connected neural network.

[0015] 1) Raw Traffic to Grayscale Module. This module graphically represents the raw traffic, generating a high-dimensional visual embedding representation, such as... Figure 1 As shown;

[0016] 2) A masked autoencoder self-supervised learning model is used as the encoder to learn effective representations from unlabeled images. The image is randomly masked, and the masked patches are reconstructed using a decoder. The reconstruction error is used to optimize the model parameters until the model converges.

[0017] 3) Original Sequence Feature Encoding Module. Based on the original encrypted traffic sequences of different types of robots, time information is introduced. In robot behavior sequence classification tasks, time information such as time intervals and relative times play a crucial role in mining robot behavior patterns. Time-LSTM is used to capture sequence data features.

[0018] 4) Fine-tuning is used to detect different types of robot behavior patterns. During training, MAE weights are adjusted, model performance is monitored, and overfitting is prevented.

[0019] 5) Learn the correlation between images and text through an interactive attention mechanism, model the relationship between visual and text features, and perform feature fusion. Construct a robot classification model, quantify the similarity of behavior combinations, and complete the recognition of different types of robot behavior patterns.

[0020] OBTT is the first algorithm to detect bot traffic on online social platforms from a network traffic perspective. It not only identifies bot traffic but also identifies specific bot types by analyzing their behavioral patterns. On the BotTraffic6 dataset, OBTT achieves 99% accuracy in identifying social bot traffic. Furthermore, it identifies different bot types with 95% accuracy.

[0021] OBTT's main contribution is utilizing data from the network layer to detect anomalies on social platforms. However, OBTT has some limitations:

[0022] OBTT focuses on detecting behavioral patterns in different categories of social bots and has demonstrated excellent performance in identifying common types of bots on social platforms. However, if the bot's behavior is very similar to that of a regular user and does not exhibit any identifiable patterns, the detection results will be poor. OBTT's detection scope does not include specific social bots, such as zero-day social bots.

[0023] The grayscale images generated from the original flow characteristics are very large. During pre-training, to consider model efficiency, the grayscale images are cropped to some extent, which may lead to the loss of some important information. Therefore, when converting the original robot flow into grayscale images, the potential information loss needs to be considered. In robot behavior pattern classification tasks, temporal information plays a crucial role in revealing the relationships between robot behaviors. To address this, this invention incorporates temporal information into the original sequence features, working together with OBTT to model the original flow characteristics.

[0024] The technical solution of this invention is as follows:

[0025] A method for discovering social bots based on multimodal feature fusion of traffic, comprising the following steps:

[0026] 1) Obtain and label the encrypted network streams of T types of robots to obtain the encrypted traffic dataset of robots on the social platform; where f z Given the behavior sequence sample of the z-th robot, construct an OBTT model, which includes a raw flow to grayscale image module, a raw sequence feature encoding module, an encoder, a decoder, a time-LSTM module, a feature fusion module, and a fully connected neural network;

[0027] 2) Train the OBTT model using unlabeled robot encrypted network streams:

[0028] 21) Obtain multiple network flows of the same robot to form the robot's behavior sequence; the original flow to grayscale module converts the robot's behavior sequence into a grayscale image; divide the grayscale image into N image patches and perform masking processing on them; embed the image patch retained after masking processing into a high-dimensional space through a linear projection layer; embed the position of each image patch in the high-dimensional space into the corresponding image patch as the input of the encoder;

[0029] 22) The encoder calculates the importance of each input image patch relative to all other image patch patches in the grayscale image, and generates a vector representation of the corresponding image patch based on the calculation result; the vector representation of the i-th image patch includes the features of the i-th image patch and its dependencies on other image patch patches;

[0030] 23) The decoder reconstructs the corresponding image patch based on the vector representation of the image patch, and generates the target image based on each reconstructed image patch;

[0031] 24) Calculate the reconstruction error based on the target image and the corresponding grayscale image, and update the OBTT model by backpropagation based on the reconstruction error;

[0032] 3) Fine-tune the OBTT model trained in step 2) using a labeled dataset of encrypted bot traffic from social media platforms:

[0033] 31) The original flow to grayscale image module converts the behavior sequence samples f z The network flow samples are graphically represented to obtain the behavioral sequence samples f. z The grayscale image; the behavioral sequence sample f z The grayscale image is divided into N image patches and each image patch is embedded into a high-dimensional space through a linear projection layer. The position of each image patch in the high-dimensional space is embedded and added to the corresponding image patch as the input of the two encoders. The two encoders share parameters.

[0034] 32) The two encoders respectively generate vector representations of each image patch and input them into the feature fusion module;

[0035] 33) The original sequence feature encoding module encodes the behavioral sequence sample f z Feature extraction is performed to obtain behavioral sequence samples f. z The original sequence features; calculate the behavioral sequence sample f z The time interval between adjacent samples is added to the behavioral sequence sample f. z From the original sequence features, behavioral sequence samples f are obtained. z The feature sequence containing time interval information is input into the time-LSTM module;

[0036] 34) The time-LSTM module extracts behavioral sequence samples f z The features are then input into the feature fusion module;

[0037] 35) The feature fusion module provides the vector representation of each image patch and the behavior sequence sample f. z The features are fused to obtain behavioral sequence samples f. z The corresponding embedded features are input into the fully connected neural network for prediction;

[0038] 36) Fine-tune the OBTT model trained in step 2) based on the error between the prediction results and the corresponding true labels;

[0039] 4) Input the encrypted network stream to be detected into the OBTT model after fine-tuning in step 3) to obtain the robot category corresponding to the encrypted network stream to be detected.

[0040] Furthermore, in step 2), the MAE paradigm is used to train the asymmetric encoder-decoder architecture consisting of the encoder and decoder.

[0041] Furthermore, the time-LSTM module is obtained by introducing one or more Time Gates into the LSTM.

[0042] Furthermore, the encoder is composed of alternating multi-head self-attention layers and feedforward layers.

[0043] A social robot discovery system based on traffic multimodal feature fusion is characterized by comprising an OBTT model, a model training unit, and a detection unit; the OBTT model includes an original traffic-to-grayscale image module, an original sequence feature encoding module, an encoder, a decoder, a time-LSTM module, a feature fusion module, and a fully connected neural network;

[0044] The model training unit is used to train the OBTT model using unlabeled bot encrypted network streams, and to fine-tune the trained OBTT model using a bot encrypted traffic dataset from a social platform; wherein, encrypted network streams of T types of bots are obtained and labeled to obtain the bot encrypted traffic dataset from the social platform; f z This is a sample of the behavior sequence of the z-th robot;

[0045] The detection unit is used to input the encrypted network stream to be detected into the fine-tuned OBTT model to obtain the robot category corresponding to the encrypted network stream; wherein, the encrypted network streams of T types of robots are obtained and labeled to obtain the robot encrypted traffic dataset on the social platform; f z This is a sample of the behavior sequence of the z-th robot;

[0046] The method for training the OBTT model using unlabeled robot encrypted network streams is as follows: 21) Obtain multiple network streams of the same robot to form the robot's behavior sequence; the original flow to grayscale module converts the robot's behavior sequence into a grayscale image; divide the grayscale image into N image patches and mask them; embed the masked image patches into a high-dimensional space through a linear projection layer; embed the position of each image patch in the high-dimensional space into the corresponding image patch as the input of the encoder; 22) The encoder calculates the importance of each input image patch relative to all other image patches in the grayscale image, and generates a vector representation of the corresponding image patch based on the calculation result; the vector representation of the i-th image patch includes the features of the i-th image patch and its dependencies with other image patches; 23) The decoder reconstructs the corresponding image patch based on the vector representation of the image patch, and generates a target image based on each reconstructed image patch; 24) Calculate the reconstruction error based on the target image and the corresponding grayscale image, and update the OBTT model by backpropagation based on the reconstruction error;

[0047] The method for fine-tuning the trained OBTT model using a labeled, encrypted traffic dataset of bots on social platforms is as follows: 31) The original traffic to grayscale image module converts the behavior sequence samples f z The network flow samples are graphically represented to obtain the behavioral sequence samples f. z The grayscale image; the behavioral sequence sample f z The grayscale image is divided into N image patches, and each image patch is embedded into a high-dimensional space through a linear projection layer. The position of each image patch in the high-dimensional space is embedded and added to the corresponding image patch as input to the two encoders; the two encoders share parameters; 32) the two encoders generate vector representations of each image patch and input them into the feature fusion module; 33) the original sequence feature encoding module processes the behavior sequence sample f z Feature extraction is performed to obtain behavioral sequence samples f. z The original sequence features; calculate the behavioral sequence sample f z The time interval between adjacent samples is added to the behavioral sequence sample f. z From the original sequence features, behavioral sequence samples f are obtained. z 34) The time-LSTM module extracts the behavioral sequence samples f containing time interval information. zThe features are input into the feature fusion module; 35) The feature fusion module inputs the vector representation of each image patch and the behavior sequence sample f z The features are fused to obtain behavioral sequence samples f. z The corresponding embedded features are input into the fully connected neural network for prediction; 36) The trained OBTT model is fine-tuned based on the error between the prediction result and the corresponding real label.

[0048] A server is characterized by comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the methods described above.

[0049] A computer-readable storage medium having a computer program stored thereon, characterized in that the computer program implements the above-described method when executed by a processor.

[0050] The advantages of this invention are as follows:

[0051] OBTT avoids using user-privacy data from packet payloads, remaining unaffected by content interference, and instead focuses on mining bot behavior patterns. First, it graphically represents encrypted traffic data from different types of bots on social platforms, providing a more intuitive view and analysis of the network behavior characteristics of various bots. Next, it uses a masked autoencoder based on the MAE architecture for pre-training on large-scale unlabeled encrypted traffic data from social platforms, followed by fine-tuning on small labeled datasets. Furthermore, it introduces temporal information into the original sequences of bot encrypted traffic to further explore potential patterns and regularities, classifying bot behavior patterns. Through joint training on multiple tasks, it improves the model's performance in multimodal tasks, enhances classification and retrieval performance in the embedding space, reduces the need for large-scale labeled data, improves the model's generalization ability, and effectively reduces computational complexity. Attached Figure Description

[0052] Figure 1 This describes the process of converting raw traffic data to grayscale.

[0053] Figure 2 This is a flowchart of the training method of the present invention.

[0054] Figure 3 Grayscale images showing the traffic conversion results for different types of robots;

[0055] (a) Grayscale image of the broadcasting robot, (b) Grayscale image of the opinion robot, (c) Grayscale image of the chat robot, (d) Grayscale image of the hybrid robot, (e) Grayscale image of the publishing robot.

[0056] Figure 4 A statistical distribution plot for improved arrival time interval sequence characteristics.

[0057] Figure 5 This is the model's convergence curve.

[0058] Figure 6 This is the result of the ablation experiment. Detailed Implementation

[0059] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0060] Besides benign bots that don't aim to mimic humans, social media platforms commonly employ malicious bots that attempt to behave like humans and mislead users. Cyber ​​manipulators often use these malicious bots to perform highly realistic behaviors, spreading false and harmful information and engaging in cyberbullying, thus disrupting the online ecosystem. Through research, we broadly categorize common social media bots into five types: automated posting bots, broadcasting bots, opinion bots, chatbots, and hybrid bots. We explore the behavioral patterns of these different types of bots to provide methodological assistance for their detection.

[0061] Fine-grained action classification on social platforms refers to the process of dividing all possible actions in a specific application into several fixed categories. Strictly speaking, each action performed by a user is unique; that is, actions of different categories are independent of each other, and it is impossible to perform two actions simultaneously. In social platforms, actions are often performed in a series, consisting of several actions arranged chronologically. In terms of data traffic, this manifests as a single collection of data containing multiple actions. Furthermore, in the HTTP / 2 protocol, all actions may be completed over a single connection, and multiple frames are not sent in sequence.

[0062] Clearly, using behavioral sequences as the research object is more accurate and has greater practical significance. This invention mainly includes three stages:

[0063] The first stage is the preprocessing stage.

[0064] The second stage is the pre-training and fine-tuning stage.

[0065] The third stage is the behavior pattern detection stage.

[0066] In this invention, the encrypted traffic dataset of bots on a social platform contains N samples (streams) and T common types of bots, where T=5. The behavioral sequence sample of the z-th bot includes n z A sample (network flow) can be represented as: Where nz It is f z The characteristic length, For the z-th robot, the nth... z One sample (network flow), b i This represents the i-th network flow sample. Each network flow sample consists of multiple data packets and can be represented as follows: Let the j-th data packet satisfy p in bytes. j ∈[0, 1500), f is the R-th data packet in the i-th network flow of the z-th robot. z The label is denoted as L z , 1≤L z ≤T. The OBTT training method of the present invention is as follows: Figure 2 As shown below, the three stages will be described in detail.

[0067] ●Preprocessing—Multimodal Flow Representation

[0068] In the task of classifying encrypted traffic from different bots on social platforms, given a sequence sample f z To automate the feature extraction process, this invention graphically represents the raw traffic and extracts the raw sequence features, instead of using statistical information from encrypted traffic.

[0069] The original traffic is graphically represented. Experiments in (Wang et al., 2017) show that feature information extracted from all layers is more representative than data extracted from the application layer of the TCP / IP model. Simulating network traffic generated by different types of bots on a real social platform, the original traffic is divided according to the same five-tuple to obtain the feature information of uplink, downlink, and bidirectional flows. This invention anonymizes the original traffic features by converting IP addresses to random addresses and setting port numbers to 0, thereby introducing bias to interfere with training. Apart from this, no other filtering is performed to ensure the authenticity of the traffic as much as possible. Converting the anonymized original traffic features into grayscale images allows for the use of existing image processing techniques and deep learning models to improve the analysis capability of complex traffic patterns. For the entire dataset, F = {f1,…,f...} z},like Figure 3 As shown, the traffic of different types of bots on social platforms in F is converted into grayscale images to visualize the differences between them.

[0070] • Raw sequence feature extraction. In the task of mining robot behavior patterns, the regularity of actions is a key influencing factor. Time information, such as time intervals and relative times, is introduced into the raw sequences of encrypted traffic from different types of robots. Since different actions (such as commenting, liking, posting, etc.) require interaction with different components, the response time to the front end varies. Each data packet (p1,…,p…) is then processed. R The arrival interval time series PTS = {s1,…,s} n-1} Convert to the improved time interval G_PTS={a 1, …,a n-1}, where s j =t j+1 -t j s j = Δt·b + Δt / 2, where b satisfies the condition Δt·b ≤ s j <Δt·(b+1), after testing, select Δt=0.005s. For example... Figure 4 As shown in the figure, the statistical distribution of the improved arrival time interval sequence features is illustrated using a box plot. Chatbots and broadcasting robots exhibit a long-tail distribution, while publishing and opinion robots show smaller interquartile ranges, indicating more concentrated time intervals and greater regularity in their actions. Hybrid robots, however, may exhibit higher dispersion in their time intervals due to the complexity of their behavioral patterns and the lack of clear regularity in their actions, resulting in a longer time interval distribution. Therefore, the improved arrival time interval is a discriminative feature.

[0071] In this way, the original robot traffic with unknown content is transformed into grayscale images and improved original sequences, providing a data foundation for model pre-training and fine-tuning.

[0072] ●OBTT pre-training and fine-tuning

[0073] Based on the characteristics of the traffic, the grayscale image is masked. Through the collaboration of the encoder module and the decoder module, OBTT performs effective self-supervised learning to achieve the purpose of classifying the grayscale image.

[0074] The input grayscale image needs to undergo a series of processing steps so that the model can learn its global representation by observing local image features. The grayscale image is resized to 100*100 pixels and divided into a series of N*N patches, similar to cutting an image into 2D blocks. Specifically, in this invention, N=20, resulting in a total of 20*20=400 image patch blocks.

[0075] Before being input into the encoder, a portion of the patches are randomly masked. These patches will not be fed into the encoder, leaving only a small number of visible patches. This is similar to BERT's strategy of masking parts of the vocabulary in natural language processing tasks, with the aim of training the model to predict these missing patches. Each unmasked image patch is embedded into a high-dimensional space through a linear projection layer. Positional embeddings are added to the image patch as input to the encoder to preserve positional information in the image space; positional encoding is crucial.

[0076] This invention adopts the MAE (He et al 2022) paradigm and an asymmetric encoder-decoder architecture. The encoder consists of alternating multi-head self-attention layers and feedforward layers, allowing the learning of dependencies between header and payload patches of different data packets—that is, calculating the importance of each patch relative to all other patches. The input vector shape can be represented as [N,D], where N is the number of input patches, and D is the dimension of each embedded patch. [N,D] undergoes three linear transformations to obtain attention weights:

[0077] Q = X·W Q K = X·W k V = X·W v

[0078]

[0079] Among them W Q W k W v It is a learnable weight matrix. The new representation of each patch after being weighted by the global context is the encoder output. These representations combine global information from the input image, enabling the model to not only understand the features of local patches but also capture the relationships between them. In the task of mining different behavioral patterns, the pre-trained encoder parameters are applied to extract packet and network flow features, respectively. The calculated attention weights are used to weight and sum the value vector V to generate a new context vector, which represents the representation of each patch after considering information from other patches. In this work, the invention uses n = 16 parallel attention heads and L = 4 alternating layers.

[0080] The encoder aims to facilitate information exchange between data packets, without focusing on the features of the masked patches. The decoder, on the other hand, is responsible for "filling in the gaps," reconstructing the original bytes and restoring the features of the masked data packets. Through the decoder's reconstruction task, the model learns how to infer the global structure from the unmasked parts, building upon the local patch information. Reconstructed grayscale images are generated from each image patch and compared with the input grayscale image (i.e., the grayscale image corresponding to the unlabeled robot encryption network stream). The reconstruction error of the masked parts (typically pixel-level mean squared error, MSE) is calculated, backpropagated, and the model parameters are updated. The decoder provides a quantifiable objective for the model's pre-training, forcing the encoder to not only focus on the details of local patches during training but also learn how to integrate these local features into a global structure, possessing global reasoning capabilities and enhancing its performance in tasks involving the discovery of different behavioral patterns.

[0081] This process does not require expensive manual labeling and can utilize a large amount of unlabeled data from real-world scenarios. After pre-training, the encoder can extract high-quality embeddings, which are retained for subsequent tasks. Experiments using this invention show that excellent detection results are achieved even at high mask rates for different types of robot behavior patterns. A high mask rate means that only a few patches are visible, which may result in insufficient raw byte information to capture dependencies between data packets. The good detection performance can be attributed to the encoder acquiring excellent global expressive power during pre-training, enabling it to capture network traffic features with minimal information, even in complex classification tasks.

[0082] To transfer the general features of the pre-trained model to tasks involving mining behavioral patterns of different types of robots and achieve better performance, further training is performed on the pre-trained model to better adapt it to specific traffic patterns. Using a small amount of encrypted network traffic data of different types of robots on labeled social platforms, a robot behavior sequence sample labeled z can be represented as follows:

[0083] In the task of classifying network traffic on social media platforms, directly training the model can take a long time, especially for complex deep learning models. Fine-tuning can significantly accelerate the convergence process, reduce computational resource consumption, and allow the model to adapt to the demands of network traffic classification tasks more quickly. Furthermore, more types of bots will undoubtedly emerge on social media platforms in the future, so it is necessary to improve the model's generalization ability. Fine-tuning reduces the risk of overfitting from training on only a small number of labeled data sets. General features learned during pre-training can help the model maintain good generalization performance even with small sample sizes. To accelerate model convergence, parameters are shared between the two encoders, improving performance by halving the number of parameters. Figure 5 As shown. Dependencies between data packets and between flows are captured during the execution of patches. The information interaction mechanism in multi-head self-attention is similar, which provides conditions for parameter sharing applications. Simply put, compared to the pre-training stage, two flow encoders double the depth of the Transformer layer. The features of each patch are subjected to row pooling and column pooling. The classification features of the robot's behavior sequence flow are input into the fully connected layer to obtain the predicted distribution. calculate Cross-entropy loss between the true label y and the actual label y

[0084] ●Behavioral Pattern Detection

[0085] It's important to note that converting flow characteristics into grayscale images is a simplified representation and may result in some information loss. The grayscale images generated from the original flow characteristics are very large, and during pre-training, they are often cropped to some extent for model efficiency, which may also lead to the loss of important information. Therefore, the potential information loss needs to be considered when converting the robot's original flow characteristics into grayscale images. In robot behavior pattern classification tasks, temporal information plays a crucial role in revealing the relationships between robot behaviors. Therefore, temporal information is incorporated into the original sequence features and used in conjunction with OBTT to model the original flow characteristics.

[0086] Using Tim e —LSTM models the improved original sequence features of different types of robot traffic, capturing long-term sequence dependencies. Specifically, it introduces one or more Timing parameters into the traditional LSTM. e Gate sWhen modeling the time intervals between adjacent actions in a robot's behavior sequence, the focus on different time steps can be dynamically adjusted. The sequence features extracted from the original encrypted traffic are combined with the corresponding time information to form the input of a single-branch network, namely X. i =[a i , Δt i For the first time step, since there is no previous time point, Δt1 can be set to 0 or Δt1 = t1. In Time-LSTM, the calculation for each time step can be expressed as: h t =LSTM([a t , Δt t ],h t-1 ), where h t It is the hidden state at the current time step, h t-1 It is the hidden state of the previous time step, [a t , Δt t [] is the input at the current time step. After processing the entire sequence, the final hidden state h is... n That is, the state of the last time step, which serves as a feature of the sample sequence.

[0087] In this way, the image-based embedding features output by the encoder are concatenated with the improved sequence features output by the time-LSTM module to obtain complete embedding features. These are then input into a fully connected neural network for training. A loss function is calculated to measure the error between the predicted results and the true labels, and the softmax function outputs the final probability distribution. It is the class label predicted by the model, P(y i ) represents the probability of category i, which ultimately completes the label prediction.

[0088] Online detection phase

[0089] The finely tuned OBTT model can be deployed on social media platforms and set up at key routing nodes to detect large amounts of encrypted traffic and identify suspicious bot traffic of different types. By inputting the encrypted network flow to be detected into the finely tuned OBTT model, the bot category corresponding to that encrypted network flow can be obtained.

[0090] Performance Analysis:

[0091] 1) Data collection

[0092] Dataset Preparation. Traffic generated and collected via laboratory mobile devices and routers provides greater flexibility and ease of use for simulation and data acquisition. Crucially, this invention comprehensively simulates the management of numerous mobile devices by a single machine, performing tasks on Twitter and executing custom bot scripts to simulate five types of bots for fully automated operation. These include five types of bots on OSN: opinion bots, reporting bots, chatbots, posting bots, and hybrid bots. By simulating a real network environment, encrypted traffic collected from social media platforms accurately characterizes the behavioral features of various bots. To mitigate ethical dilemmas and concerns related to human subjects, a custom script was used with real user traffic, with execution time and process operations fully randomized. Traffic data from typical users did not show obvious patterns, similar to the traffic of ordinary users on real social networks. This invention collected 50GB of unprocessed bot behavior traffic and 10GB of standard user behavior traffic from Twitter on an experimental platform. A large unlabeled training dataset was created using 70% of the complete dataset for pre-training. During the fine-tuning phase, 20% of the total data was used as the training dataset for supervised learning to complete different behavioral pattern recognition tasks. This invention will disclose the traffic of five types of labeled bots on Twitter.

[0093] 2) Implementation details

[0094] During the pre-training phase, the batch size was 512. The base learning rate was set to the AdamW optimizer. The mask rate for the random mask patch was set to 0.9. Then, this invention used the AdamW optimizer for 200 epochs of fine-tuning, with the base learning rate set to a certain value and the batch size set to 64. Experimental results varied under different mask rates.

[0095] 3) Evaluation Indicators

[0096] To evaluate the classification performance of this method, in addition to macro-average and weighted average, this invention also estimates the number of true positives (Tp), true negatives (Tn), false positives (Fp), and false negatives (Fn). Using the above concepts, this invention can derive recall, precision, and F1 score.

[0097] ■ Comparison with existing methods

[0098] To thoroughly evaluate the proposed method, OBTT is compared to several deep learning baseline techniques, including 2D-CNN (Wang et al. 2017), ResNet34 (He et al. 2015), and FS-Net (Liu et al. 2019). Each technique is listed below. OBTT outperforms all other comparative methods applied to this dataset. Deep learning techniques utilize unprocessed packet data for supervised learning, during which they independently extract byte features from the packets. As shown in Table 1, experimental results demonstrate that deep learning-based methods do not produce satisfactory results. Due to the impact of HTTP / 2, bots on social media platforms can generate a large number of verbose packets. These numerous packets can obscure key information contained in other packets. Pre-trained methods are able to identify potential behavioral patterns of bots and classify them effectively. Furthermore, this method does not rely on large amounts of manually labeled data; only a small amount of labeled data is needed to achieve satisfactory results in subsequent traffic classification tasks. This is thanks to a carefully crafted stream representation, model architecture, and training framework.

[0099] ■ Ablation Experiment

[0100] We conducted an in-depth analysis of the performance advantages of each individual component of OBTT. The Time-LSTM module (OBTT / t) and the Graph Embedding MAE module (OBTT / M) were removed from the system. Figure 6 The results of the evaluation are described. The absence of the time-LSTM module caused the accuracy to drop below 90% at the start of computation. The primary reason for this is that the enhanced sequence model LSTM is capable of obtaining expressive sequence representations by thoroughly extracting regular features present in the behavior of social platform bots. This result demonstrates that when attempting to identify behavioral patterns, the temporal relationship between data packets and network flow must be considered. Secondly, this strategy illustrates the benefits of packet-level attention mechanisms in contrast to the lack of pre-training. Performance may occasionally degrade in the absence of packet-level pre-trained attention, a phenomenon not observed after successful pre-training. Overfitting is more likely to occur when overemphasizing a small amount of data without any prior training.

[0101] Table 1 Comparison of experimental results

[0102]

[0103] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A method for discovering social robots based on multimodal feature fusion of traffic, comprising the following steps: 1) Obtain and label the encrypted network streams of T types of robots to obtain the encrypted traffic dataset of robots on the social platform; where f z Given the behavior sequence sample of the z-th robot, construct an OBTT model, which includes a raw flow to grayscale image module, a raw sequence feature encoding module, an encoder, a decoder, a time-LSTM module, a feature fusion module, and a fully connected neural network; 2) Train the OBTT model using unlabeled robot encrypted network streams: 21) Obtain multiple network flows of the same robot to form the robot's behavior sequence; the original flow to grayscale module converts the robot's behavior sequence into a grayscale image; divide the grayscale image into N image patches and perform masking processing on them; embed the image patch retained after masking processing into a high-dimensional space through a linear projection layer; embed the position of each image patch in the high-dimensional space into the corresponding image patch as the input of the encoder; 22) The encoder calculates the importance of each input image patch relative to all other image patch patches in the grayscale image, and generates a vector representation of the corresponding image patch based on the calculation result; the vector representation of the i-th image patch includes the features of the i-th image patch and its dependencies on other image patch patches; 23) The decoder reconstructs the corresponding image patch based on the vector representation of the image patch, and generates the target image based on each reconstructed image patch; 24) Calculate the reconstruction error based on the target image and the corresponding grayscale image, and update the OBTT model by backpropagation based on the reconstruction error; 3) Fine-tune the OBTT model trained in step 2) using a labeled dataset of encrypted bot traffic from social media platforms: 31) The original flow to grayscale image module converts the behavior sequence samples f z The network flow samples are graphically represented to obtain the behavioral sequence samples f. z The grayscale image; the behavioral sequence sample f z The grayscale image is divided into N image patches and each image patch is embedded into a high-dimensional space through a linear projection layer. The position of each image patch in the high-dimensional space is embedded and added to the corresponding image patch as the input of the two encoders. The two encoders share parameters. 32) The two encoders respectively generate vector representations of each image patch and input them into the feature fusion module; 33) The original sequence feature encoding module encodes the behavioral sequence sample f z Feature extraction is performed to obtain behavioral sequence samples f. z The original sequence features; calculate the behavioral sequence sample f z The time interval between adjacent samples is added to the behavioral sequence sample f. z From the original sequence features, behavioral sequence samples f are obtained. z The feature sequence containing time interval information is input into the time-LSTM module; 34) The time-LSTM module extracts behavioral sequence samples f z The features are then input into the feature fusion module; 35) The feature fusion module provides the vector representation of each image patch and the behavior sequence sample f. z The features are fused to obtain behavioral sequence samples f. z The corresponding embedded features are input into the fully connected neural network for prediction; 36) Fine-tune the OBTT model trained in step 2) based on the error between the prediction results and the corresponding true labels; 4) Input the encrypted network stream to be detected into the OBTT model after fine-tuning in step 3) to obtain the robot category corresponding to the encrypted network stream to be detected.

2. The method according to claim 1, characterized in that, In step 2), the MAE paradigm is used to train the asymmetric encoder-decoder architecture consisting of the encoder and decoder.

3. The method according to claim 1, characterized in that, The time-LSTM module is obtained by introducing one or more TimeGates into the LSTM.

4. The method according to claim 1, 2, or 3, characterized in that, The encoder is composed of alternating multi-head self-attention layers and feedforward layers.

5. A social robot discovery system based on multimodal feature fusion of traffic, characterized in that, It includes an OBTT model, a model training unit, and a detection unit; the OBTT model includes a raw flow to grayscale image module, a raw sequence feature encoding module, an encoder, a decoder, a time-LSTM module, a feature fusion module, and a fully connected neural network; The model training unit is used to train the OBTT model using unlabeled bot encrypted network streams, and to fine-tune the trained OBTT model using a bot encrypted traffic dataset from a social platform; wherein, encrypted network streams of T types of bots are obtained and labeled to obtain the bot encrypted traffic dataset from the social platform; f z This is a sample of the behavior sequence of the z-th robot; The detection unit is used to input the encrypted network stream to be detected into the fine-tuned OBTT model to obtain the robot category corresponding to the encrypted network stream; wherein, the encrypted network streams of T types of robots are obtained and labeled to obtain the robot encrypted traffic dataset on the social platform; f z This is a sample of the behavior sequence of the z-th robot; The method for training the OBTT model using unlabeled robot encrypted network streams is as follows: 21) Obtain multiple network streams of the same robot to form the robot's behavior sequence; the original flow to grayscale module converts the robot's behavior sequence into a grayscale image; divide the grayscale image into N image patches and mask them; embed the masked image patches into a high-dimensional space through a linear projection layer; embed the position of each image patch in the high-dimensional space into the corresponding image patch as the input of the encoder; 22) The encoder calculates the importance of each input image patch relative to all other image patches in the grayscale image, and generates a vector representation of the corresponding image patch based on the calculation result; the vector representation of the i-th image patch includes the features of the i-th image patch and its dependencies with other image patches; 23) The decoder reconstructs the corresponding image patch based on the vector representation of the image patch, and generates a target image based on each reconstructed image patch; 24) Calculate the reconstruction error based on the target image and the corresponding grayscale image, and update the OBTT model by backpropagation based on the reconstruction error; The method for fine-tuning the trained OBTT model using a labeled, encrypted traffic dataset of bots on social platforms is as follows: 31) The original traffic to grayscale image module converts the behavior sequence samples f z The network flow samples are graphically represented to obtain the behavioral sequence samples f. z The grayscale image; the behavioral sequence sample f z The grayscale image is divided into N image patches and each image patch is embedded into a high-dimensional space through a linear projection layer. The position of each image patch in the high-dimensional space is embedded and added to the corresponding image patch as the input of the two encoders. The two encoders share parameters. 32) The two encoders respectively generate vector representations of each image patch and input them into the feature fusion module; 33) The original sequence feature encoding module processes the behavior sequence sample f z Feature extraction is performed to obtain behavioral sequence samples f. z The original sequence features; calculate the behavioral sequence sample f z The time interval between adjacent samples is added to the behavioral sequence sample f. z From the original sequence features, behavioral sequence samples f are obtained. z 34) The time-LSTM module extracts the behavioral sequence samples f containing time interval information. z The features are input into the feature fusion module; 35) The feature fusion module inputs the vector representation of each image patch and the behavior sequence sample f z The features are fused to obtain behavioral sequence samples f. z The corresponding embedded features are input into the fully connected neural network for prediction; 36) The trained OBTT model is fine-tuned based on the error between the prediction result and the corresponding real label.

6. A server, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

Images