Social robot discovery method and system based on flow multi-modal feature fusion

Abstract

The invention discloses a social robot discovery method and system based on flow multi-modal feature fusion. The method comprises the following steps: 1) acquiring encrypted network flows of T types of robots and marking the encrypted network flows to obtain a robot encrypted flow data set on a social platform; wherein fz is a behavior sequence sample of the zth robot; the method comprises the following steps: constructing an OBTT model, wherein the OBTT model comprises an original traffic-to-grey-scale map module, an original sequence feature encoding module, an encoder, a decoder, a time-LSTM module, a feature fusion module and a full-connection neural network; 2) using the unmarked robot encryption network flow to train the OBTT model; 3) performing fine tuning on the trained OBTT model by using the marked robot encrypted traffic data set on the social platform; and 4) inputting a to-be-detected encrypted network flow into the fine-tuned OBTT model to obtain a corresponding robot category. According to the invention, the capability of detecting different robot behavior modes is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of network security technology and relates to a social robot discovery method and system based on traffic multimodal feature fusion. Background Art

[0002] Online social platforms have developed rapidly, with a surge in the number of users and a continuous enrichment of functions. From the initial text communication to today's multimedia interaction, technological progress and the popularization of mobile Internet have further promoted the rapid development of online social platforms, making them an important part of people's daily lives, connecting global users, and promoting information sharing and social interaction. While developing in full swing, robots on social platforms are also mixed. Robots are controlled by software, automatically perform fixed tasks, and imitate the behavior of normal users to be active on the platform. Some robots can provide users with more opinions and knowledge and break the information cocoon; however, some robots will be used by operators as a weapon for malicious network manipulation to carry out cyber violence activities - spreading false and harmful information, damaging the interests of society and even the country, and have strong network destructive power. These robots are collectively referred to as social water army robots. In a data-driven and traffic-dominated environment, social water army robots, as the main means of cyber violence, have gradually become an industrial chain. The forwarding, likes and promotion services of social media are clearly marked with prices, amplifying and spreading topic tags, and then guiding and manipulating users.

[0003] In recent years, social platform services have been widely used by users, and the platform business volume has grown rapidly. Cyber ​​violence and crimes have followed, destroying the network ecology and causing adverse effects. Robots on social platforms may be used by operators for cyber violence, financial crimes and other activities. Therefore, it is urgent to propose an effective social water army robot detection solution. Different types of robots and ordinary users have different usage purposes and behavioral preferences, so they will show different behavior patterns. Mining network behavior patterns can discover the essential laws and anomalies of robots, and provide strong support for perceiving their operating status and collecting evidence and monitoring their illegal and criminal activities.

[0004] The detection methods of robots on social platforms have strict requirements on data sources (user information, social relationships, activity history), focusing on mining users' private content data or social relationship structure, which is easy to infringe on user privacy. With the application of social platforms to combat malicious robot technology, the anti-resistance ability of operators is enhanced, and large-scale robots can be generated in a short period of time, producing highly simulated behaviors, forming complex and dense social network structures, and confusing the platform's detection of robots. Traditional network traffic analysis methods usually identify different network services based on features such as communication protocols and port numbers, but due to the complexity and variability of traffic on current social platforms, these features are no longer applicable. To solve such problems, some studies rely on expert experience to classify through statistical features and use machine learning methods, which is inefficient; some studies use more effective deep learning methods to automatically extract network flow features, but in the face of different encryption protocols, the length of data packets may fluctuate significantly, resulting in uneven information distribution, and key features in shorter data packets are difficult to capture. In addition, model training requires a large amount of labeled data for training, which is costly.

[0005] Detection methods based on text analysis aim to detect robot-related behaviors, accounts or account clusters using text information on social platforms. This text information may include explicit information of the account, such as profile information, published content, interactive information, profiles, etc., as well as implicit information, such as the account's content pattern, emotional language expression, etc. Through this text information, the user behavior portrait is constructed using text analysis semantics and emotional features to model the differences between robots and human users. This type of method has good accuracy and interpretability by capturing text features, but has the following disadvantages:

[0006] (1) Complexity of natural language. Users on social platforms have different language and cultural backgrounds and use different grammatical structures, dialects, and slang.

[0007] (2) Involving content data that is sensitive to user privacy, in-depth analysis of user-generated content may raise legal and ethical issues and is vulnerable to adversarial attacks.

[0008] (3) The model is not adaptable to the ambiguity and variability of the text, and may need to be updated frequently to maintain its effectiveness.

[0009] Detection methods based on social graph structures usually use robot accounts on online social platforms as nodes, establish connections between themselves and human users, and identify robots based on the structural gap between robot accounts and real users by analyzing the connection patterns between accounts in the social network graph rather than relying too much on text content. Such methods are not susceptible to content-oriented adversarial attacks. However, there are still some disadvantages:

[0010] (1) It can only be effectively detected when the structure of the social network changes. If the robot frequently posts, browses, etc. without interacting with other accounts, it cannot be detected.

[0011] (2) Account-level detection can only determine whether the account is a real user or a robot.

[0012] Based on network traffic detection methods. As the network environment becomes increasingly complex, ML algorithms are introduced to explore the high-dimensional statistical features of traffic to analyze complex network traffic. DL, on the other hand, analyzes traffic based on raw data packets rather than manually designed features. However, some existing DL-based methods directly use the bytes of data packets in the stream to represent the raw traffic, resulting in the bytes of long data packets overwhelming the important information in other data packets and requiring a large amount of labeled data. In addition, the current detection methods for fine-grained behaviors on social platforms are usually divided based on network flows, but in the HTTP / 2 protocol, all communications under the same domain name are completed on a single connection, which can carry any number of bidirectional data streams. It introduces HTTP header field compression and supports multiple concurrent data exchanges on a single HTTP connection. With the new feature of reset messages, clients are allowed to stop ongoing operations and restart without disconnecting and resetting HTTP connections. Flow control and priority allow multiplexing of data streams. As a result, the true label of the data stream cannot be determined, and the method of using data streams as labeled samples is not feasible. Summary of the invention

[0013] In view of the problems existing in the prior art, the purpose of the present invention is to provide a social robot discovery method and system based on the fusion of multimodal features of traffic. The present invention uses large-scale unlabeled data for pre-training, takes "different behavior patterns" as the foothold, that is, focuses on a series of actions of the robot within a time period, and uses a small amount of labeled data to fine-tune the model to achieve the purpose of detecting different behavior patterns. The traditional robot detection method based on user profiles, content or social relationships in the present invention has the risk of infringing user privacy. The OBTT (OSN Bots Traffic Transformer) learning framework is proposed, which has significant advantages in identifying the traffic of robots on social platforms and mining different types of robot behavior patterns. OBTT adopts a multimodal method to fuse the graph embedding representation of the original traffic with the sequence features, and introduces time information at the same time to explore the regularity of the robot action sequence. Use large-scale unlabeled data to fine-tune the pre-trained MAE and a small amount of labeled data to enhance the model's detection ability for different robot behavior patterns.

[0014] Attached Figure 2 As shown in the figure, OBTT uses the original traffic to grayscale image module, the original sequence feature encoding module, the encoder, the decoder, the time-LSTM module, the feature fusion module and the fully connected neural network.

[0015] 1) Raw traffic to grayscale image module. The raw traffic is graphically represented to generate a high-dimensional visual embedding representation, such as Figure 1 As shown;

[0016] 2) Use the masked autoencoder self-supervised learning model as the encoder to learn effective representations from unlabeled images. Perform random masking on the image, reconstruct the masked patches through the decoder, and use the reconstruction error to optimize the model parameters until the model converges.

[0017] 3) Original sequence feature encoding module. Time information is introduced based on the original sequence of encrypted traffic of different types of robots. In the robot behavior sequence classification task, time information such as time interval and relative time plays a key role in mining robot behavior patterns. Sequence data features are captured based on time-LSTM.

[0018] 4) Through fine-tuning, the purpose of detecting the behavior patterns of different types of robots is achieved. During the training process, the MAE weights are adjusted to monitor the model performance and prevent overfitting.

[0019] 5) Learn the correlation between images and texts through interactive attention mechanisms, model the relationship between visual and text features, and perform feature fusion. Build a robot classification model, quantify the similarity of behavior combinations, and complete the recognition of different types of robot behavior patterns.

[0020] OBTT is the first to detect robot traffic on online social platforms from the perspective of network traffic, and can also identify robot traffic as coming from a specific type of robot by analyzing the behavior patterns of different types of robots. Evaluated on the BotTraffic6 dataset, OBTT can identify social robot traffic with an accuracy of 99%. At the same time, the accuracy of detecting and identifying robot traffic as different types of robots is 95%.

[0021] The main contribution of OBTT is to use data from the network layer to detect anomalies on social platforms. However, OBTT has some limitations:

[0022] OBTT focuses on detecting behavioral patterns of different categories of social bots and has shown excellent performance in identifying common types of bots on social platforms. However, if the bot's behavior is very similar to that of ordinary users and does not show recognizable patterns, the detection results will be poor. OBTT's detection scope does not include specific social bots, such as zero-day social bots.

[0023] The grayscale image generated based on the original traffic features is very large. In order to consider the model efficiency during pre-training, the grayscale image is cropped to a certain extent, which may also lead to the loss of some important information. Therefore, when converting the grayscale image of the robot's original traffic, it is necessary to consider the possible information loss. In the robot behavior pattern classification task, time information plays a key role in revealing the relationship between robot behaviors. In view of this, the present invention integrates time information into the original sequence features, and jointly models the original traffic features with OBTT.

[0024] The technical solution of the present invention is:

[0025] A social robot discovery method based on traffic multimodal feature fusion, the steps of which include:

[0026] 1) Obtain the encrypted network flows of T types of robots and mark them to obtain the encrypted traffic dataset of robots on the social platform; where f z is the behavior sequence sample of the zth robot; construct an OBTT model, which includes an original traffic conversion grayscale image module, an original sequence feature encoding module, an encoder, a decoder, a time-LSTM module, a feature fusion module and a fully connected neural network;

[0027] 2) Using unlabeled robot encrypted network flows to train the OBTT model:

[0028] 21) obtaining multiple network flows of the same robot to form a behavior sequence of the robot; the original flow conversion grayscale image module converts the behavior sequence of the robot into a grayscale image; dividing the grayscale image into N image blocks and performing mask processing on them, embedding the image blocks retained after the mask processing into a high-dimensional space through a linear projection layer, and embedding the position of each image block in the high-dimensional space into the corresponding image block patch as the input of the encoder;

[0029] 22) the encoder calculates the importance of each input image block patch relative to all other image block patches in the grayscale image, and generates a vector representation of the corresponding image block patch according to the calculation result; the vector representation of the i-th image block patch includes the features of the i-th image block patch and its dependencies with other image block patches;

[0030] 23) The decoder reconstructs the corresponding image block according to the vector representation of the image block patch, and generates a target image according to each reconstructed image block;

[0031] 24) calculating a reconstruction error based on the target image and the corresponding grayscale image, and updating the OBTT model by back propagation according to the reconstruction error;

[0032] 3) Use the labeled encrypted traffic dataset of robots on social platforms to fine-tune the OBTT model trained in step 2):

[0033] 31) The original traffic conversion grayscale image module converts the behavior sequence sample f z The network flow samples in are graphically expressed to obtain the behavior sequence samples f z Grayscale image; the behavior sequence sample f z The grayscale image is divided into N image patches and each image patch is embedded into a high-dimensional space through a linear projection layer, and the position of each image patch in the high-dimensional space is embedded and added to the corresponding image patch as the input of the two encoders; the parameters are shared between the two encoders;

[0034] 32) The two encoders respectively generate vector representations of each image patch and input them into the feature fusion module;

[0035] 33) The original sequence feature encoding module encodes the behavior sequence sample f z Perform feature extraction to obtain behavior sequence samples f z The original sequence features; calculate the behavior sequence sample f z The time interval between adjacent samples in the behavior sequence sample f is added to the behavior sequence sample f z From the original sequence features, we get the behavior sequence sample f z The feature sequence containing time interval information is input into the time-LSTM module;

[0036] 34) The time-LSTM module extracts the behavior sequence sample f z The features are input into the feature fusion module;

[0037] 35) The feature fusion module represents the vector representation of each image patch and the behavior sequence sample f z The features of z The corresponding embedded features are input into the fully connected neural network for prediction;

[0038] 36) fine-tuning the OBTT model trained in step 2) according to the error between the predicted result and the corresponding true label;

[0039] 4) Input the encrypted network flow to be detected into the OBTT model fine-tuned in step 3) to obtain the robot category corresponding to the encrypted network flow to be detected.

[0040] Furthermore, in step 2), the MAE paradigm is used to train the asymmetric encoder-decoder architecture composed of the encoder and the decoder.

[0041] Furthermore, the time-LSTM module is obtained by introducing one or more Time Gates into the LSTM.

[0042] Furthermore, the encoder is composed of alternating multi-head self-attention layers and feed-forward layers.

[0043] A social robot discovery system based on traffic multimodal feature fusion, characterized in that it includes an OBTT model, a model training unit and a detection unit; the OBTT model includes an original traffic conversion grayscale image module, an original sequence feature encoding module, an encoder, a decoder, a time-LSTM module, a feature fusion module and a fully connected neural network;

[0044] The model training unit is used to train the OBTT model using unlabeled robot encrypted network flows, and fine-tune the trained OBTT model using a robot encrypted traffic dataset on a social platform; wherein encrypted network flows of T types of robots are obtained and labeled to obtain the robot encrypted traffic dataset on the social platform; z is the behavior sequence sample of the zth robot;

[0045] The detection unit is used to input the encrypted network flow to be detected into the fine-tuned OBTT model to obtain the robot category corresponding to the encrypted network flow to be detected; wherein, the encrypted network flows of T types of robots are obtained and marked to obtain the robot encrypted traffic data set on the social platform; z is the behavior sequence sample of the zth robot;

[0046] The method for training the OBTT model using unlabeled robot encrypted network flow is as follows: 21) obtaining multiple network flows of the same robot to form the behavior sequence of the robot; the original flow to grayscale image module converts the robot's behavior sequence into a grayscale image; dividing the grayscale image into N image block patches and performing mask processing on them, embedding the image block patches retained after mask processing into a high-dimensional space through a linear projection layer, and embedding the position of each image block in the high-dimensional space into the corresponding image block patch as the input of the encoder; 22) the encoder calculates the importance of each input image block patch relative to all other image block patches in the grayscale image, and generates a vector representation of the corresponding image block patch based on the calculation result; the vector representation of the i-th image block patch includes the features of the i-th image block patch and its dependency with other image block patches; 23) the decoder reconstructs the corresponding image block based on the vector representation of the image block patch, and generates a target image based on each reconstructed image block; 24) the reconstruction error is calculated based on the target image and the corresponding grayscale image, and the OBTT model is updated by back propagation based on the reconstruction error;

[0047] The method for fine-tuning the trained OBTT model using the labeled encrypted traffic dataset of robots on social platforms is as follows: 31) The raw traffic conversion grayscale image module converts the behavior sequence sample f z The network flow samples in are graphically expressed to obtain the behavior sequence samples f z Grayscale image; the behavior sequence sample f z The grayscale image is divided into N image patches and each image patch is embedded into a high-dimensional space through a linear projection layer, and the position of each image patch in the high-dimensional space is embedded and added to the corresponding image patch as the input of the two encoders; the parameters are shared between the two encoders; 32) the two encoders respectively generate vector representations of each image patch and input them into the feature fusion module; 33) the original sequence feature encoding module encodes the behavior sequence sample f z Perform feature extraction to obtain behavior sequence samples f z The original sequence features; calculate the behavior sequence sample f z The time interval between adjacent samples in the behavior sequence sample f is added to the behavior sequence sample f z From the original sequence features, we get the behavior sequence sample f z The feature sequence containing time interval information is input into the time-LSTM module; 34) the time-LSTM module extracts the behavior sequence sample f zThe features are input into the feature fusion module; 35) the feature fusion module vector representation of each image block patch, behavior sequence sample f z The features of z The corresponding embedded features are input into the fully connected neural network for prediction; 36) the trained OBTT model is fine-tuned according to the error between the prediction result and the corresponding true label.

[0048] A server, characterized in that it comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the above method.

[0049] A computer-readable storage medium stores a computer program, wherein the computer program implements the above method when executed by a processor.

[0050] The advantages of the present invention are as follows:

[0051] OBTT does not use data involving user privacy from the payload of the data packet and is not interfered by the content. Instead, it focuses on the mining of robot behavior patterns. First, the encrypted traffic data of different types of robots on the social platform is graphically represented to more intuitively display and analyze the network behavior characteristics of various types of robots. Then, a masked autoencoder based on the MAE architecture is used to pre-train on large-scale unlabeled encrypted traffic data of robots on social platforms, and fine-tune on a small labeled data set. In addition, based on the original sequence of the robot's encrypted traffic, time information is introduced to further explore its potential patterns and laws, and classify the robot's behavior patterns. Through the joint training of multiple tasks, the performance of the model in multimodal tasks is improved, the classification effect and retrieval performance of the embedding space are improved, the demand for large-scale labeled data is reduced, and the generalization ability of the model is improved, which effectively reduces the computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is the process of converting the original traffic into a grayscale image.

[0053] Figure 2 It is a flow chart of the training method of the present invention.

[0054] Figure 3 Grayscale images after traffic conversion for different types of robots;

[0055] (a) The grayscale image corresponding to the broadcasting robot, (b) the grayscale image corresponding to the opinion robot, (c) the grayscale image corresponding to the chat robot, (d) the grayscale image corresponding to the hybrid robot, and (e) the grayscale image corresponding to the publishing robot.

[0056] Figure 4 Statistical distribution diagram of the improved inter-arrival time sequence characteristics.

[0057] Figure 5 is the model convergence curve.

[0058] Figure 6 The ablation experiment results. DETAILED DESCRIPTION

[0059] The present invention is further described in detail below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention but not to limit the scope of the present invention.

[0060] In addition to benign robots that do not imitate humans, there are also malicious robots that try to appear to be manually operated and confuse the audience. Network manipulators often use these malicious robots to produce a series of highly simulated behaviors to spread false and harmful information and other cyber violence activities to destroy the network ecology. Through research, the common robots on social platforms are roughly divided into five categories: automatic publishing robots, broadcasting robots, public opinion robots, chat robots, and hybrid robots. The behavioral patterns of different types of robots are explored to provide methods to help detection.

[0061] Fine-grained action classification on social platforms refers to the process of dividing all possible actions in a specific application into several fixed categories. Strictly speaking, each action performed by a user is unique, that is, actions of different categories are independent of each other, and it is impossible to perform two actions at the same time. On social platforms, a series of actions are often performed, and behaviors are composed of several actions arranged in chronological order. The performance in traffic is that the traffic collected at one time contains data of multiple actions. In addition, in the http / 2 protocol, all actions may be completed on a single connection, and multiple frames are not sent in order.

[0062] Obviously, it is more accurate and more practical to take behavior sequence as the research object. The present invention mainly includes three stages:

[0063] The first stage is the preprocessing stage.

[0064] The second stage is the pre-training and fine-tuning stage.

[0065] The third stage is the behavior pattern detection stage.

[0066] In the present invention, the encrypted traffic data set of robots on the social platform includes N samples (flows) and T common types of robots, where T=5. The behavior sequence sample of the zth robot includes n z samples (network flows), which can be expressed as: Where nz Yes z The characteristic length of is the nth z samples (network flow), b i represents the i-th network flow sample. Each network flow sample includes multiple data packets and can be expressed as Assume that the jth data packet satisfies p in bytes j ∈[0, 1500), is the Rth packet in the i-th network flow of the z-th robot, f z The label is L z , 1≤L z ≤T. The training method of OBTT of the present invention is as follows Figure 2 As shown, the three stages are described in detail below.

[0067] ●Preprocessing - Multimodal Stream Representation

[0068] In the encrypted traffic classification task of different robots on social platforms, given a sequence sample f z In order to automate the feature extraction process, the present invention graphically expresses the original traffic and extracts the original sequence features without using the statistical information of the encrypted traffic.

[0069] Graphical expression of raw traffic. The results of the experiment in (Wang et al., 2017) show that compared with extracting data from the application layer of the TCP / IP model, the feature information extracted from all layers is more representative. The network traffic generated by different types of robots on real social platforms is simulated, and the raw traffic is divided according to the same five-tuple to obtain the feature information of upstream, downstream, and bidirectional flows. The present invention anonymizes the raw traffic features, specifically converting the IP address into a random address and setting the port number to 0, thereby introducing a bias to interfere with the training. In addition, no other screening is performed to ensure the authenticity of the traffic as much as possible. The anonymized raw traffic features are converted into a grayscale image, which can utilize existing image processing technology and deep learning models to improve the analysis ability of complex traffic patterns. For the entire data set, F = {f1,…,f z},like Figure 3 As shown, the traffic of different types of robots on the social platform in F is converted into grayscale images to visualize the differences between them.

[0070] Original sequence feature extraction. In the task of mining robot behavior patterns, the regularity of actions is a key influencing factor. Time information, such as time intervals and relative time, is introduced based on the original sequences of encrypted traffic of different types of robots. Since different actions (such as comments, likes, posts, etc.) need to interact with different components, the response time of different actions to the front end is different. R ) of the inter-arrival time sequence PTS = {s1,…,s n-1} is converted to the improved time interval G_PTS ​​= {a 1, …,a n-1}, where s j =t j+1 -t j ,s j =Δt·b+Δt / 2, where b satisfies the condition Δt·b≤s j <Δt·(b+1), after testing, select Δt=0.005s. Figure 4 As shown in the figure, the statistical distribution of the improved arrival time interval sequence characteristics is shown through a box plot. Among them, the chat robot and the broadcast robot have long-tail distribution characteristics, and the interquartile range of the publishing robot and the opinion robot is small, indicating that the time intervals are more concentrated and the actions are more regular. The hybrid robot may have a high degree of discreteness of the time interval due to the complexity of its behavior pattern and the lack of obvious regularity between actions, showing a longer time interval distribution. Therefore, the improved arrival time interval is a discriminative feature.

[0071] In this way, the content-agnostic raw robot traffic is converted into grayscale images and improved raw sequences, providing a data basis for model pre-training and fine-tuning.

[0072] ●OBTT pre-training and fine-tuning

[0073] According to the characteristics of the traffic, the grayscale image is masked, and the encoder module and the decoder module work together to enable OBTT to perform effective self-supervised learning to achieve the purpose of grayscale image classification.

[0074] The input grayscale image needs to go through a series of processing so that the model can learn its global representation by observing local image features. The grayscale image is resized to 100*100 and divided into a series of X patches of fixed size N*N, similar to cutting an image into 2D small blocks. Specifically, the present invention takes N=20, with a total of 20*20=400 image patches.

[0075] Before entering the encoder, a portion of the patches are randomly masked, and these patches will not be fed into the encoder, leaving only a small number of visible patches. This is similar to BERT's strategy of masking part of the vocabulary in natural language processing tasks, with the goal of training the model to predict these missing patches. Each unmasked image patch is embedded into a high-dimensional space through a linear projection layer, and the position embedding is added to the image patch as the input of the encoder to retain the position information in the image space. Position encoding is very important.

[0076] The present invention adopts the MAE (He et al 2022) paradigm and an asymmetric encoder-decoder architecture. The encoder consists of alternating multi-head self-attention layers and feedforward layers, which allows learning the dependencies between header patches and payload patches of different data packets, that is, calculating the importance of each patch relative to all other patches. The input vector shape can be expressed as [N, D], where N is the number of input patches and D is the dimension after each patch is embedded. [N, D] will be weighted by attention through three linear transformations:

[0077] Q=X·W Q , K = X·W k , V = X·W v

[0078]

[0079] Where W Q , W k , W v is a learnable weight matrix. The new representation of each patch after global context weighting is the output of the encoder. These representations combine the global information of the input image, so that the model not only understands the features of local patches, but also captures the relationship between them. In the task of mining different behavior patterns, the pre-trained encoder parameters are used to extract data packets and network flow features respectively. The value vector V is weighted and summed using the calculated attention weights to generate a new context vector, which represents the representation of each patch after comprehensive consideration of other patch information. In this work, the present invention uses n = 16 parallel attention heads and L = 4 alternating layers.

[0080] The encoder aims to promote information interaction between data packets and does not pay attention to the features of masked patches. The decoder is responsible for "filling in the gaps", reconstructing the original bytes, and restoring the masked data packet features. Through the reconstruction task of the decoder, the model learns how to infer the global structure through the unmasked part based on learning the local patch information. A reconstructed grayscale image is generated based on each reconstructed image block, and then compared with the input grayscale image (that is, the grayscale image corresponding to the unlabeled robot encrypted network flow), and the reconstruction error of the masked part is calculated (usually the mean square error, MSE, at the pixel level), and the model parameters are back-propagated and updated. The decoder provides a quantifiable goal for the pre-training of the model, forcing the encoder to not only pay attention to the detailed information of the local patch during the training process, but also learn how to integrate these local features into the global structure, have global reasoning capabilities, and enhance its performance in mining different behavior pattern tasks.

[0081] This process does not require expensive manual labeling and can use a large amount of unlabeled data from real scenes. After pre-training, the encoder can extract high-quality embeddings and retain them for subsequent tasks. Through experiments of the present invention, it can be found that even under high masking rates, the detection of different types of robot behavior patterns can achieve excellent results. A high masking rate means that only a few patches are visible, which may lead to a lack of sufficient raw byte information to capture the dependencies between data packets. Achieving better detection results can be attributed to the excellent global expression ability gained by the encoder during the pre-training process. Despite the complex classification task, it is still able to capture network traffic features with minimal information.

[0082] In order to transfer the general features of the pre-trained model to the task of mining different types of robot behavior patterns and obtain better performance, further training based on the pre-trained model makes the model better adapt to specific traffic patterns. Using a small amount of encrypted network traffic data of different types of robots on social platforms with labels, the robot behavior sequence sample labeled z can be expressed as

[0083] In the task of classifying robot network traffic on social platforms, directly training the model may require a long training time, especially for complex deep learning models. Fine-tuning can significantly accelerate the convergence process of the model, reduce the consumption of computing resources, and enable the model to adapt to the needs of network traffic classification tasks more quickly. In addition, more types of robots will definitely appear on social platforms in the future, so it is necessary to improve the generalization ability of the model. Fine-tuning can reduce the risk of overfitting caused by training only on small-scale labeled data. The general features learned in the pre-training process can help the model maintain good generalization performance in small sample conditions. In order to speed up the convergence of the model, parameters are shared between the two encoders, which improves performance while reducing the number of parameters by half, such as Figure 5 As shown. The dependencies between data packets and flows are all captured by executing the relationship between patches. The information interaction mechanism in multi-head self-attention is similar, which provides conditions for the application of parameter sharing. Simply put, compared with the pre-training stage, the two flow encoders double the depth of the Transformer layer. The features of each patch are averaged and pooled (Row Pooling) and column averaged (ColumnPooling), and the classification features of the robot behavior sequence flow are input into the fully connected layer to obtain the predicted distribution. calculate The cross entropy loss between the actual label y

[0084] ●Behavior pattern detection

[0085] It should be noted that converting traffic features into grayscale images is a simplified representation method, which may cause some information loss. The grayscale image generated based on the original traffic features is very large. In order to consider the model efficiency during pre-training, the grayscale image is cropped to a certain extent, which may also lead to the loss of some important information. Therefore, when converting the grayscale image of the robot's original traffic, it is necessary to consider the possible information loss. In the robot behavior pattern classification task, time information plays a key role in revealing the relationship between robot behaviors. To this end, the time information is integrated into the original sequence features, and the original traffic features are modeled together with OBTT.

[0086] Using Tim e —LSTM models the improved original sequence features of different types of robot traffic and captures long-term sequence dependencies. Specifically, one or more Tims are introduced into the traditional LSTM. e Gate s, which can dynamically adjust the attention paid to different time steps when modeling the time intervals between adjacent actions in the robot behavior sequence. The sequence features extracted from the original encrypted traffic and the corresponding time information are combined to form the input of a single network, namely X i =[a i , Δt i ]. For the first time step, since there is no previous time point, we can set Δt1 = 0 or Δt1 = t1. In Time-LSTM, the calculation of each time step can be expressed as: h t =LSTM([a t , Δt t ],h t-1 ). Among them, h t is the hidden state of the current time step, h t-1 is the hidden state of the previous time step, [a t , Δt t ] is the input of the current time step. After processing the entire sequence, the final hidden state h n , that is, the state of the last time step, as the feature of the sample sequence.

[0087] In this way, the encoder output is the image embedding feature, which is spliced ​​with the improved sequence feature output by the time-LSTM module to obtain the complete embedding feature, which is input into the fully connected neural network for training. The error between the predicted result and the true label is measured by calculating the loss function, and the softmax output is the final probability distribution. is the class label predicted by the model, P(y i ) is the probability of category i, and finally the label prediction is completed.

[0088] Online detection stage

[0089] The fine-tuned OBTT model is deployed on the social platform side and can be set up at key routing nodes to detect a large amount of encrypted traffic and detect suspicious robot traffic of different types of robots. By inputting the encrypted network flow to be detected into the fine-tuned OBTT model, the robot category corresponding to the encrypted network flow to be detected can be obtained.

[0090] Performance Analysis:

[0091] 1) Data Collection

[0092] Preparation of data sets. Traffic generated and collected by mobile devices and routers in the laboratory provides greater flexibility and ease of use for simulation and data collection. Crucially, the present invention comprehensively simulates the management of many mobile devices by a single machine, performs tasks on Twitter and executes custom robot scripts to simulate five types of robots to achieve fully automated operations. Including 5 types of robots on OSN: opinion robots, reporting robots, chat robots, publishing robots and hybrid robots. By simulating the real network environment, the encrypted traffic collected from the social media platform can accurately characterize the behavioral characteristics of various types of zombies. In order to alleviate the ethical dilemmas and concerns related to human subjects, a custom script is used for real user traffic, completely randomizing execution time and process operations. The traffic data of typical users does not show obvious patterns, similar to the traffic of ordinary users on actual social networks. The present invention collects 50GB of unprocessed bot behavior traffic and 10GB of standard user behavior traffic from Twitter on the experimental platform. A large number of unlabeled training datasets are created using 70% of the complete dataset for pre-training. In the fine-tuning stage, 20% of the total data is used as a training dataset for supervised learning to complete different behavioral pattern recognition tasks. The present invention will disclose the traffic of five types of labeled robots on Twitter.

[0093] 2) Implementation details

[0094] In the pre-training stage, the batch size is 512. The basic learning rate is set to AdamW optimizer. The mask rate of the random mask patch is set to 0.9. Then, the present invention uses AdamWoptimizer for 200 epochs of fine-tuning, where the basic learning rate is set to and the batch size is 64. The experimental results under different mask rates are different.

[0095] 3) Evaluation indicators

[0096] In order to evaluate the classification performance of the method, in addition to the macro average and weighted average, the present invention also estimates the number of true positives (Tp), true negatives (Tn), false positives (Fp) and false negatives (Fn). Using the above concepts, the present invention can derive the recall rate, precision rate and F1 score.

[0097] ■Comparison with existing methods

[0098] To thoroughly evaluate the proposed method, OBTT is compared with a number of deep learning baseline techniques, as follows: 2D-CNN (Wang et al. 2017), ResNet34 (He et al. 2015), and FS-Net (Liu et al. 2019). Each of the techniques is listed below. OBTT outperforms all other compared methods applied to this dataset. Deep learning techniques utilize unprocessed packet data for supervised learning, during which they independently extract byte features from packets. As shown in Table 1, the experimental results show that deep learning-based methods do not produce satisfactory results. Due to the influence of HTTP / 2, bots on social media platforms may generate a large number of lengthy packets. These large number of packets may obscure the critical information contained in other packets. The pre-training method is able to identify the potential behavior patterns of bots and classify them in an effective manner. In addition, the method does not rely on a large amount of manually labeled data; only a small amount of labeled data is required to achieve satisfactory results in the subsequent traffic classification task. This can be attributed to the carefully crafted flow representation, model architecture, and training framework.

[0099] ■Ablation experiment

[0100] In-depth analysis of the performance advantages of each individual component of OBTT. The Time-LSTM module (OBTT / t) and the Graph Embedding MAE module (OBTT / M) are removed from the system separately. Figure 6 The results of the evaluations performed are described. The absence of the time-LSTM module causes the accuracy to drop below 90% at the beginning of the computation. The main reason for this is that the enhanced sequence model LSTM is able to obtain expressive sequence representations by thoroughly extracting the regular features present in the behavior of social platform robots. This result shows that learning the temporal relationship between packets and network flows must be considered when trying to identify behavioral patterns. Second, this strategy exemplifies the benefits of the packet-level attention mechanism, which is in contrast to the lack of pre-training. In the absence of packet-level pre-trained attention, the performance may occasionally deteriorate, a phenomenon that is not observed after successful pre-training. Overfitting is more likely to occur when too much emphasis is placed on a small amount of data without any prior training.

[0101] Table 1 Comparative experimental results

[0102]

[0103] Although the specific embodiments of the present invention are disclosed for the purpose of illustration, the purpose is to help understand the content of the present invention and implement it accordingly, those skilled in the art will understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the best embodiment, and the scope of the present invention is subject to the scope defined in the claims.

Claims

1. A social robot discovery method based on traffic multimodal feature fusion, the steps of which include: 1) Obtain the encrypted network flows of T types of robots and mark them to obtain the encrypted traffic dataset of robots on the social platform; where f z is the behavior sequence sample of the zth robot; construct an OBTT model, which includes an original traffic conversion grayscale image module, an original sequence feature encoding module, an encoder, a decoder, a time-LSTM module, a feature fusion module and a fully connected neural network; 2) Using unlabeled robot encrypted network flows to train the OBTT model: 21) obtaining multiple network flows of the same robot to form a behavior sequence of the robot; the original flow conversion grayscale image module converts the behavior sequence of the robot into a grayscale image; dividing the grayscale image into N image blocks and performing mask processing on them, embedding the image blocks retained after the mask processing into a high-dimensional space through a linear projection layer, and embedding the position of each image block in the high-dimensional space into the corresponding image block patch as the input of the encoder; 22) the encoder calculates the importance of each input image block patch relative to all other image block patches in the grayscale image, and generates a vector representation of the corresponding image block patch according to the calculation result; the vector representation of the i-th image block patch includes the features of the i-th image block patch and its dependencies with other image block patches; 23) The decoder reconstructs the corresponding image block according to the vector representation of the image block patch, and generates a target image according to each reconstructed image block; 24) calculating a reconstruction error based on the target image and the corresponding grayscale image, and updating the OBTT model by back propagation according to the reconstruction error; 3) Use the labeled encrypted traffic dataset of robots on social platforms to fine-tune the OBTT model trained in step 2): 31) The original traffic conversion grayscale image module converts the behavior sequence sample f z The network flow samples in are graphically expressed to obtain the behavior sequence samples f z Grayscale image; the behavior sequence sample f z The grayscale image is divided into N image patches and each image patch is embedded into a high-dimensional space through a linear projection layer, and the position of each image patch in the high-dimensional space is embedded and added to the corresponding image patch as the input of the two encoders; the parameters are shared between the two encoders; 32) The two encoders respectively generate vector representations of each image patch and input them into the feature fusion module; 33) The original sequence feature encoding module encodes the behavior sequence sample f z Perform feature extraction to obtain behavior sequence samples f z The original sequence features; calculate the behavior sequence sample f z The time interval between adjacent samples in the behavior sequence sample f is added to the behavior sequence sample f z From the original sequence features, we get the behavior sequence sample f z The feature sequence containing time interval information is input into the time-LSTM module; 34) The time-LSTM module extracts the behavior sequence sample f z The features are input into the feature fusion module; 35) The feature fusion module represents the vector representation of each image patch and the behavior sequence sample f z The features of z The corresponding embedded features are input into the fully connected neural network for prediction; 36) fine-tuning the OBTT model trained in step 2) according to the error between the predicted result and the corresponding true label; 4) Input the encrypted network flow to be detected into the OBTT model fine-tuned in step 3) to obtain the robot category corresponding to the encrypted network flow to be detected.

2. The method according to claim 1, characterized in that In step 2), the MAE paradigm is used to train the asymmetric encoder-decoder architecture composed of the encoder and decoder.

3. The method according to claim 1, characterized in that The time-LSTM module is obtained by introducing one or more TimeGates into LSTM.

4. The method according to claim 1, 2 or 3, characterized in that: The encoder consists of alternating multi-head self-attention layers and feed-forward layers.

5. A social robot discovery system based on traffic multimodal feature fusion, characterized in that: It includes an OBTT model, a model training unit and a detection unit; the OBTT model includes an original traffic conversion grayscale image module, an original sequence feature encoding module, an encoder, a decoder, a time-LSTM module, a feature fusion module and a fully connected neural network; The model training unit is used to train the OBTT model using unlabeled robot encrypted network flows, and fine-tune the trained OBTT model using a robot encrypted traffic dataset on a social platform; wherein encrypted network flows of T types of robots are obtained and labeled to obtain the robot encrypted traffic dataset on the social platform; z is the behavior sequence sample of the zth robot; The detection unit is used to input the encrypted network flow to be detected into the fine-tuned OBTT model to obtain the robot category corresponding to the encrypted network flow to be detected; wherein, the encrypted network flows of T types of robots are obtained and marked to obtain the robot encrypted traffic data set on the social platform; z is the behavior sequence sample of the zth robot; The method for training the OBTT model using unlabeled robot encrypted network flow is as follows: 21) obtaining multiple network flows of the same robot to form the behavior sequence of the robot; the original flow to grayscale image module converts the robot's behavior sequence into a grayscale image; dividing the grayscale image into N image block patches and performing mask processing on them, embedding the image block patches retained after mask processing into a high-dimensional space through a linear projection layer, and embedding the position of each image block in the high-dimensional space into the corresponding image block patch as the input of the encoder; 22) the encoder calculates the importance of each input image block patch relative to all other image block patches in the grayscale image, and generates a vector representation of the corresponding image block patch based on the calculation result; the vector representation of the i-th image block patch includes the features of the i-th image block patch and its dependency with other image block patches; 23) the decoder reconstructs the corresponding image block based on the vector representation of the image block patch, and generates a target image based on each reconstructed image block; 24) the reconstruction error is calculated based on the target image and the corresponding grayscale image, and the OBTT model is updated by back propagation based on the reconstruction error; The method for fine-tuning the trained OBTT model using the labeled encrypted traffic dataset of robots on social platforms is as follows: 31) The raw traffic conversion grayscale image module converts the behavior sequence sample f z The network flow samples in are graphically expressed to obtain the behavior sequence samples f z Grayscale image; the behavior sequence sample f z The grayscale image is divided into N image patches and each image patch is embedded into a high-dimensional space through a linear projection layer, and the position of each image patch in the high-dimensional space is embedded and added to the corresponding image patch as the input of the two encoders; the parameters are shared between the two encoders; 32) The two encoders respectively generate vector representations of each image patch and input them into the feature fusion module; 33) The original sequence feature encoding module encodes the behavior sequence sample f z Perform feature extraction to obtain behavior sequence samples f z The original sequence features; calculate the behavior sequence sample f z The time interval between adjacent samples in the behavior sequence sample f is added to the behavior sequence sample f z From the original sequence features, we get the behavior sequence sample f z The feature sequence containing time interval information is input into the time-LSTM module; 34) the time-LSTM module extracts the behavior sequence sample f z The features are input into the feature fusion module; 35) the feature fusion module vector representation of each image block patch, behavior sequence sample f z The features of z The corresponding embedded features are input into the fully connected neural network for prediction; 36) the trained OBTT model is fine-tuned according to the error between the prediction result and the corresponding true label.

6. A server, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images