Robot user detection method, system and equipment based on combination of large language model and user relation network analysis and medium

Abstract

The invention discloses a robot user detection method, system and device based on a large language model in combination with user relation network analysis and a medium, and the method comprises the steps: firstly, optimizing the large language model through a design prompt project, and carrying out the standardized output, so as to enable the large language model to efficiently process social network data; thirdly, information including user IDs, names, positions, brief introductions and the like is collected and preprocessed to form a training set, a big language model is used for analyzing the individual brief introductions and published / commented contents of the users, and features are extracted to construct user portraits; secondly, performing feature coding on the user information, including normalization, one-hot and RoBERTa model coding, training a graph neural network model through the features, and learning and distinguishing key features of a real user and a robot; and finally, inputting user information into the model for detection, and automatically identifying a robot account. The system, the equipment and the medium are used for implementing the method. According to the method, the accuracy, adaptability and efficiency can be improved, the data processing capability and generalization capability are enhanced, and the feature purification process is simplified.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing and graph data mining, and in particular to a robot user detection method based on a large language model combined with user relationship network analysis. Background Art

[0002] With the rapid development of social media, such as the popularity of Weibo and Xiaohongshu, the speed and scope of information dissemination have reached unprecedented levels. However, this widespread information flow has also brought a series of challenges, especially in terms of information authenticity and security. This issue has attracted widespread attention at home and abroad. For example, hundreds of thousands of messages are disseminated on the famous Twitter platform abroad at all times. How to ensure their authenticity and security is a very challenging thing. Therefore, starting directly from the users who spread the information and judging their impact on the social media platform can help to ensure the authenticity and security of the disseminated information content from the source. Among these users, the ones with the greatest negative impact on social media platforms are robot users (i.e., automated social media accounts). They will quickly send negative messages in batches, which pollutes the real and safe information dissemination channels, making it difficult for users to filter out real and safe information.

[0003] The abuse of robot users has become a growing problem. They are often used to spread false information, manipulate public opinion, and commit online fraud.

[0004] Traditional robot user detection methods mainly rely on the recognition of abnormal patterns of account behavior, such as posting frequency, link sharing habits, and interaction patterns with other accounts. However, these methods have significant limitations in accuracy and efficiency. Some advanced robot users are able to circumvent these simple detection mechanisms by imitating human behavior. In addition, with the advancement of robot user generation technology, they have become more and more complex, able to better adapt to and imitate the behavior of real users, and unable to parse the differences in behavior patterns from real users from their social data, making detection more difficult. At the same time, the massive social data of robot users also poses challenges to traditional detection methods.

[0005] Wu et al. published BotTriNet: A unified and efficient embedding for socialbots detection via metric learning (J. Wu, X. Ye, and MY Yuet, "BotTriNet: A unified and efficient embedding for social bots detection via metric learning," in Proc. 11th Int. Symp. Digit. Forensics Secur., 2023, pp. 1–6.), a unified embedding framework that uses the text content posted by accounts for Twitter robot detection. The authors designed a triplet network to adjust the original embedding generated by traditional natural language processing techniques to improve classification performance, but since the input user features are only the text content of the account's posted text, the social relationships of users in the social platform are ignored, and processing all the posted text content is time-consuming and laborious, with the disadvantages of insufficient accuracy, low efficiency, lack of deep analysis, and weak generalization ability.

[0006] Feng et al. published Satar: A self-supervised approach to twitter account representation learning and its application in bot detection (S. Feng, H. Wan, N. Wang, J. Li, and M. Luo, “Satar: A self-supervised approach to twitter account representation learning and its application in bot detection,” in Proceedings of the 30th ACM International Conference on Information & Knowledge Management, 2021, pp. 3808–3817.), which introduced a self-supervised method to detect social robots by aggregating user features into sub-networks based on configuration attributes, follower relationships, and post content. However, it relies on massive data analysis and has the disadvantages of being unable to quickly process and analyze massive data and having poor real-time performance.

[0007] In summary, the prior art has the following disadvantages:

[0008] (1) Insufficient accuracy: Existing technologies mainly detect robot users by identifying abnormal patterns of account behavior, but this method has difficulty distinguishing between advanced robot users and real users because advanced robots can imitate human behavior.

[0009] (2) Inefficiency: Traditional methods are inefficient when processing massive amounts of social data, especially when it is necessary to analyze a large amount of user behavior and social network structure.

[0010] (3) Limited data processing capabilities: Existing technologies have difficulty coping with the growing amount of social data, especially when data needs to be processed and analyzed quickly to detect robot users in real time.

[0011] (4) Lack of in-depth analysis: Existing technologies do not fully utilize deep learning and social network analysis techniques to deeply understand user behavior and social network structure.

[0012] (5) Weak generalization ability: Existing methods may perform well on a specific dataset, but may perform poorly on new or unknown data and lack good generalization ability.

[0013] (6) Lack of adaptability: Existing technologies may not be able to adapt to the characteristics and changes of different social platforms and lack flexibility and universality.

[0014] (7) Difficulty in purifying user features: In massive amounts of data, it is a challenge to effectively extract features from user social data that help identify robot users. Summary of the invention

[0015] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a robot user detection method, system, device and medium based on a large language model combined with user relationship network analysis. By using a large language model and a graph neural network, an automated and intelligent detection process, combined with natural language processing and social network analysis technology, through training and regularization methods of a large amount of labeled data, robot users can be more accurately identified, manual intervention can be reduced, the efficiency of processing massive data can be improved, user social data can be more effectively processed and analyzed, user behavior and social network structure can be deeply analyzed, the depth and quality of detection can be improved, the generalization ability of the model on unknown data can be improved, the accuracy, adaptability and efficiency can be improved, the data processing ability and generalization ability can be enhanced, and the feature purification process can be simplified.

[0016] In order to achieve the above object, the technical solution adopted by the present invention is:

[0017] A robot user detection method based on a large language model combined with user relationship network analysis includes the following steps:

[0018] S1: First, we design a prompt engineering for the large language model, perform reasoning and generation of the large language model, and standardize the output of the large language model to ensure that the model can effectively process and analyze social network data;

[0019] S2: Collect user information, including user ID, public name, nickname, location information, personal profile, account protection status, number of followers, number of followings, number of lists, creation time, number of favorites, verification status, language settings, and published / commented content; pre-process and annotate user information to form a training set;

[0020] S3: Analyze the personal profile and published / commented content in the user information collected in step S2 through the prompt engineering of the large language model designed in step S1, extract user description features and published / commented content, align the results, and build a user portrait;

[0021] S4: feature encoding the user information collected in step S2, including z-score normalization of numerical features, one-hot encoding of categorical features, and encoding the user description features and posting / comment content features extracted in step S3 using the pre-trained RoBERTa model;

[0022] S5: Use the feature encoding prepared in step S4 to train the algorithm model of the graph neural network based on the user relationship network, so that the algorithm model of the graph neural network can learn the key features to distinguish between real users and robot accounts. Through training data and iterative optimization, the algorithm model of the graph neural network gradually improves its recognition accuracy.

[0023] S6: Put the actual user information into the algorithm model of the graph neural network trained in step S5 for detection, and automatically identify and mark potential robot accounts through intelligent analysis of the algorithm model of the graph neural network.

[0024] The step S1 specifically includes:

[0025] Divide the prompt engineering of designing large language models into three modules: instructions, context, and prediction requirements;

[0026] In the instruction module, the tasks of the large language model are defined, and various scenarios and requirements that need to be paid attention to are specified. The concept of role-playing is introduced in this part. The large language model is pre-specified as a community administrator of the social platform. The subsequent tasks required to be performed are to summarize the personal profile in the user information and select the most representative k posts / comments of the users.

[0027] In the context module, define the personal profile and published / commented content of the input user information;

[0028] In the demand prediction module, the output requirements and format are precisely defined. In addition, combined with the concept of One-Shot Learning, output examples are designed to help the large language model understand the task content and output format, and guide the large language model to output in JSON format.

[0029] The preprocessing in step S2 includes removing invalid data, filling missing values, and performing text cleaning;

[0030] Removing invalid data: If there is missing or garbled data in the user information about the published / commented content, as well as missing user ID, number of followers, or number of followings, it will be uniformly defined as invalid data and removed;

[0031] Filling missing values: The missing data in the user's personal profile is transmitted to the large language model through the published / commented content in the user information, and the large language model outputs the user's personal profile to fill the information;

[0032] The text cleaning: deleting irrelevant characters in the user information, including special symbols, numbers, HTML tags, and extra spaces, and uniformly setting the data encoding format to UTF-8 format to facilitate subsequent feature extraction and model training;

[0033] The redundant spaces include spaces at the beginning and end of the text, and multiple spaces between words.

[0034] In step S3, user description feature extraction includes the following steps:

[0035] S3.1: The large language model prompting project designed in step S1 is combined with the personal profile and published / commented content in the user information and input into the large language model, requiring the large language model to re-summarize the personal profile and select k most representative published / commented content, and match the k published / commented content selected by the large language model with all posts published by the user one by one, so as to accurately identify the original text of the published / commented content, and ensure that the k published / commented content selected by the large language model is aligned with the published / commented content in the original user information;

[0036] S3.2: Take the predicted demand module in step S1 as an example of the output result of the large language model, require the large language model to output the personal profile summarized in step S3.1 and select k posts / comments in Json format, and set a Json format regular expression to filter and match the output of the large language model, and convert the answer content into Json data format for easy data reading;

[0037] S3.3: For the answer of the large language model, select the currently open source large language model for local deployment and API call of the closed source large language model, and select the k selected publication / comment contents outputted from the large language model in step S3.2, where 1≤k≤10.

[0038] The feature encoding of the user information in step S4 comprises the following steps:

[0039] The z-score normalization of the numerical features is specifically as follows: the number of followers, the number of followings, the number of lists, the creation time, the number of favorites, and the verification status in the user information are set as the user's numerical features, and the average value μ and the standard deviation σ of each numerical feature are calculated, and each numerical feature is converted into a standardized z-score form using the z-score formula to eliminate the dimensional influence between different numerical features, so that they can be compared and analyzed on the same scale;

[0040] The one-hot encoding of the category feature is specifically as follows: the public name, nickname, location information, and language setting in the user information are set as the category feature of the user, and for each category feature, a binary vector is created, the length of the binary vector is equal to the number of categories of the feature, and for each user record, according to the value of its category feature, the corresponding binary vector position is marked as 1, and the remaining positions are marked as 0, and the category feature of the user information is one-hot encoded to convert it into a numerical form that can be processed by the machine learning model;

[0041] The encoding of the user's description features and the published / commented content features is specifically as follows: the user portrait constructed in step S3 is used as the user's description features, the k selected published / commented contents are used as the user's published / commented content features, and the pre-trained RoBERTa is used to encode the user's description features and the published / commented content features, and the semantic features of the text are taken to enhance the model's ability to understand the published / commented content.

[0042] The step S5 specifically comprises the following steps:

[0043] S5.1: Consider users as nodes and define two types of edges for users: following and follower, which represent two different social relationships of users respectively. User numerical features, user category features, user description features, and posting / comment content features are combined to form nodes in a heterogeneous graph. Based on the user's social relationships and the category nodes in the heterogeneous graph, a user relationship network is formed to obtain a graph neural network model.

[0044] S5.2: By labeling data and training the graph neural network model obtained in step S5.1, the node features and graph structure information are used for learning and reasoning, and the key features that distinguish real users from robot accounts are learned. During the training process, the model parameters are iteratively optimized through the back-propagation algorithm. The accuracy of the graph neural network model is improved by minimizing the difference between the predicted results and the actual labels. The network learning rate of the graph neural network model is set to θ, the dropout rate is set to σ, and the weight_decay is set to ω. The momentum optimizer is used, the cross entropy loss function is selected and different regularization methods are added. The model parameters are back-propagated using the results of the cross entropy loss function, and the gradient of each parameter is calculated. The parameters of the graph neural network model are updated using gradient descent to reduce the value of the cross entropy loss function. Cross-validation is used, and Accuracy and F1-score are used as evaluation indicators to evaluate the performance of the model to ensure the generalization ability of the model.

[0045] A robot user detection system based on a large language model combined with user relationship network analysis, comprising:

[0046] The prompt engineering module designs the prompt engineering of the large language model, performs reasoning and generation of the large language model, and standardizes the output of the large language model to ensure that the model can effectively process and analyze social network data;

[0047] The data preprocessing module collects user information, including the user's ID, public name, nickname, location information, personal profile, account protection status, number of followers, number of followings, number of lists, creation time, number of favorites, verification status, language settings, and published / commented content; preprocesses and annotates user information to form a training set;

[0048] The user portrait generation module analyzes the personal profile and published / commented content in the user information through the prompt engineering of the large language model, extracts the user description features and published / commented content, aligns the results, and constructs a user portrait;

[0049] The feature encoding module encodes the user information, including z-score normalization of numerical features, one-hot encoding of categorical features, and encoding the user description features and posting / comment content features extracted in step S3 using the pre-trained RoBERTa model;

[0050] The model training module uses feature encoding to train the algorithm model of the graph neural network based on the user relationship network, so that the algorithm model of the graph neural network can learn the key features to distinguish between real users and robot accounts. Through training data and iterative optimization, the algorithm model of the graph neural network gradually improves its recognition accuracy;

[0051] The detection execution module puts the actual user information into the algorithm model of the graph neural network for detection, and automatically identifies and marks potential robot accounts through intelligent analysis of the algorithm model of the graph neural network.

[0052] A robot user detection device based on a large language model combined with user relationship network analysis, comprising:

[0053] Memory: used to store a computer program for implementing the robot user detection method based on a large language model combined with user relationship network analysis;

[0054] Processor: used to implement the robot user detection method based on a large language model combined with user relationship network analysis when executing the computer program.

[0055] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a robot user detection method based on a large language model combined with user relationship network analysis are implemented.

[0056] Compared with the prior art, the beneficial effects of the present invention are:

[0057] 1. The present invention adopts role-playing and One-Shot Learning in the large language model prompting project, realizes the precise task specification of user information summarization and representative content selection and standardizes the output format of the large language model, and has the enhanced ability to process and analyze social network data.

[0058] 2. The present invention uses a large language model to generate user portraits, realizes in-depth analysis of user behaviors and attributes, has feature purification, and reduces the feature data that needs to be processed by the recognition model.

[0059] 3. The present invention uses z-score normalization and one-hot encoding of user information to achieve the elimination of dimensions between different features and the effective conversion of category features, which has the effect of improving the model comparison and analysis capabilities.

[0060] 4. The present invention uses a pre-trained RoBERTa model to encode user description features and published / comment content features, thereby realizing the extraction of text semantic features and enhancing the model's ability to understand content.

[0061] 5. The present invention adopts a technology that combines a large language model with user relationship network analysis to achieve efficient detection of robot users in social networks, significantly improving accuracy and adaptability.

[0062] 6. The present invention adopts a graph neural network model for training optimization and detection, realizes complex pattern recognition based on user relationship networks, enhances the robot account recognition capability, and improves the ability to self-learn and adapt to new situations.

[0063] 7. The present invention implements the detection method through an integrated system, equipment and medium, realizes the integration and convenience of the detection process, and has the convenience of system deployment and maintenance.

[0064] In summary, compared with the traditional rule-based detection method, the present invention has the advantages of strong ability to process and analyze social network data, feature purification, reducing the feature data that needs to be processed by the recognition model, improving the generalization ability of the model on unknown data, accuracy, adaptability, self-learning ability and detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a comparison chart of the simulation experiment results of the present invention and the baseline.

[0066] Figure 2 These are the network parameters and detection accuracy of SOTA and the present invention on Cresci-15 and Twibot-20 in the simulation experiment of the present invention.

[0067] Figure 3 The number of posts and detection accuracy of SOTA and the present invention on Cresci-15 and Twibot-20 in the simulation experiment of the present invention. DETAILED DESCRIPTION

[0068] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0069] The present invention provides a robot user detection method based on a large language model combined with user relationship network analysis, comprising the following steps:

[0070] S1: First, we design a prompt engineering for the large language model, perform reasoning and generation of the large language model, and standardize the output of the large language model to ensure that the model can effectively process and analyze social network data;

[0071] S2: Collect user information, including user ID, public name, nickname, location information, personal profile, account protection status, number of followers, number of followings, number of lists, creation time, number of favorites, verification status, language settings, and published / commented content; pre-process and annotate user information to form a training set;

[0072] S3: Analyze the personal profile and published / commented content in the user information collected in step S2 through the prompt engineering of the large language model designed in step S1, extract user description features and published / commented content, align the results, and build a user portrait;

[0073] S4: feature encoding the user information collected in step S2, including z-score normalization of numerical features, one-hot encoding of categorical features, and encoding the user description features and posting / comment content features extracted in step S3 using the pre-trained RoBERTa model;

[0074] S5: Use the feature encoding prepared in step S4 to train the algorithm model of the graph neural network based on the user relationship network, so that the algorithm model of the graph neural network can learn the key features to distinguish between real users and robot accounts. Through training data and iterative optimization, the algorithm model of the graph neural network gradually improves its recognition accuracy.

[0075] S6: Put the actual user information into the algorithm model of the graph neural network trained in step S5 for detection, and automatically identify and mark potential robot accounts through intelligent analysis of the algorithm model of the graph neural network.

[0076] The step S1 specifically includes:

[0077] Divide the prompt engineering of designing large language models into three modules: instructions, context, and prediction requirements;

[0078] In the instruction module, the tasks of the large language model are defined, and various scenarios and requirements that need to be paid attention to are specified. The concept of role-playing is introduced in this part. The large language model is pre-specified as a community administrator of the social platform. The subsequent tasks required to be performed are to summarize the personal profile in the user information and select the most representative k posts / comments of the users.

[0079] In the context module, define the personal profile and published / commented content of the input user information;

[0080] In the demand prediction module, the output requirements and format are precisely defined. In addition, combined with the concept of One-Shot Learning, output examples are designed to help the large language model understand the task content and output format, and guide the large language model to output in JSON format.

[0081] The preprocessing in step S2 includes removing invalid data, filling missing values, and performing text cleaning;

[0082] Removing invalid data: If there is missing or garbled data in the user information about the published / commented content, as well as missing user ID, number of followers, or number of followings, it will be uniformly defined as invalid data and removed;

[0083] Filling missing values: The missing data in the user's personal profile is transmitted to the large language model through the published / commented content in the user information, and the large language model outputs the user's personal profile to fill the information;

[0084] The text cleaning: deleting irrelevant characters in the user information, including special symbols, numbers, HTML tags, and extra spaces, and uniformly setting the data encoding format to UTF-8 format to facilitate subsequent feature extraction and model training;

[0085] The redundant spaces include spaces at the beginning and end of the text, and multiple spaces between words.

[0086] Labeling of user information: User information is categorized into two types: robot accounts and human accounts.

[0087] In step S3, user description feature extraction includes the following steps:

[0088] S3.1: The large language model prompting project designed in step S1 is combined with the personal profile and published / commented content in the user information and input into the large language model, requiring the large language model to re-summarize the personal profile and select k most representative published / commented content, and match the k published / commented content selected by the large language model with all posts published by the user one by one, so as to accurately identify the original text of the published / commented content, and ensure that the k published / commented content selected by the large language model is aligned with the published / commented content in the original user information;

[0089] S3.2: Take the predicted demand module in step S1 as an example of the output result of the large language model, require the large language model to output the personal profile summarized in step S3.1 and select k posts / comments in Json format, and set a Json format regular expression to filter and match the output of the large language model, and convert the answer content into Json data format for easy data reading;

[0090] S3.3: For the answers of the large language model, the currently open source large language model is selected for local deployment and API calls of the closed source large language model, and the k selected posts / comments output in the large language model in step S3.2 are selected, where 1≤k≤10, and k is set to 2, 5, and 10 to conduct experiments to compare the accuracy of the final classification results. Finally, the optimal large language model is selected as Vicuna-7b-v1.5, and the k value is set to 5.

[0091] The published / commented content is specifically: all published / commented content data in the user information is input into the large language model, and the large language model is allowed to select k published / commented contents that best represent the user's characteristics.

[0092] The above-mentioned result alignment and construction of user portraits are as follows: the large language model selects k posts / comments that best represent the user's characteristics and aligns them with the posts / comments in the user information. The original text cannot be tampered with or its semantic expression modified. The user portrait is constructed by combining the k posts / comments selected by the large language model and the personal profile information re-summarized by the large language model.

[0093] The feature encoding of the user information in step S4 comprises the following steps:

[0094] The z-score normalization of the numerical features is specifically as follows: the number of followers, the number of followings, the number of lists, the creation time, the number of favorites, and the verification status in the user information are set as the user's numerical features, and the average value μ and the standard deviation σ of each numerical feature are calculated, and each numerical feature is converted into a standardized z-score form using the z-score formula to eliminate the dimensional influence between different numerical features, so that they can be compared and analyzed on the same scale;

[0095] The one-hot encoding of the category feature is specifically as follows: the public name, nickname, location information, and language setting in the user information are set as the category feature of the user, and for each category feature, a binary vector is created, the length of the binary vector is equal to the number of categories of the feature, and for each user record, according to the value of its category feature, the corresponding binary vector position is marked as 1, and the remaining positions are marked as 0, and the category feature of the user information is one-hot encoded to convert it into a numerical form that can be processed by the machine learning model;

[0096] The encoding of the user's description features and the published / commented content features is specifically as follows: the user portrait constructed in step S3 is used as the user's description features, the k selected published / commented contents are used as the user's published / commented content features, and the pre-trained RoBERTa is used to encode the user's description features and the published / commented content features, and the semantic features of the text are taken to enhance the model's ability to understand the published / commented content.

[0097] The step S5 specifically comprises the following steps:

[0098] S5.1: Consider users as nodes and define two types of edges for users: following and follower, which represent two different social relationships of users respectively. User numerical features, user category features, user description features, and posting / comment content features are combined to form nodes in a heterogeneous graph. Based on the user's social relationships and the category nodes in the heterogeneous graph, a user relationship network is formed to obtain a graph neural network model.

[0099] S5.2: By labeling data and training the graph neural network model obtained in step S5.1, the node features and graph structure information are used for learning and reasoning, and the key features that distinguish real users from robot accounts are learned. During the training process, the model parameters are iteratively optimized through the back-propagation algorithm, and the accuracy of the graph neural network model is improved by minimizing the difference between the predicted results and the actual labels. The network learning rate of the graph neural network model is set to θ, the dropout rate is set to σ, and the weight_decay is set to ω. The momentum optimizer Adam is used, the cross entropy loss function is selected and the L2 regularization method is added. The model parameters are back-propagated using the results of the cross entropy loss function, and the gradient of each parameter is calculated. The parameters of the graph neural network model are updated using gradient descent to reduce the value of the cross entropy loss function. Cross-validation is used, and Accuracy and F1-score are used as evaluation indicators to evaluate the performance of the model to ensure the generalization ability of the model. In this way, the model can gradually improve its recognition accuracy and effectively distinguish normal users from potential robot accounts.

[0100] A robot user detection system based on a large language model combined with user relationship network analysis, comprising:

[0101] The prompt engineering module designs the prompt engineering of the large language model, performs reasoning and generation of the large language model, and standardizes the output of the large language model to ensure that the model can effectively process and analyze social network data, which is used to implement step S1 of a robot user detection method based on a large language model combined with user relationship network analysis;

[0102] A data preprocessing module collects user information, including the user's ID, public name, nickname, location information, personal profile, account protection status, number of followers, number of followings, number of lists, creation time, number of favorites, verification status, language settings, and published / commented content; preprocesses and annotates the user information to form a training set for implementing step S2 of a robot user detection method based on a large language model combined with user relationship network analysis;

[0103] A user portrait generation module analyzes the personal profile and published / commented content in the user information through the prompt engineering of the large language model, extracts the user description features and the published / commented content and aligns the results to construct a user portrait, which is used to implement step S3 of a robot user detection method based on a large language model combined with user relationship network analysis;

[0104] A feature encoding module performs feature encoding on the user information, including z-score normalization of numerical features, one-hot encoding of categorical features, and encoding the user description features and posting / comment content features extracted in step S3 using a pre-trained RoBERTa model, for implementing step S4 of a robot user detection method based on a large language model combined with user relationship network analysis;

[0105] A model training module uses feature coding to train the algorithm model of the graph neural network based on the user relationship network, so that the algorithm model of the graph neural network can learn the key features to distinguish between real users and robot accounts. Through training data and iterative optimization, the algorithm model of the graph neural network gradually improves its recognition accuracy, which is used to implement a robot user detection method based on a large language model combined with user relationship network analysis in step S5;

[0106] The detection execution module puts the actual user information into the algorithm model of the graph neural network for detection, and automatically identifies and marks potential robot accounts through intelligent analysis of the algorithm model of the graph neural network, so as to implement step S6 of a robot user detection method based on a large language model combined with user relationship network analysis.

[0107] A robot user detection device based on a large language model combined with user relationship network analysis, comprising:

[0108] Memory: used to store a computer program for implementing the robot user detection method based on a large language model combined with user relationship network analysis;

[0109] Processor: used to implement the robot user detection method based on a large language model combined with user relationship network analysis when executing the computer program.

[0110] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the robot user detection method based on a large language model combined with user relationship network analysis are implemented.

[0111] Simulation experiment

[0112] 1. Dataset

[0113] Cresci-15 is an early and widely used open source dataset for social bot detection. The accounts in the Cresci-15 dataset include human user accounts and fake follower accounts, which are paid to follow other accounts. In terms of data scale, Cresci-15 contains 5301 users, 2827757 tweets, and 14220 edges. For user information, Cresci-15 includes semantic and attribute information of Twitter users.

[0114] Twibot-20 is a public social robot detection dataset. It is a large-scale social robot detection benchmark in Twitter, containing 229,573 users, 33,488,192 tweets, 8,723,736 user attribute entries, and 455,958 follow relationships. From this dataset, a heterogeneous graph consisting of 229,580 nodes and 227,979 edges can be obtained. Compared with previous datasets, this dataset not only contains user attributes and semantic information, but also adds users' community relationships.

[0115] 2. Baseline comparison

[0116] DeeProBot: DeeProBot uses long short-term memory (LSTM) units and dense layers to specifically extract data features from user profiles and enhances word representations with pre-trained GloVe embeddings.

[0117] BotBuster: BotBuster is a social bot detection system that processes user metadata and textual information and uses a mixture of experts model to enhance bot detection across platforms.

[0118] RF-GNN: RF-GNN uses graph neural networks (GNNs) as base classifiers to build random forests, and uses different subgraphs to train GNN base classifiers, and finally aggregates them to produce results.

[0119] SATAR

[36] : SATAR introduces a self-supervised approach to detect Twitter bots by aggregating user features into sub-networks based on profile attributes, follower relationships, and tweet content.

[0120] BotRGCN: BotRGCN is a Twitter bot detection framework that uses relational graph convolutional networks to address community challenges by constructing a heterogeneous graph consisting of attention relations.

[0121] LMBot: LMBot is a robot detection framework based on pre-trained language models, suitable for deployment environments without graphs.

[0122] BotCS: BotCS introduces a simple multi-layer perceptron (MLP) based model without graph neural network (GNN) to detect Twitter bots, integrates social graph features, and adopts customized rules based on local bot-human interactions for prediction.

[0123] RGT: RGT introduces a user-based framework that uses edge-heterogeneous graphs to model multi-dimensional relationships and influences to effectively learn user characteristics and interactions.

[0124] BIC: BIC is a Twitter bot detection framework with text-graph interaction and semantic consistency.

[0125] like Figure 1 As shown in Figure 2, the evaluation indicators of the model of this method, namely Ours, are compared with those of the existing advanced models.

[0126] Among the research methods in recent years, the present invention surpasses most models with an accuracy of 86.81% and an F1-score of 88.48%. After the introduction of the large model, the index comparison of the present invention completely surpasses the models such as BotRGCN and LMBot, and the accuracy is slightly lower than RGT and BIC. The experimental results show that the present invention can surpass most of the optimal values ​​of other methods, and the data preprocessing of the present invention is simpler and uses fewer data feature attributes.

[0127] 2. Efficiency comparison

[0128] We have observed a notable trend in recent social robot detection research, which involves constructing and computing more data feature attributes to improve the accuracy of detection models. These methods usually adopt machine learning or deep learning techniques to learn the underlying distribution of various feature information. Although theoretically able to improve the predictive ability of the model, they have not yet considered the associated cost issues. As the number of data feature attributes increases, the computational burden and number of parameters of the model also increase, making deployment more challenging. Figure 2 As shown in the figure, compared with high-precision SOTA methods such as SATAR, RGT and LMBot, Ours in the figure is the method of the present invention, which requires fewer model parameters while maintaining higher effectiveness.

[0129] In contrast, the method of the present invention achieves excellent performance with fewer parameters, which shows that the present invention achieves superior performance while reducing parameters and training costs.

[0130] Furthermore, to demonstrate that selectively selecting and generating the most representative and unique user data can improve the accuracy of the model compared to relying on extensive and complex social data processing, the present invention is compared with related model (SOTA) methods such as BotRGCN, LMBot, BotBuster and BotCS. Figure 3 As shown, Ours, i.e., the present invention, can achieve SOTA results using only 5 posts.

[0131] By generating high-quality user data, the present invention achieves top-level results with only 5 posts, outperforming other SOTA methods for optimizing data or models in terms of effectiveness and operability.

[0132] 3. Ablation Experiment

[0133] In this part of the experiment, we will demonstrate the advantages of each module. This study will respectively demonstrate (1) the role of large language models; (2) the role of prompt engineering; and (3) the impact of different k values ​​in the large language model's selection of k most representative tweets.

[0134] 3.1 The role of large language models

[0135] In this part, this study uses a large language model to summarize and filter features of the original data set, which is called a large language model summary and filtering module. The experiment of the present invention explores the performance of the social robot detection model with and without a large language model summary and filtering module, and uses the two indicators of Accuracy and F1-score for measurement, in order to prove the effectiveness of the large language model module designed by the present invention in centrally selecting the most representative and discriminative k_tweets and generating appropriate user_description, that is, user portraits for the final social robot detection work. The comparison effect with and without a large language model is shown in Table 1 below.

[0136] Table 1: Comparison of experimental results with and without large language model enhancement

[0137] Accuracy F1-score With LLMs 0.8681 0.8848 Without LLMs 0.8462 0.8707

[0138] In Table 1, we can see that on the Twibot20 dataset, the accuracy of using large model enhancement is 86.81%, and the accuracy of not using large model enhancement is 84.62%, an improvement of about 2%. This proves that the large language model plays a positive role in the social robot detection task and that user portraits are effective for social robot detection.

[0139] 3.2 The role of prompt engineering

[0140] In the design of prompt engineering, the contextual learning framework is mainly used to improve the performance of large language models. Specifically, the concepts of role-playing and one-shot learning are combined to standardize the output style of large language models, and customized task instructions and examples are added to guide large language models to complete tasks that satisfy both generation and centralized selection. Through experiments, the necessity of role-playing and one-shot learning in prompt design will be discussed respectively.

[0141] 3.2.1 The role of role playing in the project

[0142] Role-playing in prompt engineering: Role-playing, also known as role-specific prompts, is the process of guiding a large language model to complete a task by assuming a specific role. This involves not only assigning a name or title to the model, but also deeply building a virtual character with unique perspectives, expertise, and behavior patterns. In this way, the large language model can more accurately simulate the way of thinking of a specific role, thereby providing answers that are closer to real-life scenarios. Different roles affect the content and style of the large language model's response. In this experiment, the large language model was made to assume the role of a social platform community assistant to improve its performance on social network-related tasks, while also avoiding the situation where the model refuses to generate user roles due to privacy issues. The experimental operators explored the impact of role-playing in the prompt on the final detection task and used accuracy and F1 score as indicators to measure the effect. As shown in Table 2, the impact of role-playing in prompt design on the enhancement effect of the large language model on Twibot-20 is illustrated, where "W" stands for "With" and "O" stands for "Without".

[0143] Table 2: Effects of prompt engineering on the final detection task with or without role-playing

[0144] Accuracy F1-score PromptW Role-Playing 0.8681 0.8848 PromptO Role-Playing 0.8521 0.8685

[0145] The accuracy and F1 score of using role-playing in the prompt engineering are 86.81% and 88.48%, respectively, which are 1.6% and 1.63% higher than those without role-playing. This proves that using role-playing in the prompt engineering has a positive effect on the accurate generation of user portraits.

[0146] 3.2.2 The role of one-shot learning in prompt engineering

[0147] Although large language models demonstrate amazing zero-shot capabilities, they perform poorly on complex tasks when using the zero-shot setting. In this experiment, if we only input the task description and output requirements to the large language models, they cannot meet our requirements when outputting answers. They often output redundant content and the output format does not meet the specifications, which makes it difficult to extract the corresponding data later.

[0148] Few-sample prompts can be used as a technique to enable contextual learning. The present invention only needs to put a small number of samples in the context of the input text to induce the large language model to achieve better performance. In this experiment, the present invention hopes that the large language model will output user_description and the most representative k tweets k_tweets in JSON format to facilitate subsequent input data processing. Therefore, the present invention provides an output example in JSON format for the large language model, requiring the large language model to output according to this example, thereby limiting its output. The experiment explored the impact of prompt engineering statements with or without output examples on the output of the large language model, and measured its effect with the JSON format pass rate, and the pass rate calculation formula is:

[0149] θ=Json_num / All_file (1-1)

[0150] The results of the effects of providing output examples on the output of the large language model are shown in Table 3 below:

[0151] Table 3: The impact of prompt engineering sentences with or without output examples on the output of large language models

[0152] θ PromptW One-shot Learning 0.8164 PromptO One-shot Learning 0.5203

[0153] Through few-shot learning, the user portrait data output in JSON format is 29.61% higher than the data generated without using few-shot learning. This shows that few-shot learning significantly improves the ability of large language models to build effective user portraits, and plays a key role in generating high-quality user portraits.

[0154] 3.3 The role of K value

[0155] When exploring the performance of the social robot detection model, the present invention conducts experiments on the k value, aiming to find the optimal k value when the large language model selects the most representative number of tweets, and uses Accuracy and F1-score as indicators for evaluation. This study sets three cases of K=2, K=5, and K=10, and selects three large language models for testing, namely Llama2-7b, ChatGLM3-6b, and Vicuna-7b-v1.5, thus forming 9 groups of control experiments.

[0156] These experiments show that the choice of k significantly affects the detection ability of the model. Each setting of k corresponds to a different detection accuracy, indicating that the number of posts selected is a key factor in whether the model can effectively capture the characteristics of user behavior. The experimental results show that when the k value is low, the model may not be able to fully capture the complexity of user behavior; conversely, when the k value is high, unnecessary noise may be introduced, which may impair the judgment of the model. The results show that the performance of the robot detection model (especially Vicuna-7b-v1.5) is best when k is set to 5. This finding emphasizes that in certain cases, a moderate number of posts is more effective than processing all users' posts at once, because it can better help the model identify and learn user behavior patterns, thereby improving detection accuracy. The powerful language processing capabilities of Vicuna-7b-v1.5 combined with the appropriate k value enable the model to more accurately distinguish between social robots and human users. The number of tweets and evaluation results are shown in Table 4 below.

[0157] Table 4 Precision indicators for different numbers of tweets

[0158]

[0159]

Claims

1. A robot user detection method based on a large language model combined with user relationship network analysis, characterized in that: The following steps are involved: S1: First, we design a prompt engineering for the large language model, perform reasoning and generation of the large language model, and standardize the output of the large language model to ensure that the model can effectively process and analyze social network data; S2: Collect user information, including user ID, public name, nickname, location information, personal profile, account protection status, number of followers, number of followings, number of lists, creation time, number of favorites, verification status, language settings, and published / commented content; Preprocess and label user information to form a training set; S3: Analyze the personal profile and published / commented content in the user information collected in step S2 through the prompt engineering of the large language model designed in step S1, extract user description features and published / commented content, align the results, and build a user portrait; S4: feature encoding the user information collected in step S2, including z-score normalization of numerical features, one-hot encoding of categorical features, and encoding the user description features and posting / comment content features extracted in step S3 using the pre-trained RoBERTa model; S5: Use the feature encoding prepared in step S4 to train the algorithm model of the graph neural network based on the user relationship network, so that the algorithm model of the graph neural network can learn the key features to distinguish between real users and robot accounts. Through training data and iterative optimization, the algorithm model of the graph neural network gradually improves its recognition accuracy. S6: Put the actual user information into the algorithm model of the graph neural network trained in step S5 for detection, and automatically identify and mark potential robot accounts through intelligent analysis of the algorithm model of the graph neural network.

2. According to claim 1, a robot user detection method based on a large language model combined with user relationship network analysis is characterized in that: The step S1 specifically includes: Divide the prompt engineering of designing large language models into three modules: instructions, context, and prediction requirements; In the instruction module, the tasks of the large language model are defined, and various scenarios and requirements that need to be paid attention to are specified. The concept of role-playing is introduced in this part. The large language model is pre-specified as a community administrator of the social platform. The subsequent tasks required to be performed are to summarize the personal profile in the user information and select the most representative k posts / comments of the users. In the context module, define the personal profile and published / commented content of the input user information; In the demand prediction module, the output requirements and format are precisely defined. In addition, combined with the concept of One-Shot Learning, output examples are designed to help the large language model understand the task content and output format, and guide the large language model to output in JSON format.

3. According to claim 1, a robot user detection method based on a large language model combined with user relationship network analysis is characterized in that: The preprocessing in step S2 includes removing invalid data, filling missing values, and performing text cleaning; Removing invalid data: If there is missing or garbled data in the user information about the published / commented content, as well as missing user ID, number of followers, or number of followings, it will be uniformly defined as invalid data and removed; Filling missing values: The missing data in the user's personal profile is transmitted to the large language model through the published / commented content in the user information, and the large language model outputs the user's personal profile to fill the information; The text cleaning: deleting irrelevant characters in the user information, including special symbols, numbers, HTML tags, and extra spaces, and uniformly setting the data encoding format to UTF-8 format to facilitate subsequent feature extraction and model training; The redundant spaces include spaces at the beginning and end of the text, and multiple spaces between words.

4. The robot user detection method based on a large language model combined with user relationship network analysis according to claim 1, characterized in that: In step S3, user description feature extraction includes the following steps: S3.1: The large language model prompting project designed in step S1 is combined with the personal profile and published / commented content in the user information and input into the large language model, requiring the large language model to re-summarize the personal profile and select k most representative published / commented content, and match the k published / commented content selected by the large language model with all posts published by the user one by one, so as to accurately identify the original text of the published / commented content, and ensure that the k published / commented content selected by the large language model is aligned with the published / commented content in the original user information; S3.2: Take the predicted demand module in step S1 as an example of the output result of the large language model, require the large language model to output the personal profile summarized in step S3.1 and select k posts / comments in Json format, and set a Json format regular expression to filter and match the output of the large language model, and convert the answer content into Json data format for easy data reading; S3.3: For the answer of the large language model, select the currently open source large language model for local deployment and API call of the closed source large language model, and select the k selected publication / comment contents outputted from the large language model in step S3.2, where 1≤k≤10.

5. The robot user detection method based on a large language model combined with user relationship network analysis according to claim 1, characterized in that: The feature encoding of the user information in step S4 comprises the following steps: The z-score normalization of the numerical features is specifically as follows: the number of followers, the number of followings, the number of lists, the creation time, the number of favorites, and the verification status in the user information are set as the user's numerical features, and the average value μ and the standard deviation σ of each numerical feature are calculated, and each numerical feature is converted into a standardized z-score form using the z-score formula to eliminate the dimensional influence between different numerical features, so that they can be compared and analyzed on the same scale; The one-hot encoding of the category feature is specifically as follows: the public name, nickname, location information, and language setting in the user information are set as the category feature of the user, and for each category feature, a binary vector is created, the length of the binary vector is equal to the number of categories of the feature, and for each user record, according to the value of its category feature, the corresponding binary vector position is marked as 1, and the remaining positions are marked as 0, and the category feature of the user information is one-hot encoded to convert it into a numerical form that can be processed by the machine learning model; The encoding of the user's description features and the published / commented content features is specifically as follows: the user portrait constructed in step S3 is used as the user's description features, the k selected published / commented contents are used as the user's published / commented content features, and the pre-trained RoBERTa is used to encode the user's description features and the published / commented content features, and the semantic features of the text are taken to enhance the model's ability to understand the published / commented content.

6. The robot user detection method based on a large language model combined with user relationship network analysis according to claim 5, characterized in that: The step S5 specifically comprises the following steps: S5.1: Consider users as nodes and define two types of edges for users: following and follower, which represent two different social relationships of users respectively. User numerical features, user category features, user description features, and posting / comment content features are combined to form nodes in a heterogeneous graph. Based on the user's social relationships and the category nodes in the heterogeneous graph, a user relationship network is formed to obtain a graph neural network model. S5.2: By labeling data and training the graph neural network model obtained in step S5.1, the node features and graph structure information are used for learning and reasoning, and the key features that distinguish real users from robot accounts are learned. During the training process, the model parameters are iteratively optimized through the back-propagation algorithm. The accuracy of the graph neural network model is improved by minimizing the difference between the predicted results and the actual labels. The network learning rate of the graph neural network model is set to θ, the dropout rate is set to σ, and the weight_decay is set to ω. The momentum optimizer is used, the cross entropy loss function is selected and different regularization methods are added. The model parameters are back-propagated using the results of the cross entropy loss function, and the gradient of each parameter is calculated. The parameters of the graph neural network model are updated using gradient descent to reduce the value of the cross entropy loss function. Cross-validation is used, and Accuracy and F1-score are used as evaluation indicators to evaluate the performance of the model to ensure the generalization ability of the model.

7. A detection system based on a robot user detection method based on a large language model combined with user relationship network analysis according to any one of claims 1 to 6, comprising: The prompt engineering module designs the prompt engineering of the large language model, performs reasoning and generation of the large language model, and standardizes the output of the large language model to ensure that the model can effectively process and analyze social network data; Data preprocessing module collects user information, including user ID, public name, nickname, location information, personal profile, account protection status, number of followers, number of followings, number of lists, creation time, number of favorites, verification status, language settings, and published / commented content; Preprocess and label user information to form a training set; The user portrait generation module analyzes the personal profile and published / commented content in the user information through the prompt engineering of the large language model, extracts the user description features and published / commented content, aligns the results, and constructs a user portrait; The feature encoding module encodes the user information, including z-score normalization of numerical features, one-hot encoding of categorical features, and encoding the user description features and posting / comment content features extracted in step S3 using the pre-trained RoBERTa model; The model training module uses feature encoding to train the algorithm model of the graph neural network based on the user relationship network, so that the algorithm model of the graph neural network can learn the key features to distinguish between real users and robot accounts. Through training data and iterative optimization, the algorithm model of the graph neural network gradually improves its recognition accuracy; The detection execution module puts the actual user information into the algorithm model of the graph neural network for detection, and automatically identifies and marks potential robot accounts through intelligent analysis of the algorithm model of the graph neural network.

8. A robot user detection device based on a large language model combined with user relationship network analysis, characterized in that: include: Memory: used to store a computer program for implementing a robot user detection method based on a large language model combined with user relationship network analysis as described in any one of claims 1 to 6; Processor: used to implement a robot user detection method based on a large language model combined with user relationship network analysis as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a robot user detection method based on a large language model combined with user relationship network analysis as described in any one of claims 1 to 6 are implemented.

Images