Illegal behavior-containing website description method and system based on adaptive subspace selection algorithm

Abstract

The invention provides a method and a system for describing a website containing illegal behaviors based on a self-adaptive subspace selection algorithm, and belongs to the technical field of internet network measurement. The method comprises the following steps: S1, crawling website data by utilizing a crawler; step S2, carrying out data preprocessing on the crawled website data; s3, performing text classification by using a BERT model so as to identify websites containing illegal behaviors; s4, carrying out BERT text multi-label classification based on data dimension reduction on the recognized website containing the illegal behaviors; and S5, carrying out website subject description on each type of websites containing illegal behaviors. The invention provides a solution for solving the problems of unclear description, difficulty in description and the like of website main bodies containing illegal behaviors, and improves the problems of difficulty in evidence collection, difficulty in tracing and the like of network public hazard main bodies.

Description

Technical Field

[0001] The present invention belongs to the technical field of Internet network measurement, and in particular relates to a method and system for characterizing websites containing illegal behaviors based on an adaptive subspace selection algorithm. Background Art

[0002] The unclear and difficult characterization of the subject of network nuisance is an important link in the difficulty of collecting evidence and tracing the source of the subject of network nuisance due to the complex dynamics of the subject's behavior. How to characterize the characteristics of the subject of network nuisance is actually a multi-label classification problem. In the process of feature extraction, data dimensionality reduction is generally performed for large-scale data. How to select the characteristics of the subject of nuisance from large-scale website data is the key to solving this problem.

[0003] With the development of information society and machine learning, the availability of large-scale data is increasing in many key machine learning application areas such as text classification, biomolecule analysis, scene classification, medical diagnosis, etc. However, these data often contain a large number of redundant features, which not only affects the accuracy of data classification, but also seriously wastes memory and increases the computational cost of the algorithm. Therefore, in recent years, many researchers have been exploring feature selection methods, which are effective dimensionality reduction methods that aim to identify features and eliminate redundant features in the original dataset to improve the classification performance of downstream tasks.

[0004] According to the different dimensions of the label space in the data, the feature selection problem can be divided into single-label feature selection problem and multi-label feature selection problem. Although the single-label feature selection algorithm is quite mature, it cannot be directly applied to multi-label datasets for the following reasons. (1) In multi-label problems, a sample is usually associated with multiple labels, and label correlation is the key to multi-label learning. For example, in some image datasets, the label ship has a strong correlation with the labels sea and lake, while the label desert has no correlation with the label ship. (2) In multi-label feature selection, it is necessary to maintain the consistency of the label space and the feature space. Spatial consistency and label correlation are aspects that single-label algorithms cannot effectively learn.

[0005] Therefore, many multi-label feature selection methods have emerged. According to the relationship between the feature evaluation function and the model training process, the existing multi-label feature selection methods can be divided into three categories: embedded, wrapper and filter methods. Wrapper and filter feature selection methods are performed in the preprocessing stage, in which the feature selection and model training processes are separated. Embedded feature selection methods are integrated into the model training process, and feature selection and model training conclusions are completed simultaneously. Embedded methods have gradually become the mainstream method for multi-label feature selection due to their high efficiency and effectiveness. For example, multi-label informed feature selection uses the latent semantics of multiple labels to guide the feature selection stage, performs feature selection by minimizing the joint norm, and performs joint feature selection by minimizing the joint norm of loss function and regularization. Although these methods are far superior to single-label feature selection methods by considering label correlation, due to the introduction of prior graph structure in these methods, this may be suboptimal or inaccurate, resulting in performance degradation. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a solution for characterizing websites containing illegal activities based on an adaptive subspace selection algorithm.

[0007] The first aspect of the present invention provides a method for characterizing websites containing illegal activities based on an adaptive subspace selection algorithm, the method comprising: Step S1, crawling website data using a crawler; Step S2: preprocess the crawled website data; Step S3: Use the BERT model to perform text classification to identify websites containing illegal activities; Step S4: Perform BERT text multi-label classification based on data dimension reduction on the identified websites containing illegal activities; Among them, the adaptive subspace feature extraction algorithm is used to perform data dimensionality reduction processing when extracting data features, and the original label space is converted into a low-dimensional label subspace to learn label correlation; the adaptive graph learning strategy is used to maintain the consistency of the feature space and the label subspace, and the subspace learning and label correlation learning are combined; Step S5: characterize the main body of each type of website containing illegal activities.

[0008] In step S1: configure a browser driver that is consistent with the local browser version; use Python's selenium module to download the warc.paths file in an automated manner, save the warc.paths file locally, access the corresponding addresses in sequence through website rules to perform the download task, and obtain the crawled website data.

[0009] In step S2: the crawled website data is confirmed to have missing values ​​for empty files, and the confirmed empty files are deleted. The missing values ​​are marked and the corresponding website data is retrieved, while maintaining the continuity of the date and the continuity of each file itself.

[0010] Step S3 includes: Word vector training: The BERT model uses pre-trained word vectors as input. The pre-training of word vectors includes Chinese word segmentation and part-of-speech tagging. Multi-classification layer design: The BERT model contains a multi-classification layer, and the softmax function is used to classify the feature map output by the BERT model to obtain the prediction probability of each category; Training sample preparation: The training sample includes input text and corresponding category labels; Pre-training: The BERT model is pre-trained using unlabeled text data to learn the relationship between words and context information to form the ability to represent the semantics of text; Model fine-tuning: After pre-training, use labeled text data to fine-tune the model and optimize the model parameters by minimizing the cross entropy loss function; Save parameters: save the parameters of the BERT model when training is completed; Perform classification: Use the trained BERT model to identify website data.

[0011] The second aspect of the present invention provides a system for characterizing websites containing illegal activities based on an adaptive subspace selection algorithm, the system comprising: The crawling unit is configured to: crawl website data using a crawler; The processing unit is configured to: perform data preprocessing on the crawled website data; The recognition unit is configured to: use the BERT model to perform text classification to identify websites containing illegal activities; The classification unit is configured to: perform BERT text multi-label classification based on data dimension reduction on the identified websites containing illegal activities; Among them, the adaptive subspace feature extraction algorithm is used to perform data dimensionality reduction processing when extracting data features, and the original label space is converted into a low-dimensional label subspace to learn label correlation; the adaptive graph learning strategy is used to maintain the consistency of the feature space and the label subspace, and the subspace learning and label correlation learning are combined; The characterization unit is configured to characterize the main body of each type of website containing illegal activities.

[0012] The crawling unit is specifically configured as follows: configure a browser driver that is consistent with the local browser version; use Python's selenium module to download the warc.paths file in an automated manner, save the warc.paths file locally, access the corresponding addresses in sequence through website rules to perform download tasks, and obtain crawled website data.

[0013] The processing unit is specifically configured to: confirm the missing values ​​of empty files in the crawled website data, delete the confirmed empty files, mark the missing values ​​and then reacquire the corresponding website data, while maintaining the continuity of the date and the continuity of each file itself.

[0014] The recognition unit is specifically configured to perform: Word vector training: The BERT model uses pre-trained word vectors as input. The pre-training of word vectors includes Chinese word segmentation and part-of-speech tagging. Multi-classification layer design: The BERT model contains a multi-classification layer, and the softmax function is used to classify the feature map output by the BERT model to obtain the prediction probability of each category; Training sample preparation: The training sample includes input text and corresponding category labels; Pre-training: The BERT model is pre-trained using unlabeled text data to learn the relationship between words and context information to form the ability to represent the semantics of text; Model fine-tuning: After pre-training, use labeled text data to fine-tune the model and optimize the model parameters by minimizing the cross entropy loss function; Save parameters: save the parameters of the BERT model when training is completed; Perform classification: Use the trained BERT model to identify website data.

[0015] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method for characterizing websites containing illegal behaviors based on an adaptive subspace selection algorithm described in the first aspect of the present disclosure is implemented.

[0016] The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for characterizing websites containing illegal activities based on an adaptive subspace selection algorithm as described in the first aspect of the present disclosure.

[0017] In summary, the present invention analyzes crawled website data, uses the BERT multi-classification model to extract website data containing illegal activities, then uses an adaptive subspace feature extraction algorithm to perform dimensionality reduction during feature extraction, and uses the BERT multi-label classification model to characterize the main body of websites containing illegal activities, and finally characterizes the main body of each type of website containing illegal activities. The present invention proposes solutions to solve the problems of unclear and difficult characterization of the main body of network public nuisance, and makes improvements to the problems of difficulty in obtaining evidence and tracing the main body of network public nuisance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of a method for characterizing websites containing illegal behaviors based on an adaptive subspace selection algorithm according to an embodiment of the present invention; Figure 2 A schematic diagram of data dimensionality reduction processing according to an embodiment of the present invention; Figure 3 A schematic diagram of crawling website data according to an embodiment of the present invention; Figure 4 The figure is a flow chart of identifying websites containing illegal activities based on the BERT model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0021] For ease of understanding, the main concepts involved in the present invention are clarified here: 1. Feature selection: It refers to the ability to select features with high correlation with the target variable in machine learning to improve the accuracy and efficiency of the model.

[0022] 2. Single-label feature: refers to the feature that each sample has only one label.

[0023] 3. Multi-label features: refers to the feature that each sample can have multiple labels.

[0024] 4. Multi-label informed feature selection: It is a feature selection technique for multi-label learning tasks. Its goal is to select a set of features that are helpful for improving classification performance for multiple labels.

[0025] 5. Websites containing illegal activities: refers to individuals or organizations that engage in illegal or gray area activities. In the Internet field, such entities are usually involved in various illegal activities, including but not limited to online fraud, hacker attacks, etc.

[0026] 6. BERT model: It is a pre-trained natural language processing model that can be used for various NLP tasks, including multi-classification tasks. BERT can effectively handle multi-classification problems such as text classification and sentiment analysis through its powerful language understanding ability and pre-trained representation.

[0027] 7. BERT multi-classification model: It refers to using the BERT model for multi-classification tasks, taking the pre-trained BERT model as the basis, and adding one or more fully connected layers on top to generate the final classification results.

[0028] 8. BERT multi-label classification model: refers to using the BERT model for multi-label classification tasks, taking the pre-trained BERT model as the basis, and adding one or more fully connected layers on top to generate the final classification results.

[0029] The present invention discloses a scheme for characterizing websites containing illegal activities based on an adaptive subspace selection algorithm. Data analysis is performed through crawled website data, and the BERT multi-classification model is used to extract website data containing illegal activities. Then, a feature extraction algorithm based on an adaptive subspace is used to reduce the dimension during feature extraction, and the original label space is converted into a low-dimensional label subspace to learn label correlation. An adaptive graph learning strategy is used to maintain the consistency of the feature space and the label subspace, and the subspace learning and label correlation learning are combined. The BERT multi-label classification model is used to characterize the main body of each type of website containing illegal activities, and a system for characterizing websites containing illegal activities is realized. The present invention proposes solutions to solve the problems of unclear and difficult characterization of the main body of network public nuisance, and makes improvements to the problems of difficulty in obtaining evidence and tracing the source of the main body of network public nuisance.

[0030] The first aspect of the present invention proposes a method for characterizing websites containing illegal activities based on an adaptive subspace selection algorithm, such as Figure 1 As shown, the method includes: Step S1, crawling website data using a crawler; Step S2: preprocess the crawled website data; Step S3: Use the BERT model to perform text classification to identify websites containing illegal activities; Step S4: Perform BERT text multi-label classification based on data dimension reduction on the identified websites containing illegal activities; Among them, the adaptive subspace feature extraction algorithm is used to perform data dimensionality reduction processing when extracting data features, and the original label space is converted into a low-dimensional label subspace to learn label correlation; the adaptive graph learning strategy is used to maintain the consistency of the feature space and the label subspace, and the subspace learning and label correlation learning are combined; Step S5: characterize the main body of each type of website containing illegal activities.

[0031] In step S1: configure a browser driver that is consistent with the local browser version; use Python's selenium module to download the warc.paths file in an automated manner, save the warc.paths file locally, access the corresponding addresses in sequence through website rules to perform the download task, and obtain the crawled website data.

[0032] In step S2: the crawled website data is confirmed to have missing values ​​for empty files, and the confirmed empty files are deleted. The missing values ​​are marked and the corresponding website data is retrieved, while maintaining the continuity of the date and the continuity of each file itself.

[0033] Step S3 includes: Word vector training: The BERT model uses pre-trained word vectors as input. The pre-training of word vectors includes Chinese word segmentation and part-of-speech tagging. Multi-classification layer design: The BERT model contains a multi-classification layer, and the softmax function is used to classify the feature map output by the BERT model to obtain the prediction probability of each category; Training sample preparation: The training sample includes input text and corresponding category labels; Pre-training: The BERT model is pre-trained using unlabeled text data to learn the relationship between words and context information to form the ability to represent the semantics of text; Model fine-tuning: After pre-training, use labeled text data to fine-tune the model and optimize the model parameters by minimizing the cross entropy loss function; Save parameters: save the parameters of the BERT model when training is completed; Perform classification: Use the trained BERT model to identify website data.

[0034] In some embodiments, the goal of characterizing websites containing illegal activities based on the adaptive subspace feature learning algorithm is to use crawlers to crawl large-scale website data, perform data preprocessing on large-scale website data, and then use the BERT model text classification to identify websites containing illegal activities, and perform multi-label classification on these website data. First, the original label space composed of number dealers, card dealers, data trafficking, etc. is transformed into a low-dimensional label subspace to learn label correlation; then, we use an adaptive graph learning strategy to maintain the consistency of the feature space and the label subspace. Finally, feature selection is performed by minimizing the linear mapping loss through sparse regularization, and discriminant features are selected for multi-label learning. Finally, the main body of websites containing illegal activities of different categories is characterized.

[0035] The problem addressed by the present invention is that the accuracy of the multi-label feature selection method is not high enough and the performance is not good enough. Specifically, the main concept of the method adopted by the present invention includes: A multi-label feature selection method based on adaptive subspace learning is used for the task of characterizing the characteristics of network pollution subjects. This is a feature extraction method for data dimensionality reduction, which can better optimize the above multi-label feature extraction tasks, such as Figure 2 Specifically, we first transform the original label space of the network public nuisance subject into a low-dimensional label subspace, capture label correlation by learning low-dimensional semantic representation, and select discriminative features for downstream multi-label learning. Then, during model training, we adopt an adaptive graph learning process to maintain the consistency of the label subspace and feature space instead of embedding the prior graph structure, thereby improving the robustness of the graph structure information and the effectiveness of feature selection.

[0036] In some embodiments, the characterization of websites containing illegal activities based on the adaptive subspace feature learning algorithm mainly includes crawling website data, data preprocessing, BERT text classification to identify websites containing illegal activities, converting the original label subspace into a low-dimensional label subspace, multi-label learning, and characterizing the main body of the website. The details are as follows: 1. Crawl website data. Use crawler technology to crawl website data. Data fields include URLs, domain names, survival status, titles and other fields; for example, use Python crawler programs to crawl common crawl data, or use web selenium automation technology to crawl more other website data.

[0037] 2. Data processing: Deal with missing values ​​and continuity issues of crawled website data.

[0038] 3. Identify websites containing illegal activities. First, the website data is cleaned using a rule-based method to extract websites containing keywords that are characteristic of illegal activities. Subsequently, the websites containing illegal activities are manually labeled to ensure the accuracy of the subsequent BERT model. Then, the BERT model is used for single-label learning to obtain websites under each label category.

[0039] 4. BERT text multi-label classification based on data dimensionality reduction. First, the original label subspace is converted into a low-dimensional label subspace. That is, the original label subspaces of number dealers, card dealers, data sellers, game dealers, code receiving platforms, etc. are converted into low-dimensional label subspaces. Adopting an adaptive graph learning strategy, discriminant features are selected for multi-label learning. After data dimensionality reduction, multi-label classification is performed through the BERT model, and each website containing illegal activities is characterized.

[0040] 5. Characterize the main body of websites containing illegal activities. Characterize the main body of different categories of website groups, and combine the results of step 3 with the results of step 4 to characterize the main body of several types of websites containing illegal activities.

[0041] In some embodiments, a chrome driver that is consistent with the local chrome version is first downloaded, and the warc.paths file of common crawl is downloaded in an automated manner using the selenium module of python. The file is saved locally, and then the file is used to access the corresponding addresses in sequence through the common crawl website rules to perform the download task, thereby obtaining a large amount of website data, such as Figure 3 As shown in the figure, due to the speed limit of the website, multi-threading is used to download data in parallel.

[0042] In some embodiments, the acquired website data is firstly judged as having missing values ​​in empty files, and then the empty files are deleted, and the website data is acquired again after marking the missing values, thereby ensuring the continuity of the date and the continuity of each file itself.

[0043] In some embodiments, with the rapid development of the Internet, the amount of text data continues to increase, and text classification tasks have become increasingly important. Among many text classification methods, the BERT model based on deep learning has shown strong performance. This part mainly uses the Chinese multi-classification BERT model to classify website data. First, screen out websites containing illegal activities, and then classify these websites. The process of identifying websites containing illegal activities based on the BERT text classification model is as follows: Figure 4 As shown, the details are as follows: (1) Word vector training: The Chinese multi-classification BERT model uses pre-trained word vectors as input. These word vectors need to be optimized before model training, such as Chinese word segmentation and part-of-speech tagging.

[0044] (2) Multi-classification layer design: In order to apply the Chinese multi-classification BERT model to multi-classification problems, we added a multi-classification layer based on the traditional BERT model. This layer uses the softmax function to classify the feature maps output by the BERT model and obtain the predicted probability of each category.

[0045] (3) Training sample preparation: When training the Chinese multi-classification BERT model, we need to prepare a certain amount of training samples. These samples include input text and corresponding category labels, which are used to train the model parameters.

[0046] (4) Pre-training: The Chinese multi-classification BERT model is pre-trained using large-scale unlabeled text data. In this process, the model gradually develops the ability to represent the semantics of text by learning the relationship between words and contextual information.

[0047] (5) Fine-tuning: After pre-training, we need to use labeled text data for fine-tuning. In this process, the model optimizes parameters by minimizing the cross entropy loss function, thereby improving the performance of the model on multi-classification problems.

[0048] (6) Save parameters: Save the parameters of the model currently trained to the local computer.

[0049] (7) Use the model for classification: Use the trained BERT model to classify website data.

[0050] In some embodiments, data dimensionality reduction processing is added to the BERT model. First, the original label space of the network public nuisance subject is converted into a low-dimensional label subspace, and the label correlation is captured by learning low-dimensional semantic representation, and discriminative features are selected for downstream multi-label learning. Then, during the model training process, we do not use the embedded prior graph structure, but adopt an adaptive graph learning process to maintain the consistency of the label subspace and the feature space, thereby improving the robustness of the graph structure information and the effectiveness of feature selection.

[0051] In some embodiments, the results of BERT text multi-classification and BERT text multi-label classification are combined to characterize several types of websites containing illegal activities.

[0052] Taking the common crawl website dataset as an example, we analyze the website data and characterize the main body of public nuisance websites. The specific steps are as follows: 1. Crawl website data. In this task, crawler technology is used to crawl common crawl website data. First, we should download the chrome driver 119.0.6045.124 (64-bit) that is consistent with the local chrome version, use python's selenium module to automatically download the warc.paths file of common crawl, save the file locally, and then use the file to access the corresponding addresses in sequence through the common crawl website rules to perform crawler download tasks and obtain a large amount of website data.

[0053] Due to the speed limit of the website, when downloading a certain file, the common crawl website will limit the download speed and adopt multi-process and multi-threading technology to crawl multiple sets of data in parallel.

[0054] 2. Data processing. First, the downloaded website data is checked for missing values ​​of empty files. Assuming that a file is about 1.2GB, the download path folder is traversed and the file size is calculated one by one. If the file is not less than 1.2GB, the next file size is calculated. Otherwise, the file is renamed to the original file name plus "-partial data". For example, if a file named "202406-1.warc.gz" is downloaded and the file is less than 1.2GB, the file is renamed to "202406-1-partial data.warc.gz".

[0055] When judging missing values, a second data download task is added, that is, downloading incomplete website data to ensure the continuity of the website data date and the continuity of each website data file itself.

[0056] (3) Identify websites containing illegal activities. Use a model based on BERT text classification to identify websites containing illegal activities. First, use a rule-based method to clean the website data and extract websites containing keywords that contain illegal activity characteristics. Then manually label the websites containing illegal activities to ensure the accuracy of the subsequent BERT model. Then use the BERT model for single-label learning to obtain the types of websites containing illegal activities.

[0057] (4) BERT text multi-label classification based on data dimensionality reduction. First, the original label subspace is converted into a low-dimensional label subspace. For example, the original label subspaces such as number dealers, card dealers, game dealers, platforms and users are converted into low-dimensional label subspaces. Adaptive graph learning strategy is adopted to select discriminant features for multi-label learning. After data dimensionality reduction, multi-label classification is performed through the BERT model, and each sample of a website containing illegal activities is characterized. Finally, a label group for each sample of a website containing illegal activities is obtained. The label group is [1,1,0,0,0,0,...], indicating that the website contains number dealers and card dealers, but no other information such as data trafficking, game dealers, and code receiving platforms. In the label group matrix, 1 indicates that the label is contained, and 0 indicates that the label is not contained.

[0058] (5) Characterizing the main body of Internet nuisance websites is to characterize the main body of different categories of website groups. Combining the results of identifying websites containing illegal activities with the results of BERT text multi-label classification based on data dimension reduction, the main body of several types of websites containing illegal activities is characterized. For example, each website in the website category has a label group, [1,1,0,0,0,0,...], which means that the website contains number dealers and card dealers, but no main body information such as data trafficking, game dealers, and code receiving platforms.

[0059] The second aspect of the present invention provides a system for characterizing websites containing illegal activities based on an adaptive subspace selection algorithm, the system comprising: The crawling unit is configured to: crawl website data using a crawler; The processing unit is configured to: perform data preprocessing on the crawled website data; The recognition unit is configured to: use the BERT model to perform text classification to identify websites containing illegal activities; The classification unit is configured to: perform BERT text multi-label classification based on data dimension reduction on the identified websites containing illegal activities; Among them, the adaptive subspace feature extraction algorithm is used to perform data dimensionality reduction processing when extracting data features, and the original label space is converted into a low-dimensional label subspace to learn label correlation; the adaptive graph learning strategy is used to maintain the consistency of the feature space and the label subspace, and the subspace learning and label correlation learning are combined; The characterization unit is configured to characterize the main body of each type of website containing illegal activities.

[0060] The crawling unit is specifically configured as follows: configure a browser driver that is consistent with the local browser version; use Python's selenium module to download the warc.paths file in an automated manner, save the warc.paths file locally, access the corresponding addresses in sequence through website rules to perform download tasks, and obtain crawled website data.

[0061] The processing unit is specifically configured to: confirm the missing values ​​of empty files in the crawled website data, delete the confirmed empty files, mark the missing values ​​and then reacquire the corresponding website data, while maintaining the continuity of the date and the continuity of each file itself.

[0062] The recognition unit is specifically configured to perform: Word vector training: The BERT model uses pre-trained word vectors as input. The pre-training of word vectors includes Chinese word segmentation and part-of-speech tagging. Multi-classification layer design: The BERT model contains a multi-classification layer, and the softmax function is used to classify the feature map output by the BERT model to obtain the prediction probability of each category; Training sample preparation: The training sample includes input text and corresponding category labels; Pre-training: The BERT model is pre-trained using unlabeled text data to learn the relationship between words and context information to form the ability to represent the semantics of text; Model fine-tuning: After pre-training, use labeled text data to fine-tune the model and optimize the model parameters by minimizing the cross entropy loss function; Save parameters: save the parameters of the BERT model when training is completed; Perform classification: Use the trained BERT model to identify website data.

[0063] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method for characterizing websites containing illegal behaviors based on an adaptive subspace selection algorithm described in the first aspect of the present disclosure is implemented.

[0064] The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for characterizing websites containing illegal activities based on an adaptive subspace selection algorithm as described in the first aspect of the present disclosure.

[0065] In summary, the present invention analyzes crawled website data, uses the BERT multi-classification model to extract website data containing illegal activities, then uses an adaptive subspace feature extraction algorithm to perform dimensionality reduction during feature extraction, and uses the BERT multi-label classification model to characterize the main body of websites containing illegal activities, and finally characterizes the main body of each type of website containing illegal activities. The present invention proposes solutions to solve the problems of unclear and difficult characterization of the main body of network public nuisance, and makes improvements to the problems of difficulty in obtaining evidence and tracing the main body of network public nuisance.

[0066] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above-mentioned embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

Claims

1. A method for characterizing websites containing illegal activities based on an adaptive subspace selection algorithm, characterized in that: The method comprises: Step S1, crawling website data using a crawler; Step S2: preprocess the crawled website data; Step S3: Use the BERT model to perform text classification to identify websites containing illegal activities; Step S4: Perform BERT text multi-label classification based on data dimension reduction on the identified websites containing illegal activities; Among them, the adaptive subspace feature extraction algorithm is used to perform data dimensionality reduction processing when extracting data features, and the original label space is converted into a low-dimensional label subspace to learn label correlation; the adaptive graph learning strategy is used to maintain the consistency of the feature space and the label subspace, and the subspace learning and label correlation learning are combined; Step S5: characterize the main body of each type of website containing illegal activities.

2. According to claim 1, a method for characterizing websites containing illegal activities based on an adaptive subspace selection algorithm is characterized in that: In step S1: configure a browser driver that is consistent with the local browser version; use Python's selenium module to download the warc.paths file in an automated manner, save the warc.paths file locally, access the corresponding addresses in sequence through website rules to perform the download task, and obtain the crawled website data.

3. According to claim 2, a method for characterizing websites containing illegal activities based on an adaptive subspace selection algorithm is characterized in that: In step S2: the crawled website data is confirmed to have missing values ​​for empty files, and the confirmed empty files are deleted. The missing values ​​are marked and the corresponding website data is retrieved, while maintaining the continuity of the date and the continuity of each file itself.

4. According to claim 3, a method for characterizing websites containing illegal activities based on an adaptive subspace selection algorithm is characterized in that: Step S3 includes: Word vector training: The BERT model uses pre-trained word vectors as input. The pre-training of word vectors includes Chinese word segmentation and part-of-speech tagging. Multi-classification layer design: The BERT model contains a multi-classification layer, and the softmax function is used to classify the feature map output by the BERT model to obtain the prediction probability of each category; Training sample preparation: The training sample includes input text and corresponding category labels; Pre-training: The BERT model is pre-trained using unlabeled text data to learn the relationship between words and context information to form the ability to represent the semantics of text; Model fine-tuning: After pre-training, use labeled text data to fine-tune the model and optimize the model parameters by minimizing the cross entropy loss function; Save parameters: save the parameters of the BERT model when training is completed; Perform classification: Use the trained BERT model to identify website data.

5. A system for characterizing websites containing illegal activities based on an adaptive subspace selection algorithm, characterized in that: The system comprises: The crawling unit is configured to: crawl website data using a crawler; The processing unit is configured to: perform data preprocessing on the crawled website data; The recognition unit is configured to: use the BERT model to perform text classification to identify websites containing illegal activities; The classification unit is configured to: perform BERT text multi-label classification based on data dimension reduction on the identified websites containing illegal activities; Among them, the adaptive subspace feature extraction algorithm is used to perform data dimensionality reduction processing when extracting data features, and the original label space is converted into a low-dimensional label subspace to learn label correlation; the adaptive graph learning strategy is used to maintain the consistency of the feature space and the label subspace, and the subspace learning and label correlation learning are combined; The characterization unit is configured to characterize the main body of each type of website containing illegal activities.

6. According to claim 5, a system for characterizing websites containing illegal activities based on an adaptive subspace selection algorithm is characterized in that: The crawling unit is specifically configured as follows: configure a browser driver that is consistent with the local browser version; use Python's selenium module to download the warc.paths file in an automated manner, save the warc.paths file locally, access the corresponding addresses in sequence through website rules to perform download tasks, and obtain crawled website data.

7. The system for characterizing websites containing illegal behaviors based on the adaptive subspace selection algorithm according to claim 6 is characterized in that: The processing unit is specifically configured to: confirm the missing values ​​of empty files in the crawled website data, delete the confirmed empty files, mark the missing values ​​and then reacquire the corresponding website data, while maintaining the continuity of the date and the continuity of each file itself.

8. The system for characterizing websites containing illegal behaviors based on the adaptive subspace selection algorithm according to claim 7 is characterized in that: The recognition unit is specifically configured to perform: Word vector training: The BERT model uses pre-trained word vectors as input. The pre-training of word vectors includes Chinese word segmentation and part-of-speech tagging. Multi-classification layer design: The BERT model contains a multi-classification layer, and the softmax function is used to classify the feature map output by the BERT model to obtain the prediction probability of each category; Training sample preparation: The training sample includes input text and corresponding category labels; Pre-training: The BERT model is pre-trained using unlabeled text data to learn the relationship between words and context information to form the ability to represent the semantics of text; Model fine-tuning: After pre-training, use labeled text data to fine-tune the model and optimize the model parameters by minimizing the cross entropy loss function; Save parameters: save the parameters of the BERT model when training is completed; Perform classification: Use the trained BERT model to identify website data.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the method for characterizing websites containing illegal activities based on an adaptive subspace selection algorithm as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the method for characterizing websites containing illegal activities based on an adaptive subspace selection algorithm as described in any one of claims 1 to 4.

Images