Illegal website gang identification method and system based on multi-dimensional feature fusion

Abstract

The invention provides an illegal website gang identification method and system based on multi-dimensional feature fusion, and relates to the field of network security, the method comprises the following steps: capturing network traffic data in real time through a traffic mirroring technology or a network probe to collect passive traffic data; storing the collected data, and extracting multi-dimensional features from a target website through an active request to realize feature extraction; performing feature fusion and multi-dimensional analysis on the extracted multi-dimensional features to obtain fused features and analysis results; according to the fusion features and the analysis result, training and optimizing a machine learning model to obtain a trained model; and carrying out illegal gang identification and association analysis on new website data by utilizing the trained model so as to construct a network topology structure of illegal gangs. According to the method, efficient and accurate illegal website gang identification is realized, and the real-time performance and comprehensiveness of network security monitoring are improved.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to an illegal website group identification method and system based on multi-dimensional feature fusion. Background Art

[0002] In recent years, the activities of illegal website gangs have become increasingly rampant, especially in the fields of cybercrime and data theft, which have brought great harm to social security and economic development. Traditional detection methods mainly rely on rule matching and manual review, which are not only inefficient, but also have problems such as insufficient accuracy, slow response speed, and high false positive rate. It is difficult to cope with the increasingly complex and covert operation mode of illegal website gangs.

[0003] With the development of big data analysis and artificial intelligence technology, how to accurately identify and crack down on illegal website groups through automated and intelligent means, combined with multi-dimensional feature fusion and machine learning technology, has become a key issue that needs to be urgently addressed in the current network security field. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method and system for identifying illegal website groups based on multi-dimensional feature fusion. The present invention realizes efficient and accurate identification of illegal website groups and improves the real-time and comprehensiveness of network security detection.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows: In a first aspect, a method for identifying illegal website groups based on multi-dimensional feature fusion is provided, the method comprising: Capture network traffic data in real time through traffic mirroring technology or network probes to collect passive traffic data; The collected data is saved, and multi-dimensional features are extracted from the target website through active requests to achieve feature extraction; The extracted multi-dimensional features are subjected to feature fusion and multi-dimensional analysis to obtain fusion features and analysis results; According to the fusion features and analysis results, the machine learning model is trained and optimized to obtain a trained model; By using the trained model to identify and associate new website data with illegal groups, the network topology of illegal groups can be constructed. According to the network topology of the illegal gang, anomaly detection is performed on unknown illegal websites, and the model is updated based on the anomaly detection to identify the illegal website gang.

[0006] Preferably, network traffic data is captured in real time by traffic mirroring technology or network probes to collect passive traffic data, including: Deploy traffic mirroring technology or network probes at the core nodes of the network environment; Capture network traffic data in real time through traffic mirroring technology or network probes, and combine with preliminary filtering rules to collect passive traffic data.

[0007] Preferably, the collected data is saved, and multi-dimensional features are extracted from the target website through active request to achieve feature extraction, including: Store the collected data in structured and unstructured forms; Through active requests, multi-dimensional features are extracted from the target website to achieve feature extraction, including structural features, domain name and IP feature extraction, traffic features and behavior pattern features.

[0008] Preferably, the extracted multidimensional features are subjected to feature fusion and multidimensional analysis to obtain fused features and analysis results, including: The extracted multi-dimensional features are cleaned and fused to obtain fused features; Through feature dimension reduction and data mining technology, the potential correlation between target websites is analyzed to obtain analysis results.

[0009] Preferably, the machine learning model is trained and optimized according to the fusion features and analysis results to obtain a trained model, including: Build classification models based on labeled datasets; According to the fusion features and analysis results, the model is trained through the machine learning algorithm to obtain a trained machine learning model; Tune and verify the trained machine learning model to obtain a trained model.

[0010] Preferably, the network topology of illegal gangs is constructed by using the trained model to identify and associate new website data, including: By using the trained model to analyze new website data in real time, the illegal probability is predicted to obtain the prediction result; Based on the prediction results, explore the correlation with known illegal gangs to obtain correlation analysis results; Based on the results of correlation analysis, the network topology structure of the illegal gang is constructed.

[0011] Preferably, according to the network topology of the illegal group, anomaly detection is performed on unknown illegal websites, and the model is updated according to the anomaly detection to realize the identification of the illegal website group, including: Anomaly detection of unknown illegal websites is performed through unsupervised learning algorithms to obtain detection results; Update the sample dataset and retrain the model according to the detection results to update the model; Based on the updated model, the identification of illegal website gangs can be achieved.

[0012] In the second aspect, a system for identifying illegal website groups based on multi-dimensional feature fusion includes: The acquisition module is used to capture network traffic data in real time through traffic mirroring technology or network probes to collect passive traffic data; extract multi-dimensional features from the target website through active requests to achieve feature extraction; perform feature fusion and multi-dimensional analysis on the extracted multi-dimensional features to obtain fused features and analysis results; The processing module is used to train and optimize the machine learning model according to the fusion features and analysis results to obtain a trained model; to identify illegal gangs and conduct association analysis on new website data by using the trained model to build the network topology structure of illegal gangs; to perform anomaly detection on unknown illegal websites according to the network topology structure of illegal gangs, and to update the model according to the anomaly detection to realize the identification of illegal website gangs.

[0013] According to a third aspect, a computing device includes: one or more processors; The storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method described.

[0014] In a fourth aspect, a computer-readable storage medium stores a program, and when the program is executed by a processor, the method described is implemented.

[0015] The above solution of the present invention includes at least the following beneficial effects: The present invention adopts multi-dimensional feature fusion technology and realizes accurate identification of illegal websites through machine learning models, which significantly improves the accuracy and efficiency of identification compared with traditional detection methods. Through comprehensive analysis of multiple dimensions such as structural features, domain name features, traffic features and behavioral features, it is possible to discover illegal gang behaviors hidden in complex network structures and reveal the potential connections and organizational patterns between gang members. Through anomaly detection and feedback mechanisms, unknown illegal websites and their related information are continuously discovered, and the identification model is dynamically updated to improve the coverage and robustness of the system and ensure adaptation to rapid changes in the network environment.

[0016] The network topology structure of illegal website groups constructed by the present invention provides a panoramic view for regulatory agencies, helps accurately locate the core members and key nodes of illegal website groups, and improves the ability to combat and govern illegal activities. It is not only suitable for the identification of illegal website groups, but also can be applied to the detection of domain name-based network attack events such as botnets and malware. It is a practical technical solution suitable for complex network security scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of an illegal website group identification method based on multi-dimensional feature fusion provided by an embodiment of the present invention.

[0018] Figure 2 It is a schematic diagram of an illegal website group identification system based on multi-dimensional feature fusion provided by an embodiment of the present invention.

[0019] Figure 3 It is a schematic diagram of the overall functional structure of the present invention. DETAILED DESCRIPTION

[0020] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in a form and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0021] like Figure 1 As shown, an embodiment of the present invention proposes a method for identifying illegal website groups based on multi-dimensional feature fusion, and the method includes the following steps: Step 11, capturing network traffic data in real time through traffic mirroring technology or network probes to collect passive traffic data; Step 12, saving the collected data, and extracting multi-dimensional features from the target website through active request to achieve feature extraction; Step 13, performing feature fusion and multidimensional analysis on the extracted multidimensional features to obtain fused features and analysis results; Step 14: training and optimizing the machine learning model according to the fusion features and analysis results to obtain a trained model; Step 15, using the trained model to perform illegal group identification and association analysis on new website data to construct a network topology structure of illegal groups; Step 16, based on the network topology of the illegal group, perform anomaly detection on unknown illegal websites, and update the model based on the anomaly detection to identify the illegal website group.

[0022] In the embodiment of the present invention, network traffic data is captured in real time to ensure the timeliness and freshness of the data, which provides the possibility for timely detection of illegal website gang activities; through the traffic mirroring or network probe of the core node, all relevant traffic flowing through the node can be captured. Traffic mirroring and network probe technology usually do not require direct access or modification to the target website, which reduces the impact on the operation of the target website and improves the security and reliability of data collection. Multidimensional data such as structural features, domain name information, traffic features and behavior pattern features extracted by active request provide a rich source of information for the identification of illegal website gangs; multidimensional feature extraction helps to deeply understand the behavior patterns and features of the target website; active request tools can automatically extract features, reduce manual intervention, and improve data processing efficiency and accuracy. Feature fusion and multidimensional analysis can combine features of different dimensions to form a more comprehensive feature representation, which helps to improve the accuracy of illegal website gang identification; in the process of feature fusion, the data can be cleaned and standardized to eliminate noise and redundant information and improve data quality. Multidimensional analysis helps to discover patterns and associations hidden in the data, providing important clues for gang identification. By training and optimizing the machine learning model, the model's ability to identify illegal website gangs and its robustness can be improved; the optimized model can better adapt to the ever-changing network environment and illegal website gang behavior patterns; the trained model can automatically identify illegal gangs for newly input website data, reduce manual intervention, and improve processing efficiency. The trained model can identify illegal gangs for newly input website data in real time to ensure timely discovery and response to illegal activities. Through association analysis, the association relationship between illegal website gangs can be revealed, which helps to locate the core members and key nodes of the gang. Based on the results of association analysis, the network topology of illegal gangs is constructed to provide an intuitive network view for regulatory authorities to understand and respond to illegal activities. By performing anomaly detection on unknown illegal websites, potential illegal activities can be discovered and the coverage and detection capabilities of the system can be improved; the model can be dynamically updated according to the anomaly detection results to ensure that the model can adapt to the ever-changing behavior patterns of illegal website gangs and improve the accuracy and robustness of identification; by continuously updating the model and data set, the performance of the illegal website gang identification system can be continuously improved, and its effect and value in practical applications can be improved.

[0023] In a preferred embodiment of the present invention, the above step 11, capturing network traffic data in real time by using traffic mirroring technology or network probes to collect passive traffic data, includes: Step 111, deploying traffic mirroring technology or network probes at core nodes of the network environment; Step 112, capturing network traffic data in real time through traffic mirroring technology or network probes, and combining preliminary filtering rules to collect passive traffic data.

[0024] In the embodiment of the present invention, the deployment of traffic mirroring technology or network probes at the core node can capture all network traffic flowing through the node, ensure comprehensive monitoring of the target website, and avoid missing key information; the core node is usually the convergence point of network traffic, and the traffic mirror or probe deployed here can efficiently collect a large amount of traffic data; the traffic mirroring technology and network probe have flexible deployment methods, and can select a suitable deployment location according to the network architecture and requirements to ensure the effectiveness and reliability of data collection. Real-time capture of network traffic data ensures the timeliness and freshness of the data, and can promptly discover and handle illegal website gang activities; combined with preliminary filtering rules, the captured traffic data is preliminarily screened and filtered to remove irrelevant or low-priority traffic; through filtering rules, it is ensured that the collected passive traffic data is highly relevant to the illegal website gang identification task, and the accuracy and pertinence of data analysis are improved; preliminary filtering rules help reduce unnecessary traffic data collection and processing, optimize system resource usage, and reduce operating costs.

[0025] In a specific embodiment of the present invention, step 11, the specific implementation steps include: (1) gateway / switch (core node), which is located at the core node of the network environment, is responsible for transmitting data and mirroring the traffic to the traffic collection module, which is the entrance of passive traffic collection; (2) passive traffic collection, which uses traffic mirroring technology or network probes to capture the network traffic data of the target website in real time, and only collects traffic related to the target website through preliminary filtering rules to ensure the accuracy and efficiency of subsequent analysis; (3) active request tool (web crawler), which simulates user behavior and crawls the website page content by actively requesting the target website, extracting structural features and hyperlink information; (4) active detection of traffic, using the active request tool to obtain the website's structural features (such as page layout, hyperlink structure), DNS record data and domain name information (such as registration time, operator affiliation, etc.).

[0026] In another specific embodiment of the present invention, a traffic collection module is deployed at a core node of the network environment (such as a gateway, switch or server outlet), and traffic mirroring technology or a network probe is used to capture the passing network traffic data packets in real time. The traffic collection module can be implemented based on software (such as Wireshark, Suricata, Zeek) or hardware devices; the target network traffic is copied to the traffic analysis device through a mirror port (PortMirroring); the meta information of the transmitted data packet is non-invasively obtained without affecting the normal operation of the target website; the preliminary filtering rules are deployed at the traffic mirror port, which can reduce the introduction of irrelevant traffic in the first step of data collection and reduce the complexity of subsequent data processing, wherein the preliminary filtering rules include: filtering specific network protocol traffic through a protocol type identifier (such as a protocol header), such as limiting traffic to common protocols such as HTTP, HTTPS, DNS, FTP, etc., and excluding irrelevant or low-priority protocols (such as IGMP, NetBIOS, etc.); filtering according to the source IP address or target IP address of the data packet, such as only capturing traffic from a specific subnet (such as 192.168.1.0 / 24) or only collecting traffic from the target server (such as the target IP is 10.0.0.1); filtering based on the source port or target port number of the data packet.

[0027] In a preferred embodiment of the present invention, the above step 12 saves the collected data and extracts multi-dimensional features from the target website through active request to achieve feature extraction, including: Step 121, storing the collected data in structured and unstructured forms; Step 122, through active request, extract multi-dimensional features from the target website to achieve feature extraction, including structural features and domain name and IP feature extraction, traffic features and behavior pattern features.

[0028] In the embodiment of the present invention, the storage method of structured and unstructured data can support the storage of different types of data, including text, images, videos, logs and other formats, ensuring the integrity and richness of data information. Structured data is stored through relational databases or NoSQL databases, which is convenient for efficient query, analysis and management; unstructured data can be stored through file systems, object storage and other methods to meet the processing requirements of large amounts of data. Extracting multidimensional features can comprehensively cover all aspects of the target website, including page structure, domain name and IP information, traffic behavior, user interaction mode, etc., and provide comprehensive data support for the identification of illegal website gangs; multidimensional feature extraction helps to deeply explore the characteristic information of the target website, reveal its potential behavior patterns and associations, and improve the accuracy and reliability of illegal website gang identification; the extracted multidimensional features can be flexibly combined according to specific needs to construct different feature vectors to adapt to different identification tasks and analysis scenarios; rich feature data can provide more training information for machine learning models, which helps to improve the classification performance, generalization ability and robustness of the model, and improve the efficiency and effect of illegal website gang identification.

[0029] In a specific embodiment of the present invention, step 12, the specific implementation steps include: Step 1: Network data collection: By deploying web crawler technology, we can periodically or in real time crawl the public data of websites, including web page content, HTML structure, domain name information, access records, etc. The collected data is stored in structured and unstructured forms. Let the collected original data set be: , where each Is the website i The network data collection module can combine multi-threaded crawling technology or distributed crawler framework (such as Scrapy) to improve the collection efficiency, which is especially suitable for continuous monitoring of large-scale target websites. In addition, preferably, the collection cycle can be dynamically adjusted according to the update frequency of the target website to ensure the timeliness and integrity of the data.

[0030] Step 2, structural feature extraction, use data analysis tools to analyze the captured web page content, extract information such as page content layout, page link structure, URL path features, etc. For example, use the HTML analysis library to extract the number of hyperlinks in the page, the tag nesting level, the resource loading path, etc., analyze the web page content and extract its structural feature set as follows: ,in, Represents relevant features such as the number of hyperlinks, tag nesting levels, resource loading paths, etc. In the process of structural feature extraction, web page semantic analysis technology can be combined to extract higher-level features, such as the distribution of page functional modules and the complexity of the user interface. At the same time, preferably, the parsing efficiency can be improved by the block technology of HTML content (such as DOM tree parsing), which is particularly suitable for processing scenarios with complex web page structures.

[0031] Step 3: Extract traffic features and analyze website access behaviors from passive traffic, including user access frequency, access time interval, source IP distribution, etc. Use traffic collection tools (such as Wireshark or custom traffic analysis modules) to obtain and organize relevant traffic data. The feature set is: , where each Including the visited target, visited IP, visit frequency, etc. In the process of traffic feature extraction, the time window division technology (such as sliding window) can be combined to analyze the short-term and long-term traffic behavior characteristics. In addition, preferably, a distributed traffic capture architecture can be used to achieve efficient collection and processing in a high-concurrency network environment, thereby supporting large-scale traffic feature analysis tasks.

[0032] Step 4: Extract domain name and IP features. By querying domain name registration information (such as WHOIS data), extract the domain name registrant information, registration time, registrar, etc.; obtain the IP's geographical location, affiliated operator, etc. through IP address location analysis tools. The feature set is: . Among them, each Including registration time, geographic location, operator information, etc. In the process of domain name and IP feature extraction, historical WHOIS data and IP geographical distribution change records can be combined to further analyze the evolution trend of the target website. In addition, preferably, the accuracy and coverage of the data are improved by integrating multiple public domain name information databases (such as RDAP, ARIN, APNIC).

[0033] Step 5, behavior pattern feature extraction, extract the website operation frequency, user interaction pattern, abnormal behavior pattern, etc. based on the access log and request response data. Mark multiple identical operations or batch request behaviors as potential abnormal features. The feature set is: , where each Including repeated requests, batch access patterns, abnormal path requests, etc. In the process of extracting behavioral pattern features, anomaly detection algorithms (such as isolation forests) can be combined to automatically mark high-frequency operations and batch request behaviors. In addition, preferably, for behavioral patterns across time periods, time series analysis techniques (such as LSTM) can be introduced to predict potential abnormal operation trends.

[0034] In a preferred embodiment of the present invention, the above step 13, performing feature fusion and multi-dimensional analysis on the extracted multi-dimensional features to obtain fused features and analysis results, includes: Step 131, performing data cleaning and fusion on the extracted multi-dimensional features to obtain fusion features; Step 132, analyzing the potential correlation between target websites through feature dimension reduction and data mining technology to obtain analysis results.

[0035] In the embodiment of the present invention, the data cleaning process can remove noise data, invalid data and outliers, and improve the accuracy and reliability of the data; the fusion of features of different dimensions can form a more comprehensive feature representation, which helps to capture the complex associations and potential patterns between target websites and improve the accuracy of recognition; through feature fusion, the redundant information between features can be reduced, the complexity of subsequent analysis can be reduced, and the computing efficiency can be improved; the fused features are easier to be understood and used by the machine learning model, which helps to improve the classification and prediction performance of the model. Feature dimensionality reduction technology can reduce the dimension of data while retaining the main information, reduce the amount of calculation and storage space, and improve the analysis efficiency. Data mining technology can mine hidden association rules, patterns and information from large amounts of data, reveal the potential correlation between target websites, and provide important clues for the identification of illegal website gangs.

[0036] In a specific embodiment of the present invention, step 13, the specific implementation steps include: Step 1: Feature cleaning and standardization: Clean the extracted multidimensional feature data and delete noise data and invalid data (such as irrelevant page content or erroneous traffic records). Normalize the numerical features and convert the data to a unified dimension range. Automated data cleaning tools (such as Pandas and Dask) can be introduced in the feature cleaning process to improve processing efficiency. In addition, preferably, the processing of outliers can be combined with statistical analysis methods (such as Z-score method or IQR method) to ensure that the standardized results of the data are more accurate.

[0037] Step 2, feature fusion, uses feature engineering methods to fuse multi-dimensional features. Perform one-hot encoding on categorical features, and bucket or discretize continuous features. Combined with feature importance analysis, select key features to reduce redundancy. The specific method for feature fusion is: (1) Classification feature coding Traverse the classification feature collection For each categorical feature , perform one-hot encoding, and convert each categorical variable Converted into a binary vector. For example, the input is "A","B" , the corresponding output is This method can be used to obtain the encoded classification feature matrix .

[0038] (2) Continuous feature encoding Traversing continuous feature sets For each continuous feature , perform bucketing or discretization. Optional operations include equal-width bucketing and equal-frequency bucketing. Equal-width bucketing divides the data into equal-width buckets. Intervals: , equal frequency bucketing is based on the distribution of data. Divided into intervals to ensure that the amount of data in each interval is roughly the same; discretize the processed continuous features into categorical variables. Obtain the continuous feature matrix after bucketing .

[0039] (3) Feature importance analysis The processed feature set (including and ) is combined into a preliminary feature matrix And perform feature importance analysis, by combining Lasso regression and decision tree feature importance analysis, comprehensively evaluate the features and screen out key features. The specific method is as follows: Lasso regression (L1 regularization): Perform Lasso regression to obtain the weight of each feature . The screening weight is greater than the threshold Features: ; Decision Tree Feature Importance: Calculate the importance score of each feature using a decision tree model . Filter importance above threshold Features: .

[0040] (4) Feature Fusion Combine the selected important features with the original multidimensional feature set Fusion, combining all processed features, to obtain , and finally generate a unified feature vector .

[0041] (5) Embedded feature method (processing high-dimensional sparse features) For high-dimensional sparse features (such as categorical features after One-Hot encoding): Use an embedding layer to map high-dimensional features into a low-dimensional space, where , the embedded low-dimensional features Combine with other features (such as structural features, traffic features, domain name features, and behavioral features) to form the final feature vector : , It is a multidimensional feature after feature selection. is the low-dimensional representation of the sparse features obtained by mapping the embedding layer.

[0042] Step 3: feature dimensionality reduction, using dimensionality reduction methods such as principal component analysis (PCA) or linear discriminant analysis (LDA) to reduce feature dimensions, retain key features while reducing computational complexity, where: ,in, W is a dimensionality reduction transformation matrix; in the process, clustering dimensionality reduction methods based on feature correlation (such as t-SNE and UMAP) can be combined to process nonlinearly distributed data. In addition, preferably, the dimension of dimensionality reduction target can be dynamically adjusted, and a suitable dimensionality reduction scheme can be selected according to the data scale and the need to balance computational complexity.

[0043] Step 4: Preliminary analysis of illegal association behaviors. For the integrated feature data, use data mining algorithms (such as association rule mining) to analyze and find potential associations between websites, such as similar registrant information, common access sources, etc. At the same time, combine association rule mining and weighted association rule mining in the time dimension to achieve dynamic behavior pattern recognition and association strength quantification. The following are the specific methods: (1) Multidimensional association rule mining Extract the fused feature data, analyze it using the association rule mining algorithm, and identify the highly correlated feature item sets between websites. Filter the highly correlated rules by calculating the following indicators: Support: measure the feature item set The proportion of co-occurrence in all samples, ; Confidence: A measure of the probability of a feature occurring. Under the premise of Probability: ; Lift: Measuring feature items and The strength of the association between: ; By screening the rules that meet the minimum support, confidence and lift thresholds, high-relevance rules are obtained.

[0044] (2) Dynamic correlation analysis in the time dimension Divide the sample data into multiple time windows in chronological order: ,in, and Respectively represent the start time and end time of the time window; in each time window, the support and confidence are calculated independently to extract the website association rules in the current time period: ; Compare the changes in rule strength in adjacent time windows to identify the dynamic evolution trend of association rules: Support ; Dynamic analysis based on the time dimension captures the behavioral evolution patterns and time-sensitive characteristics of illegal website gangs.

[0045] (3) Weighted association rule modeling Introduce weights based on the importance of different samples and features , perform weighted modeling on association rules to improve the accuracy of rule results. Calculate weighted support and weighted confidence respectively: Weighted support: ,in, For sample The weight of is the indicator function. Weighted confidence: .

[0046] (4) Association rule result output Output high-association rules that meet the support, confidence, and lift requirements: ; Combined with the time dimension, the dynamic evolution trend of website association rules is generated: ; Output weighted association rules to quantify the strength of association between websites: .

[0047] In a preferred embodiment of the present invention, the above step 14, training and optimizing the machine learning model according to the fusion features and the analysis results to obtain a trained model, includes: Step 141, constructing a classification model based on the labeled data set; Step 142, performing model training through a machine learning algorithm according to the fusion features and the analysis results to obtain a trained machine learning model; Step 143, tune and verify the trained machine learning model to obtain a trained model.

[0048] In the embodiment of the present invention, the classification model is constructed based on the labeled data set, which can clarify the training objectives and classification standards of the model, and ensure the applicability and effectiveness of the model in the illegal website gang identification task; the labeled data set provides rich sample data for model training, including known illegal website gang samples and legal website samples, which helps the model learn effective classification rules; by constructing the classification model, the structure and parameters of the model can be preliminarily determined. Using the fusion features and analysis results as the input of model training, the information of multidimensional features can be fully utilized to improve the recognition ability and accuracy of the model; selecting a suitable machine learning algorithm for training, such as support vector machine, random forest, deep neural network, etc., can give full play to the advantages of the algorithm and improve the classification performance of the model; through the training process, the model can learn the association rules and feature patterns between the target website and the illegal website gang. Through the tuning and verification process, the hyperparameters and structure of the model can be optimized, the classification accuracy and generalization ability of the model can be improved, and it can be better adapted to the actual application scenario; the verification process can evaluate the performance of the model on different data sets, ensure the stability and reliability of the model, and reduce the possibility of false positives and false negatives; the tuned model can better adapt to the changes in the network environment and the evolution of the behavior pattern of the illegal website gang, and maintain long-term effectiveness and practicality.

[0049] In a specific embodiment of the present invention, step 14, the specific implementation steps include: Step 1: Sample data annotation. By combining manual and automatic annotation, a labeled data set is constructed, including known illegal website gang samples and legal website samples. Illegal sample annotation is based on existing blacklists or expert analysis results. The data set is defined as: ; In order to improve the efficiency and accuracy of labeling, a semi-supervised learning method can be introduced to predictively label unlabeled samples, and expert feedback can be used to further correct the sample labeling results. In addition, preferably, the labeling data set can be continuously expanded in combination with a dynamically updated blacklist of illegal websites or the latest detection results to improve the model's ability to identify new illegal gang behaviors.

[0050] Step 2: Model selection and construction: select a machine learning model suitable for multi-classification problems, including but not limited to support vector machines, random forests, deep neural networks, etc., and build an initial model based on multi-dimensional features. Input feature set X , output the prediction results: ; In the initial model construction, the model structure suitable for high-dimensional data can be preferred. For example, deep neural networks can be combined with feature hierarchical processing to improve the ability to express complex feature interactions. At the same time, it is preferred to select key features (such as domain name registration time, traffic patterns, abnormal behavior patterns) through feature importance analysis to reduce the impact of redundant features on model performance.

[0051] Step 3, model training, divide the labeled data set into a training set and a validation set, use the training set to train the model, and improve the classification ability by optimizing the model's loss function (such as cross entropy loss). In model training, introduce data enhancement technology to perform noise simulation, feature perturbation and other processing on existing data, thereby improving the model's robustness to noisy data and diverse illegal website behaviors. In addition, preferably, the model training process can be accelerated through a distributed training framework (such as TensorFlow, PyTorch) to support the processing of large-scale data sets.

[0052] Step 4, model optimization, using grid search and cross-validation techniques to adjust model hyperparameters. The adjusted hyperparameters include but are not limited to learning rate, tree depth, regularization coefficient, Dropout ratio, etc., and the specific selection depends on the structure and characteristics of the target model. For example, for deep neural networks, it is preferred to adjust the learning rate and regularization coefficient to prevent overfitting; for random forest models, it is preferred to adjust the depth and number of trees to balance the complexity and performance of the model. In addition, preferably, the early stopping technique can be combined in the optimization process to avoid unnecessary computational overhead.

[0053] Step 5, model evaluation, use the validation set to evaluate the model performance. Preferably, the validation set is used to perform performance evaluation on the optimized model; wherein, the evaluation indicators include but are not limited to accuracy, recall rate, F1 score, etc., preferably ensuring the applicability of the model in the task of identifying illegal website gangs. During the evaluation process, the indicators can be weighted according to the task requirements, for example, a higher weight can be given to the recall rate of illegal websites to reduce underreporting. In addition, preferably, the global performance of the model can be evaluated in combination with the AUC curve, etc., to further quantify the classification ability of the model at different thresholds. In order to improve the accuracy of the evaluation, a stratified sampling method can also be used to ensure that the data distribution of the validation set is consistent with the actual scenario.

[0054] In a preferred embodiment of the present invention, the above step 15, by using the trained model to perform illegal group identification and association analysis on new website data to construct the network topology structure of the illegal group, includes: Step 151, using the trained model to perform real-time analysis on new website data, predicting its illegal probability, and obtaining a prediction result; Step 152, based on the prediction result, mining its correlation with known illegal gangs to obtain correlation analysis results; Step 153, constructing the network topology structure of the illegal gang based on the correlation analysis result.

[0055] In the embodiment of the present invention, the real-time prediction module of the present invention supports traffic analysis in a high-concurrency environment, and can process a large amount of newly input website feature data while maintaining a low latency. At the same time, the system can perform graded processing according to the probability threshold of the prediction results, for example, when the illegal probability When the preset high-risk threshold is exceeded, the alarm mechanism is immediately triggered and the relevant data is pushed to the regulatory department. In addition, the present invention allows the accuracy and adaptability of the prediction model in real-time prediction to be further improved by dynamically adjusting the parameters of the prediction model or introducing an incremental learning algorithm. The present invention can combine the edge weight calculation method in cluster analysis to quantify the strength of the association between websites, for example, according to the number of shared servers, the overlap of IP addresses or the similarity of behavior patterns, the association relationship is weighted. In addition, preferably, the clustering parameters (such as the number of clusters, the proximity distance threshold, etc.) can be dynamically adjusted to adapt to illegal gang networks of different sizes, thereby improving the accuracy and robustness of the association analysis. Preferably, the clustering analysis method combined with the time dimension can track the evolution of illegal gang behavior and provide support for subsequent dynamic network construction. The network construction process of the present invention supports dynamic updates, and can adjust the network structure in real time as the illegal website data increases or the behavior characteristics change. For example, the gang relationship diagram is updated by adding nodes and edges to ensure that it can reflect the latest illegal gang behavior patterns. In addition, preferably, the edge weight method can be combined to quantify the strength of association between members (such as the number of shared servers, the proportion of IP address overlap, etc.), so as to highlight the core nodes and key paths and provide more fine-grained information support for the precise crackdown on illegal gangs.

[0056] In a specific embodiment of the present invention, step 15, the specific implementation steps include: Step 1: Real-time prediction, use the trained model Deployed to the illegal website detection system, the model performs real-time analysis on newly input website data. Make predictions and output their illegal probability and their association with known illegal gangs. .

[0057] Step 2: Group association analysis: Use clustering algorithms (such as K-Means or DBSCAN) to further analyze the predicted illegal websites and explore the connections between the websites, such as whether they use the same server, shared IP addresses, or similar behavior patterns. ,in, represents the input feature set, C It is the clustering result, which contains the associated group information between illegal websites.

[0058] Step 3, illegal gang network construction: Based on the results of gang association analysis, the network topology of illegal website gangs is constructed using graph data structure, which intuitively displays the association between gang members in the form of a graph. This network topology uses the representation of nodes and edges, with illegal websites as nodes and the association between members as edges, to help reveal the organizational form and core nodes of gang behavior, and further provide a panoramic view and decision-making basis for regulatory authorities.

[0059] In a preferred embodiment of the present invention, the above step 16, based on the network topology of the illegal group, performs anomaly detection on unknown illegal websites, and updates the model based on the anomaly detection to realize the identification of the illegal website group, includes: Step 161, performing anomaly detection on unknown illegal websites through an unsupervised learning algorithm to obtain a detection result; Step 162, updating the sample data set and retraining the model according to the detection result to update the model; Step 163, based on the updated model, to identify illegal website groups.

[0060] In the embodiment of the present invention, the unsupervised learning algorithm can independently discover abnormal behaviors or patterns in unknown illegal websites without relying on labeled data, and expand the scope and ability of illegal website gang identification; through anomaly detection, potential threats can be discovered at the early stage of illegal website gang activities; the unsupervised learning algorithm has strong adaptability to unknown and constantly changing data, and can cope with the diversity and concealment of illegal website gang behavior. Dynamically updating the sample data set according to the anomaly detection results and incorporating newly discovered illegal websites into the training data can continuously optimize the recognition ability and accuracy of the model; retraining the model so that it can learn the latest illegal website gang characteristics and patterns, maintain the effectiveness and timeliness of the model, and adapt to changes in the network environment; through continuous updating and training, the model can accumulate more experience and knowledge, and has stronger robustness and recognition ability for complex and hidden illegal website gang behaviors. The updated model has stronger recognition capabilities and can more accurately identify illegal website groups and their members, reducing false positives and negative alerts. After the model is updated, its processing speed and efficiency will usually be improved, enabling it to analyze large amounts of data more quickly and respond to network security threats in a timely manner. By continuously updating and optimizing the model, it can achieve comprehensive coverage and continuous monitoring of the activities of illegal website groups.

[0061] In a specific embodiment of the present invention, step 16, the specific implementation steps include: Step 1, abnormal pattern detection, monitors unmarked potential illegal websites in the network environment through the anomaly detection module. Anomaly detection uses unsupervised learning algorithms (such as isolation forest or Autoencoder) to mark behaviors that are significantly different from normal patterns. The specific process is as follows: (1) Constructing a normal time series benchmark template Extract time series data of normal website access behavior , as a normal behavior template. Time series with inconsistent lengths are preprocessed by interpolation or linear expansion.

[0062] (2) Dynamic Time Warping (DTW) Static Alignment According to the time series to be detected With normal behavior template . Calculate the DTW distance. Use the DTW algorithm to calculate and The minimum alignment path between: ,in, is the alignment path, is the distance between two time points (such as Euclidean distance). Align the DTW path to On the same time axis, get the aligned time series .

[0063] (3) RNN model modeling and dynamic prediction Using normal time series Train a recurrent neural network (RNN) model to learn the time-dependent features of normal behavior: ,in, : RNN predicts the The behavior value of the moment, : Aligned time series input, : RNN model parameters. Input the time series to be detected Go to the trained RNN model and calculate the actual value With the predicted value The error between: ; Set the error threshold , judge the Whether the moment is abnormal: ; Finally, calculate the anomaly score of the entire time series : , where II is the indicator function, if the abnormal score , the website is marked as abnormal behavior.

[0064] Step 2, feedback and model update, for the newly detected illegal website data, add it to the data set as a new sample, and retrain the model to improve the recognition ability. This feedback mechanism can ensure that the model adapts to the ever-changing network environment. Preferably, the feedback mechanism can be combined with data screening rules (such as feature similarity or confidence threshold) to automatically filter mislabeled data and ensure the quality of the added data samples. In addition, preferably, an online learning algorithm (such as online gradient descent or incremental random forest) can be used to achieve real-time update of the model to reduce the time delay caused by retraining.

[0065] Step 3, alarm and response, once suspected illegal websites or illegal gang behavior is detected, the alarm mechanism is immediately triggered, and relevant information is pushed to the regulatory department or network security team to facilitate further action. Preferably, the alarm mechanism can respond in a graded manner according to the risk level (such as high, medium, and low), and high-risk behavior can trigger real-time alarms and start early warning programs. Preferably, in order to improve the operability of alarm information, visualization tools (such as real-time risk dashboards) can be combined to intuitively show users the behavioral characteristics of illegal websites and the relationship network of gangs.

[0066] An illegal website group identification system 20 based on multi-dimensional feature fusion, comprising: The acquisition module 21 is used to capture network traffic data in real time through traffic mirroring technology or network probes to collect passive traffic data; extract multi-dimensional features from the target website through active requests to achieve feature extraction; perform feature fusion and multi-dimensional analysis on the extracted multi-dimensional features to obtain fused features and analysis results; The processing module 22 is used to train and optimize the machine learning model according to the fusion features and analysis results to obtain a trained model; to identify illegal groups and conduct association analysis on new website data by using the trained model to construct the network topology structure of the illegal groups; to perform anomaly detection on unknown illegal websites according to the network topology structure of the illegal groups, and to update the model according to the anomaly detection to realize the identification of illegal website groups.

[0067] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for identifying illegal website groups based on multi-dimensional feature fusion, characterized in that: The method comprises: Capture network traffic data in real time through traffic mirroring technology or network probes to collect passive traffic data; The collected data is saved, and multi-dimensional features are extracted from the target website through active requests to achieve feature extraction; The extracted multi-dimensional features are subjected to feature fusion and multi-dimensional analysis to obtain fusion features and analysis results; According to the fusion features and analysis results, the machine learning model is trained and optimized to obtain a trained model; By using the trained model to identify and associate new website data with illegal groups, the network topology of illegal groups can be constructed. According to the network topology of the illegal gang, anomaly detection is performed on unknown illegal websites, and the model is updated based on the anomaly detection to identify the illegal website gang.

2. According to claim 1, a method for identifying illegal website groups based on multi-dimensional feature fusion, characterized in that: Capture network traffic data in real time through traffic mirroring technology or network probes to collect passive traffic data, including: Deploy traffic mirroring technology or network probes at the core nodes of the network environment; Capture network traffic data in real time through traffic mirroring technology or network probes, and combine with preliminary filtering rules to collect passive traffic data.

3. According to claim 2, a method for identifying illegal website groups based on multi-dimensional feature fusion, characterized in that: The collected data is saved and multi-dimensional features are extracted from the target website through active requests to achieve feature extraction, including: Store the collected data in structured and unstructured forms; Through active requests, multi-dimensional features are extracted from the target website to achieve feature extraction, including structural features, domain name and IP feature extraction, traffic features and behavior pattern features.

4. The method for identifying illegal website groups based on multi-dimensional feature fusion according to claim 3 is characterized in that: The extracted multi-dimensional features are subjected to feature fusion and multi-dimensional analysis to obtain fusion features and analysis results, including: The extracted multi-dimensional features are cleaned and fused to obtain fused features; Through feature dimension reduction and data mining technology, the potential correlation between target websites is analyzed to obtain analysis results.

5. The method for identifying illegal website groups based on multi-dimensional feature fusion according to claim 4 is characterized in that: According to the fusion features and analysis results, the machine learning model is trained and optimized to obtain a trained model, including: Build classification models based on labeled datasets; According to the fusion features and analysis results, the model is trained through the machine learning algorithm to obtain a trained machine learning model; Tune and verify the trained machine learning model to obtain a trained model.

6. The method for identifying illegal website groups based on multi-dimensional feature fusion according to claim 5, characterized in that: By using the trained model to identify and associate new website data with illegal groups, we can build the network topology of illegal groups, including: By using the trained model to analyze new website data in real time, the illegal probability is predicted to obtain the prediction result; Based on the prediction results, explore the correlation with known illegal gangs to obtain correlation analysis results; Based on the results of correlation analysis, the network topology structure of the illegal gang is constructed.

7. The method for identifying illegal website groups based on multi-dimensional feature fusion according to claim 6 is characterized in that: According to the network topology of the illegal gang, anomaly detection is performed on unknown illegal websites, and the model is updated based on the anomaly detection to identify the illegal website gang, including: Anomaly detection of unknown illegal websites is performed through unsupervised learning algorithms to obtain detection results; Update the sample dataset and retrain the model according to the detection results to update the model; Based on the updated model, the identification of illegal website gangs can be achieved.

8. An illegal website group identification system based on multi-dimensional feature fusion, characterized in that: include: An acquisition module is used to capture network traffic data in real time through traffic mirroring technology or network probes to collect passive traffic data; By actively requesting, multi-dimensional features are extracted from the target website to achieve feature extraction; the extracted multi-dimensional features are subjected to feature fusion and multi-dimensional analysis to obtain fused features and analysis results; The processing module is used to train and optimize the machine learning model according to the fusion features and analysis results to obtain a trained model; to identify illegal gangs and conduct association analysis on new website data by using the trained model to build the network topology structure of illegal gangs; to perform anomaly detection on unknown illegal websites according to the network topology structure of illegal gangs, and to update the model according to the anomaly detection to realize the identification of illegal website gangs.

9. A computing device, characterized in that include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images