Method for identifying unknown violation applications based on encrypted traffic behavior features and intelligence

Abstract

This invention relates to a method for identifying unknown illegal applications based on encrypted traffic behavior characteristics and intelligence, belonging to the field of network security technology. This method analyzes traffic data from network interface cards (NICs) and data packets, filters encrypted traffic, extracts features characterizing traffic behavior, and detects unknown illegal applications based on an illegal application detection model. Furthermore, it combines illegal application intelligence data to assist in the judgment of the detection results, comprehensively achieving rapid and accurate identification of unknown illegal applications.

Description

Technical Field

[0001] This invention belongs to the field of network security technology, specifically relating to a method for identifying unknown and illegal applications based on encrypted traffic behavior characteristics and intelligence. Background Technology

[0002] With the development of network communication technology, the detection and identification of these illegal and irregular applications has become more difficult.

[0003] Invention 1 (CN202110967165) proposes a method for identifying illegal applications, which detects illegal applications based on static features such as IP address and domain name, as well as content features. However, it has the following two limitations: First, with the continuous development of technology, the detection and evasion capabilities of illegal applications and websites on the Internet are becoming stronger. For example, they use methods such as dynamic changes in address and domain name and access through hidden channels to evade traditional detection methods based on static features. Second, with the application and widespread deployment of encrypted communication protocols such as TLS 1.3, more and more illegal applications use encrypted traffic communication methods, making it impossible to identify and detect illegal applications and websites based on content features.

[0004] Invention 2 (CN202410585140) proposes an online encrypted network traffic classification and unknown application identification method, which is similar to the application scenario of this invention. It can achieve the effect of identifying unknown illegal and irregular applications to a certain extent, but it has the following two limitations: On the one hand, Invention 2 identifies unknown applications based on statistical characteristics of traffic such as data packet length distribution. These characteristics work well in the identification of large categories of applications with large differences in traffic scale, such as video and audio, but cannot accurately reflect the differences in traffic behavior between different applications with similar data flow scale (such as social communication applications); on the other hand, the detection results of Invention 2 are entirely based on machine learning models such as KNN, LightGBM, and K-Means, without introducing objective and real intelligence data such as domain names, certificates, and server characteristics for secondary discrimination. When applied in the scenario of this invention, the accuracy of the detection results is difficult to guarantee. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] The technical problem this invention aims to solve is: in response to the national demand for the detection of illegal and irregular applications, and the current situation where illegal applications using encrypted transmission are emerging in an endless stream and are difficult to identify, this invention proposes a method for identifying unknown illegal and irregular applications, so as to achieve rapid and accurate identification of unknown illegal and irregular applications.

[0007] (II) Technical Solution

[0008] To address the aforementioned technical problems, this invention provides a method for identifying unknown and malicious applications based on encrypted traffic behavior characteristics and intelligence, comprising the following steps:

[0009] Step 1) For application types that can be distinguished from normal applications in terms of traffic behavior characteristics, perform unknown illegal application detection based on traffic behavior characteristics: analyze the behavior patterns of known illegal application communication traffic, design various encrypted traffic behavior characteristics, expand the number of encrypted illegal application samples using generative models such as GAN, train an illegal application detection model based on contrastive learning, and realize the detection and identification of unknown illegal applications based on encrypted traffic behavior characteristics.

[0010] Step 2) For application types that cannot be distinguished from normal applications through traffic behavior characteristics, perform unknown illegal application detection based on illegal application intelligence: collect historical illegal application intelligence data, analyze and extract multi-dimensional features, construct an illegal application feature library and detection model, and realize the detection and identification of unknown illegal applications based on illegal application features.

[0011] The present invention also provides a system for implementing the method.

[0012] The present invention also provides a network security analysis method based on the aforementioned method.

[0013] (III) Beneficial Effects

[0014] This invention proposes an unknown illegal application identification technology based on encrypted traffic behavior characteristics and intelligence, specifically including unknown illegal application detection technology based on traffic behavior characteristics and unknown illegal application detection technology based on illegal application intelligence. This technical solution achieves the detection and identification of unknown illegal applications based on the above two technologies, and has the following advantages:

[0015] First, by analyzing the behavioral patterns of illegal applications at the traffic level and extracting multi-dimensional encrypted traffic behavior features, we can effectively counter the detection evasion methods employed by illegal applications, such as dynamic changes in addresses and domain names, access through hidden channels, and encrypted transmission, thereby achieving effective identification of unknown illegal applications.

[0016] Second, historical intelligence data on illegal applications is introduced, and a knowledge graph of illegal applications is extracted from it to construct a feature database of illegal applications. Detecting illegal applications based on feature database matching can, on the one hand, detect some illegal applications without obvious traffic behavior characteristics; on the other hand, it can be combined with the detection of unknown illegal applications based on traffic behavior characteristics to effectively improve the accuracy of detecting unknown illegal applications in real-world networks. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention;

[0018] Figure 2 A schematic diagram for extracting behavioral features of encrypted traffic;

[0019] Figure 3 This is a diagram illustrating the analysis of intelligence used in violation of regulations. Detailed Implementation

[0020] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0021] This invention addresses the national need for detecting illegal and non-compliant applications, as well as the current technological reality of the proliferation and difficulty in identifying illegal applications transmitted via encryption. It proposes a method for identifying unknown illegal applications based on encrypted traffic behavior characteristics and intelligence. This method analyzes traffic data from network interface cards (NICs) and data packets, filters encrypted traffic, extracts features characterizing traffic behavior, and detects unknown illegal and non-compliant applications based on an illegal and non-compliant application detection model. Furthermore, it incorporates illegal and non-compliant application intelligence data to assist in the judgment of the detection results, comprehensively achieving rapid and accurate identification of unknown illegal and non-compliant applications.

[0022] This invention proposes a method for identifying unknown illegal applications based on encrypted traffic behavior characteristics and intelligence, approaching the issue from two angles: behavioral feature analysis of known illegal applications at the traffic level and intelligence analysis of historically discovered illegal applications. On one hand, it analyzes the behavioral patterns of known illegal applications at the traffic level to train a detection model based on encrypted traffic behavior characteristics; on the other hand, it analyzes historical illegal application intelligence to extract relevant illegal application features and construct a detection model based on these features. These two models work together from different perspectives to achieve rapid and accurate detection and identification of unknown illegal applications.

[0023] To achieve the objectives of the invention, the present invention proposes the following technical solution: a method for identifying unknown illegal applications based on encrypted traffic behavior characteristics and intelligence. This solution comprises two parts: 1) For illegal applications that differ from normal applications in their traffic behavior characteristics, the method for detecting unknown illegal applications based on these characteristics involves analyzing the behavior patterns of known illegal application communication traffic, designing traffic behavior characteristics such as data packet arrival time intervals, uplink / downlink traffic ratios, and session durations, while simultaneously expanding the number of encrypted illegal application samples using generative models such as GANs, training an illegal application detection model based on contrastive learning, and achieving the detection and identification of unknown illegal applications based on traffic behavior characteristics; 2) For illegal applications that do not show significant differences from normal applications in their traffic behavior characteristics, the method for detecting unknown illegal applications based on illegal application intelligence involves collecting historical illegal application intelligence data, analyzing and extracting multi-dimensional features such as servers, domain names, and protocols, constructing an illegal application feature library and detection model, and achieving the detection and identification of unknown illegal applications based on these features.

[0024] The following section will provide a detailed introduction to each part of the plan.

[0025] (a) Detection of Unknown Illegal Applications Based on Traffic Behavior Characteristics

[0026] This section proposes a method for detecting unknown illegal applications based on traffic behavior characteristics. This method mainly includes the following three parts: design of illegal application traffic behavior characteristics, generation of encrypted illegal application communication traffic, and training of an illegal application detection model based on contrastive learning. Each part is described below.

[0027] 1.1 Design of Encrypted Traffic Behavior Characteristics for Unauthorized Applications

[0028] The design of encrypted traffic behavior characteristics for unauthorized applications is as follows: encrypted traffic behavior characteristics that can be extracted by analyzing the behavior models of known unauthorized application traffic, including two main categories: time-series characteristics and protocol characteristics.

[0029] 1.1.1 Temporal characteristics

[0030] Timing characteristics primarily characterize the behavior patterns of encrypted traffic at the packet and session levels, including:

[0031] Data packet arrival time interval characteristics: mean, variance, range, etc. of the arrival time of consecutive data packets in statistical traffic;

[0032] Traffic rate characteristics: the rate of change of the number of data packets within a unit time window (optionally, per second), extracting sliding window statistics, including uplink traffic rate and downlink traffic rate;

[0033] Packet size distribution characteristics: Statistical distribution characteristics of packet size, including the mean, variance, transition matrix, and distribution per unit time window (optionally, per second) for the entire session;

[0034] Session distribution characteristics include the time distribution characteristics of the entire session (such as the number of sessions per hour) and the periodic characteristics of session establishment (such as reconnection behavior at fixed time intervals).

[0035] Session direction characteristics include uplink and downlink traffic ratio characteristics (such as the ratio of uplink and downlink traffic in bytes, the ratio of uplink and downlink traffic in sessions, etc.) and initial handshake direction characteristics (the transmission direction sequence pattern of the first N data packets in the connection establishment phase, forming the direction sequence encoding).

[0036] 1.1.2 Protocol Features

[0037] Protocol characteristics primarily characterize the behavioral features of encrypted traffic in terms of encryption protocols, certificates, and background traffic, including:

[0038] Features of encrypted communication protocols: These are mainly the features exhibited by SSL and other related encrypted communication protocols during the handshake phase, such as TLS version number, cipher suite, non-standard extensions, and SNI field content;

[0039] Certificate-related characteristics: These are mainly characteristics related to certificate protocols such as x509, such as self-signed certificates or certificates from non-public CAs, certificate chain length, certificate validity period, and certificate reuse rate.

[0040] Background traffic features: These are features extracted and selected from background traffic such as DNS and HTTP. DNS background traffic features include domain name features (DGA domain name detection status, differences between authoritative DNS and local resolution), query behavior features, and DNS encryption protocol features. HTTP background traffic features include User-Agent field, non-standard headers, Accept field, and Cache-Control field.

[0041] 1.2 Generation of encrypted communication traffic by unauthorized applications

[0042] This system collects encrypted communication traffic from various known (real) malicious applications in a sandbox environment. It trains generative network models such as GANs to learn the potential traffic behavior characteristics of these malicious applications, efficiently generating large amounts of encrypted traffic data (to generate more encrypted traffic behavior feature data that matches the distribution of malicious application encrypted traffic behavior characteristics). This provides training data support for unknown malicious application detection models. The system mainly consists of two parts: automated encrypted traffic capture and encrypted traffic data generation based on generative models.

[0043] 1.2.1 Automated Encrypted Traffic Capture

[0044] Based on the sandbox environment, various known non-compliant applications are collected, and deep packet inspection (DPI) technology is used to capture application-layer encrypted communication traffic in a refined manner. Simultaneously, traffic mirroring technology is used to fully record the TLS / SSL handshake process at the transport layer. During traffic capture, the encrypted traffic behavior features designed in section 1.1 are extracted to form corresponding CSV files, supporting the generation of encrypted communication traffic data.

[0045] 1.2.2 Generation of Encrypted Traffic Data Based on Generative Model

[0046] Based on the encrypted traffic behavior features obtained in the CSV file from section 1.2.1, a Generative Adversarial Network (GAN) model is trained. This training process includes a generator... Discriminator And other related content.

[0047] set up For the encrypted traffic behavior feature set collected from known violating applications, satisfying And for , For the real number field, The number of features representing the characteristics of illegal use of encrypted traffic. Potential distribution of encrypted traffic behavior characteristics for unauthorized applications.

[0048] generator This can be represented as a mapping ,in Indicates a Gaussian distribution. Indicates length is The zero vector, for identity matrix, much smaller Positive integers. That is, generators. The purpose is based on any length of random vectors Generate a corresponding length of eigenvectors The goal of training the generator is to modify its parameters so that it generates a generator that satisfies the given input for any given input. The eigenvectors of the distribution, i.e., , make the final result (G after training) satisfies: .

[0049] Discriminator This can be represented as a mapping discriminator The purpose is to judge Do the generated feature vectors follow a distribution? The discriminator is finally obtained through training. Satisfaction: For ,have:

[0050]

[0051] Based on the above process, a corresponding GAN model is trained for each type of illegal application. The GAN model training in this invention adopts the conventional GAN ​​training method and loss function, which will not be elaborated further here. After the GAN model is trained to convergence, the trained generator can be used. Quickly generate encrypted traffic data (corresponding encrypted traffic behavior features) of illegal applications to support the subsequent training of illegal application detection models based on comparative learning.

[0052] 1.3 Detection of Unknown Illegal Applications Based on Comparative Learning

[0053] Based on the encrypted traffic behavior characteristics designed in section 1.1 and the encrypted traffic data of illegal applications (encrypted traffic data of normal applications and encrypted traffic data of illegal applications) generated in section 1.2, a contrastive learning-based illegal application detection model is constructed and trained. By learning the similarities and differences between traffic samples, the model can distinguish between normal traffic and illegal traffic. The process is as follows:

[0054] 1.3.1 Constructing positive and negative sample pairs to form a dataset

[0055] (1) Positive sample pair construction: Two samples are randomly sampled from the encrypted traffic data of similar violating applications as positive sample pairs. Since it is known that the traffic of violating applications and the corresponding GAN-generated traffic have similar behavioral characteristics, they are treated as samples of the same category to construct positive sample pairs.

[0056] (2) Negative sample pair construction: Two samples are randomly sampled from the encrypted traffic data of different categories of violating applications to form negative sample pairs. This method enables the model to learn the differences in traffic behavior characteristics of different violating applications.

[0057] (3) Data augmentation: Randomly perturb the collected traffic data of known illegal applications (positive sample pairs and negative sample pairs), including adding noise, random packet loss, and adjusting timestamps, to increase the diversity of samples and improve the model's ability to detect unknown illegal applications.

[0058] 1.3.2 Illegal Application Detection Model Architecture Design: Includes a feature extraction module, a contrastive learning module, and a classifier.

[0059] (1) Feature extraction module

[0060] The feature extraction module analyzes and processes the behavioral features of the input traffic (positive and negative sample pairs constructed in 1.3.1). This module consists of a 1D convolutional neural network and a fully connected neural network, used to analyze and process the temporal relationship of each traffic sample. In addition, for protocol features, the protocol features are encoded using an embedding model and then input into the feature extraction module for analysis and processing.

[0061] (2) Comparative learning module

[0062] The traffic features processed by the feature extraction module are sent to the contrastive learning module for further processing. The contrastive learning module maps the feature representations to a low-dimensional contrastive learning space through a projection head constructed from a 3-layer fully connected neural network, making similar samples closer together and different samples further apart in the contrastive learning space.

[0063] This contrastive learning module uses a self-supervised contrastive learning loss function to optimize the projected feature representation, learning the similarities and differences between traffic samples. The self-supervised contrastive learning loss function is as follows:

[0064]

[0065] in, Represents flow samples The feature representation, namely the contrastive learning module; for Positive samples; The similarity measure between feature representations can be expressed (optionally, cosine distance, Euclidean distance, etc.). for The number of corresponding positive and negative sample pairs, that is, for any input All A number of relevant positive samples and One relevant negative sample; It is a temperature parameter used to control the scaling of the similarity metric; For indicator functions, when and If the samples are the same, the indicator function returns 0; otherwise, it returns 1.

[0066] (3) Classifier

[0067] Based on the comparative learning module, add A classifier is used to classify traffic samples as normal traffic or illegal traffic. The classifier takes as input the feature representation output by the contrastive learning module and outputs as the class probability of the traffic sample.

[0068] 1.3.3 Training of the Illegal Application Detection Model

[0069] Training the illegal application detection model includes the following two stages:

[0070] (1) Comparative learning stage

[0071] The illegal application detection model designed in section 1.3.2 is pre-trained using a contrastive loss function to optimize the training parameters of the contrastive learning module. Specifically, in each training batch, positive and negative sample pairs are randomly sampled from the dataset. The contrastive loss is calculated using the aforementioned contrastive loss function, and backpropagation is performed to update the model parameters. The similarity metric between the contrastive loss calculated using the self-supervised contrastive learning loss function and the feature representations on the validation set (extracted from the dataset) is monitored to ensure that the illegal application detection model can effectively learn the similarities and differences between traffic samples.

[0072] (2) Classification training stage

[0073] After the contrastive learning phase training converges, the training parameters of the contrastive learning module are frozen, and only the classifier parameters are optimized. Specifically, the cross-entropy loss function is used to train the classifier and optimize the classification module parameters. In each training batch, traffic samples and their corresponding labels are randomly sampled from the dataset, the classification loss is calculated, and backpropagation is performed to update the classifier parameters. The classification accuracy on the validation set is monitored, and the best-performing model is selected as the final illegal application detection model.

[0074] (ii) Detection of Unknown Illegal Applications Based on Illegal Application Intelligence

[0075] Since some illegal applications do not show significant differences from normal applications in terms of traffic behavior characteristics extracted in section 1.1, this invention introduces an intelligence database related to illegal applications. From this database, multi-dimensional features such as domain name characteristics, IP and network characteristics, and content characteristics are extracted to construct an illegal application feature database. Based on this feature database, the detection and analysis of illegal applications are then implemented.

[0076] 2.1 Collection, Analysis and Processing of Intelligence Regarding Illegal Applications

[0077] The intelligence collected related to illegal applications is analyzed from multiple dimensions, including domain name characteristics, IP and network characteristics, and content characteristics.

[0078] 2.1.1 Domain Name Feature Analysis

[0079] Obtain domain registration information and domain name resolution records of illegal applications from intelligence.

[0080] Domain registration information includes the registrant, registration email address, registration date, etc., and is used to determine the legality of the domain name.

[0081] Domain name resolution records contain the historical IP addresses corresponding to the domain name. By analyzing information such as the location of the historical IP addresses and whether they are dynamic IPs, the suspiciousness of the domain name can be further determined.

[0082] 2.1.2 IP and Network Characteristics Analysis

[0083] The intelligence extracts the IP addresses and related network characteristics of the illegal applications, including the IP address's location, open ports, network traffic characteristics, and network protocols used, to determine whether they have typical characteristics of illegal applications.

[0084] IP address geolocation characteristics: Using IP address geolocation lookup tools, you can obtain the geographical location information of an IP address. If an application's IP geolocation highly matches a known illegal application and is located in a high-risk area, or if the IP geolocation changes frequently, this can serve as important evidence for detecting illegal applications.

[0085] Open port characteristics: Use network scanning tools to scan IP addresses for open ports. Some malicious applications use specific ports for communication. For example, some virtual trading applications may use specific encrypted communication ports, while some malicious services may use common proxy ports (such as SOCKS proxy 1080, etc.).

[0086] Network traffic characteristics: These mainly include the temporal distribution of traffic volume and access frequency, as well as the encryption status of the traffic. Furthermore, if the traffic from an IP address contains a large amount of encrypted data, and the encryption protocol is used unusually (e.g., using uncommon encryption algorithms or low encryption strength), this could be an indication that a malicious application is attempting to hide the content of its data transmission.

[0087] 2.1.3 Content Feature Analysis

[0088] Extract content data (such as web page text, images, videos, user interaction information, etc.) from intelligence to analyze its themes, language style, dissemination methods, and other characteristics to determine whether it involves illegal or irregular content or behavior patterns.

[0089] Content theme characteristics: Identify sensitive information in the content through natural language processing (NLP) or keyword matching technology.

[0090] Language style characteristics: Language style refers to the content expression style commonly used by illegal applications to circumvent review, including but not limited to vague expressions (such as "XX platform" or "investment channel") and the inclusion of a large number of non-standard characters (such as pinyin abbreviations and homophones);

[0091] Propagation characteristics: Propagation methods refer to the way the app is distributed, such as through links from unknown sources, encrypted chat tools, or by using short links or redirecting domains to hide the real address.

[0092] 2.2 Construction of the Illegal Application Feature Database

[0093] A feature library of illegal applications, including the features collected in section 2.1, is constructed based on knowledge graphs.

[0094] 2.2.1 Entity Categories and Attributes in the Feature Library

[0095] The database of unauthorized applications includes the following entities:

[0096] Domain entity: Attributes include domain registrant, registration email, registration time, domain name resolution records (including IP address, resolution time, resolution status), etc.

[0097] IP address entity: Attributes include IP address, location, whether it is a dynamic IP, list of open ports, network traffic characteristics (traffic size, access frequency, encryption status, etc.), network protocol used, etc.

[0098] Content entities: Attributes include content topic, language style (vague descriptions, non-standard characters, etc.), dissemination method (links from unknown sources, encrypted chat tools, short links, etc.);

[0099] Server entity: Attributes include application name, application type, etc.

[0100] 2.2.2 Relationships between entities in the feature library

[0101] The database of features of illegal applications contains the following relationships:

[0102] Domain name and IP address relationship type: resolution (refers to the relevant content of IP and domain name in the domain name resolution record).

[0103] IP address and server relationship type: Hosted in (hosting location).

[0104] Application-domain relationship type: Use (the application uses the domain name).

[0105] Application-content relationship type: contain (application contains content).

[0106] Application-server relationship type: Hosted (the application is hosted on the server).

[0107] 2.3 Detection of Unknown Illegal Applications Based on Illegal Application Characteristics

[0108] Feature-based illegal application detection utilizes the illegal application feature library built in section 2.2 to detect new applications or web pages and determine whether they are illegal applications. The detection process includes the following steps:

[0109] 2.3.1 Feature Extraction

[0110] For new applications or web pages to be detected, relevant features are extracted from the application content and communication traffic, including domain name features, IP and network features, content features, and server features. For example, for domain name feature extraction, the domain name is extracted from the URL of the web page to be detected, and its registration information is obtained through the domain registration information query interface; for IP and network feature extraction, network scanning tools and traffic analysis equipment are used to quickly scan and monitor the target IP to obtain information such as its open ports and traffic characteristics; for content feature extraction, text analysis tools and image recognition tools are used to parse and recognize the text content, images, videos, etc. of the web page to extract features such as content theme and language style.

[0111] 2.3.2 Feature Matching

[0112] The extracted features are matched against features in the illegal application feature library (knowledge graph feature library). For each extracted feature, a matching node is searched in the illegal application feature library (knowledge graph feature library). For example, for the extracted domain name feature, a node with the same or similar domain name is searched in the illegal application feature library (knowledge graph feature library); for the extracted network feature, a node with the same or similar network feature is searched in the illegal application feature library (knowledge graph feature library).

[0113] Because some illegal applications mimic legitimate applications in terms of domain names and other aspects (e.g., domain information mimicking well-known legitimate applications like Baidu), the matching process uses fuzzy matching after whitelist filtering. For example, for domain name characteristics and network characteristics, precise filtering using a whitelist is followed by fuzzy matching.

[0114] 2.3.3 Correlation Analysis

[0115] For the matched features, their associations in the illegal application feature base (knowledge graph feature base) are further analyzed. Based on the edge type and direction defined in the knowledge graph, as well as the edge weights, the similarity between the feature combination of the application to be detected or the webpage and the feature combination of illegal applications in the illegal application feature base (knowledge graph feature base) is calculated. For example, if the domain name of the application to be detected matches a certain illegal application domain name in the illegal application feature base (knowledge graph feature base), and its resolved IP address matches the IP address of the illegal application, and this IP address has the same network traffic characteristics as the illegal application, and the webpage content contains keywords similar to those of the illegal application, then the application to be detected is considered to have a high similarity to the illegal application.

[0116] Various graph algorithms, including but not limited to shortest path algorithms and subgraph matching algorithms, can be used in association analysis to calculate the association strength and similarity between features. Optionally, the shortest path algorithm is used to calculate the shortest path length between the feature nodes of the application to be detected and the feature nodes of violating applications in the violating application feature library (knowledge graph feature library). The shorter the path length, the stronger the association. The subgraph matching algorithm is used to determine whether the feature combination of the application to be detected has a similar subgraph structure to the feature combination of violating applications in the violating application feature library (knowledge graph feature library). The higher the subgraph matching degree, the higher the similarity. Based on the results of the association analysis, unknown violating applications can be detected based on a matching threshold.

[0117] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for identifying unknown illegal applications based on encrypted traffic behavior characteristics and intelligence, characterized in that, Includes the following steps: Step 1) For application types that can be distinguished from normal applications in terms of traffic behavior characteristics, perform unknown illegal application detection based on traffic behavior characteristics: analyze the behavior patterns of known illegal application communication traffic, design various encrypted traffic behavior characteristics, expand the number of encrypted illegal application samples using generative models such as GAN, train an illegal application detection model based on contrastive learning, and realize the detection and identification of unknown illegal applications based on encrypted traffic behavior characteristics. Step 2) For application types that cannot be distinguished from normal applications by traffic behavior characteristics, perform unknown illegal application detection based on illegal application intelligence: collect historical illegal application intelligence data, analyze and extract multi-dimensional features, build an illegal application feature library and detection model, and realize the detection and identification of unknown illegal applications based on illegal application features. Step 2) specifically includes: 2.1 Collection, Analysis and Processing of Intelligence Regarding Unauthorized Use The intelligence collected related to illegal applications is analyzed from multiple dimensions, including domain name characteristics, IP and network characteristics, and content characteristics. 2.1.1 Domain Name Feature Analysis Obtain domain registration information and domain name resolution records of illegal applications from intelligence sources; Domain registration information includes the registrant, registration email address, and registration date, which are used to determine the legality of the domain name; Domain name resolution records contain the historical IP addresses corresponding to the domain name. By analyzing the location of the historical IP addresses and whether they are dynamic IP information, the suspiciousness of the domain name can be further determined. 2.1.2 IP and Network Characteristics Analysis Extract the IP addresses and related network characteristics of illegal applications from intelligence, including the IP address's location, open ports, network traffic characteristics, and network protocols used, to determine whether they have typical characteristics of illegal applications; IP address location characteristics: Obtain the geographical location information of an IP address by using an IP address location lookup tool; Open port characteristics: Use network scanning tools to scan IP addresses for open ports. Network traffic characteristics: including the temporal distribution characteristics of traffic volume and access frequency, as well as the encryption status of the traffic; 2.1.3 Content Feature Analysis Extract content data of illegal applications from intelligence, analyze their themes, language style and dissemination methods to determine whether they involve illegal or irregular content or behavior patterns. Content theme characteristics: Sensitive information in content can be identified through natural language processing or keyword matching techniques. Language style characteristics: Language style characteristics refer to the expression style of content used by illegal applications to circumvent censorship; Characteristics of dissemination methods: Dissemination methods refer to the methods of distribution within the violating application; 2.2 Construction of the Feature Database for Illegal Applications A feature library of illegal applications, containing the features collected in section 2.1, is constructed based on knowledge graphs. 2.2.1 Entity Categories and Attributes in the Feature Library The database of unauthorized applications includes the following entities: Domain entity: Attributes include domain registrant, registration email, registration time, and domain name resolution records; IP address entity: Attributes include IP address, location, whether it is a dynamic IP, list of open ports, network traffic characteristics, and network protocol used; Content entity: Attributes include content theme, language style, and dissemination method; Server entity: Attributes include application name and application type; 2.2.2 Relationships between entities in the feature library The database of features of illegal applications contains the following relationships: Domain name and IP address relationship type: resolved to; IP address and server relationship type: Hosted by; Application-domain relationship type: Use; Application-content relationship type: Contains; Application-server relationship type: Hosted on; 2.3 Detection of Unknown Illegal Applications Based on Illegal Application Characteristics The feature-based detection of illegal applications utilizes the illegal application feature library built in section 2.2 to detect new applications or web pages and determine whether they are illegal applications. The detection process in step 2.3 includes the following steps: 2.3.1 Feature Extraction For new applications or web pages to be detected, relevant features are extracted from application content and communication traffic, including domain name features, IP and network features, content features, and server features. For domain name feature extraction, the domain name is extracted from the URL of the web page to be detected, and its registration information is obtained through the domain name registration information query interface. For IP and network feature extraction, the target IP is scanned and traffic is monitored using network scanning tools and traffic analysis equipment to obtain its open ports and traffic feature information. For content feature extraction, text analysis tools and image recognition tools are used to parse and recognize the text content, images, and videos of the web page to extract content theme and language style features. 2.3.2 Feature Matching The extracted features are matched with features in the illegal application feature library; for each extracted feature, a matching node is searched in the illegal application feature library; for the extracted domain name feature, a node with the same or similar domain name is searched in the illegal application feature library; for the extracted network feature, a node with the same or similar network feature is searched in the illegal application feature library. 2.3.3 Correlation Analysis For the matched features, their association in the illegal application feature database is further analyzed. Specifically, based on the type and direction of the edges defined in the knowledge graph, as well as the edge weights, the similarity between the feature combination of the application to be detected or the webpage and the feature combination of illegal applications in the illegal application feature database is calculated. If the domain name of the application to be detected matches a certain domain name of an illegal application in the illegal application feature database, and its resolved IP address matches the IP address of the illegal application, and the IP address has the same network traffic characteristics as the illegal application, and the webpage content contains keywords similar to those of the illegal application, then the application to be detected is considered to have a high similarity to the illegal application.

2. The method as described in claim 1, wherein step 1) specifically includes: 1.1 Design of Encrypted Traffic Behavior Characteristics for Unauthorized Applications The design of encrypted traffic behavior characteristics for unauthorized applications is as follows: encrypted traffic behavior characteristics that can be extracted by analyzing the behavior models of known unauthorized application traffic, including two main categories: time-series characteristics and protocol characteristics; 1.1.1 Temporal characteristics Timing characteristics characterize the behavior patterns of encrypted traffic at the packet and session levels, including: Data packet arrival time interval characteristics: the mean, variance, and range of arrival times of consecutive data packets in statistical traffic; Traffic rate characteristics: the rate of change of the number of data packets within a unit time window, extracting sliding window statistics, including uplink traffic rate and downlink traffic rate; Packet size distribution characteristics: Statistical distribution characteristics of packet size, including the mean, variance, transition matrix, and distribution within a unit time window for the entire session; Session distribution characteristics: including the time distribution characteristics of complete sessions and the periodicity characteristics of session establishment; Session direction characteristics: including uplink and downlink traffic ratio characteristics and initial handshake direction characteristics; 1.1.2 Protocol Features Protocol characteristics characterize the behavioral features of encrypted traffic in terms of encryption protocols, certificates, and background traffic, including: Encrypted communication protocol characteristics: These are the features exhibited by the encrypted communication protocol during the handshake phase. Certificate-related characteristics: These are the characteristics related to the certificate protocol. Background traffic features: These are features extracted and selected from background traffic. DNS background traffic features include domain name features, query behavior features, DNS encryption protocol features, etc.; HTTP background traffic features include the User-Agent field, non-standard headers, Accept field, and Cache-Control field. 1.2 Generation of encrypted communication traffic by unauthorized applications The system collects encrypted communication traffic from various known illegal applications in a sandbox environment, learns the potential traffic behavior characteristics of illegal applications by training a GAN model, and generates encrypted traffic data to support the training data of unknown illegal application detection models. The system consists of two parts: automated encrypted traffic capture and encrypted traffic data generation based on generative models. 1.2.1 Automated Encrypted Traffic Capture Based on the sandbox environment, various known non-compliant applications are collected and deep packet inspection technology is used to capture application layer encrypted communication traffic. At the same time, traffic mirroring technology is used to fully record the TLS / SSL handshake process of the transport layer. During the traffic capture process, the encrypted traffic behavior features designed in step 1.1 are extracted to form corresponding CSV files to support the generation of encrypted communication traffic data. 1.2.2 Generation of Encrypted Traffic Data Based on Generative Model Based on the encrypted traffic behavior features in the CSV file obtained in 1.2.1, a generative adversarial network (GAN) model is trained to generate encrypted traffic data of illegal applications. 1.3 Detection of Unknown Illegal Applications Based on Comparative Learning Based on the encrypted traffic behavior characteristics designed in 1.1 and the encrypted traffic data of illegal applications generated in 1.2, a violation application detection model based on contrastive learning is constructed and trained using contrastive learning technology. By learning the similarity and differences between traffic samples, the violation application detection model can distinguish between normal traffic and illegal traffic.

3. The method as described in claim 2, characterized in that, Step 1.2.2 is as follows: set up For the encrypted traffic behavior feature set collected from known violating applications, satisfying And for , For the real number field, The number of features representing the characteristics of illegal use of encrypted traffic. Potential distribution of encrypted traffic behavior characteristics for unauthorized applications; generator Represented as a mapping ,in Indicates a Gaussian distribution. Indicates length is The zero vector, for identity matrix, much smaller Positive integers; i.e., generators The purpose is based on any length of random vectors Generate a corresponding length of eigenvectors Training the generator The goal is to modify its parameters so that it generates a condition that satisfies the given input for any given input. The feature vectors of the distribution, i.e., those that make the final trained generator... satisfy: ; Discriminator Represented as a mapping discriminator The purpose is to judge Do the generated feature vectors follow a distribution? The discriminator is finally obtained through training. Satisfaction: For ,have: For each type of illegal application, a corresponding GAN model is trained; after the GAN model has converged, the trained generator is used. Generate encrypted traffic data of illegal applications to support the subsequent training of illegal application detection models based on comparative learning.

4. The invention as described in claim 2, characterized in that, Step 1.3 is as follows: 1.3.1 Constructing positive and negative sample pairs to form a dataset (1) Positive sample pair construction: Randomly sample two samples from the encrypted traffic data of similar violating applications as positive sample pairs; (2) Negative sample pair construction: Randomly sample two samples from the encrypted traffic data of different categories of violating applications as negative sample pairs; (3) Data augmentation: Randomly perturb the collected known illegal application traffic data, i.e., positive and negative sample pairs, including adding noise, random packet loss and adjusting timestamps; 1.3.2 The illegal application detection model is designed as follows: it includes a feature extraction module, a contrastive learning module, and a classifier. (1) Feature extraction module The feature extraction module analyzes and processes the positive and negative sample pairs constructed in section 1.3.

1. This feature extraction module consists of a 1D convolutional neural network and a fully connected neural network, used to analyze and process the temporal relationship of each flow sample; for Protocol features are encoded using an embedding model and then input into the feature extraction module for analysis and processing. (2) Comparative learning module The traffic features processed by the feature extraction module are sent to the contrast learning module for further processing. The contrast learning module maps the feature representations to a low-dimensional contrast learning space through a projection head constructed from a 3-layer fully connected neural network, making similar samples closer in the contrast learning space and different samples further apart. This contrastive learning module uses a self-supervised contrastive learning loss function to optimize the projected feature representation, learning the similarities and differences between traffic samples; the self-supervised contrastive learning loss function is as follows: in, Represents flow samples The feature representation, namely the contrastive learning module; for Positive samples; A measure of similarity between feature representations; for The number of corresponding positive and negative sample pairs, that is, for any input All A number of relevant positive samples and One relevant negative sample; It is a temperature parameter used to control the scaling of the similarity metric; For indicator functions, when and If the samples are the same, the indicator function returns 0; otherwise, it returns 1. (3) Classifier Based on the comparative learning module, add A classifier is used to classify traffic samples as normal traffic or illegal traffic; The classifier takes as input the feature representation output by the contrastive learning module and outputs as the class probability of the traffic sample. 1.3.3 Training of the Illegal Application Detection Model Training the illegal application detection model includes the following two stages: (1) Comparative learning stage The illegal application detection model designed in 1.3.2 is pre-trained using the contrastive loss function to optimize the training parameters of the contrastive learning module. In each training batch, positive and negative sample pairs are randomly sampled from the dataset, and the contrastive loss is calculated using the contrastive loss function and backpropagation is performed to update the model parameters. (2) Classification training stage After the training converges during the contrastive learning phase, the training parameters of the contrastive learning module are frozen, and only the parameters of the classifier are optimized. Specifically, the cross-entropy loss function is used to train the classifier and optimize the parameters of the classification module. In each training batch, traffic samples and their corresponding labels are randomly sampled from the dataset, the classification loss is calculated, and backpropagation is performed to update the classifier parameters. The classification accuracy on the validation set is monitored, and the best-performing model is selected as the final illegal application detection model. The validation set is extracted from the dataset.

5. The method as described in claim 1, characterized in that, The feature matching process in step 2.3.2 is implemented using a fuzzy matching method after whitelist filtering.

6. The method as described in claim 1, characterized in that, In the association analysis process of step 2.3.3, multiple graph algorithms are used to calculate the association strength and similarity between features. Among them, the shortest path algorithm is used to calculate the shortest path length between the feature nodes of the application to be detected and the feature nodes of illegal applications in the illegal application feature library. The shorter the path length, the stronger the association. The subgraph matching algorithm is used to determine whether the feature combination of the application to be detected has a similar subgraph structure to the feature combination of illegal applications in the illegal application feature library. The higher the subgraph matching degree, the higher the similarity. Based on the results of the association analysis, the detection of unknown illegal applications is achieved by using a matching threshold method.

Images