Traffic detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a traffic detection method and device, electronic equipment and a storage medium, and relates to the field of network security. The method comprises the following steps: performing feature extraction on target traffic data to obtain a target traffic feature; the target traffic features are input into a target model, the type of target traffic data is detected based on the target traffic features according to priori knowledge of the target model in a model training stage, and the target model is obtained through training of the following three types of loss functions: a first type of loss function, a second type of loss function and a third type of loss function; the difference determination module is used for determining differences between prediction results of the neural network about training samples and training labels; the second type of loss function is used for determining the similarity between the training samples and the normal samples and classifying the training samples according to the similarity; and the third type of loss function is used for dynamically adjusting the weight of the training sample when detecting that the training sample is abnormal traffic. According to the method and the device, the technical problem of low detection coverage rate of abnormal traffic such as malicious traffic in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of network security, and more specifically, to a flow detection method, device, electronic device, and storage medium. Background Art

[0002] Currently, the industry primarily utilizes three main technical approaches for detecting malicious traffic: rule-based methods, supervised learning-based detection, and unsupervised learning-based detection. Rule-based methods primarily use predefined rules or signatures to match known malicious behavior patterns. While effective for addressing known threats, they have significant limitations when addressing unknown threats, as rule base updates often lag behind the emergence of new malicious behaviors. Furthermore, the rule matching process relies on precise rule development, which is time-consuming and complex, making it difficult to cover all potential malicious behaviors, thereby reducing detection coverage.

[0003] In recent years, leveraging the powerful feature extraction capabilities of deep learning technology, numerous supervised learning-based malicious traffic detection solutions have emerged. These methods build classification models based on historical malicious traffic samples to identify new malicious traffic. While they perform well on standardized datasets, the diverse and constantly evolving nature of malicious traffic in the real world means that labeled datasets often fail to fully cover all threat scenarios. Furthermore, the extreme imbalance in the amount of malicious traffic compared to normal traffic creates a significant challenge when training deep learning models. Models can easily bias their predictions towards normal traffic, ignoring or misclassifying a minority of malicious traffic, which in turn affects overall detection rate and accuracy.

[0004] Given the rarity and extremely uneven distribution of malicious traffic, unsupervised learning presents an attractive option. Rather than relying on labeled data, it analyzes the inherent structure and patterns of network traffic to identify anomalous behavior. However, even unsupervised learning encounters bottlenecks, particularly in traffic clustering or anomaly detection. Lacking clear guidance, the model may misclassify some malicious traffic as normal or perform poorly when faced with new, unseen malicious traffic, reducing detection coverage and robustness.

[0005] In summary, although existing technologies have dealt with malicious traffic in the network to a certain extent, they still have significant shortcomings in covering various malicious behaviors, especially in adapting to constantly evolving unknown threats. Summary of the Invention

[0006] The present application provides a traffic detection method, device, electronic device and storage medium to at least solve the technical problem of low detection coverage of abnormal traffic such as malicious traffic in the prior art.

[0007] According to one aspect of the present application, a flow detection method is provided, comprising: extracting features from target flow data to obtain target flow features; inputting the target flow features into a target model, and detecting the type of the target flow data based on the target flow features according to prior knowledge of the target model during a model training phase, wherein the target model is trained using the following three types of loss functions:

[0008] The first type of loss function is used to determine the difference between the neural network's prediction results for the training examples and the training labels;

[0009] The second type of loss function is used to determine the similarity between training samples and normal samples, and classify the training samples based on the similarity;

[0010] The third type of loss function is used to dynamically adjust the weights of training samples when the training samples are detected as abnormal traffic.

[0011] Optionally, in the process of training the target model through the first type of loss function, the predicted probability of each training sample belonging to each type of traffic by the neural network is detected; the total number of training samples, the predicted probability of each training sample belonging to each type of traffic, and the training label corresponding to each training sample are used as parameters to be calculated; the parameters to be calculated are substituted into the first type of loss function for calculation to obtain a first loss value; and the neural network is trained according to the first loss value.

[0012] Optionally, in the process of training the target model through the second type of loss function, the predicted probability of the neural network for each training sample belonging to each type of traffic is detected, wherein each type of traffic includes at least normal traffic and abnormal traffic; the central vector of each type of traffic is determined, wherein the central vector of each type of traffic represents the eigenvector at the center position of all eigenvectors corresponding to that type of traffic; the vector distance between the eigenvector of each training sample and the central vector of each type of traffic is calculated; the vector distance, the predicted probability of the neural network for each training sample belonging to each type of traffic, and the total number of training samples are substituted as parameter values into the second type of loss function for calculation to obtain a second loss value; and the neural network is trained according to the second loss value.

[0013] Optionally, in the process of training the target model through the third type of loss function, the proportion of training samples carrying each label in the training sample set is detected respectively; a dynamic adjustment benchmark value is obtained, wherein the dynamic adjustment benchmark value is used to characterize the density of the category with the highest frequency of occurrence in all abnormal traffic samples included in the training sample set; based on the proportion, the dynamic adjustment benchmark value and the third type of loss function, a third loss value is determined; and the neural network is trained based on the third loss value.

[0014] Optionally, a third type of loss function is used to represent the product of the first sub-function and the second sub-function, wherein the first sub-function is used to constrain the second sub-function to be effective for abnormal traffic samples during model training and not to be effective for normal traffic samples; the second sub-function is used to perform the following steps:

[0015] Calculating the difference between the dynamic adjustment reference value and the proportion to obtain a first value;

[0016] When it is detected that the first value is greater than 0, the ratio of the first value to the dynamically adjusted reference value is used as the second value;

[0017] When it is detected that the first value is less than or equal to 0, setting 0 as the second value;

[0018] The second value is used as a function calculation result of the second sub-function.

[0019] Optionally, the training steps of the target model include: extracting features from multiple training samples to obtain a feature vector for each training sample; dividing the feature vectors of all training samples into n feature vector sets; performing weighted sum calculation on the first type of loss function, the second type of loss function, and the third type of loss function to obtain a target loss function; iteratively training the neural network based on the target loss function and the n feature vector sets to obtain a target model.

[0020] Optionally, performing feature extraction on the target traffic data to obtain target traffic features includes: extracting at least one of the following traffic features from the target traffic data to obtain the target traffic features:

[0021] The first type of traffic features is used to characterize the statistical distribution information related to the target traffic data;

[0022] The second type of traffic features is used to characterize the transmission protocol information related to the target traffic data;

[0023] The third type of traffic features is used to characterize the raw byte information of the target traffic data;

[0024] The fourth type of traffic characteristics is used to characterize the time characteristic information corresponding to the target traffic data;

[0025] The fifth type of traffic feature is used to characterize the spatial feature information corresponding to the target traffic data.

[0026] According to another aspect of the present application, a flow detection device is provided, comprising: a feature extraction unit for extracting features from target flow data to obtain target flow features; a detection unit for inputting the target flow features into a target model, and detecting the type of the target flow data based on the target flow features according to prior knowledge of the target model during a model training phase, wherein the target model is trained using the following three types of loss functions:

[0027] The first type of loss function is used to determine the difference between the neural network's prediction results for the training examples and the training labels;

[0028] The second type of loss function is used to determine the similarity between training samples and normal samples, and classify the training samples based on the similarity;

[0029] The third type of loss function is used to dynamically adjust the weights of training samples when the training samples are detected as abnormal traffic.

[0030] According to another aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program runs, the device where the computer-readable storage medium is located executes the above-mentioned traffic detection method.

[0031] According to another aspect of the present application, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the above-mentioned traffic detection method.

[0032] As can be seen from the above content, this application proposes a malicious traffic detection method based on target traffic data feature extraction and multi-type loss function training. Compared with the existing technology, it significantly improves the coverage and accuracy of abnormal traffic detection in at least the following aspects:

[0033] First, the first-class loss function used in this application ensures that the model can accurately distinguish known normal and abnormal traffic categories during the training phase. By determining the difference between the predicted results and the actual training labels, the model can effectively learn the classification characteristics of various types of traffic, providing a solid classification foundation for abnormal traffic detection.

[0034] Secondly, the introduced separation loss function (i.e., the second-type loss function) encourages the distinction between abnormal and normal traffic by measuring the similarity between input traffic and normal samples and classifying training samples accordingly. This method is particularly effective in addressing the hidden nature of abnormal traffic. Even if abnormal and normal traffic partially overlap in the feature space, the distance between samples can be calculated to enable the model to learn the characteristic distribution of abnormal traffic, reducing false positives and improving the ability to identify new types of unknown malicious traffic.

[0035] Finally, the inclusion of a dynamic density-aware term (i.e., the third-class loss function) addresses the class imbalance problem caused by a small number of abnormal traffic samples during training. By dynamically adjusting the weights of rare abnormal traffic, the model ensures that even in the presence of severe sample imbalance, it pays full attention to all abnormal traffic. This prevents minority abnormal traffic from being overlooked or misclassified, thereby improving overall detection coverage and recognition accuracy.

[0036] By combining three loss functions, the proposed method not only excels in detecting known anomalous traffic but, more importantly, maintains a high detection rate even for unknown or novel malicious traffic. The inclusion of separation loss and dynamic density-aware terms enables the model to effectively learn key features that distinguish anomalous from normal traffic, even if these features were not present in the training data. This significantly enhances the model's generalization capabilities, enabling it to accurately detect and classify evolving malicious behavior in a deployed environment.

[0037] In summary, this application effectively addresses the problems of low coverage, insufficient identification of rare abnormal traffic, and poor model generalization in existing technologies for abnormal traffic detection through a multi-type loss function training mechanism. This method not only accurately detects known abnormal traffic but also effectively addresses unknown malicious behavior, providing a more powerful and flexible solution for real-time network monitoring and abnormal traffic identification, significantly improving network security protection capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 is a flow chart of an optional flow detection method according to an embodiment of the present application;

[0040] Figure 2 is a schematic diagram of an optional model training process according to an embodiment of the present application;

[0041] Figure 3This is a schematic diagram of a flow detection device according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0044] It should also be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected by this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0045] According to an embodiment of the present application, an embodiment of a flow detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] In an optional embodiment, a flow detection system can be used as the execution subject of the flow detection method in the embodiment of the present application, wherein the flow detection system can be a software system or an embedded system that combines software and hardware. Moreover, those skilled in the art should be aware that, in addition to using the flow detection system as the execution subject, the execution subject of the flow detection method of the present application can also be other devices, equipment, etc., and the present application does not specifically limit the specific form of the execution subject.

[0047] In order to more clearly illustrate the technical solution of the present application, the following description will use the flow detection system as the execution entity to illustrate the implementation steps of the solution.

[0048] Figure 1 is a flow chart of an optional flow detection method according to an embodiment of the present application, such as Figure 1 As shown, the method includes the following steps:

[0049] Step S101: extract features from target flow data to obtain target flow features.

[0050] Alternatively, feature extraction can be viewed as a data transformation process, converting raw, complex, and high-dimensional network traffic data into a set of low-dimensional feature vectors that are easy to understand and process. These feature vectors contain key traffic attributes, revealing similarities and differences between traffic flows and forming the basis for subsequent modeling and analysis.

[0051] The target traffic characteristics include at least one of the following characteristics:

[0052] Traffic statistics modal characteristics: These include statistical characteristics such as the number of bytes in traffic, the number of packets, the average packet size, and packet length distribution, as well as temporal characteristics such as the interval between packets and session duration. Traffic statistics reflect the intensity and regularity of communication activity and are important clues for identifying abnormal traffic.

[0053] Protocol modal features: These are used to characterize information at the network protocol level, such as transport layer protocols (TCP / UDP flags, encryption protocol type) and application layer protocols (SNI, ALPN information, TLS version, encryption algorithm type, etc.). Protocol features are crucial for understanding the semantics and context of traffic and can help models identify anomalous behavior under specific protocols.

[0054] Raw byte modal features: This involves directly processing the first x bytes of a data packet, usually selecting 64 or 128 bytes. This is the most original information in the traffic and can be serialized to capture the patterns and random distribution in encrypted traffic.

[0055] Temporal and spatial characteristics: These include the time interval sequence of traffic packets, as well as the geographic location and network topology of the traffic source and destination. These characteristics help establish abnormal patterns in time series and geographic distribution, identifying possible attack paths or signs of data leakage.

[0056] Optionally, after obtaining the target traffic features, feature engineering can be performed as needed to further improve the quality and applicability of the features. Feature engineering may include steps such as feature selection, feature transformation, and feature combination. Feature selection includes: screening out the most representative feature subset based on the importance of the features and their relevance to malicious traffic, reducing redundancy and improving model efficiency. Feature transformation includes: normalizing, standardizing, or encoding the selected features to ensure that all features are within a reasonable numerical range and avoid training skew caused by differences in numerical values. Feature combination includes: utilizing multimodal attention mechanisms or feature splicing to integrate features of different modalities into a unified space so that the model can simultaneously consider multiple perspectives of the traffic.

[0057] In step S102, the target flow characteristics are input into the target model, and the type of the target flow data is detected based on the target flow characteristics according to the prior knowledge of the target model in the model training phase.

[0058] In step S102, the target model is trained using the following three types of loss functions:

[0059] The first type of loss function is used to determine the difference between the neural network's prediction results for the training examples and the training labels;

[0060] The second type of loss function is used to determine the similarity between training samples and normal samples, and classify the training samples based on the similarity;

[0061] The third type of loss function is used to dynamically adjust the weights of training samples when the training samples are detected as abnormal traffic.

[0062] Optionally, first, the target model is a deep learning model, such as a convolutional neural network (CNN), a recurrent neural network (RNN), a Transformer model or its variants, which is specifically used to process and analyze network traffic data. After the previous feature extraction step, the feature representation of the target traffic data has been obtained. These features include but are not limited to traffic statistics, protocol information, byte sequence information, and time and space characteristics, which constitute a multi-dimensional vector describing traffic behavior. Next, the traffic detection system can input these target traffic features into the target model. The model will perform deep learning based on this information and finally output the detection results, that is, determine which type the input traffic belongs to, including but not limited to normal traffic or different malicious traffic categories.

[0063] It should be noted that during the model training phase, in order to guide model learning and optimization, this application designs and uses three types of unique loss functions:

[0064] The first type of loss function: also known as the classification loss function, this loss function is mainly used to supervise the learning process of the model to ensure that the difference between the model's prediction results on the training samples and the existing labels (training labels) is minimized as much as possible. By minimizing the classification loss, the model can learn the difference between normal traffic and malicious traffic (i.e., abnormal traffic, the malicious traffic described in the embodiments of this application all belong to abnormal traffic), as well as the characteristic differences between different types of malicious traffic, so as to accurately distinguish them during the testing phase. Typically, the classification loss function uses cross entropy loss or similar indicators to quantify the distance between the predicted probability distribution and the true label distribution.

[0065] The second type of loss function, also known as the separation loss function, is designed to enhance the model's understanding of the differences between malicious and normal traffic. By calculating the distance between training samples and normal samples, this loss function encourages the model to distinguish malicious traffic from normal traffic, enabling it to correctly classify malicious traffic even when its characteristics are somewhat similar to normal traffic. This second type of loss function enhances the model's ability to identify potentially unknown or mutated malicious traffic, improving the comprehensiveness and robustness of detection.

[0066] The third type of loss function, also known as the dynamic density-aware term, addresses the common class imbalance problem in malicious traffic detection. In training datasets, normal traffic samples far outnumber malicious traffic samples, causing model training to favor normal traffic and potentially overlook rare malicious classes. The dynamic density-aware term dynamically adjusts the weights of different class samples, specifically increasing the influence of rare or minority malicious traffic. This allows the model to prioritize learning about these rare malicious traffic types during training, ensuring a high detection rate even for rare attack patterns.

[0067] By combining these three loss functions, the target model can obtain more comprehensive and accurate learning signals during training, thereby demonstrating better malicious traffic detection performance during testing. Each loss function is optimized for different challenges in malicious traffic detection: for example, the classification loss function ensures that the model can accurately classify based on existing knowledge; the separation loss function enhances the model's ability to identify unknown malicious traffic; and the dynamic density-aware term addresses class imbalance by adjusting sample weights, ensuring that the model focuses on rare malicious types.

[0068] In summary, the target model in this application's technical solution, trained through three carefully designed loss functions, can effectively detect the type of target traffic data, maintaining high detection coverage and accuracy even when faced with encrypted or highly concealed malicious traffic. This approach not only improves the model's generalization performance, enabling it to better respond to unknown threats, but also enhances its ability to identify existing malicious traffic.

[0069] In an optional embodiment, during the process of training the target model using the first-type loss function, the traffic detection system can detect the neural network's predicted probability of each training sample belonging to each type of traffic, and then use the total number of training samples, the predicted probability of each training sample belonging to each type of traffic, and the training label corresponding to each training sample as parameters to be calculated. After substituting the parameters to be calculated into the first-type loss function for calculation and obtaining a first loss value, the neural network is trained based on the first loss value.

[0070] Optionally, a first-class loss function, i.e., a classification loss function, is used to measure the difference between the model's prediction results and the actual labels of the training samples (i.e., training labels). In deep learning, commonly used classification loss functions include cross-entropy loss. This loss function can quantify the distance between the model's predicted probability distribution and the actual label probability distribution, thereby evaluating the model's prediction accuracy. Specifically, the first-class loss function can refer to the following formula (1):

[0071]

[0072] In formula (1), y i,j represents the training label corresponding to the i-th training sample (i.e., the true category of the i-th training sample is the j-th traffic category), where i and j are both integers greater than or equal to 1. i,j It can be in coded form. For example, if the true category of the i-th training sample is the j-th traffic, then y i,j =1, and other positions are 0. =(\begin{aligned}) represents the neural network's predicted probability that the i-th training sample belongs to the j-th traffic category. N refers to the number of training samples. 1 / N is used to take the mean, so that the total loss is the average loss per sample. This ensures that the magnitude of the loss remains consistent regardless of the size of the training sample set. K is the total number of classes, including various malicious traffic types and normal traffic.

[0073] When training a neural network, the feature vector of each training example is first input into the model to obtain the model's predicted probability distribution. Subsequently, a first-class loss function (such as the cross-entropy loss function mentioned above) is used to calculate the difference between the model's prediction and the actual label, resulting in a first loss value (i.e., classification loss). This loss value reflects the model's accuracy in classifying the current batch of examples and is a key indicator for guiding model parameter updates.

[0074] After obtaining the first loss value, the backpropagation algorithm propagates the gradient of the loss value back to all layers of the model, adjusting the model's weights and bias parameters to minimize the loss value. This process is repeated until the model achieves a certain level of accuracy in classifying the training data or reaches a predetermined number of training rounds.

[0075] Neural network training is an iterative process. In each round of training, the model receives a randomly selected subset of training data (also called a batch) and calculates a loss based on the inputs and outputs of that batch. The model then adjusts its parameters based on this loss, aiming to minimize the average loss function over all training data. This process gradually optimizes the model parameters, enabling it to learn the characteristics that distinguish normal from malicious traffic from the training data, thereby improving detection performance.

[0076] By calculating the difference between the model prediction and the actual label through the first-class loss function and guiding the adjustment of model parameters based on the loss value, it ensures that the model can not only learn the characteristics of known malicious traffic, but also continuously optimize its classification ability, thereby more accurately detecting and classifying unknown or mutated malicious traffic during the deployment phase, and improving the reliability and effectiveness of network protection.

[0077] In an optional embodiment, in the process of training the target model through the second type of loss function, the traffic detection system can detect the predicted probability of the neural network for each training sample belonging to each type of traffic, wherein each type of traffic includes at least normal traffic and abnormal traffic. Then, the traffic detection system determines the center vector of each type of traffic, wherein the center vector of each type of traffic represents the feature vector at the center position of all feature vectors corresponding to that type of traffic. Subsequently, the traffic detection system calculates the vector distance between the feature vector of each training sample and the center vector of each type of traffic, and substitutes the vector distance, the predicted probability of the neural network for each training sample belonging to each type of traffic, and the total number of training samples as parameter values into the second type of loss function for calculation to obtain a second loss value. Finally, the traffic detection system trains the neural network according to the second loss value.

[0078] Optionally, before training begins, the traffic detection system needs to determine the center vector for each traffic type. The center vector can be understood as the geometric center point of the set of feature vectors for that traffic type, representing the most typical characteristics of that traffic type. For normal and abnormal traffic, the corresponding center vectors are collected and calculated separately. A common method for calculating the center vector is to average all feature vectors belonging to the same class to obtain an average vector, which is the center vector.

[0079] For each training sample, its feature vector is compared with the center vector of each type of traffic, and the vector distance between the two is calculated. Vector distance can be measured in many ways, such as Euclidean distance, Manhattan distance, or cosine similarity.

[0080] Next, the predicted probability, vector distance, and total number of training samples for each training sample are used as parameters and substituted into the second type of loss function for calculation to obtain the second loss value. The purpose of the second type of loss function is to encourage the model to predict abnormal traffic samples as abnormal while ensuring that these samples are sufficiently far from the center vector of normal traffic. The formula for the second loss function can be referred to the following formula (2):

[0081]

[0082] In formula (2), α refers to the preset constant term, It represents the predicted probability of the neural network for the i-th training sample belonging to the j-th type of traffic. N refers to the number of training samples. 1 / N is used to take the mean. K is the total number of categories, including various malicious traffic types and normal traffic. d(x i ,c j ) represents the training sample (x i ) and traffic category c j The distance between the center vectors of j is the center vector of the jth type of traffic, and the center vector of normal traffic is represented by c norma l.

[0083] By minimizing the value of the second-category loss function, the model is expected to learn a mechanism that makes the feature vectors of abnormal traffic samples farther away from the center vector of normal traffic, and these samples are more likely to be predicted as abnormal. In this way, even if abnormal traffic has similarities with normal traffic in feature space, the model can make correct classification decisions based on distance information, improving detection coverage and accuracy.

[0084] Optionally, after determining the center vector and vector distance, a second type of loss function can be used to guide the model training process. In each round of training, the model adjusts its parameters based on the calculated second loss value, hoping to reduce this loss value in the next round of training. This process is iterated until the model converges, indicating that the model can effectively distinguish normal from abnormal traffic, even in cases with complex overlap in the feature space.

[0085] Through the above steps, the second-type loss function training process is essentially a guided learning process in the feature space. It encourages the model to form clear category boundaries in the feature space, especially the boundaries between abnormal and normal traffic. This mechanism not only improves the model's classification ability, but also enhances its robustness and generalization ability when facing unknown or variant malicious traffic.

[0086] In an optional embodiment, during training of the target model using the third-type loss function, the traffic detection system may separately detect the proportion of training samples carrying each label in the training sample set, and then obtain a dynamically adjusted baseline value. The dynamically adjusted baseline value is used to represent the density of the most frequently occurring category among all abnormal traffic samples in the training sample set. The traffic detection system may determine a third loss value based on the proportion, the dynamically adjusted baseline value, and the third-type loss function, and then train the neural network based on the third loss value.

[0087] Optionally, a third loss function, also known as a dynamic density-aware term, addresses poor detection performance when certain malicious traffic classes have very few examples in the training dataset. This imbalance can easily bias the model toward frequently occurring classes, significantly compromising detection performance for infrequent or unprecedented malicious traffic.

[0088] The traffic detection system first calculates the proportion of training samples carrying each label (i.e., each category) in the training sample set. Calculating this proportion is crucial for understanding the distribution of each category. The specific calculation formula is as follows (3):

[0089]

[0090] In formula (3), II[y k =y i ] is an indicator function, indicating that the label is y i The number of samples, where y k represents the label of the kth class.

[0091] Next, the flow detection system obtains the dynamic adjustment reference value (Density max), the dynamically adjusted baseline value actually refers to the density of the category that appears most frequently among all abnormal traffic samples. This baseline value is used to determine which categories should receive extra attention because they are relatively rare in the dataset and easily ignored by the model.

[0092] Optionally, a dynamic density-aware loss, or the third loss, is calculated based on the proportion of training samples carrying each label in the training sample set and a dynamically adjusted baseline. The design philosophy of this loss function is to give higher weight to categories with fewer samples (lower density) during model training, ensuring that their influence is not diluted during training, thereby improving the model's ability to detect low-frequency malicious behavior.

[0093] By calculating the third loss, this loss term can be incorporated into the model's overall loss function, guiding the update of model parameters together with the classification loss and separation loss. At each step of model training, the system calculates the total loss for the current batch, which includes the contributions of the first, second, and third loss functions. The system then uses the backpropagation algorithm to adjust the neural network weights and biases based on this total loss to minimize the total loss.

[0094] In this way, the third-category loss function ensures that during training, the model not only learns to distinguish a large amount of common malicious traffic, but also discerns smaller, rarer anomalous traffic, thereby improving overall detection coverage and accuracy. During the training process of the third-category loss function, the dynamic density-aware term cleverly addresses the class imbalance problem in malicious traffic detection. This mechanism automatically identifies and prioritizes low-frequency malicious traffic categories, ensuring that after training, the model possesses robust detection capabilities and can effectively identify a diverse range of anomalous traffic, including malicious behaviors rarely seen in the training data, significantly enhancing the system's overall defense effectiveness.

[0095] In an optional embodiment, the third type of loss function is used to represent the product of the first sub-function and the second sub-function, wherein the first sub-function is used to constrain the second sub-function to be effective for abnormal traffic samples during model training and not to be effective for normal traffic samples; the second sub-function is used to perform the following steps:

[0096] Calculating the difference between the dynamic adjustment reference value and the proportion to obtain a first value;

[0097] When it is detected that the first value is greater than 0, the ratio of the first value to the dynamically adjusted reference value is used as the second value;

[0098] When it is detected that the first value is less than or equal to 0, setting 0 as the second value;

[0099] The second value is used as a function calculation result of the second sub-function.

[0100] Optionally, in the malicious traffic detection technology of this application, the third type of loss function is designed as the product of two parts, namely the first sub-function and the second sub-function, to dynamically adjust the model's sensitivity to abnormal traffic samples while avoiding unnecessary impact on normal traffic samples. This mechanism ensures that the model can more effectively identify abnormal traffic that occurs less frequently in the dataset through a sophisticated weight adjustment strategy, thereby improving the overall detection accuracy and coverage. The following is a detailed explanation of the first and second sub-functions involved in the third type of loss function:

[0101] The first subfunction II[y i >0] is actually an indicator function, which is used to filter out samples belonging to the abnormal traffic category. During the model training process, when the model predicts that a sample belongs to the abnormal traffic category (i.e., y i > 0), the first sub-function returns 1, so that the abnormal traffic sample can be further processed by the second sub-function; and for the sample predicted by the model to be normal traffic (i.e., y i <0), the first sub-function returns 0, which means that the second sub-function will not perform any operation on these samples, ensuring that normal traffic samples will not be misjudged or over-adjusted.

[0102] The purpose of the second sub-function is to dynamically adjust the weight of abnormal traffic samples in the loss function calculation to deal with the problem of class imbalance. Specifically, the second sub-function can refer to the following formula (4):

[0103]

[0104] The formula expression of the third type of loss function can refer to the following formula (5).

[0105]

[0106] By combining the results of the first and second sub-functions, the third loss function adaptively adjusts the model's handling of anomalous traffic samples, ensuring that rare but equally important malicious traffic types are not overlooked during model training. This mechanism, working in conjunction with the classification loss and separation loss, guides the model to balance predictive performance across classes during optimization, enabling more robust and accurate identification of anomalous traffic, particularly when faced with class imbalance.

[0107] The third loss function addresses class imbalance, a key issue in malicious traffic detection, by dynamically adjusting the weights of anomalous traffic samples. This mechanism not only improves the model's detection of low-frequency anomalous traffic types, but also, through the clever design of the first and second sub-functions, ensures that detection performance for normal traffic samples remains unaffected. This enables more comprehensive and reliable network traffic monitoring and anomaly detection in practical applications.

[0108] In an optional embodiment, the training steps of the target model include: extracting features from multiple training samples to obtain a feature vector for each training sample; dividing the feature vectors of all training samples into n feature vector sets; performing weighted sum calculation on the first type of loss function, the second type of loss function, and the third type of loss function to obtain a target loss function; iteratively training the neural network based on the target loss function and the n feature vector sets to obtain a target model.

[0109] Optionally, Figure 2 is a schematic diagram of an optional model training process according to an embodiment of the present application, such as Figure 2 As shown, multimodal feature extraction is first performed on the training samples in the training dataset, and then n feature vector sets are randomly generated in batches (n feature vector sets can be generated in batches of n equal parts). The neural network is trained using n feature vector sets, including: model parameter updates, expected category judgment, and loss calculations at each parameter iteration (mainly applying three types of loss functions to complete the calculation of classification loss, separation loss, and density loss). After multiple iterative training, if it is detected that the function value of the target loss function has entered a convergence state or the maximum number of training times has been reached, the training can be terminated to obtain the final target model.

[0110] The objective loss function can be expressed as follows:

[0111]

[0112] Where ω1 and ω2 are the weights of the first and third loss functions, respectively. In formula (6), the weight of the second loss function is set to 1.

[0113] It's important to note that dividing the feature vectors of all training samples into n sets of feature vectors is a strategy known as "batch training" in deep learning training. This approach not only significantly reduces computing resource consumption but also accelerates model convergence and improves training efficiency by optimizing parameters in each batch. This grouping allows the model to iterate quickly on smaller sets of samples, making overall training more efficient and stable.

[0114] The objective loss function combines classification loss, separation loss, and a dynamic density-aware term to achieve a balance between model accuracy and generalization. The classification loss enables the model to learn the correct traffic classifications, the separation loss forces the model to clearly distinguish between normal and abnormal traffic in the feature space, and the dynamic density-aware term addresses class imbalance, ensuring that the model pays sufficient attention to rare malicious types. Through a weighted summation calculation, the objective loss function provides comprehensive guidance for model training, ensuring that the model is optimized in all aspects.

[0115] The neural network is iteratively trained based on the target loss function. This process continuously adjusts model parameters, gradually enhancing the model's classification, separation, and density perception capabilities. Iterative training not only improves the model's fit to the training data but also enhances its adaptability to unseen data, particularly for infrequent malicious traffic. This approach significantly improves the model's robustness and generalization, enabling it to maintain high detection efficiency and accuracy in a constantly changing network environment.

[0116] In an optional embodiment, performing feature extraction on the target traffic data to obtain the target traffic feature includes: extracting at least one of the following traffic features from the target traffic data to obtain the target traffic feature:

[0117] The first type of traffic features is used to characterize the statistical distribution information related to the target traffic data;

[0118] The second type of traffic features is used to characterize the transmission protocol information related to the target traffic data;

[0119] The third type of traffic features is used to characterize the raw byte information of the target traffic data;

[0120] The fourth type of traffic characteristics is used to characterize the time characteristic information corresponding to the target traffic data;

[0121] The fifth type of traffic feature is used to characterize the spatial feature information corresponding to the target traffic data.

[0122] Optionally, the first type of traffic characteristics described above is also called traffic statistical modal characteristics. These include statistical characteristics such as the number of bytes in the traffic, the number of packets, the average packet size, and the length distribution of the packets, as well as temporal characteristics such as the interval between packets and the duration of the session. Traffic statistical characteristics can reflect the intensity and regularity of communication activity and are important clues for identifying abnormal traffic.

[0123] Optionally, the second type of traffic features mentioned above, also known as protocol modal features, are used to characterize information at the network protocol level, such as transport layer protocols (TCP / UDP flags, encryption protocol type) and application layer protocols (SNI, ALPN information, TLS version, encryption algorithm type, etc.). Protocol modal features are crucial for understanding the semantics and context of traffic and can help models identify anomalous behavior under specific protocols.

[0124] Optionally, the third type of traffic feature mentioned above is also called the original byte modal feature, which includes directly processing the first x byte sequence of the data packet, usually selecting 64 or 128 bytes. This is the most original information in the traffic, and can be captured through serialization processing to capture the pattern and random distribution in the encrypted traffic.

[0125] Optionally, the fourth type of traffic feature mentioned above is also called a time feature, which includes a time interval sequence of traffic packets.

[0126] Optionally, the fifth type of traffic feature mentioned above is also called a spatial feature, which includes the geographical location and network topology relationship between the traffic source address and the destination.

[0127] It should be noted that the first type of traffic features focuses on the statistical characteristics of target traffic data, such as packet size, packet rate, average packet length, etc. These statistical features can reflect the activity patterns and data transmission characteristics of the traffic. By analyzing this information, the system can identify statistical anomalies that are inconsistent with normal network behavior, such as sudden large traffic flows and frequent exchanges of small amounts of traffic. These anomalies are often indicative of malicious traffic. The second type of traffic features focuses on features at the transport protocol level, such as the distribution of TCP / UDP flag bits and the type of encryption protocol. Network protocol features are key clues in malicious traffic detection because malicious behavior often leaves specific traces at the protocol level. For example, some malware may tend to use specific TLS versions or encryption algorithms, and abnormal protocol field distributions (such as excessively long TCP establishment times and abnormal UDP packet structures) may also indicate potential malicious activity. The extraction of the second type of features can enhance the model's ability to identify malicious behavior at the protocol level and improve detection accuracy.

[0128] The third type of traffic feature analyzes the raw byte sequence of the target traffic data, such as packet header data, the first x byte sequence, and specific byte patterns of the TLS handshake. Raw byte information can capture detailed features in encrypted traffic, helping to identify malicious activities that attempt to evade detection through encryption. This type of feature extraction is particularly important for detecting unknown, new malicious traffic, as it may not exhibit obvious anomalies at the statistical or protocol level, but its raw byte sequence may contain characteristic patterns. By extracting the third type of features, the system can uncover deeper traffic attributes and improve its detection capabilities for encrypted malicious traffic.

[0129] The fourth type of traffic characteristics focuses on analyzing time series features, including inter-packet intervals, session durations, and traffic density. Time characteristics reveal patterns in traffic changes over time and are particularly effective for detecting time-based attack strategies (such as multi-stage attacks and time interval attacks). By observing these time characteristics, the system can identify malicious attacks that attempt to launch attacks at specific times or within specific time periods. Furthermore, changes in traffic density can reveal potential abnormal traffic activity. For example, unusually high traffic density during non-business hours may indicate malicious scanning or other network activity.

[0130] The fifth type of traffic characteristics involves the geographic location and network topology of the source and destination. Spatial characteristics can be used to categorize traffic based on geographic and network location, helping to identify traffic that may originate from malicious areas or networks. For example, some malicious activity may primarily originate from specific geographic locations or network segments. By analyzing these spatial characteristics, the system can more accurately locate and filter suspicious traffic.

[0131] The extraction and comprehensive analysis of the above five types of traffic features enable the malicious traffic detection system of this application to have the ability to gain insight into network traffic behavior from multiple angles and levels. By integrating information from multiple aspects such as statistics, protocols, byte sequences, time series, and spatial location, the system can construct a comprehensive traffic behavior profile, thereby more accurately identifying malicious traffic and avoiding false positives and missed reports. This comprehensive analysis method not only improves the accuracy and coverage of detection, but also enhances the system's generalization capabilities, enabling it to maintain a high degree of adaptability and robustness in the face of ever-changing network environments and malicious behavior.

[0132] According to another aspect of the embodiment of the present application, a flow detection device is further provided, wherein: Figure 3 is a schematic diagram of a flow detection device according to an embodiment of the present application, such as Figure 3As shown, the device includes: a feature extraction unit 301, which is used to extract features from target flow data to obtain target flow features; a detection unit 302, which is used to input the target flow features into the target model, and detect the type of target flow data based on the target flow features according to the prior knowledge of the target model in the model training stage, wherein the target model is trained by the following three types of loss functions:

[0133] The first type of loss function is used to determine the difference between the neural network's prediction results for the training examples and the training labels;

[0134] The second type of loss function is used to determine the similarity between training samples and normal samples, and classify the training samples based on the similarity;

[0135] The third type of loss function is used to dynamically adjust the weights of training samples when the training samples are detected as abnormal traffic.

[0136] Optionally, the traffic detection device includes: a first detection unit, used to detect the predicted probability of the neural network for each training sample belonging to each type of traffic; a first processing unit, used to use the total number of training samples, the predicted probability of each training sample belonging to each type of traffic, and the training label corresponding to each training sample as parameters to be calculated; a second processing unit, used to substitute the parameters to be calculated into the first type of loss function for calculation to obtain a first loss value; and a third processing unit, used to train the neural network according to the first loss value.

[0137] Optionally, the flow detection device includes: a second detection unit, used to detect the predicted probability of the neural network for each training sample belonging to each type of flow, wherein each type of flow includes at least normal flow and abnormal flow; a first determination unit, used to determine the central vector of each type of flow, wherein the central vector of each type of flow represents the eigenvector at the center position of all eigenvectors corresponding to that type of flow; a first calculation unit, used to calculate the vector distance between the eigenvector of each training sample and the central vector of each type of flow; a second calculation unit, used to substitute the vector distance, the predicted probability of the neural network for each training sample belonging to each type of flow, and the total number of training samples as parameter values into the second type of loss function for calculation to obtain a second loss value; and a first training unit, used to train the neural network according to the second loss value.

[0138] Optionally, the traffic detection device includes: a third detection unit, used to respectively detect the proportion of training samples carrying each label in the training sample set; an acquisition unit, used to obtain a dynamically adjusted reference value, wherein the dynamically adjusted reference value is used to characterize the density of the category with the highest frequency of occurrence in all abnormal traffic samples included in the training sample set; a second determination unit, used to determine a third loss value based on the proportion, the dynamic adjustment reference value and the third type of loss function; and a second training unit, used to train the neural network based on the third loss value.

[0139] Optionally, a third type of loss function is used to represent the product of the first sub-function and the second sub-function, wherein the first sub-function is used to constrain the second sub-function to be effective for abnormal traffic samples during model training and not to be effective for normal traffic samples; the second sub-function is used to perform the following steps:

[0140] Calculating the difference between the dynamic adjustment reference value and the proportion to obtain a first value;

[0141] When it is detected that the first value is greater than 0, the ratio of the first value to the dynamically adjusted reference value is used as the second value;

[0142] When it is detected that the first value is less than or equal to 0, setting 0 as the second value;

[0143] The second value is used as a function calculation result of the second sub-function.

[0144] Optionally, the flow detection device includes: a fourth processing unit, used to extract features from multiple training samples to obtain a feature vector for each training sample; a fifth processing unit, used to divide the feature vectors of all training samples into n feature vector sets; a sixth processing unit, used to perform weighted sum calculation on the first type of loss function, the second type of loss function, and the third type of loss function to obtain a target loss function; and a seventh processing unit, used to iteratively train the neural network based on the target loss function and the n feature vector sets to obtain a target model.

[0145] Optionally, the feature extraction unit 301 includes: an extraction subunit, configured to extract at least one of the following types of traffic features from the target traffic data to obtain the target traffic feature:

[0146] The first type of traffic features is used to characterize the statistical distribution information related to the target traffic data;

[0147] The second type of traffic features is used to characterize the transmission protocol information related to the target traffic data;

[0148] The third type of traffic features is used to characterize the raw byte information of the target traffic data;

[0149] The fourth type of traffic characteristics is used to characterize the time characteristic information corresponding to the target traffic data;

[0150] The fifth type of traffic feature is used to characterize the spatial feature information corresponding to the target traffic data.

[0151] According to another aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed, the device where the computer-readable storage medium is located executes the above-mentioned traffic detection method.

[0152] According to another aspect of the present application, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned traffic detection method.

[0153] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0154] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0156] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0157] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0159] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A flow detection method, characterized in that: include: Extract features from target traffic data to obtain target traffic features; The target traffic characteristics are input into the target model. Based on the prior knowledge of the target model in the model training phase, the type of the target traffic data is detected based on the target traffic characteristics. The target model is trained using the following three types of loss functions: The first type of loss function is used to determine the difference between the neural network's prediction results for the training examples and the training labels; The second type of loss function is used to determine the similarity between the training sample and the normal sample, and classify the training sample according to the similarity; The third type of loss function is used to dynamically adjust the weight of the training sample when it is detected that the training sample is abnormal traffic.

2. The method according to claim 1, characterized in that In the process of training the target model using the first type of loss function, the method further includes: Detecting the predicted probability of the neural network for each of the training samples belonging to each type of traffic; The total number of training samples, the predicted probability of each training sample belonging to each type of traffic, and the training label corresponding to each training sample are used as parameters to be calculated; Substituting the parameters to be calculated into the first type of loss function for calculation to obtain a first loss value; The neural network is trained according to the first loss value.

3. The method according to claim 1, characterized in that In the process of training the target model using the second type of loss function, the method further includes: detecting a predicted probability of the neural network, for each of the training samples, belonging to each type of traffic, wherein the various types of traffic include at least normal traffic and abnormal traffic; Determine a central vector for each type of traffic, wherein the central vector for each type of traffic represents a feature vector at a central position among all feature vectors corresponding to the type of traffic; Calculating the vector distance between the feature vector of each training sample and the center vector of each type of traffic; Substituting the vector distance, the predicted probability of each training sample belonging to each type of traffic by the neural network, and the total number of training samples as parameter values into the second type of loss function for calculation to obtain a second loss value; The neural network is trained according to the second loss value.

4. The method according to claim 1, wherein In the process of training the target model by using the third type of loss function, the method further includes: Detect the proportion of training samples carrying each label in the training sample set respectively; Obtaining a dynamically adjusted reference value, wherein the dynamically adjusted reference value is used to represent the density of the category with the highest frequency of occurrence among all abnormal traffic samples included in the training sample set; Determining a third loss value according to the proportion, the dynamically adjusted reference value, and the third type of loss function; The neural network is trained according to the third loss value.

5. The method according to claim 4, characterized in that The third type of loss function is used to represent the product of the first sub-function and the second sub-function, wherein the first sub-function is used to constrain the second sub-function to be effective for abnormal traffic samples during model training and not to be effective for normal traffic samples; the second sub-function is used to perform the following steps: Calculating a difference between the dynamic adjustment reference value and the proportion to obtain a first value; When it is detected that the first value is greater than 0, the ratio of the first value to the dynamic adjustment reference value is used as the second value; When detecting that the first value is less than or equal to 0, setting 0 as the second value; The second value is used as a function calculation result of the second sub-function.

6. The method according to claim 1, characterized in that The training steps of the target model include: Performing feature extraction on the plurality of training samples to obtain a feature vector for each training sample; Dividing the feature vectors of all the training samples into n feature vector sets; Performing weighted sum calculation on the first type loss function, the second type loss function, and the third type loss function to obtain a target loss function; The neural network is iteratively trained according to the target loss function and the n feature vector sets to obtain the target model.

7. The method according to claim 1, characterized in that Extract the target traffic data to obtain the target traffic features, including: Extract at least one of the following traffic characteristics of the target traffic data to obtain the target traffic characteristics: The first type of traffic characteristics is used to characterize the statistical distribution information related to the target traffic data; The second type of traffic characteristics is used to characterize the transmission protocol information related to the target traffic data; The third type of traffic feature is used to characterize the original byte information of the target traffic data; The fourth type of traffic characteristics is used to characterize the time characteristic information corresponding to the target traffic data; The fifth type of traffic characteristics is used to characterize the spatial characteristic information corresponding to the target traffic data.

8. A flow detection device, characterized in that: include: A feature extraction unit is used to extract features from target flow data to obtain target flow features; A detection unit is configured to input the target flow characteristics into a target model, and detect the type of the target flow data based on the target flow characteristics according to the prior knowledge of the target model during the model training phase, wherein the target model is trained using the following three types of loss functions: The first type of loss function is used to determine the difference between the neural network's prediction results for the training examples and the training labels; The second type of loss function is used to determine the similarity between the training sample and the normal sample, and classify the training sample according to the similarity; The third type of loss function is used to dynamically adjust the weight of the training sample when it is detected that the training sample is abnormal traffic.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the flow detection method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the flow detection method described in any one of claims 1 to 7.

Images