Index construction method and traffic query method, device and storage medium

Abstract

Embodiments of the present application provide an index construction method and a traffic query method, device and storage medium. In the embodiments of the present application, feature extraction is performed on historical abnormal packets to obtain feature vectors of the historical abnormal packets. Then, the feature vectors of the historical abnormal packets can be divided into a plurality of feature vector partitions, and a query index corresponding to each feature vector partition is constructed using the feature vectors in the feature vector partition, thereby completing the construction of the query index of abnormal traffic and providing a basis for the query of network attack data. Since the data volume of the feature vectors of the abnormal packets in each feature vector partition is greatly reduced compared to the feature vectors of the full volume of abnormal packets, the construction time of the query index of each partition is greatly reduced, thereby reducing the overall construction time of the query index of abnormal traffic, and the construction time of the query index is within an acceptable range.

Description

Technical Field

[0001] This application relates to the field of network security technology, and in particular to an index construction method and a traffic query method, device and storage medium. Background Technology

[0002] In the field of cybersecurity, the internet exposes a large number of payloads. A payload is code released by attackers after a successful exploitation of a vulnerability, possessing offensive capabilities and capable of achieving the attacker's objectives. Cybersecurity professionals can analyze these payloads and convert them into standardized machine- or human-readable formats. These standardized payload formats can then be used to write defense rules for network protection.

[0003] The inventors of this application have discovered that network attack data within historical traffic data also contains certain information that can be used to develop defense rules. However, the massive amount of historical traffic data makes retrieving network attack data extremely difficult, necessitating the construction of an index for historical traffic data to enable the querying of network attack data. Summary of the Invention

[0004] This application provides an index building method and a traffic query method, device, and storage medium to build a query index corresponding to a process query, providing a foundation for querying network attack data.

[0005] This application provides an index construction method, including: Obtain historical abnormal messages; extract features from the historical abnormal messages to obtain the feature vector of the historical abnormal messages; Based on at least one defined classification dimension, the feature vectors of the historical abnormal messages are classified at least once to obtain multiple feature vector partitions; the number of classifications is the same as the number of classification dimensions. For any feature vector partition among the plurality of feature vector partitions, a query index for that feature vector partition is constructed using the feature vectors in that feature vector partition.

[0006] This application also provides a traffic query method, including: Obtain the traffic sample to be queried; Feature extraction is performed on the traffic sample to be queried to obtain the feature vector of the traffic sample to be queried; From the traffic sample to be queried, obtain the value of the traffic sample to be queried on the target classification dimension; the target classification dimension is the classification dimension corresponding to multiple feature vector partitions; each feature vector partition corresponds to a query index; Based on the value of the traffic sample to be queried in the target classification dimension and the value of the classification dimension corresponding to the multiple feature vector partitions, the target feature vector partition is determined from the multiple feature vector partitions; Using the feature vector of the traffic sample to be queried and the query index of the target feature vector partition, a query is performed in the target feature vector partition to obtain the target feature vector; Obtain the historical abnormal packets corresponding to the target feature vector as similar traffic to the traffic sample to be queried.

[0007] This application embodiment also provides a computing device, including: a memory and a processor; wherein, the memory is used to store computer programs; The processor is coupled to the memory and is used to execute the computer program for performing the steps in the above-described index building method and / or traffic query method.

[0008] This application also provides a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps in the above-described index building method and / or traffic query method.

[0009] In this embodiment, feature extraction is performed on historical abnormal packets to obtain feature vectors. These feature vectors are then divided into multiple feature vector partitions. A query index corresponding to each feature vector partition is constructed using the feature vectors from that partition, completing the construction of the query index for abnormal traffic and providing a foundation for querying network attack data. Since the amount of data in the feature vectors of abnormal packets in each feature vector partition is significantly reduced compared to the feature vectors of all abnormal packets, the construction time of the query index for each partition is significantly reduced, thereby reducing the overall construction time of the query index for abnormal traffic and keeping it within an acceptable range. Furthermore, based on the above partition-based query index construction method, when querying similar traffic, the classification dimension of the feature vector partitions can be used to route the queried packet to a small number of local feature vector partitions without requiring a global query, which helps improve the efficiency of subsequent similar traffic queries. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating the index construction method provided in this application embodiment; Figure 2A schematic diagram illustrating the specific process of the index construction method provided in this application embodiment; Figure 3 A schematic diagram illustrating the principle of constructing a query index using the HNSW algorithm, provided for embodiments of this application; Figure 4 A flowchart illustrating the traffic query method provided in this application embodiment; Figure 5 A schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] In some embodiments of this application, feature extraction is performed on historical abnormal packets to obtain feature vectors for these packets. Subsequently, a query index corresponding to traffic retrieval can be constructed based on these feature vectors, providing a foundation for querying network attack data. Specifically, constructing a query index using the feature vectors of historical abnormal packets allows subsequent similar traffic queries to utilize the feature vectors of the packets to be queried within the packet database, eliminating the need for keyword-based matching and thus improving the efficiency of subsequent similar traffic searches.

[0013] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0014] It should be noted that the same reference numerals denote the same object in the following figures and embodiments. Therefore, once an object is defined in one figure or embodiment, it does not need to be discussed further in subsequent figures and embodiments.

[0015] Figure 1 This is a flowchart illustrating the index construction method provided in an embodiment of this application. Figure 1 As shown, the index construction method mainly includes: 101. Obtain historical abnormal messages; historical abnormal messages should include at least attack messages.

[0016] 102. Extract features from historical abnormal messages to obtain feature vectors of historical abnormal messages.

[0017] 103. Based on at least one defined classification dimension, classify the feature vectors of historical abnormal messages at least once to obtain multiple feature vector partitions; the number of classifications is the same as the number of classification dimensions.

[0018] 104. For any feature vector partition B among multiple feature vector partitions, construct a query index for feature vector partition B using the feature vectors in feature vector partition B.

[0019] In this embodiment, the message can be a request message or a response message. An abnormal message refers to a message that poses a risk of network attack. Abnormal messages include at least attack messages. An attack message is a message used for network attacks.

[0020] Since abnormal packets can reflect the characteristics of network attack packets to a certain extent, abnormal packets can be obtained in step 101. In this embodiment, the specific implementation method for obtaining abnormal packets is not limited.

[0021] In some embodiments, abnormal packets collected offline can be obtained as historical abnormal packets in step 101. These offline-collected abnormal packets may be manually labeled abnormal packets, or packets already blocked by an existing firewall, but are not limited to these. For embodiments that generate defense rules offline, the abnormal packets may be attack packets. In this application embodiment, the firewall may be a cloud firewall or a firewall deployed locally on a computing device.

[0022] In other embodiments, such as Figure 2 As shown, historical packets can be retrieved from the database; and normal packets can be filtered out from the historical packets to obtain the historical abnormal packets in step 101. The database stores historical packets. The database can be a local database or a cloud data warehouse, etc. This normal network traffic filtering method can reduce data storage pressure. On the other hand, since normal packets are filtered out from massive historical packets in advance, only abnormal packets are available when building the query index corresponding to abnormal traffic, which helps to improve the efficiency of building the query index for abnormal traffic.

[0023] In this application, the specific implementation method for filtering out normal messages from historical messages is not limited. Several optional implementation methods are described below by way of example.

[0024] Implementation Method 1: A packet whitelist is pre-configured. The packet whitelist contains packets that are allowed to pass through the firewall and can be considered normal packets. Accordingly, the packet whitelist can be used to filter out normal packets from historical packets to obtain historical abnormal packets.

[0025] In some embodiments, the message whitelist is characterized by the domain name and source Internet Protocol (IP) address of the message; that is, the message whitelist includes both the domain name and the source IP address. Accordingly, the domain name and source IP address of historical messages can be used to match the domain name and source IP address recorded in the message whitelist. If the domain name and source IP address of a historical message exist in the message whitelist, then the historical message is determined to be a normal message. Furthermore, messages other than normal messages in the historical messages can be identified as historical abnormal messages.

[0026] Implementation Method 2: The inventors of this application have discovered that normal packets access the same address frequently. Based on this, the access addresses of historical accesses, such as destination IP addresses and / or Uniform Resource Locator (URL) addresses, can be obtained from historical packets. Furthermore, based on the access addresses of historical packets, the access frequency of the same access address within a time window can be determined; and normal packets with access frequencies to the same access address greater than or equal to a set access frequency threshold can be filtered out from historical packets, thereby obtaining historical abnormal packets from the historical packets.

[0027] Specifically, after determining the frequency of access to the same access address within a time window, historical packets with an access frequency greater than or equal to the access frequency threshold can be identified as normal packets. Furthermore, normal packets can be filtered out from historical packets to obtain historical abnormal packets.

[0028] Implementation Method 3: A Bloom filter can be constructed based on pre-determined normal packets. The pre-determined normal packets can be packets allowed by the firewall, and / or, manually labeled normal packets, and / or, packets conforming to a packet whitelist, etc. Specifically, multiple attribute information of the normal packets can be obtained from them. Packet attributes refer to the fields contained in the packet, which may include: packet metadata, protocol type, packet header fields, and packet body content, etc.

[0029] The metadata of a message may include: message direction, encoding method, special characters contained in the message, and the corresponding data unit. Message direction refers to whether the message originates from an asset protected by the firewall or is sent to an asset protected by the firewall. For messages originating from assets protected by the firewall, the message direction can be defined as outgoing (OUT); for messages sent to assets protected by the firewall, the message direction can be defined as incoming (IN).

[0030] The encoding method of a message refers to the encoding method of the application layer data of the message, that is, what encoding algorithm is used.

[0031] The special characters contained in the message can be determined based on the pre-configured special characters. The special characters belonging to the pre-configured special characters can be extracted from the message, that is, the number of special characters contained in the abnormal message can be obtained.

[0032] The data unit corresponding to a message refers to whether the message is a single packet or a stream message. A single packet message means that data is transmitted using individual data packets. A stream message refers to a message with a larger data volume, where the data to be transmitted is divided into individual data packets and sent sequentially using a data stream method.

[0033] In this embodiment, the Bloom filter consists of a data structure composed of an m-bit array and k independent hash functions. Specifically, k hash functions can be selected; and each element of the m-bit array is set to 0; then, the n attribute information of the normal message is sequentially mapped to the positions of the bit array using the k hash functions, and the elements of the bit array at the mapped positions become 1, thus obtaining the Bloom filter. n refers to the number of attributes of the message.

[0034] Based on the above Bloom filter, normal messages can be filtered out from historical messages to obtain historical abnormal messages.

[0035] Specifically, for any historical message, multiple attribute information of that historical message can be obtained; the above k hash functions are used to map the multiple attribute information of the historical message to the positions of the bit array. If there is a non-1 in the bit mapped to the bit array, it means that the historical message does not belong to the normal message, that is, it is a historical abnormal message.

[0036] In other embodiments, the abnormal packets are those captured online. Specifically, a pre-set anomaly detection model can be used to perform anomaly detection on packets received by the firewall; if the anomaly detection model outputs a result indicating that the packets received by the firewall are abnormal, then the packets received by the firewall are determined to be abnormal packets.

[0037] Optionally, a pre-configured anomaly detection model can be used to detect the source of packets received by the firewall; if the source of the packet is a pre-marked anomaly source, the packet is determined to be an anomalous packet. The source of the packet can be represented by the source Internet Protocol (IP) address and / or source port number. Correspondingly, the pre-marked anomaly source can be a known anomalous source IP address and / or source port number. For example, a known anomalous source IP address may include: a source IP address captured by a botnet / worm monitoring system, and / or a source IP address captured by a vulnerability scanner. A known anomalous source port number may be a specific port corresponding to a high-risk asset, etc.

[0038] Specifically, for embodiments where the source of a message is represented by its source IP address, the source IP address of the message received by the firewall can be used to query among known abnormal source IP addresses; if the source IP address of the message received by the firewall is found among the known abnormal source IP addresses, it is determined that the message received by the firewall is an abnormal message.

[0039] For embodiments where the source of a message is represented by a source port number, the source port number of the message received by the firewall can be used to query among known abnormal source port numbers; if the source port number of the message received by the firewall is found among the known abnormal source port numbers, it is determined that the message received by the firewall is an abnormal message.

[0040] For embodiments where the source of a packet is indicated by its source IP address and source port number, the source IP address and source port number of the packet received by the firewall can be used to query for known abnormal source IP addresses and known abnormal source port numbers. If the source IP address of the packet received by the firewall is found in the known abnormal source IP addresses, and / or the source port number of the packet received by the firewall is found in the known abnormal source port numbers, the packet received by the firewall is determined to be an abnormal packet.

[0041] The term "offline" is used in contrast to "online." "Online" refers to the generation of defense rules in real-time or near real-time during the firewall's online operation. "Offline" refers to generating defense rules offline, separate from the firewall's online operation, and then publishing these rules to the firewall for detection or blocking of packets acquired online.

[0042] The specific implementation methods for obtaining abnormal messages shown in the above embodiments are merely illustrative and do not constitute a limitation.

[0043] After obtaining historical abnormal messages in step 101, in step 102, feature extraction can be performed on the historical abnormal messages to obtain the feature vector of the historical abnormal messages.

[0044] In this application, the specific implementation method for feature extraction from historical abnormal messages is not limited. In some embodiments, a neural network model can be used to extract features from historical abnormal messages to obtain feature vectors of the historical abnormal messages. The neural network model can be trained using known abnormal messages.

[0045] In practical applications, network traffic queries often involve searching not only the entire network packet but also specific parts of it. This necessitates distinguishing between different parts of the packet and extracting features from each part separately.

[0046] To address the aforementioned issues, this application proposes a message substructure embedding scheme in some embodiments. The method of converting words into vectors is collectively referred to as Embedding. Specifically, since messages are structured and encapsulated according to certain communication protocols, historical abnormal messages can be divided into multiple substructures according to their organizational structure. The message organizational structure may include: a request line, multiple header lines, and a message body. Optionally, historical abnormal messages can be divided into: a request line, header lines, and a message body; and the request line, header lines, and message body can each be treated as a substructure. Furthermore, each header line can also be treated as a substructure, etc.

[0047] Different parts of a message record information about different message attributes. Different message attributes are considered from different perspectives when determining traffic characteristics. For example, some message attributes do not focus on the attribute value, but only on the attribute itself. Others do not focus on the absolute value of the attribute, but only on the relative value. Based on this, multiple substructures corresponding to historical abnormal messages can be normalized to obtain multiple normalized substructures. Normalization, also known as standardization, refers to converting data into a predetermined pattern after processing it according to certain rules.

[0048] The normalization process differs depending on the message attributes of the substructure. For example, for text-based message attributes, the substructure can be converted into a specified identifier. For instance, if the message attribute of a substructure is an IP address, that IP address can be converted into a specified identifier (such as <IP>), thus achieving IP address normalization. For example, assuming the IP address recorded in the substructure is XXXX.XXX.XX.X, it can be converted to <IP>.

[0049] For example, if the message attribute of a substructure is date, it can be converted to another specified identifier, such as <DATE> + character length. Here, character length refers to the character length corresponding to the date. For instance, if the time recorded in the substructure is 2023.04.21, it can be converted to <CHAR> 10.

[0050] For example, if the message attribute of the substructure is Uniform Resource Locator (URL), the path set in the URL can be converted into another set identifier, such as <#CHAR#> + character length range. The character length range is preset. For example, the character length range is {4, 7}, etc. For example, the URLs are: (1) / paper / math / 01?date=2019-03-25&page=1; (2) / paper / math / 02?date=2019-03-25&page=10; (3) / paper / english / 01?date=2019-03-24&page=10; (4) / paper / english / 02?date=2019-03-26&page=100.

[0051] The four URLs mentioned above can be uniformly normalized to: date=\.*&page_size=NUM / paper / <#CHAR#>{4,7} / <#NUM#>{2,2}?date=<#DATE#>{10-10}&page_size=<#NUM#>{1,3}. In this example, the path “math” is normalized to <#CHAR#>{4,7}; “01?” and “02?” are normalized to <#NUM#>{2,2}?. “2019-03-25”, “2019-03-24”, and “2019-03-26” are all normalized to <#DATE#>{10-10}. “page=1”, “page=10”, and “page=100” are all normalized to <#NUM#>{1,3}.

[0052] For example, numerical normalization can be used to normalize the substructure. Numerical normalization methods include, but are not limited to, Z-score normalization, linear normalization, or standard deviation normalization.

[0053] Furthermore, a pre-trained word segmentation model can be used to segment multiple normalized substructures to obtain the words corresponding to each of the multiple message attributes of historical abnormal messages. In this embodiment, the word segmentation model is trained using corpora in the field of network security. The word segmentation model can be a Byte Pair Encoder (BPE) model, a Byte-level Byte-Pair Encoder (Byte-level BPE) model, a Bidirectional Encoder Representations from Transformers (BERT) model, or a WordPiece model, but is not limited to these.

[0054] Among them, the BPE model is a data compression algorithm used to implement variable-length subwords in a fixed-size vocabulary.

[0055] Extract words and their frequencies from the given dataset and determine the vocabulary size; split the words into character sequences, such as the 26 letters of the English alphabet plus various symbols, which serve as the initial vocabulary; add all characters from the character sequences to the vocabulary without repetition, select and merge the adjacent character pairs with the highest frequency, and repeat step 3 until the vocabulary size is met.

[0056] BBPE is a simplified form of data compression algorithm that operates at the byte level, hence the significance of byte-level encoding. In BBPE, words are not converted into character sequences, but rather into byte sequences. The BBPE model converts text into a byte-level sequence, and then applies the BPE algorithm to construct a vocabulary based on these byte-level symbol pairs.

[0057] The WordPiece model selects two words from the vocabulary and merges them into a new word each time. WordPiece chooses adjacent words that maximize the likelihood of the language model and adds them to the vocabulary. Therefore, it can merge character pairs with the highest probability in a language model trained on a given training set.

[0058] In this embodiment, the word segmentation model described above can be trained using a massive corpus of cybersecurity-related language, incorporating expert experience in the field. This cybersecurity corpus can include network packets, among other things. This training method reduces the vocabulary to approximately 50,000 words while retaining some unique terms specific to the cybersecurity domain.

[0059] Furthermore, a neural network model can be used to extract features from the words corresponding to each of the multiple message attributes to obtain word vectors corresponding to each of the multiple message attributes.

[0060] The attention mechanism in neural networks is a resource allocation scheme that, when computational power is limited, allocates computational resources to more important tasks while simultaneously addressing the problem of information overload. In neural network learning, generally speaking, the more parameters a model has, the stronger its expressive power and the greater the amount of information it stores; however, this can lead to information overload. Introducing the attention mechanism allows us to focus on information more critical to the current task from a large amount of input information, reducing attention to other information and even filtering out irrelevant information. This solves the information overload problem and improves the efficiency and accuracy of task processing.

[0061] Therefore, a neural network model with an attention mechanism can be used to extract features from the words corresponding to each of the multiple message attributes, thereby obtaining word vectors for each message attribute. Using a neural network model with an attention mechanism to extract features from the words corresponding to each of the multiple message attributes allows for focusing on message attributes that play a crucial role in determining the nature of the message, while reducing attention to other message attributes, thus improving the efficiency and accuracy of feature extraction.

[0062] Optionally, the attention mechanism can be any neural network model that incorporates an attention mechanism. Examples include the Encoder-Decoder Framework with an attention mechanism, memory networks that extend attention beyond a single input sequence, and neural network models that integrate attention mechanisms, such as Graph Attention Networks (GANs) and Transformer models. Among these, the Transformer model is computationally very friendly, using batch matrix multiplication and element-wise operations. These operations are highly optimized for parallel processing units and can be computed in parallel, thus improving computational efficiency and making it easier to train larger models compared to other models. Therefore, the Transformer model can be chosen as the network model for feature extraction from historical abnormal reports.

[0063] Accordingly, the Transformer model can be used to extract features from the words corresponding to each of the multiple attributes to obtain word vectors corresponding to each of the multiple message attributes.

[0064] Furthermore, the word vectors corresponding to each of the multiple message attributes can be weighted and summed to obtain the feature vectors corresponding to historical abnormal messages. The weights of the multiple message attributes can be empirical values, which can be determined by network security technicians based on historical prior knowledge and pre-set in the device or apparatus that executes the above index construction method.

[0065] The message substructure embedding scheme provided by the above implementation can distinguish different parts of historical abnormal messages (substructures corresponding to different message attributes) and use a neural network model to extract features from each part, that is, to perform embedding. This can reduce the dimension of the embedding vector while supporting substructure similarity search, thereby helping to reduce the time for subsequent index building.

[0066] The implementation methods for feature extraction of historical abnormal messages provided in the above embodiments are merely illustrative and do not constitute a limitation.

[0067] After extracting the feature vectors of historical abnormal messages, query indexes can be built using these feature vectors.

[0068] In this embodiment, feature extraction is performed on historical abnormal packets to obtain feature vectors. Then, a query index corresponding to traffic retrieval can be constructed based on these feature vectors, providing a foundation for querying network attack data. Specifically, constructing a query index using the feature vectors of historical abnormal packets allows subsequent similar traffic queries to use the feature vectors of the packets to be queried to search the packet database, eliminating the need for keyword-based matching and improving the efficiency of subsequent similar traffic searches.

[0069] Because historical abnormal packets are massive in volume, building a query index based on the feature vectors of these packets is time-consuming. This application proposes an abnormal packet partitioning method in some embodiments. This significantly reduces the amount of feature vector data in each partition, thereby greatly reducing the index building time for each partition and consequently reducing the overall index building time for abnormal traffic.

[0070] Specifically, in step 103, the feature vectors of historical abnormal messages can be classified at least once according to at least one set classification dimension, resulting in multiple feature vector partitions. The number of classifications is the same as the number of classification dimensions. In this embodiment, the specific implementation form of the set classification dimensions and the number of classification dimensions are not limited. There can be one or more classification dimensions. "Multiple" refers to two or more. The number of classification dimensions is determined by the time requirements for building the query index. Generally, the more classification dimensions there are, the shorter the time required to build the query index and the higher the efficiency of the query index building.

[0071] For implementation examples with multiple classification dimensions, such as Figure 2 As shown, the feature vectors of historical abnormal messages can be classified multiple times according to multiple classification dimensions to obtain multiple feature vector partitions. For example, assuming there are two classification dimensions, defined as the first classification dimension and the second classification dimension, the feature vectors of historical abnormal messages can be classified according to the first classification dimension to obtain multiple partitions classified according to the first classification dimension (e.g., ...). Figure 2 (Partitions 1 and 2 in the original text); furthermore, for any partition A among the multiple partitions, the feature vector of any partition A can be secondary classified according to the second classification dimension to obtain multiple feature vector partitions (such as partitions 1 and 2 in the original text). Figure 2 Partitions 11, 12, 21, and 22 are among the partitions in the dataset. Figure 2 The illustration uses only two classification dimensions, but this is not a limitation.

[0072] This application does not limit the specific dimensions of the classification. Optionally, the classification dimensions may be time dimension, protocol type, website framework type, etc., but are not limited to these. The following example illustrates the classification dimensions including time dimension and protocol type.

[0073] Accordingly, the feature vectors of historical abnormal messages can be divided into multiple time partitions based on the generation time of the historical abnormal messages. The size of the time window corresponding to each time partition is not limited. Optionally, the time window of a time partition can be on the order of minutes, hours, or days. For example, the time window corresponding to a time partition can be 30 minutes, 1 hour, or 2 hours, but is not limited to these.

[0074] Furthermore, for any time partition X among multiple time partitions, the protocol type of the target historical abnormal message corresponding to the feature vector in time partition X can be identified to determine the protocol type of the target historical abnormal message corresponding to time partition X. In some embodiments, a tree-based model can be used to identify the protocol type of the target historical abnormal message corresponding to the feature vector in time partition X to determine the protocol type of the target historical abnormal message corresponding to time partition X. Here, the tree-based model is a pre-trained classification model for protocol type identification. Optionally, the tree-based model can be a decision tree, etc.

[0075] Furthermore, the feature vectors of target historical abnormal messages with the same protocol type can be divided into the same feature vector partition to obtain at least one feature vector partition corresponding to that time partition X. Each time partition can be partitioned in the above manner to obtain multiple feature vector partitions.

[0076] After obtaining multiple feature vector partitions, in step 104, for any feature vector partition B, a query index for feature vector partition B can be constructed using the feature vectors in feature vector partition B. Each feature vector partition can have its corresponding query index constructed in the above manner, thus obtaining the query index corresponding to each feature vector partition and completing the construction of the query index for abnormal traffic.

[0077] Because the amount of data in the feature vector of the abnormal message in each partition is greatly reduced, the construction time of the query index for each partition is significantly reduced, thereby reducing the overall construction time of the query index for abnormal traffic and keeping the construction time of the query index within an acceptable range.

[0078] On the other hand, by constructing a query index based on the aforementioned partitions, when querying similar traffic, the packets to be queried can be routed to a small number of local feature vector partitions through the classification dimension of the feature vector partitions, without requiring a global query, which helps improve the efficiency of subsequent similar traffic queries. The process of similar traffic querying will be described in the following embodiments and will not be repeated here.

[0079] The implementation method of constructing a query index using the feature vectors of historical abnormal messages is the same as the logical process of constructing a query index for any feature vector partition B using feature vectors in that partition. Therefore, the following example illustrates the process of constructing a query index using feature vectors in any feature vector partition B. In this application embodiment, the specific implementation method of constructing a query index for any feature vector partition B using feature vectors in that partition is not limited.

[0080] In this embodiment of the application, the query index of feature vector partition B can be constructed using the approximate nearest neighbor algorithm. The approximate nearest neighbor (ANN) algorithm can be the Annoy algorithm, the locality-sensitive hashing algorithm, the vector quantization algorithm, the navigable small world (NSW) algorithm, or the hierarchical navigable small world (HNSW) algorithm, but is not limited to these. Figure 2 The illustration uses the HNSW algorithm to build a query index as an example, but it is not intended to be limiting.

[0081] The following example illustrates the specific implementation of constructing a query index for feature vector partition B using the HNSW algorithm. Accordingly, the HNSW algorithm can be used to traverse the feature vectors in feature vector partition B, and a query index for feature vector partition B can be constructed during the traversal. The process of constructing a query index for feature vector partition B using the HNSW algorithm is essentially inserting all feature vectors in feature vector partition B into a graph. HNSW is a graph-based data structure that divides the feature vectors in feature vector partition B into different levels, greedily traversing elements from the upper levels until a local minimum is reached, then switching to the next level, using the local minimum from the previous level as the new element to restart the traversal, until the lowest level has been traversed. In the graph, points represent feature vectors, and the lines connecting points represent the connections between feature vectors.

[0082] In layer 0 (L0), all feature vectors in feature vector partition B are contained. During graph construction, feature vectors are inserted one by one. For each inserted element, a layer I is randomly selected using a probability function. This probability function can be expressed as: floor(-ln(uniform(0,1)) * mL). Here, floor() represents rounding down, uniform(0,1) represents randomly selecting a value from a uniform distribution, and ln() represents taking the logarithm. The value of mL is typically set to 1 / ln(M), where M is the set number of nodes that each node can connect to. This probability function needs to be repeated in every layer except layer 0, adding the feature vector to its insertion layer I and every layer below it.

[0083] The following example illustrates the process of inserting a feature vector into a graph, using the feature vector to be inserted as Q. The input is typically: Q: The feature vector to be inserted; M: The number of connections each point needs to establish with other points in the graph. Points in the graph represent feature vectors.

[0084] ef: Size of the dynamic candidate element set; mL: The standardization factor used when selecting the number of layers for q; M_max: The maximum number of connections. If the number of other points connected to a certain point exceeds the limit, the connection needs to be reduced.

[0085] The process of inserting the feature vector Q into the graph is as follows: S1. The insertion layer I of the feature vector Q can be calculated using the above probability function.

[0086] S2. Find the nearest point among all layers from the highest layer to the insertion layer I, and use it as the entry point for Q to be inserted into layer I. Step S2 can be specifically implemented as: Steps S2.1 and S2.1: S2.1 Starting from the highest level of the graph, use a greedy algorithm to traverse the inserted feature vectors in each level and obtain the ef nearest neighbors that are closest to the feature vector Q. Figure 3 , ef = 1.

[0087] S2.2 After finding ef nearest neighbors, move down to the next level and repeat step S2.1 until the insertion level I is reached.

[0088] S3. The search proceeds from the insertion layer I to the bottom layer (i.e., layer 0, L0). In each layer, the search finds the nearest neighbors (i.e., feature vectors) to the feature vector Q, assigning them to the set W. The ef value is then added to efConstruction, meaning more nearest neighbors are returned. In S2, these nearest neighbors (feature vectors in set W) become candidate objects for connecting feature vector Q and serve as entry points to the next layer. Step S3 can be specifically implemented as follows: Steps S3.1 and S3.2: S3.1. Add M neighbors from the candidate objects contained in the set W as the connection objects of the feature vector Q in this layer. For example, the M neighbors that are closest to the feature vector Q can be selected as the connection objects of the feature vector Q in this layer.

[0089] S3.2. Check the number of connections of each of the M neighbors of feature vector Q. If the number of connections of a neighbor is greater than M_max, reduce the number of connections of that neighbor to M_max. Specifically, select the M_max nearest neighbor feature vectors from the feature vectors connected to that neighbor.

[0090] like Figure 3 As shown, the feature vectors in feature vector partition B can be divided according to... Figure 3 The nodes are organized in a specific format. When inserting a node, the feature vector representing that node is first stored in layer 0 (L0). Then, a random layer number is selected, and the process is repeated layer by layer from that layer down. In each layer, the node is inserted and connected to M nearest neighbor nodes according to certain rules, until layer 0 is reached, resulting in the query index corresponding to feature vector partition B. M is the set number of nodes that each node can connect to. Figure 3 The graph shown, composed of feature vectors, represents the query index corresponding to feature vector partition B. Figure 3 In the diagram, small circles represent eigenvectors, and the continuity between eigenvectors represents the connection relationship between them.

[0091] The HNSW indexing process involves inserting all elements into the graph. Level 0 contains all points in the dataset, and vectors are inserted one by one during graph construction. For each inserted element, a probability function `floor(-ln(uniform(0,1)) x mL)` (where `floor()` rounds down, `uniform(0,1)` randomly selects a value from a uniform distribution, and `ln()` takes the logarithm) is used to randomly select a level. The value of `mL` is typically set to 1 / ln(M). This probability function is repeated for each level (except Level 0). The vector is then added to its insertion level and to every level below it.

[0092] like Figure 3 As shown, in the query index corresponding to feature vector partition B, layer 0 (L0) contains all feature vectors in feature vector partition B. As the layer number increases, the number of feature vectors in each layer gradually decreases, following an exponential decay law. The maximum number of layers in the query index is determined by a random exponential probability decay function. A feature vector exists in all layers below the highest layer. When querying HNSW, the search starts from the highest layer. For the query index constructed by the HNSW algorithm, during similar traffic queries, the node closest to the feature vector of the query packet can be found at the top layer, then the search jumps to the second layer, and so on, finally performing similar traffic retrieval at the bottom layer (L0 layer). This layered approach greatly reduces the number of jumps, reaches the space corresponding to the feature vector of the query packet more quickly, and helps improve traffic query efficiency.

[0093] Based on the query index constructed above, similar traffic can also be retrieved or queried. The traffic query method provided in this application embodiment is illustrated below. Figure 4 As shown, this traffic query method may include: 401. Obtain the traffic sample to be queried.

[0094] 402. Extract features from the traffic sample to be queried to obtain the feature vector of the traffic sample to be queried.

[0095] 403. From the traffic sample to be queried, obtain the value of the traffic sample in the target classification dimension; the target classification dimension is the classification dimension corresponding to multiple feature vector partitions; each feature vector partition corresponds to a query index. The query index of this feature vector partition is constructed using the feature vectors of the historical abnormal packets included in this feature vector partition.

[0096] 404. Based on the value of the traffic sample to be queried in the target classification dimension and the values ​​of the classification dimensions corresponding to multiple feature vector partitions, determine the target feature vector partition from multiple feature vector partitions.

[0097] 405. Using the feature vector of the traffic sample to be queried and the query index of the target feature vector partition, perform a query in the target feature vector partition to obtain the target feature vector.

[0098] 406. Obtain the historical abnormal messages corresponding to the target feature vector as similar traffic to the traffic sample to be queried.

[0099] In this embodiment, in step 401, the traffic sample to be queried refers to sample information reflecting abnormal traffic, which may be abnormal packets, or abnormal traffic disclosed in Common Vulnerabilities & Exposures (CVE); or abnormal traffic disclosed in Proof of Concept (PoC), or abnormal traffic disclosed in zero-day vulnerabilities (0-day) and / or N-day vulnerabilities (N-day).

[0100] CVE stands for Publicly Available Vulnerabilities and Exposures, a publicly disclosed list of cybersecurity vulnerabilities. PoC usually refers to a piece of code that proves a vulnerability; it's the verification code or program used to exploit the vulnerability. PoCs are typically written by cybersecurity professionals to demonstrate the existence of a vulnerability, its severity, and the technical difficulty required to exploit it.

[0101] A zero-day vulnerability is a security flaw that exists but has not yet been publicly disclosed or widely known. Attackers can exploit these vulnerabilities to launch attacks, while defenders are not yet prepared to patch or implement defenses against them. Zero-day vulnerabilities are often a key weapon in hacker attacks and a primary focus for cybersecurity researchers seeking and fixing vulnerabilities.

[0102] N-day refers to the Nth day after a security vulnerability is publicly disclosed, at which point an attacker can exploit it to launch an attack. These vulnerabilities are publicly disclosed through vulnerability reports, hacker forums, and vulnerability announcements from security vendors.

[0103] In step 402, feature extraction can be performed on the traffic sample to be queried to obtain the feature vector of the traffic sample to be queried. For the specific implementation of feature extraction on the traffic sample to be queried, please refer to the relevant content on feature extraction of historical abnormal messages mentioned above, which will not be repeated here.

[0104] Furthermore, the feature vectors of the traffic sample to be queried and the query index constructed above can be used to perform a query in the feature vector library to obtain the target feature vector from the feature vector library. The feature vector library may include the feature vectors of the aforementioned historical abnormal packets. The query index refers to the query index constructed using the feature vectors of the historical abnormal packets; the specific construction process can be found in the relevant content of the above embodiments, and will not be repeated here.

[0105] In this embodiment, the specific implementation method of querying the feature vector library using the feature vector of the traffic sample to be queried and the query index constructed above is not limited.

[0106] In some embodiments, for the query index constructed using the HNSW algorithm described above, the HNSW algorithm can be used to query the feature vector of the traffic sample to be queried in the feature vector library according to the query index; and the target feature vector can be obtained from the feature vector library based on the distance between the feature vector of the traffic sample to be queried and the feature vector in the queried feature vector library.

[0107] Specifically, starting from the entry point of the highest level of the query index, the system searches for the feature vector closest to the feature vector of the traffic sample to be queried at each level of the query index. This feature vector serves as the entry point for the next level, continuing until level 0 of the query index. The distance between feature vectors can be Euclidean distance, cosine distance, Manhattan distance, or Hamming distance, etc. The entry point of the highest level is determined based on the graph connectivity of that level and represents the starting feature vector for the query at that level. The entry points of each level serve as the query entry point for that level and represent the starting feature vector for the query at that level. For example... Figure 3 If the highest layer is the L2 layer, then the entry point of the L2 layer is the feature vector represented by the point where the arrows are turned away.

[0108] In any layer above layer 0, starting from the entry point of that layer, traverse the neighbors of that entry point and find the feature vector 'a' that is closest to the traffic sample to be queried from the neighbors and the entry point itself. If feature vector 'a' is not the entry point itself, then the next search starts from feature vector 'a' and follows the same pattern. If feature vector 'a' and the entry point are the same feature vector, then the search for that layer ends.

[0109] After searching at levels 0 and above, the entry point at level 0 is relatively close to the traffic sample to be queried. Therefore, at level 0, starting from the entry point, we need to find the N closest first feature vectors to the traffic sample to be queried, where N is less than or equal to a set threshold W. From these N first feature vectors, we select the k closest feature vectors to the traffic sample to be queried; these are the target feature vectors. Here, k can be set arbitrarily. N W.

[0110] The HNSW algorithm described above can skip a large number of feature vectors that are far from the feature vector of the traffic sample to be queried at a higher level, and quickly locate the feature vector that is closer to the feature vector of the traffic sample to be queried, thus narrowing the search range of similar traffic and helping to improve the efficiency of similar traffic search.

[0111] In the above embodiment of constructing a query index for each feature vector partition based on feature vector partitioning, in step 403, the value of the traffic sample to be queried on the target classification dimension can also be obtained from the traffic sample to be queried; the target classification dimension is the classification dimension corresponding to multiple feature vector partitions.

[0112] Further, in step 404, the target feature vector partition can be determined from multiple feature vector partitions based on the value of the traffic sample to be queried in the target classification dimension and the values ​​of the classification dimensions corresponding to the multiple feature vector partitions. Specifically, the value of the traffic sample to be queried in the target classification dimension can be matched with the values ​​of the classification dimensions corresponding to the multiple feature vector partitions; and from the multiple feature vector partitions, the feature vector partition where the value of the traffic sample to be queried in the target classification dimension is the same as the value of the classification dimension corresponding to the multiple feature vector partitions is determined as the target feature vector partition.

[0113] For example, in an embodiment where the classification dimensions of the multiple feature vector partitions are time and protocol type, the protocol type of the traffic sample to be queried can be obtained; and the feature vector partitions whose protocol types are the same as those of the traffic sample to be queried can be used as the target feature vector partitions.

[0114] In this way, by matching the classification dimensions, the feature vectors of the traffic samples to be queried can be routed to local feature vector partitions (i.e., target feature vector partitions). Then, vector queries can be performed only in local feature vector partitions, without having to perform a global search in the feature vector library, which further narrows the query scope and helps to improve query efficiency.

[0115] Based on the above target feature vector partitioning, in step 405, the feature vector of the traffic sample to be queried and the query index of the target feature vector partition can be used to perform a query in the target feature vector partition to determine the target feature vector.

[0116] Optionally, for the query index constructed using the HNSW algorithm, the HNSW algorithm can be used to query the feature vector of the traffic sample to be queried in the target feature vector partition according to the index path corresponding to the query index of the target feature vector partition; and the target feature vector can be obtained from the target feature vector partition based on the distance between the feature vector of the traffic sample to be queried and the feature vector in the queried feature vector library. For specific implementation methods of using the HNSW algorithm for target feature vector query, please refer to the relevant content of the above embodiments, which will not be repeated here.

[0117] In other embodiments, for the example described above where historical abnormal messages are divided into substructures and the word vectors corresponding to the message attributes of multiple substructures are weighted and summed to obtain the feature vectors of historical abnormal messages, the feature vector library may include: feature vectors of historical abnormal messages. The feature vectors of historical abnormal messages are obtained by weighted summing of the word vectors corresponding to multiple message attributes of the historical abnormal messages. This feature vector extraction method supports substructure similarity queries.

[0118] In this embodiment, when extracting features from the traffic sample to be queried, the values ​​of the traffic sample under the aforementioned multiple message attributes can be obtained from the traffic sample to be queried; and the values ​​of the traffic sample under the aforementioned multiple message attributes are normalized to obtain the normalized values ​​of the traffic sample under the aforementioned multiple message attributes. Further, a pre-trained word segmentation model can be used to segment the values ​​of the traffic sample under the aforementioned multiple message attributes to obtain the words corresponding to each of the multiple message attributes; further, a neural network model can be used to extract features from the words corresponding to each of the multiple message attributes to obtain the word vectors corresponding to each of the multiple message attributes.

[0119] Furthermore, the feature vector of the traffic sample to be queried can be obtained by weighted summing of the word vectors corresponding to each of the multiple message attributes. The weights of the traffic sample in each message attribute can be flexibly set according to the actual query requirements. Message attributes that are of primary interest to the query can be given higher weights, while message attributes that are not of interest can be given lower weights, or even set to 0.

[0120] Furthermore, the feature vectors of the traffic sample to be queried and the query engine constructed above can be used to search the feature vector library to obtain the target feature vector. Since the weights of the traffic sample to be queried in multiple packet attributes can be flexibly set according to the actual query requirements, similar vector queries of substructures can be realized, solving the problem that traditional solutions cannot perform substructure queries on a specific part of network traffic.

[0121] After determining the target feature vector that is similar to the feature vector of the traffic sample to be queried, in step 406, the historical abnormal messages corresponding to the target feature vector can be obtained as similar traffic to the traffic sample to be queried, thus realizing similar traffic query.

[0122] In this embodiment, based on the pre-built query index, similar traffic is queried in the feature vector library. Due to the existence of the query index, the query can be performed according to the index path corresponding to the query index, without having to perform a global query in the feature vector library, which narrows the query scope and helps to improve the efficiency of traffic query.

[0123] After identifying similar traffic to the traffic sample to be queried, defense rules can be constructed using the traffic sample and its similar traffic. These defense rules are the same as those in the firewall, which uses them for network protection.

[0124] Specifically, defense parameters, such as IP address, URL address, and keywords, can be obtained from the traffic sample to be queried and similar traffic. Furthermore, the logical expression of the defense parameters can be edited according to the syntax format supported by the firewall to obtain the defense rules.

[0125] After generating the defense rules, you can also publish the defense rules to the firewall so that the firewall can use the defense rules for network protection.

[0126] In some embodiments, when a new vulnerability emerges in an application, a Proof-of-Concept (PoC) is released. Defense rules can be built based on the PoC for emergency response. However, rules built from a single PoC have limited defensive coverage and cannot achieve comprehensive protection. In this embodiment, the PoC can be used as a traffic sample to be queried. The traffic query method provided in the above embodiments is used to query similar traffic to the PoC. Then, defense rules can be built based on the PoC and its similar traffic, expanding the basis for generating defense rules, improving the defensive dimensions of the rules, making the defense rules more comprehensive, and helping to reduce the probability of subsequent network attacks being missed.

[0127] In other application scenarios, CVEs are disclosed, but PoCs are not. In such cases, the CVE can be used as a sample of traffic to be queried. The traffic query method provided in the above embodiments can be used to quickly retrieve similar traffic to the CVE in historical traffic. Based on similar traffic to the CVE, a complete payload can be obtained, which can provide support and basis for writing PoCs and help to improve the efficiency of obtaining PoCs after new CVEs are disclosed.

[0128] In some application scenarios, emergency response is required for 0-day or N-day traffic. In these scenarios, the internet does not disclose complete Proof-of-Concept (PoC), only some characteristics. Analyzing source code and building an environment to obtain a complete PoC would be time-consuming and labor-intensive. In this embodiment, 0-day or N-day traffic samples can be used as the traffic sample to be queried, and the traffic query method provided in the above embodiment can be used to quickly obtain similar traffic from historical traffic. Then, based on the 0-day or N-day traffic and its similar traffic, defense rules can be generated and published to the firewall for network protection. Because the traffic query method provided in the above embodiment quickly obtains similar traffic from historical traffic, it helps improve the efficiency of generating and publishing defense rules, enabling emergency response for 0-day or N-day traffic.

[0129] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 401 and 402 can be device A; or the execution subject of step 401 can be device A, and the execution subject of step 402 can be device B; and so on.

[0130] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 401, 402, etc., are merely used to distinguish different operations and do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel.

[0131] Accordingly, embodiments of this application also provide a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, cause one or more processors to perform the steps in the above-described index construction method and / or traffic query method.

[0132] Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 5 As shown, the computing device includes a memory 50a and a processor 50b. The memory 50a is used to store computer programs.

[0133] The processor 50b is coupled to the memory 50a for executing a computer program to: retrieve historical abnormal messages; the historical abnormal messages include at least attack messages; extract features from the historical abnormal messages to obtain feature vectors of the historical abnormal messages; and construct a query index using the feature vectors of the historical abnormal messages.

[0134] Optionally, when the processor 50b constructs a query index using the feature vectors of historical abnormal messages, it specifically performs the following: classifies the feature vectors of historical abnormal messages at least once according to at least one set classification dimension to obtain multiple feature vector partitions; and for any feature vector partition among the multiple feature vector partitions, constructs a query index for that feature vector partition using the feature vectors in that feature vector partition. The number of classification dimensions is the same as the number of classifications.

[0135] Optionally, the classification dimensions include: multiple classification dimensions; when the processor 50b classifies the feature vector of the historical abnormal message at least once according to at least one set classification dimension, it is specifically used to: classify the feature vector of the historical abnormal message multiple times according to multiple classification dimensions to obtain multiple feature vector partitions.

[0136] In some embodiments, there are multiple classification dimensions. These multiple classification dimensions include a time dimension and a protocol type. Accordingly, when the processor 50b performs at least one classification of the feature vectors of historical abnormal messages, it specifically performs the following: Based on the generation time of the historical abnormal messages, it divides the feature vectors of the historical abnormal messages into multiple time partitions; for any time partition among the multiple time partitions, it identifies the protocol type of the target historical abnormal message corresponding to the feature vector in that time partition to determine the protocol type of the target historical abnormal message; and it divides the feature vectors of target historical abnormal messages with the same protocol type into the same feature vector partition to obtain at least one feature vector partition corresponding to any time partition.

[0137] Optionally, when the processor 50b constructs a query index for any feature vector partition using the feature vectors in any feature vector partition, it specifically uses the Hierarchical Navigation Small World (HNSW) algorithm to traverse the feature vectors in any feature vector partition and constructs a query index for any feature vector partition during the traversal process.

[0138] In some embodiments, when the processor 50b extracts features from historical abnormal messages, it specifically performs the following steps: for any historical abnormal message, it splits the message into multiple substructures according to the message's organizational structure; it normalizes the multiple substructures based on their message attributes to obtain multiple normalized substructures; it segments the multiple normalized substructures using a pre-trained word segmentation model to obtain words corresponding to each of the multiple message attributes of any historical abnormal message; it extracts features from the words corresponding to each of the multiple message attributes using a neural network model with an attention mechanism to obtain word vectors corresponding to each of the multiple message attributes; and it performs a weighted summation of the word vectors corresponding to each of the multiple message attributes to obtain a feature vector for any historical abnormal message.

[0139] Optionally, the neural network model with the attention mechanism is a Transformer model. Accordingly, when the processor 50b uses the attention mechanism model to extract features from the words corresponding to each of the multiple message attributes, it specifically performs the following: it uses the Transformer model to extract features from the words corresponding to each of the multiple message attributes to obtain word vectors corresponding to each of the multiple message attributes.

[0140] In other embodiments, when the processor 50b obtains historical abnormal messages, it is specifically used to: obtain historical messages from the database; and filter out normal messages from the historical messages to obtain historical abnormal messages.

[0141] Furthermore, when the processor 50b filters out normal packets from historical packets to obtain historical abnormal packets, it specifically performs the following actions: using a packet whitelist to filter out normal packets from historical packets to obtain historical abnormal packets; and / or, obtaining the access addresses of historical packets from historical packets; determining the access frequency of the same access address within a time window based on the access addresses of historical packets; filtering out normal packets with access frequencies greater than or equal to a set access frequency threshold from historical packets to obtain historical abnormal packets; and / or, using a Bloom filter to filter out normal packets from historical packets to obtain historical abnormal packets; the Bloom filter is constructed based on a pre-determined set of normal packets.

[0142] In some embodiments of this application, the processor 50b is further configured to: acquire a traffic sample to be queried; extract features from the traffic sample to be queried to obtain a feature vector of the traffic sample to be queried; use the feature vector of the traffic sample to be queried and the query index to perform a query in the feature vector library to obtain a target feature vector from the feature vector library; the feature vector library includes: feature vectors of historical abnormal packets; the query index is constructed using the feature vectors of historical abnormal packets; and acquire the historical abnormal packets corresponding to the target feature vector as similar traffic to the traffic sample to be queried.

[0143] Optionally, when the processor 50b performs a retrieval in the feature vector library using the feature vector of the traffic sample to be queried and the query index, it specifically performs the following: using the HNSW algorithm to query the feature vector of the traffic sample to be queried in the feature vector library according to the index path corresponding to the query index; and obtaining the target feature vector from the feature vector library based on the distance between the feature vector of the traffic sample to be queried and the feature vector in the queried feature vector library.

[0144] Optionally, the feature vector library includes: multiple feature vector partitions; each feature vector partition corresponds to a query index. The processor 50b is further configured to: obtain the value of the traffic sample to be queried on the target classification dimension from the traffic sample to be queried; the target classification dimension is the classification dimension corresponding to the multiple feature vector partitions; and determine the target feature vector partition from the multiple feature vector partitions based on the value of the traffic sample to be queried on the target classification dimension and the values ​​of the classification dimensions corresponding to the multiple feature vector partitions. Accordingly, when the processor 50b performs a query in the feature vector library using the feature vector of the traffic sample to be queried and the query index, it specifically performs a query in the target feature vector partition using the feature vector of the traffic sample to be queried and the query index of the target feature vector partition to obtain the target feature vector.

[0145] Optionally, the processor 50b is also used to: construct defense rules using a sample of traffic to be queried and similar traffic to the sample of traffic to be queried; and publish the defense rules to the firewall to use the defense rules for network protection.

[0146] In some alternative implementations, such as Figure 5 As shown, the computing device may further include components such as a communication component 50c and a power supply component 50d. In some embodiments, the computing device may be implemented as a terminal device such as a computer, and may further include components such as a display component 50e and an audio component 50f. Figure 5 The diagram only shows some components and does not mean that the computing device must contain them. Figure 5 The inclusion of all components does not imply that a computing device can only include... Figure 5 The components shown.

[0147] The computing device provided in this embodiment can construct a query index for traffic retrieval based on the feature vectors of historical abnormal packets when building the query index, providing a foundation for querying network attack data. Specifically, by constructing the query index using the feature vectors of historical abnormal packets, subsequent similar traffic queries can utilize the feature vectors of the packets to be queried to search the packet database, eliminating the need for keyword-based matching and thus improving the efficiency of subsequent similar traffic searches.

[0148] When querying traffic, similar traffic can be queried in the feature vector library based on the pre-built query index for the traffic sample to be queried. Because of the existence of the query index, the query can be performed according to the index path corresponding to the query index, without having to perform a global query in the feature vector library, which narrows the query scope and helps to improve the efficiency of traffic query.

[0149] In this embodiment, the memory is used to store computer programs and can be configured to store various other data to support operation on its host device. The processor can execute the computer programs stored in the memory to implement corresponding control logic. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Electrically Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0150] In the embodiments of this application, the processor can be any hardware processing device capable of executing the above-described method logic. Optionally, the processor can be a central processing unit (CPU), a graphics processing unit (GPU), or a microcontroller unit (MCU); it can also be a programmable device such as a field-programmable gate array (FPGA), a programmable array logic (PAL), a general array logic (GAL), or a complex programmable logic device (CPLD); or it can be an advanced RISC machine (ARM) or a system on chip (SoC), etc., but is not limited thereto.

[0151] In this embodiment, the communication component is configured to facilitate wired or wireless communication between its host device and other devices. The device housing the communication component can access wireless networks based on communication standards, such as Wireless Fidelity (WiFi), 2G or 3G, 4G, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In another exemplary embodiment, the communication component may also be implemented based on Near Field Communication (NFC), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), or other technologies.

[0152] In embodiments of this application, the display component may include a liquid crystal display (LCD) and a touch panel (TP). If the display component includes a touch panel, the display component can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0153] In this embodiment, a power supply component is configured to provide power to various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component resides.

[0154] In embodiments of this application, the audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), which is configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals. For example, in devices with voice interaction capabilities, voice interaction with the user can be achieved through the audio component.

[0155] It should be noted that the terms "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0156] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program code.

[0157] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0158] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0159] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0160] In a typical configuration, a computing device includes one or more processors (CPU, etc.), input / output interfaces, network interfaces, and memory.

[0161] Memory may include non-persistent storage in computer-readable media, such as random-access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0162] Computer storage media are readable storage media, also known as removable media. Removable and non-removable media can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0163] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the aforementioned element.

[0164] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An index construction method, characterized in that, include: Retrieve historical abnormal messages; Feature extraction is performed on the historical abnormal messages to obtain the feature vector of the historical abnormal messages; Based on at least one defined classification dimension, the feature vectors of the historical abnormal messages are classified at least once to obtain multiple feature vector partitions; the number of classifications is the same as the number of classification dimensions; for any feature vector partition among the multiple feature vector partitions, a query index for that feature vector partition is constructed using the feature vectors in that feature vector partition. The step of constructing a query index for any feature vector partition using feature vectors from any feature vector partition includes: The approximate nearest neighbor algorithm is used to construct a query index for any feature vector partition by utilizing the feature vectors in any feature vector partition.

2. The method according to claim 1, characterized in that, The classification dimensions are multiple; these multiple classification dimensions include: time dimension and protocol type; the feature vectors of the historical abnormal messages are classified multiple times according to the multiple classification dimensions to obtain multiple feature vector partitions, including: Based on the generation time of the historical abnormal messages, the feature vectors of the historical abnormal messages are divided into multiple time partitions; For any of the multiple time partitions, the protocol type of the target historical abnormal message corresponding to the feature vector in the target historical abnormal message is identified to determine the protocol type of the target historical abnormal message; the feature vectors of target historical abnormal messages with the same protocol type are divided into the same feature vector partition to obtain at least one feature vector partition corresponding to the target historical abnormal message.

3. The method according to claim 1, characterized in that, The approximate nearest neighbor algorithm includes: the hierarchical navigation small-world HNSW algorithm; the step of using the approximate nearest neighbor algorithm to construct a query index for any feature vector partition using feature vectors in any feature vector partition includes: Using the hierarchical navigation small-world HNSW algorithm, the feature vectors in any feature vector partition are traversed, and a query index for any feature vector partition is constructed during the traversal process.

4. The method according to claim 1, characterized in that, The step of extracting features from the historical abnormal messages to obtain the feature vector of the historical abnormal messages includes: For any historical abnormal message, according to the message's organizational structure, the historical abnormal message is split into multiple substructures; Based on the message attributes corresponding to the multiple substructures, the multiple substructures are normalized to obtain multiple normalized substructures. The pre-trained word segmentation model is used to segment the multiple normalized substructures to obtain the words corresponding to the multiple message attributes of any historical abnormal message. A neural network model with an attention mechanism is used to extract features from the words corresponding to each of the multiple message attributes to obtain word vectors corresponding to each of the multiple message attributes. The word vectors corresponding to each of the multiple message attributes are weighted and summed to obtain the feature vector of any historical abnormal message.

5. The method according to claim 1, characterized in that, The acquisition of historical abnormal messages includes: Retrieve historical messages from the database; Normal messages are filtered out from the historical messages to obtain the abnormal historical messages.

6. The method according to claim 5, characterized in that, The step of filtering out normal messages from the historical messages to obtain the historical abnormal messages includes: By using a message whitelist, normal messages are filtered out from the historical messages to obtain the historical abnormal messages; And / or, Obtain the access address of the historical message from the historical message; determine the access frequency of the same access address within a time window based on the access address of the historical message; filter out normal messages from the historical message whose access frequency is greater than or equal to a set access frequency threshold to obtain the historical abnormal message. And / or, A Bloom filter is used to filter out normal messages from the historical messages to obtain the historical abnormal messages; the Bloom filter is constructed based on a predetermined set of normal messages.

7. The method according to any one of claims 1-6, characterized in that, Also includes: Obtain the traffic sample to be queried; Feature extraction is performed on the traffic sample to be queried to obtain the feature vector of the traffic sample to be queried; From the traffic sample to be queried, obtain the value of the traffic sample to be queried on the target classification dimension; the target classification dimension is the classification dimension corresponding to multiple feature vector partitions; Each feature vector partition includes: feature vectors of historical abnormal messages; and the query index corresponding to each feature vector partition is constructed using the feature vectors of that feature vector partition. Based on the value of the traffic sample to be queried in the target classification dimension and the value of the classification dimension corresponding to the multiple feature vector partitions, the target feature vector partition is determined from the multiple feature vector partitions; Using the feature vector of the traffic sample to be queried and the query index of the target feature vector partition, a query is performed in the target feature vector partition to obtain the target feature vector; Obtain the historical abnormal packets corresponding to the target feature vector as similar traffic to the traffic sample to be queried.

8. The method according to claim 7, characterized in that, The step of using the feature vector of the traffic sample to be queried and the query index of the target feature vector partition to perform a query in the target feature vector partition to obtain the target feature vector includes: Using the HNSW algorithm, the feature vector of the traffic sample to be queried is queried in the target feature vector partition according to the query index of the target feature vector partition; Based on the distance between the feature vector of the traffic sample to be queried and the feature vector in the target feature vector partition, the target feature vector is obtained from the feature vector library.

9. The method according to claim 8, characterized in that, Also includes: Defense rules are constructed using the traffic sample to be queried and similar traffic to the traffic sample to be queried; The defense rules are published to the firewall to provide network protection.

10. A traffic query method, characterized in that, include: Obtain the traffic sample to be queried; Feature extraction is performed on the traffic sample to be queried to obtain the feature vector of the traffic sample to be queried; From the traffic sample to be queried, obtain the value of the traffic sample to be queried on the target classification dimension; the target classification dimension is the classification dimension corresponding to multiple feature vector partitions; Each feature vector partition corresponds to a query index; The query index for any feature vector partition is constructed using the index construction method described in any one of claims 1-6; Based on the value of the traffic sample to be queried in the target classification dimension and the value of the classification dimension corresponding to the multiple feature vector partitions, the target feature vector partition is determined from the multiple feature vector partitions; Using the feature vector of the traffic sample to be queried and the query index of the target feature vector partition, a query is performed in the target feature vector partition to obtain the target feature vector; Obtain the historical abnormal packets corresponding to the target feature vector as similar traffic to the traffic sample to be queried.

11. A network device, characterized in that, include: A memory and a processor; wherein the memory is used to store computer programs; The processor is coupled to the memory for executing the computer program to perform the steps of the method according to any one of claims 1-10.

12. A computer-readable storage medium storing computer instructions, characterized in that, When the computer instructions are executed by one or more processors, the one or more processors are caused to perform the steps of the method according to any one of claims 1-10.

Images