IPv6 network space active address and port detection method and device

Abstract

The invention relates to an IPv6 network space active address and port detection method and device, and belongs to the technical field of network detection. The method comprises the following steps: (1) retrieving network resources; (2) active address detection is divided into a non-seed address scene, a few-seed address scene and a multi-seed address scene, a non-seed address scene generation model, a few-seed address scene generation model and a multi-seed address scene generation model are provided, and candidate addresses are generated respectively; (3) respectively carrying out alias prefix detection and alias removal on active addresses found in a non-seed address scene, a few-seed address scene and a multi-seed address scene, and adding newly found addresses after alias removal into an active IPv6 address library; and (4) predicting active ports of the active IPv6 addresses by using a decision model based on a naive Bayes classifier and utilizing all input features, and completing dynamic feedback scanning through a distributed active detection facility to realize accumulation of the active addresses and ports from nothing to existence and from few to many.

Description

Technical Field

[0001] The invention relates to an IPv6 network space active address and port detection method and device, belonging to the technical field of network detection. Background Art

[0002] The global Internet is evolving towards the IPv6 era. IPv6 network space mapping is an important means of securely managing the global IPv6 network. Currently, mapping work generally faces difficulties such as huge IPv6 address space, sparse distribution of network resources, low detection efficiency, information loss, and low coverage.

[0003] The global IPv4 network active address and port detection technology is relatively mature. The traditional method is to use tools such as ZMap, Masscan, Nmap, etc. to scan the specified port list to obtain feasible full network scanning efficiency. However, due to the huge address space of IPv6 network (2 128 ) forms an extremely sparse distribution of active hosts, so this solution cannot be directly used for active address detection in the entire IPv6 network.

[0004] It is even more difficult to further obtain the active port information on the host. Currently, only a list of once active IPv6 addresses (IPv6Hitlist) can be maintained through various open source channels, and only the commonly used ports on the active addresses in the list are detected. This method has inherent defects of information lag and low coverage. In addition, relevant research results show that most active services do not appear in the commonly used port list, so only scanning the specified or commonly used ports is not comprehensive and will lead to erroneous conclusions. For this reason, the present invention is proposed. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a method and device for detecting active addresses and ports in IPv6 network space, which realizes the accumulation of active addresses and ports from nothing to something and from few to many, solves the defects of the prior methods in terms of low hit rate, small coverage, incomplete detection of active ports and slow speed for IPv6 active address detection, and realizes efficient and high-coverage detection of IPv6 active addresses and ports.

[0006] Terminology explanation:

[0007] ICMP live host discovery technology: the source host sends an ICMP message, timestamp, address mask and other response requests to the target host. If the target host responds, the target host is considered to be a live host.

[0008] TCP live host discovery technology: refers to the use of TCP SYN Ping scanning to determine whether the target host is alive. TCP provides sending, receiving, and full-duplex services (data flows in both directions at the same time). Both communicating parties have two buffers to send and receive data. The protocol adds a progressive confirmation sequence number to the message to notify the sender to receive the next byte. If the waiting time expires, the data packet will be sent again to ensure that it is a reliable protocol.

[0009] Sparse active host distribution mechanism: refers to the data pull mechanism based on the PIM-SM (Protocol Independent Multicast-Sparse Mode) protocol through the RPT (Rendezvous Point Tree). This mechanism assumes that the hosts in the network do not need to receive multicast data by default unless explicitly requested. When a host needs to receive multicast data, it sends a join request to the RP (Rendezvous Point) to build an RPT with the RP as the root.

[0010] The technical solution of the present invention is as follows:

[0011] A method for detecting active addresses and ports in IPv6 network space, the steps are as follows:

[0012] (1) Search network resources to obtain the IPv6 active address list, seed addresses containing active port information, and detailed information on BGP prefixes;

[0013] (2) Active address detection is divided into three scenarios: no seed address scenario, few seed address scenario, and multiple seed address scenario. For the no seed address scenario, few seed address scenario, and multiple seed address scenario, a no seed address scenario generation model, a few seed address scenario generation model, and multiple seed address scenario generation model are proposed to generate candidate addresses respectively;

[0014] (3) Perform alias prefix detection and alias removal on active addresses found in scenarios with no seed addresses, scenarios with few seed addresses, and scenarios with multiple seed addresses, and add the newly found addresses after alias removal to the constructed active IPv6 address database;

[0015] (4) Based on the active IPv6 address database constructed in step (3), a decision model based on a naive Bayes classifier is used to efficiently predict the active ports of active IPv6 addresses using all input features. Dynamic feedback scanning is then completed through distributed active detection facilities to achieve the accumulation of active addresses and ports from scratch and from small to large.

[0016] Preferably, in step (1) of the present invention, the specific steps are:

[0017] First, enter the data interface and query related information based on conditions. There are two branches of query: the first is the query of IPv6 active address list and seed address containing port active information; the second branch is the query of whois information and BGP prefix information;

[0018] The first branch first determines whether it is IPv6. If it is IPv6, it searches for the IPv6 address. Then it determines whether the IP address is active. If it is active, it searches for the IPv6 active address, outputs the IPv6 active address list, and determines whether the port is active.

[0019] If it is an active port, perform an active port search;

[0020] Then determine whether the organization and ASN information are included. If the organization and ASN information are included, perform a seed address search including active port information, and output a seed address list including active port information.

[0021] The second branch first determines whether it is Whois information. If it is Whois information, it performs Whois information retrieval, outputs a Whois information list, and determines whether it contains other information such as organization, country, etc.

[0022] If it contains other information such as organization, country, etc., a BGP prefix search is performed to output detailed information of the BGP prefix.

[0023] Preferably, in step (2) of the present invention, the specific steps are as follows:

[0024] (21) using the IPv6 active address list and detailed information of the BGP prefix as input for active address detection;

[0025] (22) collecting continuous active address data for the collected IPv6 active addresses, and performing cross-detection using ICMP live host discovery technology and TCP live host discovery technology to obtain continuous active IPv6 address data;

[0026] (23) using Pyasn to query the BGP prefix corresponding to the continuously active IPv6 address, forming a dictionary D {BGP prefix: IPv6 active address list}, and then determining whether the prefix in the Whois information of the BGP prefix input in step (21) is in the dictionary D;

[0027] If in dictionary D, determine the length of the list corresponding to each BPG prefix in dictionary D:

[0028] If the list length is greater than or equal to 10, a multi-sub-address area data set is generated in the form of a dictionary D1{BGP prefix: IPv6 active address list};

[0029] If the list length is less than 10, a small number of seed address area data sets are generated in the form of dictionary D2{BGP prefix: IPv6 active address list};

[0030] The multi-seed address region dataset and the few-seed address region dataset together constitute a seed address region dataset;

[0031] If it is not in dictionary D, a seedless address area data set is generated in the form of dictionary D + dictionary D3 {seedless BGP prefix: Whois information} + dictionary D4 {seed BGP prefix: Whois information} + detailed information of BGP prefix (detailed information of BGP prefix input in step (21));

[0032] The datasets of multiple seed address regions, few seed address regions and no seed address regions together constitute the second category of datasets;

[0033] (24) respectively executing training of a no-seed address scenario generation model, a few-seed address scenario generation model, and a plurality of seed address scenario generation model;

[0034] The multiple sub-region address generation model first cleans the data of the continuously active IPv6 addresses, and then generates a tree structure based on the continuously active IPv6 address list, wherein the leaf node of the tree structure is the highest density area, and the nodes other than the leaf nodes in the tree structure have multiple child nodes, and the child nodes are split from the parent node according to the minimum entropy value of the characters. Each child node corresponds to a subset of the continuously active IPv6 address list, and the number of subsets corresponding to the child nodes is less than or equal to the threshold ε (since the value of the characters in IPv6 is 0~f, the threshold ε=16 is considered to be the most appropriate) is the leaf node, and the candidate address is dynamically generated in the process of traversing the root node of the tree structure to the leaf node in turn based on the conditional probability generation direction with the hit coefficient;

[0035] The address generation model for few-seed regions is based on the RFC classifier and KMeans clustering algorithm to classify the continuously active IPv6 addresses in a more fine-grained manner, obtain the label set, form the IPv6 address + label data set, and then use the existing technology 6Vision model to generate candidate addresses;

[0036] The RFC classifier uses the addr6 tool in the IPv6toolkit to perform pattern matching on the seed set according to the possible IPv6 address patterns proposed in the RFC 7707 document to obtain RFC-based classification results.

[0037] The seedless regional address generation model first processes the whois information of the BGP prefix, and uses the phrases of the whois information of the BGP prefix to perform semantic category annotation, named entity category recognition (using the named entity categories defined by OntoNotes and the DeBERTa model for named entity category recognition), training of difficult negative samples of phrases, construction of word-level data enhancement datasets (using the WordNet database to extract synonyms and using the FastText model to calculate similarity) and phrase-level data enhancement datasets (obtained using the Parro text conversion tool), and then converts the Whois information with seed BGP prefixes and the Whois information without seed BGP prefixes into a complete set of data. The model parameters of the pre-trained language model PEARL are fine-tuned and loaded through the SentenceTransformer package in the sentence_transformers library to obtain the encoded word embedding feature vector. The similarity between the two BGP address configuration modes is measured by calculating the similarity between the word embedding feature vector corresponding to the Whois information of the BGP prefix in the seed address area dataset and the word embedding feature vector corresponding to the Whois information of the BGP prefix in the non-seed address area dataset. Due to the similarity between the address configuration modes of the seed BGP prefix and the non-seed BGP prefix, the seed address is migrated to generate fine-grained candidate addresses.

[0038] According to the preferred embodiment of the present invention, in step (3), specifically:

[0039] The candidate addresses generated by the no-seed address scenario generation model, the few-seed address scenario generation model and the multiple-seed address scenario generation model are respectively applied to their respective models through dynamic feedback detection, thereby updating the parameters of their respective models. At the same time, the generated active IPv6 addresses are subjected to alias prefix detection to obtain new IPv6 active address data, which are updated to the IPv6 active address list, and then step (2) is repeated until the preset number of rounds is completed.

[0040] According to the present invention, further preferably, the alias prefix detection step is as follows:

[0041] (31) re-detecting the active IPv6 addresses to be detected to ensure that alias detection can be effectively performed;

[0042] (32) Based on the sparse active host distribution mechanism, a target address is randomly generated in the IPv6 network for detection. Assuming that the response possibility is very small, a BGP prefix is ​​randomly selected from the detailed information of the BGP prefix and 16 candidate addresses are generated;

[0043] (33) Using ICMP live host discovery technology, send ICMP probe packets to the target address. If all responses can be obtained from the target address, then the prefix can be determined as a candidate prefix.

[0044] (34) For the candidate prefix, an address is randomly generated under the candidate prefix, and then an ICMP probe packet is sent to the address. If a response is still received at this time, the BGP prefix is ​​considered to be an alias prefix.

[0045] Preferably, according to the present invention, in step (4), the active port prediction step is as follows:

[0046] (41) Dataset construction:

[0047] Based on the seed address of the port active information in step (1), extract the global features of active port prediction, namely, BGP prefix, organization and ASN information, and then use the DHC algorithm to generate an address space tree from the seed address, extract the address pattern corresponding to the leaf node of the tree, and use the address pattern as the local feature of active port prediction to construct an active port prediction data set of IPv6 address + port prediction global feature + port prediction local feature;

[0048] (42)Port prediction:

[0049] Based on the local features of active port prediction and the global features of active port prediction, a naive Bayes classifier is used to predict whether the port is active. Each feature corresponds to a naive Bayes classifier. It is assumed that the features are conditionally independent under given category labels to perform port prediction.

[0050] (43) Port detection:

[0051] According to step (42), a decision model of a trained naive Bayes classifier is obtained, and then all ports predicted by the model are used as the predicted ports of the active IPv6 address to be predicted (i.e., the IPv6 address list in the data set constructed in step (41)), and the active port detection is performed using the distributed active detection module, and the active port corresponding to the detected IPv6 address and its corresponding BGP prefix, organization and other information are updated in the dictionary D5.

[0052] According to the preferred embodiment of the present invention, in step (42), the specific steps are as follows:

[0053] ① Divide the data set constructed in step (41) into a model initialization data set and a model training data set, traverse the IPv6 addresses in the model training data set, and obtain the port prediction global features and port prediction local features corresponding to the target IPv6 address;

[0054] ② Based on the global features of the IPv6 addresses and their corresponding ports in the model initialization data set and the local features of the ports prediction, the detector (zmap) is used to perform a full port scan on the IPv6 addresses in the model initialization data set to obtain the real active ports corresponding to the IPv6 addresses, and the naive Bayes model is initialized based on the distribution of the real active ports;

[0055] ③ Input the port prediction global features and port prediction local features of the IPv6 address in the model training data set into the initialized naive Bayes classifier to obtain the active values ​​of each port corresponding to the target IPv6 address, sort all active values ​​in descending order, and take at least one port corresponding to the active value as the active port of the target IPv6 address, where the active value is the sum of the outputs of the naive Bayes classifiers corresponding to all features;

[0056] ④ For the IPv6 addresses in the model training data set, input them into the detector (zmap) for detection based on the determined active ports, and update the naive Bayes classifier after obtaining the detection results;

[0057] ⑤ Use multi-threaded calls to the global classifier and local classifier to find the active ports corresponding to the IPv6 addresses in the model training data set. After multiple rounds, stop the prediction until the port coverage reaches 90%.

[0058] Preferably, according to the present invention, in step (4), the distributed active detection facility includes a resource detection control node and m probe nodes, the resource detection control node is used to execute the resource detection control process, and the probe node is used to execute the distributed probe detection process;

[0059] The resource detection control process is as follows:

[0060] Ⅰ. The resource detection control node obtains the detection task of the client, parses the detection task, and extracts the total address set and port set that need to be detected;

[0061] II. Generate detection subtasks. A detection subtask contains a sub-address set and a port. The sub-address set is obtained by evenly splitting the total address set. The port comes from a port in the port set. When splitting the address set, randomization is performed, that is, the addresses in each sub-address set are made as random as possible.

[0062] III. Allocate each detection subtask to the available probe nodes reasonably through a certain algorithm. The algorithm used is: average allocation, uniform allocation according to node bandwidth, or allocation method based on optimization theory. Use the above algorithm strategy to determine how to configure subtasks and nodes, and generate a configuration list;

[0063] IV. The resource detection control node sends the assigned detection tasks to each probe node in turn according to the assigned results, and then waits for the feedback of the sent task results. If the feedback result is received, the file integrity is verified, and the task results are processed, such as whether there are errors or the results are not deduplicated, and all subtasks are checked to see if they are completed. If all are completed, the results of all subtasks are merged and stored to obtain the final result. When the client requests to access the task result, the task result is streamed to the client;

[0064] The distributed probe detection process is as follows:

[0065] First, receive tasks sent by the resource detection control node to the probe node, including verifying file integrity, obtaining network environment information, and parsing task parameters;

[0066] Second, the detection task is executed, the task to be executed is split into several times, and the detection tool is called to execute the detection subtask;

[0067] Third, the detection results are returned. After the subtask detection is completed, the task results are merged. After execution, the detection results of all subtasks are merged and returned to the resource detection control node, waiting to receive the next task.

[0068] An IPv6 network space active address and port detection device, comprising:

[0069] Network resource retrieval module: used to obtain the IPv6 active address list, seed addresses containing port active information, and detailed information of BGP prefixes;

[0070] Active address detection module: used to divide active address detection into three scenarios: no seed address scenario, few seed address scenario and multiple sub-address scenario. For the no seed address scenario, few seed address scenario and multiple sub-address scenario, a no seed address scenario generation model, a few seed address scenario generation model and multiple sub-address scenario generation model are proposed respectively to generate candidate addresses;

[0071] Active IPv6 address library construction module: used to perform alias prefix detection and alias removal on active addresses found in scenarios with no seed addresses, few seed addresses, and multiple seed addresses, and add the newly found addresses after alias removal to the constructed active IPv6 address library;

[0072] Scanning and detection module: Based on the active IPv6 address library constructed in step (3), a decision model based on the naive Bayes classifier is used to efficiently predict the active ports of active IPv6 addresses using all input features. Dynamic feedback scanning is then completed through distributed active detection facilities to achieve the accumulation of active addresses and ports from scratch and from small to large.

[0073] The beneficial effects of the present invention are:

[0074] 1. The present invention obtains an effective and reliable IPv6 active address set through an IPv6 active address detection method based on machine learning, and then uses an open port prediction method based on a naive Bayes decision model. During the entire IPv6 address space scanning process, the active ports of IPv6 active addresses with high coverage in the network space can be accurately predicted, thereby realizing the accumulation of active addresses and ports from scratch and from few to many. The defects of the existing methods in terms of low hit rate, small coverage, incomplete detection of active ports and slow speed for IPv6 active address detection are solved, thereby realizing efficient and high-coverage detection of IPv6 active addresses and ports.

[0075] 2. In the seedless address generation model, the present invention utilizes the expression capability of the pre-trained language model for phrase-level coding vectors to fully mine the association between seedless BGP prefixes and seeded BGP prefixes, and can maximize the possibility of mining active addresses from the seed address migration pattern, thereby avoiding invalid migration and improving the coverage and detection efficiency of active addresses.

[0076] 3. In order to improve the efficiency of active address detection and avoid the low hit rate of detection in the multi-seed region address generation model, the present invention proposes to increase the hit coefficient to adjust the direction of conditional probability detection, which can obtain more effective active addresses than the detection based on conditional probability alone.

[0077] 4. The present invention dynamically adds the active addresses generated by the no-seed area address generation model to the seed addresses of the multiple sub-area address generation models and the seed addresses of the few-seed area address generation model, and adds the active addresses generated by the few-seed area address generation model to the seed addresses of the multiple sub-area address generation models, which can fully tap the coverage capability of active addresses in the entire network space. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a schematic diagram of the process of the present invention;

[0079] Figure 2 is an active address detection flow chart of the present invention;

[0080] Figure 3 This is a schematic diagram of the device structure of Example 2 of the present invention. DETAILED DESCRIPTION

[0081] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but is not limited thereto.

[0082] Embodiment 1:

[0083] This embodiment provides a method for detecting active addresses and ports in an IPv6 network space, and the steps are as follows:

[0084] (1) Search network resources to obtain the IPv6 active address list, seed addresses containing active port information, and detailed information on BGP prefixes. The specific steps are as follows:

[0085] First, enter the data interface and query related information based on conditions. There are two branches of query: the first is the query of IPv6 active address list and seed address containing port active information; the second branch is the query of whois information and BGP prefix information;

[0086] The first branch first determines whether it is IPv6. If it is IPv6, it searches for the IPv6 address. Then it determines whether the IP address is active. If it is active, it searches for the IPv6 active address, outputs the IPv6 active address list, and determines whether the port is active.

[0087] If it is an active port, perform an active port search;

[0088] Then determine whether the organization and ASN information are included. If the organization and ASN information are included, perform a seed address search including active port information, and output a seed address list including active port information.

[0089] The second branch first determines whether it is Whois information. If it is Whois information, it performs Whois information retrieval, outputs a Whois information list, and determines whether it contains other information such as organization, country, etc.

[0090] If it contains other information such as organization, country, etc., a BGP prefix search is performed to output detailed information of the BGP prefix.

[0091] The data interface of the above process includes public resources and network mapping platform nodes. The network mapping platform node is a data platform deployed on the server, which mainly imports data collected from public resources into the platform. In addition, the platform completes module scheduling and data storage of the device of the present invention.

[0092] (2) Active address detection is divided into three scenarios: no seed address scenario, few seed address scenario, and multiple seed address scenario. A no seed address scenario generation model, a few seed address scenario generation model, and multiple seed address scenario generation model are proposed for the no seed address scenario, few seed address scenario, and multiple seed address scenario generation model, respectively, to generate candidate addresses. The specific steps are as follows:

[0093] (21) using the IPv6 active address list and detailed information of the BGP prefix as input for active address detection;

[0094] (22) Continuously collect active address data for the collected IPv6 active addresses, and use ICMP live host discovery technology and TCP live host discovery technology for cross-detection to obtain continuous active IPv6 address data;

[0095] Use the output IPv6 active address list obtained from network resource retrieval as the input for active address detection. Initialize the continuous active address queue Q, detection period period = 0, detection count count = 0, constant number of detections per period M, constant detection period N, number of times the address appears T in M detections. First, use ICMP live host discovery technology to detect the IPv6 hitlist list, and determine whether there is a response to the request. If there is a response, obtain the IPv6 address list with responses; otherwise, obtain the IPv6 address list without responses. Then, use TCP live host discovery technology to detect the IPv6 address list without responses. Finally, obtain the active address list and inactive address list detected in this period. The detection count count is incremented by one. If count < M times, continue to complete the detection in this period; otherwise, enter the next period of detection, and the detection period period is incremented by one. If period < N times, continue to complete the detection of the remaining periods; otherwise, obtain the list of active address lists for M detections in N periods, and perform data statistics on this list. If an IPv6 address satisfies T / (N*M) > 80%, then this address enters the queue Q until all address statistical analyses are completed to obtain the final long-term active address queue Q.

[0096] (23) Use Pyasn to query the BGP prefix corresponding to the continuously active IPv6 address to form a dictionary D {BGP prefix: IPv6 active address list}, and then determine whether the prefix in the Whois information of the BGP prefix input in step (21) is in the dictionary D;

[0097] If it is in the dictionary D, determine the length of the list corresponding to each BPG prefix in the dictionary D:

[0098] If the list length is greater than or equal to 10, generate multiple sub-address region data sets in the form of a dictionary D1 {BGP prefix: IPv6 active address list};

[0099] If the list length is less than 10, generate a small number of sub-address region data sets in the form of a dictionary D2 {BGP prefix: IPv6 active address list};

[0100] The multiple sub-address region data sets and the small number of sub-address region data sets together constitute the seeded address region data set;

[0101] If it is not in dictionary D, a seedless address area data set is generated in the form of dictionary D + dictionary D3 {seedless BGP prefix: Whois information} + dictionary D4 {seed BGP prefix: Whois information} + detailed information of BGP prefix (detailed information of BGP prefix input in step (21));

[0102] The datasets of multiple seed address regions, few seed address regions and no seed address regions together constitute the second category of datasets;

[0103] (24) respectively executing training of a no-seed address scenario generation model, a few-seed address scenario generation model, and a plurality of seed address scenario generation model;

[0104] The multiple sub-region address generation model first cleans the data of the continuously active IPv6 addresses, and then generates a tree structure based on the continuously active IPv6 address list, wherein the leaf node of the tree structure is the highest density area, and the nodes other than the leaf nodes in the tree structure have multiple child nodes, and the child nodes are split from the parent node according to the minimum entropy value of the characters. Each child node corresponds to a subset of the continuously active IPv6 address list, and the number of subsets corresponding to the child nodes is less than or equal to the threshold ε (since the value of the characters in IPv6 is 0~f, the threshold ε=16 is considered to be the most appropriate) is the leaf node, and the candidate address is dynamically generated in the process of traversing the root node of the tree structure to the leaf node in turn based on the conditional probability generation direction with the hit coefficient;

[0105] The conditional probability with hit coefficient means that the definition of conditional probability is based on a space tree (such as Figure 3 As shown in the figure, each child node includes a first probability R of finding an active address in the address block corresponding to the child node and a second probability Q of the parent node selecting the child node, wherein the first probability is determined by the relative size of the second probability in all brother nodes, and the probability P of a child node being more active than other brother nodes is proportional to the probability of the parent node selecting the child node. Since the density distribution of seed addresses is not necessarily consistent with the density distribution of active addresses in the actual network, it is easy to have a high proportion of active addresses in the address space corresponding to the node, but a low hit rate of detecting new active addresses in the generation of candidate addresses. Therefore, the hit coefficient is added to the conditional probability, thereby more accurately adjusting the direction of generating candidate addresses, and the hit coefficient δ = (the number of new active addresses of child nodes detected each time A / the cumulative sum of the number of new active addresses of child nodes detected each time by other brother nodes B). In summary, R = δ*Q / P;

[0106] The address generation model for few-seed regions is based on the RFC classifier and KMeans clustering algorithm to classify the continuously active IPv6 addresses in a more fine-grained manner, obtain the label set, form the IPv6 address + label data set, and then use the existing technology 6Vision model to generate candidate addresses;

[0107] The RFC classifier uses the addr6 tool in the IPv6toolkit to perform pattern matching on the seed set according to the possible IPv6 address patterns proposed in the RFC 7707 document to obtain RFC-based classification results.

[0108] The seedless regional address generation model first processes the whois information of the BGP prefix, and uses the phrases of the whois information of the BGP prefix to perform semantic category annotation, named entity category recognition (using the named entity categories defined by OntoNotes and the DeBERTa model for named entity category recognition), training of difficult negative samples of phrases, construction of word-level data enhancement datasets (using the WordNet database to extract synonyms and using the FastText model to calculate similarity) and phrase-level data enhancement datasets (obtained using the Parro text conversion tool), and then converts the Whois information with seed BGP prefixes and the Whois information without seed BGP prefixes into a complete set of data. The model parameters of the pre-trained language model PEARL are fine-tuned and loaded through the SentenceTransformer package in the sentence_transformers library to obtain the encoded word embedding feature vector. The similarity between the two BGP address configuration modes is measured by calculating the similarity between the word embedding feature vector corresponding to the Whois information of the BGP prefix in the seed address area dataset and the word embedding feature vector corresponding to the Whois information of the BGP prefix in the non-seed address area dataset. Due to the similarity between the address configuration modes of the seed BGP prefix and the non-seed BGP prefix, the seed address is migrated to generate fine-grained candidate addresses.

[0109] (3) Alias ​​prefix detection and alias removal are performed on active addresses found in scenarios with no seed addresses, scenarios with few seed addresses, and scenarios with multiple seed addresses, and the newly found addresses after alias removal are added to the constructed active IPv6 address database. Specifically:

[0110] The candidate addresses generated by the no-seed address scenario generation model, the few-seed address scenario generation model and the multiple-seed address scenario generation model are respectively applied to their respective models through dynamic feedback detection, thereby updating the parameters of their respective models. At the same time, the generated active IPv6 addresses are subjected to alias prefix detection to obtain new IPv6 active address data, which are updated to the IPv6 active address list, and then step (2) is repeated until the preset number of rounds is completed.

[0111] The steps for alias prefix detection are as follows:

[0112] (31) re-detecting the active IPv6 addresses to be detected to ensure that alias detection can be effectively performed;

[0113] (32) Based on the sparse active host distribution mechanism, a target address is randomly generated in the IPv6 network for detection. Assuming that the response possibility is very small, a BGP prefix is ​​randomly selected from the detailed information of the BGP prefix and 16 candidate addresses are generated;

[0114] (33) Using ICMP live host discovery technology, send ICMP probe packets to the target address. If all responses can be obtained from the target address, then the prefix can be determined as a candidate prefix.

[0115] (34) For the candidate prefix, an address is randomly generated under the candidate prefix, and then an ICMP probe packet is sent to the address. If a response is still received at this time, the BGP prefix is ​​considered to be an alias prefix.

[0116] (4) Based on the active IPv6 address database constructed in step (3), a decision model based on a naive Bayes classifier is used to efficiently predict the active ports of active IPv6 addresses using all input features. Dynamic feedback scanning is then completed through distributed active detection facilities to achieve the accumulation of active addresses and ports from scratch and from small to large.

[0117] The steps for active port prediction are as follows:

[0118] (41) Dataset construction:

[0119] Based on the seed address of the port active information in step (1), extract the global features of active port prediction, namely, BGP prefix, organization and ASN information, and then use the DHC algorithm to generate an address space tree from the seed address, extract the address pattern corresponding to the leaf node of the tree, and use the address pattern as the local feature of active port prediction to construct an active port prediction data set of IPv6 address + port prediction global feature + port prediction local feature;

[0120] (42)Port prediction:

[0121] Based on the local features of active port prediction and the global features of active port prediction, a naive Bayes classifier is used to predict whether the port is active. Each feature corresponds to a naive Bayes classifier. It is assumed that the features are conditionally independent under given category labels to perform port prediction.

[0122] (43) Port detection:

[0123] According to step (42), a decision model of a trained naive Bayes classifier is obtained, and then all ports predicted by the model are used as the predicted ports of the active IPv6 address to be predicted (i.e., the IPv6 address list in the data set constructed in step (41)), and the active port detection is performed using the distributed active detection module, and the active port corresponding to the detected IPv6 address and its corresponding BGP prefix, organization and other information are updated in the dictionary D5.

[0124] In step (42), the specific steps are as follows:

[0125] ① Divide the data set constructed in step (41) into a model initialization data set and a model training data set, traverse the IPv6 addresses in the model training data set, and obtain the port prediction global features and port prediction local features corresponding to the target IPv6 address;

[0126] The model initialization data set is used to initialize the parameters of the naive Bayes classifier, and the model training data set is used to train the model after model initialization. Each IPv6 address in the model training data set does not contain or contains incomplete corresponding active ports, and the target IPv6 address contains at least one unscanned port.

[0127] ② Based on the global features of the IPv6 addresses and their corresponding ports in the model initialization data set and the local features of the ports prediction, the detector (zmap) is used to perform a full port scan on the IPv6 addresses in the model initialization data set to obtain the real active ports corresponding to the IPv6 addresses, and the naive Bayes model is initialized based on the distribution of the real active ports;

[0128] ③ Input the port prediction global features and port prediction local features of the IPv6 address in the model training data set into the initialized naive Bayes classifier to obtain the active values ​​of each port corresponding to the target IPv6 address, sort all active values ​​in descending order, and take at least one port corresponding to the active value as the active port of the target IPv6 address, where the active value is the sum of the outputs of the naive Bayes classifiers corresponding to all features;

[0129] ④ For the IPv6 addresses in the model training data set, input them into the detector (zmap) for detection based on the determined active ports, and update the naive Bayes classifier after obtaining the detection results;

[0130] ⑤ Use multi-threaded calls to the global classifier and local classifier to find the active ports corresponding to the IPv6 addresses in the model training data set. After multiple rounds, stop the prediction until the port coverage reaches 90%.

[0131] In step (4), the distributed active detection facility belongs to the infrastructure module of active address and port detection. The logical module of the distributed active detection module mainly includes resource detection control and distributed probe detection. Its physical architecture is a "one master and multiple slaves" type, that is, a control node and several probe nodes. The facility takes the new IPv6 active address list obtained by active address detection and the IPv6 active address to be detected and the corresponding predicted port obtained by active port detection as input.

[0132] The distributed active detection facility includes a resource detection control node and m probe nodes. The resource detection control node is used to execute the resource detection control process, and the probe node is used to execute the distributed probe detection process.

[0133] The resource detection control process is as follows: upload the resources to be detected to the resource detection control node, and then call the task scheduling process. The resource detection control node verifies the specified probe information, creates tasks, verifies the integrity of data files, and executes the task creation sub-process according to a certain resource allocation strategy, and returns the request task result link to the client. This sub-process mainly distributes the detection sub-tasks generated by the task configuration module to each probe node reasonably, and issues tasks. After the task is issued, the sub-process ends and returns to the main process to wait for feedback on the results of the issued tasks. If the feedback result is received, verify the file integrity, process the task results, such as whether there are errors or the results are not deduplicated, etc., and check whether all sub-tasks are completed. If all are completed, merge the results of all sub-tasks and store them to obtain the final result. When the client requests to access the task result, the task result is streamed to the client. Specifically:

[0134] Ⅰ. The resource detection control node obtains the detection task of the client, parses the detection task, and extracts the total address set and port set that need to be detected;

[0135] II. Generate detection subtasks. A detection subtask contains a sub-address set and a port. The sub-address set is obtained by evenly splitting the total address set. The port comes from a port in the port set. When splitting the address set, randomization is performed, that is, the addresses in each sub-address set are made as random as possible.

[0136] III. Allocate each detection subtask to the available probe nodes reasonably through a certain algorithm. The algorithm used is: average allocation, uniform allocation according to node bandwidth, or allocation method based on optimization theory. Use the above algorithm strategy to determine how to configure subtasks and nodes, and generate a configuration list;

[0137] IV. The resource detection control node sends the assigned detection tasks to each probe node in turn according to the assigned results, and then waits for the feedback of the sent task results. If the feedback result is received, the file integrity is verified, and the task results are processed, such as whether there are errors or the results are not deduplicated, and all subtasks are checked to see if they are completed. If all are completed, the results of all subtasks are merged and stored to obtain the final result. When the client requests to access the task result, the task result is streamed to the client;

[0138] The distributed probe detection process is as follows: first, the probe receives the task issued by the resource control node, creates a task process, verifies the file integrity, and then obtains the network environment information according to the resource configuration file, and creates a detection sub-process. The sub-process first splits the task to be executed, and then calls the detection sub-tasks based on detection tools such as ping and ZMap. Finally, the detection results of all sub-tasks are merged and returned to the resource detection control node. The sub-process ends and returns to the main process, waiting to receive the next task.

[0139] in:

[0140] Ping detection tool refers to the use of ping to determine whether the local host can successfully exchange (send and receive) data packets with another host. Then, based on the returned information, it can be inferred whether the TCP / IP parameters are set correctly, whether the operation is normal, whether the network is unobstructed, etc. The specific operation is to first verify the connection with one or more remote computers by sending ICMP (Internet Control Message Protocol) echo packets to the computer and listening for echo reply packets, then wait for up to one second for each sent packet, and finally print the number of packets transmitted and received;

[0141] ZMap detection tool refers to a scanning software developed by a research team at the University of Michigan led by Durumeric. This tool can scan the entire public Internet in one hour, displaying information about nearly 4 billion online devices. The scan results can show which websites cannot defend against specific vulnerabilities;

[0142] Distributed probe detection is a three-step process that includes receiving detection tasks, executing detection tasks, and returning detection results. The specific steps are as follows:

[0143] First, receive tasks sent by the resource detection control node to the probe node, including verifying file integrity, obtaining network environment information, and parsing task parameters;

[0144] Second, the detection task is executed, the task to be executed is split into several times, and the detection tool is called to execute the detection subtask;

[0145] Third, the detection results are returned. After the subtask detection is completed, the task results are merged. After execution, the detection results of all subtasks are merged and returned to the resource detection control node, waiting to receive the next task.

[0146] Embodiment 2:

[0147] This embodiment provides an IPv6 network space active address and port detection device, including:

[0148] Network resource retrieval module: used to obtain the IPv6 active address list, seed addresses containing port active information, and detailed information of BGP prefixes;

[0149] Active address detection module: used to divide active address detection into three scenarios: no seed address scenario, few seed address scenario and multiple sub-address scenario. For the no seed address scenario, few seed address scenario and multiple sub-address scenario, a no seed address scenario generation model, a few seed address scenario generation model and multiple sub-address scenario generation model are proposed respectively to generate candidate addresses;

[0150] Active IPv6 address library construction module: used to perform alias prefix detection and alias removal on active addresses found in scenarios with no seed addresses, few seed addresses, and multiple seed addresses, and add the newly found addresses after alias removal to the constructed active IPv6 address library;

[0151] Scanning and detection module: Based on the active IPv6 address library constructed in step (3), a decision model based on the naive Bayes classifier is used to efficiently predict the active ports of active IPv6 addresses using all input features. Dynamic feedback scanning is then completed through distributed active detection facilities to achieve the accumulation of active addresses and ports from scratch and from small to large.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting active addresses and ports in IPv6 network space, characterized in that: Here are the steps: (1) Search network resources to obtain the IPv6 active address list, seed addresses containing active port information, and detailed information on BGP prefixes; (2) Active address detection is divided into three scenarios: no seed address scenario, few seed address scenario, and multiple seed address scenario. For the no seed address scenario, few seed address scenario, and multiple seed address scenario, a no seed address scenario generation model, a few seed address scenario generation model, and multiple seed address scenario generation model are proposed to generate candidate addresses respectively; (3) Perform alias prefix detection and alias removal on active addresses found in scenarios with no seed addresses, scenarios with few seed addresses, and scenarios with multiple seed addresses, and add the newly found addresses after alias removal to the active IPv6 address database; (4) Based on the active IPv6 address database, a decision model based on the naive Bayes classifier is used to utilize all input features to predict the active ports of active IPv6 addresses. Then, dynamic feedback scanning is completed through distributed active detection facilities to achieve the accumulation of active addresses and ports from scratch and from small to large.

2. The method for detecting active addresses and ports in IPv6 network space according to claim 1, characterized in that: In step (1), the specific steps are: First, enter the data interface and query related information based on conditions. There are two branches of query: the first is the query of IPv6 active address list and seed address containing port active information; the second branch is the query of whois information and BGP prefix information; The first branch first determines whether it is IPv6. If it is IPv6, it searches for the IPv6 address. Then determine whether the IP address is active. If the IP address is active, perform an IPv6 active address search, output an IPv6 active address list, and determine whether the port is active. If it is an active port, perform an active port search; Then determine whether the organization and ASN information are included. If the organization and ASN information are included, perform a seed address search including active port information, and output a seed address list including active port information. The second branch first determines whether it is Whois information. If it is Whois information, it performs Whois information retrieval, outputs a Whois information list, and determines whether it contains organizational information; If it contains organizational information, a BGP prefix search is performed to output detailed information of the BGP prefix.

3. The method for detecting active addresses and ports in IPv6 network space according to claim 2, characterized in that: In step (2), the specific steps are as follows: (21) using the IPv6 active address list and detailed information of the BGP prefix as input for active address detection; (22) collecting continuous active address data for the collected IPv6 active addresses, and performing cross-detection using ICMP live host discovery technology and TCP live host discovery technology to obtain continuous active IPv6 address data; (23) using Pyasn to query the BGP prefix corresponding to the continuously active IPv6 address, forming a dictionary D {BGP prefix: IPv6 active address list}, and then determining whether the prefix in the Whois information of the BGP prefix input in step (21) is in the dictionary D; If in dictionary D, determine the length of the list corresponding to each BPG prefix in dictionary D: If the list length is greater than or equal to 10, a multi-sub-address area data set is generated in the form of a dictionary D1{BGP prefix: IPv6 active address list}; If the list length is less than 10, a small number of seed address area data sets are generated in the form of dictionary D2{BGP prefix: IPv6 active address list}; The multi-seed address region dataset and the few-seed address region dataset together constitute a seed address region dataset; If it is not in dictionary D, a seedless address area data set is generated in the form of dictionary D + dictionary D3 {seedless BGP prefix: Whois information} + dictionary D4 {seed BGP prefix: Whois information} + detailed information of BGP prefix; The datasets of multiple seed address regions, few seed address regions and no seed address regions together constitute the second category of datasets; (24) respectively executing training of a no-seed address scenario generation model, a few-seed address scenario generation model, and a plurality of seed address scenario generation model; The multiple sub-region address generation model first cleans the data of the continuously active IPv6 addresses, and then generates a tree structure based on the continuously active IPv6 address list, where the leaf nodes of the tree structure are the highest density areas. The nodes in the tree structure other than the leaf nodes have multiple child nodes, and the child nodes are split from the parent nodes according to the minimum entropy value of the characters. Each child node corresponds to a subset of the continuously active IPv6 address list. The child node is a leaf node if the number of subsets corresponding to the child node is less than or equal to the threshold ε. Based on the conditional probability generation direction with a hit coefficient, the candidate address is dynamically generated in the process of traversing the root node of the tree structure to the leaf node in sequence; The few-seed region address generation model is based on the RFC classifier and KMeans clustering algorithm to classify the continuously active IPv6 addresses in a more fine-grained manner, obtain the label set, form the IPv6 address + label data set, and then use the 6Vision model to generate candidate addresses; The seedless regional address generation model first processes the whois information of the BGP prefix, and uses the phrases of the whois information of the BGP prefix to perform semantic category labeling, named entity category recognition, phrase difficult negative sample training, word-level data augmentation dataset construction and phrase-level data augmentation dataset construction. The Whois information with seed BGP prefix and the Whois information without seed BGP prefix are respectively fine-tuned using the model parameters of the pre-trained language model PEARL to obtain the encoded word embedding feature vector. The similarity between the word embedding feature vector corresponding to the Whois information of the BGP prefix in the seed address regional dataset and the word embedding feature vector corresponding to the Whois information of the BGP prefix in the seed address regional dataset is calculated to measure the similarity between the two BGP address configuration modes. Since the address configuration modes of the seeded BGP prefix and the seedless BGP prefix are similar, the seed address is migrated to generate fine-grained candidate addresses.

4. The method for detecting active addresses and ports in IPv6 network space according to claim 3, characterized in that: In step (3), specifically: The candidate addresses generated by the no-seed address scenario generation model, the few-seed address scenario generation model and the multiple-seed address scenario generation model are respectively applied to their respective models through dynamic feedback detection, thereby updating the parameters of their respective models. At the same time, the generated active IPv6 addresses are subjected to alias prefix detection to obtain new IPv6 active address data, which are updated to the IPv6 active address list, and then step (2) is repeated until the preset number of rounds is completed.

5. The method for detecting active addresses and ports in IPv6 network space according to claim 4, characterized in that: The steps for alias prefix detection are as follows: (31) re-detecting the active IPv6 address to be detected; (32) Based on the sparse active host distribution mechanism, a target address is randomly generated in the IPv6 network for detection, a BGP prefix is ​​randomly selected from the detailed information of the BGP prefix and 16 candidate addresses are generated; (33) Using ICMP live host discovery technology, send ICMP probe packets to the target address. If all responses can be obtained from the target address, then the prefix can be determined as a candidate prefix. (34) For the candidate prefix, an address is randomly generated under the candidate prefix, and then an ICMP probe packet is sent to the address. If a response is still received at this time, the BGP prefix is ​​considered to be an alias prefix.

6. The method for detecting active addresses and ports in IPv6 network space according to claim 5, characterized in that: In step (4), the active port prediction steps are as follows: (41) Dataset construction: Based on the seed address of the port active information, the global features of active port prediction are extracted, namely the BGP prefix, organization and ASN information. Then, the DHC algorithm is used to generate an address space tree from the seed address, and the address pattern corresponding to the leaf node of the tree is extracted. The address pattern is used as the local feature of active port prediction to construct an active port prediction data set of IPv6 address + port prediction global feature + port prediction local feature. (42)Port prediction: Based on the local features of active port prediction and the global features of active port prediction, a naive Bayes classifier is used to predict whether the port is active. Each feature corresponds to a naive Bayes classifier. It is assumed that the features are conditionally independent under given category labels to perform port prediction. (43) Port detection: According to step (42), a decision model of a trained naive Bayes classifier is obtained, and then all ports predicted by the model are used as the predicted ports of the active IPv6 address to be predicted, and the active port detection is performed using the distributed active detection module, and the active port corresponding to the detected IPv6 address and its corresponding BGP prefix and organization are updated to the dictionary D5.

7. The method for detecting active addresses and ports in IPv6 network space according to claim 6, characterized in that: In step (42), the specific steps are as follows: ① Divide the data set constructed in step (41) into a model initialization data set and a model training data set, traverse the IPv6 addresses in the model training data set, and obtain the port prediction global features and port prediction local features corresponding to the target IPv6 address; ② Based on the global features of the IPv6 addresses and their corresponding ports in the model initialization data set and the local features of the ports prediction, the detector is used to perform a full port scan on the IPv6 addresses in the model initialization data set to obtain the real active ports corresponding to the IPv6 addresses, and the naive Bayes model is initialized based on the distribution of the real active ports; ③ Input the port prediction global features and port prediction local features of the IPv6 address in the model training data set into the initialized naive Bayes classifier to obtain the active values ​​of each port corresponding to the target IPv6 address, sort all active values ​​in descending order, and take at least one port corresponding to the active value as the active port of the target IPv6 address, where the active value is the sum of the outputs of the naive Bayes classifiers corresponding to all features; ④ For the IPv6 addresses in the model training data set, input them into the detector based on the determined active ports for detection, and update the naive Bayes classifier after obtaining the detection results; ⑤ Use multi-threaded calls to the global classifier and local classifier to find the active ports corresponding to the IPv6 addresses in the model training data set. After multiple rounds, stop the prediction until the port coverage reaches 90%.

8. The method for detecting active addresses and ports in IPv6 network space according to claim 7, characterized in that: In step (4), the distributed active detection facility includes a resource detection control node and m probe nodes. The resource detection control node is used to execute the resource detection control process, and the probe node is used to execute the distributed probe detection process.

9. The method for detecting active addresses and ports in IPv6 network space according to claim 8, characterized in that: The resource detection control process is as follows: Ⅰ. The resource detection control node obtains the detection task of the client, parses the detection task, and extracts the total address set and port set that need to be detected; II. Generate detection subtasks. A detection subtask contains a sub-address set and a port. The sub-address set is obtained by evenly splitting the total address set. The port comes from a port in the port set. When splitting the address set, random processing is performed; III. Allocate each detection subtask to the probe node through a certain algorithm, using the following algorithms: average allocation, uniform allocation according to node bandwidth, or allocation method based on optimization theory; IV. The resource detection control node sends the assigned detection tasks to each probe node in turn according to the assigned results, and then waits for the result feedback of the sent tasks. If the feedback result is received, the file integrity is verified, the task result is processed, and all subtasks are checked to see if they are all completed. If all are completed, the results of all subtasks are merged and stored to obtain the final result. When the client requests to access the task result, the task result is streamed to the client. The distributed probe detection process is as follows: First, receive tasks sent by the resource detection control node to the probe node, including verifying file integrity, obtaining network environment information, and parsing task parameters; Second, the detection task is executed, the task to be executed is split into several times, and the detection tool is called to execute the detection subtask; Third, the detection results are returned. After the subtask detection is completed, the task results are merged. After execution, the detection results of all subtasks are merged and returned to the resource detection control node, waiting to receive the next task.

10. An IPv6 network space active address and port detection device, characterized in that: include: Network resource retrieval module: used to obtain the IPv6 active address list, seed addresses containing port active information, and detailed information of BGP prefixes; Active address detection module: used to divide active address detection into three scenarios: no seed address scenario, few seed address scenario and multiple sub-address scenario. For the no seed address scenario, few seed address scenario and multiple sub-address scenario, a no seed address scenario generation model, a few seed address scenario generation model and multiple sub-address scenario generation model are proposed respectively to generate candidate addresses; Active IPv6 address library construction module: used to perform alias prefix detection and alias removal on active addresses found in scenarios with no seed addresses, few seed addresses, and multiple seed addresses, and add the newly found addresses after alias removal to the constructed active IPv6 address library; Scanning and detection module: Based on the active IPv6 address library constructed in step (3), a decision model based on the naive Bayes classifier is used to efficiently predict the active ports of active IPv6 addresses using all input features. Dynamic feedback scanning is then completed through distributed active detection facilities to achieve the accumulation of active addresses and ports from scratch and from small to large.

Images