IPv6 network space active address and port detection method and device

Abstract

The application relates to an IPv6 network space active address and port detection method and device, and belongs to the technical field of network detection. The method steps are as follows: (1) network resource retrieval; (2) active address detection is divided into a no-seed address scene, a few-seed address scene and a multi-seed address scene, a no-seed address scene generation model, a few-seed address scene generation model and a multi-seed address scene generation model are proposed, and candidate addresses are respectively generated; (3) the active addresses discovered in the no-seed address scene, the few-seed address scene and the multi-seed address scene are respectively subjected to alias prefix detection and alias removal, and the newly discovered addresses after the alias removal are added to an active IPv6 address library; (4) a decision model based on a naive Bayes classifier is used, all input features are utilized, active ports of the active IPv6 addresses are predicted, dynamic feedback scanning is completed through a distributed active detection facility, and accumulation of the active addresses and the ports from nothing to something and from few to many is realized.

Description

TECHNICAL FIELD

[0001] The application relates to an IPv6 network space active address and port detection method and device and belongs to the technical field of network detection. BACKGROUND

[0002] The global Internet is evolving towards the IPv6 era, and IPv6 network space mapping is an important means for the security management of global IPv6 networks. At present, the mapping work generally faces the problems of huge IPv6 address space, sparse network resource distribution, low detection efficiency, information loss and low coverage.

[0003] Global IPv4 network active address and port detection technology is relatively mature, and the traditional method is to use ZMap, Masscan, Nmap and other tools to scan the specified port list to obtain the feasible network scanning efficiency. However, because the huge address space (2 128 ) of the IPv6 network forms an extremely sparse active host distribution, the scheme cannot be directly used for IPv6 network active address detection.

[0004] It is even more difficult to further obtain the active port information on the host. At present, only a once-active IPv6 address list (IPv6Hitlist) can be maintained through various open source channels, and only the commonly used ports on the active addresses in the list are detected. This method has the defects of information lag and low coverage. Moreover, relevant research results show that most active services do not appear in the commonly used port list, so scanning the specified or commonly used ports is not comprehensive and may lead to incorrect conclusions. Therefore, the application is proposed. SUMMARY

[0005] In view of the deficiencies of the prior art, the application provides an IPv6 network space active address and port detection method and device, which realizes the accumulation of active addresses and ports from nothing to more, and solves the defects of the prior art, such as low hit rate, small coverage, incomplete detection of active ports and slow speed for IPv6 active address detection, and realizes efficient and high-coverage detection of IPv6 active addresses and ports.

[0006] Term explanation:

[0007] ICMP alive host discovery technology: refers to that a source host sends an ICMP message, a time stamp, an address mask and the like to a target host, and if a response of the target host is obtained, the target host is considered as an alive host;

[0008] TCP alive host discovery technology: refers to using TCP SYN Ping scanning to determine whether the target host is in the alive state, TCP provides sending, receiving, full duplex service (data flows simultaneously in both directions), both parties have two buffers to send and receive data, the protocol adds a progressive acknowledgement sequence number to the message to inform the sender that the next byte is received, if the waiting time is timeout, the data packet will be sent again, so as to ensure that it is a reliable protocol.

[0009] Sparse active host distribution mechanism: refers to the pull mechanism of data through RPT (Rendezvous Point Tree) based on PIM-SM (Protocol Independent Multicast-Sparse Mode) protocol, which assumes that the hosts in the network do not need to receive multicast data by default, unless explicitly requested. When a host needs to receive multicast data, it sends a join request to the RP (Rendezvous Point), and builds an RPT with the RP as the root.

[0010] The technical scheme of the application is as follows:

[0011] An IPv6 network space active address and port detection method, the steps are as follows:

[0012] (1) Network resource retrieval, obtaining the IPv6 active address list, seed address containing port active information and detailed information of BGP prefix;

[0013] (2) The active address detection is divided into three scenes: no seed address scene, few seed address scene and multiple seed address scene, and the no seed address scene generation model, the few seed address scene generation model and the multiple seed address scene generation model are proposed respectively for the no seed address scene, the few seed address scene and the multiple seed address scene, and the candidate addresses are generated respectively;

[0014] (3) The active addresses discovered in the no seed address scene, the few seed address scene and the multiple seed address scene are respectively subjected to alias prefix detection and alias removal, and the newly discovered addresses after removing the alias are added to the constructed active IPv6 address library;

[0015] (4) Based on the active IPv6 address library constructed in step (3), using the decision model based on the Naive Bayes classifier, all input features are used to efficiently predict the active ports of the active IPv6 addresses, and then the dynamic feedback scanning is completed through the distributed active detection facility, so as to realize the accumulation of the active addresses and ports from nothing to something, and from few to many.

[0016] According to the application, in step (1), the specific steps are as follows:

[0017] Firstly, enter the data interface, and based on the conditions, relevant information is queried, which is divided into two branches, the first branch is IPv6 active address list and seed address containing port active information query, and the second branch is whois information and BGP prefix information query;

[0018] In the first branch, firstly, whether it is IPv6 is judged, if it is IPv6, IPv6 address retrieval is carried out, then whether the IP address is active is judged, if the IP address is active, IPv6 active address retrieval is carried out, IPv6 active address list is outputted, and whether the port is active is judged;

[0019] If it is an active port, active port retrieval is carried out;

[0020] Then, whether it contains organization and ASN information is judged, if it contains organization and ASN information, seed address containing port active information retrieval is carried out, and seed address containing port active information list is outputted;

[0021] In the second branch, firstly, whether it is Whois information is judged, if it is Whois information, Whois information retrieval is carried out, Whois information list is outputted, and whether it contains organization, country and other information is judged;

[0022] If it contains organization, country and other information, BGP prefix retrieval is carried out, and detailed information of the BGP prefix is outputted.

[0023] According to the application, in step (2), the specific steps are as follows:

[0024] (21) IPv6 active address list and detailed information of the BGP prefix are taken as input of active address detection;

[0025] (22) continuous active address data collection is carried out for the collected IPv6 active address, cross detection is carried out by using ICMP alive host discovery technology and TCP alive host discovery technology, and continuous active IPv6 address data is obtained;

[0026] (23) Pyasn is used to query the BGP prefix corresponding to the continuous active IPv6 address, dictionary D {BGP prefix: IPv6 active address list} is formed, then whether the prefix in the Whois information of the BGP prefix inputted in step (21) is in the dictionary D is judged;

[0027] If it is in the dictionary D, the length of the list corresponding to each BPG prefix in the dictionary D is judged:

[0028] If the length of the list is greater than or equal to 10, multiple seed address region data sets are generated, in the form of dictionary D1 {BGP prefix: IPv6 active address list};

[0029] If the list length is less than 10, a few-seed address region dataset is generated in the form of a dictionary D2{BGP prefix: IPv6 active address list};

[0030] The multi-seed address region dataset and the few-seed address region dataset together constitute a seed address region dataset;

[0031] If not in the dictionary D, a no-seed address region dataset is generated in the form of a dictionary D+dictionary D3{no-seed BGP prefix: Whois information}+dictionary D4{seed BGP prefix: Whois information}+detailed information of the BGP prefix (detailed information of the BGP prefix input in step (21));

[0032] The multi-seed address region dataset, the few-seed address region dataset and the no-seed address region dataset together constitute a two-class dataset;

[0033] (24) respectively train the no-seed address scene generation model, the few-seed address scene generation model and the multi-seed address scene generation model;

[0034] The multi-seed region address generation model is to first clean the data of the continuously active IPv6 address, and then generate a tree structure according to the continuously active IPv6 address list, wherein the leaf node of the tree structure is the highest density region, the nodes other than the leaf node in the tree structure have multiple child nodes, the child nodes are split from the parent node according to the minimum entropy value of the character, each child node corresponds to a subset of the continuously active IPv6 address list, and the number of subsets corresponding to the child nodes is less than or equal to a threshold value ε (since the value of the character in IPv6 is 0-f, it is considered that the threshold value ε=16 is the most appropriate), which is the leaf node. The process of guiding the dynamic generation of candidate addresses in the process of sequentially traversing the root node to the leaf node of the tree structure based on the conditional probability with the hit coefficient;

[0035] The few-seed region address generation model is to classify the continuously active IPv6 address in a more fine-grained manner based on the RFC classifier and the KMeans clustering algorithm to obtain a label set, form an IPv6 address+label dataset, and then use the existing technology 6Vision model to generate candidate addresses;

[0036] The RFC classifier refers to using the addr6 tool in the IPv6 toolkit to perform pattern matching on the seed set according to the possible IPv6 address mode proposed in the RFC 7707 document to obtain the classification result based on the RFC;

[0037] The seed-free address region address generation model is to process the whois information of the BGP prefix, and use the phrases of the whois information of the BGP prefix to perform semantic category labeling, named entity category identification (using the named entity categories defined by OntoNotes and using the DeBERTa model for named entity category identification), training of difficult negative samples of phrases, construction of word-level data enhancement data sets (using the WordNet database to extract synonyms and using the FastText model to calculate similarity), and construction of phrase-level data enhancement data sets (using the Parro text conversion tool to obtain), respectively using the model parameters of the pre-trained language model PEARL fine-tuned, loading through the SentenceTransformer package in the sentence_transformers library to obtain the encoded word embedding feature vectors, calculating the similarity between the word embedding feature vectors corresponding to the whois information of the BGP prefix of the seed address region data set and the word embedding feature vectors corresponding to the whois information of the BGP prefix of the seed-free address region data set, and the similarity between the two BGP address configuration modes, since there is similarity between the address configuration modes of the seed BGP prefix and the seed-free BGP prefix, the seed address is migrated, and the candidate address with fine granularity is generated.

[0038] According to the application, in step (3), specifically:

[0039] The candidate addresses generated by the seed-free address scene generation model, the few-seed address scene generation model and the multi-seed address scene generation model are respectively affected by dynamic feedback detection to the respective models, so as to update the parameters of the respective models, and at the same time, the generated active IPv6 addresses are detected by the alias prefix, new IPv6 active address data is obtained, and is updated to the IPv6 active address list, and then step (2) is repeated until the preset number of rounds is completed.

[0040] According to the application, the alias prefix detection step is as follows:

[0041] (31) The generated active IPv6 address to be detected is probed again to ensure that the alias detection can be effectively performed;

[0042] (32) Based on the sparse active host distribution mechanism, a target address is randomly generated in the IPv6 network for detection, and a response possibility is very small, which is a hypothesis, and one BGP prefix is randomly selected from the detailed information of the BGP prefix and 16 candidate addresses are generated;

[0043] (33) Using ICMP alive host discovery technology to send ICMP probe packets to the target address, if all responses can be obtained from the target address, then the prefix can be determined as a candidate prefix;

[0044] (34) For the candidate prefix, a random address is generated under the candidate prefix, and then an ICMP probe packet is sent to the address, and if a response is still received at this time, the BGP prefix is considered as an alias prefix.

[0045] According to the present application, preferably, in step (4), the active port prediction step is as follows:

[0046] (41) Data set construction:

[0047] Based on the seed address of the port active information in step (1), the active port prediction global feature is extracted, that is, the BGP prefix, organization and ASN information, and then the seed address is generated into an address space tree using the DHC algorithm, the address mode corresponding to the leaf node of the tree is extracted, and the address mode is taken as the active port prediction local feature, and the active port prediction data set of IPv6 address + port prediction global feature + port prediction local feature is constructed;

[0048] (42) Port prediction:

[0049] Based on the active port prediction local feature and the active port prediction global feature, a naive Bayes classifier is used to predict whether the port is active, each feature corresponds to a naive Bayes classifier, and it is assumed that the features are conditionally independent given the class label, and the port is predicted;

[0050] (43) Port detection:

[0051] According to the decision model of the trained naive Bayes classifier obtained in step (42), all ports predicted by the model are used as the predicted ports of the active IPv6 address to be predicted (that is, the IPv6 address list in the data set constructed in step (41)), and the distributed active detection module is used for active port detection, and the active ports detected by the IPv6 address and the corresponding BGP prefix, organization and other information are updated to the dictionary D5.

[0052] According to the present application, preferably, in step (42), the specific steps are as follows:

[0053] ① The data set constructed in step (41) is divided into a model initialization data set and a model training data set, the IPv6 addresses in the model training data set are traversed, and the port prediction global feature and the port prediction local feature corresponding to the target IPv6 address are obtained;

[0054] ②According to the model initialization data set, the IPv6 address and the corresponding port prediction global feature and the port prediction local feature are predicted, the IPv6 address of the model initialization data set is scanned by using a detector (zmap), the real active port corresponding to the IPv6 address is obtained, and the naive Bayes model is initialized based on the real active port distribution;

[0055] ③The port prediction global feature and the port prediction local feature of the IPv6 address of the model training data set are input into the initialized naive Bayes classifier, the active value of each port corresponding to the target IPv6 address is obtained, all active values are sorted in descending order, and at least one port corresponding to the active value is taken as the active port of the target IPv6 address, wherein the active value is the sum of the outputs of all features corresponding to the naive Bayes classifier;

[0056] ④For the IPv6 address of the model training data set, the determined active port is input into the detector (zmap) for detection, and the naive Bayes classifier is updated after the detection result is obtained;

[0057] ⑤The global classifier and the local classifier are called by using multi-threading, the respective active ports of the IPv6 addresses in the model training data set are found, and after multiple rounds, the prediction is stopped until the port coverage rate reaches 90%.

[0058] According to the application, in step (4), the distributed active detection facility includes a resource detection control node and m probe nodes, the resource detection control node is used for executing a resource detection control process, and the probe nodes are used for executing a distributed probe detection process.

[0059] The resource detection control process is as follows:

[0060] I. The resource detection control node obtains a detection task of a client, the detection task is analyzed by the resource detection control node, and a total address set and a port set that need to be detected are extracted;

[0061] II. A detection subtask is generated, one detection subtask includes a subaddress set and a port, the subaddress set is obtained by uniformly splitting the total address set, and the port comes from a certain port in the port set; when the address set is split, randomization processing is performed, that is, the addresses in each subaddress set are made to be random as much as possible;

[0062] III. Each detection subtask is reasonably distributed to an available probe node by using a certain algorithm, the algorithm is: average distribution, uniform distribution according to node bandwidth or a distribution method based on optimization theory, the above algorithm strategies are used to determine how to configure the subtask and the node, and a configuration list is generated;

[0063] IV. The resource detection control node sequentially issues the assigned detection tasks to each probe node according to the allocation results, and then waits for the feedback of the issued task results. If the feedback results are received, the file integrity is verified, the task results are processed, such as whether there are errors or the results are not deduplicated, and it is checked whether all subtasks are completed. If all subtasks are completed, the results of all subtasks are combined and stored to obtain the final results. When the client requests to access the task results, the task results are streamed to the client;

[0064] The distributed probe detection process is as follows:

[0065] First, the task issued by the resource detection control node to the probe node is received, including verifying the file integrity, obtaining the network environment information, and parsing the task parameters;

[0066] Second, the detection task is executed, and the task to be executed is split into several times, and the detection tool is called to execute the subtask;

[0067] Third, the detection result is returned. After the subtask detection is completed, the task results are combined, and after the execution is completed, the detection results of all subtasks are combined and returned to the resource detection control node, and the next task is received.

[0068] An IPv6 network space active address and port detection device, comprising:

[0069] A network resource retrieval module is used to obtain an IPv6 active address list, a seed address containing port active information, and detailed information of a BGP prefix;

[0070] An active address detection module is used to divide active address detection into three scenarios: no seed address scenario, few seed address scenario, and multiple seed address scenario. A no seed address scenario generation model, a few seed address scenario generation model, and a multiple seed address scenario generation model are proposed for the no seed address scenario, the few seed address scenario, and the multiple seed address scenario, respectively, to generate candidate addresses;

[0071] An active IPv6 address library construction module is used to perform alias prefix detection and alias removal on the active addresses discovered in the no seed address scenario, the few seed address scenario, and the multiple seed address scenario, respectively, and add the newly discovered addresses after removing the aliases to the constructed active IPv6 address library;

[0072] A scanning detection module is used to use a decision model based on a naive Bayes classifier to efficiently predict the active ports of the active IPv6 addresses based on the active IPv6 address library constructed in step (3) and all input features, and to complete dynamic feedback scanning through a distributed active detection facility to realize the accumulation of active addresses and ports from nothing to something and from few to many.

[0073] The application has the beneficial effects that:

[0074] 1、The application obtains an effective and reliable IPv6 active address set through the IPv6 active address detection method based on machine learning, and then uses the open port prediction method based on the naive Bayes decision model to accurately predict the active ports of the high-coverage IPv6 active address in the network space in the process of scanning the entire IPv6 address space, realize the accumulation of active addresses and ports from nothing to many, and solve the defects of low hit rate, small coverage, incomplete detection of active ports, and slow speed of the existing method for IPv6 active address detection, and realize efficient and high-coverage detection of IPv6 active addresses and ports.

[0075] 2、In the seedless address generation model, the pre-training language model is used to fully mine the correlation between the seedless BGP prefix and the seed BGP prefix, and the possibility of active address migration is maximized from the seed address migration mode, so that the invalid migration is avoided, and the coverage and detection efficiency of the active address are improved.

[0076] 3、In the multi-seed regional address generation model, in order to improve the efficiency of active address detection and avoid low hit rate of detection, the hit coefficient is increased to adjust the direction of detection based on conditional probability, and more effective active addresses can be obtained compared with simple detection based on conditional probability.

[0077] 4、The application dynamically supplements the active address generated by the seedless regional address generation model to the seed address of the multi-seed regional address generation model and the seed address of the few-seed regional address generation model, and supplements the active address generated by the few-seed regional address generation model to the seed address of the multi-seed regional address generation model, which can fully mine the coverage ability of the active address in the entire network space. BRIEF DESCRIPTION OF DRAWINGS

[0078] Figure 1 The flowchart of the application is shown in the figure;

[0079] Figure 2 The active address detection flowchart of the application is shown in the figure;

[0080] Figure 3 The device structure diagram of embodiment 2 of the application is shown in the figure. DETAILED DESCRIPTION

[0081] The application will be further described below through embodiments and in conjunction with the accompanying drawings, but is not limited thereto.

[0082] Embodiment 1:

[0083] The embodiment provides an IPv6 network space active address and port detection method, and steps are as follows:

[0084] (1) Network resource retrieval, obtaining an IPv6 active address list, a seed address containing port active information and detailed information of a BGP prefix, and specific steps are as follows:

[0085] First, enter the data interface, and query related information based on conditions, which is divided into two branch queries, the first is IPv6 active address list and seed address containing port active information query, and the second branch is Whois information and BGP prefix information query;

[0086] The first branch is that whether it is IPv6 is first judged, if it is IPv6, then IPv6 address retrieval is carried out, whether the IP address is active is further judged, if the IP address is active, then IPv6 active address retrieval is carried out, an IPv6 active address list is output, and whether the port is active is judged;

[0087] If it is an active port, then active port retrieval is carried out;

[0088] Whether the information contains organization and ASN is further judged, if the information contains organization and ASN, then seed address containing port active information retrieval is carried out, and a seed address list containing port active information is output;

[0089] The second branch is that whether it is Whois information is first judged, if it is Whois information, then Whois information retrieval is carried out, a Whois information list is output, and whether the information contains organization, country and other information is judged;

[0090] If the information contains organization, country and other information, then BGP prefix retrieval is carried out, and detailed information of a BGP prefix is output.

[0091] The data interface of the above process includes public resources and a network mapping platform node, the network mapping platform node is a data platform deployed on a server, mainly data collected from public resources is imported into the platform. In addition, the platform completes module scheduling and data storage of the device and the like.

[0092] (2) The active address detection is divided into three scenes, a seed address-free scene, a few seed address scene and a multiple seed address scene, a seed address-free scene generation model, a few seed address scene generation model and multiple seed address scene generation model are respectively proposed for the seed address-free scene, the few seed address scene and the multiple seed address scene, and candidate addresses are respectively generated, and specific steps are as follows:

[0093] (21) The IPv6 active address list and the detailed information of the BGP prefix are taken as inputs of the active address detection.

[0094] (22) Collecting IPv6 active addresses, using ICMP and TCP alive host discovery technology to cross-probe, obtaining persistent active IPv6 address data;

[0095] The output IPv6 active address list obtained from network resource retrieval is used as the input of active address detection, initializing the persistent active address queue Q, the detection period period = 0, the detection count count = 0, the constant number of detections per period M, the constant detection period N, the number of times T that the address appears in M detections, first using the ICMP alive host discovery technology to probe the IPv6 hitlist list, judging whether the request has a response, if there is a response, obtaining the responsive IPv6 address list, otherwise obtaining the non-responsive IPv6 address list, then using the TCP alive host discovery technology to probe the non-responsive IPv6 address list, finally obtaining the active address list and the inactive address list detected in this period, the detection count count is incremented by one, if count < M times, continue to complete the detection in this period, otherwise enter the next period of detection, the detection period period is incremented by one, if period < N times, continue to complete the remaining period detection, otherwise obtain the list of active address lists of M detections in N periods, and perform data statistics on the list, if the IPv6 address appears T / (N*M)>80%, the address enters the queue Q, until all address statistical analysis is completed, obtaining the final long-term active address queue Q.

[0096] (23) Using Pyasn to query the BGP prefix corresponding to the persistent active IPv6 address, forming a dictionary D{BGP prefix: IPv6 active address list}, then judging whether the prefix in the Whois information of the BGP prefix input in step (21) is in the dictionary D;

[0097] If it is in the dictionary D, judge the length of the list corresponding to each BPG prefix in the dictionary D:

[0098] If the list length is greater than or equal to 10, generate multiple seed address region data sets in the form of dictionary D1{BGP prefix: IPv6 active address list};

[0099] If the list length is less than 10, generate a few seed address region data sets in the form of dictionary D2{BGP prefix: IPv6 active address list};

[0100] The multiple seed address region data sets and the few seed address region data sets together constitute the seed address region data set;

[0101] If not in the dictionary D, a seedless address region dataset is generated in the form of dictionary D + dictionary D3 {seedless BGP prefix: Whois information} + dictionary D4 {seeded BGP prefix: Whois information} + detailed information of the BGP prefix (detailed information of the BGP prefix input in step (21));

[0102] The multi-seed address region dataset, the few-seed address region dataset and the seedless address region dataset jointly constitute two types of datasets;

[0103] (24) respectively training the seedless address scene generation model, the few-seed address scene generation model and the multi-seed address scene generation model;

[0104] The multi-seed address region generation model is to perform data cleaning on the continuously active IPv6 addresses, and then generate a tree structure according to the continuously active IPv6 address list, wherein the leaf node of the tree structure is the highest density region, the nodes other than the leaf node in the tree structure have multiple child nodes, the child nodes are split from the parent node according to the minimum entropy value of the character, each child node corresponds to a subset of the continuously active IPv6 address list, and the number of subsets corresponding to the child nodes is less than or equal to a threshold value ε (since the value of the character in IPv6 is 0-f, it is considered that the threshold value ε = 16 is the most appropriate), and the child node is the leaf node. The process of guiding the dynamic generation of candidate addresses in the process of sequentially traversing the root node to the leaf node of the tree structure based on the conditional probability with the hit coefficient;

[0105] The conditional probability with the hit coefficient refers to the definition of the conditional probability in it is based on a space tree (such as Figure 3 As shown) Each child node includes the first probability R of finding an active address by the address block corresponding to the child node, and the second probability Q of the parent node selecting the child node, wherein the first probability is determined by the relative size of the second probability among all sibling nodes, and the probability P of the child node being more active than other sibling nodes is proportional to the probability of the parent node selecting the child node. Since the density distribution of the seed address is not necessarily consistent with the density distribution of the active address in the actual network, it is easy to appear that the proportion of the active address in the address space corresponding to the node is very high, but the hit rate of detecting new active addresses in the generated candidate address is low. Therefore, the hit coefficient is added in the conditional probability, and the direction of generating the candidate address is adjusted more accurately, and the hit coefficient δ = (the number of new active addresses A detected each time in the child node / the cumulative sum B of the number of new active addresses detected each time in the child node of other sibling nodes). In summary, R = δ * Q / P;

[0106] The few-seed region address generation model is based on the RFC classifier and the KMeans clustering algorithm to classify the continuously active IPv6 addresses in a more fine-grained manner, obtain a label set, form an IPv6 address + label data set, and then use the existing technology 6Vision model to generate candidate addresses.

[0107] The RFC classifier refers to the mode matching of the seed set using the addr6 tool in the IPv6 toolkit according to the possible IPv6 address mode proposed in the RFC 7707 document, to obtain the classification result based on the RFC.

[0108] The no-seed region address generation model is to first process the whois information of the BGP prefix, and use the phrases of the whois information of the BGP prefix to do semantic category labeling, named entity category recognition (using the named entity categories defined by OntoNotes and using the DeBERTa model for named entity category recognition), training of difficult negative samples of phrases, construction of word-level data augmentation data set (using the WordNet database to extract synonyms, and using the FastText model to calculate the similarity), and construction of phrase-level data augmentation data set (using the Parro text conversion tool to obtain). The whois information of the BGP prefix with seed and the whois information of the BGP prefix without seed are respectively loaded by using the model parameters of the pre-trained language model PEARL after fine-tuning, to obtain the encoded word embedding feature vector, and the similarity between the two BGP address configuration modes is calculated by calculating the similarity between the word embedding feature vectors corresponding to the whois information of the BGP prefix of the seed address region data set and the word embedding feature vectors corresponding to the whois information of the BGP prefix of the no-seed address region data set. Since there is similarity between the address configuration modes of the BGP prefix with seed and the BGP prefix without seed, the seed address is migrated to generate fine-grained candidate addresses.

[0109] (3) The discovered active addresses in the no-seed address scenario, few-seed address scenario and multi-seed address scenario are respectively subjected to alias prefix detection and alias removal, and the newly discovered addresses after removing the alias are added to the constructed active IPv6 address library. Specifically:

[0110] The candidate addresses generated by the seed address scene generation model, the few-seed address scene generation model and the multi-seed address scene generation model respectively are subjected to dynamic feedback type detection to the respective models, thereby updating the parameters of the respective models, and the generated active IPv6 addresses are subjected to alias prefix detection to obtain new IPv6 active address data, which is updated to the IPv6 active address list, and then step (2) is repeated until a preset number of rounds is completed.

[0111] The alias prefix detection step is as follows:

[0112] (31) The generated active IPv6 addresses to be detected are probed again to ensure that the alias detection can be effectively performed;

[0113] (32) Based on the sparse active host distribution mechanism, a target address is randomly generated in the IPv6 network for detection, and a BGP prefix is randomly selected from the detailed information of the BGP prefix and 16 candidate addresses are generated;

[0114] (33) The ICMP alive host discovery technology is used to send an ICMP probe packet to the target address, and if all responses can be obtained from the target address, it can be determined that the prefix is a candidate prefix;

[0115] (34) For the candidate prefix, an address is randomly generated under the candidate prefix, and an ICMP probe packet is sent to the address. If a response is still received at this time, the BGP prefix is considered to be an alias prefix.

[0116] (4) Based on the active IPv6 address library constructed in step (3), a decision model based on a naive Bayes classifier is used to efficiently predict the active ports of the active IPv6 addresses, and a distributed active detection facility is used to complete dynamic feedback scanning to realize the accumulation of active addresses and ports from nothing to something and from few to many.

[0117] The active port prediction step is as follows:

[0118] (41) Data set construction:

[0119] Based on the seed address of the port active information in step (1), global features for active port prediction are extracted, i.e. BGP prefix, organization and ASN information, and then the DHC algorithm is used to generate an address space tree from the seed address, the address pattern corresponding to the leaf nodes of the tree is extracted, and the address pattern is used as a local feature for active port prediction to construct an active port prediction data set of IPv6 address + global feature for port prediction + local feature for port prediction.

[0120] (42) Port prediction:

[0121] Based on the active port prediction local feature and the active port prediction global feature, the active port is predicted by using the Naive Bayes classifier, each feature corresponds to a Naive Bayes classifier, assuming that the features are conditionally independent given the class label, and the port is predicted;

[0122] (43) Port detection:

[0123] According to the decision model of the trained Naive Bayes classifier obtained in step (42), all the ports predicted by the model are used as the predicted ports of the active IPv6 addresses (i.e. the IPv6 address list in the data set constructed in step (41)), and the distributed active port detection module is used for active port detection, and the detected active ports and the corresponding BGP prefix, organization, etc. Information of IPv6 address is updated to dictionary D5.

[0124] In step (42), the specific steps are as follows:

[0125] ①The data set constructed in step (41) is divided into model initialization data set and model training data set, and the IPv6 addresses in the model training data set are traversed to obtain the port prediction global feature and the port prediction local feature corresponding to the target IPv6 address;

[0126] Among them, the model initialization data set is used to initialize the parameters of the Naive Bayes classifier, and the model training data set is used for model training after model initialization, each IPv6 address in the model training data set does not contain or contains not all corresponding active ports, and the target IPv6 address contains at least one unscanned port;

[0127] ②According to the IPv6 address and the corresponding port prediction global feature and the port prediction local feature of the model initialization data set, the IPv6 address of the model initialization data set is scanned by the detector (zmap) to obtain the real active port corresponding to the IPv6 address, and the Naive Bayes model is initialized based on the real active port distribution;

[0128] ③The port prediction global feature and the port prediction local feature of the IPv6 address of the model training data set are input into the initialized Naive Bayes classifier to obtain the active value of each port corresponding to the target IPv6 address, and all the active values are sorted in descending order, and at least one port corresponding to the active value is used as the active port of the target IPv6 address, wherein the active value is the sum of the outputs of all features corresponding to the Naive Bayes classifier;

[0129] ④For the IPv6 address of the model training data set, the determined active port is input into the detector (zmap) for detection, and the Naive Bayes classifier is updated after obtaining the detection result.

[0130] ⑤Using multi-threaded call global classifier and local classifier, find the corresponding active port in each model training data set IPv6 address, after several rounds, until the port coverage rate reaches 90% stop prediction.

[0131] In step (4), the distributed active probe facility belongs to the infrastructure module of active address and port detection. The logical module of the distributed active probe module mainly includes resource detection control and distributed probe detection. Its physical architecture is "one master and multiple slaves", that is, one control node and several probe nodes. This facility takes the new IPv6 active address list obtained by active address detection and the IPv6 active address to be detected obtained by active port detection and the corresponding predicted port as input.

[0132] The distributed active probe facility includes a resource detection control node and m probe nodes. The resource detection control node is used to execute the resource detection control process, and the probe node is used to execute the distributed probe detection process.

[0133] The resource detection control process is as follows: upload the resource to be detected to the resource detection control node, then call the task scheduling process, the resource detection control node verifies the specified probe information, creates a task, verifies the integrity of the data file, and executes the task creation sub-process according to a certain resource allocation strategy, and links the request task result back to the client. This sub-process mainly allocates the detection sub-tasks generated by the task configuration module to each probe node reasonably, and issues the task. After the task is issued, this sub-process ends and returns to the main process, waits for the feedback of the issued task result. If the feedback result is received, the file integrity is verified, the task result is processed, such as whether there is an error or the result is not deduplicated, etc., and whether all sub-tasks are completed is checked. If all sub-tasks are completed, the results of all sub-tasks are combined to obtain the final result. When the client requests to access the task result, the task result is transmitted to the client through streaming.

[0134] I. The resource detection control node obtains the detection task of the client, and the resource detection control node parses the detection task to extract the total address set and the port set that need to be detected;

[0135] II. Generate a detection sub-task, which contains a sub-address set and a port. The sub-address set is obtained by uniformly splitting the total address set, and the port comes from a certain port in the port set. When splitting the address set, randomization processing is performed, that is, the addresses in each sub-address set are made as random as possible.

[0136] III. Allocate each probe subtask to available probe nodes using a specific algorithm. The algorithm used may be: average allocation, uniform allocation based on node bandwidth, or allocation method based on optimization theory. The algorithm strategy is used to determine how to configure the subtasks and nodes and generate a configuration list.

[0137] IV. The resource detection control node will distribute the assigned detection tasks to each probe node in sequence according to the allocation results, and then wait for the feedback of the task results. If the feedback results are received, the integrity of the file will be verified, the task results will be processed, such as whether there are errors or whether the results are not deduplicated, and all sub-tasks will be checked to see if they are all completed. If they are all completed, the results of all sub-tasks will be merged and stored to obtain the final result. When the client requests access to the task results, the task results will be streamed to the client.

[0138] The distributed probe detection process is as follows: First, the probe receives the task issued by the resource control node, creates a task process, verifies file integrity, then obtains network environment information based on the resource configuration file, and creates a detection sub-process. This sub-process first splits the task to be executed, then calls detection tools such as ping and ZMap to execute the detection sub-tasks, and finally merges the detection results of all sub-tasks and sends them back to the resource detection control node. The sub-process ends, returns to the main process, and waits to receive the next task.

[0139] in:

[0140] Ping probes are tools that use ping to determine whether a local host can successfully exchange (send and receive) data packets with another host. Based on the returned information, one can infer whether TCP / IP parameters are set correctly, whether the system is functioning properly, and whether the network is working smoothly. Specifically, the process involves first sending ICMP (Internet Control Message Protocol) echo packets to the computer and listening for echo reply packets to verify the connection with one or more remote computers. Then, each sent data packet is waited for a maximum of one second, and finally, the number of transmitted and received data packets is printed.

[0141] ZMap is a scanning software developed by a research team at the University of Michigan led by Durumeric. This tool can scan the entire public internet within an hour, displaying information on nearly 4 billion online devices. The scan results can show which websites cannot defend against specific vulnerabilities.

[0142] Distributed probe detection consists of three main processes: receiving probe tasks, executing probe tasks, and returning probe results. The specific steps are as follows:

[0143] First, receive the task issued by the resource detection control node to the probe node, including verifying file integrity, obtaining network environment information and parsing task parameters;

[0144] Second, execute the detection task, split the to-be-executed task into several times, and call a detection tool to execute a detection subtask;

[0145] Third, return the detection result, combine the task results after the subtask detection is completed, combine the detection results of all subtasks after the execution is completed, and return to the resource detection control node, waiting for receiving the next task.

[0146] Embodiment 2:

[0147] The embodiment provides an IPv6 network space active address and port detection device, which comprises:

[0148] A network resource retrieval module is used to obtain an IPv6 active address list, a seed address containing port active information and detailed information of a BGP prefix;

[0149] An active address detection module is used to divide active address detection into three scenarios, i.e., a no-seed address scenario, a few-seed address scenario and a multiple-seed address scenario, and propose a no-seed address scenario generation model, a few-seed address scenario generation model and a multiple-seed address scenario generation model for the no-seed address scenario, the few-seed address scenario and the multiple-seed address scenario respectively, and generate candidate addresses respectively;

[0150] An active IPv6 address library construction module is used to perform alias prefix detection and alias removal on the active addresses discovered in the no-seed address scenario, the few-seed address scenario and the multiple-seed address scenario respectively, and add the newly discovered addresses after the alias removal to the constructed active IPv6 address library;

[0151] A scanning detection module is used to use a decision model based on a naive Bayes classifier, utilize all input features, efficiently predict active ports of active IPv6 addresses, and then complete dynamic feedback scanning through a distributed active detection facility, so that the active address and port are accumulated from nothing to something and from few to many.

[0152] The above merely describes preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting active addresses and ports in IPv6 network space, characterized in that, The steps are as follows: (1) Network resource retrieval: obtain a list of active IPv6 addresses, seed addresses containing port activity information, and detailed information on BGP prefixes; (2) Active address detection is divided into three scenarios: seedless address scenario, few seed address scenario and multiple sub-address scenario. Seedless address scenario generation model, few seed address scenario generation model and multiple sub-address scenario generation model are proposed for seedless address scenario, few seed address scenario and multiple sub-address scenario respectively, and candidate addresses are generated respectively. The specific steps are as follows: (21) Use the IPv6 active address list and BGP prefix details as input for active address probing; (22) Collect continuous active address data for the collected IPv6 active addresses, and use ICMP live host discovery technology and TCP live host discovery technology to cross-probe to obtain continuous active IPv6 address data; (23) Use Pyasn to query the BGP prefix corresponding to the continuously active IPv6 address to form a dictionary D{BGP prefix: list of active IPv6 addresses}, and then determine whether the prefix in the Whois information of the BGP prefix input in step (21) is in dictionary D; If we have a dictionary D, we need to determine the length of the list corresponding to each BPG prefix in dictionary D: If the list length is greater than or equal to 10, multiple sub-address region datasets are generated, in the form of a dictionary D1{BGP prefix: list of active IPv6 addresses}. If the list length is less than 10, a few-seed address region dataset is generated, in the form of a dictionary D2{BGP prefix: list of active IPv6 addresses}; Multiple sub-address region datasets and few seed address region datasets together constitute a seeded address region dataset; If it is not in dictionary D, then a seedless address region dataset is generated, in the form of dictionary D + dictionary D3 {seedless BGP prefix: Whois information} + dictionary D4 {seed BGP prefix: Whois information} + detailed information of BGP prefix; Multiple sub-address region datasets, few-seed address region datasets, and seedless address region datasets together constitute two types of datasets; (24) Train the seedless address scene generation model, the few seed address scene generation model, and the multiple sub-address scene generation model respectively; The multi-sub-region address generation model first cleans the continuously active IPv6 addresses, and then generates a tree structure based on the list of continuously active IPv6 addresses. The leaf nodes of the tree structure are the highest density regions. Each node in the tree structure, except for the leaf nodes, has multiple child nodes. The child nodes are split from the parent node based on the minimum entropy value of the characters. Each child node corresponds to a subset of the list of continuously active IPv6 addresses. If the number of subsets corresponding to a child node is less than or equal to the threshold ε, it is a leaf node. The candidate addresses are dynamically generated by traversing the tree structure from the root node to the leaf node in the conditional probability generation direction with a hit coefficient. The seedless region address generation model is based on the RFC classifier and KMeans clustering algorithm to classify continuously active IPv6 addresses in a more granular way, obtain a set of labels, form an IPv6 address + label dataset, and then use the 6Vision model to generate candidate addresses. The seedless address generation model first processes the WHOIS information of BGP prefixes. It then performs semantic category labeling, named entity category recognition, training with difficult negative samples for phrases, and constructs word-level and phrase-level data augmentation datasets. The WHOIS information with and without seeded BGP prefixes is then fine-tuned using the pre-trained PEARL language model to obtain encoded word embedding feature vectors. The similarity between the word embedding feature vectors corresponding to the BGP prefixes in the seeded address region dataset and those in the seedless address region dataset is measured. Since there is similarity between the seeded and seedless BGP prefix address configurations, seed address migration is performed to generate fine-grained candidate addresses. (3) Perform alias prefix detection and alias removal on active addresses found in scenarios with no seed address, few seed address, and multiple sub-addresses respectively, and add the newly discovered addresses after removing aliases to the active IPv6 address database. The candidate addresses generated by the seedless address generation model, the few-seed address generation model, and the multiple-sub-address generation model are dynamically fed back to their respective models to update their parameters. At the same time, the generated active IPv6 addresses are subjected to alias prefix detection to obtain new IPv6 active address data, which is then updated to the IPv6 active address list. Then, step (2) is repeated until the preset number of rounds is completed. (4) Based on the active IPv6 address database, a decision model based on Naive Bayes classifier is used to predict the active ports of active IPv6 addresses. Then, a dynamic feedback scan is completed through distributed active probing facilities to achieve the accumulation of active addresses and ports from nothing to something, and from few to many.

2. The method for detecting active IPv6 network addresses and ports as described in claim 1, characterized in that, In step (1), the specific steps are as follows: First, enter the data interface and query relevant information based on the conditions. The query is divided into two branches: the first is the query of the IPv6 active address list and the seed address containing port active information, and the second branch is the query of whois information and BGP prefix information. The first branch first determines whether it is IPv6. If it is IPv6, then performs an IPv6 address search. Next, determine whether the IP address is active. If the IP address is active, perform an IPv6 active address search, output a list of active IPv6 addresses, and determine whether the port is active. If it is an active port, then perform an active port search; Next, determine whether it contains organizational and ASN information. If it does, perform a seed address retrieval that contains port activity information and output a list of seed addresses containing port activity information. The second branch first determines whether it is Whois information. If it is Whois information, it performs Whois information retrieval, outputs a list of Whois information, and determines whether it contains organizational information. If the information includes organizational information, a BGP prefix search is performed, and detailed information about the BGP prefix is ​​output.

3. The method for detecting active IPv6 network addresses and ports as described in claim 2, characterized in that, The steps for alias prefix detection are as follows: (31) Re-probe the active IPv6 addresses generated during the detection process; (32) Based on the sparse active host distribution mechanism, a target address is randomly generated in the IPv6 network for probing. A BGP prefix is ​​randomly selected from the detailed information of the BGP prefix and 16 candidate addresses are generated. (33) Use ICMP live host discovery technology to send ICMP probe packets to the target address. If all responses can be obtained from the target address, then the prefix can be determined as a candidate prefix. (34) For the candidate prefix, another address is randomly generated under the candidate prefix, and then an ICMP probe packet is sent to the address. If a response is still received at this time, the BGP prefix is ​​considered to be an alias prefix.

4. The method for detecting active IPv6 network space addresses and ports as described in claim 3, characterized in that, In step (4), the active port prediction steps are as follows: (41) Dataset Construction: Based on the seed address of port activity information, global features for active port prediction are extracted, namely BGP prefix, organization and ASN information. Then, the DHC algorithm is used to generate an address space tree from the seed address, and the address pattern corresponding to the leaf node of the tree is extracted. This address pattern is used as the local feature for active port prediction, and an active port prediction dataset consisting of IPv6 address + global port prediction features + local port prediction features is constructed. (42) Port prediction: Based on the local and global features of active port prediction, a Naive Bayes classifier is used to predict whether a port is active. Each feature corresponds to a Naive Bayes classifier. It is assumed that the features are conditionally independent given the class label, and port prediction is performed. (43) Port detection: Based on step (42), a trained Naive Bayes classifier decision model is obtained. Then, all ports predicted by the model are used as predicted ports of the active IPv6 addresses to be predicted. Active ports are detected using a distributed active detection module, and the active ports corresponding to the detected IPv6 addresses, their corresponding BGP prefixes, and organizations are updated in dictionary D5.

5. The method for detecting active IPv6 network addresses and ports as described in claim 4, characterized in that, In step (42), the specific steps are as follows: ① Divide the dataset constructed in step (41) into the model initialization dataset and the model training dataset. Traverse the IPv6 addresses in the model training dataset to obtain the global port prediction features and local port prediction features corresponding to the target IPv6 address. ② Based on the IPv6 addresses and their corresponding port prediction global features and port prediction local features of the model initialization dataset, a full port scan is performed on the IPv6 addresses of the model initialization dataset using a detector to obtain the real active ports corresponding to the IPv6 addresses. The Naive Bayes model is then initialized based on the distribution of the real active ports. ③ Input the global and local port prediction features of the IPv6 address in the model training dataset into the Naive Bayes classifier that has been initialized to obtain the activity value of each port corresponding to the target IPv6 address. Sort all the activity values ​​in descending order and take at least one port corresponding to the activity value as the active port of the target IPv6 address. The activity value is the sum of the outputs of the Naive Bayes classifiers corresponding to all features. ④ The IPv6 addresses of the model training dataset are input into the detector based on the determined active ports for detection. After obtaining the detection results, the Naive Bayes classifier is updated. ⑤ Use multi-threaded calls to the global and local classifiers to find the corresponding active ports in the IPv6 addresses of the model training dataset. After multiple rounds, stop prediction when the port coverage reaches 90%.

6. The method for detecting active addresses and ports in IPv6 network space as described in claim 5, characterized in that, In step (4), the distributed active detection facility includes a resource detection control node and m probe nodes. The resource detection control node is used to execute the resource detection control process, and the probe nodes are used to execute the distributed probe detection process.

7. The method for detecting active IPv6 network space addresses and ports as described in claim 6, characterized in that, The resource detection and control process is as follows: Ⅰ. The resource detection control node obtains the client's detection task, parses the detection task, and extracts the total set of addresses and ports to be detected; II. Generate probe subtasks. A probe subtask contains a sub-address set and a port. The sub-address set is obtained by uniformly splitting the total address set, and the port comes from a port in the port set. Randomization is performed when splitting the address set. III. Allocate each probe subtask to the probe nodes using a certain algorithm, such as: average allocation, uniform allocation based on node bandwidth, or allocation method based on optimization theory. IV. The resource detection control node will distribute the assigned detection tasks to each probe node in sequence according to the allocation results, and then wait for the feedback of the task results. If the feedback results are received, the file integrity will be verified, the task results will be processed, and all sub-tasks will be checked to see if they are all completed. If they are all completed, the results of all sub-tasks will be merged and stored to obtain the final result. When the client requests access to the task results, the task results will be streamed to the client. The distributed probe detection process is as follows: First, receive tasks from the resource exploration control node to the probe node, including verifying file integrity, obtaining network environment information, and parsing task parameters; Second, the detection task is executed by breaking down the task to be executed into several parts and calling the detection tool to execute the detection sub-tasks. Third, the detection results are returned. After the sub-task detection is completed, the task results are merged. After the execution is completed, the detection results of all sub-tasks are merged and sent back to the resource detection control node, waiting to receive the next task.

8. An IPv6 network space active address and port detection device, applied to the IPv6 network space active address and port detection method according to claim 1, characterized in that, include: Network resource retrieval module: used to obtain a list of active IPv6 addresses, seed addresses containing port activity information, and detailed information on BGP prefixes; The active address detection module is used to divide active address detection into three scenarios: seedless address scenario, few seed address scenario, and multiple sub-address scenario. For the seedless address scenario, few seed address scenario, and multiple sub-address scenario, respectively, a seedless address scenario generation model, a few seed address scenario generation model, and a multiple sub-address scenario generation model are proposed to generate candidate addresses respectively. Active IPv6 address database construction module: This module is used to perform alias prefix detection and alias removal on active addresses discovered in scenarios with no seed address, few seed address, and multiple sub-addresses, and adds the newly discovered addresses after alias removal to the constructed active IPv6 address database. Scanning and probing module: Based on the active IPv6 address database built in step (3), the decision model based on the Naive Bayes classifier is used to efficiently predict the active ports of active IPv6 addresses by utilizing all input features. Then, dynamic feedback scanning is completed through distributed active probing facilities to achieve the accumulation of active addresses and ports from nothing to something, and from few to many.

Images