IP positioning landmark mining system, method, electronic device and storage medium

Abstract

The present invention provides an IP positioning landmark mining system, method, electronic device and storage medium, and relates to the field of IP positioning technology. The system includes: an uncontrollable landmark mining device, which is used to retrieve a first candidate website set from the entire network using a search engine, and screen out a second candidate website set from the first candidate website set using a multi-classifier voting method; each candidate website in the candidate website set includes a suspected uncontrollable landmark; a controllable landmark mining device, which is used to classify the public probe website set using a semi-supervised classification method, and determine the candidate public probe website set based on the classification result; the candidate public probe website in the candidate public probe website set includes a suspected controllable landmark; a landmark generator, which is used to retrieve from the second candidate website set and the candidate public probe website set according to a preset search condition, and generate an IP positioning landmark set based on the search result. The present invention can greatly improve the number, mining efficiency and functional richness of IP positioning landmarks.

Description

Technical Field

[0001] The present invention relates to the field of IP positioning technology, and in particular to an IP positioning landmark mining system, method, electronic equipment and storage medium. Background Art

[0002] At present, Internet Protocol (IP) positioning technology maps IP addresses to real-world geographic locations and has been widely used in areas such as precise online advertising delivery, privacy protection, tracking and tracing, information push, and account protection.

[0003] An IP location landmark represents a network node with a determined IP address and its geographic location. The conventional method for obtaining IP location landmarks is to search for information containing locations in active network nodes and mine their geographic locations, such as methods based on network content mining, methods based on online maps, and methods based on network measurement platforms. These methods all have the disadvantages of a small number of IP location landmarks, low mining efficiency, and limited functions. Summary of the invention

[0004] The present invention provides an IP location landmark mining system, method, electronic device and storage medium to solve the defects of the prior art such as small number of IP location landmarks, low mining efficiency and limited functions, and achieve the purpose of greatly improving the number of IP location landmarks, mining efficiency and functional richness.

[0005] The present invention provides an IP location landmark mining system, comprising:

[0006] An uncontrollable landmark miner is used to retrieve a first candidate website set from the entire network using a search engine, and select a second candidate website set from the first candidate website set using a multi-classifier voting method; each candidate website in the candidate website set includes a suspected uncontrollable landmark, and the uncontrollable landmark is a landmark that is only accessible through the network;

[0007] A controllable landmark miner is used to classify the public probe website set using a semi-supervised classification method, and determine a candidate public probe website set based on the classification result; the candidate public probe websites in the candidate public probe website set include suspected controllable landmarks, and the controllable landmarks are landmarks that have both network accessibility and detection capabilities;

[0008] A landmark generator is used to search from the second candidate website set and the candidate public probe website set according to preset search conditions, and generate an IP positioning landmark set based on the search results.

[0009] According to an IP positioning landmark mining system provided by the present invention, the uncontrollable landmark mining device includes:

[0010] A search term construction module, used to extract a plurality of high-frequency feature words from a device website set including an uncontrollable landmark, and construct a search term based on the plurality of high-frequency feature words; the high-frequency feature words are feature words with a word frequency higher than a preset word frequency;

[0011] A first retrieval module, configured to input the search term into the search engine, and use the search engine to retrieve the first candidate website set from the entire network;

[0012] A classifier training module, used to extract a target feature vector of each device website sample in the device website sample set, and train a plurality of different classifiers based on the target feature vector;

[0013] A screening module is used to calculate, for each candidate website in the first candidate website set, the voting probability that the candidate website includes an uncontrollable landmark by using the trained multiple classifiers, calculate the target voting probability that the candidate website includes an uncontrollable landmark based on the voting weights and the voting probabilities corresponding to each of the trained multiple classifiers, and screen out the second candidate website set based on the target voting probabilities corresponding to each of the candidate websites in the first candidate website set.

[0014] According to an IP location landmark mining system provided by the present invention, the search term construction module includes:

[0015] A first feature extraction unit is used to input the device website set including the uncontrollable landmark into the TF-IDF model for processing, and obtain the high-frequency feature words in the URL, body and title of the device website;

[0016] The combining unit is used to combine the high-frequency characteristic words in the URL, body and title of the device website by using a full arrangement method to obtain the search word.

[0017] According to an IP location landmark mining system provided by the present invention, the classifier training module includes:

[0018] A second feature extraction unit is used to extract a feature vector of a URL, a body and a title of each device website sample in the device website sample set from the device website sample;

[0019] A concatenation unit, used for concatenating the feature vectors of the URL, body and title of the device website sample to obtain the target feature vector;

[0020] A training unit is used to characterize positive samples with the target feature vector of the device website samples including uncontrollable landmarks, and to characterize negative samples with the target feature vector of the device website samples not including uncontrollable landmarks, so as to train the plurality of classifiers.

[0021] According to an IP positioning landmark mining system provided by the present invention, the landmark generator includes:

[0022] A second retrieval module, configured to retrieve a plurality of landmark attributes from the second candidate website set and the candidate public probe website set;

[0023] A landmark determination module, used to determine a landmark having multiple landmark attributes as an IP positioning landmark;

[0024] The landmark generation module is used to generate the IP location landmark set including each of the IP location landmarks.

[0025] According to an IP positioning landmark mining system provided by the present invention, the plurality of landmark attributes include landmark type, device type, IP address and geographic location;

[0026] The second retrieval module is specifically used to retrieve the landmark type, the device type, the IP address and the geographic location in turn from the second candidate website set and the candidate public probe website set, and stop searching the geographic location if the retrieval of the landmark type, the device type and the IP address fails.

[0027] According to an IP location landmark mining system provided by the present invention, the second retrieval module includes:

[0028] A first retrieval unit is configured to retrieve a landmark type from each website in the second candidate website set and the candidate public probe website set, and generate a first website set based on each website of the retrieved landmark type;

[0029] A second retrieval unit is used to retrieve device types from each website in the first website collection, determine the device type with the largest number of occurrences as the device type of the landmark, and generate a second website collection based on the websites of the device type of the landmark retrieved;

[0030] A third retrieval unit is used to extract the IP address of the landmark using a regular expression for each website in the second website set, or extract the IP address of the landmark from the online video, audio or control stream of the website, and generate a third website set based on each website for which the IP address of the landmark is retrieved;

[0031] The fourth retrieval unit is used to extract the geographical location of the device for each website in the third website collection using regular expressions, or to extract the place name corresponding to the device using natural language processing, and obtain the geographical location corresponding to the place name through an online map application program interface.

[0032] The present invention also provides an IP location landmark mining method, comprising:

[0033] A search engine is used to retrieve a first candidate website set from the entire network, and a multi-classifier voting method is used to screen out a second candidate website set from the first candidate website set; each candidate website in the candidate website set includes a suspected uncontrollable landmark, and the uncontrollable landmark is a landmark that is only accessible through the network;

[0034] The public probe website set is classified by using a semi-supervised classification method, and a candidate public probe website set is determined based on the classification result; the candidate public probe websites in the candidate public probe website set include suspected controllable landmarks, and the controllable landmarks are landmarks that have both network accessibility and detection capabilities;

[0035] A search is performed from the second candidate website set and the candidate public probe website set according to preset search conditions, and an IP location landmark set is generated based on the search results.

[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the IP location landmark mining method as described above is implemented.

[0037] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the IP location landmark mining method as described above is implemented.

[0038] The present invention also provides a computer program product, including a computer program, which implements the IP location landmark mining method as described above when executed by a processor.

[0039] The present invention provides an IP positioning landmark mining system, method, electronic device and storage medium, the system includes an uncontrollable landmark mining device, a controllable landmark mining device and a landmark generator; wherein, the uncontrollable landmark mining device can use a search engine to retrieve a first candidate website set from the entire network, no longer limited to a specific data set, greatly improving the number of uncontrollable landmarks, and using a multi-classifier voting method to screen out a second candidate website set including suspected uncontrollable landmarks from the first candidate website set, and can screen out a second candidate website set including suspected uncontrollable landmarks from the huge first candidate website set, which can improve the mining efficiency of uncontrollable landmarks with only network accessibility; the controllable landmark mining device can classify the public probe website set using a semi-supervised classification method, and determine the candidate public probe website set including suspected controllable landmarks based on the classification results, greatly improving the number of uncontrollable landmarks with both network accessibility and detection capabilities; the landmark generator can perform a deep search from the second candidate website set and the candidate public probe website set according to the preset search conditions, and generate an IP positioning landmark set based on the search results, which can mine IP positioning landmarks with richer functions. Therefore, the present invention can greatly improve the number, mining efficiency and functional richness of IP positioning landmarks. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 It is a schematic diagram of the system architecture of the IP positioning landmark mining system provided by an embodiment of the present invention.

[0042] Figure 2 It is a schematic diagram of the feature extraction principle of the uncontrollable landmark miner provided by an embodiment of the present invention.

[0043] Figure 3 It is a flow chart of the IP location landmark mining method provided by an embodiment of the present invention.

[0044] Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] At present, conventional methods for obtaining IP location landmarks include: methods based on network content mining, methods based on online maps, and methods based on network measurement platforms. These three conventional methods are introduced below.

[0047] 1) Method based on web content mining: GeoCAM (geographic camera) regularly crawls the information of online webcams on manually selected portal websites. Since these online webcams usually disclose their IP addresses and geographic locations to provide functions such as sharing images, GeoCAM analyzes the structure and composition of these websites and uses natural language processing technology to extract the IP addresses and longitude and latitude coordinates of the webcams for large-scale generation of IP location landmarks.

[0048] This method only considers a specific range and number of web pages, resulting in a limited number of landmarks, and expansion can only rely on manual addition and annotation. In addition, the mining of geographic locations relies heavily on expert experience, which makes the recognition efficiency low. With the addition of new types of websites, the landmark extraction strategy will change greatly, which will eventually reduce the usability of the method. Finally, since the public nodes in the website only have connectivity, the functions of landmarks are very limited.

[0049] 2) Online map-based method: The target area is divided according to the size of the returned area in one query. At the center of each partitioned area, the online map searches for a list of institutions around the center based on their coordinates. In the returned list of institutions, for each institution item, the domain name is converted to an IP address and the geographic location is converted to longitude and latitude. The online map obtains street-level landmarks by associating IP addresses with longitude and latitude.

[0050] This method only considers a single source, online maps, and the number of landmarks is limited. In addition, with the addition of various privacy protection mechanisms in online maps, this landmark acquisition method has become increasingly limited. At the same time, this method also has the problem of limited landmark functions.

[0051] 3) Method based on network measurement platform: Common network measurement platforms usually provide some network nodes with geographical locations. These nodes are used to perform distributed network measurements and obtain IP location landmarks.

[0052] The number of landmarks obtained by this method is the least among the three methods. Due to the management and maintenance of the network measurement platform, this method is becoming increasingly difficult to use and cannot meet the needs of positioning.

[0053] Based on this, the present invention provides an IP positioning landmark mining system, method, electronic device and storage medium, which are described in detail below.

[0054] Combine the following Figure 1-Figure 2 Describe the IP location landmark mining system of the present invention.

[0055] Please refer to Figure 1 , Figure 1 Schematic diagram of the system architecture of the IP location landmark mining system provided by the embodiment of the present invention. Figure 1 As shown, the system may include: an uncontrollable landmark miner 1, a controllable landmark miner 2 and a landmark generator 3.

[0056] The uncontrollable landmark miner 1 is used to use a search engine to retrieve a first candidate website set from the entire network, and use a multi-classifier voting method to screen out a second candidate website set from the first candidate website set; each candidate website in the candidate website set includes suspected uncontrollable landmarks, and the uncontrollable landmark is a landmark that only has network accessibility.

[0057] Specifically, the functional type of IP-located landmarks is defined, and landmarks that are only accessible through the network are defined as "uncontrollable landmarks." First, a search engine is used to retrieve candidate websites that may include uncontrollable landmarks from the entire network, that is, candidate websites that include suspected uncontrollable landmarks. All retrieved candidate websites are classified into the first candidate website set, which is no longer restricted to a specific data set and has the ability to operate for a long time, greatly increasing the number of uncontrollable landmarks. Then, since the first candidate website set is relatively large, a multi-classifier voting method is used to screen out candidate websites that are most likely to include uncontrollable landmarks from the huge first candidate website set, that is, candidate websites that include suspected uncontrollable landmarks. All screened candidate websites are classified into the second candidate website set, which can improve the mining efficiency of uncontrollable landmarks that are only accessible through the network.

[0058] In one embodiment, the uncontrollable landmark miner 1 includes: a search term construction module, a first retrieval module, a classifier training module and a screening module.

[0059] A search term construction module is used to extract multiple high-frequency feature words from a device website set including uncontrollable landmarks, and construct search terms based on the multiple high-frequency feature words; high-frequency feature words are feature words with a frequency higher than a preset frequency.

[0060] Specifically, the device website set includes a small number of manually selected device websites including uncontrollable landmarks. Device websites refer to websites of network devices (such as online webcams, etc.). Device websites usually have unique text content and words, from which multiple high-frequency feature words with a frequency higher than a preset frequency are extracted, and these high-frequency feature words are selected to construct search terms.

[0061] In one embodiment, the search term construction module includes: a first feature extraction unit and a combination unit.

[0062] The first feature extraction unit is used to input the device website set including the uncontrollable landmark into a term frequency-inverse document frequency index (TF-IDF) model for processing to obtain high-frequency feature words in the URL, body and title of the device website.

[0063] Specifically, Figure 2 As shown, the URL, body, and title of the device website including uncontrollable landmarks are input into the TF-IDF model. The TF-IDF model can efficiently extract words unique to the device website, that is, high-frequency feature words in the URL, body, and title of the device website.

[0064] The combination unit is used to combine the high-frequency feature words in the URL, main body and title of the device website by using the full arrangement method to obtain the search word.

[0065] Specifically, Figure 2 As shown, the full permutation method is used to combine the high-frequency feature words in the URL, body and title of the device website output by the TF-IDF model to form the search terms for retrieving uncontrollable landmarks.

[0066] The first search module is used to input a search term into a search engine, and use the search engine to search for a first candidate website set from the entire network.

[0067] Specifically, the search terms are input into the search engine, and the search engine is used to retrieve candidate websites that may include uncontrollable landmarks from the entire network, that is, candidate websites that include suspected uncontrollable landmarks. All retrieved candidate websites are classified into the first candidate website set. It is no longer restricted to a specific data set and has the ability to operate for a long time, which greatly increases the number of uncontrollable landmarks.

[0068] The classifier training module is used to extract the target feature vector of each device website sample in the device website sample set, and train multiple different classifiers based on the target feature vector.

[0069] In one embodiment, the classifier training module includes: a second feature extraction unit, a splicing unit and a training unit.

[0070] The second feature extraction unit is used to extract the feature vector of the URL, body and title of the device website sample from each device website sample in the device website sample set.

[0071] Specifically, Figure 2 As shown, the URL, body and title of the device website sample are input into the Word2Vec (Word to Vector) model, and the feature vectors of the URL, body and title of the device website sample can be extracted respectively through the Word2Vec model. The Word2Vec model is a group of related models used to generate word vectors.

[0072] The concatenation unit is used to concatenate the feature vectors of the URL, body and title of the device website sample to obtain a target feature vector.

[0073] Specifically, Figure 2 As shown, the feature vectors of the URL, body and title of the device website sample are combined in sequence to obtain a high-dimensional vector, namely, the target feature vector of the device website sample.

[0074] The training unit is used to characterize positive samples with target feature vectors of device website samples including uncontrollable landmarks, and to characterize negative samples with target feature vectors of device website samples not including uncontrollable landmarks, so as to train multiple classifiers.

[0075] Exemplarily, the multiple classifiers may include: decision tree, random forest, support vector machine (SVM), XGBoost and LightGBM, but the present embodiment is not limited thereto.

[0076] The device website sample set includes positive samples and negative samples. The positive samples are manually selected device website samples that include uncontrollable landmarks, and the negative samples are device website samples that do not include uncontrollable landmarks in Google Hot 100. Both positive and negative samples are represented by the target feature vector of the device website sample. Five classifiers, including decision tree, random forest, support vector machine, XGBoost and LightGBM, are trained respectively.

[0077] The screening module is used to calculate the voting probability that the candidate website includes an uncontrollable landmark for each candidate website in the first candidate website set by using multiple trained classifiers, calculate the target voting probability that the candidate website includes an uncontrollable landmark based on the voting weights and voting probabilities corresponding to each of the multiple trained classifiers, and screen out the second candidate website set based on the target voting probabilities corresponding to each candidate website in the first candidate website set.

[0078] Specifically, for each candidate website in the first candidate website set, the target voting probability that the candidate website includes an uncontrollable landmark can be calculated by the following expression (1):

[0079] (1)

[0080] in, Indicates The classifiers calculate the voting probability that the candidate sites include uncontrollable landmarks. Indicates The voting weight of each classifier is represents the total number of classifiers, represents the target voting probability.

[0081] Exemplarily, the voting weight of the decision tree is 0.8933, the voting weight of the random forest is 0.9147, the voting weight of the support vector machine is 0.9652, the voting weight of XGBoost is 0.9532, and the voting weight of LightGBM is 0.9699, but this embodiment is not limited to this.

[0082] The target voting probabilities corresponding to the candidate websites in the first candidate website set are sorted from large to small, and the candidate websites with the highest sorting are used as the second candidate website set.

[0083] The controllable landmark miner 2 is used to classify the public probe website set using a semi-supervised classification method, and determine a candidate public probe website set based on the classification results; the candidate public probe websites in the candidate public probe website set include suspected controllable landmarks, and the controllable landmarks are landmarks that have both network accessibility and detection capabilities.

[0084] Specifically, the functional type of IP location landmarks is defined, and landmarks that have both network accessibility and detection capabilities are defined as "controllable landmarks". The public probe website set includes a series of public probe websites, a small number of which have geographic location tags in their Hypertext Markup Language (HTML) files. The URLs, bodies, and titles of the labeled public probe websites are input into the Word2Vec model. The Word2Vec model can extract the feature vectors of the URLs, bodies, and titles of the labeled public probe websites respectively, and the feature vectors of the URLs, bodies, and titles of the labeled public probe websites are combined to obtain a high-dimensional vector of the labeled public probe website to characterize the labeled public probe website. The public probe website set is classified using a semi-supervised classification method (such as PU-Bagging). During the classification process, a threshold is set to ensure that each unlabeled public probe website becomes a positive sample only when it has a high probability of containing geographic information. After the classification is completed, the candidate public probe website set including suspected controllable landmarks is determined based on the classification results.

[0085] The landmark generator 3 is used to search from the second candidate website set and the candidate public probe website set according to the preset search conditions, and generate an IP positioning landmark set based on the search results.

[0086] In one embodiment, the landmark generator 3 includes: a second retrieval module, a landmark determination module and a landmark generation module.

[0087] The second retrieval module is used to retrieve multiple landmark attributes from the second candidate website set and the candidate public probe website set.

[0088] Specifically, for each website in the second candidate website set and the candidate public probe website set, a plurality of landmark attributes are retrieved.

[0089] In one embodiment, multiple landmark attributes include landmark type, device type, IP address and geographic location; the second retrieval module is specifically used to: retrieve the landmark type, device type, IP address and geographic location in turn from the second candidate website set and the candidate public probe website set, and stop retrieving the geographic location if the retrieval of the landmark type, device type and IP address fails.

[0090] Specifically, from the second candidate website set that may include uncontrollable landmarks and the candidate public probe website set that may include controllable landmarks, the landmark type, device type, IP address and geographic location are retrieved in turn. Since the retrieval overhead of the geographic location is the largest, once the retrieval of the landmark type, device type and IP address fails, the retrieval of the geographic location is stopped immediately, which can reduce the retrieval overhead.

[0091] In one embodiment, the second retrieval module includes: a first retrieval unit, a second retrieval unit, a third retrieval unit and a fourth retrieval unit.

[0092] The first retrieval unit is used to retrieve a landmark type from each website in the second candidate website set and the candidate public probe website set, and generate a first website set based on each website with the retrieved landmark type.

[0093] Specifically, for each website in the second candidate website set and the candidate public probe website set, if the landmark attribute of the website includes the landmark type, the website is retained, and all the retained websites are used as the first website set.

[0094] The second retrieval unit is used to retrieve the device type from each website in the first website collection, determine the device type with the largest number of occurrences as the landmark device type, and generate a second website collection based on each website with the retrieved landmark device type.

[0095] Specifically, the device type can refer to the device type definition provided in the YouThings dataset. For each website in the first website set, the predefined device type is retrieved from the website, and the device type with the largest number of occurrences is determined as the device type of the landmark. If the landmark attribute of the website includes the device type, the website is retained, and all the retained websites are used as the second website set.

[0096] The third retrieval unit is used to extract the IP address of the landmark for each website in the second website collection using a regular expression, or to extract the IP address of the landmark from the online video, audio or control stream of the website, and generate a third website collection based on the websites for which the IP addresses of the landmarks are retrieved.

[0097] Specifically, for each website in the second website collection, if the HTML file of the website includes the IP address of the landmark, a regular expression is used to extract the IP address of the landmark. If the HTML file of the website does not include the IP address of the landmark, a data packet is captured from the online video, audio or control stream by the IP online streaming method and the destination IP address is extracted as the IP address of the landmark. If the landmark attribute of the website includes the IP address of the landmark, the website is retained, and all the retained websites are used as the third website collection.

[0098] The fourth retrieval unit is used to extract the geographical location of the device for each website in the third website collection using regular expressions, or to extract the place name corresponding to the device using natural language processing, and obtain the geographical location corresponding to the place name through an online map application program interface.

[0099] Specifically, for each website in the third website collection, if the HTML file of the website includes the geographical location (latitude and longitude coordinates) of the device, a regular expression is used to extract the geographical location of the device. If the HTML file of the website does not include the geographical location of the device, a natural language processing method is used to extract possible place names, and the geographical location of the device is obtained through an online map application program interface.

[0100] The landmark determination module is used to determine a landmark that has multiple landmark attributes as an IP positioning landmark.

[0101] Specifically, only a landmark that has multiple landmark attributes at the same time is considered to be a qualified IP positioning landmark.

[0102] The landmark generation module is used to generate an IP location landmark set including various IP location landmarks.

[0103] An IP positioning landmark mining system provided by an embodiment of the present invention includes an uncontrollable landmark mining device, a controllable landmark mining device and a landmark generator; wherein the uncontrollable landmark mining device can use a search engine to retrieve a first candidate website set from the entire network, and is no longer limited to a specific data set, which greatly increases the number of uncontrollable landmarks, and uses a multi-classifier voting method to screen out a second candidate website set including suspected uncontrollable landmarks from the first candidate website set, and can screen out a second candidate website set including suspected uncontrollable landmarks from a huge first candidate website set, which can improve the mining efficiency of uncontrollable landmarks that only have network accessibility; the controllable landmark mining device can be used to search for ... is no longer limited to a specific data set, which greatly increases the number of uncontrollable landmarks, and is no longer limited to a specific data set, which greatly increases the number of uncontrollable landmarks, and is no longer limited to a specific data set, which greatly increases the number of uncontrollable landmarks, and is no longer limited to a specific data set, which greatly increases the number of uncontrollable landmarks, and is no longer limited to a specific data set, which greatly increases the number of uncontrollable landmarks, and is no longer limited to a specific data set, which greatly increases the number of uncontrollable landmarks, and is no longer limited to a specific data set, which greatly increases the number of uncontrollable landmarks, and is no longer limited to a specific data The semi-supervised classification method can be used to classify the public probe website set, and the candidate public probe website set including suspected controllable landmarks can be determined based on the classification results, which greatly increases the number of uncontrollable landmarks that have both network accessibility and detection capabilities; the landmark generator can perform in-depth retrieval from the second candidate website set and the candidate public probe website set according to the preset retrieval conditions, and generate an IP positioning landmark set based on the retrieval results, which can mine IP positioning landmarks with richer functions, namely uncontrollable landmarks and controllable surfaces, breaking through the technical bottleneck of the single function of landmarks discovered by the prior art (only uncontrollable landmarks), and can greatly improve the accuracy of positioning services. Therefore, the embodiments of the present invention can greatly increase the number, mining efficiency and functional richness of IP positioning landmarks.

[0104] Furthermore, the embodiments of the present invention have extremely strong scalability and deployability. The entire process is highly automated and does not require manual intervention and expert experience in the feature engineering and landmark generation stages. It can run for a long time and provide a steady stream of multifunctional positioning landmarks for IP positioning services.

[0105] The IP location landmark mining method provided by the present invention is described below. The IP location landmark mining method described below and the IP location landmark mining system described above can be referenced to each other.

[0106] Please refer to Figure 3 , Figure 3 FIG. 1 is a flow chart of an IP location landmark mining method provided by an embodiment of the present invention. Figure 3 As shown, the method may include the following steps:

[0107] Step 301: using a search engine to retrieve a first candidate website set from the entire network, and using a multi-classifier voting method to screen out a second candidate website set from the first candidate website set; each candidate website in the candidate website set includes a suspected uncontrollable landmark, and the uncontrollable landmark is a landmark that only has network accessibility;

[0108] Step 302: classify the public probe website set using a semi-supervised classification method, and determine a candidate public probe website set based on the classification result; the candidate public probe websites in the candidate public probe website set include suspected controllable landmarks, and the controllable landmarks are landmarks that have both network accessibility and detection capabilities;

[0109] Step 303: Search from the second candidate website set and the candidate public probe website set according to the preset search conditions, and generate an IP location landmark set based on the search results.

[0110] In one embodiment, step 301 includes:

[0111] Step 3011: extract multiple high-frequency feature words from the device website set including the uncontrollable landmark, and construct search words based on the multiple high-frequency feature words; the high-frequency feature words are feature words with a word frequency higher than a preset word frequency;

[0112] Step 3012: input the search term into the search engine, and use the search engine to retrieve the first candidate website set from the entire network;

[0113] Step 3013: extracting a target feature vector of each device website sample in the device website sample set, and training a plurality of different classifiers based on the target feature vector;

[0114] Step 3014: For each candidate website in the first candidate website set, use the trained multiple classifiers to calculate the voting probability that the candidate website includes an uncontrollable landmark, calculate the target voting probability that the candidate website includes an uncontrollable landmark based on the voting weights and voting probabilities corresponding to the trained multiple classifiers, and screen out the second candidate website set based on the target voting probabilities corresponding to each candidate website in the first candidate website set.

[0115] In one embodiment, step 3011 includes:

[0116] The device website collection including uncontrollable landmarks is input into the TF-IDF model for processing to obtain high-frequency feature words in the URL, body and title of the device website;

[0117] The high-frequency feature words in the URL, main body and title of the equipment website are combined by full permutation method to obtain the search words.

[0118] In one embodiment, step 3013 includes:

[0119] For each device website sample in the device website sample set, extract a feature vector of the URL, body and title of the device website sample from the device website sample;

[0120] The feature vectors of the URL, body and title of the device website sample are concatenated to obtain the target feature vector;

[0121] Positive samples are represented by target feature vectors of device website samples including uncontrollable landmarks, and negative samples are represented by target feature vectors of device website samples excluding uncontrollable landmarks, and multiple classifiers are trained.

[0122] In one embodiment, step 303 includes:

[0123] Step 3031, searching for multiple landmark attributes from the second candidate website set and the candidate public probe website set;

[0124] Step 3032: determine a landmark with multiple landmark attributes as an IP positioning landmark;

[0125] Step 3033: Generate an IP location landmark set including each IP location landmark.

[0126] In one embodiment, the plurality of landmark attributes include landmark type, device type, IP address, and geographic location;

[0127] Step 3031 includes: searching landmark type, device type, IP address and geographic location in sequence from the second candidate website set and the candidate public probe website set, and stopping searching for geographic location when the retrieval of landmark type, device type and IP address fails.

[0128] In one embodiment, step 3031 includes:

[0129] For each website in the second candidate website set and the candidate public probe website set, retrieve a landmark type from the website, and generate a first website set based on each website of the retrieved landmark type;

[0130] For each website in the first website collection, retrieve the device type from the website, determine the device type with the most occurrences as the device type of the landmark, and generate a second website collection based on the websites of the device type of the landmark retrieved;

[0131] For each website in the second website set, a regular expression is used to extract the IP address of the landmark, or the IP address of the landmark is extracted from the online video, audio or control stream of the website, and a third website set is generated based on each website for which the IP address of the landmark is retrieved;

[0132] For each website in the third website collection, a regular expression is used to extract the geographical location of the device, or a natural language processing method is used to extract the place name corresponding to the device, and the geographical location corresponding to the place name is obtained through an online map application program interface.

[0133] An IP positioning landmark mining method provided by an embodiment of the present invention, first, a search engine is used to retrieve a first candidate website set from the entire network, which is no longer limited to a specific data set, greatly improving the number of uncontrollable landmarks, and a multi-classifier voting method is used to screen out a second candidate website set including suspected uncontrollable landmarks from the first candidate website set, and a second candidate website set including suspected uncontrollable landmarks can be screened out from the huge first candidate website set, which can improve the mining efficiency of uncontrollable landmarks with only network accessibility; then, a semi-supervised classification method is used to classify the public probe website set, and a candidate public probe website set including suspected controllable landmarks is determined based on the classification results, which greatly improves the number of uncontrollable landmarks with both network accessibility and detection capabilities; finally, a deep search is performed from the second candidate website set and the candidate public probe website set according to the preset search conditions, and an IP positioning landmark set is generated based on the search results, which can mine IP positioning landmarks with richer functions. Therefore, the embodiment of the present invention can greatly improve the number, mining efficiency and functional richness of IP positioning landmarks.

[0134] Please refer to Figure 4 , Figure 4 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the IP location landmark mining method, which includes: using a search engine to retrieve a first candidate website set from the entire network, and using a multi-classifier voting method to filter out a second candidate website set from the first candidate website set; each candidate website in the candidate website set includes a suspected uncontrollable landmark, and the uncontrollable landmark is a landmark that only has network accessibility; using a semi-supervised classification method to classify the public probe website set, and determining the candidate public probe website set based on the classification result; the candidate public probe website in the candidate public probe website set includes a suspected controllable landmark, and the controllable landmark is a landmark that has both network accessibility and detection capabilities; searching from the second candidate website set and the candidate public probe website set according to the preset search conditions, and generating an IP location landmark set based on the search results.

[0135] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0136] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the IP positioning landmark mining method provided by the above-mentioned method embodiments, which includes: using a search engine to retrieve a first candidate website set from the entire network, and using a multi-classifier voting method to screen out a second candidate website set from the first candidate website set; each candidate website in the candidate website set includes suspected uncontrollable landmarks, and uncontrollable landmarks are landmarks that only have network accessibility; using a semi-supervised classification method to classify the public probe website set, and determining the candidate public probe website set based on the classification results; the candidate public probe websites in the candidate public probe website set include suspected controllable landmarks, and controllable landmarks are landmarks that have both network accessibility and detection capabilities; searching from the second candidate website set and the candidate public probe website set according to preset search conditions, and generating an IP positioning landmark set based on the search results.

[0137] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the IP positioning landmark mining method provided by the above-mentioned method embodiments, the method comprising: using a search engine to retrieve a first candidate website set from the entire network, and using a multi-classifier voting method to screen out a second candidate website set from the first candidate website set; the candidate websites in each candidate website set include suspected uncontrollable landmarks, which are landmarks that only have network accessibility; using a semi-supervised classification method to classify the public probe website set, and determining the candidate public probe website set based on the classification results; the candidate public probe websites in the candidate public probe website set include suspected controllable landmarks, which are landmarks that have both network accessibility and detection capabilities; searching from the second candidate website set and the candidate public probe website set according to preset search conditions, and generating an IP positioning landmark set based on the search results.

[0138] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An IP location landmark mining system, characterized in that: include: An uncontrollable landmark miner is used to retrieve a first candidate website set from the entire network using a search engine, and select a second candidate website set from the first candidate website set using a multi-classifier voting method; each candidate website in the candidate website set includes a suspected uncontrollable landmark, and the uncontrollable landmark is a landmark that is only accessible through the network; A controllable landmark miner is used to classify the public probe website set using a semi-supervised classification method, and determine a candidate public probe website set based on the classification result; the candidate public probe websites in the candidate public probe website set include suspected controllable landmarks, and the controllable landmarks are landmarks that have both network accessibility and detection capabilities; A landmark generator, used to search from the second candidate website set and the candidate public probe website set according to a preset search condition, and generate an IP location landmark set based on the search results; Wherein, the uncontrollable landmark miner includes: A search term construction module, used to extract a plurality of high-frequency feature words from a device website set including an uncontrollable landmark, and construct a search term based on the plurality of high-frequency feature words; the high-frequency feature words are feature words with a word frequency higher than a preset word frequency; A first retrieval module, configured to input the search term into the search engine, and use the search engine to retrieve the first candidate website set from the entire network; A classifier training module, used to extract a target feature vector of each device website sample in the device website sample set, and train a plurality of different classifiers based on the target feature vector; A screening module is used to calculate, for each candidate website in the first candidate website set, the voting probability that the candidate website includes an uncontrollable landmark by using the trained multiple classifiers, calculate the target voting probability that the candidate website includes an uncontrollable landmark based on the voting weights and the voting probabilities corresponding to each of the trained multiple classifiers, and screen out the second candidate website set based on the target voting probabilities corresponding to each of the candidate websites in the first candidate website set.

2. The IP location landmark mining system according to claim 1, characterized in that: The search term building module includes: A first feature extraction unit is used to input the device website set including the uncontrollable landmark into the TF-IDF model for processing, and obtain the high-frequency feature words in the URL, body and title of the device website; The combining unit is used to combine the high-frequency characteristic words in the URL, body and title of the device website by using a full arrangement method to obtain the search word.

3. The IP location landmark mining system according to claim 1, characterized in that: The classifier training module comprises: A second feature extraction unit is used to extract a feature vector of a URL, a body and a title of each device website sample in the device website sample set from the device website sample; A concatenation unit, used for concatenating the feature vectors of the URL, body and title of the device website sample to obtain the target feature vector; A training unit is used to characterize positive samples with the target feature vector of the device website samples including uncontrollable landmarks, and to characterize negative samples with the target feature vector of the device website samples not including uncontrollable landmarks, so as to train the plurality of classifiers.

4. The IP location landmark mining system according to claim 1, characterized in that: The landmark generator includes: A second retrieval module, configured to retrieve a plurality of landmark attributes from the second candidate website set and the candidate public probe website set; A landmark determination module, used to determine a landmark having multiple landmark attributes as an IP positioning landmark; The landmark generation module is used to generate the IP location landmark set including each of the IP location landmarks.

5. The IP location landmark mining system according to claim 4, characterized in that: The plurality of said landmark attributes comprises a landmark type, a device type, an IP address and a geographic location; The second retrieval module is specifically used to retrieve the landmark type, the device type, the IP address and the geographic location in turn from the second candidate website set and the candidate public probe website set, and stop searching the geographic location if the retrieval of the landmark type, the device type and the IP address fails.

6. The IP location landmark mining system according to claim 5, characterized in that: The second retrieval module comprises: A first retrieval unit is configured to retrieve a landmark type from each website in the second candidate website set and the candidate public probe website set, and generate a first website set based on each website of the retrieved landmark type; A second retrieval unit is used to retrieve device types from each website in the first website collection, determine the device type with the largest number of occurrences as the device type of the landmark, and generate a second website collection based on the websites for which the landmark device types are retrieved; A third retrieval unit is used to extract the IP address of the landmark using a regular expression for each website in the second website set, or extract the IP address of the landmark from the online video, audio or control stream of the website, and generate a third website set based on each website for which the IP address of the landmark is retrieved; The fourth retrieval unit is used to extract the geographical location of the device for each website in the third website collection using regular expressions, or to extract the place name corresponding to the device using natural language processing, and obtain the geographical location corresponding to the place name through an online map application program interface.

7. A method for mining IP location landmarks, characterized in that: include: A search engine is used to retrieve a first candidate website set from the entire network, and a multi-classifier voting method is used to screen out a second candidate website set from the first candidate website set; each candidate website in the candidate website set includes a suspected uncontrollable landmark, and the uncontrollable landmark is a landmark that is only accessible through the network; The public probe website set is classified by using a semi-supervised classification method, and a candidate public probe website set is determined based on the classification result; the candidate public probe websites in the candidate public probe website set include suspected controllable landmarks, and the controllable landmarks are landmarks that have both network accessibility and detection capabilities; Searching from the second candidate website set and the candidate public probe website set according to a preset search condition, and generating an IP location landmark set based on the search results; The method of using a search engine to retrieve a first candidate website set from the entire network and using a multi-classifier voting method to select a second candidate website set from the first candidate website set includes: Extracting a plurality of high-frequency feature words from a collection of device websites including uncontrollable landmarks, and constructing search words based on the plurality of high-frequency feature words; the high-frequency feature words are feature words with a word frequency higher than a preset word frequency; Input the search term into the search engine, and use the search engine to retrieve the first candidate website set from the entire network; Extracting a target feature vector of each device website sample in the device website sample set, and training a plurality of different classifiers based on the target feature vector; For each candidate website in the first candidate website set, the voting probability that the candidate website includes an uncontrollable landmark is calculated using the trained multiple classifiers respectively, and the target voting probability that the candidate website includes an uncontrollable landmark is calculated based on the voting weights and the voting probabilities corresponding to each of the trained multiple classifiers, and the second candidate website set is screened out based on the target voting probabilities corresponding to each of the candidate websites in the first candidate website set.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the IP location landmark mining method as described in claim 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the IP location landmark mining method as described in claim 7 is implemented.

Images