Method, system and server for extracting useful information from HTTP data

Abstract

The present invention discloses a method, system and server for extracting useful information from HTTP data, wherein the method for extracting useful information from HTTP data includes the following steps: obtaining HTTP data and extracting the payload information of a message from a message of HTTP data; determining whether the payload information matches a filtering condition in a filtering library; if not, terminating the process; if so, parsing the information to be parsed in the payload information to obtain the corresponding terminal features to be identified; matching the extracted terminal features to be identified with the terminal feature library; when a match is determined, outputting the terminal model in the terminal feature group; when a match is determined, storing the payload information in a database. The present invention increases the way to obtain terminal information, makes up for the shortcomings of the existing methods that plain text information cannot be obtained or is invalid, and UA information cannot be accurately obtained and parsed, increases the way to identify terminal models, and is conducive to improving the terminal model recognition rate.

Description

Technical Field

[0001] The present invention relates to the field of network data analysis and utilization, in particular to a method, system and server for extracting useful information from HTTP data. Background Art

[0002] The DPI (deep packet inspection) platform can collect and store in real time the HTTP data of users when using the network. The HTTP data includes a large amount of useful information, such as terminal model information, mobile phone number information, etc. Therefore, it is necessary to extract useful information such as terminal model and mobile phone number from the HTTP data.

[0003] In the existing terminal model identification method, the terminal model identification is mainly realized by regular matching based on the plain text message information (device fingerprint information) generated by the user accessing the network. The plain text message information is, for example, DHCP HostName, DHCP Vendor, and UA (User-Agent) in the request packet; DHCP, Dynamic Host Configuration Protocol, is a protocol for automatically assigning IP addresses to devices in a local network. Among them, HostName (Option 12) is the host name of the device, which is a string used to identify the device in the network. Vendor (Option 60) is used to identify the device manufacturer of the DHCP client.

[0004] However, due to some encryption technologies, there are situations where the device fingerprint information cannot be obtained, resulting in the inability to identify the terminal model.

[0005] In the existing mobile phone number recognition methods, the mobile phone number is extracted from the HTTP GET message sent by the mobile network mainly by manually combing the mobile phone number feature information in the URL and the feature parameters of the URL, or by aggregating the sub-text data of each URL into a local file pool, uploading the number text data to the cloud distributed file system, and then using the Hadoop data warehouse tool for distributed computing to extract 11 consecutive digits similar to the mobile phone number.

[0006] However, although the former method can accurately extract mobile phone numbers, the efficiency of this method is low, and since the URL is unique and the characteristic parameters in the URLs of different websites may be different, it is impossible to achieve batch extraction of mobile phone numbers through this method. Although the latter method can achieve batch extraction of mobile phone numbers, various similar data can be matched in the URL according to the characteristic that the mobile phone number has 11 consecutive digits, such as the time value on the LINUX server, so the accuracy of this method is low. Summary of the invention

[0007] The purpose of the present invention is to solve the above problems existing in the prior art and to provide a method, system and server for extracting useful information from HTTP data.

[0008] The purpose of the present invention is achieved through the following technical solutions:

[0009] The method for extracting useful information from HTTP data includes the following steps:

[0010] S1, obtaining HTTP data and extracting payload information of the message from a message of the HTTP data;

[0011] S2, determining whether the payload information matches a filtering condition in the filtering library;

[0012] S3, when it is determined that the payload information does not match any filtering conditions in the filtering library, the process ends;

[0013] S4, when it is determined that the payload information matches a filtering condition in the filtering library, extracting the terminal feature to be identified from the terminal information parsed from the payload information;

[0014] S5, matching the extracted terminal features to be identified with a terminal feature library;

[0015] S6, when it is determined that the terminal feature to be identified matches a terminal feature group in the terminal feature library, outputting the terminal model in the terminal feature group;

[0016] S7: When it is determined that the terminal feature to be identified does not match any terminal feature group in the terminal feature library, the payload information is stored in a database.

[0017] Preferably, each filtering condition includes hex code position information and hex code;

[0018] In S2, the following steps are used to determine whether the payload information matches a filtering condition in the filtering library:

[0019] S21, determining a selected hex code in the hex data corresponding to the payload information according to the hex code position information in the filtering condition, and judging whether the selected hex code is the same as the hex code in the filtering condition;

[0020] S22, if not, determining that the payload information does not match the filtering condition;

[0021] S23: If yes, determine whether the payload information matches the filtering condition.

[0022] Preferably, the filtering condition also includes port information and protocol information corresponding to the payload information;

[0023] Before executing S21, determining whether the port information and protocol information corresponding to the payload information are consistent with the port information and protocol information in the filtering condition;

[0024] If not, determining that the payload information does not match the filtering condition;

[0025] If yes, execute S21.

[0026] Preferably, the filtering condition also includes field type information;

[0027] In S23, it is first determined whether the field type information in the parsed terminal information is consistent with the field type information corresponding to the filtering condition. If so, it is determined that the payload information is consistent with the filtering condition; otherwise, it is determined that the payload information is inconsistent with the filtering condition.

[0028] Preferably, in S4, the terminal information is information converted from the selected hex code.

[0029] Preferably, the method for extracting useful information from HTTP data further includes the following process:

[0030] S10, processing the acquired HTTP data to obtain a batch of feature combinations to be identified, wherein the feature combinations to be identified include a key field value to be identified and its corresponding host field value to be identified, a parameter type field value to be identified, and a value field value to be identified;

[0031] S20, matching each feature combination to be identified with a reliable feature library; the reliable feature library includes a set of valid feature combinations, each valid feature combination includes a key field value and its corresponding host field value and parameter type field value; each valid feature combination in the reliable feature library is a feature combination to be determined that meets the requirements after verification by screening parameters;

[0032] S30, when it is determined that a feature combination to be identified is the same as a valid feature combination in the reliable feature library, the value of the value field to be identified of the feature combination to be identified is determined as the mobile phone number corresponding to the feature combination to be identified.

[0033] Preferably, a set of the screening parameters includes segment library matching rate, terminal device distribution rate, household distribution rate, number effectiveness rate, single-number device coverage number and single-number area coverage number.

[0034] Preferably, when it is determined that the segment library matching rate of the value field value that meets the requirements corresponding to a feature combination to be determined is not less than 98%, the number efficiency is not less than 98%, the terminal device distribution rate is between 0.7-1.3, the household distribution rate is between 0.7-1.3, the single-number device coverage number is less than 10, and the single-number area coverage number is less than 5, the feature combination to be determined is determined to be a valid feature combination; otherwise, the feature combination to be determined is added to the invalid feature library.

[0035] A system for extracting useful information from HTTP data, including:

[0036] A data extraction unit, used to obtain HTTP data and extract payload information of the message from a message of the HTTP data;

[0037] A judgment unit, used to determine whether the payload information matches a filtering condition in a filtering library;

[0038] An ending unit, used for ending the process when it is determined that the payload information does not match any filtering conditions in the filtering library;

[0039] A feature extraction unit, configured to extract a feature of a terminal to be identified from the terminal information parsed from the payload information when it is determined that the payload information matches a filtering condition in the filtering library;

[0040] A matching unit, used for matching the extracted terminal feature to be identified with a terminal feature library;

[0041] A matching result output unit, configured to output a terminal model in a terminal feature group when it is determined that the terminal feature to be identified matches a terminal feature group in a terminal feature library;

[0042] The matching failure processing unit is used to store the payload information in a database when it is determined that the terminal feature to be identified does not match any terminal feature group in the terminal feature library.

[0043] The server comprises a memory and a processor, wherein the memory stores a program that can be processed by the processor, and when the program is executed, any of the above methods for extracting useful information from HTTP data is implemented.

[0044] The advantages of the technical solution of the present invention are mainly reflected in:

[0045] The present invention conducts in-depth research and analysis on the payload information in the message, and thus constructs a terminal feature library on this basis, and performs terminal model identification of newly acquired message data based on the terminal feature library, thereby increasing the ways of obtaining terminal information, making up for the shortcomings of the existing methods that plaintext information cannot be obtained or is invalid, and UA information cannot be accurately obtained and parsed, increasing the ways of terminal model identification, and being beneficial to improving the terminal model recognition rate.

[0046] The present invention can also identify feature combinations based on multiple aspects of HTTP data, and can obtain a reliable feature library with more feature combinations, which is beneficial to improving the extraction range and extraction ability of mobile phone numbers. It does not require manual combing of features, which reduces the difficulty of establishing a feature library. Moreover, when screening feature combinations, multiple screening parameters are used for selection, which can effectively improve the accuracy of the extracted feature combinations, thereby helping to improve the accuracy of mobile phone number extraction, and can extract multiple mobile phone numbers at one time, which is beneficial to improving the extraction rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the process of identifying the terminal model in the method for extracting useful information from HTTP data of the present invention;

[0048] Figure 2 is a process diagram of step S2 in the method of the present invention;

[0049] Figure 3 It is a schematic diagram of the process of identifying mobile phone numbers in the method for extracting useful information from HTTP data of the present invention. DETAILED DESCRIPTION

[0050] The purpose, advantages and features of the present invention will be illustrated and explained by the non-limiting description of the following preferred embodiments. These embodiments are only typical examples of the application of the technical solution of the present invention, and any technical solution formed by equivalent replacement or equivalent transformation falls within the scope of protection claimed by the present invention.

[0051] In the description of the scheme, it should be noted that the terms "center", "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience and simplification of description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0052] Example 1

[0053] The method for extracting useful information from HTTP data disclosed by the present invention is described below in conjunction with the accompanying drawings. Figure 1 As shown, it includes the following steps:

[0054] S1, obtaining HTTP data and extracting payload information of the message from a message of the HTTP data;

[0055] S2, determining whether the payload information matches a filtering condition in the filtering library;

[0056] S3, when it is determined that the payload information does not match any filtering conditions in the filtering library, the process ends;

[0057] S4, when it is determined that the payload information matches a filtering condition in the filtering library, extracting the terminal feature to be identified from the terminal information parsed from the payload information;

[0058] S5, matching the extracted terminal features to be identified with a terminal feature library;

[0059] S6, when it is determined that the terminal feature to be identified matches a terminal feature group in the terminal feature library, outputting the terminal model in the terminal feature group;

[0060] S7: When it is determined that the terminal feature to be identified does not match any terminal feature group in the terminal feature library, the payload information is stored in a database.

[0061] The method for extracting useful information from HTTP data is based on a predetermined filter library and a terminal feature library when performing terminal model identification. The filter library and the terminal feature library are determined according to the following process:

[0062] Historical traffic data is obtained through a known DPI platform, HTTP data is extracted from the obtained traffic data, batches of messages are obtained from the HTTP data, payload information (effective load) of each message is extracted, all the extracted payload information is decoded and converted into hex data, and then this batch of hex data is converted into corresponding characters according to the ASCII code table, and the characters include visible characters and invisible characters; then, whether each of the payload information includes terminal information related to the terminal model is determined according to the visible characters.

[0063] If the visible characters after parsing a payload message contain terminal information related to the terminal model, then the hex code corresponding to the terminal information of the payload message and the hex code position information of the hex code in the hex data corresponding to the payload message are determined as a filtering condition. Of course, the port information in the header data of the message corresponding to each payload message and the protocol information determined based on the port information can also be manually determined and stored together with the hex code position information and the hex code corresponding to the payload information as a filtering condition. The specific technology of manually determining the port information and the protocol information corresponding to the port information from the header data of the message is a known technology and will not be described in detail here.

[0064] The terminal information includes but is not limited to MAC address, manufacturer, terminal model, brand, software version number, hardware version number, ssid, device id and other fields. The terminal information after parsing different payload information will be different. The corresponding terminal model can be determined according to the terminal information, and the field that can uniquely reflect the terminal model can be selected from the terminal information and the determined field and the terminal model can be stored as an associated terminal feature group in the terminal feature library, or all terminal information and the determined terminal model can be stored as an associated terminal feature group in the terminal feature library.

[0065] Each of the filtering conditions may also include field type information. The field type information in each filtering condition is determined based on the terminal information parsed from each payload information. For example, the field type information is fields such as brand and manufacturer. The field type information of different filtering conditions may differ.

[0066] For example, the following payload information is available:

[0067] “=”.

[0068] After the payload information is converted into hex data by base64 decoding, and then the hex data is converted into corresponding characters according to the ASCII code table, the following terminal information is found:

[0069] MAC address: 5A:E3:B6:D3:42:D2;

[0070] Manufacturer: Fiberxxx;

[0071] Model:HGXXX.

[0072] Through manual analysis, it is determined that the hex code corresponding to the terminal information is 6F70743D31323026646174613D, and the hex code is the first 26 bits of the hex data converted by decoding the payload information.

[0073] Therefore, the filtering conditions determined according to the payload information may include the following information:

[0074] The location information of the hex code is the first 26 bits of the hex data converted from the payload information;

[0075] The hex code is 6F70743D31323026646174613D.

[0076] The field type information can be identified as Manufacturer: Fiberxxx.

[0077] Correspondingly, the model: HGXXX and the determined terminal model: FiberHome HGXXX fusion gateway can be extracted as an associated terminal feature group and added to the terminal feature library, or the manufacturer: Fiberxxx and model: HGXXX in the parsed terminal information can be extracted from the terminal model as an associated terminal feature group and added to the terminal feature library.

[0078] When there are several filtering conditions in the filtering library and several terminal features and corresponding terminal models in the terminal feature library, they can be used to identify the terminal model.

[0079] In S1, the technology of collecting traffic data in real time through the DPI platform, obtaining HTTP data from the traffic data, and extracting messages from the HTTP data and extracting payload information from the messages is a known technology, which is not an innovation of the present invention and will not be described here.

[0080] As attached Figure 2 As shown, in S2, whether the payload information matches a filtering condition in the filtering library is determined according to the following steps:

[0081] S21, determining a selected hex code in the hex data corresponding to the payload information according to the hex code position information in the filtering condition, and judging whether the selected hex code is the same as the hex code in the filtering condition;

[0082] S22, if not, determining that the payload information does not match the filtering condition;

[0083] S23: If yes, determine whether the payload information matches the filtering condition.

[0084] Of course, if the filtering condition also includes port information and protocol information, then after a message is obtained in real time, the port information is first determined according to the header data of the message, and then the corresponding protocol information is determined according to the port information and the relationship table between the port information and the protocol information. The determined port information and protocol information are the port information and protocol information corresponding to the payload information extracted from the message, and before executing S21, it is first determined whether the port information and protocol information corresponding to the payload information are consistent with the port information and protocol information in the filtering condition; if not, it is determined that the payload information does not meet the filtering condition. If so, S21 is executed.

[0085] Furthermore, when the filtering condition also includes field type information, the selected hex code can be first converted into characters, and the selected hex code is also converted into corresponding characters according to the ASCII code table, and the field type information corresponding to the selected hex code is extracted from the characters converted from the selected hex code according to the field type information of the filtering condition. Moreover, in S23, it is first determined whether the field type information corresponding to the selected hex code is consistent with the field type information corresponding to the filtering condition. If so, it is determined that the payload information is consistent with the filtering condition; otherwise, it is determined that the payload information is inconsistent with the filtering condition.

[0086] Still taking the payload information of the above example for terminal identification, after converting the payload information into hex data, it is determined that the selected hex code 6F70743D31323026646174613D is consistent with the hex code of the filtering condition according to the hex code position information of a filtering condition (the first 26 bits of the hex code), and then the selected hex code is parsed to obtain the following terminal information: MAC address: 5A:E3:B6:D3:42:D2, manufacturer: Fiberxxx, model: HGXXX.

[0087] Therefore, the manufacturer information is extracted from the parsed terminal information according to the field type information of the filtering condition, and the extracted manufacturer Fiberxxx is compared with the manufacturer in the filtering condition. If the two are determined to be consistent, it is determined that the payload information meets the filtering condition. Conversely, if the extracted manufacturer Fiberxxx is inconsistent with the manufacturer in the filtering condition, it is determined that the payload information does not meet the filtering condition.

[0088] In S4, the terminal information parsed from the payload information is the character converted from the selected hex code. All fields of the terminal information can be extracted as the terminal features to be identified, or the terminal features to be identified can be extracted from the parsed terminal information according to the terminal model and its associated terminal features determined by the payload information based on the filtering condition that the payload information meets.

[0089] In S5, when matching the terminal feature to be identified with the terminal feature library, the terminal feature to be identified is compared with the terminal feature groups in the terminal feature library in sequence.

[0090] When it is determined in the terminal feature library that all terminal features except the terminal model in a terminal feature group match the terminal feature to be identified, it is determined that the terminal feature to be identified matches the terminal feature group in the terminal feature library, thereby outputting the terminal model in the terminal feature group.

[0091] On the contrary, when it is determined that in the terminal feature library, at least one of the terminal features other than the terminal model in each terminal feature group does not match the terminal feature to be identified, it is determined that the terminal feature to be identified does not match the terminal feature library.

[0092] For example, in the terminal feature library, the terminal model of the terminal feature group: the terminal feature associated with the FiberHome HGXXX fusion gateway is model: HGXXX; if the terminal feature to be identified is manufacturer: Fiberxxx, model: HGXXX, then the terminal feature associated with the terminal model matches the terminal feature to be identified, then it is determined that the terminal feature to be identified matches a terminal feature group in the terminal feature library, and the terminal model is output: FiberHome HGXXX fusion gateway.

[0093] On the contrary, in the terminal feature library, the terminal model of the terminal feature group: the terminal feature associated with the Fenghuo HGXXX fusion gateway is model: HGXXX; if the terminal feature to be identified includes model: HG, at this time, it is determined that the terminal feature associated with the terminal model does not match the terminal feature to be identified, then it is determined that the terminal feature to be identified does not match the terminal feature group.

[0094] In S7, the data in the database can be used for subsequent updating of the filter library and the terminal feature library. The specific updating process refers to the above-mentioned process of determining the filter library and the terminal feature library, which will not be described in detail here.

[0095] Furthermore, the method for extracting useful information from HTTP data can be used to identify not only the terminal model but also the mobile phone number. Figure 3As shown, the method for extracting useful information from HTTP data also includes the following process:

[0096] S10, processing the acquired HTTP data to obtain a batch of feature combinations to be identified, wherein the feature combinations to be identified include a key field value to be identified and its corresponding host field value to be identified, a parameter type field value to be identified, and a value field value to be identified;

[0097] S20, matching each feature combination to be identified with a reliable feature library; the reliable feature library includes a set of valid feature combinations, each valid feature combination includes a key field value and its corresponding host field value and a parameter type field value; the parameter type field includes Cookie, URL, Referer, and each valid feature combination in the reliable feature library is a feature combination to be determined that meets the requirements after verification through screening parameters;

[0098] S30, when it is determined that a feature combination to be identified is the same as a valid feature combination in the reliable feature library, the value of the value field to be identified of the feature combination to be identified is determined as the mobile phone number corresponding to the feature combination to be identified.

[0099] The reliable feature library is determined based on the acquired historical HTTP data:

[0100] As attached Figure 2 As shown, historical HTTP data for a period of time is obtained. The period of time can be selected as needed, preferably 1 week.

[0101] The historical HTTP data acquired over a period of time is automatically formatted through software programming to obtain a feature combination table, which includes at least a host (host name) field, a parameter type field, a key field, and a value field, etc. The parameter type field includes Cookie, URL, Referer, and the value field contains at least 11 consecutive digits. Table 1 provides an example of a partial record of a feature combination table.

[0102] Table 1:

[0103]

[0104] Each row in the feature combination table is defined as a record. When extracting each record from the historical HTTP data, the value field value containing at least 11 consecutive digits is extracted through regular matching. Each extracted value field value containing at least 11 consecutive digits is defined as a value field value that meets the requirements. All value field values ​​that meet the requirements and their corresponding key field values, host field values, and parameter type field values ​​are added to the feature combination table.

[0105] The key field value, host field value, and parameter type field value of each record are defined as a feature combination to be determined. A value field value that meets the requirements corresponds to a feature combination to be determined, and a feature combination to be determined can correspond to multiple different value field values ​​that meet the requirements.

[0106] Subsequently, the effective feature combination for extracting the mobile phone number can be determined based on the value field value that meets the requirements corresponding to each feature combination to be determined. When determining the effective feature combination for extracting the mobile phone number, it is specifically determined whether the value field value that meets the requirements corresponding to each feature combination to be determined meets a set of screening parameters, and the screening parameters include the number segment library matching rate, terminal device distribution rate, household distribution rate, number effectiveness rate, single-number device coverage number and single-number area coverage number.

[0107] (1) The number segment library matching rate is the ratio of the number of value field values ​​that meet the requirements and are successfully matched with the mobile phone number segment library among all value field values ​​that meet the requirements extracted under a feature combination to be determined to the total number of value field values ​​that meet the requirements extracted under the feature combination to be determined.

[0108] When all the value field values ​​that meet the requirements extracted under a combination of features to be determined are matched with the mobile phone number segment library, the first 7 digits (counting from left to right) of each value field value that meets the requirements are matched with the first 7 digits of the mobile phone number in the mobile phone number segment library. The mobile phone number segment library can be obtained in advance and stored in the system for call. The first 7 digits of each mobile phone number reflect the operator and province and city information of a mobile phone number, which can effectively reflect the possibility that a value field value that meets the requirements is a mobile phone number. When the first 7 digits of the value field value that meets the requirements are the same as the first 7 digits of a mobile phone number in the mobile phone number segment library, it is considered that the extracted value field value that meets the requirements is matched successfully.

[0109] The higher the matching rate of the number segment library, the higher the possibility that the value field value corresponding to the feature combination is a mobile phone number. Conversely, the lower the matching rate of the number segment library, the lower the possibility that the value field value corresponding to the feature combination is a mobile phone number. For example, the number of value field values ​​that meet the requirements extracted under a feature combination to be determined is 1000, but the number of value field values ​​that meet the requirements and are successfully matched with the mobile phone number segment library is 10, then the number segment library matching rate is 0.01, which means that the possibility that the value field value that meets the requirements extracted under this feature combination is a mobile phone number is very low. Therefore, such a feature combination to be determined cannot be used for mobile phone number extraction.

[0110] (2) The terminal device distribution rate reflects how many terminal devices all the value field values ​​that meet the requirements extracted from a feature combination to be determined appear on within the period of time. It is the ratio of the number of all the value field values ​​that meet the requirements extracted from a feature combination to be determined to the number of terminal devices on which the feature combination to be determined appears.

[0111] Since a mobile phone number usually does not appear in multiple terminal devices, the higher the terminal device distribution rate, the smaller the possibility that the value field value that meets the requirements corresponding to the feature combination to be determined is a mobile phone number. Conversely, the lower the terminal device distribution rate, the higher the possibility that the value field value that meets the requirements corresponding to the feature combination to be determined is a mobile phone number.

[0112] (3) The household distribution rate reflects how many households all the value field values ​​that meet the requirements extracted from a feature combination to be determined appear in the period of time. It is the ratio of the number of all the value field values ​​that meet the requirements extracted from a feature combination to be determined to the number of home gateways where the feature combination to be determined appears.

[0113] Usually, a mobile phone number will not appear in multiple households within a period of time. Therefore, the higher the household distribution rate, the less likely it is that the value field value that meets the requirements for the feature combination to be determined is a mobile phone number. Conversely, the lower the household distribution rate, the more likely it is that the value field value that meets the requirements for the feature combination to be determined is a mobile phone number.

[0114] (4) The number efficiency is the ratio of the number of qualified value field values ​​that meet the mobile phone number regularity to the total number of qualified value field values ​​extracted under a combination of features to be determined.

[0115] The higher the number efficiency rate, the higher the possibility that the value field value that meets the requirements and corresponds to the feature combination to be determined is a mobile phone number; conversely, the lower the number efficiency rate, the lower the possibility that the value field value that meets the requirements and corresponds to the feature combination to be determined is a mobile phone number.

[0116] (5) The single-number device coverage number is the maximum value of each value field value that meets the requirements and is extracted from a feature combination to be determined and appears in the terminal devices within the period of time. The number of terminal devices in which each value field value that meets the requirements appears in the period of time is specifically obtained by determining the number of terminal MAC addresses in which each value field value that meets the requirements appears.

[0117] The larger the number of single-number device coverage, the lower the possibility that the value field value that meets the requirements of the feature combination to be determined is a mobile phone number. Conversely, the smaller the number of single-number device coverage, the higher the possibility that the value field value that meets the requirements of the feature combination to be determined is a mobile phone number. Taking into account the impact of one-to-many scenarios such as random MAC addresses and router hanging (connecting one router to another router or switch), when the single-number device coverage is less than 10, the possibility that the value field value that meets the requirements of the feature combination to be determined is a mobile phone number is higher.

[0118] (6) The single number area coverage number reflects the maximum value of each value field value that meets the requirements extracted by a feature combination to be determined and appears in the area within the period of time. The area where each value field value that meets the requirements appears can be determined according to the following process:

[0119] First, determine from which gateway device the record to which the value field value that meets the requirements belongs obtains the data;

[0120] Then, the area where the gateway device is located is determined according to the IP information of the gateway device as the area where the value field value that meets the requirements appears.

[0121] The area is generally a province and / or city. The larger the single-number area coverage number, the lower the possibility that the value field value that meets the requirements of the feature combination to be determined is a mobile phone number. Conversely, the smaller the single-number area coverage number, the higher the possibility that the value field value that meets the requirements of the feature combination to be determined is a mobile phone number. Taking into account the impact of one-to-many scenarios such as random MAC and router hanging, when the single-number area coverage number is less than 5, the possibility that the value field value that meets the requirements of the feature combination to be determined is a mobile phone number is higher.

[0122] By determining whether the value field value that meets the requirements corresponding to each feature combination to be determined meets the above screening parameters, it is determined whether the feature combination to be determined can be used to extract the mobile phone number. Specifically, if the segment library matching rate of the value field value that meets the requirements corresponding to a feature combination to be determined is not less than 98%, the number efficiency is not less than 98%, the terminal device distribution rate is between 0.7-1.3, the family distribution rate is between 0.7-1.3, the single-number device coverage number is less than 10, and the single-number area coverage number is less than 5, then the possibility that the vlue parameter value corresponding to the feature combination to be determined is a mobile phone number is high, so the feature combination to be determined is stored in the reliable feature library as a valid feature combination, and at the same time, the extraction time of each valid feature combination will be recorded in the reliable feature library; on the contrary, when at least one of the screening parameters corresponding to a feature combination to be determined does not meet the above requirements, the possibility that the value field value that meets the requirements extracted by its corresponding value parameter value is a mobile phone number is low, so the feature combination to be determined is stored in the invalid feature library as an invalid feature combination, and the extraction time of the invalid feature combination is recorded at the same time. Even better, the number segment library matching rate is not less than 99% and the number efficiency is not less than 99%.

[0123] After obtaining the reliable feature library, the reliable feature library can be used to extract the mobile phone number to be extracted from the HTTP data of the mobile phone number to be extracted. In S10, each key field value containing at least 11 digits is extracted from the HTTP data obtained in real time or the new HTTP data for a period of time according to the above-mentioned method of extracting each record as a key field value to be identified, and the host field value, parameter type field value, and value field value corresponding to each key field value containing at least 11 digits are used as a feature combination to be identified.

[0124] As attached Figure 3 As shown, after determining that all feature combinations to be identified are matched with the reliable feature library, the mobile phone number corresponding to the extracted feature combination to be identified is output. Of course, after extracting the mobile phone number corresponding to the feature combination to be identified, the mobile phone number corresponding to the extracted feature combination to be identified can be matched to determine whether there are repeated mobile phone numbers. If so, when outputting the mobile phone number, only the number of times it appears is output.

[0125] Furthermore, after extracting the mobile phone number corresponding to the feature combination to be identified, the mobile phone number corresponding to the feature combination to be identified can also be compared with the mobile phone number segment library. If it is determined that the mobile phone number corresponding to the feature combination to be identified matches a mobile phone number in the mobile phone number segment library, the mobile phone number corresponding to the feature combination to be identified is output; if it is determined that the mobile phone number corresponding to the feature combination to be identified does not match a mobile phone number in the mobile phone number segment library, the mobile phone number corresponding to the feature combination to be identified is not output. The reason for this operation is that there may be a situation where the user enters an incorrect mobile phone number, which can effectively ensure the accuracy of the output mobile phone number.

[0126] Furthermore, during use, in order to ensure the reliability of the reliable feature library, it is necessary to iteratively update the reliable feature library and the invalid feature library. Specifically, the latest HTTP data obtained is used to verify whether each valid feature combination in the reliable feature library is still valid and whether each invalid feature combination in the invalid feature library is still invalid. For example, the newly collected HTTP data every day, week or month can be used for verification.

[0127] When newly acquired HTTP data is used to verify a valid feature combination in a reliable feature library, a value field value that meets the requirements and corresponds to the valid feature combination is extracted from the latest acquired HTTP data, and the newly extracted value field value that meets the requirements is combined with the previously extracted value field value that meets the requirements to re-determine whether it meets the above-mentioned screening parameters, thereby determining whether a valid feature combination still meets the requirements. If the newly extracted and previously extracted value field values ​​that meet the requirements corresponding to a valid feature combination still meet the requirements after re-verification with the screening parameters, it is determined that the valid feature combination is still valid and the valid feature combination is retained in the reliable feature library. Conversely, if the newly extracted and previously extracted value field values ​​that meet the requirements corresponding to a valid feature combination do not meet the requirements after re-verification with the screening parameters, it is determined that the valid feature combination is invalid and the valid feature combination is transferred to the invalid feature library as an invalid feature combination.

[0128] When newly acquired HTTP data is used to check an invalid feature combination in the invalid feature library, a value field value that meets the requirements and corresponds to the invalid feature combination is extracted from the latest acquired HTTP data, and the newly extracted value field value that meets the requirements is combined with the previously extracted value field value that meets the requirements to re-determine whether it meets the above-mentioned screening parameters, thereby determining whether an invalid feature combination still does not meet the requirements. If the newly extracted and previously extracted value field values ​​that meet the requirements corresponding to an invalid feature combination still do not meet the requirements after re-verification with the screening parameters, it is determined that the invalid feature combination is still invalid and the feature combination is retained in the invalid feature library; conversely, if the newly extracted and previously extracted value field values ​​that meet the requirements corresponding to an invalid feature combination meet the requirements after re-verification with the screening parameters, it is determined that the invalid feature combination is valid and the invalid feature combination is transferred to the reliable feature library as a valid feature combination.

[0129] In order to distinguish valid feature combinations from invalid feature combinations, a valid state label is configured for each valid feature combination, and an invalid state label is configured for each invalid feature combination. For example, the valid state label can be set to 1, and the invalid state label can be set to 0. In addition, the verification time of each feature combination is recorded during each verification.

[0130] Example 2

[0131] This embodiment discloses a system for extracting useful information from HTTP data, including:

[0132] A data extraction unit, used to obtain HTTP data and extract payload information of the message from a message of the HTTP data;

[0133] A judgment unit, used to determine whether the payload information matches a filtering condition in a filtering library;

[0134] An ending unit, used for ending the process when it is determined that the payload information does not match any filtering conditions in the filtering library;

[0135] A feature extraction unit, configured to extract a feature of a terminal to be identified from the terminal information parsed from the payload information when it is determined that the payload information matches a filtering condition in the filtering library;

[0136] A matching unit, used for matching the extracted terminal feature to be identified with a terminal feature library;

[0137] A matching result output unit, configured to output a terminal model in a terminal feature group when it is determined that the terminal feature to be identified matches a terminal feature group in a terminal feature library;

[0138] The matching failure processing unit is used to store the payload information in a database when it is determined that the terminal feature to be identified does not match any terminal feature group in the terminal feature library.

[0139] Example 3

[0140] This embodiment discloses a server, including a memory and a processor, wherein the memory stores a program that can be processed by the processor, and when the program is executed, the method for extracting useful information from HTTP data as described above is implemented.

[0141] There are many implementation methods of the present invention, and all technical solutions formed by equivalent transformation or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A method for extracting useful information from HTTP data, characterized in that: The steps include: S1, obtain HTTP data and extract the payload information of the message from a message of HTTP data; S2, determining whether the payload information matches a filtering condition in the filtering library; S3, when it is determined that the payload information does not match any filtering conditions in the filtering library, the process ends; S4, when it is determined that the payload information matches a filtering condition in the filtering library, extracting the terminal feature to be identified from the terminal information parsed from the payload information; S5, matching the extracted terminal features to be identified with a terminal feature library; S6, when it is determined that the terminal feature to be identified matches a terminal feature group in the terminal feature library, outputting the terminal model in the terminal feature group; S7: When it is determined that the terminal feature to be identified does not match any terminal feature group in the terminal feature library, the payload information is stored in a database.

2. The method for extracting useful information from HTTP data according to claim 1, characterized in that: Each filter condition includes hex code location information and hex code; In S2, the following steps are used to determine whether the payload information matches a filtering condition in the filtering library: S21, determining a selected hex code in the hex data corresponding to the payload information according to the hex code position information in the filtering condition, and judging whether the selected hex code is the same as the hex code in the filtering condition; S22, if not, determining that the payload information does not match the filtering condition; S23: If yes, determine whether the payload information matches the filtering condition.

3. The method for extracting useful information from HTTP data according to claim 2, characterized in that: The filtering condition also includes port information and protocol information corresponding to the payload information; Before executing S21, determining whether the port information and protocol information corresponding to the payload information are consistent with the port information and protocol information in the filtering condition; If not, determining that the payload information does not match the filtering condition; If yes, execute S21.

4. The method for extracting useful information from HTTP data according to claim 2, characterized in that: The filtering condition also includes field type information; In S23, first determine whether the field type information in the parsed terminal information is consistent with the field type information corresponding to the filtering condition, and if so, determine whether the payload information is consistent with the filtering condition; Otherwise, it is determined that the payload information does not match the filtering condition.

5. The method for extracting useful information from HTTP data according to claim 2, characterized in that: In the above S4, the terminal information is information converted from the selected hex code.

6. The method for extracting useful information from HTTP data according to any one of claims 1 to 5, characterized in that: It also includes the following processes: S10, processing the acquired HTTP data to obtain a batch of feature combinations to be identified, wherein the feature combinations to be identified include a key field value to be identified and its corresponding host field value to be identified, a parameter type field value to be identified, and a value field value to be identified; S20, matching each feature combination to be identified with a reliable feature library; the reliable feature library includes a set of valid feature combinations, each valid feature combination includes a key field value and its corresponding host field value and parameter type field value; each valid feature combination in the reliable feature library is a feature combination to be determined that meets the requirements after verification by screening parameters; S30, when it is determined that a feature combination to be identified is the same as a valid feature combination in the reliable feature library, the value of the value field to be identified of the feature combination to be identified is determined as the mobile phone number corresponding to the feature combination to be identified.

7. The method for extracting useful information from HTTP data according to claim 6, characterized in that: A set of screening parameters includes number segment library matching rate, terminal device distribution rate, household distribution rate, number effectiveness rate, single number device coverage number and single number area coverage number.

8. The method for extracting useful information from HTTP data according to claim 7, characterized in that: When it is determined that the segment library matching rate of the value field value that meets the requirements corresponding to a feature combination to be determined is not less than 98%, the number efficiency is not less than 98%, the terminal device distribution rate is between 0.7-1.3, the household distribution rate is between 0.7-1.3, the single-number device coverage number is less than 10, and the single-number area coverage number is less than 5, the feature combination to be determined is determined to be a valid feature combination; otherwise, the feature combination to be determined is added to the invalid feature library.

9. A system for extracting useful information from HTTP data, characterized in that: include: A data extraction unit for obtaining HTTP data and extracting payload information of the message from a message of HTTP data; A judgment unit, used to determine whether the payload information matches a filtering condition in a filtering library; An ending unit, used for ending the process when it is determined that the payload information does not match any filtering conditions in the filtering library; A feature extraction unit, configured to extract a feature of a terminal to be identified from the terminal information parsed from the payload information when it is determined that the payload information matches a filtering condition in the filtering library; A matching unit, used for matching the extracted terminal feature to be identified with a terminal feature library; A matching result output unit, configured to output a terminal model in a terminal feature group when it is determined that the terminal feature to be identified matches a terminal feature group in a terminal feature library; The matching failure processing unit is used to store the payload information in a database when it is determined that the terminal feature to be identified does not match any terminal feature group in the terminal feature library.

10. A server, comprising a memory and a processor, wherein the memory stores a program that can be processed by the processor, wherein: When the program is executed, the method for extracting useful information from HTTP data as described in any one of claims 1-8 is implemented.

Images