A method, device, equipment and readable medium for querying network traffic information

Abstract

The present invention discloses a network traffic information query method, device, equipment and readable medium, wherein the method includes the following steps: obtaining network traffic index data and formatting the index data to generate a data field in a fixed-length format; storing the data field in columnar form to obtain multiple arrangement fields; writing an inverted index file for the multiple arrangement fields to construct an inverted index, and writing at least one index feature value for a specified field in the multiple arrangement fields; selecting a query field in the arrangement field containing the index feature value, setting a filter condition based on the query field, and performing a data query based on the filter condition. The storage method provided by the present invention can not only read valid data fields in a targeted manner according to the query field, reduce disk operations, and reduce system resource consumption; but also can quickly locate data records that meet the filter conditions through field indexes and feature value indexes, thereby improving query performance and reducing response time.

Description

Technical Field

[0001] The present invention relates to the technical field of computer networks, and particularly to a method, device, equipment and readable medium for querying network traffic information. Background Art

[0002] With the development of computer network technology and the rapid expansion of the Internet scale, the storage of network traffic information data has become increasingly complex. Currently, computers usually use databases to store network traffic information data. For example, the database storage systems used in the prior art are general systems and do not have targeted design and optimization for the data characteristics of network traffic. Therefore, they have disadvantages such as low storage efficiency and slow query speed, which are infinitely magnified as the amount of data increases. The data storage problem has become a bottleneck for improving the performance of the entire system. Especially in some scenarios with extremely high performance requirements, it is often necessary to increase physical devices to expand processing capabilities, which not only introduces more procurement costs, but also more physical devices and more complex deployment methods pose very severe challenges to deployment and operation and maintenance personnel.

[0003] Therefore, there is still a need to improve the methods for storing and querying computer network traffic information in the prior art. Summary of the Invention

[0004] In view of this, an object of an embodiment of the present invention is to provide a method, device, equipment and readable medium for querying network traffic information.

[0005] Based on the above object, on the one hand, an embodiment of the present invention provides a method for querying network traffic information, including the following steps:

[0006] Obtain network traffic metric data and format the metric data to generate data fields in a fixed-length format;

[0007] Store the data fields in columns to obtain multiple arranged fields;

[0008] Write an inverted index file for the multiple arranged fields to construct an inverted index, and write at least one index feature value for a specified field in the multiple arranged fields;

[0009] Select a query field in the arranged fields containing index feature values, set a filtering condition based on the query field, and perform data query based on the filtering condition.

[0010] In some embodiments, obtaining network traffic metric data and formatting the metric data to generate data fields in a fixed-length format includes:

[0011] Obtain network traffic metric data, store the metric data as a file package, where the file package is updated and created at fixed time intervals and the name of the file package is attached with a timestamp;

[0012] Format the metric data in the file package to generate data fields in a fixed-length format.

[0013] In some embodiments, when the metric data has not been used for an overdue period, delete the metric data and / or compress the metric data.

[0014] In some embodiments, perform columnar storage on the data fields to obtain multiple arranged fields including:

[0015] Perform binary processing on the data fields and perform columnar storage to obtain multiple arranged fields, and each arranged field is named in the field order.

[0016] In some embodiments, the index feature values include values greater than a first threshold and / or less than a second threshold and / or equal to a third threshold.

[0017] In some embodiments, the query methods of the query include non-index feature value query and index feature value query.

[0018] In some embodiments, the query method of the index feature value query includes:

[0019] Index the index feature values in the arranged fields based on the filtering conditions, read the index results, and aggregate the index results into a filtered result set.

[0020] On the other hand, the present invention also provides a network traffic information query device, including:

[0021] A data processing module, configured to obtain network traffic metric data and format the metric data to generate data fields in a fixed-length format;

[0022] A data storage module, configured to perform columnar storage on the data fields to obtain multiple arranged fields;

[0023] An index construction module, configured to write an inverted index file for the multiple arranged fields to construct an inverted index and write at least one index feature value for a specified field in the multiple arranged fields;

[0024] A data query module, configured to select a query field from the index feature values, set a filtering condition based on the query field, and perform data query based on the filtering condition.

[0025] In another aspect of the embodiments of the present invention, there is also provided a computer device, including: at least one processor; and a memory storing computer instructions executable on the processor, and when the instructions are executed by the processor, the steps of the above method are implemented.

[0026] In another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above method.

[0027] The present invention has at least the following beneficial technical effects:

[0028] The present invention is a storage and query method designed for network traffic data. Various index data formats of network traffic are formatted into a unified fixed-length format, and each field is then written into a storage file in a columnar storage manner, and an index is established for the specified field. This method can not only read valid data fields targeted according to the query field, reduce disk operations, and reduce system resource consumption; but also quickly locate data records that meet the filtering conditions through field indexes and eigenvalue indexes, improve query performance, and reduce response time. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention, and for those of ordinary skill in the art, other embodiments can be obtained based on these drawings without creative efforts.

[0030] Figure 1 It is a schematic diagram of the network traffic information query method provided by the embodiments of the present invention;

[0031] Figure 2 It is a schematic diagram of columnar storage provided by the embodiments of the present invention;

[0032] Figure 3 It is a schematic diagram of one-dimensional index eigenvalue provided by the embodiments of the present invention;

[0033] Figure 4 It is a schematic diagram of multi-dimensional index eigenvalue provided by the embodiments of the present invention;

[0034] Figure 5 It is a schematic diagram of eigenvalue index query provided by the embodiments of the present invention;

[0035] Figure 6 It is a flowchart of the network traffic information query method provided by the embodiments of the present invention;

[0036] Figure 7Schematic diagram of the network traffic information query device provided by the embodiment of the present invention. Detailed implementation manners

[0037] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the following further elaborates on the embodiments of the present invention in detail with reference to specific embodiments and the accompanying drawings.

[0038] It should be noted that the specific features and advantages of the present invention will be described below by way of examples. However, all descriptions are only for the purpose of explanation and should not be construed as any limitation to the present invention. In addition, any single technical feature described or implied in the embodiments mentioned in this article, or any single technical feature shown or implied in the accompanying drawings, can still be arbitrarily combined or deleted between these technical features (or their equivalents), so as to obtain more other embodiments of the present invention that may not be directly mentioned in this article.

[0039] In the first aspect of the embodiment of the present invention, an embodiment of the network information traffic query method is proposed. Figure 1 The figure shows a schematic diagram of the network information traffic storage method provided by the embodiment of the present invention. As Figure 1 shown, the network information traffic storage method of the embodiment of the present invention includes the following steps:

[0040] S1. Obtain network traffic metric data and format the metric data to generate data fields in a fixed-length format;

[0041] S2. Store the data fields column by column to obtain a plurality of arranged fields;

[0042] S3. Write an inverted index file for the plurality of arranged fields to construct an inverted index, and write at least one index feature value for a specified field in the plurality of arranged fields;

[0043] S4. Select a query field from the arranged fields containing index feature values, set a filtering condition based on the query field, and perform data query based on the filtering condition.

[0044] Among them, the network traffic metric data obtained in S1 is numerical data such as packet loss rate, time, and number of data packets. As Figure 2 shown in the schematic diagram of columnar storage provided in this embodiment, Filed 1 is the original traffic metric data obtained. By writing the field values into a file in binary format, the numerical values of data fields in a fixed-length format are generated, and at the same time, storage space can be saved

[0045] S2 stores the data fields in columnar format to obtain multiple arranged fields. Filed 1 is stored columnarly as multiple files Filed 1, Filed 2, Filed 3... Filed n, where each field is written into a separate file, and the file name is named after the field name. Since the field values are formatted in a fixed-length format in S1, the nth value can be located in the file according to the numerical offset. The traffic metric data is split into file packages at time intervals. For example, a new storage file is created every hour, and the timestamp is appended to the file package name.

[0046] In this embodiment, the system records the storage time of the stored metric data. When the metric data expires due to long-term non-use, the expired metric data is aged to save disk space, including two ways of data aging:

[0047] (1) Aging data deletion: Delete the old data and no longer use it;

[0048] (2) Aging data compression: Compress the old data for future use.

[0049] Since each file stores only the value of one field and is in a unified format, the compression ratio is very high, which can save a large amount of storage space. When the data in this time period is occasionally used, it can be directly decompressed. After decompression, the data further refreshes the aging time.

[0050] S3 writes an inverted index file for multiple arranged fields to build an inverted index, and writes at least one index feature value for a specified field among the multiple arranged fields. The eigenvalue index reduces the waiting time of users during query by indexing the values that exist and meet the specified conditions in the specified field. For example, Figure 3 The figure shows a schematic diagram of the one-dimensional index eigenvalue provided by the embodiment of the present invention. For example, in the nth field Filed n, index features are added based on the numerical characteristics. As Figure 3 The right part shows the field file after adding index features. According to the numerical characteristics of Filed n, its numerical values are classified. Among them, 0: 1, 4 means that the first row and the fourth row have a numerical value of 0 in this field file, and 1: 2, 3, 5, 6 means that the second, third, fifth, and sixth rows have a numerical value of 1 in this field file. (

[0051] Furthermore, as Figure 4 The figure shows a schematic diagram of the multi-dimensional index eigenvalue provided by the embodiment of the present invention. For example, for the numerical values of the Filed 3 field, multi-dimensional eigenvalue indexes are added. Among them, assuming the numerical value is v, based on the environment, user, and product requirements, developers can add multi-dimensional additional eigenvalue to v. As Figure 4The right part shows multiple files constructed according to eigenvalue indexes. Among them, index3: v = 4 is 1, 4, which means that under the condition that the value in Filed 3 is equal to 4, the index results are the first row and the fourth row; index3: v > 10 is 3, which means that under the condition that the value is greater than 3, the index result is the third row, that is, the value 13; index3: v > 15 && v < 10 and v > 15 || v < 10 are the same. The method of the embodiment of the present invention constructs a multi-dimensional eigenvalue index for a specified field, classifies massive data, and more quickly filters data that meets the conditions during querying.

[0052] S4 Selects the query field, sets the filtering condition based on the query field, and performs data query based on the filtering condition. For example, Figure 5 The schematic diagram of eigenvalue index query provided by the embodiment of the present invention. Select the query field, for example, Figure 5 sets the query fields as Filed 1, Filed 3, Field n, only processes the query field information, reduces the total number of reads and processes, and then sets the filtering condition based on the field. The filtering can be field value filtering and / or multi-field filtering. For field value filtering, specifically, it can be numerical type filtering. For example, the value is greater than, less than a certain threshold, or equal to a certain value, or the value satisfies being in a certain interval, etc. For numerical type filtering, mixed operations are supported; field value filtering can also be string type, such as exact match, containment match, wildcard match, regular expression match, etc. In the embodiment of the present invention, the filtering condition is set as numerical type filtering (Field 3 < 5 || Field 3 > 10) && Field n > 0.

[0053] Performs data query based on the filtering condition. The data query process is divided into single field value filtering and multi-field filtering. For single field value filtering, it is divided into two levels:

[0054] (1) Index-free filtering: From the field file, read the field values in sequence for filtering and matching, and cache the record numbers of the hits;

[0055] (2) Index-based filtering: Read the data numbers that meet the conditions from the index file of the field.

[0056] For multi-field filtering, it is necessary to merge the filtering results. The merging includes "AND" merging and "OR" merging. Among them, "AND" merging obtains the final filtering result set by extracting the record numbers commonly included in each result set; "OR" merging: accumulate each result set and remove duplicates to obtain the final filtering result set.

[0057] According to the embodiment of the present invention Figure 5As can be seen from the eigenvalue index query schematic diagram given in [reference], this index is a multi-field index. First, single-field filtering is performed on each field separately, and then the filtering results are merged to generate a filtered result set. Based on the filtering conditions (Field 3 < 5 || Field 3 > 10) && Field n > 0, queries are performed in Field 3 and Field n respectively. According to two eigenvalue index files established during storage, the filtering results of Filed 3 in the eigenvalue index file are 1, 3, 4, and the filtering results of Field n are 2, 3, 5, 6. The two filtering results are merged by "AND" to generate a final filtered result set of 3.

[0058] The query results are displayed based on the filtered result set. The filtered result set contains the data serial numbers that meet the filtering conditions. Since each field of the data record has a fixed length, the data offset can be calculated from the data serial number, and the position of this data record in the field file can be directly located by the offset.

[0059] Furthermore, the embodiment of the present invention only extracts the data of the query columns, and does not process other columns, thereby reducing the amount of data processed and disk operations.

[0060] Furthermore, in the context of massive data, in order to improve the query performance as much as possible and reduce the response time, the present invention supports the eigenvalues of preset fields to accelerate the query process for the eigenvalues of relevant fields.

[0061] Furthermore, the present invention can also establish a separate index for the eigenvalues. The filtering conditions matching the eigenvalue index can directly obtain the corresponding filtered result set during the query process, further accelerating the query process.

[0062] In some embodiments of the present invention, eigenvalue indexes are partially established in the network information data. Therefore, during the query process, it is necessary to further query the fields with eigenvalue indexes and those without eigenvalue indexes, specifically as Figure 6The flowchart of the network traffic information query method provided by the embodiment of the present invention is shown, which shows the entire query process. The entire process starts with setting query columns and filtering conditions, and matches each relevant field in the filtering conditions one by one to determine whether there is an index for this field. If the determination is yes, the filtering serial number set of this field is obtained from the index file; if the determination is no, the data file is read, the values of this field are matched, and its filtering serial number set is obtained. Further, the filtering mode is confirmed. The filtering mode includes "AND" and "OR". If the "AND" mode is selected for filtering, it is further determined whether the filtering result of any field is an empty set. If the determination result is yes, the query result is an empty set and the query ends; if the determination result is no, the filtering serial number sets of each field are merged and the serial number set is mapped to the data set, and the query ends; if the "OR" mode is selected for filtering, it is further determined whether the filtering result of any field is the entire set. If the determination result is yes, the query result is the entire set and the query ends; if the determination result is no, the filtering serial number sets of each field are merged, and the serial number set is mapped to the data set, and the query ends.

[0063] When querying columnar storage data in the traditional technology, the system needs to scan the values of all rows to generate the filtering result by filtering each field's file, and its scanning and filtering speed is very slow, and the user waiting time is long. The query method based on the network traffic information storage method of the present invention introduces eigenvalue indexing, effectively improving the query performance and reducing the response time.

[0064] It should be particularly noted that each step in each embodiment of the above network information traffic storage method can be crossed, replaced, added, or deleted with each other. Therefore, these reasonable permutation and combination transformations also belong to the protection scope of the present invention for the network information traffic storage method, and the protection scope of the present invention should not be limited to the embodiments.

[0065] In the third aspect of the embodiment of the present invention, a network information storage device is proposed. Figure 7 The schematic diagram of the embodiment of the network information query device provided by the present invention is shown. As Figure 7 shown, the network information query device of the embodiment of the present invention includes the following modules:

[0066] A data processing module 001, configured to obtain network traffic index data and format the index data to generate data fields in a fixed-length format;

[0067] A data storage module 002, configured to perform columnar storage on the data fields to obtain multiple arranged fields;

[0068] An index construction module 003, the index construction module 003 is configured to write an inverted index file for the multiple permutation fields to construct an inverted index, and write at least one index feature value for a specified field in the multiple permutation fields;

[0069] A data query module 004, the data query module 004 is configured to select a query field, set a filtering condition based on the query field, and perform a data query based on the filtering condition.

[0070] Finally, it should be noted that those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program of the network traffic information storage and query method can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium of the program can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc. The embodiments of the above computer programs can achieve the same or similar effects as the corresponding foregoing method embodiments.

[0071] In addition, the method disclosed according to the embodiments of the present invention can also be implemented as a computer program executed by a processor, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the above functions defined in the method disclosed in the embodiments of the present invention are executed.

[0072] In addition, the above method steps and system units can also be implemented by using a controller and a computer-readable storage medium for storing a computer program that enables the controller to implement the above steps or unit functions.

[0073] Those skilled in the art will also understand that the various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, a general description of the functions of various illustrative components, blocks, modules, circuits, and steps has been given. Whether this function is implemented as software or hardware depends on the specific application and the design constraints imposed on the overall system. The functions that those skilled in the art can implement in various ways for each specific application, but this implementation decision should not be construed as causing a departure from the scope of the disclosure of the embodiments of the present invention.

[0074] In one or more exemplary designs, the functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one location to another. The storage media may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a general purpose or special purpose computer or a general purpose or special purpose processor. Additionally, any connection is properly termed a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0075] The above are exemplary embodiments disclosed by the present invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments disclosed by the present invention as defined by the claims. The functions, steps, and / or actions of the method claims according to the disclosed embodiments herein need not be performed in any particular order. In addition, although the elements disclosed by the embodiments of the present invention may be described or claimed in individual form, they may also be understood as plural unless explicitly limited to the singular.

[0076] It should be understood that, as used herein, unless the context clearly supports the exception, the singular form "a" is also intended to include the plural form. It should also be understood that the "and / or" used herein refers to any and all possible combinations of one or more of the associated listed items.

[0077] The serial numbers of the disclosed embodiments of the present invention above are only for description and do not represent the superiority or inferiority of the embodiments.

[0078] Those of ordinary skill in the art can understand that all or part of the steps to implement the above embodiments can be completed by hardware or by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium, and the above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disc, or the like.

[0079] Those of ordinary skill in the art should understand that the discussion of any above embodiment is only exemplary and is not intended to imply that the scope (including the claims) disclosed by the embodiments of the present invention is limited to these examples; under the idea of the embodiments of the present invention, the technical features in the above embodiments or different embodiments can also be combined, and there are many other variations in different aspects of the above embodiments of the present invention, which are not provided in detail for the sake of brevity. Therefore, any omission, modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present invention shall be included in the protection scope of the embodiments of the present invention.

Claims

1. A method for querying network traffic information, characterized in that, It includes the following steps: Obtain network traffic metric data and format the metric data to generate data fields in a fixed-length format; Perform columnar storage on the data fields to obtain multiple arranged fields; Write an inverted index file for the multiple arranged fields to build an inverted index, and write at least one index feature value for a specified field in the multiple arranged fields; Select a query field from the arranged fields containing the index feature value, set a filtering condition based on the query field, and perform data query based on the filtering condition; Among them, writing at least one index feature value for a specified field in the multiple arranged fields includes: classifying based on the numerical value of the data field and writing the row number where the classified numerical value is located in the arranged field as the index feature value, and writing multi-dimensional feature values related to the environment, user, and product requirements as the index feature value; Selecting a query field from the arranged fields containing the index feature value, setting a filtering condition based on the query field, and performing data query based on the filtering condition includes: selecting a query field from the arranged fields containing the index feature value, and setting the numerical value v to be equal to a preset value, greater than a preset value, and / or less than a preset value based on the query field to perform data query; Performing columnar storage on the data fields to obtain multiple arranged fields includes: performing binary processing on the data fields and performing columnar storage to obtain multiple arranged fields, and each arranged field is named in the order of the fields.

2. The network traffic information query method according to claim 1, wherein Obtaining network traffic metric data and formatting the metric data to generate data fields in a fixed-length format includes: Obtain network traffic metric data, store the metric data as a file package, the file package is updated and created at fixed time intervals and the name of the file package is attached with a timestamp; Format the metric data in the file package to generate data fields in a fixed-length format.

3. The network traffic information query method according to claim 1, wherein It also includes: When the metric data has not been used for an overdue period, delete the metric data and / or compress the metric data.

4. The network traffic information query method according to claim 1, wherein The index feature value includes a value greater than a first threshold and / or less than a second threshold and / or equal to a third threshold.

5. The network traffic information query method according to claim 1, wherein The query method of the query includes query without an index feature value and query with an index feature value.

6. The network traffic information query method according to claim 5, wherein, The query method of the query with an index feature value includes: Index the index feature value in the arranged field based on the filtering condition, read the index result, and aggregate the index results into a filtered result set.

7. A network traffic information query device, characterized in that, It includes: A data processing module configured to obtain network traffic metric data and format the metric data to generate data fields in a fixed-length format; A data storage module configured to perform columnar storage on the data fields to obtain multiple arranged fields; An index construction module configured to write an inverted index file for the multiple arranged fields to build an inverted index, and write at least one index feature value for a specified field in the multiple arranged fields; A data query module, which is configured to select query fields from the index feature values, set filtering conditions based on the query fields, and perform data queries based on the filtering conditions; Wherein, the index construction module is further configured to classify based on the numerical values of the data fields and write the row numbers where the classified numerical values are located in the arrangement fields as index feature values, and write multi-dimensional feature values related to the environment, user, and product requirements as index feature values; The data query module is further configured to select query fields from the arrangement fields containing index feature values, and set the numerical value v to be equal to a preset value, greater than a preset value, and / or less than a preset value based on the query fields to perform data queries; The data storage module is further configured to perform binary processing on the data fields and perform columnar storage to obtain a plurality of arrangement fields, and each of the arrangement fields is named in the order of the fields.

8. A computer device, characterized in that, Comprising: At least one processor; And A memory, the memory stores computer instructions that can be run on the processor, and when the instructions are executed by the processor, the steps of the method according to any one of claims 1-6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1-6 are implemented.

Images