Efficient big data storage file structure and data storage method

Abstract

The application relates to a high-efficiency large-data storage file structure and a data storage method, which comprises multiple files, each file comprising a file header unit, a data unit and an index unit; the file header unit is used for recording the check magic number of the file, the file fileNum, the index quantity size and the data byte size of each page; the data unit comprises multiple data pages, each data page is used for storing data in a binary form and is compactly arranged according to the fingerprint order, and sequentially comprises a fingerprint, the data length and the data quantity contained by the fingerprint and the data itself; the index unit is used for compactly arranging each data page into an array according to the fingerprint order, each data page in the array is marked with the page index of the corresponding data page, and the data position is found through the page index. The application can effectively store and manage a large quantity of fingerprint data, improves the efficiency during data query and retrieval, can quickly position and retrieve the required fingerprint data, and improves the efficiency of data query and retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data security, in particular to an efficient big data storage file structure and data storage method. BACKGROUND

[0002] The traditional fingerprint data storage method usually uses a relational database or a file system for storage and management. Based on the storage mode of the relational database, the fingerprint data is stored as a binary large object (BLOB) or the fingerprint feature points are extracted as a feature vector, and stored in the form of a table in the database, and SQL statements are used for query and retrieval. This way has certain flexibility and scalability in data storage and management, but the data storage efficiency is low, the storage space is wasted, and the data query and retrieval speed is slow, which is not suitable for large-scale fingerprint data storage and management.

[0003] The storage mode based on the file system stores and manages the fingerprint data in the form of a file, and each file represents a fingerprint. The file name is usually the fingerprint ID, and the file content is the binary data of the fingerprint feature points. The file system is used for query and retrieval. This way can improve the data storage efficiency and query speed, but due to the simple storage method, it cannot perform advanced query and retrieval operations, and also has the problem of storage space waste.

[0004] The fingerprint data storage method based on hash index maps the fingerprint data to a hash value, and stores and manages it through a hash table, which can improve the data query and retrieval efficiency. However, in the case of hash collision, the data query and retrieval efficiency may decrease. The fingerprint data storage method based on compression algorithm compresses the fingerprint data to reduce the storage space and improve the data storage efficiency. However, due to the large amount of time and calculation in compression and decompression, the data query and retrieval speed may decrease. The fingerprint data storage method based on distributed storage stores the fingerprint data on multiple nodes and uses a distributed query method for data query and retrieval. This way can improve the efficiency of data storage and query, but it needs to be planned and deployed for distributed storage and management, which has certain technical difficulty. Therefore, for large-scale fingerprint data, there are still problems such as storage space waste and low query and retrieval efficiency.

[0005] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The present application aims to overcome the shortcomings of the prior art and provides an efficient big data storage file structure and data storage method, which solves the deficiencies of the existing methods.

[0007] The application discloses a high-efficiency big data storage file structure.

[0008] The file header unit is used for recording the check magic number, file fileNum, index quantity size and each page data byte size of the file.

[0009] The data unit comprises a plurality of data pages, each data page is used for storing data in a binary form and is compactly arranged in a fingerprint order, and sequentially comprises a fingerprint, data length and data quantity contained by the fingerprint and data itself.

[0010] The index unit is used for compactly arranging each data page into an array in a fingerprint order, and each data page in the array is marked with a page index of the corresponding data page.

[0011] The index unit comprises the following steps: when the data is searched through the fingerprint, first, the starting position offest of the fingerprint data is determined through binary search after all the index information is loaded into the memory, and then the data is read by directly randomly accessing the starting position of the data page corresponding to the index mark.

[0012] A data storage method based on the high-efficiency big data storage file structure.

[0013] S1, traffic data is captured through a traffic feature extraction algorithm, a fingerprint is generated according to a rule, and the required file quantity is determined according to the size of the data quantity.

[0014] S2, the fingerprint data is sequentially read, and a fingerprint and all corresponding data are taken as a unit and written into a data page of a file, the data of the unit sequentially comprises a fingerprint and subsequent data length, data quantity and data stored in a compact arrangement form.

[0015] Whether the current written data page is the beginning of a data page, if yes, the index value corresponding to the data page is equal to the fingerprint value, if not, whether the size of the remaining storage capacity of the current data page is greater than the data corresponding to the fingerprint is judged, if yes, the data corresponding to the fingerprint is written into the data page, otherwise, the data page is skipped and the next data page is written.

[0016] If the size of the data corresponding to the fingerprint is greater than the storage capacity of a data page, that is, the data corresponding to the fingerprint spans a plurality of data pages, the index value corresponding to each spanned data page is increased by 1.

[0017] The file includes a file header unit, a data unit and an index unit; the file header unit is used for recording a check magic number of the file, a file fileNum, an index quantity size and a data byte size per page; the data unit includes multiple data pages, each data page is used for storing data in binary form and is arranged compactly in a fingerprint order, and sequentially includes a fingerprint, a data length contained by the fingerprint and a data number and data itself, when searching, the data positioned according to the index is read according to the determined data size order; the index unit is used for arranging each data page compactly in an array according to the fingerprint order, each data page in the array is marked with a page index of the corresponding data page, and the data position is found through the page index.

[0018] The data storage method further includes the step of quickly searching the data position according to a search rule, which is executed after step S2.

[0019] The step of quickly searching the data position according to the search rule includes:

[0020] The file is determined to be read according to the fingerprint hash value, the index is positioned to the data page where the fingerprint data is located through index binary search, and the starting position of the data page is positioned according to the determined data byte size per page and the index quantity size;

[0021] It is judged whether the real-time position of the data page is the beginning of a fingerprint data, if not, the data page is reduced by 1, until the condition is met, and then the data is read;

[0022] The length of one fingerprint is read to judge whether the read fingerprint is the searched fingerprint, if yes, four bytes, i.e. the length of the data corresponding to the fingerprint, are read, and the part of data is read, if not, four bytes, i.e. the length of the data corresponding to the fingerprint, are read, the length bytes are skipped, and if the read fingerprint is greater than the searched fingerprint, it is indicated that the fingerprint data does not exist.

[0023] The present application has the following advantages: a high-efficiency large data storage file structure and data storage method, which can effectively store and manage a large amount of fingerprint data, and improve the efficiency in data query and search; can save storage space and reduce data storage cost without affecting data storage; can quickly locate and search the required fingerprint data, and improve the efficiency of data query and search. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 It is a schematic diagram of the file structure of the present application;

[0025] Figure 2 It is a flowchart of writing the fingerprint data into the file of the present application;

[0026] Figure 3 Flowchart of searching data for the present application;

[0027] Figure 4 Example diagram of indexing for the present application. DETAILED DESCRIPTION

[0028] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided with reference to the accompanying drawings is not intended to limit the protection scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the protection scope of the present application. The present application will be further described below with reference to the accompanying drawings.

[0029] As shown in Figure 1 one of the real-time modes of the present application involves a high-efficiency large-data storage file structure, which includes multiple files, each file including a header, data, and an index.

[0030] The header is a file header, recording the basic information of the file, including a check magic number, a file number fileNum, an index number size, and a data byte size per page.

[0031] The data is data, arranged compactly according to the fingerprint order, in turn including a fingerprint, the length of the data contained by the fingerprint, the number of data contained, and the data. The data is stored in binary form next to each other, so a large amount of data space is saved, and in the retrieval, the data positioned according to the index is read according to the determined data size order.

[0032] Each page of data is not necessarily full, and there will be a certain amount of white space left over, which is to ensure that the page where the fingerprint is located can be positioned faster in the retrieval, and then the retrieval starts from the start of the page.

[0033] The index is an index, which is arranged in an array in a compact order according to the fingerprint sequence, and the array subscript is the corresponding pageindex. The index data is the key of the index. When the data is searched through the fingerprint, the index information is first loaded into the memory, and then the binary search is performed to determine the starting position offset of the fingerprint data in the page. Because the size of each page is determined, the starting position of the page can be directly randomly accessed according to the index subscript, that is, the page number, to read the data.

[0034] Another embodiment of the application relates to a data storage method based on an efficient large data storage file structure, the data storage method comprising:

[0035] S1, the traffic data is captured by a traffic feature extraction algorithm to generate fingerprints according to rules, and the number of files required is determined according to the size of the data volume;

[0036] The traffic feature extraction algorithm is a technology for extracting features from network traffic data and generating fingerprint information. These fingerprint information can be used to identify and classify different network traffic, such as network attacks and normal network communication. Common traffic features include: packet header information, packet size, protocol type, source IP address and destination IP address, etc. These features can be extracted by various algorithms and techniques. The fingerprint information of the traffic feature extraction algorithm usually includes a combination of multiple feature values, which can represent different aspects of network traffic.

[0037] Further, taking a piece of traffic feature fingerprint as an example, it mainly includes the hash value, source IP, destination IP, protocol and other information of the piece of traffic feature. For example, the data of a piece of traffic feature is as follows, and each array is the hash value, source IP and destination IP of the piece of traffic feature.

[0038] S2, as shown in Figure 2 The fingerprint data is sequentially read, and a fingerprint and all corresponding data are written into a data page (page) of a file as a unit. The data of the unit includes the fingerprint and the subsequent data length, the number of data and the data stored in a compact arrangement;

[0039] Whether the current data page is the beginning of a one-page data page, if yes, the index value corresponding to the data page is equal to the fingerprint value, if not, whether the size of the remaining storage capacity of the current data page is greater than the data corresponding to the fingerprint, if yes, the data corresponding to the fingerprint is written into the data page, otherwise, the data page is skipped and the next data page is written;

[0040] If the data size corresponding to the fingerprint is greater than the storage capacity of a data page, i.e. the data corresponding to the fingerprint spans multiple data pages, the index value corresponding to each spanned data page is increased by 1; this is to enable locating the page where the fingerprint data is located when performing a binary search of the index page by the fingerprint during retrieval.

[0041] Further, the data storage method further comprises a step of quickly searching the data location according to a search rule, which is performed after step S2.

[0042] As shown in Figure 3 the step of quickly searching the data location according to the search rule comprises:

[0043] determining the file to be read according to the fingerprint hash value, performing a binary search of the index to locate the data page where the fingerprint data is located, and locating the start position of the data page according to the determined data byte size per page and index number size;

[0044] determining whether the real-time position of the data page is the start of a fingerprint data according to index[page]&totalFileNum==fileNumber, if not, then the data page is decreased by 1 until the condition is met, and then the data is read;

[0045] wherein index[page]&totalFileNum==fileNumber is to determine whether the start of the data page is the start of a complete fingerprint data, for example, the total number of files to be stored by a system is 32, and totalFileNum=31 starting from 0, then in the file with fileNumber 16, according to the file format set previously, as long as the index value corresponding to a page (the index and the data page are one-to-one) divided by the total number of files 32 has a remainder not equal to 16, index[page]&totalFileNum==fileNumber is equivalent to taking the remainder of fileNumber divided by totalFileNum+1, for example, 17%32!=17, which means that the data start of the page is not the start of a complete fingerprint data, and only the page above can be searched for a page that starts with a complete fingerprint data for retrieval.

[0046] reading the length of a fingerprint to determine whether the read fingerprint is the searched fingerprint, if yes, reading four bytes, i.e. the length of the data corresponding to the fingerprint, and starting to read the part of data, if not, reading four bytes, i.e. the length of the data corresponding to the fingerprint, skipping the length bytes, and directly finding or reading the fingerprint greater than the searched fingerprint, which means that the fingerprint data does not exist.

[0047] The present application first finds a page by binary search, but this page is not necessarily the page where the data is located, because some fingerprint data can span multiple pages, resulting in the start position of each page not necessarily being the start of a fingerprint data, as shown in Figure 4 Taking the file with fileNumer of 0 and total file number n of 8 as an example, when searching for data with fingerprint 24, binary search first locates to fp2, and then finds that the value of fp2 is 17, (17&n-1)! = 0, so page is reduced to fp1, and then data is read from the start position corresponding to fp1, until the matched fingerprint or data greater than the fingerprint (indicating that the fingerprint does not exist) is read.

[0048] The above only describes the preferred embodiments of the present application, and it should be understood that the present application is not limited to the forms disclosed herein, and should not be considered as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concepts described herein by the above teachings or related art or knowledge. Any modification and change made by those skilled in the art without departing from the spirit and scope of the present application shall be within the protection scope of the claims of the present application.

Claims

1. A data storage method based on an efficient big data storage file structure, characterized in that: The data storage method includes: S1. Capture traffic data using a traffic feature extraction algorithm, generate fingerprints according to rules, and determine the required number of files based on the size of the data. S2. Read fingerprint data sequentially, and write a fingerprint and all its corresponding data as a unit into the data page of the file. The data in this unit includes the fingerprint and the length of the subsequent data. The number of data entries and the data are stored in a compact arrangement. Determine whether the data page being written is the beginning of a data page. If it is, the index value of the data page is equal to the fingerprint value. If not, continue to determine whether the remaining storage capacity of the current data page is greater than the data corresponding to the fingerprint. If it is, write the data corresponding to the fingerprint into the data page. Otherwise, skip the data page and start writing from the next data page. If the data size corresponding to the fingerprint is greater than the storage capacity of a data page, that is, if the data corresponding to the fingerprint spans multiple data pages, then the index value corresponding to it is incremented by 1 for each additional data page it spans. The file includes a file header unit, a data unit, and an index unit. The file header unit records the file's checksum magic number, fileNum, index size, and the size of each page in bytes. The data unit includes multiple data pages, each used to store data in binary form and arranged compactly according to fingerprint order. Each data page includes the fingerprint, the length and number of data entries contained in the fingerprint, and the data itself. During retrieval, the data located by the index is read sequentially according to the determined data size. The index unit arranges each data page into an array in fingerprint order. Each data page in the array is identified by its page index, and the data location is found through the page index.

2. The data storage method based on an efficient big data storage file structure according to claim 1, characterized in that: The data storage method further includes a step of quickly retrieving data locations according to retrieval rules, which is performed after step S2.

3. The data storage method based on an efficient big data storage file structure according to claim 2, characterized in that: The steps for quickly retrieving data locations based on retrieval rules include: The file to be read is determined based on the fingerprint hash value. The index is used to perform a binary search to locate the data page containing the fingerprint data, and the starting position of the data page is located based on the determined size of each page in bytes and the number of indexes. Determine if the real-time position of the data page is the start of a fingerprint data. If not, decrement the data page by 1 until the condition is met, then start reading data. The length of a fingerprint is read to determine if the read fingerprint is the fingerprint to be retrieved. If it is, four bytes are read, which is the length of the data corresponding to the fingerprint, and the data for that part is read. If it is not, four bytes are read, which is the length of the data corresponding to the fingerprint, and the length bytes are skipped. If the fingerprint found or read is greater than the fingerprint to be retrieved, it means that the fingerprint data does not exist.

Images