Massive file data stream storage method, storage system and big data processing system

Abstract

The application relates to the technical field of computer application, and discloses a mass file data stream storage method, a storage system and a big data processing system. A rotation state conversion model is arranged between a data stream and a storage node, and storage pressure is dispersed through multi-node caching. The storage method comprises the following steps: presetting a file size demarcation point; inputting mass file data; judging whether the size of the input file is smaller than or equal to the demarcation point; storing file data smaller than or equal to the demarcation point in a selected storage node through the rotation state conversion model; generating index information of all small files, and merging the index information into an index file; organizing data in the form of a B+ tree for each index file to obtain an index data bucket; storing a plurality of index data buckets in the form of data blocks in the storage node; and saving all index data buckets in sequence as an index data bucket queue. The application can effectively solve the problems of traditional distributed storage systems, such as write performance decline and low efficiency when facing mass files.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer application, and in particular to a mass file data stream storage method, a storage system and a big data processing system. BACKGROUND

[0002] Under the background of the rapid development of technologies such as Internet of Things, cloud computing, artificial intelligence and big data, the amount of data is showing an explosive growth. These data not only require a huge amount of storage space, but also have the outstanding characteristics of various data types, large data size changes, and fast data flow. Generally, data files can be divided into structured files, semi-structured files and unstructured files, and the file sizes are different. That is, these file data often contain a large number of large files and hundreds of millions of mass small files. These file data have bottlenecks in metadata management, storage efficiency and access performance, especially the storage problem of mass small files (LSOF, lots of small files), which is a recognized problem in the current industry and academia. The traditional storage calling system cannot meet the increasing storage demand and access speed requirements of file data. On the one hand, the ordinary hard disk storage calling technology has slow read-write speed and is not suitable for large-scale data storage, while the solid state disk and flash storage calling technology has high cost although it has fast read-write speed. On the other hand, the traditional relational database has certain limitations in processing mass data, and although the distributed storage system provides some solutions, it still faces great challenges in performance optimization, cost control and scalability.

[0003] Hadoop is the most popular big data processing architecture at present, which provides a processing framework for managing and analyzing mass data. Hadoop Distributed File System (HDFS) is the core component for storing data in the Hadoop system. It performs well in storing and managing large files. However, it performs poorly in processing a large number of small files. This is because the existing distributed file system represented by HDFS generally adopts a master-slave node architecture. The client node issues a write request, and the slave node DataNode is responsible for storing data, and the master node NameNode (the management node of the HDFS file system, which maintains the file directory tree and data block index, and the correspondence between data blocks and data nodes) is responsible for maintaining the entire namespace and recording all modifications of the namespace. Each file stored by HDFS needs to store meta information in NameNode, regardless of the size of the file itself, the meta information occupies 150 bytes. Therefore, when facing a large number of small files, the master node needs to maintain too much metadata, and the memory resource consumption is rapid, which causes great pressure on the master node, making the write performance of HDFS decrease seriously and the access to small files inefficient.

[0004] Therefore, there is an urgent need to propose a method that can efficiently store massive amounts of small files. Summary of the Invention

[0005] The purpose of this invention is to provide a storage method, storage system, and big data processing system for massive file data streams. By providing an ORFS round-robin state transition model and strategy architecture design, an optimized round-robin intermediate layer is built between the small file data stream and the underlying storage nodes. This distributes the data stream across multiple disks on multiple nodes, fully utilizing the performance of storage devices and improving cluster utilization. Simultaneously, a B+ tree is used to generate the index structure and is integrated with ORFS. Additional capabilities enhance the access performance of individual small files, simplify data operations, and effectively reduce NameNode memory consumption, providing convenience for efficient storage and analysis of massive small files.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a method for storing massive file data streams, which deploys a round-robin state transition model between the data stream and storage nodes, and distributes storage pressure through multi-node caching; the storage method includes the following steps:

[0008] S1. Preset file size dividing point;

[0009] S2. Input massive amounts of file data, and check whether the size of each input file is less than or equal to the boundary point. If so, execute S3 to S4.

[0010] S3. File data less than or equal to the boundary point is stored in the selected storage node through a round-robin state transition model;

[0011] S4. Generate index information for all files less than or equal to the boundary point, and merge the index information into the index file. Each index file organizes data in the form of a B+ tree to obtain index data buckets. Multiple index data buckets are stored in the form of data blocks on the storage node, and all index data buckets are saved sequentially as an index data bucket queue.

[0012] As one possible implementation, the round-robin state transition model has m data buckets, which are encoded into a queue; the first k data buckets in the queue are data filling buckets, k < m, and the other data buckets are idle waiting buckets;

[0013] The round-robin state transition model stores file data less than or equal to the boundary point in selected storage nodes, specifically including the following steps:

[0014] S30. Write file data to multiple data filling buckets in parallel;

[0015] S31. When the data filling bucket is full, remove the full data filling bucket from the data filling bucket queue and move the full data filling bucket into the write waiting bucket queue. That is, the state of the full data filling bucket is changed to write waiting bucket. Remove a new bucket from the idle waiting bucket to the data filling bucket queue to accept subsequent file data writing.

[0016] S32. The round-robin state transition model is equipped with a data bucket scheduling module, which is used to poll storage nodes in a round-robin fashion and select storage nodes based on a load balancing strategy;

[0017] S33. Using hashing techniques, the write wait bucket is routed to the selected storage node for data dumping preparation. At this time, the state of the write wait bucket is changed to the data dumping bucket.

[0018] S34. After the data in the data dump bucket is written to the selected storage node, the status of the data dump bucket changes to idle waiting bucket, waiting for task distribution.

[0019] As one possible implementation, each selected storage node corresponds to a waiting queue consisting of multiple write waiting buckets. The write waiting buckets are refreshed one by one, and the write waiting buckets to be refreshed are simultaneously rewritten as data dumping buckets. The refreshed data dumping buckets are rewritten as idle waiting buckets.

[0020] As one possible implementation, the storage methods for storage nodes include:

[0021] Receive the data dump bucket, determine if there is enough remaining memory to hold the file data in the data dump bucket. If not, send a retransmission request and wait for retransmission. If yes, add the data dump bucket to the write queue.

[0022] After the write thread completes the previous write task, it retrieves the first data dump bucket from the write queue and hands it over to the write thread to complete the task of writing the file data in the data dump bucket to the storage node.

[0023] As one possible implementation, check each input file size to see if it is less than or equal to the boundary point. If not, proceed with the following steps:

[0024] S5. Determine whether the file data needs to be split. If yes, execute S6-S7; otherwise, execute S8.

[0025] S6. Split file data;

[0026] S7. Determine whether the data of the split file is less than or equal to the dividing point. If yes, execute S3 to S4; if no, execute S5 to S7; until the result of S5 is that no split is needed, then execute S8.

[0027] S8. File data is stored in HDFS.

[0028] As one possible implementation, the dividing point is 4.35MB.

[0029] Secondly, the present invention provides a storage system for massive file data streams, comprising:

[0030] The input port is used to receive massive file data streams;

[0031] The first judgment module is configured with a boundary point to ensure that the size of the input file is less than or equal to the boundary point.

[0032] The second judgment module is used to determine whether file data larger than the dividing point needs to be split.

[0033] The splitting module is used to split file data that is larger than the dividing point and needs to be split.

[0034] HDFS is used to store file data that is larger than the delimiter and does not need to be split.

[0035] The rotation state transition module is used to store file data that is less than or equal to the boundary point to the selected storage node;

[0036] The index information generation module is used to generate index information for all files less than or equal to the boundary point, and merge the index information into the index file. Each index file organizes the data in the form of a B+ tree to obtain index data buckets. Multiple index data buckets are stored in the form of data blocks on the storage node, and all index data buckets are saved sequentially as an index data bucket queue.

[0037] Storage nodes are used to store file data less than or equal to the boundary point, as well as multiple index data buckets.

[0038] As one possible implementation, the round-robin state transition module has m data buckets, which are encoded into a queue; the first k data buckets in the queue are data filling buckets, k < m, and the other data buckets are idle waiting buckets; when storing file data smaller than the boundary point in the selected storage node through the round-robin state transition model, S30 to S34 as described in claim 2 are executed.

[0039] As one possible implementation, the storage system has a master node and storage nodes. The master node controls the rotation state transition model, and the storage nodes are responsible for data storage.

[0040] Thirdly, the present invention provides a big data processing system that uses the storage method provided in the first aspect to store massive file data streams.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] 1. The storage method for massive file data streams provided by this invention designs a round-robin state transition model, which is deployed between the high-speed data stream and the underlying storage nodes. By distributing storage pressure through multi-node caching, the high-speed data stream write pressure of a single node can be effectively alleviated, making full use of each node in the cluster to maximize the development of the cluster's storage performance.

[0043] 2. The storage method for massive file data streams provided by this invention deploys a round-robin state transition model on top of the storage node, so it is not affected by database version iteration updates, and many storage optimization efforts can still be applied.

[0044] 3. The storage method for massive file data streams provided by this invention uses a round-robin state transition model to organize index information into a B+ tree for storage. The B+ tree-based storage mode can quickly retrieve index information of small files, thereby effectively improving the overall performance of processing small files.

[0045] 4. This invention provides a round-robin state transition model ORFS, uses B+ trees to generate index structures, and integrates with ORFS, which can effectively solve the problems of severe write performance degradation and low efficiency in accessing small files in traditional distributed storage systems when faced with massive files. Attached Figure Description

[0046] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0047] Figure 1 This is a data bucket rotation state transition diagram showing the four states of idle waiting, data filling, write waiting, and data dumping in an embodiment of the present invention;

[0048] Figure 2 A flowchart illustrating a method for storing massive file data streams provided in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram illustrating the working principle of the rotation state transition model provided in this embodiment of the invention;

[0050] Figure 4 A schematic diagram of a storage system for massive file data streams provided in an embodiment of the present invention;

[0051] Figure 5 The rotation process of the master node in the storage system for massive file data streams provided in this embodiment of the invention;

[0052] Figure 6 A flowchart of the storage node operation of a storage system for massive file data streams provided in an embodiment of the present invention.

[0053] Figure label:

[0054] 10 - Input port, 20 - First judgment module, 30 - Second judgment module, 40 - Segmentation module, 50 - HDFS, 60 - Round-robin state transition module, 70 - Index information generation module, 80 - Storage node. Detailed Implementation

[0055] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0056] It should be noted that when a component is referred to as "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as "connected to" another component, it can be directly connected to or indirectly connected to that other component.

[0057] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0058] In the description of this invention, it should be understood that the terms "upper" and "lower" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0059] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral connection; it can refer to a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0060] This invention proposes a storage method, storage system, and big data processing system for massive file data streams. It employs an efficient storage and retrieval technology for massive data streams based on an optimized round-robin strategy, specifically for the efficient storage and retrieval of massive small file data streams. Unlike traditional solutions, this invention provides a round-robin state transition model (ORFS) and a strategy architecture design. This model is deployed between the high-speed data stream and the underlying storage nodes. By distributing storage pressure through multi-node caching, it alleviates the high-speed data stream write pressure on a single node, fully utilizing each node in the cluster to maximize the cluster's storage performance. Because it is deployed on top of the storage nodes, it is not affected by database version updates, allowing many storage optimization efforts to remain applicable. Simultaneously, ORFS organizes index information into B+ trees for storage. This B+ tree-based storage model enables rapid retrieval of index information for small files, thereby effectively improving the overall performance of small file processing.

[0061] In a first aspect, embodiments of the present invention provide a method for storing massive file data streams, deploying a round-robin state transition model between the data stream and storage nodes, and distributing storage pressure through multi-node caching.

[0062] As one possible implementation, the round-robin state transition model has m data buckets, which are organized into a queue; the first k data buckets in the queue are data filling buckets, k < m, and the other data buckets are idle waiting buckets.

[0063] The round-robin algorithm, also known as the rotation algorithm, is a widely used optimization strategy in computer science. It optimizes data access and improves data processing efficiency by rotating the elements of a dataset according to certain rules. The core idea of ​​the round-robin algorithm is to rotate the elements of the dataset in a specific order, making data access more efficient. The basic principle of the round-robin algorithm is as follows:

[0064] (1) Define the rotation direction: Determine the rotation direction of the data set, which can be left rotation or right rotation.

[0065] (2) Determine the number of rotations: Determine the number of rotations based on actual needs, i.e. how many positions to rotate.

[0066] (3) Perform rotation operation: Rotate the data set according to the defined rotation direction and number of rotations.

[0067] Depending on the state of the data bucket, it can be divided into four states: idle waiting, data filling, write waiting, and data dumping. See the state transition diagram for details. Figure 1Idle Waiting: When a data bucket is in an idle waiting state, it is empty and ready to accept the insertion of new data streams. In this embodiment, all data buckets are initially set to this idle waiting state. Data Filling: When a data bucket is in a data filling state, all data insertions are directed to that data bucket. This means that such a data bucket is the one the system actually writes to during runtime. In this implementation, at least one data bucket must be in a data filling state; otherwise, there will be no available data bucket to accommodate the newly inserted data, triggering the system stop mechanism. Another key point is that when data filling causes a data bucket to be full, it cannot accept new data. In this case, the data bucket will enter a write waiting state. Simultaneously, the system will automatically select one from the idle waiting data buckets and use it as the new data filling bucket. Write Waiting: Data buckets in the write waiting state are converted from data filling buckets; that is, they are all filled with newly written temporary storage data and are waiting to be persistently stored on the disk or SSD. The system will not immediately persist a full data bucket because the write bandwidth of the underlying disk or SSD is limited. Therefore, all full buckets of data must wait until the system schedules them for persistence. Each storage node has a wait queue to store data buckets in a write-waiting state that have not yet been persisted. The system refreshes the data buckets in the wait queue one by one. Data dumping: When data in a bucket in a write-waiting state is about to be persisted, its state changes to data dumped. This means that the data temporarily stored in that bucket will be written to persistent storage. Once all data in the data dumped bucket has been written to persistent storage, the system changes its state to idle and waiting. The system can have multiple data dumped buckets persisting data simultaneously.

[0068] This embodiment defines numerous data buckets at the upper layer of the round-robin state transition model, with each data bucket representing the memory space of a node in the data center. Therefore, all data buckets constitute a massive distributed memory space. The key idea behind this patent design is to utilize this vast distributed memory space to accept rapidly arriving massive data streams.

[0069] See Figure 2 The method for storing massive file data streams provided in this embodiment includes the following steps:

[0070] S1. Preset file size delimiter; for example, the delimiter is 4.35MB.

[0071] S2. Input massive amounts of file data, and check whether the size of each input file is less than or equal to the boundary point. If so, execute S3 to S4.

[0072] In this embodiment, files are distinguished according to their size. Files larger than 4.35MB are defined as large files, and files smaller than or equal to 4.35MB are defined as small files. Small files are stored in a non-clustered index, that is, their index information is stored in the round-robin state transition model, and the data information is stored in the selected storage node through the round-robin state transition model.

[0073] As an example, the data structure for a small collection of files is as follows:

[0074] {

[0075] "_id":ObjectId("IDENTIFIER"),

[0076] "filename":"FILENAME",

[0077] "format":"FORMAT",

[0078] "uploadDate":UPLOAD_TIMESTAMP,

[0079] "length":INTEGER,

[0080] "downloadCount":INTEGER,

[0081] "data":BINARY DATA

[0082] }

[0083] Files larger than 4.35MB are defined as large files. If a large file is indivisible, it will be stored directly in HDFS without further processing. If a large file needs to be split, it will be split first, then the process will return to the previous step to determine if it is a small file before proceeding to the next step. The data structure for large files is shown below:

[0084] {

[0085] "_id":ObjectId("IDENTIFIER"),

[0086] "fileId":ObjeetId("FILE_IDENTIFIER"),

[0087] "n":INTEGER,

[0088] "data":BINARY DATA

[0089] }

[0090] S3. File data less than or equal to the boundary point is stored in the selected storage node through a round-robin state transition model;

[0091] For example, each selected storage node corresponds to a waiting queue consisting of multiple write waiting buckets. The write waiting buckets are refreshed one by one, and the write waiting buckets to be refreshed are simultaneously rewritten as data dumping buckets. The refreshed data dumping buckets are rewritten as idle waiting buckets.

[0092] See Figure 3 As one possible implementation, S3 includes the following steps:

[0093] S30. Write file data to multiple data filling buckets in parallel;

[0094] S31. When the data filling bucket is full, remove the full data filling bucket from the data filling bucket queue and move the full data filling bucket into the write waiting bucket queue. That is, the state of the full data filling bucket is changed to write waiting bucket. Remove a new bucket from the idle waiting bucket to the data filling bucket queue to accept subsequent file data writing.

[0095] It should be noted that the size of the data filling bucket is determined by the system configuration and depends on the specific data distribution strategy. For example, if the data filling bucket size is set to 10GB, then the data filling bucket can be considered full when the data size in each data filling bucket reaches the range of 10000M to 10240M.

[0096] S32. The round-robin state transition model is equipped with a data bucket scheduling module, which is used to poll storage nodes in a round-robin fashion and select storage nodes based on a load balancing strategy;

[0097] S33. Using hashing techniques, the write wait bucket is routed to the selected storage node for data dumping preparation. At this time, the state of the write wait bucket is changed to the data dumping bucket.

[0098] S34. After the data in the data dump bucket is written to the selected storage node, the status of the data dump bucket is changed to idle waiting bucket, waiting for task distribution;

[0099] S4. Generate index information for all files less than or equal to the boundary point, and merge the index information into the index file. Each index file organizes data in the form of a B+ tree to obtain index data buckets. Multiple index data buckets are stored in the form of data blocks on the storage node, and all index data buckets are saved sequentially as an index data bucket queue.

[0100] As one possible implementation, the storage methods for storage nodes include:

[0101] Receive the data dump bucket, determine if there is enough remaining memory to hold the file data in the data dump bucket. If not, send a retransmission request and wait for retransmission. If yes, add the data dump bucket to the write queue.

[0102] After the write thread completes the previous write task, it retrieves the first data dump bucket from the write queue and hands it over to the write thread to complete the task of writing the file data in the data dump bucket to the storage node.

[0103] As one possible implementation, if the size of the input file is determined to be greater than the delimiter, then the following steps are performed:

[0104] S5. Determine whether the file data needs to be split. If yes, execute S6-S7; otherwise, execute S8.

[0105] For example, the criterion for large file splitting is that if a large file can be split into several smaller files and stored on storage nodes, and each smaller file can be retrieved and quickly merged for use, then it is split; otherwise, the large file is not split and is directly stored in the HDFS system. Generally, the greater the regularity contained in the data, the easier it is to split.

[0106] S6. Split file data;

[0107] As an example, this method utilizes iterative partitioning algorithms to divide large file data, improving data processing efficiency through parallel computing and distributed storage. The basic principle is as follows:

[0108] (1) Data partitioning: Dividing a large dataset into multiple smaller datasets, each of which contains a part of the dataset;

[0109] (2) Parallel computing: The segmented dataset is distributed to multiple computing nodes, and each dataset is computed independently;

[0110] (3) Result merging: The calculation results of each computing node are merged to obtain the final result.

[0111] S7. Determine whether the data of the split file is less than or equal to the dividing point. If yes, execute S3 to S4; if no, execute S5 to S7; until the result of S5 is that no split is needed, then execute S8.

[0112] S8. File data is stored in HDFS.

[0113] Secondly, embodiments of the present invention provide a storage system for massive file data streams, see [link to relevant documentation]. Figure 4 The storage system includes:

[0114] Input port 10 is used to receive massive file data streams;

[0115] The first judgment module 20 is configured with a boundary point, which is used to check whether the size of the input file is less than or equal to the boundary point.

[0116] The second judgment module 30 is used to determine whether file data larger than the dividing point needs to be split.

[0117] The segmentation module 40 is used to segment file data that is larger than the dividing point and needs to be segmented;

[0118] HDFS50 is used to store file data that is larger than the delimiter and does not need to be split.

[0119] The rotation state transition module 60 is used to store file data less than or equal to the boundary point to the selected storage node;

[0120] As an example, the round-robin state transition module 60 has m data buckets, which are encoded into a queue; the first k data buckets in the queue are data filling buckets, k < m, and the other data buckets are idle waiting buckets; when storing file data less than or equal to the boundary point in the selected storage node through the round-robin state transition model, the first aspect S30 to S34 are executed.

[0121] The index information generation module 70 is used to generate index information for all files less than or equal to the boundary point, and merge the index information into the index file. Each index file organizes data in the form of a B+ tree to obtain index data buckets. Multiple index data buckets are stored in the form of data blocks on the storage node, and all index data buckets are saved sequentially as an index data bucket queue.

[0122] Storage node 80 is used to store file data less than or equal to the boundary point, as well as multiple index data buckets.

[0123] As one possible implementation, the storage system has a master node and storage nodes. The master node controls the round-robin state transition model, and the storage nodes are responsible for data storage.

[0124] See Table 1 for specifications of the storage system parameters:

[0125] Parameter Explanation c Data stream size t Time k Number of storage nodes B t ]]> Bucket data bucket carrying data stream at time t b Data bucket size i Storage node label (i ∈ [0, k-1])

[00002] D i ]] The ith storage node WriteQueue i ]] Write queue on the ith storage node WriteThread i ]] Write thread on the ith storage node

[0126] Among them, B t This refers to the bucket that carries the data stream at time t; D i WriteQueue is the i-th storage node for persistent data storage. i WriteThread is the write queue on the i-th storage node. i This refers to the write thread on the i-th storage node.

[0127] See the master node rotation process. Figure 5 ,include:

[0128] (1) i is marked as a storage node and initialized to 0;

[0129] (2) Data flows in continuously, and at time t, the master node allocates a predefined data bucket space B. t After successful allocation, the data will be temporarily stored in B. t If allocation fails, the master node will be unable to allocate a new bucket, and the system will fail and exit.

[0130] (3) When data bucket B t Once the storage is full, calculate the storage node number i to be sent, and send the data bucket to D. i Meanwhile, in the transition phase (4); after the transmission is complete, release data bucket B. t The data writing task is completed by the node itself; if the sending fails, it will wait for a period of time and then resend (the reason for failure may be that the write work queue of the lower-level node is full or the network transmission is faulty).

[0131] (4) Increment i sequentially and open new data buckets to continuously receive data streams.

[0132] For the storage strategy of the underlying storage nodes, please refer to Figure 6 Because in the round-robin state transition model, if the number of nodes does not meet the minimum requirement, the data will temporarily reside in memory, making memory resources very precious. Therefore, this system chooses to use a single thread to complete the write throughput. The write process is as follows:

[0133] (1) Receive data bucket B from the upper layer t Next, determine if there is enough remaining memory to accommodate the data bucket. If not, send a message to the upper layer and wait for resending. If there is still remaining memory, add the data bucket to the write queue (WriteQueue). i middle;

[0134] (2) Waiting for the WriteThread i After completing the previous write task, the first bucket in the write waiting queue is retrieved and handed over to the write thread to continue completing the task.

[0135] Typically, the index information for each small file has fixed fields and lengths, as shown in Table 2:

[0136] Table 2 Small File Index Information

[0137] Field name Description Length File name hash File name hash of small file 32 bytes File size Size information of small file 32 bytes Merged file information Information about merged file 16 bytes Offset Offset position of small file 16 bytes

[0138] B+ trees are a variant of B-trees, the main difference being that all actual data is stored in the leaf nodes, while non-leaf nodes are only used for indexing. This design allows B+ trees to significantly improve query efficiency when handling large amounts of data, especially performing exceptionally well in range queries. Leaf nodes in a B+ tree are connected by linked lists, facilitating range searches and traversals. This patent uses the size information of small files as an index to construct the index system and ensures the ordered nature of the data within the index system. Simultaneously, the hash value of the small filename is stored in the metadata to ensure that small files can be uniquely identified. Through an index queue, ORFS can obtain the index information of small files and merge this index information into index files. Each index file organizes data in the form of a B+ tree. Such index files are called index data buckets. Since the metadata of each small file occupies 96 bytes, our index data buckets are also stored in ORFS in the form of data blocks. Therefore, in a system with the default data block size (128MB), one index data bucket can store information for approximately one million small files. This might not be enough for tens of millions of small files, so this patent splits the buckets and stores the metadata of each bucket in a bucket queue, ensuring the order of the index data buckets stored in the queue. This storage method increases the number of small file indexes stored in the indexing system without affecting access efficiency.

[0139] Thirdly, the present invention provides a big data processing system that uses the storage method provided in the first aspect to store massive file data streams.

[0140] In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0141] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for storing massive file data streams, characterized in that, A round-robin state transition model is deployed between data streams and storage nodes to distribute storage pressure through multi-node caching; the storage method includes the following steps: S1. Preset file size dividing point; S2. Input massive amounts of file data, and check whether the size of each input file is less than or equal to the boundary point. If so, execute S3 to S4. S3. File data less than or equal to the boundary point is stored in the selected storage node through a round-robin state transition model; The round-robin state transition model has m data buckets, which are organized into a queue; the first k data buckets in the queue are data filling buckets, k < m, and the other data buckets are idle waiting buckets. The round-robin state transition model stores file data less than or equal to the boundary point in selected storage nodes, specifically including the following steps: S30. Write file data to multiple data filling buckets in parallel; S31. When a data fill bucket is full, remove the full data fill bucket from the data fill bucket queue and move the full data fill bucket into the write waiting bucket queue. That is, the state of the full data fill bucket is changed to write waiting bucket. Remove a new bucket from the idle waiting bucket to the data fill bucket queue to accept subsequent file data writing. S32. The round-robin state transition model is configured with a data bucket scheduling module, which is used to poll storage nodes in a round-robin fashion and select storage nodes based on a load balancing strategy; S33. Using hashing, the write wait bucket is routed to the selected storage node for data dumping preparation. At this point, the state of the write wait bucket is changed to the data dump bucket. S34. After the data in the data dump bucket is written to the selected storage node, the status of the data dump bucket is changed to idle waiting bucket, waiting for task distribution; S4. Generate index information for all files less than or equal to the boundary point, and merge the index information into the index file. Each index file organizes the data in the form of a B+ tree to obtain index data buckets. Multiple index data buckets are stored in the form of data blocks on the storage node, and all index data buckets are saved sequentially as an index data bucket queue.

2. The method for storing massive file data streams according to claim 1, characterized in that, Each selected storage node corresponds to a waiting queue consisting of multiple write waiting buckets. The write waiting buckets are refreshed one by one. The write waiting buckets to be refreshed are simultaneously rewritten as data dumping buckets. The refreshed data dumping buckets are rewritten as idle waiting buckets.

3. The method for storing massive file data streams according to claim 1, characterized in that, Storage methods for storage nodes include: Receive the data dump bucket, determine if there is enough remaining memory to hold the file data in the data dump bucket. If not, send a retransmission request and wait for retransmission. If yes, add the data dump bucket to the write queue. After the write thread completes the previous write task, it retrieves the first data dump bucket from the write queue and hands it over to the write thread to complete the task of writing the file data in the data dump bucket to the storage node.

4. The method for storing massive file data streams according to claim 1, characterized in that, Check each input file size for less than or equal to the delimiter. If not, proceed with the following steps: S5. Determine whether the file data needs to be split. If yes, execute S6-S7; otherwise, execute S8. S6. Split file data; S7. Determine whether the data of the split file is less than or equal to the dividing point. If yes, execute S3 to S4; if no, execute S5 to S7; until the result of S5 is that no split is needed, then execute S8. S8. File data is stored in HDFS.

5. The method for storing massive file data streams according to any one of claims 1 to 4, characterized in that, The dividing point is 4.35 MB.

6. A storage system for massive file data streams, characterized in that, include: The input port is used to receive massive file data streams; The first judgment module is configured with a boundary point to ensure that the size of the input file is less than or equal to the boundary point. The second judgment module is used to determine whether file data larger than the dividing point needs to be split. The splitting module is used to split file data that is larger than the dividing point and needs to be split. HDFS is used to store file data that is larger than the delimiter and does not need to be split. The rotation state transition module is used to store file data that is less than or equal to the boundary point to the selected storage node; The index information generation module is used to generate index information for all files less than or equal to the boundary point, and merge the index information into the index file. Each index file organizes the data in the form of a B+ tree to obtain index data buckets. Multiple index data buckets are stored in the form of data blocks on the storage node, and all index data buckets are saved in order as an index data bucket queue. A storage node is used to store file data less than or equal to the boundary point and multiple index data buckets; the round-robin state transition module has m data buckets, which are organized into a queue; the first k data buckets in the queue are data filling buckets, k < m, and the other data buckets are idle waiting buckets; when storing file data less than the boundary point in the selected storage node through the round-robin state transition model, S30 to S34 as described in claim 1 are executed.

7. The storage system for massive file data streams according to claim 6, characterized in that, The storage system has a master node and storage nodes. The master node controls the rotation state transition model, and the storage nodes are responsible for data storage.

8. A big data processing system, characterized in that, The storage method described in any one of claims 1 to 4 is used to store massive file data streams.

Images