System and method for realizing distributed big data cluster design and deployment optimization based on hierarchical index

Abstract

The invention relates to a system for realizing distributed big data cluster design and deployment optimization based on hierarchical indexing, and the system comprises a high-speed index cluster which is used for independently storing all index data and completing the mapping processing from a query condition to a unique internal serial number of each data record; the data storage cluster is used for storing data on a plurality of data storage nodes and taking a random number prefix of a unique internal serial number as a slice key to be in one-to-one correspondence with the plurality of data storage nodes; and the master controller server (group) is used for receiving all read and write requests in the system, synchronizing the high-speed index cluster and the data storage cluster to complete the read and write requests, and returning the result of the read and write requests to a request calling party. The invention also relates to a corresponding method. By the adoption of the system and method for achieving distributed big data cluster design and deployment optimization based on hierarchical indexing, the data storage capacity of a single data node is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data cluster design, in particular to the fields of security and server clusters, and specifically refers to a system and method for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing. Background Art

[0002] Database (DB) refers to a warehouse that stores data in a certain format in a computer storage device. In order to optimize query efficiency, the database management system introduces database index, which is a sorted data structure that arranges the original irregularly arranged raw data according to a specific data structure to form a new sorted + raw data structure. This data structure is mainly based on b tree or b+tree. Query based on index can reduce the number of I / O times and speed up the retrieval speed, but there are the following problems:

[0003] -The time spent on creating and maintaining indexes increases as the amount of data increases;

[0004] -Index requires physical space. In addition to the data space occupied by the data table, each index also occupies a certain amount of physical space;

[0005] -Index will reduce the efficiency of data table modification operations (deletion, addition, modification), because the index table needs to be modified at the same time as the data table;

[0006] -Each index-based query request requires loading the index as a prerequisite;

[0007] As we all know, the efficiency of disk I / O operations is very low. When index data is stored on disk, all data cannot be loaded into memory at once during query. Disk pages can only be loaded one by one. Each disk page corresponds to a b-tree or b+tree node, resulting in a large number of disk I / O operations (the height of the tree in the worst case). When the current data volume is huge, the depth of the b-tree or b+tree is too large, which can easily cause disk I / O reading and writing to be too frequent, leading to low efficiency. Therefore, existing databases, whether distributed or traditional single-point servers, whether SQL or NOSQL, have a limited amount of data carried by a single data storage node.

[0008] With the development of electronicization and informatization, the total amount of various types of data has exploded, and traditional databases have been unable to cope with the needs of big data writing and querying; traditional database vendors, such as Oracle and IBM, have very expensive big data solutions based on traditional database management systems and are difficult to use widely; emerging distributed database solutions can only rely on linear expansion due to the limited amount of data that a single data node can carry, and deployment costs are rising. Summary of the invention

[0009] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a system and method for realizing distributed big data cluster design and deployment optimization based on hierarchical indexing.

[0010] In order to achieve the above objectives, the system and method for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing of the present invention are as follows:

[0011] The main features of the system for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing are as follows:

[0012] High-speed index cluster, used to store all index data separately and complete the mapping process from query conditions to the unique internal serial number of each data record;

[0013] A data storage cluster is used to store data on multiple data storage nodes, and uses a random number prefix of a unique internal serial number as a shard key, which corresponds one-to-one with multiple data storage nodes; and

[0014] The main controller server (group) is connected to the high-speed index cluster and the data storage cluster, and is used to receive all read and write requests in the system, and after synchronizing the high-speed index cluster and the data storage cluster to complete the read and write requests, return the results of the read and write requests to the request caller.

[0015] Preferably, the high-speed index cluster includes an index update request processing unit, an index query request processing unit, an _id_index field pair warehouse, a warehouse index module and a first network interface, wherein the index update request processing unit updates the _id_index field pair warehouse and the warehouse index module according to the received forwarded write data request and / or update data request, and the index query request processing unit converts the query condition into a set of unique internal serial numbers that meet the conditions according to the received query request, and returns it.

[0016] Preferably, when the index update request processing unit receives a data write request issued by the system, it extracts the unique internal serial number and all fields marked as indexes or composite indexes, adds them to the _id_index field pair warehouse, and uses them to update the warehouse index module; when the index update request processing unit receives a data update request, if the update field is an index or composite index field, the correspondence between the new index and the internal serial number is updated to the _id_index field pair warehouse, and then the warehouse index module is updated.

[0017] Preferably, after receiving the read request issued by the system, the index query request processing unit extracts the query conditions and sorting fields contained in the read request. If the field exceeds the index and compound index field processing range, the index query request processing unit directly returns empty; if the field is within the index and compound index field processing range, the index query request processing unit searches for qualified data records in the warehouse index module and returns them according to the sorted combination serial number result set.

[0018] Preferably, the data storage cluster includes a read engine, a write engine, a data warehouse, a serial number (_id) index library and a second network interface, wherein:

[0019] The reading engine is used to read the data record and return the result after completing the read request operation;

[0020] The write engine is used to return a success or failure result to the system after completing the write request operation;

[0021] The serial number (_id) index library is used to help the data warehouse efficiently obtain the data structure of the data according to the internal serial number.

[0022] Preferably, when the reading engine receives a query request based on the internal sequence number of the system, the reading engine queries the sequence number (_id) index library 340 and searches for the data record storage offset address corresponding to the internal sequence number, and reads the data record from the address and returns it; when the reading engine receives a request containing other query conditions besides the sequence number, the reading engine traverses the data warehouse to search for records that meet the conditions and returns them;

[0023] When the write engine receives a write request from the system, the write engine adds the new data record to the end of the data warehouse and updates the serial number (_id) index library before returning.

[0024] Preferably, the master controller server (group) includes a read request processing module, a write request processing module, a serial number generation strategy module and a third network interface, wherein the read request processing module and the write request processing module are responsible for receiving all read and write requests and returning the results to the caller; the serial number generation strategy module is used to generate a unique serial number within the system.

[0025] Preferably, the system also includes a data storage control cluster, which includes a sub-controller, a metadata cluster and a fourth network interface, wherein the sub-controller is responsible for receiving all read and write requests forwarded by the main controller server (group) and returning the results to the main controller server (group), and the metadata cluster is used to create and maintain the mapping relationship between the serial number prefix and the data node.

[0026] The method for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing using the above system is mainly characterized in that the method comprises the following steps:

[0027] (1) In the system, a high-speed index cluster is configured to store data index information, and a data storage cluster is configured to store data on multiple data storage nodes;

[0028] (2) The high-speed index cluster completes the mapping of each data record to the internal sequence number according to the query conditions; the data storage cluster completes the loading of the internal sequence number to each data;

[0029] (3) configuring the master controller server (group) to receive all read and write requests in the system, and synchronize the high-speed index cluster and the data storage cluster to complete the read and write requests, and return the results of the read and write requests to the request caller;

[0030] (4) The data storage cluster is configured as a distributed data storage cluster, data is stored on multiple data storage nodes, and the serial number generation strategy of the master controller server (group) is configured to ensure the sequential growth of the serial number on each data node.

[0031] Preferably, the step (3) is specifically as follows:

[0032] (3.1) After receiving the write request, the master controller server (group) updates the index library on the high-speed index cluster and writes the data into the data storage cluster;

[0033] (3.2) After receiving the read request, the master controller server (group) searches for all serial numbers that meet the query conditions according to the high-speed index cluster and sorts and combines them, and then sends them to the data storage cluster, and returns them to the caller after the data is loaded.

[0034] The system and method for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing of the present invention are adopted. By introducing a shard key generation algorithm and a high-speed index cluster, the index tree of the index field is avoided to be maintained on the ordinary disk of the data unit, thereby effectively avoiding a large number of random read and write operations of the ordinary disk caused by the query and update of the index tree, and greatly improving the data storage capacity of a single data node. According to actual measurements, in the actual application of this technical solution, the data carrying capacity of its data node can be increased from about 20 to 30 million to the order of magnitude of nearly 100 million. Therefore, under the same data carrying demand, the number of servers required to use the system described in this technical solution is only about 30% of the number of servers required for an ordinary system, which has certain technical advantages and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of the structure of the system for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing in the present invention. DETAILED DESCRIPTION

[0036] In order to more clearly describe the technical content of the present invention, further description is given below in conjunction with specific embodiments.

[0037] Before describing in detail embodiments according to the present invention, it should be noted that, hereinafter, the terms "comprises", "includes" or any other variations are intended to cover non-exclusive inclusion, whereby a process, method, article or apparatus comprising a series of elements includes not only these elements, but also other elements not explicitly listed or inherent to such process, method, article or apparatus.

[0038] See also Figure 1 As shown, the system for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing, wherein the system includes:

[0039] High-speed index cluster, used to store all index data separately and complete the mapping process from query conditions to the unique internal serial number of each data record;

[0040] A data storage cluster is used to store data on multiple data storage nodes, and uses a random number prefix of a unique internal serial number as a shard key, which corresponds one-to-one with multiple data storage nodes; and

[0041] The main controller server (group) is connected to the high-speed index cluster and the data storage cluster, and is used to receive all read and write requests in the system, and after synchronizing the high-speed index cluster and the data storage cluster to complete the read and write requests, return the results of the read and write requests to the request caller.

[0042] As a preferred embodiment of the present invention, the high-speed index cluster includes an index update request processing unit, an index query request processing unit, an _id_index field pair warehouse, a warehouse index module and a first network interface, wherein the index update request processing unit updates the _id_index field pair warehouse and the warehouse index module according to the received forwarded write data request and / or update data request, and the index query request processing unit converts the query condition into a set of unique internal serial numbers that meet the conditions according to the received query request, and returns it.

[0043] As a preferred embodiment of the present invention, when the index update request processing unit receives a data write request issued by the system, it extracts the unique internal serial number and all fields marked as indexes or composite indexes, adds them to the _id_index field pair warehouse, and uses them to update the warehouse index module; when the index update request processing unit receives a data update request, if the update field is an index or composite index field, the correspondence between the new index and the internal serial number is updated to the _id_index field pair warehouse, and then the warehouse index module is updated.

[0044] As a preferred embodiment of the present invention, after receiving the read request issued by the system, the index query request processing unit extracts the query conditions and sorting fields contained in the read request. If the field exceeds the index and compound index field processing range, the index query request processing unit directly returns empty; if the field is within the index and compound index field processing range, the index query request processing unit queries the warehouse index module for data records that meet the conditions and returns them according to the sorted combination serial number result set.

[0045] As a preferred embodiment of the present invention, the data storage cluster includes a read engine, a write engine, a data warehouse, a sequence number (_id) index library and a second network interface, wherein:

[0046] The reading engine is used to read the data record and return the result after completing the read request operation;

[0047] The write engine is used to return a success or failure result to the system after completing the write request operation;

[0048] The serial number (_id) index library is used to help the data warehouse efficiently obtain the data structure of the data according to the internal serial number.

[0049] As a preferred embodiment of the present invention, when the reading engine receives a query request based on the internal sequence number of the system, the reading engine queries the sequence number (_id) index library 340 and searches for the data record storage offset address corresponding to the internal sequence number, and reads the data record from the address and returns it; when the reading engine receives a request containing other query conditions besides the sequence number, the reading engine traverses the data warehouse to search for records that meet the conditions and returns them;

[0050] When the write engine receives a write request from the system, the write engine adds the new data record to the end of the data warehouse and updates the serial number (_id) index library before returning.

[0051] As a preferred embodiment of the present invention, the master controller server (group) includes a read request processing module, a write request processing module, a serial number generation strategy module and a third network interface, wherein the read request processing module and the write request processing module are responsible for receiving all read and write requests and returning the results to the caller; the serial number generation strategy module is used to generate a unique serial number within the system.

[0052] As a preferred embodiment of the present invention, the system also includes a data storage control cluster, which includes a sub-controller, a metadata cluster and a fourth network interface, wherein the sub-controller is responsible for receiving all read and write requests forwarded by the master controller server (group) and returning the results to the master controller server (group), and the metadata cluster is used to create and maintain the mapping relationship between the serial number prefix and the data node.

[0053] In practical applications, please refer to Figure 1 As shown, the system for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing of the present technical solution includes a general controller server (group) 100, a data storage control cluster 200, a data storage cluster 300 and a high-speed index cluster 400.

[0054] The master controller server (group) 100 includes a read request processing module 110, a write request processing module 120, a sequence number generation strategy module 130 and a third network interface 140. The read request processing module 110 and the write request processing module 120 are responsible for receiving all read and write requests and returning the results to the caller; the sequence number generation strategy module 130 generates an internal unique sequence number (_id).

[0055] After the main controller server (group) 100 receives the read request, the read request processing module 110 extracts the query conditions and sorting fields contained in the read request. If these fields exceed the processing range of the index and compound index fields, the read request processing module 110 sends the read request directly to the data storage control cluster 200 through the network interface 130. After the data storage control cluster 200 completes the query, it returns the result to the read request processing module 110; if these fields are within the processing range of the index and compound index fields, the read request processing module 110 sends the read request to the high-speed index cluster 400. The index query request processing 420 efficiently extracts the serial numbers (_id) of all qualified records according to the query conditions and sorting fields in the read request and returns them to the read request processing module 110 in the desired sorting combination. The read request processing module 110 pushes the serial number (_id) set to the data storage control cluster 200, and the sub-controller 210 completes the query and returns the final data result to the read request processing module 110.

[0056] After the main controller server (group) 100 receives the write request, the write request processing module 120 distributes the request to the data storage control cluster controller 200 and the high-speed index cluster 400. After these two modules have processed the request successfully, they return a write success to the request caller; if any module fails to process, the other side returns a write failure to the request caller after the operation is rolled back.

[0057] In a recommended implementation of the present invention, the master controller server 100 is deployed in a mode of a series server (group) cluster followed by a high availability (HA) distributor to cope with a huge amount of concurrent read and write requests.

[0058] In a recommended implementation of the present invention, the serial number (_id) is a unique serial number of internal data, which is invisible to the outside and includes a prefix and a suffix. The prefix is ​​a uniformly distributed random number, and the suffix is ​​a sequentially increasing serial number.

[0059] In a recommended implementation of the present invention, the serial number (_id) may also be specified by an external application, which ensures that the serial number (_id) is globally unique and includes a uniformly distributed random number prefix and a sequentially increasing serial number suffix.

[0060] In a recommended implementation of the present invention, the third network interface 140 uses an optical module interface to cope with the situation of writing and reading a large amount of data, so as to avoid the network becoming a system bottleneck.

[0061] The data storage control cluster 200 includes a sub-controller 210, a metadata cluster 220 and a fourth network interface 230. The sub-controller 210 is responsible for receiving all read and write requests forwarded by the controller 100 and returning the results to the controller 100. The metadata cluster creates and maintains the mapping relationship between the serial number (_id) prefix (shard key) and the data node.

[0062] After the sub-controller 210 receives the forwarded read request, if the query is a query based on the serial number (_id), that is, the query and result sorting have been completed and converted into a qualified serial number (_id) set through the high-speed index cluster, the sub-controller 210 queries the metadata cluster 220 for the data storage node where the serial number (_id) is located and sends the query based on the serial number (_id) separately to the data storage node 300 where it is located and returns the result; if the query request contains query conditions that are not serial numbers (_id), that is, the query conditions exceed the index / compound index field range and cannot be converted into a qualified serial number (_id) set through the high-speed index cluster, the data storage control cluster 200 broadcasts the query request to all data storage nodes 300, performs an inefficient scanning query, and then returns the query results after comprehensive sorting.

[0063] After receiving the write request, the sub-controller 210 queries the metadata cluster 220 for the data storage node corresponding to the serial number (_id) prefix and sends the write request to the data storage node separately and returns the write result.

[0064] In a recommended implementation of the present invention, the data storage control cluster 200 can be configured to achieve a one-to-one correspondence between the prefix (shard key) of the serial number (_id) and the data storage node serial number. When the prefix of the serial number (_id) of the written record data is a uniformly distributed random number and the start and end ranges are equal to the start and end ranges of the data storage node serial numbers, writing records can achieve automatic load balancing to ensure that data is evenly distributed to all data nodes.

[0065] In a recommended implementation of the present invention, the data storage control cluster 200 and the data storage cluster 300 are implemented by Hadoop+Hbase.

[0066] In a recommended implementation of the present invention, the data storage control cluster 200 and the data storage cluster 300 are implemented by distributed clusters such as MongoDB / Redis / ScyllaDB / PostgreSQL / CockroachDB.

[0067] The data cluster node 300 includes a read engine 310 , a write engine 320 , a data warehouse 330 , a sequence number (_id) index library 340 , and a second network interface 350 .

[0068] The write engine 310 services the write request and returns success or failure.

[0069] The read engine 320 services the read request, reads the data record and returns it.

[0070] The serial number (_id) index library 340 is a data structure that helps the data warehouse 330 to efficiently obtain data based on the serial number (_id), and is also the unique index of the data warehouse. When the reading engine 310 receives a query request based on the serial number (_id), the reading engine 310 queries the serial number (_id) index library 340, finds the data record storage offset address corresponding to the serial number (_id), reads the data record from the address and returns it; when the reading engine 310 receives a request containing query conditions other than the serial number (_id), the reading engine 310 traverses the data warehouse 320, finds records that meet the conditions and returns them.

[0071] When the write engine 320 receives a write request, the write engine 320 adds the new record to the end of the data warehouse 330 and updates the sequence number (_id) index library 340 before returning;

[0072] In a recommended implementation of the present invention, the data storage node 300 may be deployed in primary-secondary / dual-copy / multiple-copy mode to achieve automatic failover and disaster recovery during data storage.

[0073] In a recommended implementation of the present invention, the serial number (_id) is a unique serial number generated by the serial number (_id) generation strategy module 130 or an external application according to a specified strategy, and includes a prefix and a suffix, and the suffix is ​​a sequentially increasing serial number. Therefore, the serial numbers (_id) of all records in the same data storage node are sequentially increasing, and the reconstruction of the serial number (_id) index library 340 due to the writing of new data is avoided, thereby greatly improving the processing capacity of continuous concurrent writes that a single data node can withstand when it is already carrying tens of millions of data records.

[0074] In a recommended implementation of the present invention, the data storage node 300 is composed of low-cost SATA HDD disks and servers with poor CPU performance to reduce the total cost of cluster deployment;

[0075] In a recommended implementation of the present invention, the serial numbers (_id) of all records in the data storage cluster 300 increase sequentially, so it has the characteristic of continuous high concurrent writing under high data capacity. However, when loading the serial number (_id) data, it is still necessary to randomly load the serial number (_id) index library, so it is suitable for application scenarios where a large amount of data is written and limited concurrent queries are required.

[0076] The high-speed index cluster 400 includes an index update request processing unit 410, an index query request processing unit 420, an _id_index field pair warehouse 430, a warehouse index library 440, and a first network interface 450. The index update request processing unit 410 updates the _id_index field pair warehouse 430 and the warehouse index library 440 according to the received and forwarded write and update data requests, and the index query request processing unit 420 receives the query request, converts the query condition into a set of sequence numbers (_id) that meet the condition and returns it.

[0077] When the index update request processing unit 410 receives a data write request, it extracts the serial number (_id) and all fields marked as indexes or composite indexes, adds them to the _id_index field pair warehouse 430, and then updates the warehouse index library 440; when receiving a data update request, if the update field is an index or composite index field, the correspondence between the new index and the serial number (_id) is updated to the _id_index field pair warehouse 430, and then the warehouse index library 440 is updated.

[0078] After receiving the read request, the index query request processing unit 420 extracts the query conditions and sorting fields contained in the read request. If these fields are beyond the processing range of the index and compound index fields, the index query request processing unit 420 directly returns empty; if these fields are within the processing range of the index and compound index fields, the index query request processing unit 420 queries the warehouse index library 440, finds the records that meet the query conditions and returns them according to the sorted combined serial number (_id) result set.

[0079] In a recommended implementation of the present invention, the high-speed index cluster 400 is composed of SSD disks and servers with strong CPU performance and more memory, such as 512G, to complete efficient index query and update;

[0080] In a recommended implementation of the present invention, the fields that need to be queried, filtered, and sorted account for a very small proportion, and a high-performance, high-cost high-speed index cluster requires a small number of servers to complete deployment. A large number of data storage server clusters are constructed by relatively low-cost storage servers, thereby effectively reducing the overall deployment cost of the system.

[0081] In a recommended implementation of the present invention, the high-speed index cluster 400 is composed of a memory cluster to achieve more efficient query and update.

[0082] The method for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing using the above-mentioned system, wherein the method comprises the following steps:

[0083] (1) In the system, a high-speed index cluster is configured to store data index information, and a data storage cluster is configured to store data on multiple data storage nodes;

[0084] (2) The high-speed index cluster completes the mapping of each data record to the internal sequence number according to the query conditions; the data storage cluster completes the loading of the internal sequence number to each data;

[0085] (3) configuring the master controller server (group) to receive all read and write requests in the system, and synchronize the high-speed index cluster and the data storage cluster to complete the read and write requests, and return the results of the read and write requests to the request caller;

[0086] (4) The data storage cluster is configured as a distributed data storage cluster, data is stored on multiple data storage nodes, and the serial number generation strategy of the master controller server (group) is configured to ensure the sequential growth of the serial number on each data node.

[0087] As a preferred embodiment of the present invention, the step (3) is specifically as follows:

[0088] (3.1) After receiving the write request, the master controller server (group) updates the index library on the high-speed index cluster and writes the data into the data storage cluster;

[0089] (3.2) After receiving the read request, the master controller server (group) searches for all serial numbers that meet the query conditions according to the high-speed index cluster and sorts and combines them, and then sends them to the data storage cluster, and returns them to the caller after the data is loaded.

[0090] In a specific embodiment of the present invention, the method for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing specifically includes the following steps:

[0091] 1. Configure two clusters, the index cluster (high-speed cluster) and the data cluster (ordinary cluster), which are used to store data index information and data itself respectively.

[0092] -The index cluster is used to store all index data separately; the data cluster stores the actual data records, each of which contains a unique internal serial number (_id). The index cluster completes the mapping of query conditions to _id; the data cluster completes the loading of _id to data.

[0093] 2. Configure the master controller to receive all read and write requests, synchronize the two clusters to complete the read and write requests, and return the results of the read and write requests to the request caller;

[0094] -After receiving the write request, the master controller updates the index library on the index cluster and then writes the data to the data cluster;

[0095] -After receiving the read request, the main controller searches for all serial numbers (_id) that meet the query conditions according to the index cluster and combines them in order, then sends them to the data cluster, loads the data, and returns to the caller.

[0096] 3. Configure the data cluster as a distributed data storage cluster and store data on multiple data storage nodes to ensure high-concurrency write processing, large-capacity data storage, and linear system expansion. At the same time, by configuring the _id generation strategy on the master controller, ensure that the _id on each data node grows sequentially, ensuring that a single data node still maintains extremely high write processing capabilities under large data volumes.

[0097] In a typical application scenario of the present invention, the data structure is determined by the national standard (GB / T) or the industry standard (GA / T) specification and will not be increased or decreased at will. However, complex engineering situations occur after repeated reconstruction of the index cluster and the data storage control cluster.

[0098] In a typical application scenario of the present invention, the data structure contains a large number of auxiliary information fields that do not need to be queried and a small number of fields that need to be queried, such as time. For example, in the GA / T 1400 video image information application system, face objects and vehicle objects contain a large amount of auxiliary field information, but the fields used for query filtering are limited to a limited number of fields such as time and device ID.

[0099] In a typical application scenario of the present invention, data writing is a large-scale, high-concurrency intensive operation, while data reading is a limited operation. For example, a large number of servers write log data intensively and concurrently into a central log repository, but the business operations to apply these data are all manual operations, and the number of requests is relatively limited.

[0100] Furthermore, the deployment optimization method of this technical solution is as follows:

[0101] 1. Configure a high-speed cluster to store a large amount of frequently written / read index data, and store ordinary data in an ordinary data cluster.

[0102] 2. Configure the master controller to control the two clusters to complete read and write synchronization requests. When processing write requests, the two clusters are updated synchronously to complete the write. When processing read requests, all internal serial numbers (_id) that meet the query conditions are searched according to the index cluster and sorted and combined according to the expected order. Then, they are sent to the data cluster, loaded with data, and returned to the caller.

[0103] 3. Configure a distributed data cluster, using a random number prefix of a unique internal serial number (_id) as the shard key, corresponding to the data storage node one by one;

[0104] 4. Configure the internal serial number (_id) generation strategy on the master controller to ensure that its prefix is ​​a uniformly distributed random number, implement distributed request distribution for read and write requests, and achieve linear parallel expansion of the concurrent read and write processing capabilities of the entire cluster and the affordable data capacity;

[0105] 5. Configure the internal serial number (_id) generation strategy on the main controller to ensure that its suffix is ​​a sequentially increasing serial number, realize the sequential growth of the internal serial number (_id) of each record on each data storage node, eliminate the reconstruction update of the _id index library caused by the addition of new records on the data storage node, and realize that the write processing capability of a single data storage node will not be attenuated when carrying large-capacity data.

[0106] Furthermore, the high-speed cluster is composed of high-performance servers to achieve efficient query and update.

[0107] Furthermore, each data record contains a unique internal serial number (_id), which contains a prefix and a suffix. The prefix is ​​a uniformly distributed random number, and the suffix is ​​a sequentially increasing serial number. The index cluster completes the mapping of query conditions to _id; the data cluster completes the loading of _id to data;

[0108] Furthermore, the software on the data storage node eliminates possible performance bottlenecks of writing data. Therefore, if the read request demand is not high, the data storage node can be built with low-cost servers to reduce the overall deployment cost.

[0109] On this basis, the present invention proposes a distributed big data cluster design and deployment optimization method and system with hierarchical indexing for high concurrent writing and reading platform applications with large data volumes, which isolates most index data from ordinary data and implements hierarchical storage. The present invention has the following features:

[0110] 1. Maintain high write performance under the premise of high data load;

[0111] -When the network throughput meets the requirements, the write performance of the entire distributed cluster is a linear superposition of the write capacity of a single data node. The present invention relates to a competitive distributed big data cluster design and deployment optimization method and system that can greatly provide the concurrent write processing capability of a single data node, especially when it already carries a large amount of data. Therefore, it can effectively improve the data carrying capacity of the entire cluster and the concurrent write processing capability under the premise of high data carrying capacity will not decay with the accumulation of data.

[0112] 2.Higher readout performance;

[0113] -Data storage clusters that store large amounts of data only need to load data based on internal sequence numbers, thus improving query efficiency;

[0114] 3. Low-cost facilities. Index data that is frequently read and updated is stored in a cluster with higher performance, and ordinary data is stored in a cluster with lower performance and lower cost. This reduces the overall deployment cost of the system without affecting performance.

[0115] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0116] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.

[0117] A person of ordinary skill in the art may understand that all or part of the steps of the method for implementing the above-mentioned embodiment may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0118] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0119] In the description of this specification, the description with reference to the terms "an embodiment", "some embodiments", "example", "specific example", or "embodiment" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0120] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

[0121] The system and method for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing of the present invention are adopted. By introducing a shard key generation algorithm and a high-speed index cluster, the index tree of the index field is avoided to be maintained on the ordinary disk of the data unit, thereby effectively avoiding a large number of random read and write operations of the ordinary disk caused by the query and update of the index tree, and greatly improving the data storage capacity of a single data node. According to actual measurements, in the actual application of this technical solution, the data carrying capacity of its data node can be increased from about 20 to 30 million to the order of magnitude of nearly 100 million. Therefore, under the same data carrying demand, the number of servers required to use the system described in this technical solution is only about 30% of the number of servers required for an ordinary system, which has certain technical advantages and practical value.

[0122] In this specification, the present invention has been described with reference to specific embodiments thereof. However, it is apparent that various modifications and variations may be made without departing from the spirit and scope of the present invention. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A system for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing, characterized in that: The system comprises: High-speed index cluster, used to store all index data separately and complete the mapping process from query conditions to the unique internal serial number of each data record; A data storage cluster is used to store data on multiple data storage nodes, and uses a random number prefix of a unique internal serial number as a shard key, which corresponds one-to-one with multiple data storage nodes; and The main controller server (group) is connected to the high-speed index cluster and the data storage cluster, and is used to receive all read and write requests in the system, and after synchronizing the high-speed index cluster and the data storage cluster to complete the read and write requests, return the results of the read and write requests to the request caller.

2. The system for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing according to claim 1 is characterized in that: The high-speed index cluster includes an index update request processing unit, an index query request processing unit, an _id_index field pair warehouse, a warehouse index module and a first network interface, wherein the index update request processing unit updates the _id_index field pair warehouse and the warehouse index module according to the received forwarded write data request and / or update data request, and the index query request processing unit converts the query condition into a set of unique internal serial numbers that meet the conditions according to the received query request, and returns it.

3. The system for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing according to claim 2 is characterized in that: When the index update request processing unit receives a data write request issued by the system, it extracts the unique internal serial number and all fields marked as indexes or composite indexes, adds them to the _id_index field pair warehouse, and uses them to update the warehouse index module; when the index update request processing unit receives a data update request, if the update field is an index or composite index field, the correspondence between the new index and the internal serial number is updated to the _id_index field pair warehouse, and then the warehouse index module is updated.

4. The system for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing according to claim 2 is characterized in that: After receiving the read request sent by the system, the index query request processing unit extracts the query condition and sorting field contained in the read request. If the field exceeds the processing range of the index or composite index field, the index query request processing unit directly returns null. If the field is within the processing range of the index and composite index field, the index query request processing unit searches for eligible data records in the warehouse index module and returns the result set according to the sorted combination serial number.

5. The system for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing according to claim 2 is characterized in that: The data storage cluster includes a read engine, a write engine, a data warehouse, a serial number (_id) index library and a second network interface, wherein: The reading engine is used to read the data record and return the result after completing the read request operation; The write engine is used to return a success or failure result to the system after completing the write request operation; The serial number (_id) index library is used to help the data warehouse efficiently obtain the data structure of the data according to the internal serial number.

6. The system for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing according to claim 5 is characterized in that: When the reading engine receives a query request based on the internal sequence number of the system, the reading engine queries the sequence number (_id) index library 340 and searches for the data record storage offset address corresponding to the internal sequence number, and reads the data record from the address and returns it; When the reading engine receives a request containing other query conditions except the sequence number, the reading engine traverses the data warehouse to find records that meet the conditions and returns them; When the write engine receives a write request from the system, the write engine adds the new data record to the end of the data warehouse and updates the serial number (_id) index library before returning.

7. The system for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing according to claim 5 is characterized in that: The master controller server (group) includes a read request processing module, a write request processing module, a serial number generation strategy module and a third network interface, wherein the read request processing module and the write request processing module are responsible for receiving all read and write requests and returning the results to the caller; the serial number generation strategy module is used to generate a unique serial number within the system.

8. The system for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing according to claim 7 is characterized in that: The system also includes a data storage control cluster, which includes a sub-controller, a metadata cluster and a fourth network interface, wherein the sub-controller is responsible for receiving all read and write requests forwarded by the main controller server (group) and returning the results to the main controller server (group), and the metadata cluster is used to create and maintain the mapping relationship between the serial number prefix and the data node.

9. A method for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing using the system of claim 8, characterized in that: The method comprises the following steps: (1) In the system, a high-speed index cluster is configured to store data index information, and a data storage cluster is configured to store data on multiple data storage nodes; (2) The high-speed index cluster completes the mapping of each data record to the internal sequence number according to the query conditions; the data storage cluster completes the loading of the internal sequence number to each data; (3) configuring the master controller server (group) to receive all read and write requests in the system, and synchronize the high-speed index cluster and the data storage cluster to complete the read and write requests, and return the results of the read and write requests to the request caller; (4) The data storage cluster is configured as a distributed data storage cluster, data is stored on multiple data storage nodes, and the serial number generation strategy of the master controller server (group) is configured to ensure the sequential growth of the serial number on each data node.

10. The method for implementing distributed big data cluster design and deployment optimization based on hierarchical indexing according to claim 9, characterized in that: The step (3) is specifically as follows: (3.1) After receiving the write request, the master controller server (group) updates the index library on the high-speed index cluster and writes the data into the data storage cluster; (3.2) After receiving the read request, the master controller server (group) searches for all serial numbers that meet the query conditions according to the high-speed index cluster and sorts and combines them, and then sends them to the data storage cluster, and returns them to the caller after the data is loaded.

Images