Method and system for realizing data archiving processing based on multiple distributed cluster groups

Abstract

The invention relates to a method for achieving data archiving processing based on multiple distributed cluster groups. The method comprises the following steps that a hot data cluster is configured to be used for storing hot data of a platform, and an archiving / dormancy cluster is configured to be used for storing archiving data; the hot data and the archived data are subjected to full-process data life cycle flow configuration processing; the configuration master controller receives all read and write requests of the system, and returns returned data to a calling party after request positioning is completed; and performing internal configuration processing on the archiving / dormancy cluster to complete storage, retrieval and retrieval of archived data. The invention also relates to a corresponding system. The method and the system for realizing data archiving processing based on the multiple distributed cluster groups have extremely high robustness, economical efficiency and high efficiency, support data capacity without an upper limit, and can ensure that irrelevant data nodes are kept in a dormant state while providing archived data retrieval and retrieval services.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed cluster processing, and in particular to data archiving processing, and specifically refers to a method and system for implementing data archiving processing based on multiple distributed cluster groups. Background Art

[0002] The lifecycle management of massive relational (SQL) and non-relational (NOSQL) data involves data creation, reading, updating and deletion (CRUD) operations, as well as data backup, recovery and migration. This type of data lifecycle management is the basis for digital archive systems that implement digital asset access, application, archiving, offline, sealing, retrieval, and online re-application. Currently, there are many types of massive data lifecycle management in the industry that can achieve similar migration, archiving, retrieval, and online re-application functions, but the following problems are common:

[0003] - To apply archived data again, you need to enable the archive device first, including connecting the server to storage devices such as disks and tapes; reading the data from the storage device and importing it into the application system before applying the data. If there is a problem in any link of this processing line, such as incompatibility between the server and storage devices such as disks and tapes, incompatibility of interface protocols, incompatibility between the data format read by the server and the new application system, etc., the application of archived data will fail;

[0004] -The data loading and application process is slow, and the data needs to be imported as a whole before being applied;

[0005] -Archived data is imported into the application system as external data and cannot form seamless integrated data with existing platform data.

[0006] With the advent of the big data era, the total amount of various types of data has exploded, and the demand for occasional access to archived data from many years ago has become increasingly strong. Business needs are concentrated on ultra-large capacity, multiple occasional access, short access time, and unlimited traceability. Traditional data backup, recovery, and migration libraries are no longer able to cope with this. Traditional database vendors, such as Oracle and IBM, have very expensive big data lifecycle management solutions based on traditional database management systems, which cannot flexibly respond to the above needs and are difficult to be widely used. Emerging distributed database solutions also only support passive data import and export, making it difficult to cope with flexible data role switching online and offline at any time. Summary of the invention

[0007] The purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art and provide a method and system for realizing data archiving processing based on multiple distributed cluster groups.

[0008] In order to achieve the above-mentioned purpose, the method and system for realizing data archiving processing based on multiple distributed cluster groups of the present invention are as follows:

[0009] The method for realizing data archiving processing based on multiple distributed cluster groups has the following main features:

[0010] (1) Configure a hot data cluster to store the hot data of the platform, and configure an archive / dormant cluster to store archive data;

[0011] (2) The hot data and archived data are all processed by full-process data life cycle flow configuration;

[0012] (3) Configure the master controller to receive all read and write requests from the system, and return the returned data to the caller after completing the request positioning;

[0013] (4) Performing internal configuration processing on the archive / dormant cluster to complete the storage, retrieval and review of archived data.

[0014] Preferably, the step (2) is specifically as follows:

[0015] The hot data and archived data are accessed, processed, archived in hot data clusters, migrated to archive / dormant clusters after cooling down, archive / dormant clusters enter a dormant state, archive / dormant clusters are awakened when needed and connected to the main controller cluster to provide data services, and continue to enter a dormant state after the services are completed, including full-process data lifecycle flow configuration processing.

[0016] Preferably, the step (3) specifically comprises the following steps:

[0017] After receiving the read request, the master controller described in (3.1) locates the corresponding request to the hot data cluster and the archive / dormant cluster respectively according to the internal master metadata cluster, and then returns the returned data to the caller;

[0018] (3.2) After receiving the write request, the master controller forwards the write request to the hot data cluster to complete the write service, and then returns the corresponding write service execution result to the application caller;

[0019] The master controller described in (3.3) performs automatic data migration and metadata update according to the data lifecycle flow configuration to ensure that data read requests and write requests are located on the correct cluster.

[0020] Preferably, the step (4) comprises:

[0021] A distributed data storage cluster consisting of an index cluster, a metadata cluster and multiple data nodes is configured in the archive / hibernate cluster, wherein the index cluster is a fast wake-up cluster, and the metadata cluster and the distributed data storage cluster are slow clusters, which are respectively used to store data index information and the data itself; and a sub-controller is also configured in the archive / hibernate cluster.

[0022] Preferably, the step (4) comprises performing the following processing:

[0023] Each data record of the archive / dormant cluster contains a unique internal serial number (_id), the index cluster completes the mapping of query conditions to _id, and the metadata cluster completes the loading of _id to data;

[0024] The sub-controller is configured to receive all read requests, and after synchronizing the index cluster and metadata cluster to complete the read request for loading archived data, the result of the read request is returned to the request caller. After the main controller receives the read request, it searches for all serial numbers (_id) that meet the query conditions according to the index cluster and sorts them, and then sends them to the metadata cluster for data loading and returns them to the caller.

[0025] Preferably, the archive / dormant cluster is internally configured as a distributed data storage cluster for storing data on multiple data nodes to achieve separation of massive data.

[0026] The main feature of the system for implementing the above-mentioned method and realizing data archiving processing based on multiple distributed cluster groups is that the system includes:

[0027] Hot data cluster, used to store the hot data of the platform;

[0028] Archive / hibernation clusters for storage, retrieval, and review of archived data; and

[0029] The main controller is connected to the hot data cluster and the archive / dormant cluster, and is used to receive the read and write requests in the system and return the corresponding execution results to the caller.

[0030] Preferably, the hot data cluster is specifically:

[0031] Implemented by a distributed cluster of MongoDB, Redis, ScyllaDB, PostgreSQL, or CockroachDB;

[0032] It consists of SSD disks and servers to complete frequent queries on hot data and real-time insertion of new data.

[0033] Preferably, the archive / hibernation cluster includes: a sub-controller, an index cluster, a metadata cluster, multiple data nodes and a network interface, wherein the sub-controller is used to receive all read and write requests forwarded by the main controller and return the results to the main controller, the index cluster converts the query conditions and result sorting into a set of data unique serial numbers (_id), and the metadata cluster is used to create and maintain the mapping relationship between the unique serial number (_id) and the data node.

[0034] Preferably, the unique serial number (_id) is a unique serial number generated by the metadata cluster according to a specified strategy, including a prefix and a suffix, the prefix is ​​the shard key, which is a uniform random number in the range of 1-N of the data node serial number, and the suffix is ​​a sequentially increasing serial number, which is a unique index of each internal data node.

[0035] Preferably, the master controller includes a read request processing module, a write request processing module, a metadata library and a network interface, wherein the read request processing module and the write request processing module are used to receive all read and write requests in the system and return the results to the caller; the metadata library is used to maintain the total cluster index and forward the read and write requests of data within different time ranges to different clusters.

[0036] Preferably, the hot data cluster and archive / dormant cluster are sharded according to time, and the sharding information is stored in the metadata database. The read request processing module queries the metadata database to forward different requests unidirectionally to the corresponding cluster.

[0037] Preferably, the network interfaces all use optical module interfaces to cope with large amounts of data writing and reading.

[0038] The method and system for realizing data archiving processing based on multiple distributed cluster groups of the present invention have the following technical advantages:

[0039] 1. Public security data with high reliability includes data that can be archived and backed up, that is, a digital archive system for the access, application, archiving, offline, sealing, retrieval, and online application of various types of public security data;

[0040] 2. When searching, the system can query whether this part of data (file) is in the current archive room, just like searching the file details of the traditional archive room;

[0041] 3. When accessing, the system can locate a certain file cabinet and load this part of data (files) just like accessing files in a traditional archive room;

[0042] 4. Like traditional archives rooms, electronic archives rooms have national standard safety protection plans such as fire prevention, waterproofing, and earthquake resistance;

[0043] 5. When a city-level earthquake, flood, or regional fire occurs, the electronic archive room can put historical data online. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the structure of the system for realizing data archiving processing based on multiple distributed cluster groups of the present invention. DETAILED DESCRIPTION

[0045] In order to more clearly describe the technical content of the present invention, further description is given below in conjunction with specific embodiments.

[0046] Before describing in detail embodiments according to the present invention, it should be noted that, hereinafter, the terms "comprises", "includes" or any other variations are intended to cover non-exclusive inclusion, whereby a process, method, article or apparatus comprising a series of elements includes not only these elements, but also other elements not explicitly listed or inherent to such process, method, article or apparatus.

[0047] The method for implementing data archiving processing based on multiple distributed cluster groups, wherein the method comprises the following steps:

[0048] (1) Configure a hot data cluster to store the hot data of the platform, and configure an archive / dormant cluster to store archive data;

[0049] (2) The hot data and archived data are all processed by full-process data life cycle flow configuration;

[0050] (3) Configure the master controller to receive all read and write requests from the system, and return the returned data to the caller after completing the request positioning;

[0051] (4) Performing internal configuration processing on the archive / dormant cluster to complete the storage, retrieval and review of archived data.

[0052] As a preferred embodiment of the present invention, the step (2) is specifically as follows:

[0053] The hot data and archived data are accessed, processed, archived in hot data clusters, migrated to archive / dormant clusters after cooling down, archive / dormant clusters enter a dormant state, archive / dormant clusters are awakened when needed and connected to the main controller cluster to provide data services, and continue to enter a dormant state after the services are completed, including full-process data lifecycle flow configuration processing.

[0054] As a preferred embodiment of the present invention, the step (3) specifically comprises the following steps:

[0055] After receiving the read request, the master controller described in (3.1) locates the corresponding request to the hot data cluster and the archive / dormant cluster respectively according to the internal master metadata cluster, and then returns the returned data to the caller;

[0056] (3.2) After receiving the write request, the master controller forwards the write request to the hot data cluster to complete the write service, and then returns the corresponding write service execution result to the application caller;

[0057] The master controller described in (3.3) performs automatic data migration and metadata update according to the data lifecycle flow configuration to ensure that data read requests and write requests are located on the correct cluster.

[0058] As a preferred embodiment of the present invention, the step (4) comprises:

[0059] A distributed data storage cluster consisting of an index cluster, a metadata cluster and multiple data nodes is configured in the archive / hibernate cluster, wherein the index cluster is a fast wake-up cluster, and the metadata cluster and the distributed data storage cluster are slow clusters, which are respectively used to store data index information and the data itself; and a sub-controller is also configured in the archive / hibernate cluster.

[0060] As a preferred embodiment of the present invention, the step (4) comprises the following processing:

[0061] Each data record of the archive / dormant cluster contains a unique internal serial number (_id), the index cluster completes the mapping of query conditions to _id, and the metadata cluster completes the loading of _id to data;

[0062] The sub-controller is configured to receive all read requests, and after synchronizing the index cluster and metadata cluster to complete the read request for loading archived data, the result of the read request is returned to the request caller. After the main controller receives the read request, it searches for all serial numbers (_id) that meet the query conditions according to the index cluster and sorts them, and then sends them to the metadata cluster for data loading and returns them to the caller.

[0063] As a preferred embodiment of the present invention, the archive / dormant cluster is internally configured as a distributed data storage cluster for storing data on multiple data nodes to achieve separation of massive data.

[0064] In practical applications, the method of implementing data archiving processing based on multiple distributed cluster groups in the technical solution specifically includes the following implementation steps:

[0065] 1. Configure two clusters, a hot data cluster (high-speed cluster) and an archive data cluster, to store hot data and archive data respectively.

[0066] 2. Each piece of data goes through the following life cycle: access, processing, archiving in the hot data cluster, migration to the archive cluster after it becomes cold, the archive cluster enters a dormant state, the archive cluster is awakened when needed and seamlessly connected to the main cluster to provide data services, and then goes back to a dormant state after the service is completed.

[0067] 3. Configure the master controller and the master metadata cluster to receive all read and write requests. After synchronizing the two clusters to complete the read and write requests, the results of the read and write requests are returned to the request caller.

[0068] -After receiving the read request, the master controller locates the request to two different clusters based on the master metadata cluster and returns the returned data to the caller;

[0069] -After receiving the write request, the main controller forwards the write request to the hot data cluster to complete the write service, and then returns the write service execution result to the application caller;

[0070] -The general metadata server automatically migrates data and updates metadata according to the data lifecycle flow configuration to ensure that data read and write requests are located on the correct cluster.

[0071] 4. Two subclusters are configured in the archive cluster, the index cluster (fast wake-up cluster) and the data cluster (slow cluster), which are used to save data index information and save the data itself respectively.

[0072] 5. Each record in the data cluster of the archive cluster contains a unique internal serial number (_id). The index cluster completes the mapping of query conditions to _id; the data cluster completes the loading of _id to data.

[0073] 6. Configure the sub-controller in the archive cluster to receive all read requests, synchronize the index cluster and the data cluster to complete the read request for archival data loading, and return the result of the read request to the request caller;

[0074] -After receiving the read request, the main controller searches for all serial numbers (_id) that meet the query conditions according to the index cluster and combines them in order, then sends them to the data cluster, loads the data, and returns to the caller.

[0075] 7. Configure the data cluster in the archive cluster as a distributed data storage cluster, and store the data on multiple data nodes to achieve massive data separation. To retrieve archived data once, you only need to wake up the data node where the data is located.

[0076] In a typical application scenario of the present invention, the data structure stored in the archive cluster contains a large number of auxiliary information fields that do not need to be queried and a small number of fields that need to be queried, such as time. For example, in the GA / T 1300 public security video image information application system, face objects and vehicle objects contain a large amount of auxiliary field information, but the fields used for retrieval and sorting and that need to be stored in the index cluster are limited to a limited number of fields such as time and device ID.

[0077] In a typical application scenario of the present invention, the number of archived data queries is relatively small, and fewer operations are required to load and read the archived data.

[0078] See also Figure 1 As shown, the system for implementing the above-mentioned method to realize data archiving processing based on multiple distributed cluster groups, wherein the system includes:

[0079] Hot data cluster, used to store the hot data of the platform;

[0080] Archive / hibernation clusters for storage, retrieval, and review of archived data; and

[0081] The main controller is connected to the hot data cluster and the archive / dormant cluster, and is used to receive the read and write requests in the system and return the corresponding execution results to the caller.

[0082] As a preferred embodiment of the present invention, the hot data cluster is specifically:

[0083] Implemented by a distributed cluster of MongoDB, Redis, ScyllaDB, PostgreSQL, or CockroachDB;

[0084] It consists of SSD disks and servers to complete frequent queries on hot data and real-time insertion of new data.

[0085] As a preferred embodiment of the present invention, the archive / hibernation cluster includes: a sub-controller, an index cluster, a metadata cluster, multiple data nodes and a network interface, wherein the sub-controller is used to receive all read and write requests forwarded by the main controller and return the results to the main controller, the index cluster converts the query conditions and result sorting into a set of data unique serial numbers (_id), and the metadata cluster is used to create and maintain the mapping relationship between the unique serial number (_id) and the data node.

[0086] As a preferred embodiment of the present invention, the unique serial number (_id) is a unique serial number generated by the metadata cluster according to a specified strategy, including a prefix and a suffix, the prefix is ​​the shard key, which is a uniform random number ranging from 1 to N of the data node serial number, and the suffix is ​​a sequentially increasing serial number, which is a unique index of each internal data node.

[0087] As a preferred embodiment of the present invention, the master controller includes a read request processing module, a write request processing module, a metadata library and a network interface, wherein the read request processing module and the write request processing module are used to receive all read and write requests in the system and return the results to the caller; the metadata library is used to maintain the total cluster index and forward the read and write requests for data within different time ranges to different clusters.

[0088] As a preferred embodiment of the present invention, the hot data cluster and archive / dormant cluster are sharded according to time, and the sharding information is stored in the metadata database. The read request processing module queries the metadata database to forward different requests unidirectionally to the corresponding cluster.

[0089] As a preferred implementation mode of the present invention, the network interfaces all adopt optical module interfaces to cope with the situation of writing and reading a large amount of data.

[0090] See also Figure 1 As shown, in a specific embodiment of the present invention, the system for implementing data archiving processing based on multiple distributed cluster groups of the present technical solution includes: a master controller 100, a hot data cluster 200 and an archiving / dormant cluster 300.

[0091] The master controller 100 includes a read request processing module 110, a write request processing module 120, a metadata repository 130, and a network interface 140. The read request processing module 110 and the write request processing module 120 are responsible for receiving all read and write requests and returning the results to the caller; the metadata repository 130 maintains the total cluster index and forwards data read and write requests of different time ranges to different clusters.

[0092] After the main controller 100 receives the read request, the read request processing module 110 extracts the query conditions and sorting fields contained in the read request, forwards the request to the hot data cluster 200 and the archive / dormant cluster 300, merges the results returned by each according to the sorting, and returns them to the caller.

[0093] After the main controller 100 receives the write request, the write request processing module 120 distributes the request to the hot data cluster 200, and after the processing is completed, returns the write request result to the request caller.

[0094] In a recommended implementation of the present invention, the hot data cluster 200 and the archive / dormant cluster 300 are sharded according to time, and the sharding information is stored in the metadata database 130. The read request processing module 110 can forward different requests to the corresponding clusters "single point" by querying the metadata database 130, without broadcasting all read requests to all data clusters. In this way, the archive cluster can be dormant when there is no forwarding request, and can be partially awakened and seamlessly connected to the main controller 100 when there is a request.

[0095] In a recommended implementation of the present invention, the network interface 140 uses an optical module interface to cope with large amounts of data writing and reading, so as to avoid the network becoming a system bottleneck.

[0096] Hot data cluster 200 includes a network interface 210 .

[0097] In a recommended implementation of the present invention, the hot data cluster 200 is deployed according to a typical distributed big data platform and can be implemented by a distributed cluster such as MongoDB / Redis / ScyllaDB / PostgreSQL / CockroachDB.

[0098] In a recommended implementation of the present invention, the hot data cluster 200 is composed of SSD disks and servers with strong CPU performance and more memory, such as 512GB, and has completed frequent queries for hot data and real-time insertion of new data;

[0099] In a recommended implementation of the present invention, data older than 3-6 months is cold data that needs to be archived, that is, it should be automatically migrated from the hot data cluster 200 to the archive / dormant cluster 300 to ensure that the hot data cluster 200 has enough space to accommodate new real-time data;

[0100] The archive / hibernation cluster 300 includes a sub-controller 310, an index cluster 320, a metadata cluster 330, data nodes 1-N (340-350) and a network interface 360. The sub-controller 310 is responsible for receiving all read and write requests forwarded by the main controller 100 and returning the results to the main controller 100. The index cluster 320 converts the query conditions and result sorting into a data unique serial number (_id) set. The metadata cluster 330 creates and maintains the mapping relationship between the unique serial number (_id) and the data nodes 340-350.

[0101] After receiving the write request for data migration, the sub-controller 310 requests the metadata database 330 to generate a new unique serial number (_id) and queries the data node corresponding to the new serial number; distributes the write request to the data node and the index cluster 320, and returns a write success to the request caller after both modules have processed successfully; if any module fails to process, the other side returns a write failure to the request caller after the operation is rolled back.

[0102] After receiving the forwarded read request, the sub-controller 310 queries the index cluster 320, converts the query conditions and sorting conditions into a unique serial number (_id) set, and directly returns the empty result set to the main controller 100 when the query result is empty. If the query result is not empty, the sub-controller 310 wakes up the metadata database 330 and queries the metadata database 330 for the set of data nodes 340-350 where the unique serial number (_id) set is located. After waking up all corresponding data nodes, the query based on the unique serial number (_id) is sent separately to the data node where it is located and the result is returned.

[0103] When the index cluster 320 receives a data write request, it extracts the serial number (_id) and all fields marked as indexes or composite indexes, and adds them to the _id to index field dictionary; after receiving a read request, it extracts the search conditions and sorting fields contained in the read request, finds records that meet the query conditions based on the id to index field dictionary, and returns them according to the sorted combined serial number (_id) result set.

[0104] In a recommended implementation of the present invention, the unique serial number (_id) of the data is a data structure that helps the data nodes 340-350 to efficiently obtain data according to the serial number (_id), and is also the unique index of the data nodes 340-350. When the data nodes 340-350 receive a query request based on the serial number (_id), the data nodes 340-350 search for the data record storage offset address corresponding to the serial number (_id), and read the data record from the address and return it; the query of the data nodes 340-350 should not contain any query conditions other than the serial number (_id) to avoid the data nodes 340-350 traversing the entire data node to find records that meet the conditions.

[0105] In a recommended implementation of the present invention, the serial number (_id) is a unique serial number generated by the metadata database 330 according to a specified strategy, including a prefix and a suffix, the prefix is ​​the shard key, which is a uniform random number with a value range of 1-N for the data node serial number, and the suffix is ​​a sequentially increasing serial number. Therefore, automatic load balancing can be achieved to ensure that data is evenly distributed to all data nodes. At the same time, the serial numbers (_id) of all records of the same data node are sequentially increasing, and the index library of the unique index (_id) is avoided from being reconstructed due to the writing of new data, thereby greatly improving the processing capacity of continuous concurrent writes that a single data node can withstand when it already carries tens of millions of data records.

[0106] In a recommended implementation of the present invention, the data nodes 340-350 of the archive data cluster are composed of low-cost SATA HDD disks and servers or storage devices with poor CPU performance to reduce the total cost of cluster deployment;

[0107] In a recommended implementation of the present invention, the index cluster 320, metadata repository 330 and data nodes 340-350 are implemented by a distributed cluster such as MongoDB / Redis / ScyllaDB / PostgreSQL / CockroachDB.

[0108] In a recommended implementation of the present invention, the archive cluster is divided into a request processing part, namely the sub-controller 310, the index cluster 320 and a data retrieval processing part, namely the metadata library 330 and the data nodes 340-350. Therefore, when an archive query request arrives, only the request processing part is awakened and completes the service, and all data nodes continue to remain in a dormant state.

[0109] In a recommended implementation of the present invention, the data nodes in the archive cluster are sharded according to unique serial numbers (_id) to achieve separation of massive data. When an archive data loading request arrives, only some of the data nodes involved in the data are awakened and complete the service, and other data nodes continue to remain in a dormant state.

[0110] In a recommended implementation of the present invention, data nodes 340 - 350 may be deployed in primary-secondary / dual-copy / multiple-copy modes to achieve disaster recovery and automatic failover during data storage.

[0111] In a recommended implementation of the present invention, all access to archived data requires awakening the sub-controller 310, the index cluster 320, and the metadata cluster 330. Therefore, in the server cluster of archived data, the sub-controller 310, the index cluster 320, and the metadata cluster 330 are composed of servers with high performance that can be awakened at any time to complete efficient archived data queries.

[0112] In a recommended implementation of the present invention, massive data is stored on servers corresponding to data nodes 340-350, which are awakened only when hit to provide data loading services. Data nodes 340-350 are constructed by storage servers with relatively poor performance and relatively low cost, thereby effectively reducing the overall deployment cost of the system.

[0113] In a recommended implementation of the present invention, the network interface 360 ​​uses an optical module interface to cope with large amounts of data writing and reading, so as to avoid the network becoming a system bottleneck.

[0114] In a recommended implementation of the present invention, the hot data cluster 200 can be further subdivided into two types: real-time data cluster and ordinary data cluster. The real-time data only stores the most frequently called data of the past month, and is saved by the server (group) with the best performance; the ordinary data cluster and the archive data cluster 300 are composed of servers with the same architecture. After the ordinary data service period reaches the archive period, it can directly sleep and enter the archive data mode to avoid large-scale data migration.

[0115] In a recommended implementation of the present invention, the server controls the sleep state of the server by adjusting the server power management settings, thereby achieving automatic dormancy of the server in the absence of service.

[0116] In a recommended implementation of the present invention, the system uses Intel AMT technology (Intel Active Management Technology) to enable the server to be awakened when service is required.

[0117] In a recommended implementation of the present invention, the system uses Intel AMT technology (Intel Active Management Technology) to periodically wake up all servers to prevent magnetic field attenuation and electronic device aging caused by long-term power-down of servers and mechanical hard disks.

[0118] In a recommended implementation of the present invention, the disk status of the archive cluster is checked regularly to ensure the integrity of the archived data.

[0119] Furthermore, the digital archivable system based on multiple distributed cluster group deployment of the technical solution includes:

[0120] A master controller and two or more distributed data clusters including a data cluster and an archive cluster are configured in the digital application system, wherein the data cluster is used to store platform data; and the archive cluster is used to store, retrieve and read archive data;

[0121] An index subcluster is configured separately in the archive cluster to index and separately save the retrieved and sorted data; a distributed data subcluster is configured to save the data on different data nodes in the data subcluster;

[0122] Configure the distributed data sub-clusters within the archive cluster. Each data record is saved with a unique serial number (_id) and is used as a unique index and sharded. Different data nodes save the data and provide data loading services when the data is retrieved.

[0123] Configure the archive cluster controller to control the two subclusters to complete data migration and retrieval requests respectively. When processing the write request of data migration, the two subclusters are updated synchronously to complete the writing. When processing the read request of archive data retrieval, the unique serial number (_id) of all data that meets the query conditions is searched according to the index subcluster and combined according to the expected sorting. Then, it is sent to the data subcluster, loaded with data, and returned to the caller.

[0124] Configure the master controller to send query requests to the archive cluster sub-controller only when the query involves the time range of the archive data cluster;

[0125] In a preferred embodiment of the present invention, the index subcluster is separated from the data subcluster, and the index subcluster is awakened separately during archived data retrieval to provide retrieval and sorting services.

[0126] In a preferred embodiment of the present invention, when archived data is accessed, the data node storing the accessed archived data is awakened and provides data loading service, while other data nodes continue to remain in a dormant state.

[0127] In a preferred embodiment of the present invention, when responding to a request containing retrieval conditions and sorting conditions, the archive cluster controller wakes up the index server (group) without waking up the data node, and queries whether there is data that meets the retrieval conditions. If so, a set of unique serial numbers (_id) that meet the retrieval conditions and sorting conditions are sent to the distributed data sub-cluster, and the corresponding data nodes are extracted. After waking up, a data retrieval request is sent, and the returned results are assembled and returned;

[0128] In a preferred embodiment of the present invention, when archived data is accessed, the entire distributed data cluster is partially awakened and provides data loading services only when it is determined that the data exists and is specifically accessed.

[0129] On this basis, the method and system for realizing data archiving based on multiple distributed cluster groups of the technical solution support overall sleep, wake-up and seamless access of local clusters. It has the following features:

[0130] 1. Enter the dormant state as a distributed cluster;

[0131] -The entire distributed cluster is deployed based on the main controller + main metadata server + subordinate multiple sub-controllers + subordinate sub-metadata clusters + subordinate sub-data node clusters. When the archived data goes offline and enters hibernation, it is not just the data itself that enters the hibernation state, but the operating system, service software and all system-related software of the entire metadata database cluster and data node cluster. When the system is awakened, it is not just the data that is restored, but the entire archived data retrieval, review and loading service.

[0132] 2. Sleep in low power state. During the sleep period, the power consumption of the cluster can be ignored;

[0133] 3. The dormant archive cluster can be awakened at any time and provide access services;

[0134] -The dormant cluster can be awakened when receiving a query request, seamlessly connected to the main cluster, and provide query services;

[0135] -The dormant cluster is divided into multiple sub-clusters. Most of the sub-clusters only need to be awakened when data is loaded, and are in a power-saving dormant state the rest of the time.

[0136] As a branch of the overall cluster, the archive data cluster can be dormant at any time, and wake up and seamlessly connect to the overall cluster when it is retrieved. While providing archive data retrieval and retrieval services, it ensures that irrelevant data nodes remain dormant. The system has high query efficiency while taking into account low cost and power-saving operation characteristics. It is suitable for scenarios where there is a need for long-term archiving of massive relational (SQL) and non-relational (NOSQL) data, such as message conversations, pictures, video clips, server cluster logs, etc.

[0137] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0138] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.

[0139] A person of ordinary skill in the art may understand that all or part of the steps of the method for implementing the above-mentioned embodiment may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0140] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0141] In the description of this specification, the description with reference to the terms "an embodiment", "some embodiments", "example", "specific example", or "embodiment" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0142] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

[0143] The method and system for realizing data archiving processing based on multiple distributed cluster groups of the present invention have the following technical advantages:

[0144] 1. Public security data with high reliability includes data that can be archived and backed up, that is, a digital archive system for the access, application, archiving, offline, sealing, retrieval, and online application of various types of public security data;

[0145] 2. When searching, the system can query whether this part of data (file) is in the current archive room, just like searching the file details of the traditional archive room;

[0146] 3. When accessing, the system can locate a certain file cabinet and load this part of data (files) just like accessing files in a traditional archive room;

[0147] 4. Like traditional archives rooms, electronic archives rooms have national standard safety protection plans such as fire prevention, waterproofing, and earthquake resistance;

[0148] 5. When a city-level earthquake, flood, or regional fire occurs, the electronic archive room can put historical data online.

[0149] In this specification, the present invention has been described with reference to specific embodiments thereof. However, it is apparent that various modifications and variations may be made without departing from the spirit and scope of the present invention. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A method for implementing data archiving processing based on multiple distributed cluster groups, characterized in that: The method comprises the following steps: (1) Configure a hot data cluster to store the hot data of the platform, and configure an archive / dormant cluster to store archive data; (2) The hot data and archived data are all processed by full-process data life cycle flow configuration; (3) Configure the master controller to receive all read and write requests from the system, and return the returned data to the caller after completing the request positioning; (4) Performing internal configuration processing on the archive / dormant cluster to complete the storage, retrieval and review of archived data.

2. The method for realizing data archiving processing based on multiple distributed cluster groups according to claim 1, characterized in that: The step (2) is specifically as follows: The hot data and archived data are accessed, processed, archived in hot data clusters, migrated to archive / dormant clusters after cooling down, archive / dormant clusters enter a dormant state, archive / dormant clusters are awakened when needed and connected to the main controller cluster to provide data services, and continue to enter a dormant state after the services are completed, including full-process data lifecycle flow configuration processing.

3. The method for realizing data archiving processing based on multiple distributed cluster groups according to claim 1, characterized in that: The step (3) specifically comprises the following steps: After receiving the read request, the master controller described in (3.1) locates the corresponding request to the hot data cluster and the archive / dormant cluster respectively according to the internal master metadata cluster, and then returns the returned data to the caller; (3.2) After receiving the write request, the master controller forwards the write request to the hot data cluster to complete the write service, and then returns the corresponding write service execution result to the application caller; The master controller described in (3.3) performs automatic data migration and metadata update according to the data lifecycle flow configuration to ensure that data read requests and write requests are located on the correct cluster.

4. The method for realizing data archiving processing based on multiple distributed cluster groups according to claim 1, characterized in that: The step (4) comprises: A distributed data storage cluster consisting of an index cluster, a metadata cluster and multiple data nodes is configured in the archive / hibernate cluster, wherein the index cluster is a fast wake-up cluster, and the metadata cluster and the distributed data storage cluster are slow clusters, which are respectively used to store data index information and the data itself; and a sub-controller is also configured in the archive / hibernate cluster.

5. The method for realizing data archiving processing based on multiple distributed cluster groups according to claim 4, characterized in that: The step (4) comprises the following steps: Each data record of the archive / dormant cluster contains a unique internal serial number (_id), the index cluster completes the mapping of query conditions to _id, and the metadata cluster completes the loading of _id to data; The sub-controller is configured to receive all read requests, and after synchronizing the index cluster and metadata cluster to complete the read request for loading archived data, the result of the read request is returned to the request caller. After the main controller receives the read request, it searches for all serial numbers (_id) that meet the query conditions according to the index cluster and sorts them, and then sends them to the metadata cluster for data loading and returns them to the caller.

6. The method for realizing data archiving processing based on multiple distributed cluster groups according to claim 4, characterized in that: The archive / dormant cluster is internally configured as a distributed data storage cluster for storing data on multiple data nodes to achieve separation of massive data.

7. A system for implementing the method of claim 5 and realizing data archiving processing based on multiple distributed cluster groups, characterized in that: The system comprises: Hot data cluster, used to store the hot data of the platform; Archive / hibernation clusters for storage, retrieval, and review of archived data; and The main controller is connected to the hot data cluster and the archive / dormant cluster, and is used to receive the read and write requests in the system and return the corresponding execution results to the caller.

8. The system for implementing data archiving processing based on multiple distributed cluster groups according to claim 7, characterized in that: The hot data cluster is specifically: Implemented by a distributed cluster of MongoDB, Redis, ScyllaDB, PostgreSQL, or CockroachDB; It consists of SSD disks and servers to complete frequent queries on hot data and real-time insertion of new data.

9. The system for implementing data archiving processing based on multiple distributed cluster groups according to claim 7, characterized in that: The archive / hibernation cluster includes: a sub-controller, an index cluster, a metadata cluster, multiple data nodes and a network interface, wherein the sub-controller is used to receive all read and write requests forwarded by the main controller and return the results to the main controller, the index cluster converts the query conditions and result sorting into a data unique serial number (_id) set, and the metadata cluster is used to create and maintain the mapping relationship between the unique serial number (_id) and the data node.

10. The system for implementing data archiving processing based on multiple distributed cluster groups according to claim 9, characterized in that: The unique serial number (_id) is a unique serial number generated by the metadata cluster according to the specified strategy, including a prefix and a suffix. The prefix is ​​the shard key, which is a uniform random number in the range of 1-N for the data node serial number. The suffix is ​​a sequentially increasing serial number, which is a unique index for each internal data node.

11. The system for implementing data archiving processing based on multiple distributed cluster groups according to claim 9, characterized in that: The master controller includes a read request processing module, a write request processing module, a metadata library and a network interface, wherein the read request processing module and the write request processing module are used to receive all read and write requests in the system and return the results to the caller; the metadata library is used to maintain the total cluster index and forward the read and write requests of data within different time ranges to different clusters.

12. The system for implementing data archiving processing based on multiple distributed cluster groups according to claim 11, characterized in that: The hot data cluster and the archive / dormant cluster are sharded according to time, and the sharding information is stored in the metadata database. The read request processing module queries the metadata database to forward different requests unidirectionally to the corresponding cluster.

13. The system for implementing data archiving processing based on multiple distributed cluster groups according to claim 11, characterized in that: The network interfaces described all use optical module interfaces to cope with the situation of writing and reading a large amount of data.

Images