A full-flow storage cluster management method and device for separating hot and cold data

Abstract

The present application provides a full-flow storage cluster management method and device with hot and cold data separation, including a second node receiving cold data, determining the virtual domain to which the cold data belongs and the physical storage area to be stored in the virtual domain according to configuration information, and sending the cold data to the first node to which the physical storage area to be stored belongs; the second node obtains the hot data of the cold data according to the received cold data, and stores the hot data in the local storage area, and the first node receives the cold data sent by the second node, and stores the received cold data in the corresponding physical storage area. In the present invention, users only need to pay attention to the storage configuration at the business level. The increase or decrease of storage nodes will only affect the total capacity, and will not have any impact on the existing data, thus avoiding the risk of data migration when adding or reducing nodes.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of network technology, and in particular, to a method and device for managing a full-flow storage cluster with hot and cold data separation. Background Art

[0002] Currently, in the field of network traffic data storage, deploying a single traffic storage device on a network node can no longer meet storage needs. The storage of all traffic is achieved by deploying a cluster of traffic storage devices. Storing network traffic in each traffic storage node in the cluster, especially when multiple applications or users try to access the same data or resources at the same time, may encounter resource preemption problems, resulting in reduced retrieval performance. In addition, when it is necessary to increase or reduce storage nodes, the management node needs to sort out the data that needs to be migrated and perform migration operations to ensure load balancing of each traffic storage node in the entire cluster. Summary of the invention

[0003] The present application describes a full-flow storage cluster management method and device with cold and hot data separation, which can solve the above-mentioned technical problems.

[0004] According to a first aspect, a full-flow storage cluster management method for separating cold and hot data is provided, wherein the full-flow storage cluster includes at least one management node, a first subcluster and a second subcluster, wherein the first subcluster includes at least one first node, and the second subcluster includes at least one second node, wherein the management node is used to manage and monitor the first subcluster and the second subcluster, wherein the first subcluster is used to store cold data, and the second subcluster is used to store hot data;

[0005] The method comprises:

[0006] The management node sends configuration information to the first subcluster and the second subcluster, wherein the configuration information includes information of virtual domains, the virtual domains are used to store traffic data packets of different service attributes in different logical storage spaces, the logical storage space of a virtual domain includes a physical storage area in at least one first node in the first subcluster, the physical storage space of the first node is divided into at least one physical storage area, and each physical storage area belongs to a virtual domain;

[0007] At least one of the second nodes receives the cold data, determines the virtual domain to which the cold data belongs and the physical storage area to be stored in the virtual domain according to the configuration information, and sends the cold data to the first node to which the physical storage area to be stored belongs;

[0008] At least one of the second nodes obtains hot data of the cold data according to the received cold data, and stores the hot data in a local storage area, wherein the cold data is a traffic data packet, and the hot data is a traffic log and statistical data generated based on the traffic data packet;

[0009] At least one of the first nodes receives the cold data sent by the second node, and stores the received cold data in a corresponding physical storage area.

[0010] According to a second aspect, a full-flow storage cluster device for separating cold and hot data is provided, wherein the full-flow storage cluster device includes at least one management node, a first subcluster and a second subcluster, wherein the first subcluster includes at least one first node, and the second subcluster includes at least one second node, wherein the management node is used to manage and monitor the first subcluster and the second subcluster, wherein the first subcluster is used to store cold data, and the second subcluster is used to store hot data;

[0011] The management node is used to send configuration information to the first subcluster and the second subcluster, wherein the configuration information includes information of virtual domains, the virtual domains are used to store traffic data packets of different service attributes in different logical storage spaces, the logical storage space of a virtual domain includes a physical storage area in at least one first node in the first subcluster, the physical storage space of the first node is divided into at least one physical storage area, and each physical storage area belongs to a virtual domain;

[0012] at least one of the second nodes is configured to receive cold data, determine the virtual domain to which the cold data belongs and the physical storage area to be stored in the virtual domain according to the configuration information, and send the cold data to the first node to which the physical storage area to be stored belongs;

[0013] at least one of the second nodes, configured to obtain hot data of the cold data according to the received cold data, and store the hot data in a local storage area, wherein the cold data is a traffic data packet, and the hot data is a traffic log and statistical data generated based on the traffic data packet;

[0014] At least one of the first nodes is used to receive the cold data sent by the second node, and store the received cold data in a corresponding physical storage area.

[0015] According to a third aspect, a full-flow storage cluster system for separating cold and hot data is provided, the full-flow storage cluster system comprising at least one management node, a first subcluster and a second subcluster, the first subcluster comprising at least one first node, the second subcluster comprising at least one second node, the management node being used to manage and monitor the first subcluster and the second subcluster, the first subcluster being used to store cold data, and the second subcluster being used to store hot data;

[0016] The management node is used to send configuration information to the first subcluster and the second subcluster, wherein the configuration information includes information of virtual domains, the virtual domains are used to store traffic data packets of different service attributes in different logical storage spaces, the logical storage space of a virtual domain includes a physical storage area in at least one first node in the first subcluster, the physical storage space of the first node is divided into at least one physical storage area, and each physical storage area belongs to a virtual domain;

[0017] at least one of the second nodes is configured to receive cold data, determine the virtual domain to which the cold data belongs and the physical storage area to be stored in the virtual domain according to the configuration information, and send the cold data to the first node to which the physical storage area to be stored belongs;

[0018] at least one of the second nodes, configured to obtain hot data of the cold data according to the received cold data, and store the hot data in a local storage area, wherein the cold data is a traffic data packet, and the hot data is a traffic log and statistical data generated based on the traffic data packet;

[0019] At least one of the first nodes is used to receive the cold data sent by the second node, and store the received cold data in a corresponding physical storage area.

[0020] In the above-mentioned system and method provided in the embodiments of this specification, the concept of virtual domain is introduced in the field of full-flow cluster storage, and users only need to pay attention to the storage configuration at the business level. The cluster node storage is regarded as a whole space, and the virtual domain is a part of the space on it. The increase or decrease of nodes will only affect the total capacity and will not have any impact on the existing data, thus avoiding the risk of data migration when adding or reducing nodes. The hot and cold data are separated, and when querying hot data, resources can be exclusively used to speed up the retrieval performance, which solves the problem of resource preemption that may occur and lead to reduced retrieval performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A schematic diagram showing the structure of a full-flow storage cluster of a full-flow storage cluster management method for cold and hot data separation provided in an embodiment of this specification;

[0023] Figure 2 A schematic diagram showing a flow chart of storage node space acquisition of a full-flow storage cluster management method for cold and hot data separation provided in an embodiment of this specification;

[0024] Figure 3 A schematic diagram showing a virtual domain configuration of a full-flow storage cluster management method for separation of hot and cold data provided by an embodiment of this specification;

[0025] Figure 4 A schematic diagram showing capacity distribution calculation of a full-flow storage cluster management method for cold and hot data separation provided in an embodiment of this specification;

[0026] Figure 5 A schematic diagram showing a virtual domain capacity increase sorting principle of a full-flow storage cluster management method for cold and hot data separation provided by an embodiment of this specification;

[0027] Figure 6 A schematic diagram showing a principle of decreasing capacity of virtual domains in a full-flow storage cluster management method for separation of hot and cold data provided by an embodiment of this specification;

[0028] Figure 7 A schematic diagram showing the structure of traffic collection and storage of a full-traffic storage cluster management method with cold and hot data separation provided in an embodiment of this specification;

[0029] Figure 8 A schematic diagram showing a flow chart of a full-flow storage cluster management method for separating hot and cold data provided in an embodiment of this specification;

[0030] Fig. 9 A schematic diagram showing the structure of a full-flow storage cluster management device with cold and hot data separation provided in an embodiment of this specification;

[0031] Fig.10 A schematic diagram of the structure of a full-flow storage cluster management system with cold and hot data separation provided in an embodiment of this specification is shown. DETAILED DESCRIPTION

[0032] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0034] In the field of network traffic data storage, with the development of various network technologies, a single device can no longer meet the storage needs of traffic storage devices deployed on network nodes, so cluster deployment is required to achieve full traffic storage.

[0035] The storage and retrieval device for intelligent network full-flow distribution of the present invention realizes the storage of full-flow based on cluster mode. Among them, intelligent distribution is to store network flow data on each node in the cluster according to rules and algorithms to achieve data balance among each node in the cluster. When retrieving data, each node concurrently queries, improving the retrieval performance from the overall system level.

[0036] In the full-flow storage cluster mode, traffic data is divided into cold data and hot data. The concepts of cold and hot data here are different from those in traditional storage systems. In traditional storage systems, cold and hot data mainly refer to the access popularity of data, and the data source is the same.

[0037] The data sources of the network full-traffic storage and retrieval scenarios are different. Among them, cold data refers to traffic data packets. In actual applications, users seldom directly view the most original data packets. They are more often used for tracing and evidence collection after an incident occurs. They belong to the bottom-level data in the retrieval chain, and the amount of such data is huge, which consumes a lot of resources.

[0038] Hot data refers to session logs, protocol logs, and statistical data generated based on data packets, which are often displayed to users in a visual form, making them intuitive and easy to understand. The space occupied by data packets is not of the same magnitude, and the ratio of hot and cold data is 30,000:1.

[0039] When hot and cold data coexist, how to improve the overall storage and retrieval performance of the cluster is a key issue that the network full-flow storage and retrieval device needs to pay attention to and solve.

[0040] The device of the present invention is suitable for full-flow storage clusters, such as Figure 1 As shown, it mainly includes management nodes, storage nodes and traffic processing nodes.

[0041] The management node is responsible for managing the storage nodes and traffic processing nodes in the cluster, maintaining the node status through heartbeats, collecting other node information, including space capacity, CPU cores, total memory, network port information, etc., sending virtual domain configurations to each node, and synchronizing the IP and network port information of all nodes in the cluster to each node. HA dual-machine hot standby solution can be used for deployment to ensure high availability of the system.

[0042] The storage node is responsible for storing cold data such as traffic packets within the cluster. Through the management node configuration, the traffic data of the specified network port can be received. The data is stored in order on the disk by chunk, and a three-level index is generated for each chunk to improve the retrieval performance. The storage node is organized into a data packet storage subcluster through the management node. When querying data packets, only the nodes on this subcluster are requested.

[0043] The traffic processing node is mainly responsible for capturing network traffic and forwarding the traffic data to the specified virtual domain through the specified network port according to the configuration of the management node. Each virtual domain includes multiple storage nodes. The local storage session logs, protocol logs and statistics of the traffic processing node are stored in high IO storage media such as SSD or NVME to accelerate access.

[0044] According to the virtual domain configuration issued by the management node, the traffic processing node records the storage node IP, allocated capacity, and forwarded data volume in the virtual domain, and calculates the space utilization rate of each node in the virtual domain. The captured data packets are forwarded to the node with the lowest space utilization rate. Subsequent data packets of the same session are stored on the same node by default, which ensures that the space utilization rate of each node in the virtual domain is basically balanced, and the data of the same session can be stored on the same node for easy retrieval and query. The traffic processing node will be formed into a hot data storage subcluster by the management node, and only the nodes on this subcluster will be requested when querying logs and statistical hot data.

[0045] The administrator configures the virtual domain on the management node according to the scenario requirements. The core configuration of the virtual domain is the storage capacity. The virtual domain is distributed to multiple storage nodes according to the capacity. The minimum allocation unit is 1TB, which can also be adjusted according to the capacity scale.

[0046] The traffic processing node only focuses on the virtual domain and stores the traffic data in different virtual domains according to the traffic attributes. Once the data is stored, it will not be moved, and there is no storage migration across virtual domains.

[0047] like Figure 2 As shown, in a full-traffic storage cluster, the storage node specifies the management node during deployment, automatically registers with the management node after startup, and reports the total storage space of the node. After the connection is established, the heartbeat reports the space usage of the node at regular intervals.

[0048] The management node automatically divides the collected storage space into the default storage space, that is, the default virtual domain, to ensure that data can be stored normally in the initial stage of the system. When the storage node capacity increases, the expanded capacity will be automatically divided into the default storage space. When a storage node fails and is removed from the cluster, the space of the corresponding virtual domain will be reduced according to the configuration based on the division of space on the node.

[0049] like Figure 3 As shown, the default virtual domain exists in the initial stage of the management node and cannot be deleted. The administrator creates N virtual domains according to the scenario requirements and any dimensions, such as traffic data area attributes, traffic business attributes, and attention. The space of the virtual domain is divided from the default virtual domain. When adding a virtual domain, the space is reduced from the default virtual domain. When deleting a virtual domain, the space is automatically returned to the default virtual domain.

[0050] The main attributes of a virtual domain include the name of the virtual domain, the size of the available space, the remaining space alarm, and whether the space is written in a loop; the size of the space can be adjusted as needed, and the minimum unit of the space division can be adjusted according to the actual situation. When the space capacity is increased, it will be divided from the default virtual domain; when the space capacity is reduced, the reduced capacity will be automatically returned to the default virtual domain. When the space is reduced, the earliest traffic data packet will be deleted first. When the free space of the virtual domain is insufficient, an alarm will be generated, and the virtual domain will be expanded as needed. When the virtual domain space is full, the oldest data packet data will be automatically aged to ensure that new data can be written all the time; when circular writing is not configured, new traffic data will be automatically discarded when the space is full.

[0051] like Figure 4 , the capacity distribution calculation of virtual domains is generally divided into two categories: increasing capacity and reducing capacity. There are two scenarios for increasing capacity: creating a new virtual domain, that is, the capacity of the virtual domain increases from 0 to the specified size, and the capacity of an existing virtual domain is expanded; there are two scenarios for reducing capacity: deleting a virtual domain, that is, the capacity of the virtual domain is reduced from the allocated size to 0, and the capacity of an existing virtual domain is reduced. When increasing capacity, first determine whether the overall remaining unallocated space of the cluster is sufficient. If not, it cannot be configured. If it is satisfied, it is divided into two situations: if the capacity of an existing virtual domain is to be increased, the allocated storage nodes are obtained first, and the remaining space of the nodes is continued to be divided until there is not enough space, and then new storage nodes are obtained according to the sorting method below;

[0052] The new virtual domain directly selects the nodes to be allocated according to the order below. According to the order principle below, the node with the most remaining resources in the cluster can be selected.

[0053] like Figure 5The priority of the virtual domain capacity increase sorting principle is from high to low: sort by total space in descending order, sort by remaining unallocated space in descending order, sort by the number of virtual domains on the node in ascending order, and sort by computing power in descending order (hard disk IO, number of CPU cores, total memory, etc.). According to the sorting, the top 8 storage nodes with the highest matching degree are taken as pre-selected nodes for capacity division. If the number is insufficient, the existing available nodes are taken. The TopN can be adjusted according to the actual situation. According to the increased capacity, a fixed number of storage nodes are polled and traversed in the minimum unit of 1TB. Each time a unit is allocated, the pre-selected nodes must be re-sorted until the space is allocated to each node. If the capacity of the node to be allocated is insufficient, the node with the highest matching degree is taken from the cluster for capacity allocation according to the sorting principle, and the cycle is repeated until the space allocation is completed.

[0054] The present invention can evenly distribute the capacity to each node when the storage node configuration is consistent. When the configuration is inconsistent, it can also maintain a balanced resource utilization rate of each node based on the principle of "the greater the ability, the greater the responsibility", thereby ensuring the load balance of the storage nodes in the cluster as much as possible.

[0055] When reducing capacity, in order to maintain the load balance of storage nodes, it is also necessary to reduce the allocated node capacity by units according to the sorting principle. The reduced capacity will eventually be automatically reclaimed by the default virtual domain. Figure 6 The priority is from high to low, and the virtual domain capacity is sorted in descending order: sort in descending order according to the allocated capacity of each node that has been divided into the virtual domain, sort in ascending order according to the remaining unallocated space of the node, sort in descending order according to the number of virtual domains on the node, and sort in ascending order according to computing power (hard disk IO, number of CPU cores, total memory, etc.).

[0056] like Figure 7 As shown in the figure, after the virtual domain is successfully configured, the management node will send the capacity distribution of the virtual domain to other nodes in the cluster. The main content includes: the IP addresses of all storage nodes included in the virtual domain, and the capacity allocated to each node. Then follow the steps below to collect and store traffic in the virtual domain:

[0057] First, the management node needs to configure one or more network ports of the traffic processing node as traffic receiving network ports. The traffic receiving network ports can be combined across multiple nodes and called a "network". Then configure or use the default traffic collection strategy, such as matching the five-tuple information and message content of the message, bind the network to the strategy, and send it to the traffic processing node. Subsequently, the traffic receiving network port under the network will capture the traffic that meets the characteristics to the traffic processing node for analysis and processing.

[0058] Secondly, the management node also needs to configure one and only one network port of the traffic processing node as a traffic forwarding port. This network port will be the only network port that the collection device can use to forward data packets. By default, the network will be in accordance with the default message processing strategy, which can match the metadata parsed from the message and specify the virtual domain binding for message storage. At this time, the traffic processing node will forward and store the data packet through the only forwarding port to the default virtual domain. It can also forward and store messages with different characteristics to different virtual domains by binding a custom message processing strategy. A network can be configured with multiple message processing strategies. According to the pre-configured policy priority matching hit strategy, the configuration will eventually be sent to the traffic processing node corresponding to the network port in the network.

[0059] After confirming the target virtual domain for storage, the traffic processing node organizes the virtual domain storage node IPs and allocated capacity issued by the management node, and records the amount of forwarded data in real time, calculates the space utilization rate of each node in the virtual domain, and forwards the captured data packets that meet the characteristics to the node with the lowest space utilization rate in the virtual domain. Subsequent data packets of the same session are stored in the same node by default.

[0060] In actual usage scenarios, the cluster will be expanded according to business needs, and the present invention implements separate expansion of different types of nodes.

[0061] When you need to increase the cluster traffic collection capacity or extend the hot data storage period, you can add traffic processing nodes separately. After the newly added traffic processing nodes are deployed in the actual environment, they are registered to the management node and the collection port is specified to start collecting traffic, thereby increasing the overall traffic collection capacity of the cluster; the traffic collection nodes store the logs and statistical data generated by the traffic locally, and the management node adds the new nodes to the hot data storage subcluster. For the entire system, the storage capacity of hot data increases and the storage period becomes longer. When you need to expand the storage period of cold data, that is, data packets, you can add data packet storage nodes separately. The added nodes will be added to the data packet storage subcluster by the management node, which is equivalent to adding part of the capacity of a whole space. After the newly added capacity is allocated to the virtual domains, the capacity of each virtual domain increases, and the storage period of data packets under the same traffic background becomes longer.

[0062] The present invention combines manual configuration of virtual domain configuration and automatic configuration of virtual domain capacity distribution calculation, ensures the intelligence of storage diversion, and can also be manually intervened according to actual scenarios, finding a balance between automatic and manual.

[0063] The present invention regards cluster node storage as a whole space, and the virtual domain is a part of the space on it. The increase or decrease of nodes will only affect the total capacity, and will not have any impact on the existing data, thus avoiding the risk of data migration in the consistent HASH scenario. The present invention separates hot and cold data. Due to the physical isolation of resources, when querying hot data, it can exclusively use resources and speed up retrieval performance. The concept of virtual domain is introduced in the field of full-flow cluster storage. Users only need to pay attention to the configuration at the business level. The storage cluster is a whole block of storage for users.

[0064] When dividing storage, set the minimum division unit, and allocate space on each storage node according to unit polling. To avoid data being too dispersed, set the starting storage node pool according to the actual situation, prioritize the division of space in the nodes in the storage node pool, and add nodes one by one when insufficient. While ensuring node load balancing, it also avoids data being too dispersed. Appropriate data dispersion helps improve the performance of concurrent queries and reduces the risk of data loss due to the increased probability of abnormal nodes due to too many query hits. Hot data is stored independently by the traffic processing node itself, and the exclusive use of hardware resources improves query performance. In the cluster, traffic processing nodes and data packet storage nodes can be expanded independently. When it is necessary to increase the traffic collection capacity and hot data storage cycle, traffic processing nodes can be added separately; when it is necessary to expand the storage cycle of cold data, that is, data packets, data packet storage nodes can be added separately.

[0065] Figure 8 A flow chart of a full-flow storage cluster management method with cold and hot data separation provided in an embodiment of this specification is shown.

[0066] There is at least one management node, a first subcluster and a second subcluster in the full-flow storage cluster. The first subcluster includes at least one first node, and the second subcluster includes at least one second node. The management node is used to manage and monitor the first subcluster and the second subcluster. The first subcluster is used to store cold data, and the second subcluster is used to store hot data.

[0067] The following steps are involved:

[0068] 110. The management node sends configuration information to the first subcluster and the second subcluster.

[0069] Among them, the configuration information includes information of the virtual domain, and the virtual domain is used to store traffic data packets of different business attributes in different logical storage spaces. The logical storage space of a virtual domain includes a physical storage area in at least one first node in the first sub-cluster. The physical storage space of the first node is divided into at least one physical storage area, and each physical storage area belongs to a virtual domain.

[0070] 120. At least one second node receives the cold data, determines the virtual domain to which the cold data belongs and the physical storage area to be stored in the virtual domain according to the configuration information, and sends the cold data to the first node to which the physical storage area to be stored belongs.

[0071] 130. At least one second node obtains hot data of the cold data based on the received cold data, and stores the hot data in a local storage area, wherein the cold data is a traffic data packet, and the hot data is a traffic log and statistical data generated based on the traffic data packet.

[0072] 140. At least one first node receives the cold data sent by the second node, and stores the received cold data in a corresponding physical storage area.

[0073] Based on the above embodiment, further, step 120 specifically includes:

[0074] The second node parses the cold data, obtains metadata of the cold data, and uses the metadata of the cold data to search for a first message processing strategy, wherein the first message processing strategy is a virtual domain storage strategy configured according to dimensional information of the traffic message.

[0075] If the first message processing policy is matched, the virtual domain to which the cold data belongs is determined according to the priority of the matched policy.

[0076] If the first message processing strategy is not matched, the second message processing strategy is searched to determine the virtual domain to which the cold data belongs, and the second message processing strategy is the virtual domain storage strategy of the system default flow message.

[0077] Based on the physical storage area of ​​the first node in the virtual domain and the amount of traffic data that has been forwarded to the first node in the virtual domain, the space utilization rate of the physical storage area of ​​the first node in the virtual domain is calculated, and the physical storage area of ​​the first node with the lowest space utilization rate in the virtual domain is used as the physical storage area where cold data will be stored in the virtual domain.

[0078] Based on the above embodiment, further comprising:

[0079] When a new first node is added, the new first node sends a registration request to the management node, and the registration request message includes the physical storage space size of the new first node.

[0080] The management node adds the newly added first node to the first sub-cluster, divides the physical storage space of the newly added first node, and allocates each physical storage area of ​​the newly added first node to different virtual domains.

[0081] Based on the above embodiment, further comprising:

[0082] The management node receives the capacity increase request and determines whether the capacity of the unallocated physical storage space in the first sub-cluster satisfies the storage capacity requested to be increased in the capacity increase request.

[0083] If so, it is determined whether to increase storage space for an existing virtual domain or to increase storage space for a newly added virtual domain.

[0084] If the storage space is to be increased for an existing virtual domain, the remaining physical storage space in all the first nodes in the existing virtual domain is preferentially allocated.

[0085] If the remaining physical storage space in the first node in the existing virtual domain meets the storage capacity requested to be increased, the storage space of the first node in the existing virtual domain is divided.

[0086] If the remaining physical storage space in each first node in the existing virtual domain does not satisfy the storage capacity requested to be increased, the physical storage space of the first node that does not belong to the existing virtual domain is divided.

[0087] If the storage space is to be increased for the newly added virtual domain, the physical storage space of the first node that does not belong to the existing virtual domain is divided.

[0088] Based on the above embodiment, further dividing the physical storage space of the first node that does not belong to the existing virtual domain specifically includes:

[0089] S51. Sort the first nodes in the first subcluster in descending order according to the priorities of the total storage space capacity of the first nodes, the remaining unallocated storage space capacity of the first nodes, the number of virtual domains existing on the first nodes, and the computing power of the first nodes, to obtain an allocation sequence, wherein a preset number of first nodes are included in the allocation sequence as pre-selected nodes.

[0090] Among them, the first nodes in the first subcluster are sorted from high to low according to the priority of descending order of total storage space capacity of the first node, descending order of remaining unallocated storage space capacity in the first node, ascending order of number of virtual domains existing on the first node and descending order of computing power of the first node, specifically including: firstly sorting the first nodes in descending order according to the total storage space capacity on the node to obtain a first sequence, sorting the first nodes in parallel positions in the first sequence in descending order according to remaining unallocated storage space capacity, if the first nodes in parallel positions still have the same remaining unallocated storage space capacity, then sorting them in ascending order according to the number of existing virtual domains, if the first nodes in parallel positions still have the same number of virtual domains, then sorting them in descending order according to computing power.

[0091] S52, divide the physical storage capacity of the pre-selected nodes, increase the capacity of the storage space according to the request, poll and traverse the pre-selected nodes in the allocation sequence according to the minimum storage space unit, and divide the minimum storage space unit on the physical storage space of each pre-selected node.

[0092] S53: Determine whether the remaining storage space capacity of the pre-selected node is greater than the minimum storage space unit.

[0093] If so, after each minimum storage space unit is allocated on each pre-selected node, the pre-selected nodes in the allocation sequence are re-sorted in descending order according to the total storage space capacity of the first node, in descending order according to the remaining unallocated storage space capacity in the first node, in ascending order according to the number of virtual domains existing on the first node, and in descending order according to the computing power of the first node, the allocation sequence is updated, and step S52 is repeated until the capacity of the requested increase in storage space is fully allocated.

[0094] Otherwise, remove the pre-selected nodes whose remaining storage space capacity is less than the minimum storage space unit from the current allocation sequence, re-sort the pre-selected nodes in the current allocation sequence according to the priority of descending order of the total storage space capacity of the first node, descending order of the remaining unallocated storage space capacity in the first node, ascending order of the number of virtual domains existing on the first node, and descending order of the computing power of the first node, update the allocation sequence, and repeat step S52 until the capacity of the requested increase in storage space is fully allocated.

[0095] Based on the above embodiment, further comprising:

[0096] The management node receives a request to reduce the capacity of the virtual domain, and reduces the capacity of the physical storage area of ​​the first node in the virtual domain by storage units according to the sequence of the first node in the recovery sequence obtained by the second preset sorting method, and the reduced physical storage area is reclaimed by the default virtual domain, wherein the second preset sorting method is to sort in descending order according to the allocated capacity of each first node allocated to the virtual domain, in ascending order according to the remaining unallocated space of the first node, in descending order according to the number of virtual domains existing on the first node, and in ascending order according to the computing power, and sort from high to low priority to obtain the recovery sequence.

[0097] Sorting is performed from high to low priority according to the allocated capacity in descending order of each first node that has been allocated to the virtual domain, sorting in ascending order according to the remaining unallocated space of the first node, sorting in descending order according to the number of virtual domains existing on the first node, and sorting in ascending order according to computing power, specifically including: firstly sorting the first nodes in descending order according to the total storage space capacity on the node to obtain a second sequence, sorting the first nodes in parallel positions in the second sequence in descending order according to the remaining unallocated storage space capacity, if the remaining unallocated storage space capacity of the first nodes in parallel positions is the same, then sorting is performed in ascending order according to the number of existing virtual domains, and if the number of virtual domains of the first nodes in parallel positions is the same, then sorting is performed in descending order according to computing power.

[0098] Based on the above embodiment, further comprising:

[0099] The first node sends a heartbeat message to the management node, where the heartbeat message includes usage information of the physical storage space of the first node.

[0100] When the storage capacity of the first node increases, the first node sends a capacity expansion request to the management node, and the management node divides the expanded storage capacity into the default storage space of the virtual domain to which the first node belongs.

[0101] When the first node goes offline, the management node reduces the logical storage space of each virtual domain according to the virtual domains to which the physical storage space of the first node is divided.

[0102] Based on the above embodiment, further comprising:

[0103] When the deployment of the newly added second node is completed, the newly added second node sends a registration request to the management node, the management node specifies the collection port of the newly added second node, and the newly added second node receives cold data from the specified collection port.

[0104] In this embodiment, the concept of virtual domain is introduced in the field of full-flow cluster storage, and users only need to pay attention to the storage configuration at the business level. The cluster node storage is regarded as a whole space, and the virtual domain is a part of the space on it. The increase or decrease of nodes will only affect the total capacity and will not have any impact on the existing data, avoiding the risk of data migration when adding or reducing nodes. The hot and cold data are separated, and when querying hot data, resources can be exclusively used to speed up the retrieval performance, which solves the problem of resource preemption that may cause reduced retrieval performance.

[0105] Corresponding to the above method provided by the present invention, Fig. 9 A schematic diagram of the structure of a full-flow storage cluster device for separating hot and cold data provided in an embodiment of this specification is shown.

[0106] The full-flow storage cluster device includes at least one management node, a first subcluster and a second subcluster. The first subcluster includes at least one first node, and the second subcluster includes at least one second node. The management node is used to manage and monitor the first subcluster and the second subcluster. The first subcluster is used to store cold data, and the second subcluster is used to store hot data.

[0107] A management node is used to send configuration information to the first subcluster and the second subcluster, wherein the configuration information includes information of virtual domains, the virtual domains are used to store traffic data packets of different business attributes in different logical storage spaces, the logical storage space of a virtual domain includes a physical storage area in at least one first node in the first subcluster, the physical storage space of the first node is divided into at least one physical storage area, and each physical storage area belongs to a virtual domain.

[0108] At least one second node is used to receive cold data, determine the virtual domain to which the cold data belongs and the physical storage area to be stored in the virtual domain according to the configuration information, and send the cold data to the first node to which the physical storage area to be stored belongs.

[0109] At least one second node is used to obtain hot data of the cold data based on the received cold data, and store the hot data in a local storage area, wherein the cold data is a traffic data packet, and the hot data is a traffic log and statistical data generated based on the traffic data packet.

[0110] At least one first node is used to receive the cold data sent by the second node, and store the received cold data in a corresponding physical storage area.

[0111] Based on the above embodiment, further, the second node is specifically used to parse the cold data, obtain the metadata of the cold data, and use the metadata of the cold data to find the first message processing strategy, wherein the first message processing strategy is a virtual domain storage strategy configured according to the dimensional information of the traffic message.

[0112] If the first message processing policy is matched, the virtual domain to which the cold data belongs is determined according to the priority of the matched policy.

[0113] If the first message processing strategy is not matched, the second message processing strategy is searched to determine the virtual domain to which the cold data belongs, and the second message processing strategy is the virtual domain storage strategy of the system default flow message.

[0114] Based on the physical storage area of ​​the first node in the virtual domain and the amount of traffic data that has been forwarded to the first node in the virtual domain, the space utilization rate of the physical storage area of ​​the first node in the virtual domain is calculated, and the physical storage area of ​​the first node with the lowest space utilization rate in the virtual domain is used as the physical storage area where cold data will be stored in the virtual domain.

[0115] Based on the above embodiment, further, when a new first node is added, the new first node sends a registration request to the management node, and the registration request message includes the physical storage space capacity of the new first node.

[0116] The management node adds the newly added first node to the first sub-cluster, divides the physical storage space of the newly added first node, and allocates the physical storage area in the newly added first node to different virtual domains.

[0117] Based on the above embodiment, further, the management node is configured to receive a capacity increase request and determine whether the capacity of the unallocated physical storage space in the first sub-cluster satisfies the storage capacity requested to be increased in the capacity increase request.

[0118] If so, it is determined whether to increase storage space for an existing virtual domain or to increase storage space for a newly added virtual domain.

[0119] If the storage space is to be increased for an existing virtual domain, the remaining physical storage space in all the first nodes in the existing virtual domain is preferentially allocated.

[0120] If the remaining physical storage space in the first node in the existing virtual domain meets the storage capacity requested to be increased, the storage space of the first node in the existing virtual domain is divided.

[0121] If the remaining physical storage space in each first node in the existing virtual domain does not satisfy the storage capacity requested to be increased, the physical storage space of the first node that does not belong to the existing virtual domain is divided.

[0122] If the storage space is to be increased for the newly added virtual domain, the physical storage space of the first node that does not belong to the existing virtual domain is divided.

[0123] Based on the above embodiments, further, the management node is specifically used to execute step S51, and sorts the first nodes in the first sub-cluster from high to low according to the priority of descending order of total storage space capacity of the first node, descending order of remaining unallocated storage space capacity in the first node, ascending order of the number of virtual domains existing on the first node, and descending order of computing power of the first node, to obtain an allocation sequence, which includes a preset number of first nodes as pre-selected nodes.

[0124] Among them, the first nodes in the first subcluster are sorted from high to low according to the priority of descending order of total storage space capacity of the first node, descending order of remaining unallocated storage space capacity in the first node, ascending order of number of virtual domains existing on the first node and descending order of computing power of the first node, specifically including: firstly sorting the first nodes in descending order according to the total storage space capacity on the node to obtain a first sequence, sorting the first nodes in parallel positions in the first sequence in descending order according to remaining unallocated storage space capacity, if the first nodes in parallel positions still have the same remaining unallocated storage space capacity, then sorting them in ascending order according to the number of existing virtual domains, if the first nodes in parallel positions still have the same number of virtual domains, then sorting them in descending order according to computing power.

[0125] Execute step S52 to divide the physical storage capacity of the pre-selected nodes, increase the capacity of the storage space according to the request, poll and traverse the pre-selected nodes in the allocation sequence according to the minimum storage space unit, and divide the minimum storage space unit on the physical storage space of each pre-selected node.

[0126] Step S53 is executed to determine whether the remaining storage space capacity of the pre-selected node is greater than the minimum storage space unit.

[0127] If so, after each minimum storage space unit is allocated on each pre-selected node, the pre-selected nodes in the allocation sequence are re-sorted in descending order according to the total storage space capacity of the first node, in descending order according to the remaining unallocated storage space capacity in the first node, in ascending order according to the number of virtual domains existing on the first node, and in descending order according to the computing power of the first node, the allocation sequence is updated, and step S52 is repeated until the capacity of the requested increase in storage space is fully allocated.

[0128] Otherwise, remove the pre-selected nodes whose remaining storage space capacity is less than the minimum storage space unit from the current allocation sequence, re-sort the pre-selected nodes in the current allocation sequence according to the priority of descending order of the total storage space capacity of the first node, descending order of the remaining unallocated storage space capacity in the first node, ascending order of the number of virtual domains existing on the first node, and descending order of the computing power of the first node, update the allocation sequence, and repeat step S52 until the capacity of the requested increase in storage space is fully allocated.

[0129] Based on the above embodiment, further, the management node is used to receive a capacity reduction request of a virtual domain, and reduce the capacity of the physical storage area of ​​the first node in the virtual domain by storage units according to the sequence of the first node in the recovery sequence obtained by the second preset sorting method, and the reduced physical storage area is reclaimed by the default virtual domain, wherein the second preset sorting method is to sort in descending order according to the allocated capacity of each first node allocated to the virtual domain, in ascending order according to the remaining unallocated space of the first node, in descending order according to the number of virtual domains existing on the first node, and in ascending order according to the computing power, and sort from high to low priority to obtain the recovery sequence.

[0130] Sorting is performed from high to low priority according to the allocated capacity in descending order of each first node that has been allocated to the virtual domain, sorting in ascending order according to the remaining unallocated space of the first node, sorting in descending order according to the number of virtual domains existing on the first node, and sorting in ascending order according to computing power, specifically including: firstly sorting the first nodes in descending order according to the total storage space capacity on the node to obtain a second sequence, sorting the first nodes in parallel positions in the second sequence in descending order according to the remaining unallocated storage space capacity, if the remaining unallocated storage space capacity of the first nodes in parallel positions is the same, then sorting is performed in ascending order according to the number of existing virtual domains, and if the number of virtual domains of the first nodes in parallel positions is the same, then sorting is performed in descending order according to computing power.

[0131] Based on the above embodiment, further, the first node is used to send a heartbeat message to the management node, and the heartbeat message includes usage information of the physical storage space of the first node.

[0132] When the storage capacity of the first node increases, the first node sends a capacity expansion request to the management node, and the management node divides the expanded storage capacity into the default storage space of the virtual domain to which the first node belongs.

[0133] When the first node goes offline, the management node reduces the logical storage space of each virtual domain according to the virtual domains to which the physical storage space of the first node is divided.

[0134] Based on the above embodiment, further, when the newly added second node is deployed, the newly added second node is used to send a registration request to the management node, the management node specifies the collection port of the newly added second node, and the newly added second node receives cold data from the specified collection port.

[0135] In this embodiment, the concept of virtual domain is introduced in the field of full-flow cluster storage, and users only need to pay attention to the storage configuration at the business level. The cluster node storage is regarded as a whole space, and the virtual domain is a part of the space on it. The increase or decrease of nodes will only affect the total capacity and will not have any impact on the existing data, avoiding the risk of data migration when adding or reducing nodes. The hot and cold data are separated, and when querying hot data, resources can be exclusively used to speed up the retrieval performance, which solves the problem of resource preemption that may cause reduced retrieval performance.

[0136] Corresponding to the above method provided by the present invention, Fig.10 A schematic diagram of the structure of a full-flow storage cluster system with cold and hot data separation provided in an embodiment of this specification is shown.

[0137] The full-flow storage cluster system includes at least one management node, a first subcluster and a second subcluster. The first subcluster includes at least one first node, and the second subcluster includes at least one second node. The management node is used to manage and monitor the first subcluster and the second subcluster. The first subcluster is used to store cold data, and the second subcluster is used to store hot data.

[0138] A management node is used to send configuration information to the first subcluster and the second subcluster, wherein the configuration information includes information of virtual domains, the virtual domains are used to store traffic data packets of different business attributes in different logical storage spaces, the logical storage space of a virtual domain includes a physical storage area in at least one first node in the first subcluster, the physical storage space of the first node is divided into at least one physical storage area, and each physical storage area belongs to a virtual domain.

[0139] At least one second node is used to receive cold data, determine the virtual domain to which the cold data belongs and the physical storage area to be stored in the virtual domain according to the configuration information, and send the cold data to the first node to which the physical storage area to be stored belongs.

[0140] At least one second node is used to obtain hot data of the cold data based on the received cold data, and store the hot data in a local storage area, wherein the cold data is a traffic data packet, and the hot data is a traffic log and statistical data generated based on the traffic data packet.

[0141] At least one first node is used to receive the cold data sent by the second node, and store the received cold data in a corresponding physical storage area.

[0142] Based on the above embodiment, further, the second node is specifically used to parse the cold data, obtain the metadata of the cold data, and use the metadata of the cold data to find the first message processing strategy, wherein the first message processing strategy is a virtual domain storage strategy configured according to the dimensional information of the traffic message.

[0143] If the first message processing policy is matched, the virtual domain to which the cold data belongs is determined according to the priority of the matched policy.

[0144] If the first message processing strategy is not matched, the second message processing strategy is searched to determine the virtual domain to which the cold data belongs, and the second message processing strategy is the virtual domain storage strategy of the system default flow message.

[0145] Based on the physical storage area of ​​the first node in the virtual domain and the amount of traffic data that has been forwarded to the first node in the virtual domain, the space utilization rate of the physical storage area of ​​the first node in the virtual domain is calculated, and the physical storage area of ​​the first node with the lowest space utilization rate in the virtual domain is used as the physical storage area where cold data will be stored in the virtual domain.

[0146] Based on the above embodiment, further, when a new first node is added, the new first node sends a registration request to the management node, and the registration request message includes the physical storage space capacity of the new first node.

[0147] The management node adds the newly added first node to the first sub-cluster, divides the physical storage space of the newly added first node, and allocates the physical storage area in the newly added first node to different virtual domains.

[0148] Based on the above embodiment, further, the management node is configured to receive a capacity increase request and determine whether the capacity of the unallocated physical storage space in the first sub-cluster satisfies the storage capacity requested to be increased in the capacity increase request.

[0149] If so, it is determined whether to increase storage space for an existing virtual domain or to increase storage space for a newly added virtual domain.

[0150] If the storage space is to be increased for an existing virtual domain, the remaining physical storage space in all the first nodes in the existing virtual domain is preferentially allocated.

[0151] If the remaining physical storage space in the first node in the existing virtual domain meets the storage capacity requested to be increased, the storage space of the first node in the existing virtual domain is divided.

[0152] If the remaining physical storage space in each first node in the existing virtual domain does not satisfy the storage capacity requested to be increased, the physical storage space of the first node that does not belong to the existing virtual domain is divided.

[0153] If the storage space is to be increased for the newly added virtual domain, the physical storage space of the first node that does not belong to the existing virtual domain is divided.

[0154] Based on the above embodiments, further, the management node is specifically used to execute step S51, and sorts the first nodes in the first sub-cluster from high to low according to the priority of descending order of total storage space capacity of the first node, descending order of remaining unallocated storage space capacity in the first node, ascending order of the number of virtual domains existing on the first node, and descending order of computing power of the first node, to obtain an allocation sequence, which includes a preset number of first nodes as pre-selected nodes.

[0155] Among them, the first nodes in the first subcluster are sorted from high to low according to the priority of descending order of total storage space capacity of the first node, descending order of remaining unallocated storage space capacity in the first node, ascending order of number of virtual domains existing on the first node and descending order of computing power of the first node, specifically including: firstly sorting the first nodes in descending order according to the total storage space capacity on the node to obtain a first sequence, sorting the first nodes in parallel positions in the first sequence in descending order according to remaining unallocated storage space capacity, if the first nodes in parallel positions still have the same remaining unallocated storage space capacity, then sorting them in ascending order according to the number of existing virtual domains, if the first nodes in parallel positions still have the same number of virtual domains, then sorting them in descending order according to computing power.

[0156] Execute step S52 to divide the physical storage capacity of the pre-selected nodes, increase the capacity of the storage space according to the request, poll and traverse the pre-selected nodes in the allocation sequence according to the minimum storage space unit, and divide the minimum storage space unit on the physical storage space of each pre-selected node.

[0157] Step S53 is executed to determine whether the remaining storage space capacity of the pre-selected node is greater than the minimum storage space unit.

[0158] If so, after each minimum storage space unit is allocated on each pre-selected node, the pre-selected nodes in the allocation sequence are re-sorted in descending order according to the total storage space capacity of the first node, in descending order according to the remaining unallocated storage space capacity in the first node, in ascending order according to the number of virtual domains existing on the first node, and in descending order according to the computing power of the first node, the allocation sequence is updated, and step S52 is repeated until the capacity of the requested increase in storage space is fully allocated.

[0159] Otherwise, remove the pre-selected nodes whose remaining storage space capacity is less than the minimum storage space unit from the current allocation sequence, re-sort the pre-selected nodes in the current allocation sequence according to the priority of descending order of the total storage space capacity of the first node, descending order of the remaining unallocated storage space capacity in the first node, ascending order of the number of virtual domains existing on the first node, and descending order of the computing power of the first node, update the allocation sequence, and repeat step S52 until the capacity of the requested increase in storage space is fully allocated.

[0160] Based on the above embodiment, further, the management node is used to receive a capacity reduction request of a virtual domain, and reduce the capacity of the physical storage area of ​​the first node in the virtual domain by storage units according to the sequence of the first node in the recovery sequence obtained by the second preset sorting method, and the reduced physical storage area is reclaimed by the default virtual domain, wherein the second preset sorting method is to sort in descending order according to the allocated capacity of each first node allocated to the virtual domain, in ascending order according to the remaining unallocated space of the first node, in descending order according to the number of virtual domains existing on the first node, and in ascending order according to the computing power, and sort from high to low priority to obtain the recovery sequence.

[0161] Sorting is performed from high to low priority according to the allocated capacity in descending order of each first node that has been allocated to the virtual domain, sorting in ascending order according to the remaining unallocated space of the first node, sorting in descending order according to the number of virtual domains existing on the first node, and sorting in ascending order according to computing power, specifically including: firstly sorting the first nodes in descending order according to the total storage space capacity on the node to obtain a second sequence, sorting the first nodes in parallel positions in the second sequence in descending order according to the remaining unallocated storage space capacity, if the remaining unallocated storage space capacity of the first nodes in parallel positions is the same, then sorting is performed in ascending order according to the number of existing virtual domains, and if the number of virtual domains of the first nodes in parallel positions is the same, then sorting is performed in descending order according to computing power.

[0162] Based on the above embodiment, further, the first node is used to send a heartbeat message to the management node, and the heartbeat message includes usage information of the physical storage space of the first node.

[0163] When the storage capacity of the first node increases, the first node sends a capacity expansion request to the management node, and the management node divides the expanded storage capacity into the default storage space of the virtual domain to which the first node belongs.

[0164] When the first node goes offline, the management node reduces the logical storage space of each virtual domain according to the virtual domains to which the physical storage space of the first node is divided.

[0165] Based on the above embodiment, further, when the newly added second node is deployed, the newly added second node is used to send a registration request to the management node, the management node specifies the collection port of the newly added second node, and the newly added second node receives cold data from the specified collection port.

[0166] Those skilled in the art should be aware that in one or more of the above examples, the functions described in this application can be implemented by hardware, software, firmware, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0167] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation methods of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the scope of protection of the present application.

Claims

1. A full-flow storage cluster management method with hot and cold data separation, characterized in that: The full-flow storage cluster includes at least one management node, a first subcluster and a second subcluster, the first subcluster includes at least one first node, the second subcluster includes at least one second node, the management node is used to manage and monitor the first subcluster and the second subcluster, the first subcluster is used to store cold data, and the second subcluster is used to store hot data; The method comprises: The management node sends configuration information to the first subcluster and the second subcluster, wherein the configuration information includes information of virtual domains, the virtual domains are used to store traffic data packets of different service attributes in different logical storage spaces, the logical storage space of a virtual domain includes a physical storage area in at least one first node in the first subcluster, the physical storage space of the first node is divided into at least one physical storage area, and each physical storage area belongs to a virtual domain; At least one of the second nodes receives the cold data, determines the virtual domain to which the cold data belongs and the physical storage area to be stored in the virtual domain according to the configuration information, and sends the cold data to the first node to which the physical storage area to be stored belongs; At least one of the second nodes obtains hot data of the cold data according to the received cold data, and stores the hot data in a local storage area, wherein the cold data is a traffic data packet, and the hot data is a traffic log and statistical data generated based on the traffic data packet; At least one of the first nodes receives the cold data sent by the second node, and stores the received cold data in a corresponding physical storage area.

2. The method according to claim 1, characterized in that: The at least one second node receives the cold data, and determines, according to the configuration information, a virtual domain to which the cold data belongs and a physical storage area to be stored in the virtual domain, specifically including: The second node parses the cold data to obtain metadata of the cold data, and uses the metadata of the cold data to search for a first message processing strategy, wherein the first message processing strategy is a virtual domain storage strategy configured according to dimension information of the traffic message; If the first message processing strategy is hit, determining the virtual domain to which the cold data belongs according to the priority of the hit strategy; If the first message processing strategy is not hit, then searching for a second message processing strategy to determine the virtual domain to which the cold data belongs, wherein the second message processing strategy is a virtual domain storage strategy for a system default flow message; Based on the physical storage area of ​​the first node in the virtual domain and the amount of traffic data that has been forwarded to the first node in the virtual domain, the space utilization rate of the physical storage area of ​​the first node in the virtual domain is calculated, and the physical storage area of ​​the first node with the lowest space utilization rate in the virtual domain is used as the physical storage area where the cold data will be stored in the virtual domain.

3. The method according to claim 1, characterized in that The method further comprises: When a new first node is added, the new first node sends a registration request to the management node, and the registration request includes the physical storage space capacity of the new first node; The management node adds the newly added first node to the first sub-cluster, divides the physical storage space of the newly added first node, and allocates the physical storage area in the newly added first node to different virtual domains.

4. The method according to claim 1, characterized in that The method further comprises: The management node receives the capacity increase request, and determines whether the capacity of the unallocated physical storage space in the first sub-cluster satisfies the storage capacity requested to be increased in the capacity increase request; If so, determine whether to increase storage space for an existing virtual domain or to increase storage space for a newly added virtual domain; If the storage space is increased for an existing virtual domain, the remaining physical storage space in all the first nodes in the existing virtual domain is preferentially allocated; If the remaining physical storage space in the first node in the existing virtual domain meets the storage capacity requested for increase, dividing the storage space of the first node in the existing virtual domain; If the remaining physical storage space in each first node in the existing virtual domain does not satisfy the storage capacity requested to be increased, dividing the physical storage space of the first node that does not belong to the existing virtual domain; If the storage space is to be increased for the newly added virtual domain, the physical storage space of the first node that does not belong to the existing virtual domain is divided.

5. The method according to claim 4, characterized in that The dividing of the physical storage space of the first node that does not belong to the existing virtual domain specifically includes: S51, sorting the first nodes in the first sub-cluster in descending order according to the priorities of the total storage space capacity of the first nodes, the remaining unallocated storage space capacity of the first nodes, the number of virtual domains existing on the first nodes, and the computing power of the first nodes, from high to low, to obtain an allocation sequence, wherein the allocation sequence includes a preset number of first nodes as pre-selected nodes; Among them, the first nodes in the first sub-cluster are sorted from high to low according to the priority of descending order of total storage space capacity of the first node, descending order of remaining unallocated storage space capacity in the first node, ascending order of number of virtual domains existing on the first node and descending order of computing power of the first node, specifically including: firstly sorting the first nodes in descending order according to the total storage space capacity on each node to obtain a first sequence, sorting the first nodes in parallel positions in the first sequence in descending order according to remaining unallocated storage space capacity, if the first nodes in parallel positions still have the same remaining unallocated storage space capacity, then sorting them in ascending order according to the number of existing virtual domains, if the first nodes in parallel positions still have the same number of virtual domains, then sorting them in descending order according to computing power; S52, dividing the physical storage capacity of the pre-selected nodes, increasing the capacity of the storage space according to the request, polling and traversing the pre-selected nodes in the allocation sequence according to the minimum storage space unit, and dividing the minimum storage space unit on the physical storage space of each pre-selected node; S53, determining whether the remaining storage space capacity of the pre-selected node is greater than the minimum storage space unit; If so, after each minimum storage space unit is allocated on each of the pre-selected nodes, the pre-selected nodes in the allocation sequence are re-sorted in descending order according to the total storage space capacity of the first node, in descending order according to the remaining unallocated storage space capacity of the first node, in ascending order according to the number of virtual domains existing on the first node, and in descending order according to the computing power of the first node, and the allocation sequence is updated, and step S52 is repeatedly executed until the capacity of the requested increase in storage space is fully allocated; Otherwise, remove the pre-selected nodes whose remaining storage space capacity is less than the minimum storage space unit from the current allocation sequence, re-sort the pre-selected nodes in the current allocation sequence according to the priority of descending order of the total storage space capacity of the first node, descending order of the remaining unallocated storage space capacity in the first node, ascending order of the number of virtual domains existing on the first node, and descending order of the computing power of the first node, update the allocation sequence, and repeat step S52 until the capacity of the requested increase in storage space is fully allocated.

6. The method according to claim 1, characterized in that The method further comprises: The management node receives a request to reduce the capacity of the virtual domain, and according to the sequence of the first node in the recovery sequence obtained in a second preset sorting method, the capacity of the physical storage area of ​​the first node in the virtual domain is reduced by storage units, and the reduced physical storage area is recovered by the default virtual domain, wherein the second preset sorting method is to sort in descending order according to the allocated capacity of each first node that has been divided into the virtual domain, in ascending order according to the remaining unallocated space of the first node, in descending order according to the number of virtual domains existing on the first node, and in ascending order according to the computing power, and the recovery sequence is obtained by sorting from high to low priority; Sorting is performed from high to low priority according to the descending order of the allocated capacity of the first node that has been allocated to the virtual domain, the ascending order of the remaining unallocated space of the first node, the descending order of the number of virtual domains existing on the first node, and the ascending order of computing power, specifically including: firstly sorting the first nodes of the virtual domain in descending order according to the total storage space capacity on the node to obtain a second sequence, sorting the first nodes in a parallel position in the second sequence in descending order according to the remaining unallocated storage space capacity, if the first nodes in a parallel position still have the same remaining unallocated storage space capacity, then sorting them in ascending order according to the number of existing virtual domains, and if the first nodes in a parallel position still have the same number of virtual domains, then sorting them in descending order according to computing power.

7. The method according to claim 1, characterized in that The method further comprises: The first node sends a heartbeat message to the management node, wherein the heartbeat message includes usage information of the physical storage space of the first node; When the storage capacity of the first node increases, the first node sends a capacity expansion request to the management node, and the management node divides the expanded storage capacity into a default storage space of the virtual domain to which the first node belongs; When the first node goes offline, the management node reduces the logical storage space of each virtual domain according to the virtual domains to which the physical storage space of the first node is divided.

8. The method according to claim 1, characterized in that: The method further comprises: When the deployment of the newly added second node is completed, the newly added second node sends a registration request to the management node, the management node specifies a collection port of the newly added second node, and the newly added second node receives the cold data from the specified collection port.

9. A full-flow storage cluster device with hot and cold data separation, characterized in that: The full-flow storage cluster device includes at least one management node, a first subcluster and a second subcluster, the first subcluster includes at least one first node, the second subcluster includes at least one second node, the management node is used to manage and monitor the first subcluster and the second subcluster, the first subcluster is used to store cold data, and the second subcluster is used to store hot data; The management node is used to send configuration information to the first subcluster and the second subcluster, wherein the configuration information includes information of virtual domains, the virtual domains are used to store traffic data packets of different service attributes in different logical storage spaces, the logical storage space of a virtual domain includes a physical storage area in at least one first node in the first subcluster, the physical storage space of the first node is divided into at least one physical storage area, and each physical storage area belongs to a virtual domain; at least one of the second nodes is configured to receive cold data, determine the virtual domain to which the cold data belongs and the physical storage area to be stored in the virtual domain according to the configuration information, and send the cold data to the first node to which the physical storage area to be stored belongs; at least one of the second nodes, configured to obtain hot data of the cold data according to the received cold data, and store the hot data in a local storage area, wherein the cold data is a traffic data packet, and the hot data is a traffic log and statistical data generated based on the traffic data packet; At least one of the first nodes is used to receive the cold data sent by the second node, and store the received cold data in a corresponding physical storage area.

10. A full-flow storage cluster system with hot and cold data separation, characterized in that: The full-flow storage cluster system includes at least one management node, a first subcluster and a second subcluster, the first subcluster includes at least one first node, the second subcluster includes at least one second node, the management node is used to manage and monitor the first subcluster and the second subcluster, the first subcluster is used to store cold data, and the second subcluster is used to store hot data; The management node is used to send configuration information to the first subcluster and the second subcluster, wherein the configuration information includes information of virtual domains, the virtual domains are used to store traffic data packets of different service attributes in different logical storage spaces, the logical storage space of a virtual domain includes a physical storage area in at least one first node in the first subcluster, the physical storage space of the first node is divided into at least one physical storage area, and each physical storage area belongs to a virtual domain; at least one of the second nodes is configured to receive cold data, determine the virtual domain to which the cold data belongs and the physical storage area to be stored in the virtual domain according to the configuration information, and send the cold data to the first node to which the physical storage area to be stored belongs; at least one of the second nodes, configured to obtain hot data of the cold data according to the received cold data, and store the hot data in a local storage area, wherein the cold data is a traffic data packet, and the hot data is a traffic log and statistical data generated based on the traffic data packet; At least one of the first nodes is used to receive the cold data sent by the second node, and store the received cold data in a corresponding physical storage area.

Images