An auxiliary storage method of a big data platform

Abstract

The application discloses an auxiliary storage method of a big data platform, wherein the big data platform is composed of information nodes and a plurality of distributed data storage nodes, and comprises the following steps: a. big data platform cluster information detection; b. auxiliary storage node configuration; c. judging whether an auxiliary storage operation is triggered; and d. respectively executing different auxiliary storage operations according to trigger conditions. According to different problems of the big data platform, the application realizes targeted auxiliary storage operation, reasonably plans the circulation of data blocks among different distributed data storage nodes, reduces unnecessary and repeated read-write operations among original data storage nodes of the big data platform, reduces the read-write pressure and storage pressure of each node, and greatly shortens the data storage time.

Description

Technical Field

[0001] This invention belongs to the field of big data storage technology, and specifically relates to an auxiliary storage method for a big data platform. Background Technology

[0002] In recent years, with the rapid development of information technology, all industries are constantly generating massive amounts of data, resulting in an explosive growth in data volume. Traditional data storage and processing models can no longer meet the demands of today's era for high-speed and sensitive responses. In order to better utilize data based on data analysis, distributed and virtualization technologies are typically used to build big data platforms for storing this massive amount of data.

[0003] However, with the continuous growth of system business scale and user numbers, the data storage capacity and access frequency of big data platforms are increasing exponentially. Existing traditional distributed big data platform solutions involve a large amount of repetitive data read / write work during system expansion, data balancing, fault recovery, and replication to ensure data integrity and consistency. This places enormous additional data read / write and storage pressure on each distributed storage node, resulting in extremely slow platform response times, or even the collapse of the entire big data platform, leading to widespread data loss.

[0004] To address these issues, existing solutions (such as patent CN202011262921.7) employ platform expansion, reducing read / write and storage pressure on existing nodes in the cluster by adding compute and storage nodes. This approach lacks consideration for the bottlenecks of individual nodes and fails to optimize and integrate existing node resources, resulting in resource waste. Another solution (such as patent CN201611029700.9) uses asynchronous I / O and structured data processing, improving the data storage rate of the big data platform through data distribution. This approach optimizes data before it enters the big data platform and distributes it to appropriate big data platform nodes, but it lacks optimization strategies for data after it enters the platform and does not optimize the data processing methods and processes of the existing big data platform nodes. Summary of the Invention

[0005] This invention provides an auxiliary storage method for big data platforms, which avoids a large amount of repetitive data transmission between distributed storage nodes in big data platforms, and reduces the read / write pressure and storage pressure of each node.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An auxiliary storage method for a big data platform, wherein the big data platform comprises information nodes and multiple distributed data storage nodes, wherein the information nodes manage the namespace and data block mapping information of the platform's storage system, and the data storage nodes store the actual data blocks and perform specific data block read and write operations, the method comprising the following steps:

[0008] Step 1: Big Data Platform Cluster Information Detection. The big data platform cluster information includes, but is not limited to, information related to big data platform information nodes and distributed data storage nodes, platform storage resource occupancy rate, replication status, service health status of data storage nodes, storage resource occupancy of data storage nodes, and load status of data storage nodes (IO, CPU, and memory resource occupancy).

[0009] Step 2, Auxiliary storage node configuration. Based on the information detected in Step 1, configure a corresponding auxiliary storage node for each distributed data storage node, and configure a platform auxiliary storage node for the entire big data platform.

[0010] Step 3: Determine whether auxiliary storage operations are triggered. The determination conditions include, but are not limited to, whether the overall big data platform storage resource utilization rate Si exceeds the threshold Ts, whether there is a data storage node failure or disconnection, whether a new data storage node is added, and whether the load pressure of a data storage node exceeds the threshold.

[0011] Furthermore, the load pressure of the data storage node includes the utilization rate of the server's CPU, memory, disk space, and disk I / O rate, represented by ci, mi, di, and io, respectively. Fi = MAX(ci / Tc, mi / Tm, di / Td, io / Tio), where Tc, Tm, Td, and Tio are the pressure alarm thresholds for the data storage node's CPU, memory, disk space, and disk I / O, respectively. When the Fi of the data storage node is greater than or equal to 1, it indicates that the load pressure of the data storage node is too high, and auxiliary storage operations will be triggered; otherwise, they will not.

[0012] Step 4: Perform different auxiliary storage operations according to the triggering conditions in Step 3.

[0013] Furthermore, when the overall big data platform storage resource utilization Si exceeds the threshold Ts, the auxiliary storage operation includes the following steps:

[0014] Step 401: Start the platform auxiliary storage node service and write all newly added data blocks into the platform auxiliary storage node;

[0015] Step 402: Feed back the mapping information of the newly added data block to the information node of the big data platform for overwriting;

[0016] Step 403: Alarm feedback, reminding users to expand the storage capacity of the big data platform; the alarm methods include, but are not limited to, email alarms, SMS alarms, and big data platform interface alarms.

[0017] Furthermore, when a data storage node fails and becomes unreachable, the auxiliary storage operation includes the following steps:

[0018] Step 411: Obtain information on the faulty and disconnected data storage node; the storage node information includes the mapping information of the data blocks in the node;

[0019] Step 412: Enable the auxiliary storage node service for the corresponding storage node;

[0020] Step 413: Copy the data blocks from the original failed and disconnected data storage node from the replicas of other data storage nodes on the platform to the corresponding auxiliary storage node;

[0021] Step 414: Transfer the data blocks originally intended to be written to the faulty and disconnected data storage node to the corresponding auxiliary storage node;

[0022] Step 415: Feed back the data block mapping information in the auxiliary storage node to the information node of the big data platform for overwriting;

[0023] Step 416: Wait for the faulty and disconnected data storage node to recover;

[0024] Step 417: Compare the data block differences between the faulty and disconnected storage node and the corresponding auxiliary storage node, and incrementally write the difference data blocks into the recovered storage node;

[0025] Step 418: Feed back the mapping information of the difference data blocks to the information nodes of the big data platform for overlay;

[0026] Furthermore, when a new data storage node is added, the auxiliary storage operation includes the following steps:

[0027] Step 421: Determine whether the storage resource utilization rate Si of the entire big data platform exceeds the threshold Ts;

[0028] Step 422: Write all data blocks from the already activated platform auxiliary storage nodes into the newly added big data platform data storage node;

[0029] Step 423: Feed back the mapping information of the corresponding data block to the information node of the big data platform for overlay;

[0030] Step 424: The big data platform performs a balanced adjustment of the data block distribution in the existing data storage nodes and the newly added data storage nodes;

[0031] Step 425: Configure the corresponding auxiliary storage node for the newly added data storage node;

[0032] Furthermore, when the load pressure on a data storage node exceeds a threshold, the auxiliary storage operation includes the following steps:

[0033] Step 431: Obtain information on data storage nodes whose load pressure exceeds the threshold;

[0034] Step 432: Enable the auxiliary storage node service for the corresponding storage node;

[0035] Step 433: Transfer the data blocks originally intended to be written to the high-load data storage node to the corresponding auxiliary storage node;

[0036] Step 434: Feed back the data block mapping information in the auxiliary storage node to the information node of the big data platform for overwriting;

[0037] Step 435: Wait for the pressure on the high-load data storage nodes to fall below the threshold;

[0038] Step 436: Write the data blocks in the auxiliary storage node into the corresponding data storage node in the big data platform;

[0039] Step 437: Feed back the mapping information of the corresponding data block to the information node of the big data platform for overlay;

[0040] To address the aforementioned technical problems, the present invention also provides an auxiliary storage device, which specifically includes: a detection module, a synchronization module, a processing module, and a storage module;

[0041] The detection module is used for big data platform cluster information detection to obtain information related to big data platform information nodes and distributed storage nodes.

[0042] Furthermore, the detection module is also used to collect information on the occupancy rate of storage resources across the entire big data platform;

[0043] Furthermore, the detection module is also used to monitor the health status of data storage nodes on the big data platform;

[0044] Furthermore, the detection module is also used to monitor changes in data storage resources of the big data platform; the changes in storage resources refer to the addition or reduction of data storage nodes.

[0045] Furthermore, the detection module is also used to collect information related to storage resource occupancy and load pressure of data storage nodes on the big data platform;

[0046] The synchronization module is used to synchronize data block mapping information in the information nodes of the big data platform. On the one hand, it acquires the mapping information of data blocks in the data storage nodes of the big data platform; on the other hand, it synchronizes the data block mapping information in the auxiliary storage nodes to the big data platform information nodes.

[0047] The processing module connects to the detection module, acquires relevant data, and determines whether auxiliary storage operation needs to be initiated.

[0048] Furthermore, the processing module is also used to interface with the synchronization module to obtain the specific mapping information of the data block, and to determine the business logic of the auxiliary storage operation and the specific flow of the data block;

[0049] Furthermore, the processing module is also used to assist in the specific implementation of storage operation business logic;

[0050] The storage module interfaces with the underlying hardware storage and performs specific data block read and write operations and data block compression optimizations according to the operation instructions of the processing module.

[0051] The beneficial effects of this invention compared to the prior art are:

[0052] 1) Based on the different problems of big data platforms, this invention rationally plans the flow of data blocks between different distributed data storage nodes through targeted auxiliary storage operations, reducing unnecessary and repetitive read and write operations between data storage nodes of the original big data platform, alleviating the read and write pressure and storage pressure of each node, and greatly shortening the data storage time.

[0053] 2) This invention enables the monitoring of storage resource occupancy across the entire big data platform. When the occupancy rate exceeds a threshold, auxiliary storage operations are initiated to store the overflowing data into the platform's auxiliary storage nodes, preventing large-scale data loss. Simultaneously, an alarm mechanism is provided for notification.

[0054] 3) This invention enables the monitoring of storage pressure and load pressure on each data storage node of the big data platform. When the pressure is too high, auxiliary storage operation is initiated, diverting the storage pressure and data read / write load pressure to the corresponding auxiliary storage node, thereby reducing the pressure on the data storage nodes of the big data platform and preventing the big data platform from experiencing task lag. Attached Figure Description

[0055] Figure 1 This is a flowchart of the big data platform auxiliary storage method described in this invention;

[0056] Figure 2 This is the auxiliary storage operation process described in this invention when the storage resource occupancy rate of the entire big data platform exceeds a threshold.

[0057] Figure 3 This is the auxiliary storage operation process described in this invention when a data storage node fails and becomes disconnected;

[0058] Figure 4This is the auxiliary storage operation process described in this invention when a new data storage node is added;

[0059] Figure 5 This is the auxiliary storage operation process described in this invention when the load pressure of a data storage node exceeds a threshold;

[0060] Figure 6 A schematic diagram of the auxiliary storage device module for the big data platform provided by the present invention; Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar components or components having the same or similar functions throughout. The described embodiments are some, but not all, of the embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0062] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0063] Example 1

[0064] Embodiment 1 of the present invention provides a method for auxiliary storage of a big data platform, the steps of which are as follows: Figure 1 As shown, the specific steps include:

[0065] Step S101: Big Data Platform Cluster Information Detection.

[0066] In this example, the big data platform cluster information is obtained in two ways. One way is by calling the relevant API interfaces of the big data platform management platform Ambari to obtain relevant parameters of the big data platform, such as information node information (master node: 10.188.0.10, standby node: 10.188.0.11) and information related to distributed data storage nodes (data nodes: 10.188.0.12-10.188.0.17), platform storage resource utilization (67.5%), replication status (3 replicas), and service health status of data storage nodes. The other way is by monitoring storage node server resources to obtain the storage resource utilization and load pressure of data storage nodes.

[0067] Step S102: Configure auxiliary storage nodes.

[0068] Based on the data storage node information obtained in step S101, a corresponding auxiliary storage node is configured for each distributed data storage node. Simultaneously, a platform auxiliary storage node is configured for the entire big data platform. The configuration of the auxiliary storage node is based on the storage space and load pressure of its corresponding data storage node; while the configuration of the platform auxiliary storage node is based on the data scale and expansion efficiency requirements of the entire big data platform. In this example, auxiliary storage nodes 10.66.0.12-10.66.0.17 are configured for data nodes 10.188.0.12-10.188.0.17, with system resource sizes identical to the corresponding data nodes. Platform auxiliary storage node 10.66.0.250 is configured, with a system resource size 30% of the total resource size of all data nodes on the platform; the 10.188.0 segment and the 10.66.0 segment are interconnected.

[0069] Step S103: Determine whether an auxiliary storage operation is triggered.

[0070] In practical applications, step S104, the auxiliary storage operation, will be triggered if one or more of the following conditions occur; otherwise, it will not.

[0071] The overall storage resource utilization rate Si of the big data platform exceeds the threshold Ts=90% in this embodiment of the invention;

[0072] The system detected that a data storage node on the big data platform was out of service, such as data node 10.188.0.17.

[0073] The big data platform has detected the addition of a new data storage node, such as a new data node 10.188.0.18.

[0074] The calculations show that some data storage nodes on the big data platform are under load exceeding a threshold.

[0075] Furthermore, the formula for calculating whether the load pressure on the data storage node exceeds the threshold is expressed as follows:

[0076] Fi=MAX(ci / Tc,mi / Tm,di / Td,io / Tio), where ci, mi, di, and io represent the CPU, memory, disk space utilization, and disk I / O rate of the data storage node server, respectively, and Tc, Tm, Td, and Tio are the pressure alarm thresholds for the CPU, memory, disk space, and disk I / O of the data storage node, respectively.

[0077] In this example, in step S101, the mi of data node 10.188.0.13 was detected to be 91%, while the system Tm was 90%. It was calculated that Fi was greater than 1 at this time, indicating that the load pressure on data node 10.188.0.13 was too high.

[0078] Step S104: Perform different auxiliary storage operations according to the triggering conditions.

[0079] Based on the four different judgment conditions in step S103, four targeted auxiliary storage operations will be triggered in this example; in actual applications, one or more of these auxiliary storage operations may be performed simultaneously.

[0080] Figure 2 The auxiliary storage operation process described in this invention when the storage resource utilization rate of the entire big data platform exceeds a threshold specifically includes the following steps:

[0081] Step S201: Start the platform auxiliary storage node service and write all newly added data blocks into the platform auxiliary storage node;

[0082] In this example, the platform auxiliary storage node 10.66.0.250 has already been configured in step S102, while step S201 completes the startup of the specific services and the allocation of storage resources for the platform auxiliary storage node 10.66.0.250, including the formatting of the storage disk and the initialization and startup of the DataNode service. The big data platform then writes the newly added data block BP-451827885 into the DataNode service of 10.66.0.250.

[0083] Step S202: Feed back the mapping information of the newly added data block to the information node of the big data platform for overwriting;

[0084] In this example, the mapping information for the newly added data block, specifically the storage location information of the newly added data block in the platform's auxiliary storage node (10.66.0.250: / data / hdfs / current / BP-451827885), is sent to the NameNode node in the big data platform for storage, so that it can be used for data access throughout the big data platform.

[0085] Step S203: Alarm feedback, reminding users to expand the storage capacity of the big data platform;

[0086] In practical applications, alarm methods include email alarms, SMS alarms, and alarms via big data platform interfaces;

[0087] Figure 3 The auxiliary storage operation process described in this invention when a data storage node fails and becomes unreachable specifically includes the following steps:

[0088] Step S301: Obtain information on the faulty and disconnected data storage node;

[0089] In this example, the storage node information specifically refers to the data blocks stored in the failed and disconnected data storage node 10.188.0.17, obtained from the NameNode node in the big data platform, along with their corresponding storage location information (e.g., if 10.188.0.17 stores data block BP-451325774, its three replica storage locations are: 10.188.0.17: / data / hdfs / current / BP-451325774, 10.188.0.12: / data / hdfs / current / BP-451325774, and 10.188.0.14: / data / hdfs / current / BP-451325774).

[0090] Step S302: Enable the auxiliary storage node service for the corresponding storage node;

[0091] In this example, the corresponding auxiliary storage node 10.66.0.17 has been configured in step S102, while step S302 completes the startup of the specific services and the allocation and occupation of storage resources for the corresponding auxiliary storage node 10.66.0.17.

[0092] Step S303: Copy the data blocks from the original failed and disconnected data storage node from the replicas of other data storage nodes on the platform to the corresponding auxiliary storage node;

[0093] In this example, step S301 reveals the replica storage locations of all data blocks in the failed, disconnected data storage node 10.188.0.17, allowing direct replication to the corresponding secondary storage node. (For example, copying data block BP-451325774 directly from replica 10.188.0.12: / data / hdfs / current / BP-451325774 to 10.66.0.17).

[0094] Step S304: Transfer the data block BP-451325792, which was originally intended to be written to the faulty and disconnected data storage node 10.188.0.17, to the corresponding auxiliary storage node 10.66.0.17;

[0095] Step S305: Feed back the data block mapping information (10.66.0.17: / data / hdfs / current / BP-451325792) in the auxiliary storage node to the NameNode of the big data platform for overwriting;

[0096] Step S306: Wait for the failed data storage node to recover;

[0097] In this example, the present invention periodically detects the health status of the data storage node 10.188.0.17. If the 10.188.0.17 node is still out of service, the process will jump to step S303 to continue execution; otherwise, the process will jump to step S307.

[0098] Step S307: Compare the data block differences between the failed and disconnected storage node and the corresponding auxiliary storage node, and incrementally write the difference data blocks into the recovered storage node;

[0099] Step S308: Feed back the mapping information of the differential data blocks to the information nodes of the big data platform for overlay.

[0100] In this example, the incremental data block BP-451325792 in the secondary storage node 10.66.0.17 will be written to the recovered faulty and disconnected data storage node 10.188.0.17, and the data block mapping information (10.188.0.17: / data / hdfs / current / BP-451325792) will be returned to the NameNode node in the big data platform for storage.

[0101] Figure 4 The auxiliary storage operation process described in this invention when a new data storage node is added specifically includes the following steps:

[0102] Step S401: Determine whether the storage resource utilization rate Si of the entire big data platform exceeds the threshold Ts=90%.

[0103] Step S402: Write all data blocks (such as BP-451827885, etc.) in the already enabled platform auxiliary storage node 10.66.0.250 into the newly added big data platform data storage node 10.188.0.18.

[0104] Adding new data storage nodes to a big data platform could be an expansion that exceeds the cluster's storage resource limits, or it could be a preventative expansion that does not exceed the limits. In this example, for an expansion that exceeds the limits, the overflowing data has already been stored in the platform's auxiliary storage node 10.66.0.250 in step S201; for a preventative expansion that does not exceed the limits, since it will not cause additional data read / write pressure on the platform, it is carried out in the traditional big data platform expansion method.

[0105] Step S403: Feed back the mapping information of the corresponding data block BP-451827885 (10.188.0.18: / data / hdfs / current / BP-451827885) to the NameNode of the big data platform for overwriting.

[0106] Step S404: The big data platform performs a balanced adjustment of the data block distribution in the original data storage nodes 10.188.0.12-10.188.0.17 and the newly added data storage node 10.188.0.18.

[0107] Step S405: Configure the corresponding auxiliary storage node 10.66.0.18 for the newly added data storage node.

[0108] Figure 5 The auxiliary storage operation process described in this invention when the load pressure of a data storage node exceeds a threshold specifically includes the following steps:

[0109] Step S501: Obtain information on data storage nodes whose load pressure exceeds the threshold (e.g., data node 10.188.0.13).

[0110] Step S502: Enable the auxiliary storage node service 10.66.0.13 for the corresponding storage node.

[0111] Step S503: Transfer the data block BP-451827932, which was originally intended to be written to the high-load data storage node 10.188.0.13, to the corresponding auxiliary storage node 10.66.0.13.

[0112] Step S504: Feed back the data block mapping information (10.66.0.13: / data / hdfs / current / BP-451827932) in the auxiliary storage node 10.66.0.13 to the NameNode of the big data platform for overwriting.

[0113] Step S505: Wait for the pressure on the high-load data storage node to fall below the threshold.

[0114] In this example, the present invention periodically detects the pressure of data storage node 10.188.0.13. When the load pressure is higher than the threshold (i.e., Fi is greater than 1), it will jump to step S503 to continue execution; otherwise, it will jump to step S506.

[0115] Step S506: Write data block BP-451827932 from auxiliary storage node 10.66.0.13 to the corresponding data storage node 10.188.0.13 in the big data platform.

[0116] Step S507: Feed back the mapping information of the corresponding data block BP-451827932 (10.188.0.13: / data / hdfs / current / BP-451827932) to the NameNode of the big data platform for overwriting.

[0117] Example 2

[0118] To achieve the above objectives, Example 2 of the present invention also provides an auxiliary storage device for a big data platform, the component modules of which are shown in the schematic diagram below. Figure 6 As shown, it specifically includes: a detection module 61, a synchronization module 62, a processing module 63, and a storage module 64.

[0119] The detection module 61 is used to collect information related to information nodes and distributed storage nodes of the big data platform.

[0120] The detection module 61 is also used to collect information on the occupancy rate of storage resources in the entire big data platform.

[0121] The detection module 61 is also used to monitor the health status of data storage nodes on the big data platform.

[0122] The detection module 61 is also used to monitor the increase or decrease of data storage nodes in the big data platform.

[0123] The detection module 61 is also used to collect information related to the storage resource occupancy and load pressure of the data storage nodes of the big data platform.

[0124] Synchronization module 62 is used to synchronize data block mapping information in the information node of the big data platform, including obtaining the mapping information of data blocks in the data storage node of the big data platform and synchronizing the data block mapping information in the auxiliary storage node to the information node of the big data platform.

[0125] Processing module 63 is used to determine whether auxiliary storage operation needs to be initiated.

[0126] The processing module 63 is also used to determine the business logic of the auxiliary storage operation and the specific flow of data blocks.

[0127] The processing module 63 is also used to assist in the specific implementation of the business logic of storage operations and the copying, migration and deletion operations of data blocks.

[0128] Storage module 64 is used to interface with the underlying hardware storage to perform specific data block read and write operations and data block compression and other optimization processes.

[0129] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An auxiliary storage method of a big data platform, characterized by The big data platform consists of information nodes and multiple distributed data storage nodes, and includes the following steps: a. Big data platform cluster information detection; b. Auxiliary storage node configuration; c. Determine if an auxiliary storage operation has been triggered; d. Execute different auxiliary storage operations according to the triggering conditions; The big data platform cluster information mentioned in step a includes: information related to big data platform information nodes and distributed data storage nodes, platform storage resource utilization rate, replication status, service health status of data storage nodes, storage resource utilization of data storage nodes, and load status of data storage nodes. The auxiliary storage nodes mentioned in step b include auxiliary storage nodes corresponding to each distributed data storage node and a platform auxiliary storage node for the entire big data platform. 2.The auxiliary storage method of a big data platform of claim 1, wherein, The configuration of the corresponding auxiliary storage node described in step b is based on the storage space and load pressure of the corresponding data storage node; the configuration of the platform's auxiliary storage node is based on the data scale and expansion efficiency requirements of the entire big data platform. 3.The auxiliary storage method of a big data platform of claim 1, wherein, Step c determines whether to trigger an auxiliary storage operation based on at least one of the following four criteria: Does the overall storage resource utilization rate Si of the big data platform exceed the threshold Ts? Does the big data platform have any data storage nodes that have failed or become unreachable? Has the big data platform added any new data storage nodes? Does the big data platform have data storage nodes whose load pressure exceeds the threshold? 4. The auxiliary storage method of a big data platform according to claim 3, characterized in that, The criterion for determining whether the data storage node load pressure exceeds the threshold in step c is as follows: Fi=MAX(ci / Tc,mi / Tm,di / Td,io / Tio), where ci, mi, di, and io represent the CPU, memory, disk space utilization, and disk I / O rate of the data storage node server, respectively, and Tc, Tm, Td, and Tio are the pressure alarm thresholds for the CPU, memory, disk space, and disk I / O of the data storage node, respectively. When the Fi of the data storage node is greater than or equal to 1, it indicates that the load pressure of the data storage node exceeds the threshold, and auxiliary storage operations will be triggered; otherwise, they will not.

5. The auxiliary storage method of a big data platform according to claim 1, characterized in that, Step d includes auxiliary storage operations for when the overall big data platform's storage resource utilization exceeds a threshold. The specific methods are as follows: d01. Enable the platform auxiliary storage node service and write all newly added data blocks to the platform auxiliary storage node; d02. Feedback the mapping information of the newly added data blocks to the information nodes of the big data platform for overwriting; d03. Alarm feedback to remind users to expand the storage capacity of the big data platform; the alarm feedback includes, but is not limited to, email alarms, SMS alarms, and big data platform interface alarms.

6. The auxiliary storage method of a big data platform according to claim 1, characterized in that, Step d also includes auxiliary storage operations for data storage node failures and outages, the specific methods of which are as follows: d11. Obtain information on faulty and disconnected data storage nodes; the storage node information includes, but is not limited to, the mapping information of data blocks in the node; d12. Enable the secondary storage node service for the corresponding storage node; d13. Copy the data blocks from the original failed and disconnected data storage node from the replicas of other data storage nodes on the platform to the corresponding auxiliary storage node; d14. Transfer the data blocks originally intended for writing to the faulty or disconnected data storage node to the corresponding auxiliary storage node; d15. Feed back the data block mapping information in the auxiliary storage node to the information node of the big data platform for overwriting; d16. Determine if the failed or disconnected data storage node has recovered; d17. Compare the data block differences between the failed and disconnected storage node and the corresponding auxiliary storage node, and incrementally write the difference data blocks into the recovered storage node; d18. Feed back the mapping information of the differential data blocks to the information nodes of the big data platform for overlay.

7. The auxiliary storage method of a big data platform according to claim 1, characterized in that, Step d also includes additional auxiliary storage operations for the data storage node, the specific methods of which are as follows: d21. Determine whether the overall storage resource utilization rate Si of the big data platform exceeds the threshold Ts; d22. Write all data blocks from the already enabled platform auxiliary storage nodes into the newly added big data platform data storage node; d23. Feed back the mapping information of the corresponding data block to the information nodes of the big data platform for overwriting; d24. The big data platform performs a balanced adjustment of the data block distribution in existing and newly added data storage nodes; d25. Configure the corresponding auxiliary storage node for the newly added data storage node. 8.The auxiliary storage method of a big data platform of claim 1, wherein, Step d also includes auxiliary storage operations for data storage nodes whose load pressure exceeds a threshold, as detailed below: d31. Obtain information on data storage nodes whose load pressure exceeds the threshold; d32. Enable the secondary storage node service for the corresponding storage node; d33. Transfer the data blocks originally intended for writing to the high-load data storage node to the corresponding auxiliary storage node; d34. Feed back the data block mapping information in the auxiliary storage node to the information node of the big data platform for overwriting; d35. Wait for the pressure on high-load data storage nodes to fall below the threshold; d36. Write the data blocks from the auxiliary storage node to the corresponding data storage node in the big data platform; d37. Feed back the mapping information of the corresponding data block to the information nodes of the big data platform for overwriting.

Images