A dynamic resource scheduling method to improve Hadoop data throughput in CTyunOS

Abstract

The purpose of the present invention is to provide a dynamic resource scheduling method for improving Hadoop data throughput in CTyunOS, the method comprising: docking Hadoop with CTyunOS; predicting the load; using Hadoop task scheduling logic and CTyunOS underlying resource scheduling mechanism to perform dynamic resource scheduling according to the predicted load results. The present invention develops a dynamic resource scheduling mechanism by utilizing the kernel-level resource scheduling capability of CTyunOS to achieve dynamic resource scheduling, which can respond to changes in resource demand in real time, improve resource utilization, enhance the interaction between Hadoop task scheduling logic and CTyunOS underlying resource scheduling mechanism, and make full use of the optimization features provided by the system. Reduce high latency and increase data throughput, especially in a high-concurrency environment, by optimizing the load distribution between nodes and improving data transmission efficiency to improve the overall performance of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of dynamic resource allocation, and in particular to a dynamic resource scheduling method for improving Hadoop data throughput in CTyunOS. Background Art

[0002] CTyunOS is a cloud operating system designed to provide efficient resource management and scheduling to support large-scale data processing and cloud computing applications. Hadoop is an open source framework widely used in large-scale data processing scenarios. Its core components include a distributed file system (HDFS) and a resource scheduling framework (YARN). Hadoop implements task allocation and data storage on computing nodes through these components. However, Hadoop's performance improvement mainly depends on the efficient management of underlying resources and the reasonable scheduling of tasks. In traditional deployment environments, Hadoop is usually based on a general-purpose operating system and does not fully utilize the optimization capabilities of cloud computing platforms and specific operating systems, which leads to the following problems: Static resource allocation strategies cannot dynamically adapt to real-time load changes, resulting in low resource utilization. Data throughput is easily affected by uneven node loads, resulting in overall performance fluctuations. There is a lack of optimization mechanisms supported by the underlying operating system, such as memory optimization, CPU isolation, and I / O priority management.

[0003] Some existing resource scheduling optimization methods have alleviated these problems to a certain extent. For example, dynamic load balancing algorithms improve the problem of uneven load by reallocating tasks, and predictive scheduling based on historical data can allocate resources in advance to cope with expected loads. However, these methods are difficult to fully improve the performance of Hadoop in complex cloud environments, especially in scenarios involving multi-tenants and dynamic resource demands. In addition, these solutions usually do not fully utilize the optimization capabilities at the operating system level. Traditional Hadoop cannot dynamically respond to fluctuations in resource demand, resulting in node overload during peak periods and waste of resources during off-peak periods. Low system coupling: Hadoop task scheduling logic lacks interaction with the underlying operating system resource scheduling, and cannot fully utilize the system optimization features. High latency and low throughput: Uneven load between nodes and data transmission bottlenecks lead to performance degradation, especially in high-concurrency environments. Summary of the invention

[0004] The purpose of the present invention is to provide a dynamic resource scheduling method for improving Hadoop data throughput in CTyunOS. The method develops a dynamic resource scheduling mechanism by utilizing the kernel-level resource scheduling capability of CTyunOS, realizes the dynamic resource scheduling, can respond to changes in resource demand in real time, improve resource utilization, enhance the interaction between Hadoop task scheduling logic and CTyunOS underlying resource scheduling mechanism, and make full use of the optimization characteristics provided by the system. Reduce high latency and increase data throughput, especially in a high-concurrency environment, by optimizing the load distribution between nodes and improving data transmission efficiency to improve the overall system performance.

[0005] A dynamic resource scheduling method for improving Hadoop data throughput in CTyunOS, comprising: Connect Hadoop with CTyunOS; Forecasting of load; Hadoop task scheduling logic and CTyunOS underlying resource scheduling mechanism are used to perform dynamic resource scheduling based on predicted load results.

[0006] Preferably, before docking Hadoop with CTyunOS, the process also includes deploying the CTyunOS resource environment, specifically: Create a containerized instance in CTyunOS and run each Hadoop node in a separate container; Based on the virtualization management module of CTyunOS, the resources of the physical server are pooled into dynamically allocated computing resource pools, storage resource pools, and network bandwidth pools; Independent bandwidth is allocated to each independent container through a network namespace technology.

[0007] Preferably, the docking of Hadoop with CTyunOS includes: By extending Hadoop's YARN scheduler and calling the CTyunOS monitoring API, real-time information is obtained, including: CPU load, which limits the proportion of CPU time used by containers based on control groups; Memory usage: Use the built-in memory.stat file in the container to monitor memory usage trends; IO throughput, monitoring the disk read and write speed through the block layer of the CTyunOS kernel; Data processing,Monitoring data is collected through Prometheus and stored in a time series database.

[0008] Preferably, predicting the load includes: The exponential smoothing algorithm of the sliding window is used to predict the historical load of the node, which is expressed as: ; in, is the smoothing coefficient, is the current load value, is the predicted value.

[0009] Preferably, the dynamic resource scheduling based on the predicted load results using Hadoop task scheduling logic and CTyunOS underlying resource scheduling mechanism includes: Calculate the weight of tasks to be processed; Allocating nodes according to the task weights; If the occupancy rate of the allocated node exceeds the threshold, the CTyunOS underlying resource scheduling mechanism is used to migrate the pending tasks to the new node.

[0010] Preferably, the calculating the weight of the task to be processed includes: The weight factor is set based on the task type, data volume and priority, and the task priority is calculated, which is expressed as: Priority(Task) = f(Size, Deadline, Compute\_Complexity); Assign high-priority tasks to low-load nodes.

[0011] Preferably, if the occupancy rate of the allocated node exceeds a threshold, the CTyunOS underlying resource scheduling mechanism is used to migrate the pending tasks to a new node, including: When the node resource usage exceeds 85%, dynamic expansion is triggered; Use CTyunOS's hot migration technology to move part of the task to the new node, and automatically select the replica closest to the target node to read the data during migration; When the node resource utilization rate is lower than 30%, the container and corresponding resources are actively released.

[0012] Preferably, allocating high-priority tasks to low-load nodes includes: Based on the bandwidth allocation strategy of CTyunOS, higher network bandwidth is allocated to high-priority tasks first; Use the token bucket algorithm to limit bandwidth consumption of low-priority tasks, so that the network of high-priority tasks is unobstructed; The Snappy compression algorithm is enabled during data transmission to merge small files into large files through Hadoop Archive.

[0013] Preferably, the deployment of the CTyunOS resource environment also includes: Introduce a distributed cache system at the CTyunOS layer to cache hot data; Pre-read frequently accessed HDFS blocks into the cache.

[0014] Preferably, after the dynamic resource scheduling is performed according to the predicted load results by using the Hadoop task scheduling logic and the CTyunOS underlying resource scheduling mechanism, the method further includes: The heartbeat detection mechanism of CTyunOS is used to monitor the node status, detect container failure, and isolate the faulty node; The tasks on the failed node are marked as "failed" in the Hadoop scheduler, triggering the retry mechanism; Prioritize the nearest copy to start retrying; Use the task snapshot function of CTyunOS to record the intermediate status of the restart task; For HDFS replica damage caused by node failure, CTyunOS starts an automatic replica repair process to redistribute replica storage among healthy nodes.

[0015] The beneficial effects of the present invention are: 1. The present invention utilizes the kernel characteristics of CTyunOS, such as CPU binding, dynamic memory adjustment and network traffic control, to achieve efficient resource allocation of Hadoop tasks, thereby significantly improving system performance. 2. The present invention uses the monitoring API of CTyunOS to obtain and analyze the resource usage data of Hadoop nodes in real time, dynamically adjust the resource allocation strategy, and ensure that the system can flexibly respond to load changes and optimize resource utilization and data throughput. 3. The present invention innovatively applies dynamic bandwidth allocation and data compression strategies to optimize transmission efficiency based on the data characteristics of HDFS, ensure high data transmission rates in high-load environments, and reduce network bottlenecks. 4. The present invention combines the heartbeat detection and task snapshot functions of CTyunOS to achieve rapid isolation of faulty nodes and automatic restart of tasks, enhance the reliability and stability of the system, and reduce performance losses caused by failures. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor. Figure 1 A flow chart of a dynamic resource scheduling method for improving Hadoop data throughput in CTyunOS of the present invention; Figure 2 A schematic diagram of the structure of a dynamic resource scheduling system for improving Hadoop data throughput in CTyunOS according to the present invention; Figure 3 This is a schematic diagram of the CTyunOS resource environment deployment process of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0019] In addition, the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0020] Some existing resource scheduling optimization methods have alleviated these problems to a certain extent. For example, dynamic load balancing algorithms improve the problem of uneven load by reallocating tasks, and predictive scheduling based on historical data can allocate resources in advance to cope with expected loads. However, these methods are difficult to fully improve the performance of Hadoop in complex cloud environments, especially in scenarios involving multi-tenants and dynamic resource demands. In addition, these solutions usually do not fully utilize the optimization capabilities at the operating system level. Traditional Hadoop cannot dynamically respond to fluctuations in resource demand, resulting in node overload during peak periods and waste of resources during off-peak periods. Low system coupling: Hadoop task scheduling logic lacks interaction with the underlying operating system resource scheduling, and cannot fully utilize the system optimization features. High latency and low throughput: Uneven load between nodes and data transmission bottlenecks lead to performance degradation, especially in high-concurrency environments.

[0021] The present invention utilizes the kernel characteristics of CTyunOS, such as CPU binding, dynamic memory adjustment, and network traffic control, to achieve efficient resource allocation for Hadoop tasks, thereby significantly improving system performance. The present invention uses the monitoring API of CTyunOS to obtain and analyze the resource usage data of Hadoop nodes in real time, dynamically adjust the resource allocation strategy, and ensure that the system can flexibly respond to load changes and optimize resource utilization and data throughput. The present invention innovatively applies dynamic bandwidth allocation and data compression strategies to optimize transmission efficiency based on the data characteristics of HDFS, ensure high data transmission rates in high-load environments, and reduce network bottlenecks. The present invention combines the heartbeat detection and task snapshot functions of CTyunOS to achieve rapid isolation of faulty nodes and automatic restart of tasks, enhance the reliability and stability of the system, and reduce performance losses caused by faults.

[0022] Example 1 A dynamic resource scheduling method to improve Hadoop data throughput in CTyunOS, reference Figure 1 and Figure 2 ,include: S100, connects Hadoop to CTyunOS; CtyunOS is a cloud operating system designed to provide efficient resource management and scheduling to support large-scale data processing and cloud computing applications. Features of CTyunOS: Virtualization and container management capabilities: CTyunOS supports efficient virtualization and containerized operating environments, and can dynamically adjust resource allocation, including CPU, memory, storage, and network bandwidth. Real-time resource monitoring: Built-in resource monitoring module supports real-time collection and analysis of resource utilization, system load, and task status. Flexible task scheduling interface: Provides an API interface to support deep linkage with Hadoop's resource scheduling layer.

[0023] Hadoop is an open source software framework for distributed storage and processing of large-scale data sets. Hadoop's core components include HDFS (Hadoop Distributed File System) and the MapReduce computing model. Hadoop's features: Distributed computing framework: Hadoop performs large-scale data computing based on the MapReduce model and relies on YARN (YetAnother Resource Negotiator) to manage cluster resources. High IO density: Tasks consume a lot of disk and network bandwidth, which can easily lead to performance bottlenecks. Simple scheduling: The default resource scheduling strategy is static allocation, and there is a lack of support for dynamic adjustment of tasks. The integration of the two: Dynamic expansion: Hadoop can achieve dynamic expansion and recovery of resources by connecting to the CTyunOS API, breaking through the limitations of static allocation. Unified monitoring: CTyunOS's monitoring data can provide Hadoop with more accurate node load information and guide scheduling optimization. Containerization support: The container migration and isolation technology provided by CTyunOS helps to improve the flexibility and reliability of Hadoop task scheduling.

[0024] S200, predicting the load; Predicting the load can help the system evenly distribute computing tasks to the nodes in the cluster to avoid overloading certain nodes, thereby improving the efficiency of the overall system.

[0025] S300,adopts Hadoop task scheduling logic and CTyunOS underlying resource scheduling,mechanism to perform dynamic resource scheduling based on the predicted,load results.

[0026] The present invention ensures efficient execution of tasks by managing and allocating computing resources, such as CPU, memory, network bandwidth, etc. In an embodiment of the present invention, the CTyunOS core service layer supports the entire optimization mechanism, including resource management, real-time monitoring and network optimization capabilities. The resource monitoring module deeply integrates the real-time monitoring capabilities of CTyunOS with the resource scheduling of Hadoop to provide a basis for dynamic scheduling. The task scheduling module optimizes the resource utilization efficiency of Hadoop tasks through dynamic allocation and migration mechanisms. The data transmission optimization module focuses on solving the HDFS transmission efficiency bottleneck and improving throughput. The fault-tolerant module uses the high reliability mechanism provided by CTyunOS to ensure the stability and rapid recovery of tasks.

[0027] Preferably, S100, before connecting Hadoop to CTyunOS, also includes deploying the CTyunOS resource environment, specifically: S001, create a containerized instance in CTyunOS and run each Hadoop node in an independent container; Create containerized instances in CTyunOS, and each Hadoop node runs in an independent container to isolate resource usage for different tasks.

[0028] S002, based on the virtualization management module of CTyunOS, pools the resources of the physical server into dynamically allocated computing resource pools, storage resource pools, and network bandwidth pools; S003, allocate independent bandwidth to each independent container through network namespace technology.

[0029] Each container is allocated independent bandwidth through the network namespace technology to avoid network resource conflicts between tasks.

[0030] Preferably, docking Hadoop with CTyunOS includes: By extending Hadoop's YARN scheduler and calling the CTyunOS monitoring API, real-time information is obtained, including: The ‌YARN scheduler‌ is the core component of the YARN (Yet Another Resource Negotiator) system, responsible for the allocation and management of cluster resources. YARN is the resource manager of the Hadoop cluster, mainly used to coordinate the resource requirements of different jobs, optimize resource utilization, and improve cluster utilization and job execution efficiency. YARN's resource management mechanism is organized in the form of resource pools, and each resource pool corresponds to a queue. Users can specify a queue when submitting a job. If no queue is specified, the default queue is used by default. The queues in YARN form a tree structure. All applications run in leaf queues, and the resources of child queues use the resources of the parent queue.

[0031] In actual deployment, it is crucial to configure the scheduler properly to optimize cluster performance and meet the needs of different jobs. By intelligently allocating resources, the YARN scheduler can maximize the use of cluster resources, avoid the impact of hot spots on applications, coordinate the operation of a large number of applications in the cluster, and solve problems such as resource competition.

[0032] CPU load, which limits the proportion of CPU time used by containers based on control groups; Memory usage: Use the built-in memory.stat file in the container to monitor memory usage trends; IO throughput, monitoring the disk read and write speed through the block layer of the CTyunOS kernel; Data processing,Monitoring data is collected through Prometheus and stored in a time series database,such as InfluxDB.

[0033] Preferably, S200, predicting the load includes: The exponential smoothing algorithm of the sliding window is used to predict the historical load of the node, which is expressed as: ; in, is the smoothing coefficient, is the current load value, is the predicted value.

[0034] The forecast results will be used as a basis for decision-making on dynamic adjustment of resources.

[0035] Preferably, S300, using Hadoop task scheduling logic and CTyunOS underlying resource scheduling mechanism to perform dynamic resource scheduling according to the predicted load results includes: S310, calculating the weight of the task to be processed; S320, allocating nodes according to task weights; S330: If the occupancy rate of the allocated node exceeds the threshold, the CTyunOS underlying resource scheduling mechanism is used to migrate the pending tasks to the new node.

[0036] Preferably, S310, calculating the weight of the task to be processed includes: The weight factor is set based on the task type, data volume and priority, and the task priority is calculated, which is expressed as: Priority(Task) = f(Size, Deadline, Compute\_Complexity); Assign high-priority tasks to low-load nodes.

[0037] High-priority tasks are assigned to low-load nodes to reduce waiting time and resource conflicts. Based on real-time monitoring data, the node with the lowest resource utilization is selected for task scheduling.

[0038] Preferably, in S330, if the occupancy rate of the allocated node exceeds a threshold, the CTyunOS underlying resource scheduling mechanism is used to migrate the pending tasks to a new node, including: S331, when the node resource occupancy rate exceeds 85%, dynamic expansion is triggered; S332, uses CTyunOS's hot migration technology to move part of the task to the new node, and automatically selects the replica closest to the target node to read the data during migration; The hot migration technology of Docker containers (CRIU - Checkpoint / Restore in Userspace) is used to achieve seamless migration of tasks from high-load nodes to low-load nodes. Data consistency is guaranteed by the HDFS copy mechanism: during migration, the copy closest to the target node is automatically selected to read data. Based on the built-in network flow control module of CTyunOS, the transmission path is dynamically adjusted to select the link with the lowest latency.

[0039] S333: When the node resource utilization rate is lower than 30%, the container and corresponding resources are actively released.

[0040] When the node resource utilization rate is lower than 30%, the system actively releases containers and corresponding resources to reduce waste.

[0041] Preferably, allocating high-priority tasks to low-load nodes includes: Based on the bandwidth allocation strategy of CTyunOS, higher network bandwidth is allocated to high-priority tasks first; Use the token bucket algorithm to limit bandwidth consumption of low-priority tasks, so that the network of high-priority tasks is unobstructed; ‌‌The token bucket algorithm is a commonly used current limiting algorithm used to limit the number of requests per unit time. ‌The algorithm maintains a token bucket with a fixed capacity and puts a certain number of tokens into the bucket every second. When a request arrives, if there are enough tokens in the token bucket, the request is allowed to pass and consumes a token from the bucket; otherwise, the request is rejected. ‌

[0042] The Snappy compression algorithm is enabled during data transmission to merge small files into large files through Hadoop Archive.

[0043] The Snappy compression algorithm is used to optimize performance when processing large amounts of data. The goal of Snappy is to provide a reasonable compression rate while ensuring compression speed, rather than pursuing the maximum compression rate or compatibility with other compression algorithms. Snappy compresses data through the following steps: Matching process: Snappy looks for consecutive identical byte sequences, and when at least 4 consecutive identical bytes are found, these bytes are replaced with 3 integer tags. This replacement mechanism reduces the size of the data, but at the same time increases the number of tags. In order to optimize storage space, Snappy compresses these integers through bit operations to ensure that the tags do not take up too much space. Encoding and decoding: Snappy's encoding and decoding processes are very efficient and can provide reasonable compression effects while ensuring speed. When encoding, Snappy converts the input data into a uint8 array and replaces the tags when a consecutive identical byte sequence is found; when decoding, Snappy reconstructs the original data based on these tags. In the embodiment of the present invention, the Snappy compression algorithm is enabled during data transmission, and the compression ratio reaches 50%-70%, which greatly reduces the transmission traffic.

[0044] Preferably, deploying the CTyunOS resource environment also includes: Introduce a distributed cache system at the CTyunOS layer to cache hot data; A distributed cache system is a cache system that stores data on multiple servers, aiming to improve application performance by reducing the number of database accesses. A distributed cache system is usually deployed in an independent application process and on a different machine from the application process, so data read and write operations need to be completed over the network.

[0045] The distributed cache system reduces the database load by caching frequently accessed data and reducing the number of database queries. Cached data is stored in memory, which can significantly increase the data access speed and improve the user experience. In scenarios with a wide range of user groups and high concurrency, the distributed cache system can effectively disperse server pressure and improve the overall performance of the system.

[0046] Pre-read frequently accessed HDFS blocks into the cache.

[0047] Pre-read frequently accessed HDFS blocks into the cache to avoid disk read delays.

[0048] Preferably, after S300, using Hadoop task scheduling logic and CTyunOS underlying resource scheduling mechanism to perform dynamic resource scheduling according to the predicted load results, it also includes: S410 uses the heartbeat detection mechanism of CTyunOS to monitor the node status, detect container failure, and isolate the faulty node; Heartbeat detection is a method of sending signals regularly to confirm whether each node in the system is in normal working condition. The heartbeat signal is similar to the human heartbeat. Each beat represents a "breath" of the system to ensure that the system is running continuously. This process is usually carried out by one party sending a heartbeat signal and the other party receiving and confirming it. The basic principle of heartbeat detection is to ensure that the communication link between nodes is unobstructed by sending signals at a fixed frequency. If a node does not receive a heartbeat signal for a period of time, it can be judged that the node may have failed, and corresponding treatment measures can be taken. When reporting heartbeats, some additional status information and metadata are usually carried, such as the load and operation status of the node, so that the management system can better understand the health status of the entire distributed system. In a distributed system, nodes usually report their status to other nodes at a fixed frequency. The advantage of this method is that it is simple and easy to implement, and can effectively monitor the health status of nodes. Generally, a node sends a heartbeat signal at each time period (for example, every second). When a node sends a heartbeat signal, if it does not receive a response within a predetermined time, it will be considered that there may be a problem with the target node. This mechanism is called timeout detection. Specifically, if a node does not receive any heartbeat response within a specified time (for example, 3 seconds), the timeout processing mechanism will be triggered. The core of the timeout processing mechanism is to set a reasonable timeout time, which needs to be adjusted according to network latency and node processing capabilities. If the timeout time is too short, a normal node may be mistakenly judged as a failed node; if the timeout time is too long, it will affect the timeliness of fault detection.

[0049] S420, the task on the failed node is marked as "failed" in the Hadoop scheduler, triggering a retry mechanism; S430, prioritize the most recent copy and start retrying; S440, use the task snapshot function of CTyunOS to record the intermediate status of the restart task; The task snapshot function of CTyunOS is a data protection and rapid recovery tool. It completely copies the disk of the cloud host at a specific point in time to generate a data copy that is exactly the same as the state at that time, which is the so-called "snapshot". When the cloud host fails or data is lost, the data can be quickly restored by rolling back to the state of the snapshot. The snapshot function can quickly restore data in the event of data loss or corruption, reduce data loss and downtime, and provide higher reliability and availability. By creating snapshots regularly, an effective disaster recovery plan can be established to help deal with emergencies such as hardware failures and natural disasters. Using snapshots can provide an effective deployment and testing environment to ensure the consistency of applications in different environments and conduct safe functional testing and performance evaluation.

[0050] S450: For HDFS replica damage caused by node failure, CTyunOS starts the automatic replica repair process to redistribute replica storage among healthy nodes.

[0051] CTyunOS starts the automatic replica repair process, which mainly includes the following steps: Start the repair process: During the CTyunOS startup process, the system will automatically detect the replica status. If the replica is found to be damaged or missing, the system will trigger the automatic repair process. Replica detection: The system will check the status of all replicas to determine which ones are damaged or missing. This step is the basis of the repair process and ensures that the system can accurately identify the replicas that need to be repaired. Data recovery: Once the replica that needs to be repaired is determined, the system will recover the data from other healthy replicas. This usually involves copying data from other replicas to the damaged or missing replica to ensure the integrity and consistency of the data.

[0052] Replica synchronization: After the data recovery is complete, the system will ensure that the data is synchronized between all replicas. This includes updating the data of all replicas to ensure that they are in a consistent state. Replica verification: After the repair is complete, the system will verify all replicas to ensure that they are working properly and the data is consistent. This step is critical to ensure that the repair is successful. Restart the system: After all replicas are repaired and verified, the system will restart to ensure that the new replica state is correctly loaded and applied.

[0053] The advantages of the present invention over the prior art include: Significantly improved data throughput: Through dynamic resource scheduling and underlying optimization, the resource competition problem is effectively solved and the performance of the Hadoop cluster is improved. Higher resource utilization: Real-time monitoring and intelligent scheduling make resource allocation more refined and reduce waste. Enhanced system stability: When resources are tight or nodes fail, dynamic adjustment and task migration mechanisms are used to ensure that task execution is not interrupted.

[0054] The present invention utilizes the kernel characteristics of CTyunOS, such as CPU binding, dynamic memory adjustment, and network traffic control, to achieve efficient resource allocation for Hadoop tasks, thereby significantly improving system performance. The present invention uses the monitoring API of CTyunOS to obtain and analyze the resource usage data of Hadoop nodes in real time, dynamically adjust the resource allocation strategy, and ensure that the system can flexibly respond to load changes and optimize resource utilization and data throughput. The present invention innovatively applies dynamic bandwidth allocation and data compression strategies to optimize transmission efficiency based on the data characteristics of HDFS, ensure high data transmission rates in high-load environments, and reduce network bottlenecks. The present invention combines the heartbeat detection and task snapshot functions of CTyunOS to achieve rapid isolation of faulty nodes and automatic restart of tasks, enhance the reliability and stability of the system, and reduce performance losses caused by faults.

[0055] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A dynamic resource scheduling method for improving Hadoop data throughput in CTyunOS, characterized in that: include: Connect Hadoop with CTyunOS; Forecasting of load; Hadoop task scheduling logic and CTyunOS underlying resource scheduling mechanism are used to perform dynamic resource scheduling based on predicted load results.

2. A dynamic resource scheduling method for improving Hadoop data throughput in CTyunOS according to claim 1, characterized in that: Before connecting Hadoop with CTyunOS, the CTyunOS resource environment is deployed, specifically: Create a containerized instance in CTyunOS and run each Hadoop node in a separate container; Based on the virtualization management module of CTyunOS, the resources of the physical server are pooled into dynamically allocated computing resource pools, storage resource pools, and network bandwidth pools; Independent bandwidth is allocated to each independent container through a network namespace technology.

3. According to claim 1, a dynamic resource scheduling method for improving Hadoop data throughput in CTyunOS is characterized in that: The connection between Hadoop and CTyunOS includes: By extending Hadoop's YARN scheduler and calling the CTyunOS monitoring API, real-time information is obtained, including: CPU load, which limits the proportion of CPU time used by containers based on control groups; Memory usage: Use the built-in memory.stat file in the container to monitor memory usage trends; IO throughput, monitoring the disk read and write speed through the block layer of the CTyunOS kernel; Data processing,Monitoring data is collected through Prometheus and stored in a time series database.

4. According to claim 1, a dynamic resource scheduling method for improving Hadoop data throughput in CTyunOS is characterized in that: The load prediction includes: The exponential smoothing algorithm of the sliding window is used to predict the historical load of the node, which is expressed as: ； in, is the smoothing coefficient, is the current load value, is the predicted value.

5. A dynamic resource scheduling method for improving Hadoop data throughput in CTyunOS according to claim 1, characterized in that: The method of using Hadoop task scheduling logic and CTyunOS underlying resource scheduling mechanism to perform dynamic resource scheduling according to the predicted load results includes: Calculate the weight of tasks to be processed; Allocating nodes according to the task weights; If the occupancy rate of the allocated node exceeds the threshold, the CTyunOS underlying resource scheduling mechanism is used to migrate the pending tasks to the new node.

6. A dynamic resource scheduling method for improving Hadoop data throughput in CTyunOS according to claim 5, characterized in that: The calculation of the weight of the task to be processed includes: The weight factor is set based on the task type, data volume and priority, and the task priority is calculated, which is expressed as: Priority(Task) = f(Size, Deadline, Compute\_Complexity); Assign high-priority tasks to low-load nodes.

7. A dynamic resource scheduling method for improving Hadoop data throughput in CTyunOS according to claim 5, characterized in that: If the occupancy rate of the allocated node exceeds the threshold, the CTyunOS underlying resource scheduling mechanism is used to migrate the pending tasks to the new node, including: When the node resource usage exceeds 85%, dynamic expansion is triggered; Use CTyunOS's hot migration technology to move part of the task to the new node, and automatically select the replica closest to the target node to read the data during migration; When the node resource utilization rate is lower than 30%, the container and corresponding resources are actively released.

8. A dynamic resource scheduling method for improving Hadoop data throughput in CTyunOS according to claim 6, characterized in that: The allocating high-priority tasks to low-load nodes comprises: Based on the bandwidth allocation strategy of CTyunOS, higher network bandwidth is allocated to high-priority tasks first; Use the token bucket algorithm to limit the bandwidth consumption of low-priority tasks, so that the network of high-priority tasks can be unblocked; The Snappy compression algorithm is enabled during data transmission to merge small files into large files through Hadoop Archive.

9. A dynamic resource scheduling method for improving Hadoop data throughput in CTyunOS according to claim 2, characterized in that: The deployment of the CTyunOS resource environment also includes: Introduce a distributed cache system at the CTyunOS layer to cache hot data; Pre-read frequently accessed HDFS blocks into the cache.

10. A dynamic resource scheduling method for improving Hadoop data throughput in CTyunOS according to claim 1, characterized in that: After the Hadoop task scheduling logic and the CTyunOS underlying resource scheduling mechanism are used to perform dynamic resource scheduling according to the predicted load results, the method further includes: The heartbeat detection mechanism of CTyunOS is used to monitor the node status, detect container failure, and isolate the faulty node; The tasks on the failed node are marked as "failed" in the Hadoop scheduler, triggering the retry mechanism; Prioritize the nearest copy to start retrying; Use the task snapshot function of CTyunOS to record the intermediate status of the restart task; For HDFS replica damage caused by node failure, CTyunOS starts an automatic replica repair process to redistribute replica storage among healthy nodes.

Images