Intelligent resource scheduling and dynamic data priority management method and system

Abstract

The invention discloses an intelligent resource scheduling and dynamic data priority management method and system, and the method comprises the steps: setting priorities for data files or data blocks based on the business importance of the data files or the data blocks, setting independent queues for tasks with different priorities, and dynamically adjusting the resource distribution according to the priorities; the task type, the access frequency and the flow condition of the current task are monitored and analyzed, the high-load task is automatically identified, and bandwidth allocation is dynamically adjusted; labels are set for different data attributes in the HDFS, the use conditions of data are monitored in real time, the use conditions comprise the access frequency, the modification frequency and the data size, and the labels and the priorities of the data are automatically adjusted. According to the method and the system, through fine-grained priority management, self-adaptive bandwidth allocation and a data intelligent label mechanism, flexible resource scheduling is realized, and the access efficiency of key data is improved, so that the response speed of the system is optimized, and the problems of resource allocation and access challenges under high-concurrency and multi-priority services are solved.

Description

Technical Field

[0001] The present invention relates to the field of resource scheduling, and in particular to an intelligent resource scheduling and dynamic data priority management method and system. Background Art

[0002] As the data in Hadoop big data clusters increases, the system faces multiple challenges such as processing efficiency, resource allocation, and data access optimization. Especially when multiple services with different priorities are running at the same time, the system is prone to uneven resource allocation and low data access efficiency. Specifically, it will face the following problems:

[0003] (1) A sudden increase in traffic causes system overload.

[0004] The HDFS system is prone to overload during traffic peaks, especially due to the uncertainty of large-scale concurrent requests, and the system needs to handle a large number of read and write tasks. As the core node for metadata management, NameNode needs to quickly respond to millions of requests under high traffic conditions. However, when requests surge, both NameNode and DataNode may experience response delays or even be unable to process requests due to resource pressure, resulting in a significant drop in the performance of the entire file system.

[0005] (2) Data is unevenly distributed, and some data has a great impact on the system.

[0006] Although HDFS attempts to balance data distribution through the replication mechanism, the access requirements for some data in key business scenarios are much higher than other data. These "hot data" are usually concentrated on a few key nodes, resulting in an unbalanced I / O load. Once these data are delayed in processing, important business tasks will be interrupted, affecting system stability. Even if these data only account for 5% of the total, their high traffic characteristics create a burden between nodes, making it difficult for traditional replication balancing mechanisms to meet actual needs.

[0007] (3) Business development and resource supply growth are not synchronized.

[0008] As new businesses and functions continue to be launched, the demand for data storage is growing exponentially. Faced with this continuous growth, it is often difficult for the platform to expand resources in a timely manner. Insufficient resource supply not only affects the execution efficiency of existing tasks, but may also lead to nonlinear growth of system load, resulting in delays, failures, and even data loss. The mismatch between resources and business needs has become a bottleneck restricting the stable operation of the system. Summary of the invention

[0009] In order to solve the resource allocation and access challenges under high concurrency and multi-priority services, the present invention provides an intelligent resource scheduling and dynamic data priority management method and system, which realizes flexible resource scheduling and improves the access efficiency of key data through fine-grained priority management, adaptive bandwidth allocation and data intelligent label mechanism, thereby optimizing the system response speed and solving the above problems.

[0010] To achieve the above object, the present invention adopts the following technical solutions:

[0011] In one embodiment of the present invention, a method for intelligent resource scheduling and dynamic data priority management is proposed, the method comprising:

[0012] Set priorities for data files or data blocks based on their business importance, set up independent queues for tasks of different priorities, and dynamically adjust resource allocation based on priority;

[0013] Monitor and analyze the task type, access frequency, and traffic of the current task, automatically identify high-load tasks, and dynamically adjust bandwidth allocation;

[0014] Set labels for different data attributes in HDFS, monitor data usage in real time, including access frequency, modification frequency, and data size, and automatically adjust data labels and their priorities.

[0015] Furthermore, a priority upgrade mechanism is set to allow low-priority tasks to gradually increase their priority if they have not been processed for a long time.

[0016] Furthermore, the following steps are performed to analyze the traffic situation of the current task, automatically identify high-load tasks and dynamically adjust bandwidth allocation:

[0017] Regularly process historical traffic data to extract daily bandwidth requirements and traffic peak characteristics of different tasks;

[0018] Set a traffic threshold for each task based on historical traffic peak characteristics;

[0019] Monitor the traffic of the current task in real time. When the traffic of the task approaches the traffic threshold, predict that the traffic will soon reach the peak.

[0020] Once the traffic of the task continues to rise and exceeds the traffic threshold, an alert is triggered and the task is marked as a high-load task;

[0021] Enables adaptive bandwidth allocation to reallocate bandwidth from low-load tasks to high-load tasks.

[0022] Furthermore, for tasks with large fluctuations in access frequency, dynamic priorities are set; when it is detected that multiple tasks have reached a high load, bandwidth is allocated in sequence according to the task priorities.

[0023] In one embodiment of the present invention, an intelligent resource scheduling and dynamic data priority management system is also proposed, the system comprising:

[0024] Fine-grained priority management module, used to set priorities for data files or data blocks based on their business importance, set up independent queues for tasks of different priorities, and dynamically adjust resource allocation according to priority;

[0025] Adaptive bandwidth allocation module, which is used to monitor and analyze the task type, access frequency and traffic of the current task, automatically identify high-load tasks and dynamically adjust bandwidth allocation;

[0026] The data intelligent label module is used to set labels for different data attributes in HDFS, monitor data usage in real time, including access frequency, modification frequency, and data size, and automatically adjust data labels and their priorities.

[0027] Furthermore, a priority upgrade mechanism is set to allow low-priority tasks to gradually increase their priority if they have not been processed for a long time.

[0028] Furthermore, the following steps are performed to analyze the traffic situation of the current task, automatically identify high-load tasks and dynamically adjust bandwidth allocation:

[0029] Regularly process historical traffic data to extract daily bandwidth requirements and traffic peak characteristics of different tasks;

[0030] Set a traffic threshold for each task based on historical traffic peak characteristics;

[0031] Monitor the traffic of the current task in real time. When the traffic of the task approaches the traffic threshold, predict that the traffic will soon reach the peak.

[0032] Once the traffic of the task continues to rise and exceeds the traffic threshold, an alert is triggered and the task is marked as a high-load task;

[0033] Enables adaptive bandwidth allocation to reallocate bandwidth from low-load tasks to high-load tasks.

[0034] Furthermore, for tasks with large fluctuations in access frequency, dynamic priorities are set; when it is detected that multiple tasks have reached a high load, bandwidth is allocated in sequence according to the task priorities.

[0035] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned intelligent resource scheduling and dynamic data priority management are implemented.

[0036] In one embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program for executing intelligent resource scheduling and dynamic data priority management.

[0037] Beneficial effects:

[0038] 1. The present invention refines priority management to the file and data block level, not just the task level. This refined priority scheduling can ensure that critical data gets priority resource support in a high-load environment, reduce system latency, and achieve more accurate resource management.

[0039] 2. The present invention automatically identifies high-load tasks and flexibly adjusts bandwidth by real-time monitoring of task traffic and access frequency. This adaptive allocation mechanism optimizes system resource utilization and is particularly suitable for scenarios with frequent load fluctuations in data centers.

[0040] 3. The present invention dynamically adjusts resource allocation and priority according to real-time data characteristics and business needs, enhances the system's adaptive ability, and reduces manual intervention. By adjusting data priority and resource allocation in real time, the system can flexibly respond to high concurrency and burst traffic situations, thereby improving the overall service quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of a method for intelligent resource scheduling and dynamic data priority management according to an embodiment of the present invention;

[0042] Figure 2 It is a schematic diagram of the structure of the intelligent resource scheduling and dynamic data priority management system of the present invention;

[0043] Figure 3 It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION

[0044] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0045] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, system, device, method or computer program product. Therefore, the present disclosure may be implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0046] According to the implementation mode of the present invention, a method and system for intelligent resource scheduling and dynamic data priority management are proposed. Through fine-grained priority management, adaptive bandwidth allocation and data intelligent label mechanism, flexible resource scheduling is achieved, and the access efficiency of key data is improved, thereby optimizing the system response speed and solving the difficult problems of resource allocation and access challenges under high concurrency and multi-priority services.

[0047] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0048] The present invention proposes an intelligent resource scheduling and dynamic data priority management method, comprising:

[0049] (1) Fine-grained priority mechanism and multi-level scheduling queue.

[0050] In order to ensure that important tasks are processed first, a fine-grained priority management mechanism can be introduced in HDFS. In addition to setting priorities at the file level, priority management is further refined to the block level to achieve refined control of resource scheduling. Specifically, multi-level priority queues can place high-priority critical data blocks at the front to ensure that they are given priority to processing resources even when resource competition is fierce. This design can dynamically adjust the scheduling strategy, give priority to supporting key businesses when traffic surges or resources are tight, and reduce system latency.

[0051] (2) Workload-based adaptive bandwidth allocation mechanism.

[0052] The adaptive bandwidth allocation mechanism can allocate bandwidth based on real-time workload, allowing the system to flexibly adapt to the needs of different tasks. The mechanism autonomously allocates appropriate bandwidth by analyzing the current task type, request frequency, and network load, ensuring that critical tasks are given priority and sufficient bandwidth support. This not only avoids delays in high-priority tasks due to insufficient resources, but also prevents idle bandwidth during low loads. This mechanism enables the system to manage resources more flexibly and efficiently, thereby alleviating the problem of insufficient or uneven use of bandwidth resources.

[0053] (3)Data intelligent labeling mechanism.

[0054] The data intelligent label mechanism adds labels to the data in HDFS to monitor the data usage in real time, including information such as access frequency, modification frequency, and data size. Based on this information, the system can automatically adjust the data priority, such as labeling data with high access frequency as "hot data" to ensure that they are given priority during storage and access. Through this dynamic priority adjustment, the system can allocate resources more effectively, ensure the access efficiency of key data, reduce system load problems caused by hot data, and effectively improve the intelligence level of data scheduling.

[0055] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0056] In order to explain the above intelligent resource scheduling and dynamic data priority management method more clearly, a specific embodiment is used for illustration below. However, it should be noted that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation to the present invention.

[0057] like Figure 1 As shown, the process of implementing an intelligent resource scheduling and dynamic data priority management method is as follows:

[0058] 1. Fine-grained priority mechanism and multi-level scheduling queue

[0059] Implementation process:

[0060] 1. Priority definition

[0061] Native Hadoop and HDFS do not directly implement fine-grained task priorities, task classification, resource allocation strategies, and dynamic adjustments. To implement these features, additional development or external scheduling systems are usually required.

[0062] Custom development: To implement priority management of files and data blocks, a customized module needs to be developed on HDFS. This module can set priorities for data files or data blocks based on their business importance. For example, financial transaction log files can be marked as high priority, ordinary business data can be marked as medium priority, and cold data can be marked as low priority.

[0063] Priority storage: Priority information can be stored in the metadata of HDFS by extending the metadata or adding a tag field. Add a field to the metadata storage structure of the NameNode to save the priority of each data file or data block.

[0064] 2. Multi-level scheduling queue design

[0065] Scheduler selection: On YARN (Hadoop resource management framework), you can use CapacityScheduler or FairScheduler schedulers, which can allocate different resource queues for different tasks. You can set up independent queues for tasks of different priorities so that resources are allocated according to task priorities during scheduling.

[0066] Queue division: Create multiple queues, such as high priority queue, medium priority queue, and low priority queue. You can set the resource allocation ratio of the high priority queue to be greater than that of the medium and low priority queues. In this way, when a high priority task enters the queue, it can be ensured that it will get priority in computing and storage resources.

[0067] Dynamic scheduling: Dynamically adjust resource allocation through the configuration of scheduling strategies. For example, when there is a surge in tasks in the high-priority queue, the system can temporarily borrow resources from the low-priority queue and release the resources back to the low-priority queue after the high-priority tasks are processed.

[0068] 3. Multi-level priority and scheduling queue workflow

[0069] (1) Task submission: Tasks are submitted to queues of different priorities based on the importance of the data files. For example, tasks involving high-frequency trading are submitted to the high-priority queue, and ordinary business tasks are submitted to the medium-priority queue.

[0070] (2) Priority Scheduling: When NameNode receives multiple task requests, it first schedules high-priority tasks and allocates sufficient resources. Meanwhile, medium- and low-priority tasks wait until high-priority tasks are completed before obtaining resources.

[0071] (3) Task execution and resource monitoring: During task execution, NameNode monitors resource usage in real time. If it is found that low-priority tasks are consuming a large amount of resources, the system can adjust the resource allocation strategy according to the priority to ensure that high-priority tasks are not affected by low-priority tasks.

[0072] (4) Priority upgrade mechanism: To prevent low-priority tasks from waiting for a long time, a priority upgrade mechanism can be set up to allow low-priority tasks to gradually increase their priority if they are not processed for a long time, thus ensuring the fairness of the system.

[0073] Example:

[0074] File A (payment log, priority: high): submitted to the high priority queue, and receives 70% of the computing and storage resources.

[0075] File B (normal business log, priority: medium): submitted to the medium priority queue and allocated 20% of resources.

[0076] File C (historical data, priority: low): submitted to the low priority queue and only gets 10% of the resources.

[0077] 2. Workload-based adaptive bandwidth allocation mechanism

[0078] 1. Bandwidth monitoring and analysis

[0079] (1) Real-time monitoring tools: The system can use network monitoring tools (such as NetFlow or a custom bandwidth monitoring module) to track the task traffic, bandwidth requirements, and actual usage in real time. This module is usually located between the NameNode and the DataNode and continuously collects task traffic statistics, including the task's access frequency, request size, and bandwidth usage.

[0080] (2) Access frequency monitoring: For example, the system counts the access frequency of each data block and records the number of accesses of each task in a specific time period. If the number of accesses reaches a certain threshold (for example, more than 500 accesses per minute), the system will mark the task as a high-load task.

[0081] (3) Traffic peak determination: Traffic peak determination is a key link in adaptive bandwidth allocation. It mainly helps the system identify traffic peaks in advance by comparing real-time traffic with historical traffic, and then prioritizes the bandwidth requirements of high-load tasks.

[0082] Historical traffic records: The system will record the traffic conditions of each task in different time periods in the past. For example, record the average bandwidth requirement of file A during daily operation (such as 10Mbps).

[0083] Data analysis tools: Use data analysis tools (such as Hadoop, Spark, etc.) to regularly process historical traffic data and extract daily bandwidth requirements and traffic peak characteristics of different tasks. For example, the log data of file A usually has a traffic surge during business peak hours (such as 11:00-13:00 and 19:00-21:00 every day).

[0084] Real-time traffic monitoring module: The system monitors the traffic of the current task in real time and uses network traffic analysis tools (such as NetFlow, sFlow or custom modules) to count the bandwidth usage of each task. The real-time traffic monitoring module will regularly report the current traffic to the NameNode or bandwidth management module.

[0085] Task traffic statistics: For each task, the system will count its bandwidth usage in real time. For example, if the system detects that the current traffic of file A reaches 80Mbps, which is close to the historical traffic peak, it can be determined that the task is about to enter a traffic peak.

[0086] Traffic peak prediction:

[0087] Threshold setting: Based on the historical traffic peak characteristics, the system sets a traffic threshold for each task. For example, the bandwidth requirement of file A is usually 10Mbps, but it usually reaches 100Mbps during the transaction peak period, so its traffic threshold can be set to 80Mbps.

[0088] Real-time comparison and prediction: When the real-time traffic monitoring module detects that the traffic of a task is close to or exceeds the threshold (such as the current traffic of file A exceeds 80Mbps), the system will predict that the traffic is about to reach the peak and determine whether the task needs to increase bandwidth allocation.

[0089] Peak warning mechanism: The system can set up a warning mechanism. Once the task traffic continues to rise and exceeds the threshold, the system triggers a warning and marks the task as a peak load task. The system then starts adaptive bandwidth allocation and reallocates bandwidth from low-load tasks to high-load tasks.

[0090] 2. Task type analysis

[0091] Identification of high-load tasks: Define task types in NameNode, such as high-bandwidth demand tasks and low-bandwidth demand tasks. In daily access, the system dynamically evaluates the load of tasks based on access frequency and traffic size. For example, if a payment system log task has a continuous access frequency of 500 times per minute, the system will classify it as a high-bandwidth demand task.

[0092] Access frequency threshold: The task type analysis module in the system sets the access threshold. For example, a task with an access frequency of more than 500 times is classified as "high load". When the system finds that the number of accesses to a data block is higher than the threshold, it can not only adjust the priority of the task, but also increase its bandwidth allocation to ensure smoothness.

[0093] Dynamic task update: The load classification of tasks is dynamic. If a task (such as log data of file A) is a high-load task during peak hours, but the access frequency decreases during off-peak hours, the system will reduce its bandwidth requirement to normal.

[0094] 3. Dynamic bandwidth allocation

[0095] Bandwidth adjustment module: Implements a bandwidth allocation policy module in NameNode to allocate available bandwidth among tasks. The system allocates a higher proportion of bandwidth to tasks with high bandwidth requirements and reclaims resources from low-load tasks.

[0096] Bandwidth allocation rules: Assuming the total system bandwidth is 100Gbps, when a high-load task is identified, the system will dynamically increase the bandwidth allocation for the task (such as increasing it to 50Gbps), while reducing the bandwidth of low-priority tasks to 20Gbps.

[0097] Dynamic adjustment example: If the system detects that the access frequency of file A reaches the threshold, the system increases its bandwidth from 30Gbps to 50Gbps and reduces the bandwidth of file B (normal task) from 20Gbps to 10Gbps to prioritize the needs of high-load tasks.

[0098] Priority processing: For tasks with large fluctuations in traffic, the system can set dynamic priorities. When multiple tasks are detected to be under high load, bandwidth is allocated in order of task priority. For example, payment system tasks will get more bandwidth than log processing tasks.

[0099] Example:

[0100] Assume that the system currently has 100Gbps of bandwidth resources, and the allocation is as follows:

[0101] High bandwidth demand task (File A): The initial bandwidth is 30 Gbps, which increases to 50 Gbps during the access frequency peak period.

[0102] Normal task (File B): Initial bandwidth is 20 Gbps, which is reduced to 10 Gbps during peak hours.

[0103] Low priority task (file C): initial bandwidth 10 Gbps, reduced to 5 Gbps at peak times.

[0104] 3. Data Intelligent Labeling Mechanism

[0105] In the HDFS hierarchical security system, the data intelligent label mechanism is mainly used to identify and classify different types of data, and dynamically adjust the data priority according to its attributes such as access frequency, modification frequency, and data size.

[0106] 1. Label Type

[0107] Set labels for different data attributes. For example:

[0108] Access frequency label: used to identify the access frequency of data. High access frequency is marked as hot data, and low access frequency is marked as cold data.

[0109] Modification frequency label: Mark the frequency of data modification. For example, frequently modified data can be marked as high-change data, and infrequently modified data can be marked as low-change data.

[0110] Data size label: Identify the storage usage of data. For example, large files are marked as large data volume, and small files are marked as small data volume.

[0111] Automatic label generation: The system automatically generates labels for each data block or file. For example, data accessed more than 500 times per minute can be marked as high-frequency access, so that labels can help the system prioritize different data blocks.

[0112] 2. Real-time monitoring and dynamic label adjustment

[0113] Data monitoring: The system monitors the access, modification frequency, and data size of each data block in real time. The data monitoring module can obtain relevant information by scanning logs regularly or directly from HDFS metadata.

[0114] Label adjustment: When the attributes of certain data change (such as an increase in access frequency), the system will automatically adjust the label. For example, if the number of accesses to a data block increases from 10 times per hour to 100 times per hour, the system will adjust it from "cold data" to "hot data" and increase its priority accordingly.

[0115] Threshold setting: Define thresholds for each tag. For example, the threshold for the "High Access Frequency" tag can be set to more than 50 visits per hour. If this threshold is exceeded, the system will automatically mark it as hot data.

[0116] 3. Tag management and priority adjustment

[0117] Priority strategy: The system sets the priority based on the tag attributes of the data. For example, data with high-frequency access and high-change tags has a higher priority, and the system will allocate resources first.

[0118] Dynamic resource allocation: When a data block is marked as high priority, the system will allocate more bandwidth or I / O resources to it. Low-priority data will be temporarily accessed or delayed when resources are tight.

[0119] Tag caching and fast access: The system can cache high-priority data blocks in memory to improve access speed and reduce I / O consumption. For example, data with frequently accessed tags can stay in the cache longer.

[0120] Example:

[0121] File A: The file has a high access frequency and high change rate, and has a high priority. The system allocates bandwidth to it first.

[0122] File B: Labeled as low access frequency, low change, low priority, and delayed processing when resources are insufficient.

[0123] File C: Labeled as large data volume, low volatility, high storage optimization requirements, and low priority.

[0124] Initial label setting: File A is initially accessed 100 times per hour, and the system sets a high access frequency label for it.

[0125] Real-time monitoring and adjustment: When the number of accesses to file A drops to 5 times per hour, the system automatically adjusts it to a low access frequency and lowers its priority.

[0126] Tag priority takes effect: When the access volume of file A increases again to exceed the threshold, the tag automatically returns to high access frequency, and the bandwidth and I / O priority are increased again.

[0127] Based on the same inventive concept, the present invention also proposes an intelligent resource scheduling and dynamic data priority management system. The implementation of the system can refer to the implementation of the above method, and the repeated parts will not be repeated. The term "module" used below can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0128] Figure 2 This is a schematic diagram of the structure of the intelligent resource scheduling and dynamic data priority management system of the present invention. Figure 2 As shown, the system includes:

[0129] A fine-grained priority management module 101 is used to set priorities for data files or data blocks based on their business importance, set independent queues for tasks of different priorities, and dynamically adjust resource allocation according to the priorities;

[0130] Set up a priority upgrade mechanism to allow low-priority tasks to gradually increase their priority if they have not been processed for a long time.

[0131] The adaptive bandwidth allocation module 102 is used to monitor and analyze the task type, access frequency and traffic of the current task, automatically identify high-load tasks and dynamically adjust bandwidth allocation;

[0132] The following steps are used to analyze the traffic of the current task, automatically identify high-load tasks, and dynamically adjust bandwidth allocation:

[0133] Regularly process historical traffic data to extract daily bandwidth requirements and traffic peak characteristics of different tasks;

[0134] Set a traffic threshold for each task based on historical traffic peak characteristics;

[0135] Monitor the traffic of the current task in real time. When the traffic of the task approaches the traffic threshold, predict that the traffic will soon reach the peak.

[0136] Once the traffic of the task continues to rise and exceeds the traffic threshold, an alert is triggered and the task is marked as a high-load task;

[0137] Enables adaptive bandwidth allocation to reallocate bandwidth from low-load tasks to high-load tasks.

[0138] The data intelligent label module 103 is used to set labels for different data attributes in HDFS, monitor the usage of data in real time, including access frequency, modification frequency and data size, and automatically adjust the data label and its priority;

[0139] For tasks with large fluctuations in access frequency, dynamic priorities are set; when multiple tasks are detected to be under high load, bandwidth is allocated in sequence according to task priority.

[0140] It should be noted that although several modules of the intelligent resource scheduling and dynamic data priority management system are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for concretization.

[0141] Based on the above invention concept, Figure 3 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, the aforementioned intelligent resource scheduling and dynamic data priority management are implemented.

[0142] Based on the aforementioned inventive concept, the present invention further proposes a computer-readable storage medium, which stores a computer program for executing the aforementioned intelligent resource scheduling and dynamic data priority management.

[0143] The intelligent resource scheduling and dynamic data priority management method and system proposed in the present invention have the following highlights:

[0144] 1. Refine priority management to the file and data block level, not just the task level. This refined priority scheduling can ensure that critical data is given priority resource support in a high-load environment, reduce system latency, and achieve more accurate resource management.

[0145] 2. Compared with the traditional static bandwidth allocation method, the bandwidth allocation is dynamically adjusted according to the task traffic monitored in real time. By monitoring the task traffic and access frequency in real time, the system can automatically identify high-load tasks and flexibly adjust the bandwidth. This adaptive allocation mechanism optimizes the resource utilization of the system and is particularly suitable for dealing with scenarios where the load in the data center fluctuates frequently.

[0146] 3. This solution dynamically adjusts resource allocation and priority according to real-time data characteristics and business needs, enhancing the system's adaptive capabilities and reducing manual intervention. By adjusting data priority and resource allocation in real time, the system can flexibly respond to high concurrency and burst traffic, thereby improving the overall service quality.

[0147] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements contained in the spirit and scope of the attached claims.

[0148] Regarding the limitation of the protection scope of the present invention, those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. An intelligent resource scheduling and dynamic data priority management method, characterized in that: The method includes: Set priorities for data files or data blocks based on their business importance, set up independent queues for tasks of different priorities, and dynamically adjust resource allocation based on priority; Monitor and analyze the task type, access frequency, and traffic of the current task, automatically identify high-load tasks, and dynamically adjust bandwidth allocation; Set labels for different data attributes in HDFS, monitor data usage in real time, including access frequency, modification frequency, and data size, and automatically adjust data labels and their priorities.

2. The intelligent resource scheduling and dynamic data priority management method according to claim 1 is characterized in that: Set up a priority upgrade mechanism to allow low-priority tasks to gradually increase their priority if they have not been processed for a long time.

3. The intelligent resource scheduling and dynamic data priority management method according to claim 1 is characterized in that: The following steps are used to analyze the traffic of the current task, automatically identify high-load tasks, and dynamically adjust bandwidth allocation: Regularly process historical traffic data to extract daily bandwidth requirements and traffic peak characteristics of different tasks; Set a traffic threshold for each task based on historical traffic peak characteristics; Monitor the traffic of the current task in real time. When the traffic of the task approaches the traffic threshold, predict that the traffic will soon reach the peak. Once the traffic of the task continues to rise and exceeds the traffic threshold, an alert is triggered and the task is marked as a high-load task; Enables adaptive bandwidth allocation to reallocate bandwidth from low-load tasks to high-load tasks.

4. The intelligent resource scheduling and dynamic data priority management method according to claim 1, characterized in that: For tasks with large fluctuations in access frequency, dynamic priorities are set; when multiple tasks are detected to be under high load, bandwidth is allocated in sequence according to task priority.

5. An intelligent resource scheduling and dynamic data priority management system, characterized in that: The system includes: Fine-grained priority management module, used to set priorities for data files or data blocks based on their business importance, set up independent queues for tasks of different priorities, and dynamically adjust resource allocation according to priority; Adaptive bandwidth allocation module, which is used to monitor and analyze the task type, access frequency and traffic of the current task, automatically identify high-load tasks and dynamically adjust bandwidth allocation; The data intelligent label module is used to set labels for different data attributes in HDFS, monitor data usage in real time, including access frequency, modification frequency, and data size, and automatically adjust data labels and their priorities.

6. The intelligent resource scheduling and dynamic data priority management system according to claim 5, characterized in that: Set up a priority upgrade mechanism to allow low-priority tasks to gradually increase their priority if they have not been processed for a long time.

7. The intelligent resource scheduling and dynamic data priority management system according to claim 5, characterized in that: The following steps are used to analyze the traffic of the current task, automatically identify high-load tasks, and dynamically adjust bandwidth allocation: Regularly process historical traffic data to extract daily bandwidth requirements and traffic peak characteristics of different tasks; Set a traffic threshold for each task based on historical traffic peak characteristics; Monitor the traffic of the current task in real time. When the traffic of the task approaches the traffic threshold, predict that the traffic will soon reach the peak. Once the traffic of the task continues to rise and exceeds the traffic threshold, an alert is triggered and the task is marked as a high-load task; Enables adaptive bandwidth allocation to reallocate bandwidth from low-load tasks to high-load tasks.

8. The intelligent resource scheduling and dynamic data priority management system according to claim 5, characterized in that: For tasks with large fluctuations in access frequency, dynamic priorities are set; when multiple tasks are detected to be under high load, bandwidth is allocated in sequence according to task priority.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 4.

Images