System and method for hybrid processing of mass offline data and mass real-time data

Abstract

The invention relates to the technical field of electric data processing, and discloses a system and a method for hybrid processing of mass offline data and mass real-time data, and the system comprises a queue management module which receives task requests, identifies and marks task types, distributes tasks to a stream processing queue or a batch processing queue, and sorts the tasks according to priorities; the task types comprise flow tasks and batch tasks; the thread pool management module establishes a thread pool, sequentially processes tasks according to priorities and collects real-time parameters; the calculation module is responsible for calculating task priorities and calculating task jitter factors based on real-time parameters; the scheduling decision module compares the task jitter factor with a reference value, determines a weight adjustment coefficient, corrects the priority and formulates a scheduling strategy; the alarm module judges whether to trigger an alarm. According to the invention, by introducing the dynamic priority scheduling and preemptive processing mechanism driven by the jitter factor, the real-time response capability and the resource utilization efficiency of the system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electrical data processing, and more particularly, to a system and method for hybrid processing of massive offline data and massive real-time data. Background Art

[0002] With the rapid development of big data technology, there are a large number of offline batch processing tasks and real-time stream processing tasks in modern information systems. Offline tasks usually involve complex calculations of massive data, emphasizing the integrity and accuracy of data processing; while stream tasks focus on the fast response and low-latency processing of real-time data. These two types of tasks often compete for system resources. How to efficiently coordinate and hybrid process them in the same system has become an important challenge in the current data processing field.

[0003] Usually, fixed priorities or static resource partitioning methods are adopted to schedule batch tasks and stream tasks respectively, resulting in low system resource utilization efficiency, and in the peak period of tasks, there are often phenomena such as too long response delay of stream tasks or blocked execution of batch tasks. In addition, there is a lack of a dynamic monitoring and adjustment mechanism for task status and real-time parameters of system operation, making it difficult to adapt to complex and changeable business loads and unable to effectively solve the problems of jitter and priority imbalance in task scheduling.

[0004] Therefore, how to implement an intelligent scheduling system that can dynamically perceive the real-time state of the system, flexibly adjust task priorities and scheduling strategies, and take into account the efficient hybrid processing of massive offline data and real-time data has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] In view of this, the present invention proposes a system and method for hybrid processing of massive offline data and massive real-time data to solve the problems of low scheduling efficiency, large task response delay, and insufficient dynamic adjustment of priorities caused by resource competition between stream tasks and batch tasks in the prior art.

[0006] On the one hand, a system for hybrid processing of massive offline data and massive real-time data proposed by the present invention includes: A queue management module, configured to receive task requests, identify task types and label them, and according to the labeling results, allocate tasks to a stream processing queue and a batch processing queue, and then sort the tasks according to priorities; the task types include stream tasks and batch tasks; A thread pool management module, configured to establish a stream task thread pool and a batch task thread pool, process tasks in sequence according to priorities, and collect real-time parameters; A calculation module, respectively connected to the queue management module and the thread pool management module, the calculation module is configured to calculate priorities, and calculate a task jitter factor according to real-time parameters; The scheduling decision module is configured to determine a weight adjustment coefficient according to the magnitude relationship between the task jitter factor and the jitter reference value, correct the priority according to the weight adjustment coefficient, and determine a scheduling policy according to the corrected priority; The alarm module is configured to compare the real-time parameters with the corresponding threshold ranges to determine whether to issue an alarm.

[0007] Further, the priority is calculated and obtained through the following relationship: ; where is the priority of the task, is the task waiting time, is the constant value corresponding to the task type, is the remaining value of the agreed time in the SLA agreement, is the agreed time in the SLA agreement, is the number of task failure retries, is a constant, takes [1, 3], , , and represent the initial weight coefficients, ; After the calculation module obtains the priority, the queue management module sorts the tasks in the stream processing queue and the batch processing queue respectively according to the priority.

[0008] Further, the real-time parameters include: the current delay and historical delay of the task, the resource usage status of the thread pool, and the priority change trend of the task. The task jitter factor is calculated and obtained through the following relationship: ; ; where is the task jitter factor, is the current delay in the T-th batch of real-time data, is the average value of the current delays in a total of T batches of real-time data, is the current thread pool resource utilization exception term, is the historical task jitter factor of the i-th batch, i ≥ 3, is the current thread pool occupancy rate, is the exception threshold of the current thread pool usage rate, = 0.85, is the conversion coefficient, indicating the sensitivity of the exception degree, , , and They are the first to the fourth jitter adjustment coefficients in sequence, + + + = 1.

[0009] Furthermore, according to the magnitude relationship between the task jitter factor and the jitter reference value, determining the weight adjustment coefficient includes: Calculating the difference between the task jitter factor and the jitter reference value and the ratio of the task jitter factor to the jitter reference value, and the weight adjustment coefficient satisfies the following relationship: ; wherein, is the weight adjustment coefficient, is the jitter reference value, = 10 -6 .

[0010] Furthermore, correcting the priority according to the weight adjustment coefficient includes: multiplying the initial weight coefficient by the weight adjustment coefficient respectively and recalculating the priority to obtain the corrected priority, denoted as the corrected priority, and re - sorting the tasks according to the corrected priority.

[0011] Furthermore, determining the scheduling strategy according to the corrected priority includes: Obtaining the task type. If it is a streaming task, select the currently idle or least - loaded thread in sequence according to the re - sorted queue for immediate execution; when all threads are busy, add the current task to the preemptive scheduling candidate queue; If it is a batch task, divide the processor time into several time slices, and allocate a time slice to each task in sequence according to the re - sorted queue. When the time slice is exhausted, if the task is not completed, put the current task back to the end of the queue, and if the task is completed, remove it from the queue and schedule the next task to continue execution.

[0012] Furthermore, when there is a situation where a streaming task is in the preemptive scheduling candidate queue, obtain the remaining computing resources of the current device. When the remaining computing resources are greater than or equal to the computing resources required by the tasks in the preemptive scheduling candidate queue, suspend processing the batch tasks and start processing the tasks in the preemptive scheduling candidate queue; When the remaining computing resources are less than the computing resources required by the tasks in the preemptive scheduling candidate queue, suspend processing the batch tasks and streaming tasks in the stream processing queue and the batch processing queue; When there is a suspended task, start calculating the processing duration and obtain the historical maximum suspension time. When the processing duration is greater than or equal to the historical maximum suspension time, cancel the suspension and mark the currently processed streaming task.

[0013] Furthermore, it also includes: After obtaining the correction priority value, compare the current task jitter factor with the jitter threshold. If the current task jitter factor is greater than or equal to the jitter threshold, mark the current task and send it to the alarm module.

[0014] Further, when the alarm module determines that the real-time parameter is greater than or equal to its corresponding threshold range, calculate the range excess amount, and issue an alarm according to the range excess amount in different levels; The alarm module is also used to issue an alarm for the marked task.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: Through the queue management module, real-time annotation and classification of task types are carried out, effectively distinguishing stream tasks and batch tasks, and refined allocation according to priorities, improving the pertinence of task processing and the utilization efficiency of system resources.

[0016] Comprehensively considering latency, thread pool utilization rate, and historical fluctuation trends, calculate the task jitter factor, and accordingly correct the task priority, realizing the intelligent self-adaptation of the scheduling strategy, and enhancing the stability of scheduling and the system's load response ability.

[0017] In scenarios where resources are scarce or high-priority tasks suddenly occur, batch tasks can be dynamically suspended and computing resources can be yielded to high-priority stream tasks, thereby ensuring the timely processing of critical real-time tasks and enhancing the elasticity and real-time response ability of the system.

[0018] The system not only schedules based on the initial task information, but also real-time collects parameters such as the thread pool status, task latency, and priority changes, and dynamically adjusts the scheduling strategy to adapt to the complex and changeable task load environment.

[0019] When real-time parameters are abnormal, the system can automatically mark high-risk tasks and issue alarms in different levels, effectively supporting operation and maintenance monitoring, and preventing task backlog, system congestion, or service collapse.

[0020] The present invention integrates the stream processing and batch processing models, combines the thread pool and time slice scheduling mechanisms, enhances the system's support ability for processing large-scale heterogeneous data, and is particularly suitable for the efficient management and intelligent scheduling of massive data.

[0021] On the other hand, the present invention also proposes a method for hybridly processing massive offline data and massive real-time data, including: S1: Receive a task request, identify the task type and perform annotation. According to the annotation result, allocate the task to the stream processing queue and the batch processing queue, and then sort the tasks according to the priority; the task types include stream tasks and batch tasks; S2: Establish a stream task thread pool and a batch task thread pool, process tasks in sequence according to the priority, and collect real-time parameters; S3: Calculate the priority and calculate the task jitter factor according to the real-time parameters; S4: Determine the weight adjustment coefficient according to the magnitude relationship between the task jitter factor and the jitter reference value, correct the priority according to the weight adjustment coefficient, and determine the scheduling strategy according to the corrected priority; S5: Compare the real-time parameters with the corresponding threshold range to determine whether to issue an alarm.

[0022] It should be noted that a method and system for hybrid processing of massive offline data and massive real-time data provided by the present invention have the same beneficial effects, which will not be elaborated here. Description of the Drawings

[0023] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings: Figure 1 It is a functional block diagram of a system for hybrid processing of massive offline data and massive real-time data provided by an embodiment of the present invention.

[0024] Figure 2 It is a flowchart of a method for hybrid processing of massive offline data and massive real-time data provided by an embodiment of the present invention. Detailed Embodiments

[0025] The exemplary embodiments disclosed in the present application will be described in more detail below with reference to the drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art. It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0026] Refer to Figure 1 As shown, an embodiment of the present invention provides a system for hybrid processing of massive offline data and massive real-time data, including: A queue management module, configured to receive task requests, identify task types and perform annotations, and according to the annotation results, allocate tasks to a stream processing queue and a batch processing queue, and then sort the tasks according to the priority; the task types include stream tasks and batch tasks; A thread pool management module, configured to establish a stream task thread pool and a batch task thread pool, process tasks in sequence according to priorities, and collect real-time parameters; A calculation module, respectively connected to the queue management module and the thread pool management module, the calculation module is configured to calculate priorities and calculate a task jitter factor according to real-time parameters; A scheduling decision module, configured to determine a weight adjustment coefficient according to the magnitude relationship between the task jitter factor and a jitter reference value, correct the priority according to the weight adjustment coefficient, and determine a scheduling strategy according to the corrected priority; An alarm module, configured to compare real-time parameters with corresponding threshold ranges to determine whether to issue an alarm.

[0027] It should be noted that in this embodiment, by setting the queue management module to accurately identify and classify and schedule stream tasks and batch tasks, parallel processing of massive heterogeneous data is achieved, avoiding resource conflicts; through the calculation module, priorities and task jitter factors are dynamically calculated, introducing a real-time feedback mechanism to effectively perceive system state fluctuations; the scheduling decision module adjusts task priorities based on the jitter factor and the reference value, and formulates an adaptive scheduling strategy to ensure the timely processing of critical tasks during peak system loads or resource shortages; at the same time, the equipped alarm module can timely warn of potential anomalies based on the change trend of real-time parameters, improving the overall robustness and security of the system; integrating the above mechanisms, the present invention significantly improves the task processing efficiency, resource utilization rate and stability of the stream-batch integrated system, and is particularly suitable for data processing scenarios with high requirements for low latency, high throughput and strong real-time performance.

[0028] In some embodiments of the present application, the priority is calculated and obtained through the following relationship: ; Wherein, is the priority of the task, is the task waiting time, is a constant value corresponding to the task type, is the remaining value of the time stipulated in the SLA agreement, is the time stipulated in the SLA agreement, is the number of task failure retries, is a constant, takes [1, 3], , , and represent initial weight coefficients, ; After the calculation module obtains the priority, the queue management module sorts the tasks in the stream processing queue and the batch processing queue according to the priority respectively.

[0029] It should be noted that in the above formula, if each symbol in the formula has a unit, it is uniformly converted to the International System of Units, and then only the units are calculated during the calculation, and the units are deleted. In addition, for the symbol B, preferably, when the task type is a streaming task, the corresponding constant value is 1.5; when the task type is a batch task, the corresponding constant value is 1.0. The specific value of the symbol B can be automatically tuned using a genetic algorithm or reinforcement learning.

[0030] It can be understood that by introducing multi-dimensional key factors such as task waiting time, task type weight, remaining SLA time, and number of failed retries, and using an adjustable initial weight coefficient to weight and integrate them, a refined dynamic evaluation of task priority is achieved; among them, the differential setting of the task type constant value (such as a higher weight for streaming tasks) can accurately reflect the time sensitivity of different tasks, which is beneficial to improving the intelligence and time response ability of the overall system scheduling; in addition, by automatically optimizing the setting of parameter B with a genetic algorithm or reinforcement learning, the limitation of manual experience setting is avoided, enabling the priority model to continuously adaptively adjust according to the actual operating state, thereby realizing the reasonable scheduling and allocation of system resources, reducing task latency, and increasing the task completion rate, which is particularly suitable for mixed data processing scenarios with high task complexity, strong resource constraints, and prominent real-time requirements.

[0031] In some embodiments of the present application, the real-time parameters include: the current delay and historical delay of the task, the resource usage status of the thread pool, and the change trend of the task priority. The task jitter factor is calculated through the following relationship: ; ; where is the task jitter factor, is the current delay in the T-th batch of real-time data, is the average value of the current delays in a total of T batches of real-time data, is the current thread pool resource utilization anomaly term, is the historical task jitter factor of the i-th batch, i≥3, is the current thread pool occupancy rate, is the anomaly threshold of the current thread pool usage rate, =0.85, is the conversion coefficient, indicating the sensitivity of the anomaly degree, , , and are the first to fourth jitter adjustment coefficients in sequence, + + + =1.

[0032] It should be noted that the jitter factor of a task is dynamically calculated through real-time parameters to measure the stability and scheduling priority fluctuations during the task execution process. The real-time parameters include: the difference between the current task delay and the historical delay, the resource usage status of the thread pool, and the change trend of the task priority, etc.

[0033] The jitter factor of a task consists of the following three parts: The first part is: the difference between the current task delay and the historical average delay, which is used to measure whether there is an obvious abnormality in the current task execution. The second part is: the abnormal item of thread pool resource utilization, which indicates whether the current thread pool resource occupancy has exceeded the set threshold. When the resource utilization rate exceeds the set threshold, its exceeding degree is expressed by a ratio, and it is zero when it does not exceed the set threshold. Specifically, if the current resource occupancy rate is higher than the set threshold, it is calculated according to the weighted product of the abnormal degree sensitivity parameter and the exceeding ratio; if it does not exceed the set threshold, this item is zero. The third part is: the average value of the continuous change of the task scheduling priority, which reflects the drastic fluctuation degree of the task priority in multiple time slices. Through the weighted synthesis of the above three parts of information, the comprehensive jitter factor of the task can be obtained. Among them, the weights of each part can be flexibly adjusted according to the task type, system load or policy requirements, and the sum of all weight coefficients is one.

[0034] In specific implementation, the calculated task jitter factor is not only used to measure whether the current task state is stable, but also can be used as a key basis for adjusting the scheduling strategy. The scheduling decision module dynamically adjusts the scheduling priorities of various tasks by judging the relationship between the current task jitter factor and the reference threshold, so as to optimize the overall resource scheduling and task response performance of the system.

[0035] By introducing multi-dimensional real-time parameters (such as delay, resource utilization rate, priority change range, etc.) to construct a comprehensive jitter factor, it can comprehensively reflect the task execution state and scheduling stability, effectively avoiding misjudgment problems caused by single-index judgment; using the relationship between the resource utilization rate and the threshold to express the abnormal degree, it has the ability to adjust sensitivity, improving the system's recognition and response ability to sudden loads; adding the evaluation of the task priority change trend helps to detect potential risk tasks with unstable scheduling in advance, so as to carry out active intervention; the weight coefficient is adjustable and has strong adaptability, and can be adaptively optimized according to different operation stages or business requirements; combining the task jitter factor to correct the scheduling priority enables the entire system to maintain an efficient and stable operation state when facing a mixed load of a large number of streaming and batch tasks, greatly improving the task completion rate and system robustness.

[0036] In some embodiments of the present application, determining the weight adjustment coefficient according to the size relationship between the task jitter factor and the jitter reference value includes: Calculate the difference between the task jitter factor and the jitter reference value, as well as the ratio of the task jitter factor to the jitter reference value. The weight adjustment coefficient satisfies the following relationship: ; where, is the weight adjustment coefficient, is the jitter reference value, = 10 -6 .

[0037] It should be noted that in some embodiments of the present application, to further improve the dynamic adaptability and accuracy of task scheduling, a method for calculating the weight adjustment coefficient based on the relationship between the task jitter factor and the reference value is designed. This method quantifies the deviation degree between the current state and the ideal state of the task, and then dynamically adjusts the weight parameters in the task priority sorting to achieve a more sensitive and stable scheduling strategy.

[0038] The specific calculation process is as follows: According to the difference between the task jitter factor and the jitter reference value, as well as the relative ratio between the two, a weight adjustment factor is constructed. This factor satisfies the following mathematical relationship: Subtract the set reference jitter factor value from the current jitter factor value of the task as the offset; then divide the current jitter factor value by the sum of the reference value and a small constant as the normalization ratio; multiply the above two and then normalize with the sum of one plus the difference between the current jitter factor value and the reference value to finally obtain the weight adjustment coefficient corresponding to the task. Among them: The weight adjustment coefficient represents the factor used to amplify or converge the influence of the current task sorting in the task scheduling priority sorting; the jitter reference value is the benchmark value of the task jitter factor in the ideal state obtained from historical statistics; the small constant is one multiplied by ten to the power of negative six, which is used to prevent division by zero errors and stabilize the calculation process.

[0039] By calculating the deviation degree between the current task and the reference state in real time and dynamically adjusting its contribution to the sorting priority, the scheduling accuracy is effectively improved; when the task state deviates severely, by increasing or decreasing its sorting influence weight, the system scheduling becomes more robust and the jitter propagation is reduced; this adjustment mechanism has a unified calculation logic for different types of tasks and adapts to multiple types of loads such as stream processing and batch processing; both the reference value and the small constant can be set through policies and optimized by learning models, enabling the scheduling strategy to have stronger scenario adaptation capabilities; through the jitter feedback adjustment mechanism, the scheduling logic is made to tend to be reasonable in real time, reducing resource waste and delay backlog, and ensuring the optimal system performance.

[0040] In some embodiments of the present application, the priority is corrected according to the weight adjustment coefficient, including: multiplying the initial weight coefficient by the weight adjustment coefficient respectively and recalculating the priority to obtain the corrected priority, denoted as the corrected priority, and re-sorting the tasks according to the corrected priority.

[0041] It should be noted that by introducing a weight adjustment coefficient to correct the initial task priority, the priority weight can be adjusted in real time according to the current operating state of the system, avoiding the scheduling rigidity caused by static weights, and effectively improving the adaptability of the scheduling strategy to dynamic change scenarios. Different tasks may face different resource bottlenecks and service quality requirements during operation. By dynamically adjusting the weights and recalculating the priorities, the response efficiency of high-priority tasks can be precisely controlled to ensure that critical tasks are executed first, improving the overall service quality. When the system resources are tight, by adjusting the weights, the queuing impact of low-priority tasks is weakened, resource contention and queuing blockage are alleviated, and the congestion problem of the scheduling queue is relieved, improving the overall operation efficiency and stability of the system. If some tasks affect the fairness and stability of system scheduling due to abnormal behaviors (such as excessive jitter or frequent retries), then by attenuating the weights of their priorities, the interference of these tasks to the scheduling order can be effectively suppressed, improving the robustness and fault tolerance of the system. By using the corrected priority as the final sorting basis, the sorting result can more truly reflect the current system load situation and task urgency, so as to achieve more accurate and reasonable scheduling decisions and avoid task "starvation" or "over-preemption".

[0042] In some embodiments of the present application, determining a scheduling strategy according to the corrected priority includes: Obtain the task type. If it is a streaming task, select the currently idle or least-loaded thread from the re-sorted queue in turn for immediate execution; when all threads are busy, add the task to the preemptive scheduling candidate queue; If it is a batch task, divide the processor time into several time slices, and allocate a time slice to each task in turn according to the re-sorted queue. When the time slice is exhausted, if the task is not completed, put the current task back to the end of the queue; if the task is completed, remove it from the queue and schedule the next task to continue execution.

[0043] It should be noted that this solution classifies tasks into "streaming tasks" and "batch tasks", and adopts the immediate execution and time slice rotation strategies respectively, making the scheduling logic more in line with the characteristics of the tasks themselves, effectively improving the compatibility of the system with different task models, and enhancing the generality and adaptability of the scheduling module.

[0044] By preferentially selecting the idle or least-loaded thread to immediately execute the streaming task, the waiting time of the streaming task is compressed to the greatest extent; and when all thread resources are occupied, the streaming task is added to the preemptive scheduling candidate queue and given priority when resources are available, effectively ensuring the timeliness and stability of real-time sensitive tasks.

[0045] For batch tasks, the round-robin scheduling strategy is adopted. On the premise of ensuring system fairness, it is ensured that each task can obtain processing time, avoiding individual tasks from occupying resources for a long time or being starved due to low priority. At the same time, when a task is completed within a time slice, resources can be released in a timely manner, improving resource utilization efficiency.

[0046] By scheduling in combination with the current thread load information, it helps to dynamically balance the load pressure of each processing thread, reduce thread queuing blockage and resource contention, and improve concurrent scheduling ability and task throughput efficiency.

[0047] In the overall strategy, the scheduling method is driven by priority and segmented according to the differences in task types, making the scheduling control more refined and the strategy more flexible. Parameters and execution logic can be further optimized according to specific system goals (such as minimizing response time and maximizing resource utilization).

[0048] Through a reasonable scheduling mechanism, enabling stream tasks to respond in a timely manner and batch tasks to be processed fairly, helps to maintain the overall service quality (QoS) of the system, reduce the service default rate (such as exceeding the SLA agreement time), and improve the end-user experience.

[0049] In some embodiments of the present application, when there is a situation where a stream task is in the preemptive scheduling candidate queue, obtain the remaining computing resources of the current device. When the remaining computing resources are greater than or equal to the computing resources required by the tasks in the preemptive scheduling candidate queue, suspend the processing of batch tasks and start processing the tasks in the preemptive scheduling candidate queue; When the remaining computing resources are less than the computing resources required by the tasks in the preemptive scheduling candidate queue, suspend the processing of batch tasks and stream tasks in the stream processing queue and batch processing queue; When there are suspended tasks, start calculating the processing duration and obtain the historical maximum suspension time. When the processing duration is greater than or equal to the historical maximum suspension time, cancel the suspension and mark the currently processed stream task.

[0050] It should be noted that by setting up a preemption mechanism, actively monitor the resource situation when a stream task enters the preemptive candidate queue, give priority to ensuring tasks with high real-time requirements, so that they can obtain processing opportunities first when resources are available, significantly improving the system's instant response ability to critical tasks and meeting the service quality requirements in real-time processing scenarios.

[0051] Dynamically evaluate the current remaining computing resources of the device, compare with the resources required by tasks, accurately adjust the scheduling strategy, effectively avoid batch tasks from occupying resources for a long time or causing stream task delays due to resource waste, realize the on-demand allocation and fine-grained scheduling of resources, and improve the overall computing resource utilization rate.

[0052] Automatically pause the processing of batch tasks and streaming tasks when resources are insufficient, start monitoring the processing duration of the paused tasks and compare it with the historical maximum pause time to ensure that tasks will not be indefinitely delayed, and recover the tasks in a timely manner through the timeout recovery strategy to ensure the stability and reliability of the system operation.

[0053] Identify and manage long-term paused streaming tasks through a marking mechanism, providing a decision-making basis for subsequent scheduling optimization (such as priority improvement, resource reallocation), which helps the system automatically adapt to complex scenarios such as resource tension and task congestion, and improves the system's adaptive scheduling ability under high load.

[0054] This method takes into account both "fairness" and "timeliness" in the preemption mechanism, that is, when resource conditions permit, let the streaming tasks be executed first, and when resources are scarce, pause all tasks and dynamically monitor them, forming a fair waiting and wake-up mechanism, so that different types of tasks can still obtain reasonable processing opportunities under high load, effectively coordinating the scheduling goal conflicts between fairness and real-time.

[0055] With the dynamic acquisition and comparison of the "historical maximum pause time", provide a data-driven learning basis for the task scheduling strategy, and can further adaptively adjust the scheduling threshold in combination with machine learning algorithms, promoting the system to evolve towards intelligent and autonomous decision-making scheduling.

[0056] In some embodiments of the present application, it further includes: When the corrected priority value is obtained, compare the current task jitter factor with the jitter threshold. If the current task jitter factor is greater than or equal to the jitter threshold, mark the current task and send it to the alarm module.

[0057] The alarm module judges that when the real-time parameter is greater than or equal to its corresponding threshold range, calculate the range excess, and issue an alarm according to the range excess in different levels; The alarm module is also used to issue an alarm for the marked tasks.

[0058] It should be noted that the dynamic monitoring and alarm management of the task status are realized through the following steps: First, when the system obtains the corrected priority value, it will compare the jitter factor of the current task with the pre-set jitter threshold. The jitter factor reflects the degree of fluctuation of the task during operation. If the jitter factor is greater than or equal to the threshold, it means that the task status is abnormal or fluctuates greatly. At this time, the system will immediately mark the task as abnormal and send its information to the alarm module.

[0059] After receiving the marked task information, the alarm module further determines whether the real-time parameters related to the task exceed the corresponding threshold range. If the real-time parameters are greater than or equal to the threshold, the alarm module will calculate the specific magnitude of the threshold (i.e., the range excess). According to the size of the excess, the system will perform graded processing and issue different levels of alarms in stages. The classification of alarm levels helps operation and maintenance personnel to understand the severity of the problem and make corresponding response measures.

[0060] In addition, the alarm module not only issues alarms for real-time parameter anomalies, but also issues alarm reminders for previously marked tasks, ensuring that all tasks that may pose risks receive timely attention.

[0061] By comparing the task jitter factor with the threshold, the system can keenly capture the subtle but important abnormal fluctuations in the task status. As a quantitative indicator to measure task fluctuations, the jitter factor makes anomaly detection no longer rely on simple static judgments, but combines dynamic change trends, which is more scientific and reasonable. This effectively avoids the possibility of ignoring potential risks and improves the accuracy of anomaly detection.

[0062] The alarm module not only determines whether the task is abnormal, but also issues graded alarms based on the extent to which the real-time parameters exceed the threshold. This grading strategy helps to distinguish the urgency of the problem. For example, a slight excess can trigger a low-level alarm to alert the monitoring personnel to pay attention; while a serious excess can trigger a high-level alarm, indicating that immediate intervention is required. This grading mechanism improves the pertinence and practicality of the alarm, and avoids the "alarm fatigue" problem caused by all abnormalities being reported.

[0063] By marking abnormal tasks and sending information to the alarm module, the system can realize active early warning of task abnormalities. The alarm module can continuously track the status of these tasks and issue alarms in time to ensure that abnormal information is not missed. This mechanism makes operation and maintenance management more proactive and forward-looking, reducing the risk of passively waiting for failures to occur.

[0064] Timely detection of abnormal tasks and multi-level alarm mechanisms help to discover potential system risks in advance, promote rapid response and processing, avoid the expansion of abnormalities or the occurrence of more serious failures, and ensure the overall stable operation and security of the system.

[0065] Automatically compare task jitter factors and thresholds, automatically calculate range-exceeding amounts and generate graded alarms, reducing reliance on manual monitoring. The alarm module uniformly manages abnormal tasks and real-time parameter alarms, improving the automated management capabilities of operation and maintenance, allowing managers to focus more on the analysis and decision-making of important issues, and improving overall work efficiency.

[0066] By quantifying the jitter factor and the range exceedance of real-time parameters and combining hierarchical alarms, the system can manage the running status of tasks in a more detailed and comprehensive manner. This not only helps to detect anomalies in a timely manner but also accumulates anomaly data, providing data support for subsequent optimization of task scheduling and system performance.

[0067] Specifically, regarding the range exceedance, multiple preset values that increase numerically in sequence are set. When the range exceedance is between the first preset value and the second preset value, the alarm level is set as a low-level alarm. When the range exceedance is between the second preset value and the third preset value, the alarm level is set as a medium-level alarm, and so on.

[0068] Refer to Figure 2 As shown, an embodiment of the present invention provides a method for hybrid processing of massive offline data and massive real-time data, including: S1: Receive a task request, identify the task type and perform annotation. According to the annotation result, allocate the task to the stream processing queue and the batch processing queue, and then sort the tasks according to the priority; the task types include stream tasks and batch tasks; S2: Establish a stream task thread pool and a batch task thread pool, process the tasks in sequence according to the priority, and collect real-time parameters; S3: Calculate the priority and calculate the task jitter factor according to the real-time parameters; S4: Determine the weight adjustment coefficient according to the magnitude relationship between the task jitter factor and the jitter reference value, correct the priority according to the weight adjustment coefficient, and determine the scheduling strategy according to the corrected priority; S5: Compare the real-time parameters with the corresponding threshold range to determine whether to issue an alarm.

[0069] It can be understood that by accurately identifying and classifying the task types, allocating stream tasks and batch tasks to independent processing queues and thread pools respectively, resource competition and mutual interference between the two types of tasks are avoided, thereby improving the concurrent processing ability and execution efficiency of the overall system.

[0070] Based on task annotation and priority calculation, the system can reasonably sort tasks, ensure that high-priority tasks obtain priority scheduling resources, improve the response speed and processing efficiency of critical tasks, and meet the timeliness requirements of different services.

[0071] By collecting relevant parameters of task execution in real time and calculating the jitter factor, the system can dynamically evaluate the stability of task execution, correct the priority in combination with the weight adjustment coefficient, realize intelligent optimization of the task scheduling strategy, and improve the scheduling flexibility and system adaptability.

[0072] When there are significant fluctuations in tasks or changes in the system environment, the dynamic correction of priorities ensures that the scheduling strategy can be adjusted in a timely manner, avoiding resource waste or delays in critical tasks caused by abnormal fluctuations, and enhancing the robustness and stability of the system.

[0073] The real-time parameter comparison with thresholds and the hierarchical alarm mechanism can detect task anomalies in a timely manner, issue differentiated alarms according to the severity of the signals, assist operation and maintenance personnel in quickly locating problems, reducing the system fault downtime, and improving the system's security guarantee ability.

[0074] The entire method automatically completes task classification, priority calculation, scheduling strategy adjustment, and alarm issuance, greatly reducing the dependence on manual monitoring, enhancing the system's automation and intelligent management level, and reducing operation and maintenance costs.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications or equivalent replacements can still be made to the specific embodiments of the present invention. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

Claims

1. A system for hybrid processing of massive offline data and massive real-time data, characterized in that, Including: A queue management module, configured to receive task requests, identify task types and perform annotation, and according to the annotation results, allocate tasks to a stream processing queue and a batch processing queue, and then sort the tasks according to the priority; the task types include stream tasks and batch tasks; A thread pool management module, configured to establish a stream task thread pool and a batch task thread pool, process tasks sequentially according to the priority, and collect real-time parameters; A calculation module, connected to the queue management module and the thread pool management module respectively, the calculation module is configured to calculate the priority and calculate the task jitter factor according to the real-time parameters; A scheduling decision module, configured to determine a weight adjustment coefficient according to the magnitude relationship between the task jitter factor and the jitter reference value, correct the priority according to the weight adjustment coefficient, and determine a scheduling strategy according to the corrected priority; An alarm module, configured to compare the real-time parameters with the corresponding threshold range to determine whether to issue an alarm.

2. The system for hybrid processing of massive offline data and massive real-time data according to claim 1, wherein The priority is calculated and obtained through the following relationship: ； wherein, is the priority of the task, is the waiting time of the task, is the constant value corresponding to the task type, is the remaining value of the agreed time in the SLA agreement, is the agreed time in the SLA agreement, is the number of task failure retries, is a constant, takes values in [1, 3], and and and represent the initial weight coefficient, ; After the calculation module obtains the priority, the queue management module sorts the tasks in the stream processing queue and the batch processing queue respectively according to the priority.

3. The system for hybrid processing of massive offline data and massive real-time data according to claim 2, wherein The real-time parameters include: the current delay and historical delay of the task, the resource usage status of the thread pool and the priority change trend of the task, and the task jitter factor is calculated and obtained through the following relationship: ； ； Among them, is the task jitter factor, is the current latency in the T-th batch of real-time data, is the average value of the current latency in a total of T batches of real-time data, is the current thread pool resource utilization exception item, is the historical task jitter factor of the i-th batch, i ≥ 3, is the current thread pool occupancy rate, is the abnormal threshold of the current thread pool usage rate, = 0.85, is the conversion coefficient, indicating the sensitivity of the abnormal degree, , , and are the first to fourth jitter adjustment coefficients in sequence, + + + = 1.

4. The system for hybrid processing of massive offline data and massive real-time data according to claim 3, wherein Determining the weight adjustment coefficient according to the magnitude relationship between the task jitter factor and the jitter reference value includes: Calculating the difference between the task jitter factor and the jitter reference value and the ratio of the task jitter factor to the jitter reference value, and the weight adjustment coefficient satisfies the following relationship: ； Among them, is the weight adjustment coefficient, is the dithering reference value, = 10 -6 .

5. The system for hybrid processing of massive offline data and massive real-time data according to claim 4, wherein Correcting the priority according to the weight adjustment coefficient includes: multiplying the initial weight coefficient by the weight adjustment coefficient respectively and recalculating the priority to obtain the corrected priority, denoted as the corrected priority, and re-sorting the tasks according to the corrected priority.

6. The system for hybrid processing of massive offline data and massive real-time data according to claim 5, wherein Determining the scheduling strategy according to the corrected priority includes: Obtaining the task type. If it is a stream task, select the currently idle or least-loaded thread from the re-sorted queue to execute immediately in sequence; when all threads are busy, add the task to the preemptive scheduling candidate queue; If it is a batch task, divide the processor time into several time slices, allocate a time slice to each task in sequence according to the re-sorted queue. When the time slice is exhausted, if the task is not completed, put the current task back to the end of the queue. If the task is completed, remove it from the queue and schedule the next task to continue execution.

7. The system for hybrid processing of massive offline data and massive real-time data according to claim 6, wherein, When there is a situation where a stream task is in the preemptive scheduling candidate queue, obtain the remaining computing resources of the current device. When the remaining computing resources are greater than or equal to the computing resources required by the tasks in the preemptive scheduling candidate queue, suspend processing the batch tasks and start processing the tasks in the preemptive scheduling candidate queue; When the remaining computing resources are less than the computing resources required by the tasks in the preemptive scheduling candidate queue, suspend processing the batch tasks and stream tasks in the stream processing queue and the batch processing queue; When there is a suspended task, start calculating the processing duration and obtain the historical maximum suspension time. When the processing duration is greater than or equal to the historical maximum suspension time, cancel the suspension and mark the current streaming task being processed.

8. The system for hybrid processing of massive offline data and massive real-time data according to claim 7, characterized in that, It further includes: After obtaining the correction priority value, compare the current task jitter factor with the jitter threshold. If the current task jitter factor is greater than or equal to the jitter threshold, mark the current task and send it to the alarm module.

9. The system for hybrid processing of massive offline data and massive real-time data according to claim 8, wherein The alarm module determines that when the real-time parameter is greater than or equal to its corresponding threshold range, calculates the range excess, and issues an alarm in grades according to the range excess; The alarm module is also used to issue an alarm for the marked task.

10. A method for hybrid processing of massive offline data and massive real-time data, applied to the system for hybrid processing of massive offline data and massive real-time data according to any one of claims 1-9, characterized in that, The method includes: S1: Receive a task request, identify the task type and perform annotation. According to the annotation result, allocate the task to the streaming processing queue and the batch processing queue, and then sort the tasks according to the priority; the task types include streaming tasks and batch tasks; S2: Establish a streaming task thread pool and a batch task thread pool, process the tasks in sequence according to the priority, and collect real-time parameters; S3: Calculate the priority and calculate the task jitter factor according to the real-time parameters; S4: Determine the weight adjustment coefficient according to the magnitude relationship between the task jitter factor and the jitter reference value, correct the priority according to the weight adjustment coefficient, and determine the scheduling policy according to the corrected priority; S5: Compare the real-time parameter with the corresponding threshold range to determine whether to issue an alarm.

Images