A system and method for hybrid processing of mass offline data and mass real-time data

Abstract

The application relates to the technical field of electric data processing, and discloses a system and a method for mixed processing of mass offline data and mass real-time data. The system comprises a queue management module, a thread pool management module, a calculation module, a scheduling decision module and an alarm module. The queue management module receives a task request, identifies and labels a task type, distributes the task to a stream processing queue or a batch processing queue, and sorts the task according to priorities. The task type comprises a stream task and a batch task. The thread pool management module establishes a thread pool, processes the tasks according to priorities, and collects real-time parameters. The calculation module is responsible for calculating a task priority and a task jitter factor based on real-time parameters. The scheduling decision module compares the task jitter factor with a reference value, determines a weight adjustment coefficient, corrects the priority, and formulates a scheduling strategy. The alarm module judges whether an alarm is triggered. The application introduces a dynamic priority scheduling driven by a jitter factor and a preemptive processing mechanism, thereby improving the real-time response capability and resource utilization efficiency of the system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric data processing, in particular to a system and method for mixed processing of mass offline data and mass real-time data. BACKGROUND

[0002] With the rapid development of big data technology, there are a large number of offline batch processing tasks and real-time stream processing tasks in modern information systems. Offline tasks usually involve complex calculations of mass data, emphasizing the integrity and accuracy of data processing; while stream tasks focus on fast response and low latency processing of real-time data. These two types of tasks often compete for system resources, and how to efficiently coordinate mixed processing in the same system has become an important challenge in the current data processing field.

[0003] Usually, fixed priority or static resource partitioning is used to schedule batch tasks and stream tasks, resulting in low system resource utilization efficiency, and in the task peak period, the stream task response delay is too long or the batch task execution is blocked. In addition, there is a lack of dynamic monitoring and adjustment mechanism for task state and system running real-time parameters, which is difficult to adapt to complex and variable business loads, and cannot effectively solve the jitter and priority imbalance problem in task scheduling.

[0004] Therefore, how to realize an intelligent scheduling system that can dynamically perceive the real-time state of the system, flexibly adjust the priority and scheduling strategy of the task, and efficiently mix process mass offline data and real-time data has become a technical problem to be solved. SUMMARY

[0005] In view of this, the present application provides a system and method for mixed processing of mass offline data and mass real-time data to solve the problems of low scheduling efficiency, large task response delay and insufficient dynamic adjustment of priority caused by resource competition between stream tasks and batch tasks in the prior art.

[0006] In one aspect, the present application provides a system for mixed processing of mass offline data and mass real-time data, comprising:

[0007] The queue management module is configured to receive task requests, identify task types and mark, distribute tasks to stream processing queues and batch processing queues according to the marking results, and sort tasks according to priority; the task type includes stream tasks and batch tasks;

[0008] The thread pool management module is configured to establish stream task thread pool and batch task thread pool, process tasks in order according to priority, and collect real-time parameters;

[0009] A computing module is connected with the queue management module and the thread pool management module respectively, and is configured to calculate a priority and a task jitter factor according to real-time parameters;

[0010] A scheduling decision module is configured to determine a weight adjustment coefficient according to a size relationship between the task jitter factor and a jitter reference value, correct the priority according to the weight adjustment coefficient, and determine a scheduling strategy according to the corrected priority;

[0011] An alarm module is configured to compare the real-time parameters with corresponding threshold ranges to determine whether to issue an alarm.

[0012] Further, the priority is calculated according to the following relationship:

[0013] ;

[0014] Wherein, is the priority of the task, is the waiting time of the task, is a constant value corresponding to the type of the task, is a remaining value of a time agreed in an SLA agreement, is the time agreed in the SLA agreement, is the number of retries of the task, is a constant, is taken as [1, 3], , , and indicates an initial weight coefficient, ;

[0015] When the computing module obtains the priority, the queue management module sorts the tasks in the stream processing queue and the batch processing queue according to the priority.

[0016] Further, the real-time parameters include a current delay and a historical delay of the task, a thread pool resource usage state, and a priority change trend of the task, and the task jitter factor is calculated according to the following relationship:

[0017] ;

[0018] ;

[0019] Wherein, is the task jitter factor, is a current delay in the Tth batch of real-time data, is an average value of the current delay in the T batches of real-time data, is a current thread pool resource utilization anomaly item, is the jitter factor of the i-th batch of historical tasks, i≥3, is the current thread pool occupancy rate, is the abnormal threshold of the current thread pool usage rate, =0.85, is the conversion coefficient, indicating the sensitivity of the abnormal degree, , , and are the first to fourth jitter adjustment coefficients, respectively, =1.

[0020] Further, according to the size relationship between the task jitter factor and the jitter reference value, the weight adjustment coefficient is determined, comprising:

[0021] The difference between the task jitter factor and the jitter reference value and the ratio of the task jitter factor to the jitter reference value are calculated, and the weight adjustment coefficient satisfies the following relationship:

[0022] ;

[0023] wherein, is the weight adjustment coefficient, is the jitter reference value, =10 -6 .

[0024] Further, according to the weight adjustment coefficient, the priority is modified, comprising: multiplying the initial weight coefficient by the weight adjustment coefficient respectively, and then recalculating the priority to obtain the modified priority, denoted as the modified priority, and reordering the tasks according to the modified priority.

[0025] Further, according to the modified priority, a scheduling strategy is determined, comprising:

[0026] The task type is obtained, if it is a stream task, the current idle or lowest load thread is selected according to the reordered queue to perform instant execution in turn; when all threads are in a busy state, the task is added to a preemption scheduling candidate queue;

[0027] If it is a batch task, the processor time is divided into several time slices, and each task is allocated a time slice in turn according to the reordered queue, when the time slice is exhausted, if the task is not completed, the current task is put back to the end of the queue, if the task is completed, it is removed from the queue, and the next task is scheduled to continue execution.

[0028] ​​​Further, when there is a stream task in the preemption scheduling candidate queue, the remaining computing resources of the current device are obtained, and when the remaining computing resources are greater than or equal to the required computing resources of the task in the preemption scheduling candidate queue, the processing of the batch task is suspended and the task in the preemption scheduling candidate queue is started to be processed.

[0029] When the remaining computing resources are less than the required computing resources of the task in the preemption scheduling candidate queue, the batch tasks and stream tasks in the stream processing queue and the batch processing queue are suspended.

[0030] When there is a suspended task, the processing duration is started to be calculated, and the historical maximum suspension time is obtained, and when the processing duration is greater than or equal to the historical maximum suspension time, the suspension is cancelled and the current processed stream task is marked.

[0031] Further, further comprising:

[0032] When the modified priority value is obtained, the current task jitter factor is compared with the jitter threshold value, and if the current task jitter factor is greater than or equal to the jitter threshold value, the current task is marked and sent to the alarm module.

[0033] Further, the alarm module judges when the real-time parameter is greater than or equal to the threshold range corresponding thereto, calculates the range exceeding amount, and issues an alarm according to the range exceeding amount;

[0034] The alarm module is also used to issue an alarm for the marked task.

[0035] Compared with the prior art, the beneficial effects of the present application are that:

[0036] Through real-time labeling and classification of the task type by the queue management module, the stream task and the batch task are effectively distinguished, and fine allocation is performed according to the priority, thereby improving the pertinence of task processing and the utilization efficiency of system resources.

[0037] The delay, thread pool utilization rate and historical fluctuation trend are comprehensively considered to calculate the task jitter factor, and the task priority is corrected according to the task jitter factor, so that the intelligent self-adaptation of the scheduling strategy is realized, and the stability of scheduling and the load response capability of the system are improved.

[0038] In the scene of resource shortage or high-priority task burst, the batch task can be dynamically suspended and the computing resources can be given to the high-priority stream task, so that the timely processing of the key real-time task is ensured, and the flexibility and real-time response capability of the system are enhanced.

[0039] The system not only schedules based on initial task information, but also collects parameters such as thread pool state, task delay and priority change in real time, dynamically adjusts the scheduling strategy, and adapts to complex and variable task load environment.

[0040] When the real-time parameter is abnormal, the system can automatically mark high-risk tasks and issue alarms in stages, effectively supporting operation and maintenance monitoring, and preventing task accumulation, system congestion or service collapse.

[0041] The fusion of stream processing and batch processing models, combined with the thread pool and time slice scheduling mechanism, improves the support capability of the system for large-scale heterogeneous data processing, and is particularly suitable for efficient management and intelligent scheduling of massive data.

[0042] In another aspect, the application also provides a method for mixed processing of massive offline data and massive real-time data, comprising:

[0043] S1: receiving a task request, identifying the task type and marking, according to the marking result, distributing the task to a stream processing queue and a batch processing queue, and then sorting the tasks according to the priority; the task type includes stream tasks and batch tasks;

[0044] S2: establishing a stream task thread pool and a batch task thread pool, processing the tasks in order according to the priority, and collecting real-time parameters;

[0045] S3: calculating the priority and the task jitter factor according to the real-time parameters;

[0046] S4: determining the weight adjustment coefficient according to the size relationship between the task jitter factor and the jitter reference value, correcting the priority according to the weight adjustment coefficient, and determining the scheduling strategy according to the corrected priority;

[0047] S5: comparing the real-time parameters with the corresponding threshold range to determine whether to issue an alarm.

[0048] It should be noted that the method and system for mixed processing of massive offline data and massive real-time data provided by the application have the same beneficial effects, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0049] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are for purposes of illustration only and are not considered a limitation of the present application. Moreover, like reference numerals are used to designate identical components throughout the specification. In the drawings:

[0050] Figure 1 The functional block diagram of the system for mixed processing of massive offline data and massive real-time data provided by the embodiment of the application.

[0051] Figure 2 The flowchart of the method for mixed processing of massive offline data and massive real-time data provided by the embodiment of the application. DETAILED DESCRIPTION

[0052] Exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0053] Referring to Figure 1 As shown in the drawings, the embodiments of the present application provide a system for processing mass offline data and mass real-time data, comprising:

[0054] A queue management module is configured to receive a task request, identify a task type and mark it, distribute the task to a stream processing queue and a batch processing queue according to the marking result, and sort the tasks according to the priority; the task type includes a stream task and a batch task;

[0055] A thread pool management module is configured to establish a stream task thread pool and a batch task thread pool, process the tasks in sequence according to the priority, and collect real-time parameters;

[0056] A calculation module is connected with the queue management module and the thread pool management module, and is configured to calculate the priority and calculate a task jitter factor according to the real-time parameters;

[0057] A scheduling decision module is configured to determine a weight adjustment coefficient according to the size relationship between the task jitter factor and a jitter reference value, correct the priority according to the weight adjustment coefficient, and determine a scheduling strategy according to the corrected priority;

[0058] An alarm module is configured to compare the real-time parameters with the corresponding threshold range to determine whether to issue an alarm.

[0059] It should be noted that the embodiment realizes parallel processing of massive heterogeneous data by setting the queue management module to accurately identify and classify and schedule the stream task and the batch task, avoids resource conflicts; the priority and the task jitter factor are dynamically calculated by the calculation module, a real-time feedback mechanism is introduced, the system state fluctuation is effectively perceived; the scheduling decision module adjusts the task priority based on the jitter factor and the reference value, and formulates an adaptive scheduling strategy to ensure timely processing of critical tasks during system load peaks or resource shortages; the alarm module equipped at the same time can timely warn potential abnormalities based on real-time parameter changes, improve the overall robustness and safety of the system; in combination with the above mechanisms, the application significantly improves the task processing efficiency, resource utilization and stability of the stream-batch integrated system, and is particularly suitable for data processing scenarios with low delay, high throughput and high real-time requirements.

[0060] In some embodiments of the present application, the priority is obtained by the following relationship:

[0061] ;

[0062] wherein, is the priority of the task, is the task waiting time, is a constant value corresponding to the task type, is the remaining value of the time agreed in the SLA agreement, is the time agreed in the SLA agreement, is the number of task failure retries, is a constant, is taken as [1, 3], , , and represent the initial weight coefficient, ;

[0063] When the calculation module obtains the priority, the queue management module sorts the tasks in the stream processing queue and the batch processing queue according to the priority.

[0064] It should be noted that in the above formula, if each symbol in the formula has a unit, it is uniformly converted into the International System of Units, and then only the unit is calculated and the unit is deleted. In addition, for symbol B, preferably, when the task type is a stream task, the corresponding constant value is 1.5; when the task type is a batch task, the corresponding constant value is 1.0. The specific value of symbol B can be automatically tuned using genetic algorithm or reinforcement learning.

[0065] It can be understood that by introducing the multi-dimensional key factors such as task waiting time, task type weight, SLA remaining time and failure retry times, and using adjustable initial weight coefficients to weight and integrate them, the fine dynamic evaluation of task priority is realized; wherein the difference setting of task type constant value (such as higher flow task weight) can accurately reflect the time-sensitive sensitivity of different tasks, which is beneficial to improve the intelligence and time response ability of the overall system scheduling; in addition, with the help of genetic algorithm or reinforcement learning to automatically optimize the setting of parameter B, the limitations of artificial experience setting are avoided, so that the priority model can continuously adapt to the actual running state, so as to realize the reasonable scheduling and allocation of system resources, reduce task delay, improve task completion rate, and is especially suitable for mixed data processing scenes with high task complexity, strong resource constraints and high real-time requirements.

[0066] In some embodiments of the present application, the real-time parameters include: task current delay and historical delay, thread pool resource usage state and task priority change trend, and the task jitter factor is calculated by the following relationship:

[0067] ;

[0068] ;

[0069] wherein, is the task jitter factor, is the current delay in the Tth batch of real-time data, is the average value of the current delay in the T batches of real-time data, is the current thread pool resource utilization anomaly item, is the historical task jitter factor of the ith batch, i≥3, is the current thread pool occupancy rate, is the abnormal threshold of the current thread pool usage rate, =0.85, is the conversion coefficient, which represents the sensitivity of the abnormality degree, 、 、 and are the first to fourth jitter adjustment coefficients, respectively, + + + =1.

[0070] It should be noted that the jitter factor of the task is dynamically calculated by real-time parameters to measure the stability and scheduling priority fluctuation in the task execution process. The real-time parameters include: the difference between the task current delay and the historical delay, the thread pool resource usage state and the task priority change trend, and other information.

[0071] The task jitter factor consists of three components: The first component is the difference between the current task latency and the historical average latency, which measures whether the current task execution exhibits significant anomalies. The second component is the thread pool resource utilization anomaly, which indicates whether the current thread pool resource usage has exceeded a set threshold. When resource utilization exceeds the threshold, the degree of excess is expressed as a ratio; otherwise, it is zero. Specifically, if the current resource utilization exceeds the threshold, the factor is calculated as the weighted product of the anomaly sensitivity parameter and the excess ratio. If the threshold is not exceeded, the factor is zero. The third component is the average value of the continuous change in task scheduling priority, which reflects the degree of fluctuation in the task's priority across multiple time slices. The weighted combination of these three components yields the task's comprehensive jitter factor. The weights assigned to each component can be flexibly adjusted based on task type, system load, or policy requirements; the sum of all weight coefficients is one.

[0072] In practice, the calculated task jitter factor not only measures the stability of the current task state but also serves as a key basis for adjusting scheduling strategies. The scheduling decision module dynamically adjusts the scheduling priority of various tasks by determining the relationship between the current task jitter factor and a reference threshold, thereby optimizing the overall system resource scheduling and task responsiveness.

[0073] By introducing multi-dimensional real-time parameters (such as latency, resource utilization, and priority change amplitude) to construct a comprehensive jitter factor, it can comprehensively reflect the task execution status and scheduling stability, effectively avoiding the misjudgment problem caused by single indicator judgment; the relationship between resource utilization and threshold is used to express the degree of abnormality, with sensitivity adjustment capability, which improves the system's ability to identify and respond to sudden loads; adding an assessment of task priority change trends helps to discover potential risk tasks with unstable scheduling in advance, so as to take proactive intervention; the weight coefficient is adjustable and highly adaptable, and can perform strategy adaptive optimization according to different operation stages or business needs; combining the task jitter factor to correct the scheduling priority enables the entire system to maintain an efficient and stable operation state when facing a large number of mixed loads of streaming and batch tasks, greatly improving the task completion rate and system robustness.

[0074] In some embodiments of the present application, determining the weight adjustment coefficient according to the relationship between the task jitter factor and the jitter reference value includes:

[0075] Calculate the difference between the task jitter factor and the jitter reference value, as well as the ratio of the task jitter factor to the jitter reference value. The weight adjustment coefficient satisfies the following relationship:

[0076] ;

[0077] in, is the weight adjustment coefficient, a jitter reference value, = 10 -6 .

[0078] It should be noted that in some embodiments of the present application, in order to further improve the dynamic adaptability and accuracy of task scheduling, a weight adjustment coefficient calculation method based on the relationship between task jitter factor and reference value is designed. This method quantifies the deviation between the current state and the ideal state of the task, and then dynamically adjusts the weight parameter in the task priority ranking, to achieve a more sensitive and stable scheduling strategy.

[0079] The specific calculation process is as follows: according to the difference between the task jitter factor and the jitter reference value, and the relative proportion between the two, a weight adjustment factor is constructed. This factor satisfies the following mathematical relationship: subtract the set reference jitter factor value from the current jitter factor value of the task as the offset; then divide the current jitter factor value by the reference value plus a small constant as the normalization ratio; multiply the above two and then normalize with the sum of the current jitter factor value minus the reference value, to finally obtain the weight adjustment coefficient corresponding to the task. Wherein: the weight adjustment coefficient represents the factor used to amplify or converge the influence of the current task ranking in the task scheduling priority ranking; the jitter reference value is the benchmark value of the task jitter factor in the ideal state obtained by historical statistics; the small constant is one times ten to the minus sixth power, which is used to prevent division by zero error and stabilize the calculation process.

[0080] By calculating the deviation between the current task and the reference state in real time, the contribution of the task to the ranking priority is dynamically adjusted, which effectively improves the scheduling accuracy; when the task state deviates seriously, the ranking influence weight is increased or decreased, so that the system scheduling is more stable and the jitter propagation is reduced; this adjustment mechanism has unified calculation logic for different types of tasks, and is suitable for multiple types of loads such as stream processing and batch processing; the reference value and the small constant can be set by strategy and optimized by learning model, so that the scheduling strategy has stronger scene adaptation ability; through the jitter feedback adjustment mechanism, the scheduling logic tends to be reasonable in real time, reduces resource waste and delay backlog, and guarantees the optimal system performance.

[0081] In some embodiments of the present application, the priority is corrected according to the weight adjustment coefficient, including: multiplying the initial weight coefficient by the weight adjustment coefficient respectively to recalculate the priority to obtain the corrected priority, denoted as the corrected priority, and reordering the tasks according to the corrected priority.

[0082] It should be noted that by introducing the weight adjustment coefficient to correct the initial priority of the task, the priority weight can be adjusted in real time according to the current running state of the system, avoiding the scheduling rigidity caused by static weight, and effectively improving the adaptability of the scheduling strategy to the dynamic change scene. Different tasks may face different resource bottlenecks and quality of service requirements during running. By dynamically adjusting the weight and recalculating the priority, the response efficiency of high-priority tasks can be accurately controlled, ensuring that critical tasks are executed first and improving the overall service quality. When the system resources are tight, by adjusting the weight, the queuing influence of low-priority tasks is weakened, the resource contention and queuing block are slowed down, the congestion problem of the scheduling queue is alleviated, and the overall running efficiency and stability of the system are improved. If some tasks affect the fairness and stability of system scheduling due to abnormal behavior (such as excessive jitter and frequent retries), the priority of these tasks can be adjusted by weight attenuation, effectively suppressing the interference of these tasks to the scheduling order, and improving the robustness and fault tolerance of the system. By taking the corrected priority as the final sorting basis, the sorting result can better reflect the current system load situation and task urgency, so as to realize more accurate and reasonable scheduling decision, and avoid task "starvation" or "excessive preemption".

[0083] In some embodiments of the present application, the scheduling strategy is determined according to the corrected priority, including:

[0084] If it is a stream task, the current idle or lowest load thread is selected for immediate execution according to the reordered queue; when all threads are in a busy state, the task is added to the preemptive scheduling candidate queue;

[0085] If it is a batch task, the processor time is divided into several time slices, and each task is allocated a time slice in turn according to the reordered queue. When the time slice is exhausted, if the task is not completed, the current task is put back to the end of the queue, and if the task is completed, it is removed from the queue, and the next task is executed.

[0086] It should be noted that this scheme divides tasks into "stream tasks" and "batch tasks", and uses immediate execution and time slice rotation strategy respectively, so that the scheduling logic is more in line with the characteristics of the task itself, effectively improving the compatibility of the system to different task models, and enhancing the universality and adaptability of the scheduling module.

[0087] By preferentially selecting an idle or lowest load thread to execute a stream task immediately, the waiting time of the stream task is compressed to the maximum extent; when all thread resources are occupied, the stream task is added to the preemptive scheduling candidate queue, and is executed first when resources are available, effectively guaranteeing the timeliness and stability of real-time sensitive tasks.

[0088] For batch tasks, a time slice round-robin scheduling strategy is adopted to ensure that each task can obtain processing time under the premise of ensuring system fairness, and to avoid individual tasks from occupying resources for a long time or being starved due to low priority. At the same time, when the task is completed within the time slice, the resource can be released in time to improve the resource utilization efficiency.

[0089] By combining the current thread load information for scheduling, it is helpful to dynamically balance the load pressure of each processing thread, reduce thread queuing block and resource contention, and improve concurrent scheduling capability and task throughput efficiency.

[0090] In the overall strategy, the scheduling mode is driven by priority and is divided according to the differences in task types, so that the scheduling control is more fine and the strategy is more flexible. The parameters and execution logic can be further optimized according to the specific system goals (such as minimizing response time and maximizing resource utilization).

[0091] Through reasonable scheduling mechanism, the stream task is timely responded and the batch task is handled fairly, which helps to maintain the overall quality of service (QoS) of the system, reduce the service violation rate (such as exceeding the SLA agreement time), and improve the end user experience.

[0092] In some embodiments of the present application, when there is a stream task in the preemption scheduling candidate queue, the remaining computing resources of the current device are obtained, and when the remaining computing resources are greater than or equal to the required computing resources of the tasks in the preemption scheduling candidate queue, the batch tasks are suspended and the tasks in the preemption scheduling candidate queue are started to be processed;

[0093] When the remaining computing resources are less than the required computing resources of the tasks in the preemption scheduling candidate queue, the batch tasks and stream tasks in the stream processing queue and batch processing queue are suspended;

[0094] When there are suspended tasks, the processing duration is started to be calculated and the historical maximum suspension time is obtained, and when the processing duration is greater than or equal to the historical maximum suspension time, the suspension is cancelled and the current processed stream task is marked.

[0095] It should be noted that by setting the preemption mechanism, the resource situation is actively monitored when the stream task enters the preemption candidate queue, the tasks with high real-time requirements are preferentially guaranteed, the processing opportunity is preferentially obtained when the resources are available, the real-time response capability of the system to critical tasks is significantly improved, and the quality of service demand in the real-time processing scene is met.

[0096] The current remaining computing resources of the device are dynamically evaluated and compared with the required resources of the tasks, the scheduling strategy is accurately controlled, the batch tasks are effectively prevented from occupying resources for a long time or causing stream task delay due to resource waste, on-demand allocation and fine scheduling of resources are realized, and the overall computing resource utilization rate is improved.

[0097] The automatic suspension of batch task and stream task processing is performed when resources are insufficient, and the processing duration of the suspended task is monitored and compared with the historical maximum suspension time to ensure that the task is not indefinitely delayed, and the task is timely recovered through a timeout recovery strategy to ensure the stability and reliability of system operation.

[0098] The long-suspended stream task is identified and managed through the marking mechanism to provide decision basis for subsequent scheduling optimization (such as priority promotion and resource reallocation), which helps the system to automatically adapt to complex scenarios such as resource shortage and task congestion, and improves the adaptive scheduling capability of the system under high load.

[0099] The method takes into account "fairness" and "timeliness" in the preemption mechanism, that is, the stream task is given priority in the case of resource conditions, and all tasks are suspended and dynamically monitored in the case of resource shortage, forming a fair waiting and awakening mechanism, so that different types of tasks can still obtain reasonable processing opportunities under high load, effectively coordinating the conflict between fairness and real-time scheduling goals.

[0100] With the dynamic acquisition and comparison of the "historical maximum suspension time", the task scheduling strategy provides data-driven learning basis, which can further adaptively adjust the scheduling threshold combined with machine learning algorithm, and promote the evolution of the system towards intelligence and autonomous decision scheduling.

[0101] In some embodiments of the present application, it further comprises:

[0102] When the corrected priority value is obtained, the current task jitter factor is compared with the jitter threshold value, and if the current task jitter factor is greater than or equal to the jitter threshold value, the current task is marked and sent to the alarm module.

[0103] The alarm module judges when the real-time parameter is greater than or equal to the threshold range corresponding thereto, calculates the range exceeding amount, and issues an alarm according to the range exceeding amount;

[0104] The alarm module is also used to issue an alarm for the marked task.

[0105] It should be noted that the dynamic monitoring and alarm management of the task state are realized through the following steps:

[0106] Firstly, when the system obtains the corrected priority value, the jitter factor of the current task is compared with the pre-set jitter threshold value. The jitter factor reflects the fluctuation degree of the task in the running process, and if the jitter factor is greater than or equal to the threshold value, it means that the task state is abnormal or fluctuates greatly. At this time, the system will immediately mark the task as abnormal and send its information to the alarm module.

[0107] After receiving the marked task information, the alarm module further judges whether the real-time parameter related to the task exceeds its corresponding threshold range. If the real-time parameter is greater than or equal to the threshold, the alarm module calculates the specific amplitude of exceeding the threshold (i.e. the range exceeding amount). According to the size of the exceeding amount, the system will be processed in stages, and different levels of alarms will be issued in stages. The classification of alarm levels helps the operation and maintenance personnel to clearly understand the severity of the problem, so as to make corresponding response measures.

[0108] In addition, the alarm module not only alarms for real-time parameter abnormalities, but also alarms for previously marked tasks to ensure that all tasks that may have risks are timely focused.

[0109] By comparing the task jitter factor with the threshold, the system can sensitively capture subtle but important abnormal fluctuations in the task state. The jitter factor as a quantitative indicator to measure task fluctuations makes abnormal detection no longer rely on simple static judgment, but combines dynamic change trend, which is more scientific and reasonable. This effectively avoids the possibility of ignoring potential risks and improves the accuracy of abnormal detection.

[0110] The alarm module not only judges whether the task is abnormal, but also classifies the alarm based on the amplitude of the real-time parameter exceeding the threshold. This hierarchical strategy is beneficial to distinguish the urgency of the problem. For example, slight over-standard can trigger a low-level alarm to remind the monitoring personnel to pay attention; while serious over-standard triggers a high-level alarm, prompting immediate intervention. This hierarchical mechanism improves the pertinence and practicality of the alarm, avoiding the problem of "alarm fatigue" caused by all abnormal alarms.

[0111] By marking abnormal tasks and sending information to the alarm module, the system realizes the proactive warning of task abnormalities. The alarm module can continuously track the status of these tasks and timely issue alarms to ensure that abnormal information is not missed. This mechanism makes the operation and management more proactive and forward-looking, reducing the risk of passive waiting for faults to occur.

[0112] Timely detection of abnormal tasks and multi-level alarm mechanism help to discover potential system risks in advance, promote rapid response and processing, avoid the expansion of abnormalities or cause more serious faults, and ensure the overall stable operation and safety of the system.

[0113] Automatic comparison of task jitter factor and threshold, automatic calculation of range exceeding amount and hierarchical alarm, reduces the dependence on manual monitoring. The alarm module uniformly manages abnormal tasks and real-time parameter alarms, improves the automation management capability of operation and maintenance, so that the management personnel can focus more on the analysis and decision of important problems, and improves the overall work efficiency.

[0114] Through quantifying the jitter factor and the range exceeding amount of real-time parameters, combined with hierarchical alarms, the system can more carefully and comprehensively manage the running status of tasks. This not only helps to discover abnormalities in time, but also accumulates abnormal data to provide data support for subsequent optimization of task scheduling and system performance.

[0115] Specifically, regarding the range exceeding amount, a plurality of preset values sequentially increasing in value are set, when the range exceeding amount is between a first preset value and a second preset value, the alarm level is set to a low-level alarm, when the range exceeding amount is between the second preset value and a third preset value, the alarm level is set to a middle-level alarm, and so on.

[0116] Referring to Figure 2 The embodiment of the present application provides a method for mixed processing of mass offline data and mass real-time data, comprising:

[0117] S1: receiving a task request, identifying a task type and performing labeling, according to the labeling result, distributing the task to a stream processing queue and a batch processing queue, and then sorting the tasks according to priorities; the task type includes a stream task and a batch task;

[0118] S2: establishing a stream task thread pool and a batch task thread pool, processing the tasks in sequence according to the priorities, and collecting real-time parameters;

[0119] S3: calculating a priority, and calculating a task jitter factor according to the real-time parameters;

[0120] S4: determining a weight adjustment coefficient according to the size relationship between the task jitter factor and a jitter reference value, modifying the priority according to the weight adjustment coefficient, and determining a scheduling strategy according to the modified priority;

[0121] S5: comparing the real-time parameters with corresponding threshold ranges to determine whether to issue an alarm.

[0122] It can be understood that by accurately identifying and classifying the task type, the stream task and the batch task are respectively distributed to independent processing queues and thread pools, avoiding resource competition and mutual interference between the two types of tasks, thereby improving the concurrent processing capability and execution efficiency of the overall system.

[0123] Based on task labeling and priority calculation, the system can reasonably sort tasks to ensure that high-priority tasks are given priority in scheduling resources, improve the response speed and processing efficiency of critical tasks, and meet the timeliness requirements of different businesses.

[0124] By collecting relevant parameters of task execution in real time and calculating the jitter factor, the system can dynamically evaluate the stability of task execution, modify the priority by combining the weight adjustment coefficient, realize intelligent optimization of the task scheduling strategy, and improve the scheduling flexibility and system adaptability.

[0125] When there is a large fluctuation in the task or a change in the system environment, the dynamic correction of the priority ensures that the scheduling strategy can be adjusted in time, avoiding resource waste or critical task delay caused by abnormal fluctuations, and improving the robustness and stability of the system.

[0126] The real-time parameter comparison and hierarchical alarm mechanism can timely discover task abnormalities, and differentially alarm according to the severity of the signal, assisting the operation and maintenance personnel to quickly locate the problem, reducing the system downtime, and improving the safety guarantee capability of the system.

[0127] The entire method automatically completes task classification, priority calculation, scheduling strategy adjustment and alarm sending, greatly reducing the dependence on manual monitoring, improving the automation and intelligent management level of the system, and reducing the operation and maintenance cost.

[0128] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it, although the present application has been described in detail with reference to the above examples, those skilled in the art should understand that: the specific embodiments of the present application can still be modified or replaced by the equivalent, without departing from the spirit and scope of the present application, any modification or equivalent replacement, which should be covered within the protection scope of the claims of the present application.

Claims

1. A system for hybrid processing of massive offline data and massive real-time data, characterized in that: include: The queue management module is configured to receive task requests, identify and label the task types, assign the tasks to stream processing queues and batch processing queues based on the labeling results, and then sort the tasks according to priority; the task types include stream tasks and batch tasks; The thread pool management module is configured to establish a stream task thread pool and a batch task thread pool, process tasks in order of priority, and collect real-time parameters; a calculation module, connected to the queue management module and the thread pool management module respectively, and configured to calculate the priority and the task jitter factor according to the real-time parameters; a scheduling decision module configured to determine a weight adjustment coefficient based on a relationship between the task jitter factor and the jitter reference value, modify the priority according to the weight adjustment coefficient, and determine a scheduling strategy based on the modified priority; an alarm module configured to compare the real-time parameter with a corresponding threshold range to determine whether to issue an alarm; The real-time parameters include: the current delay and historical delay of the task, the thread pool resource usage status and the priority change trend of the task. The task jitter factor is calculated and obtained through the following relationship: in, is the task jitter factor, is the current delay in the Tth batch of real-time data, is the current average delay in T batches of real-time data, This is an exception in the current thread pool resource utilization. is the jitter factor of the historical tasks in the i-th batch, i≥3, is the current thread pool occupancy, is the abnormal threshold of the current thread pool usage, =0.85, is the conversion coefficient, which indicates the sensitivity of the abnormality. 、 、 and These are the first to fourth jitter adjustment coefficients, + + + =1; Determine the scheduling strategy based on the revised priority, including: Get the task type. If it is a stream task, select the currently idle or least loaded thread according to the reordered queue for immediate execution. When all threads are busy, add the task to the preemptive scheduling candidate queue. If it is a batch task, the processor time is divided into several time slices, and a time slice is allocated to each task in turn according to the reordered queue. When the time slice is exhausted, if the task is not completed, the current task is put back to the end of the queue. If the task is completed, it is removed from the queue and the next task is scheduled to continue execution.

2. The system for hybrid processing of massive offline data and massive real-time data according to claim 1, characterized in that: The priority is calculated by the following relationship: in, is the priority of the task, A is the task waiting time, B is the constant value corresponding to the task type, is the remaining value of the agreed time in the SLA agreement, The time agreed in the SLA agreement, The number of retries for failed tasks. is a constant, Take [1, 3], 、 、 and represents the initial weight coefficient, ; After the calculation module obtains the priority, the queue management module sorts the tasks in the stream processing queue and the batch processing queue according to the priority.

3. The system for hybrid processing of massive offline data and massive real-time data according to claim 2, characterized in that: Determining the weight adjustment coefficient according to the magnitude relationship between the task jitter factor and the jitter reference value includes: The difference between the task jitter factor and the jitter reference value and the ratio of the task jitter factor to the jitter reference value are calculated, and the weight adjustment coefficient satisfies the following relationship: in, is the weight adjustment coefficient, is the jitter reference value, =10 -6 .

4. The system for hybrid processing of massive offline data and massive real-time data according to claim 3, characterized in that: The priority is modified according to the weight adjustment coefficient, including: multiplying the initial weight coefficient by the weight adjustment coefficient respectively, recalculating the priority to obtain a modified priority, recorded as the modified priority, and reordering the tasks according to the modified priority.

5. The system for hybrid processing of massive offline data and massive real-time data according to claim 4, characterized in that: When there is a stream task in the preemptive scheduling candidate queue, obtain the remaining computing resources of the current device. When the remaining computing resources are greater than or equal to the computing resources required by the tasks in the preemptive scheduling candidate queue, suspend the processing of batch tasks and start processing the tasks in the preemptive scheduling candidate queue. When the remaining computing resources are less than the computing resources required by the tasks in the preemptive scheduling candidate queue, suspending the processing of the batch tasks and stream tasks in the stream processing queue and the batch processing queue; When there is a paused task, the processing duration is calculated and the historical maximum pause time is obtained. When the processing duration is greater than or equal to the historical maximum pause time, the pause is canceled and the currently processed stream task is marked.

6. The system for hybrid processing of massive offline data and massive real-time data according to claim 5, characterized in that: Also includes: After obtaining the corrected priority value, the jitter factor of the current task is compared with the jitter threshold. If the jitter factor of the current task is greater than or equal to the jitter threshold, the current task is marked and sent to the alarm module.

7. The system for hybrid processing of massive offline data and massive real-time data according to claim 6, characterized in that: The alarm module determines that when the real-time parameter is greater than or equal to the corresponding threshold range, calculates the range exceeding amount, and issues an alarm according to the level of the range exceeding amount; The alarm module is further configured to issue an alarm for the marked task.

8. A method for hybrid processing of massive offline data and massive real-time data, applied to the system for hybrid processing of massive offline data and massive real-time data according to any one of claims 1 to 7, characterized in that: The method comprises: S1: Receive task requests, identify and label task types, assign tasks to stream processing queues and batch processing queues based on the labeling results, and then sort tasks according to priority; the task types include stream tasks and batch tasks; S2: Establish a stream task thread pool and a batch task thread pool, process tasks in order of priority, and collect real-time parameters; S3: Calculates the priority and the task jitter factor based on real-time parameters; S4: determining a weight adjustment coefficient according to a magnitude relationship between the task jitter factor and the jitter reference value, modifying the priority according to the weight adjustment coefficient, and determining a scheduling strategy according to the modified priority; S5: Compare the real-time parameters with the corresponding threshold range to determine whether to issue an alarm.

Images