Real-time stream computing and service scheduling method and system based on distributed architecture platform

Abstract

The invention discloses a real-time stream computing and service scheduling method and system based on a distributed architecture platform, and belongs to the technical field of real-time stream computing and scheduling strategies. The problems that in the prior art, a traditional real-time flow calculation and service scheduling method is low in resource utilization rate and low in calculation efficiency are solved. According to the invention, a real-time stream computing and service scheduling system based on a distributed architecture platform is established; in a self-defined task scheduling center of a real-time flow calculation and service scheduling system, a workflow scheduling and management platform is integrated, a directed acyclic graph workflow is defined, and scheduling rules and strategies are formulated; and integrating a stream computing engine of a resolver for real-time stream data processing in the self-defined stream computing processing center, performing stream computing, storing a computing result into a DolphinDB database, and completing real-time stream computing and service scheduling. The method effectively improves the data processing efficiency and the service response speed, and can be applied to large-scale data stream processing.

Description

Technical Field

[0001] The present invention relates to a real-time stream computing and service scheduling method and system, and in particular to a real-time stream computing and service scheduling method and system based on a distributed architecture platform, belonging to the technical field of real-time stream computing and scheduling strategy. Background Art

[0002] In the current distributed architecture platform, service scheduling and real-time stream computing are two core functions. However, with the continuous surge in data volume and the improvement of real-time requirements, existing technologies and methods have gradually exposed some significant problems. In terms of platform service scheduling, the existing technologies often have problems such as low resource utilization, single scheduling strategy, and lack of dynamic adjustment mechanism in scheduling strategies. In terms of real-time stream computing, traditional computing models are often unable to effectively cope with large-scale continuous data streams. Bottlenecks are easily generated during the transmission and processing of data between computing nodes, resulting in increased computing delays and failure to meet real-time requirements.

[0003] In summary, a real-time stream computing and service scheduling method and system based on a distributed architecture platform is needed. Summary of the invention

[0004] A brief overview of the present invention is provided below in order to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0005] In view of this, in order to solve the problems of low resource utilization and low computing efficiency of traditional real-time stream computing and service scheduling methods in the prior art, the present invention provides a real-time stream computing and service scheduling method and system based on a distributed architecture platform.

[0006] Technical solution 1 is as follows: A real-time stream computing and service scheduling method based on a distributed architecture platform includes the following steps:

[0007] S1. Build a real-time stream computing and service scheduling system based on a distributed architecture platform;

[0008] S2. Integrate Apache Airflow components in the custom task scheduling center of the real-time stream computing and service scheduling system, define directed acyclic graph workflows, and formulate scheduling rules and strategies;

[0009] S3. In the custom stream computing processing center of the real-time stream computing and service scheduling system, the stream computing engine of the StreamEngineParser component is integrated to perform stream computing, store the calculation results in the DolphinDB database, and complete the real-time stream computing and service scheduling.

[0010] Furthermore, the step S2 specifically includes the following steps:

[0011] S21. In the custom task scheduling center, Apache Airflow components are called through Java to build a directed acyclic graph workflow, where each node represents a task and the edge represents the dependency relationship between tasks;

[0012] S22. Add task nodes of the directed acyclic graph workflow, which include data source acquisition, data preprocessing, feature extraction, real-time analysis, and define the execution order and dependencies of tasks;

[0013] S23. Setting the properties of the directed acyclic graph workflow to ensure that the directed acyclic graph workflow starts executing according to the predetermined cycle and time point;

[0014] S24. According to the constructed directed acyclic graph workflow and the dependency relationship of the tasks, the scheduler in the custom task scheduling center is used to schedule the task execution, and the comprehensive priority calculation is introduced to obtain the scheduling comprehensive priority;

[0015] S25. According to the comprehensive scheduling priority, design the exception handling and fault tolerance mechanism, obtain the scheduling rules and strategies, and implement the automatic retry mechanism of the task. When the task fails to execute, the task is automatically rescheduled according to the preset number of retries and delay time interval;

[0016] In S24, the process of calculating the comprehensive priority is as follows:

[0017] S241. Using the normalization method, the urgency of the task is expressed as the relative value of the difference between the current time and the deadline, and the task urgency priority P is calculated. u,i ;

[0018] Task urgency priority u,i It is expressed as:

[0019]

[0020] Among them, D i represents the deadline of the current task i, T represents the current time, and D j Indicates the deadline of any task;

[0021] S242. By comparing the task resource requirement with the maximum resource requirement in the task set, the resource requirement priority P is calculated. r,i ;

[0022] Resource requirement priority P r,i It is expressed as:

[0023]

[0024] Among them, R i represents the amount of resources required for the current task i, R j Indicates the amount of resources required for any task;

[0025] S243. Use the in-degree and out-degree of the task as the measure of the task dependency priority to calculate the task dependency priority P d,i ;

[0026] The dependency priority of the task P d,i It is expressed as:

[0027]

[0028] Among them, De i Denotes the dependency factor corresponding to the current task i, De j Represents the dependency factor corresponding to any task;

[0029] S244. Calculate the comprehensive priority P t,i ;

[0030] P t,i =a·P u,i +b·P r,i +c·P d,i

[0031] Among them, a represents the first weight coefficient, which is used to adjust the proportion of urgency in the priority calculation, b represents the first weight coefficient, which is used to adjust the proportion of resource requirements in the priority calculation, c represents the first weight coefficient, which is used to adjust the proportion of task dependencies in the priority calculation, and a+b+c=1.

[0032] Furthermore, the step S3 specifically includes the following steps:

[0033] S31. Configure optional stream computing engines for the DolphinDB database, including the responsive state engine, cross-section engine, and time series engine, and integrate the StreamEngineParser component into the DolphinDB database environment by configuring a JSON file;

[0034] S32. Define the stream computing parameters including input table, output table and custom algorithm in DolphinDB database;

[0035] S33. Write a script in the DolphinDB database that subscribes to the input table, calls the custom algorithm, and publishes the results to the output table;

[0036] S34. Build a pipeline through the StreamEngineParser component to convert the user-defined algorithm into efficient computing instructions that can be directly understood and executed by the underlying hardware. The computing instructions include data reading instructions, data processing instructions, and data output instructions. The computing instructions are optimized and the optimized computing instructions are executed to obtain the computing results.

[0037] S35. Store the calculation results in the DolphinDB database in real time, or send the results to a specified output destination according to business needs;

[0038] In S32, an input table is first defined, the input table includes an indicator name, an indicator value and a timestamp, and then an output table is defined, the output table includes a calculated predicted value and a corresponding timestamp, and finally a custom algorithm is defined in which the user sets the algorithm logic according to business needs.

[0039] Technical solution 2 is as follows: A real-time stream computing and service scheduling system based on a distributed architecture platform, used to implement the real-time stream computing and service scheduling method based on a distributed architecture platform described in technical solution 1, including a custom task scheduling center, a custom stream computing processing center, a distributed storage center and a data acquisition center;

[0040] The custom task scheduling center is connected to the custom stream computing processing center and the distributed storage center respectively, and the custom stream computing processing center is connected to the distributed storage center and the data acquisition center respectively.

[0041] The beneficial effects of the present invention are as follows: the present invention proposes a real-time stream computing and service scheduling method, which relies on a distributed architecture platform and integrates the workflow management of the Apache Airflow component, the stream processing and parsing capabilities of the StreamEngineParser component, and the high-performance time series database of DolphinDB to achieve an efficient, flexible and scalable real-time data processing and service scheduling solution; the present invention effectively solves the problems of delay, uneven resource allocation and system complexity in large-scale data stream processing, and significantly improves data processing efficiency and service response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0043] Figure 1 A flowchart of a real-time stream computing and service scheduling method based on a distributed architecture platform;

[0044] Figure 2 It is a structural diagram of the real-time stream computing and service scheduling system based on the distributed architecture platform;

[0045] Figure 3 The present invention is a schematic diagram of an embodiment of a real-time stream computing and service scheduling method and system based on a distributed architecture platform.

[0046] Figure numerals: 1. Customized task scheduling center; 2. Customized stream computing processing center; 3. Distributed storage center; 4. Data acquisition center. DETAILED DESCRIPTION

[0047] In order to make the technical solutions and advantages of the embodiments of the present invention more clearly understood, the exemplary embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0048] Example 1: Reference Figure 1-Figure 3 The present embodiment is described in detail, a real-time stream computing and service scheduling method based on a distributed architecture platform, specifically comprising the following steps:

[0049] S1. Build a real-time stream computing and service scheduling system based on a distributed architecture platform;

[0050] S2. Integrate Apache Airflow components (workflow scheduling and management platform) in the custom task scheduling center of the real-time stream computing and service scheduling system, define directed acyclic graph workflows (DAGs), and formulate scheduling rules and strategies;

[0051] S3. In the stream computing processing center of the real-time stream computing and service scheduling system, the stream computing engine of the StreamEngineParser component (a parser for real-time stream data processing) is integrated to perform stream computing and store the calculation results in the DolphinDB database for subsequent data analysis, business decision-making or visualization.

[0052] Furthermore, the step S2 specifically includes the following steps:

[0053] S21. In the custom task scheduling center, Apache Airflow components are called through Java to build a directed acyclic graph workflow, where each node represents a task and the edge represents the dependency relationship between tasks;

[0054] S22. Add task nodes of the directed acyclic graph workflow, which include data source acquisition, data preprocessing, feature extraction, real-time analysis, etc., and define the execution order and dependencies of tasks;

[0055] S23. Setting properties of the directed acyclic graph workflow, such as task owner, start date, scheduling interval, etc., to ensure that the directed acyclic graph workflow starts execution according to the predetermined cycle and time point;

[0056] S24. According to the constructed directed acyclic graph workflow and the dependency relationship of the tasks, the scheduler in the custom task scheduling center is used to schedule the task execution, and the comprehensive priority calculation is introduced to obtain the scheduling comprehensive priority;

[0057] S25. According to the comprehensive scheduling priority, design the exception handling and fault tolerance mechanism, obtain the scheduling rules and strategies, and implement the automatic retry mechanism of the task. When the task execution fails, the task is automatically rescheduled according to the preset number of retries and delay time interval. The monitoring and alarm system is integrated into the real-time stream computing and service scheduling system to monitor the task execution status and system resource usage in real time, and issue alarms in time for abnormal situations and trigger the corresponding processing flow;

[0058] In S24, the process of calculating the comprehensive priority is as follows:

[0059] S241. Using the normalization method, the urgency of the task is expressed as the relative value of the difference between the current time and the deadline, and the task urgency priority P is calculated. u,i ;

[0060] Task urgency priority u,i It is expressed as:

[0061]

[0062] Among them, D i represents the deadline of the current task i, T represents the current time, and D j Indicates the deadline of any task;

[0063] S242. By comparing the task resource requirement with the maximum resource requirement in the task set, the resource requirement priority P is calculated. r,i ;

[0064] Resource requirement priority P r,i It is expressed as:

[0065]

[0066] Among them, R i represents the amount of resources required for the current task i, R j Indicates the amount of resources required for any task;

[0067] S243. Use the in-degree (i.e., the number of other tasks that depend on the task) and out-degree (i.e., the number of other tasks that the task depends on) of the task as the measure of the dependency priority to calculate the dependency priority P of the task. d,i ;

[0068] The dependency priority of the task P d,i It is expressed as:

[0069]

[0070] Among them, De i Denotes the dependency factor corresponding to the current task i, De j Represents the dependency factor corresponding to any task;

[0071] S244. Calculate the comprehensive priority P t,i ;

[0072] P t,i =a·P u,i +b·P r,i +c·P d,i

[0073] Among them, a represents the first weight coefficient, which is used to adjust the proportion of urgency in the priority calculation, b represents the first weight coefficient, which is used to adjust the proportion of resource requirements in the priority calculation, c represents the first weight coefficient, which is used to adjust the proportion of task dependencies in the priority calculation, and a+b+c=1.

[0074] Furthermore, the step S3 specifically includes the following steps:

[0075] S31. Configure optional stream computing engines for the DolphinDB database, including ReactiveStateEngine, CrossSectionalEngine, and TimeSeriesEngine, and integrate the StreamEngineParser component into the DolphinDB database environment by configuring a JSON file;

[0076] S32. Define the stream computing parameters including input table, output table and custom algorithm in DolphinDB database;

[0077] S33. Write a script in the DolphinDB database that subscribes to the input table, calls the custom algorithm, and publishes the results to the output table;

[0078] S34. Build a pipeline through the StreamEngineParser component to convert the user-defined algorithm into efficient computing instructions that can be directly understood and executed by the underlying hardware. The computing instructions include data reading instructions (data source access, data flow control, etc.), data processing instructions (computing instructions, data conversion instructions, window operation instructions, etc.), data output instructions (result storage instructions, data flow end instructions), etc., and optimize the computing instructions, execute the optimized computing instructions, and obtain the computing results;

[0079] S35. Store the calculation results in real time in the distributed time series database of DolphinDB, or send the results to a specified output destination such as a message queue, database table, visualization interface, etc. according to business needs for subsequent analysis and application;

[0080] In S32, an input table is first defined, which is used to receive real-time data streams. The input table includes fields such as indicator name, indicator value, and timestamp. Then, an output table is defined, which is used to store the results of stream calculation. The output table includes the calculated prediction value (MAP) and the corresponding timestamp. Finally, a user-defined algorithm is defined for setting the algorithm logic according to business needs.

[0081] Specifically, in this embodiment, in step S32, the specific settings of the custom algorithm are as follows:

[0082] The IoT system contains a large number of temperature sensors and humidity sensors deployed in different locations. It is necessary to calculate the comprehensive environmental index CEI of each location in real time. This index comprehensively considers the changes in temperature and humidity and has a complex nonlinear relationship.

[0083] The comprehensive environmental index CEI is expressed as:

[0084]

[0085] Wherein, T is the current temperature, H is the current humidity, T0 is the reference value of temperature, T0=25°C, H0 is the reference value of humidity, H0=50%RH, α0 is the first model parameter, α0=1.2, α1 is the second model parameter, α1=1.5, α2 is the third model parameter, α2=0.8, σ is the fourth model parameter, σ=0.5, β is the fifth model parameter, β=0.01;

[0086] In step S34, the optimization method of calculating the instruction is as follows:

[0087] S341. Use loop unrolling and instruction parallelism to reduce the overhead of loop control instructions and increase instruction-level parallelism. For a loop instruction, it needs to iterate m times to execute a complex computing task;

[0088] The iterative process of performing a complex computing task is expressed as:

[0089]

[0090] Where A represents the first matrix, B represents the second matrix, C represents the third matrix, m represents the number of iterations, and j represents a task other than the current task i;

[0091] During the calculation process, the code in the loop body is copied multiple times through loop unrolling, thereby reducing the number of loop iterations; each iteration can calculate multiple elements at the same time, improving calculation efficiency;

[0092] The loop unrolling process is expressed as:

[0093] C[i][j]=(A[i][0])×B[0][j]+(A[i][1])×B[1][j])

[0094] +(A[i][2])×B[2][j]+(A[i][3])×B[3][j])+...

[0095] S342. Use vectorized single instruction multiple data (SIMD) instruction set to process multiple instructions simultaneously;

[0096] For functions that need to operate on arrays, the array is divided into multiple vectors through vectorization, each vector contains multiple elements, and then the vectors are processed in parallel using single instruction multiple data (SIMD) instructions;

[0097] The parallel processing process is as follows:

[0098] First, divide the array X into multiple vectors, such as each vector contains 4 elements;

[0099] A vector V containing 4 elements K It is expressed as:

[0100] V K =(X[4k],X[4k+1],X[4k+2],X[4k+3])

[0101] Use SIMD instructions to perform operations on a vector V containing 4 elements. K Perform parallel summation to obtain the processing results S of multiple instructions, that is, the optimized calculation instructions;

[0102] The processing result S of multiple instructions is expressed as:

[0103]

[0104] Among them, SIMD_SUM means using single instruction multiple data instructions to vector V k Perform a sum operation.

[0105] Example 2: Reference Figure 2-Figure 3 This embodiment is described in detail. A real-time stream computing and service scheduling system based on a distributed architecture platform is used to implement the real-time stream computing and service scheduling method based on a distributed architecture platform described in Example 1, including a custom task scheduling center 1, a custom stream computing processing center 2, a distributed storage center 3 and a data acquisition center 4;

[0106] The custom task scheduling center 1 is connected to the custom stream computing processing center 2 and the distributed storage center 3 respectively, and the custom stream computing processing center 2 is connected to the distributed storage center 3 and the data acquisition center 4 respectively.

[0107] Specifically, the custom stream computing processing center 2 relies on the collaboration of each node to perform complex real-time computing tasks;

[0108] The custom task scheduling center 1 is used to divide into master and slave nodes and intelligently allocate tasks according to system load and resource status;

[0109] Distributed storage center 3 is responsible for persistent storage of data, and distributed storage center 3 is DolphinDB database;

[0110] The component nodes work together through efficient data transmission and scheduling mechanisms.

[0111] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.

Claims

1. A real-time stream computing and service scheduling method based on a distributed architecture platform, characterized in that: The following steps are involved: S1. Build a real-time stream computing and service scheduling system based on a distributed architecture platform; S2. Integrate Apache Airflow components in the custom task scheduling center of the real-time stream computing and service scheduling system, define directed acyclic graph workflows, and formulate scheduling rules and strategies; S3. In the custom stream computing processing center of the real-time stream computing and service scheduling system, the stream computing engine of the StreamEngineParser component is integrated to perform stream computing, store the calculation results in the DolphinDB database, and complete the real-time stream computing and service scheduling.

2. A real-time stream computing and service scheduling method based on a distributed architecture platform according to claim 1, characterized in that: The S2 specifically includes the following steps: S21. In the custom task scheduling center, Apache Airflow components are called through Java to build a directed acyclic graph workflow, where each node represents a task and the edge represents the dependency relationship between tasks; S22. Add task nodes of the directed acyclic graph workflow, which include data source acquisition, data preprocessing, feature extraction, real-time analysis, and define the execution order and dependencies of tasks; S23. Setting the properties of the directed acyclic graph workflow to ensure that the directed acyclic graph workflow starts executing according to the predetermined cycle and time point; S24. According to the constructed directed acyclic graph workflow and the dependency relationship of the tasks, the scheduler in the custom task scheduling center is used to schedule the task execution, and the comprehensive priority calculation is introduced to obtain the scheduling comprehensive priority; S25. According to the comprehensive scheduling priority, design the exception handling and fault tolerance mechanism, obtain the scheduling rules and strategies, and implement the automatic retry mechanism of the task. When the task fails to execute, the task is automatically rescheduled according to the preset number of retries and delay time interval; In S24, the process of calculating the comprehensive priority is as follows: S241. Using the normalization method, the urgency of the task is expressed as the relative value of the difference between the current time and the deadline, and the task urgency priority P is calculated. u,i ; Task urgency priority u,i It is expressed as: Among them, D i represents the deadline of the current task i, T represents the current time, and D j Indicates the deadline of any task; S242. By comparing the task resource requirement with the maximum resource requirement in the task set, the resource requirement priority P is calculated. r,i ; Resource requirement priority P r,i It is expressed as: Among them, R i represents the amount of resources required for the current task i, R j Indicates the amount of resources required for any task; S243. Use the in-degree and out-degree of the task as the measure of the task dependency priority to calculate the task dependency priority P d,i ; The dependency priority of the task P d,i It is expressed as: Among them, De i Denotes the dependency factor corresponding to the current task i, De j Represents the dependency factor corresponding to any task; S244. Calculate the comprehensive priority P t,i ; P t,i =a·P u,i +b·P r,i +c·P d,i Among them, a represents the first weight coefficient, which is used to adjust the proportion of urgency in the priority calculation, b represents the first weight coefficient, which is used to adjust the proportion of resource requirements in the priority calculation, c represents the first weight coefficient, which is used to adjust the proportion of task dependencies in the priority calculation, and a+b+c=1.

3. A real-time stream computing and service scheduling method based on a distributed architecture platform according to claim 2, characterized in that: The S3 specifically includes the following steps: S31. Configure optional stream computing engines for the DolphinDB database, including the responsive state engine, cross-section engine, and time series engine, and integrate the StreamEngineParser component into the DolphinDB database environment by configuring a JSON file; S32. Define the stream computing parameters including input table, output table and custom algorithm in DolphinDB database; S33. Write a script in the DolphinDB database that subscribes to the input table, calls the custom algorithm, and publishes the results to the output table; S34. Build a pipeline through the StreamEngineParser component to convert the user-defined algorithm into efficient computing instructions that can be directly understood and executed by the underlying hardware. The computing instructions include data reading instructions, data processing instructions, and data output instructions. The computing instructions are optimized and the optimized computing instructions are executed to obtain the computing results. S35. Store the calculation results in the DolphinDB database in real time, or send the results to a specified output destination according to business needs; In S32, an input table is first defined, the input table includes an indicator name, an indicator value and a timestamp, and then an output table is defined, the output table includes a calculated predicted value and a corresponding timestamp, and finally a custom algorithm is defined in which the user sets the algorithm logic according to business needs.

4. A real-time stream computing and service scheduling system based on a distributed architecture platform, characterized in that: A method for implementing the real-time stream computing and service scheduling method based on a distributed architecture platform as described in any one of claims 1 to 3, comprising a custom task scheduling center (1), a custom stream computing processing center (2), a distributed storage center (3) and a data acquisition center (4); The custom task scheduling center (1) is connected to the custom stream computing processing center (2) and the distributed storage center (3) respectively, and the custom stream computing processing center (2) is connected to the distributed storage center (3) and the data collection center (4) respectively.

Images