Link transmission load adjustment method and system for station stream batch integrated data in data

Abstract

The invention provides a link transmission load adjustment method and system for station stream batch integrated data in data. The method comprises the following steps: acquiring various stream batch data which arrives at a data center within a period of time and requests link transmission; based on each type of flow batch data, calculating the dynamic priority of the flow batch data according to the initial priority, the task waiting degree and the emergency degree of the flow batch data; adding the flow batch data into a task waiting queue of the corresponding link according to the dynamic priority of the flow batch data, the load of each link and the task response condition; on the basis of each link, a dynamic double-layer scheduling mechanism is adopted, and load dynamic adjustment is conducted on task waiting queues of the links; and transmitting the stream batch data in the adjusted task waiting queue by using a link. The task execution priority can be determined when the link load and the task time are different, so that the overall transmission efficiency is improved, and the requirements of response capability and decision support of a power system are met.

Description

Technical Field

[0001] This invention relates to the fields of smart grid and big data platform and computer network engineering technology, specifically to a method and system for adjusting the load of data link transmission in a data middle platform that integrates batch and stream processing. Background Technology

[0002] In the context of the increasingly complex power industry, multiple challenges arise, including resource allocation, demand management, and data processing. With the continuous development and application of smart grid technology, the role of the data platform is becoming increasingly important. A data platform is an integrated data management platform designed to integrate data from different business systems, achieving efficient data sharing and utilization. Its functions include improving data governance capabilities, supporting real-time monitoring and intelligent analysis, thereby optimizing power operations, enhancing decision support, and driving the digital transformation of the power industry.

[0003] Current technical solutions include traditional architectures that separate data stream and batch processing, and unified stream and batch scheduling frameworks and technologies based on the Lambda architecture. In the traditional architecture, streaming and batch data are processed independently, leading to uneven resource allocation and impacting data processing efficiency. The Lambda architecture maintains two separate systems for streaming and batch processing, which run independently and may result in redundant computations, further reducing data processing efficiency.

[0004] In summary, current technical solutions suffer from low processing efficiency when scheduling integrated batch and stream data, which affects the responsiveness and decision support of the power system. Summary of the Invention

[0005] To overcome the shortcomings of traditional solutions in terms of low processing efficiency when scheduling integrated streaming and batch data, which affects the responsiveness and decision support of the power system, this invention provides a method for adjusting the link transmission load of integrated streaming and batch data in a data middleware platform, comprising:

[0006] Acquire multiple batches of data that arrive at the data center within a certain period and request link transmission;

[0007] Based on each type of batch data, the dynamic priority of the batch data is calculated according to its initial priority, task waiting level, and urgency; and the batch data is added to the task waiting queue of the corresponding link according to its dynamic priority, the load of each link, and the task response status.

[0008] Based on each of the links, a dynamic two-layer scheduling mechanism is adopted to dynamically adjust the load of the task waiting queue of the link; and the link is used to transmit the batch data in the adjusted task waiting queue.

[0009] Optionally, calculating the dynamic priority of the batch data based on its initial priority, task waiting level, and urgency includes:

[0010] Based on the waiting time and execution time of the batch data at the current moment, and combined with the starvation protection mechanism, the task waiting degree of the batch data is calculated;

[0011] The urgency of the batch data is calculated based on the distance from the current time to the deadline and the execution time of the batch data.

[0012] The initial priority, task waiting level, and urgency level of the batch data are weighted and summed to calculate the dynamic priority of the batch data at the current time.

[0013] Optionally, the task waiting level of the batch data satisfies the following formula:

[0014] Where β is the scaling factor for the starvation protection mechanism, W i (t) is the waiting time of the batch data at the current moment, B i This refers to the execution time of the batch data.

[0015] The urgency of the batch data satisfies the following formula:

[0016] The median value of urgency D i -t represents the time from the current time t to the deadline D of the batch data. i The distance, B i This refers to the execution time of the batch data.

[0017] The dynamic priority of the batch data at the current moment satisfies the following formula:

[0018] Where P i (t) represents the dynamic priority of the batch data at the current time t, P i (0) is the initial priority of the batch data, and a is the weight of the initial priority. is the task waiting level of the batch data, and b is the weight of the task waiting level. is the urgency level of the batch data, and c is the weight of the urgency level.

[0019] Optionally, adding the batch data to the task waiting queue of the corresponding link based on the dynamic priority of the batch data, the load of each link, and the task response status includes:

[0020] Based on the dynamic priority of the batch data, the load and task response status of each link, and the link load balancing strategy that minimizes the sum of transmission load and expected completion time, a task allocation link is selected in each link.

[0021] Add the batch data to the task waiting queue of the task allocation link;

[0022] The link load balancing strategy satisfies the following formula:

[0023] The selected link satisfies the transmission load L. j and the estimated completion time T j The sum is minimized, and j represents link j.

[0024] Optionally, the dynamic two-layer scheduling mechanism for dynamically adjusting the load of the task waiting queue of the link includes:

[0025] Obtain the link utilization and remaining bandwidth of the link;

[0026] Based on the link utilization and dynamic two-level scheduling mechanism, the optimal adjustment strategy for the link is determined.

[0027] If the optimal adjustment strategy for the link is the waiting queue adjustment strategy, then based on the link quality factor and remaining bandwidth of the link, the task allocation ratio for the link is adjusted using the task queue allocation rules.

[0028] If the optimal adjustment strategy for the link is a link bandwidth allocation strategy, then based on the link quality factor and total bandwidth resources, the bandwidth allocation ratio for the link is adjusted using bandwidth allocation rules.

[0029] Optionally, the task allocation ratio satisfies the following formula:

[0030] Where M j W represents the task allocation ratio of link j. j W n The weights of links j and n, C j C n Q represents the remaining bandwidth of links j and n. j Q n is the link quality factor for links j and n, and N is the total number of links;

[0031] The bandwidth allocation ratio satisfies the following formula:

[0032] Among them B j B represents the bandwidth allocation ratio for link j.total This represents the total bandwidth resources.

[0033] Optionally, the initial priority of the batch data is determined based on multiple dimensions, including the urgency, data type, and real-time nature of the batch data.

[0034] Optionally, after acquiring the various batches of data that arrive at the data center within a certain period of time and request link transmission, the method further includes:

[0035] If there is an urgent batch of data among the various batches of data, then the urgent batch of data will be allocated to an idle link for transmission.

[0036] On the other hand, the present invention also provides a link transmission load adjustment system for integrated batch and stream data in a data middle platform, comprising:

[0037] The batch data acquisition module is used to acquire various batch data that arrive at the data center within a certain period of time and request link transmission.

[0038] The dynamic priority adjustment module is used to calculate the dynamic priority of each batch of data based on its initial priority, task waiting level, and urgency; and to add the batch of data to the task waiting queue of the corresponding link based on its dynamic priority, the load of each link, and the task response status.

[0039] The two-layer scheduling module is used to dynamically adjust the load of the task waiting queue of each link using a dynamic two-layer scheduling mechanism; and to use the links to transmit batch data in the adjusted task waiting queue.

[0040] Optional, dynamic priority adjustment module, specifically used for:

[0041] Based on the waiting time and execution time of the batch data at the current moment, and combined with the starvation protection mechanism, the task waiting degree of the batch data is calculated;

[0042] The urgency of the batch data is calculated based on the distance from the current time to the deadline and the execution time of the batch data.

[0043] The initial priority, task waiting level, and urgency level of the batch data are weighted and summed to calculate the dynamic priority of the batch data at the current time.

[0044] Optionally, the task waiting level of the batch data satisfies the following formula:

[0045] Where β is the scaling factor for the starvation protection mechanism, W i(t) is the waiting time of the batch data at the current moment, B i This refers to the execution time of the batch data.

[0046] The urgency of the batch data satisfies the following formula:

[0047] The median value of urgency D i -t represents the time from the current time t to the deadline D of the batch data. i The distance, B i This refers to the execution time of the batch data.

[0048] The dynamic priority of the batch data at the current moment satisfies the following formula:

[0049] Where P i (t) represents the dynamic priority of the batch data at the current time t, P i (0) is the initial priority of the batch data, and a is the weight of the initial priority. is the task waiting level of the batch data, and b is the weight of the task waiting level. is the urgency level of the batch data, and c is the weight of the urgency level.

[0050] Optional, dynamic priority adjustment module, specifically used for:

[0051] Based on the dynamic priority of the batch data, the load and task response status of each link, and the link load balancing strategy that minimizes the sum of transmission load and expected completion time, a task allocation link is selected in each link.

[0052] Add the batch data to the task waiting queue of the task allocation link;

[0053] The link load balancing strategy satisfies the following formula:

[0054] The selected link satisfies the transmission load L. j and the estimated completion time T j The sum is minimized, and j represents link j.

[0055] Optional, a two-layer scheduling module, specifically used for:

[0056] Obtain the link utilization and remaining bandwidth of the link;

[0057] Based on the link utilization and dynamic two-level scheduling mechanism, the optimal adjustment strategy for the link is determined.

[0058] If the optimal adjustment strategy for the link is the waiting queue adjustment strategy, then based on the link quality factor and remaining bandwidth of the link, the task allocation ratio for the link is adjusted using the task queue allocation rules.

[0059] If the optimal adjustment strategy for the link is a link bandwidth allocation strategy, then based on the link quality factor and total bandwidth resources, the bandwidth allocation ratio for the link is adjusted using bandwidth allocation rules.

[0060] Optionally, the task allocation ratio satisfies the following formula:

[0061] Where M j W represents the task allocation ratio of link j. j W n The weights of links j and n, C j C n Q represents the remaining bandwidth of links j and n. j Q n is the link quality factor for links j and n, and N is the total number of links;

[0062] The bandwidth allocation ratio satisfies the following formula:

[0063] Among them B j B represents the bandwidth allocation ratio for link j. total This represents the total bandwidth resources.

[0064] Optionally, the initial priority of the batch data is determined based on multiple dimensions, including the urgency, data type, and real-time nature of the batch data.

[0065] Optional, also includes:

[0066] An emergency scheduling module is used to allocate the emergency batch data to an idle link for transmission if there is emergency batch data among the multiple batch data streams.

[0067] On the other hand, the present invention also provides a computer device, characterized in that it includes: one or more processors;

[0068] The processor is used to store one or more programs;

[0069] When the one or more programs are executed by the one or more processors, the link transmission load adjustment method for integrated data streaming and batch processing in the data middle platform described in any one of the above-mentioned methods is implemented.

[0070] On the other hand, the present invention also provides a computer-readable storage medium, characterized in that it stores a computer program thereon, wherein when the computer program is executed, it implements the link transmission load adjustment method for integrated data streaming and batch processing in the data middle platform as described in any one of the above.

[0071] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0072] This invention provides a method and system for adjusting the load of integrated batch and stream data transmission links in a data middleware platform. The method involves acquiring various batch and stream data arriving at the data center within a certain period and requesting link transmission; calculating the dynamic priority of each batch and stream data based on its initial priority, task waiting level, and urgency; adding the batch and stream data to the corresponding link's task waiting queue based on the dynamic priority, the load of each link, and the task response status; dynamically adjusting the load of the link's task waiting queue using a dynamic two-layer scheduling mechanism for each link; and transmitting the batch and stream data in the adjusted task waiting queue using the links. This invention utilizes the initial priority, task waiting level, and urgency of the batch and stream data to dynamically adjust priorities, combined with a dynamic two-layer scheduling mechanism. This ensures that the priority order of task execution is determined when different conditions arise in link load and task time, thereby improving overall transmission efficiency and meeting the responsiveness and decision support requirements of power systems. Attached Figure Description

[0073] Figure 1 This is a schematic diagram of the link transmission load adjustment method for integrated batch and stream data in the data middle platform according to the present invention.

[0074] Figure 2 This is a schematic diagram of the overall structure of the model based on the dynamic priority adjustment scheduling algorithm of the present invention;

[0075] Figure 3 This is a flowchart of the dynamic adjustment strategy for the two-layer link waiting queue of the present invention;

[0076] Figure 4 A comparison chart of task set transmission completion time and deadline scheduling success rate of different algorithms of the present invention;

[0077] Figure 5 A comparison chart showing the latency required for scheduling transmission under the same transmission task set using different algorithms of the present invention;

[0078] Figure 6 This is a comparison chart of the average latency and jitter of high-priority tasks using different algorithms of this invention;

[0079] Figure 7 This is a schematic diagram of the data middle platform integrated batch and stream data link transmission load adjustment system of the present invention;

[0080] Figure 8 This is a schematic diagram of the electronic device of the present invention. Detailed Implementation

[0081] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0082] Example 1:

[0083] This invention provides a method for adjusting the link transmission load of integrated batch and stream data in a data middleware platform, such as... Figure 1 As shown, the flowchart includes:

[0084] Step 101: Obtain various batches of data that arrive at the data center within a certain period of time and request link transmission;

[0085] Step 102: Based on each type of batch data, calculate the dynamic priority of the batch data according to its initial priority, task waiting level, and urgency; add the batch data to the task waiting queue of the corresponding link according to its dynamic priority, the load of each link, and the task response status.

[0086] Step 103: Based on each link, a dynamic two-layer scheduling mechanism is adopted to dynamically adjust the load of the task waiting queue of that link; using that link, the batch data in the adjusted task waiting queue is transmitted.

[0087] In this embodiment of the invention, the initial priority of the batch data, the waiting level of the task, and the urgency level are used to dynamically adjust the priority. Combined with a dynamic two-layer scheduling mechanism, the priority order of task execution can be determined when different conditions occur in link load and task time, thereby improving the overall transmission efficiency and meeting the needs of power system response capability and decision support.

[0088] The various batch data obtained in step 101 above can be batch data received by the data center within a certain period of time, or it can be several (e.g., but not limited to 10,000) batch data generated based on the characteristics of batch sample transmission data (optionally provided by the State Grid Big Data Center) to simulate various batch data that arrive at the data platform and request link transmission data within a certain period of time.

[0089] Streaming and batch data processing encompasses both high-frequency streaming data and traditional batch data. Power data includes real-time streaming data and offline batch data, with increasingly different processing requirements. Streaming data demands immediate transmission and processing, while batch data has higher throughput requirements and needs to be processed on a large scale during off-peak periods. The mixed processing of streaming and batch data requires an efficient scheduling mechanism to meet timeliness requirements while avoiding system resource waste or overload. One possible scenario involves the current need for real-time processing of massive amounts of data. In critical tasks such as power dispatching, monitoring, and fault diagnosis, it is necessary not only to process high-frequency streaming data (such as real-time power load and voltage data) but also to combine it with traditional batch data (such as historical operating records and reports) for offline analysis and decision support.

[0090] In step 102 above, the initial priority of the batch data is determined based on multiple dimensions, including the urgency, data type, and real-time nature of the batch data. The initial priority is the original priority of the batch data, such as the highest priority, high priority, medium-high priority, medium priority, and low priority, as shown in Table 1.

[0091] Table 1

[0092] Priority level describe Initial priority value Level 1 (Priority 1) highest priority 9 to 10 Level 2 (Priority 2) High priority 7 to 8 Level 3 (Priority 3) Medium to high priority 5 to 6 Level 4 (Priority 4) Medium priority 3 to 4 Level 5 (Priority 5) low priority 0 to 2

[0093] The following is an example of a five-priority classification based on the above criteria:

[0094] Priority 1 (highest priority), type: streaming data, such as real-time status of the power grid, fault detection, status of protection devices, etc.; urgency: highest, requiring immediate response; real-time: millisecond level.

[0095] Priority 2 (high priority), type: batch data, such as load forecasts and power generation plans for a future period of time; urgency: high, affecting grid dispatch and requiring rapid response; real-time: minute level.

[0096] Priority 3 (Medium-High Priority): Equipment status monitoring data. Type: Batch data, such as the operating status and maintenance information of equipment like transformers and lines; Urgency: Medium, involving equipment health and requiring timely response; Real-time: Hourly level.

[0097] Priority 4 (Medium Priority): Historical operational data. Type: Batch data, including historical load, power consumption, fault records, etc.; Urgency: Low, mainly used for analysis, some delay is acceptable; Real-time: Daily level.

[0098] Priority 5 (Low Priority): Market transaction and reporting data. Type: Batch data, such as electricity market transaction data, statistical reports, etc.; Urgency: Lowest, mainly used for compliance and review, no need for immediate processing, delays can be tolerated; Real-time: Weekly or monthly level.

[0099] The initial priority classification method described above categorizes streaming and batch data based on their timeliness and importance, ensuring that high-priority data is transmitted preferentially along the data link. This helps power system managers prioritize the processing of the most urgent and important data during data processing and response, thereby improving the reliability and efficiency of the power grid. These priorities can be further adjusted according to specific business needs to obtain dynamic priorities for streaming and batch data.

[0100] For example, each row of data in the batch stream includes the following fields: task_id, data number (sequence number); initial_priority, initial priority; arrival_time, arrival time; burst_time, execution time; deadline, deadline; completion_time, completion time; waiting_time, waiting time; turnaround_time, turnaround time (total time from when data enters the waiting queue to when data transmission ends); and priority, dynamic priority.

[0101] This invention proposes a dynamic scheduling strategy (also known as a dynamic scheduling model). In step 102 above, this dynamic scheduling strategy is used to dynamically adjust the priority of tasks (in this case, batch data). During dynamic adjustment, factors such as the initial priority of the task, the task's waiting level (related to waiting time), and its urgency level (indicating the degree of approaching timeout) can be considered.

[0102] In one possible implementation, in step 102 above, when calculating the dynamic priority of the batch data based on its initial priority, task waiting level, and urgency: the task waiting level of the batch data can be calculated based on its current waiting time and execution time, combined with a starvation protection mechanism; the urgency of the batch data can be calculated based on its distance from the current time to the deadline and its execution time; and the dynamic priority of the batch data at the current time is calculated by weighted summation of its initial priority, task waiting level, and urgency.

[0103] For example, the task waiting time for batch data satisfies the following formula:

[0104] Where β is the scaling factor of the starvation protection mechanism, used to adjust the degree to which waiting time affects priority. In other words, by adjusting the weight of the waiting time percentage, it can be dynamically adjusted based on overall performance during the scheduling process. W i (t) is the waiting time of batch data i (or task i) at the current time t, B i This is the execution time (burst_time) of batch data i (or task i), used to normalize the waiting time and avoid excessive impact from the difference in waiting time between long and short tasks. To prevent some tasks from being unable to execute for an extended period, the formula incorporates a percentage of the waiting time. This reflects the degree of waiting time a task has relative to its own execution time. Through this mechanism, tasks with longer waiting times gradually have their priority increased. Multiplying the percentage of waiting time by β affects the overall priority, as W... i As (t) increases, the dynamic priority Pi(t) of the task will also increase, gradually gaining execution opportunities.

[0105] The urgency of batch data satisfies the following formula:

[0106] The median value of urgency D i -t represents the time from the current time t to the deadline D for batch data i (or task i). i The distance, B i It is the execution time of batch data i (or task i); when D i The smaller the -t value, the closer the task is to the deadline, the greater the urgency of the task as it approaches the deadline, and the higher the dynamic priority Pi(t) of the task will be. If the task has already timed out, i.e., U... i (t)<=0, then This will quickly increase the priority, allowing the task to be processed as soon as possible.

[0107] The dynamic priority of the current batch data satisfies the following formula:

[0108] Where P i (t) is the dynamic priority of batch data i (or task i) at time t, P i (0) is the initial priority of batch data i (or task i), representing the basic importance of the task, and a is the weight of the initial priority. is the task waiting level of batch data i (or task i), and b is the weight of the task waiting level. is the urgency of batch data i (or task i), and c is the weight of the urgency. a, b, and c are used to dynamically adjust the proportions of the initial priority, waiting time, and urgency in the priority calculation. Tasks with higher priority updates will have a higher priority weight due to their urgency (i.e., lower Ui(t)).

[0109] Furthermore, in the task allocation in step 102, when adding the batch data to the task waiting queue of the corresponding link based on the dynamic priority of the batch data, the load of each link, and the task response status, the task allocation link can be selected in each link based on the dynamic priority of the batch data, the load of each link, and the task response status, combined with the link load balancing strategy that minimizes the sum of transmission load and expected completion time; and the batch data can be added to the task waiting queue of the task allocation link.

[0110] In this example, tasks are dynamically allocated based on the current load of the link and the estimated completion time. Let L be the current transmission load of link j. j The current time is T. j If task i is assigned to link j, then the link load balancing strategy is satisfied. The link load balancing strategy satisfies the following formula:

[0111] The selected link satisfies the transmission load L. j and the estimated completion time T j The link load balancing strategy selects the link with the minimum sum of the estimated completion times of the current load to allocate tasks, thereby balancing the load on the links and avoiding overload.

[0112] In step 102, a strategy of processing data periodically is adopted. Based on the data type (original priority) of the batch data (e.g., highest priority, high priority, medium-high priority, medium priority, and low priority batch data as shown in Table 1), as well as factors such as data size, data waiting time, and data transmission deadline, the data is further classified into secondary priorities. The resulting priorities are dynamic priorities. Then, based on the dynamic priorities, the data is distributed to the corresponding links (e.g., idle links or links with low load) for data transmission. During implementation, the response speed of tasks about to time out is improved, reducing the number of timeout tasks and increasing the on-time completion rate; a starvation protection mechanism is added for tasks with excessively long waiting times, thereby reducing the average waiting time of data transmission tasks on the data platform links; and task allocation is dynamically adjusted to balance link load and improve overall throughput.

[0113] Step 102 shows the flow of the scheduling algorithm based on dynamic priority adjustment as follows: Figure 2As shown, after the input of streaming data and batch data, an initial priority determination is performed to obtain Q1 to Q5, which correspond to priority 1 to priority 5 in Table 1, respectively. Based on the determined initial priority, DPAS (the data platform streaming-batch integrated link optimization transmission scheduling strategy proposed in this embodiment of the invention) priority scheduling is performed, outputting dynamic priorities and allocating tasks. In one possible scenario, if there is urgent streaming-batch data among multiple streaming-batch data, the urgent streaming-batch data can also be allocated to idle links for transmission.

[0114] In this step, initial priorities are assigned to batch data, and then the priority and scheduling strategy are dynamically adjusted based on factors such as task type, load, and urgency. Its flexibility lies in its ability to comprehensively consider various tasks and determine which tasks to prioritize under conditions of uneven system load and varying task times, thereby improving the overall transmission efficiency of the system. Specifically, the initial priority division method for tasks based on the characteristics of State Grid data, the dynamic priority adjustment strategy including a starvation protection mechanism, and a multi-factor fusion-based efficient batch-stream hybrid scheduling (dynamic adjustment of proportional coefficients) allows the system to flexibly adjust the parameters in the priority calculation formula according to the characteristics of the scheduled tasks and the current link status, further improving the efficiency and robustness of the scheduling algorithm. The goal of dynamic weighted priority adjustment scheduling is to ensure that, during actual operation, task priorities can be flexibly adjusted according to changes in tasks and the load status of the link. This allows for reasonable bandwidth allocation during peak network load periods or link failures, ensuring that critical tasks are processed in a timely manner. Weighting coefficients are used to dynamically adjust the proportions of initial priority, waiting time, and urgency in the priority calculation, enabling the system to flexibly and efficiently schedule tasks under different loads and scheduling requirements.

[0115] In step 103 above, the task waiting queue adjustment strategy and the link bandwidth allocation strategy are combined to form a dynamic two-layer adjustment mechanism to ensure load balancing and task priority guarantee under different link states. In one implementation, step 103 employs a dynamic two-layer scheduling mechanism. When dynamically adjusting the load on the link's task waiting queue, the link utilization and remaining bandwidth can be obtained. Based on the link utilization and the dynamic two-layer scheduling mechanism, the optimal adjustment strategy for the link is determined. If the optimal adjustment strategy is a waiting queue adjustment strategy, the task allocation ratio is adjusted based on the link quality factor and remaining bandwidth using task queue allocation rules. If the optimal adjustment strategy is a link bandwidth allocation strategy, the bandwidth allocation ratio is adjusted based on the link quality factor and total bandwidth resources using bandwidth allocation rules. In this implementation, at the data platform center, to better achieve load balancing of data transmission links, the dynamic adjustment strategy of the data platform's link transmission waiting queue can be combined to further optimize bandwidth allocation and task scheduling. This strategy ensures a balance between link resource allocation and task demand by dynamically adjusting the task waiting queue length, bandwidth allocation weight, and task priority, thereby maximizing overall transmission efficiency and improving link resource utilization.

[0116] In the two-layer scheduling design, the first layer involves dynamic adjustment of the task waiting queue; prioritizing dynamic adjustment of task allocation ratios based on link load and status to meet load balancing and priority requirements. The second layer involves dynamic allocation of link bandwidth; when task adjustments cannot resolve bottlenecks, more bandwidth is allocated to critical links to optimize performance. Combined with the data platform's dynamic adjustment strategy for link transmission waiting queues, bandwidth allocation and task scheduling are further optimized. By dynamically adjusting the task waiting queue length, bandwidth allocation weight, and task priority, a balance between link resource allocation and task demands is ensured, thereby maximizing data transmission link load balancing and overall transmission efficiency.

[0117] In the first layer, the task waiting queue is dynamically adjusted using a (task) waiting queue adjustment strategy. The task queue allocation rules employed can dynamically adjust the task allocation ratio based on the link status and remaining bandwidth. The triggering conditions for prioritizing task waiting queue adjustment include: a. Link utilization (LF). i <0.8 (Link utilization is not close to saturation). Here, 0.8 is just an example. b. Link quality factor Q characterizing the link status. i Q avg (Link quality factor is higher than average Q) avg During adjustments, high-priority tasks can be prioritized and assigned to higher-priority tasks. i On the link; for low-priority tasks, they can be assigned to links with lower load to avoid preempting high-performance link resources.

[0118] In the second layer, dynamic link bandwidth allocation employs a link bandwidth allocation strategy. The bandwidth allocation rules used can dynamically adjust the bandwidth allocation ratio based on the link's status and total bandwidth resources. Triggering conditions for prioritizing link bandwidth adjustment include: a. Link congestion: the link utilization rate (LF) is [not specified]. i >0.8, where 0.8 is just an example; b. Performance degradation: Link latency of a certain link. (Link latency exceeds the maximum link latency limit) or the link's transmission load (Transmission load exceeds the link's maximum load limit), c. Priority conflict: High-priority tasks cannot meet real-time requirements. During adjustments: a. Bandwidth resource transfer: Dynamically reduce bandwidth allocation on low-priority links and allocate it to links handling high-priority tasks. b. Bandwidth reservation mechanism: Reserve 20%-30% of total bandwidth for the highest priority tasks.

[0119] When merging the first and second layer scheduling strategies, the optimal adjustment strategy can be selected in real time based on the link status as the trigger logic: a. Prioritize task queue adjustment: When link resources are not exhausted and the load is relatively balanced, adjust the task allocation ratio. b. Adjust link bandwidth: When some links have excessively high utilization or experience performance bottlenecks, adjust the link bandwidth allocation ratio. For example, if the link utilization is LF... i >0.8: Prioritize adjusting link bandwidth to be greater than; if LF i ≤0.8: Prioritize adjusting the task waiting queue.

[0120] When employing a two-level cooperative scheduling approach, the following steps may be included:

[0121] Step 1: Monitor link status: Obtain link utilization (LF) in real time. i D i Current transmission load L i Remaining bandwidth of the link C i .

[0122] Step 2: Determine the adjustment strategy:

[0123] If LF i >0.8: Prioritize adjusting link bandwidth.

[0124] If LF i ≤0.8: Prioritize adjusting the task waiting queue.

[0125] Step 3: Dynamically allocate resources: Adjust the task allocation ratio P according to the formula. i and bandwidth allocation ratio B i .

[0126] For example, the task allocation ratio satisfies the following formula:

[0127] Where M j W represents the task allocation ratio of link j. j W n The weights of links j and n, C j C n Q represents the remaining bandwidth of links j and n. j Q n is the link quality factor for links j and n, and N is the total number of links;

[0128] The bandwidth allocation ratio satisfies the following formula:

[0129] Among them B j B represents the bandwidth allocation ratio for link j. total This represents the total bandwidth resources.

[0130] The overall flowchart of the two-layer link adjustment and allocation in step 103 is as follows: Figure 3 As shown, for a certain link, if the task data queue waiting for transmission in the sprocket is not empty, the task is allocated to the link. After allocation, it is determined whether the link utilization is close to saturation. If it is close to saturation, the bandwidth is adjusted using the link bandwidth allocation strategy before transmission. If it is not close to saturation, the task waiting queue adjustment strategy is used before transmission. Assuming that the data transmission in the task data queue is completed, the process ends. Assuming that the data in the task data queue is not completed, the link allocation is redistributed until the task data queue is empty.

[0131] The goal of dynamically adjusting the batch data waiting queue in this step is to ensure that tasks on the link are evenly distributed under fluctuating load conditions, avoiding excessive backlog that could lead to task delays and ensuring efficient system operation. Based on factors such as current link load, task priority, and queue length, queue management is optimized through the following scheduling strategies: The first layer implements dynamic adjustment of the task waiting queue, prioritizing dynamic adjustments to the task allocation ratio based on link load and status to meet load balancing and priority requirements. The second layer implements dynamic allocation of link bandwidth; when task adjustments cannot resolve bottlenecks, more bandwidth is allocated to critical links to achieve performance optimization.

[0132] In this embodiment of the invention, the above-described batch and stream data transmission experiments were used. Different algorithms were employed under the same data platform link configuration for integrated batch and stream task transmission scheduling. A dynamic adjustment strategy for the link transmission waiting queue was also implemented to compare the performance of different algorithms in integrated batch and stream task transmission scheduling. The scheduling algorithms involved in the experiment included Shortest Job First (SJF), Shortest Remaining Time (SRTF), First Come, First Served (FCFS), Highest Response Ratio First (HRRN), and the data platform integrated batch and stream link optimized transmission scheduling strategy (DPAS) proposed in this project.

[0133] according to Figure 4 The comparison results of task set transmission completion time and deadline scheduling success rate of different algorithms shown demonstrate that the DPAS algorithm proposed in this embodiment of the invention achieves a better deadline scheduling success rate than SJF, SRTF, FCFS, and HRRN. According to... Figure 5 The comparison of the latency required for scheduling transmission under the same transmission task set by different algorithms shows that the total latency of the DPAS algorithm proposed in this embodiment of the invention is lower than that of SJF, SRTF, FCFS, and HRRN. According to... Figure 6 The comparison of average delay and jitter for high-priority tasks shown in the examples demonstrates that the DPAS algorithm proposed in this embodiment of the invention achieves lower execution time and better performance for high-priority tasks compared to SJF, SRTF, FCFS, and HRRN. Furthermore... Figure 4 The transmission completion time shown is the average latency. Combined with the average latency index, it can be seen that the DPAS algorithm can not only respond to high-priority tasks in a timely manner, but also take into account medium and low-priority tasks and tasks with excessively long waiting times.

[0134] This invention proposes a dynamic scheduling and transmission optimization process for streaming and batch data, aiming to solve problems such as uneven link load, improper task prioritization, and transmission delays caused by link congestion and failures in existing data transmission systems. Comparison results of the algorithms show that, through strategies such as priority partitioning, dynamic weight adjustment, and link load balancing, this invention can effectively optimize the transmission scheduling of streaming and batch data, ensuring the real-time requirements of high-priority tasks and balancing the load of each link, thereby improving the utilization efficiency of network resources. Simultaneously, it can automatically adjust task allocation when link failures or congestion occur, ensuring stable system operation and high reliability. This method is of great significance for fields such as smart grids and big data platforms, significantly improving the efficiency, stability, and intelligence of data transmission, and providing a practical solution for efficient transmission and real-time processing in complex data environments.

[0135] The following comparison of the State Grid data platform stream-batch integrated optimized link transmission strategy proposed in this embodiment with the retrieved traditional technologies illustrates the effectiveness of this embodiment.

[0136] Several technical solutions have attempted to combine streaming data and batch data processing, and these solutions can be broadly categorized as follows:

[0137] Traditional architectures separating data streaming and batch processing: Traditional power system data processing architectures typically separate data streaming and batch processing. Streaming data requires immediate response and usually prioritizes link resources, but peak loads can lead to link congestion. Batch data is usually transmitted during low-load periods, but due to the isolation of link and resource allocation, batch processing tasks fail to fully utilize idle link resources. This separation leads to uneven resource allocation, especially when streaming and batch data need to be processed simultaneously in the same queue, easily causing transmission bottlenecks or load imbalances. In the independent processing mode of streaming and batch data, resources may be over-allocated or wasted, failing to fully utilize warehouse and computing resources. Furthermore, processing streaming and batch data on the same shelf can lead to shelf imbalance, particularly since the high frequency of streaming load data can affect the throughput efficiency of batch data. High-priority streaming tasks may crowd out batch data resources, causing batch task delays. Link resources are not dynamically shared, resulting in resource waste and uneven distribution.

[0138] Integrated batch and stream processing scheduling frameworks (such as the Lambda architecture): The Lambda architecture is a solution that combines batch processing and stream processing, primarily used for large-scale data analysis scenarios. In power systems, the Lambda architecture combines streaming data processing with batch data processing, providing low-latency real-time data processing and high-throughput batch data processing. The Lambda architecture divides data processing into three layers: the Batch Layer, the Speed ​​Layer, and the Serving Layer. The disadvantage is that the Lambda architecture requires maintaining two separate systems for stream processing and batch processing, calculating and storing the results of the two types of tasks separately, increasing architectural complexity and making maintenance and expansion more difficult. Furthermore, because the stream processing and batch processing layers operate independently, issues such as the same data potentially being read and processed repeatedly by both layers increase computational and storage overhead, leading to resource waste and low data processing efficiency.

[0139] Unified Streaming and Batch Scheduling and Dynamic Resource Adjustment Technology: Some emerging technology frameworks (such as Apache Pulsar and Apache Kafka Streams) have proposed solutions to optimize data transmission queues and resource allocation through the integrated scheduling of streaming and batch data. These solutions typically achieve unified streaming and batch scheduling in the following ways: Task priority allocation: Dynamically adjust the bandwidth allocation of streaming and batch data based on the real-time requirements and importance of tasks; Intelligent resource scheduling: Combine queue load monitoring and task scheduling strategies to dynamically adjust the allocation ratio of streaming and batch tasks based on the current system load, avoiding queue congestion or resource waste. The disadvantages are that while dynamic scheduling can address resource pressure to a certain extent when the load queue is large, it may still fail to avoid load imbalances or latency issues in complex task queues and scenarios with multiple tasks. Furthermore, the technology is very difficult to implement, requiring real-time monitoring of the status of each instruction and task, involving the implementation of complex algorithms and scheduling strategies, system design, and maintenance requirements. Current systems often assume that links are always available, with insufficient consideration for link failures or congestion. Tasks on failed links cannot be migrated to other links in a timely manner, affecting the reliability of data transmission. Task backlog or loss prevents the fulfillment of real-time requirements; low-priority tasks are not effectively allocated to certain idle links. Existing scheduling schemes fail to dynamically detect differences in link resource utilization, leading to resource idleness. Idle links are not fully utilized, while other links may be overloaded, resulting in a decrease in overall system throughput.

[0140] Existing technical solutions, especially the Lambda architecture and the integrated streaming and batch scheduling framework, have made some progress in combining streaming and batch data, but they still face problems such as uneven resource allocation, high system complexity, and task priority. To solve these problems, the new scheduling scheme needs to more accurately control the transmission priority of streaming and batch data, and dynamically adjust bandwidth and resource allocation based on the queue load status to achieve load balancing. This is the technical innovation of this invention.

[0141] In summary, existing technical solutions have the following drawbacks. Therefore, this invention proposes a dynamic priority scheduling strategy and a dynamic adjustment of the waiting queue to efficiently achieve collaborative transmission of streaming and batch data.

[0142] Disadvantage 1: Uneven resource utilization. Separate streaming and batch processing architecture: Separating streaming and batch data processing systems leads to ineffective resource sharing. Streaming data may consume excessive bandwidth under high-priority conditions, while batch data fails to fully utilize bandwidth resources when the link is idle. Lambda architecture: Streaming and batch processing systems operate independently, resulting in redundant computation and storage, leading to resource waste. To address Disadvantage 1, this invention employs a dynamic task scheduling mechanism, combining the transmission requirements of streaming and batch data, to achieve resource sharing and rational allocation. Specifically, low-priority tasks are dynamically allocated when the link is idle to avoid resource idleness. This resource sharing and dynamic adjustment improves resource utilization, thus solving the problem of insufficient resource sharing in the separate streaming and batch data processing architecture.

[0143] Disadvantage 2: Uneven link load. Traditional stream-batch separation architecture: Because streaming and batch data are processed on different links, the load distribution is uneven, which may lead to some links being overloaded while others are idle, resulting in resource waste or link congestion. Integrated stream-batch scheduling framework: The load balancing mechanism is not flexible enough and fails to respond to link load and task scheduling needs in real time, resulting in low task processing efficiency. To address this disadvantage 2, this embodiment of the invention adopts an intelligent load balancing algorithm to dynamically adjust task allocation and bandwidth allocation according to the real-time status of the links. Specifically, it includes: dynamically adjusting task priority and bandwidth allocation according to the link health status and bandwidth utilization; automatically migrating tasks to lighter links when a link is overloaded to ensure overall system load balance. Optimizing link load balancing improves system throughput, thereby solving the link load imbalance problem and improving overall transmission efficiency.

[0144] Disadvantage 3: Insufficient handling of link failures and congestion. Existing solutions fail to adequately consider link failures or congestion, leading to task interruptions or delays and reduced system stability and reliability when link failures occur. The lack of link health monitoring and task migration mechanisms means that if a link malfunctions, a rapid response is impossible, resulting in task backlog or loss. To address this disadvantage 3, this invention introduces a link health monitoring mechanism. Upon detecting a link failure or congestion, the task migration mechanism is immediately triggered, reallocating tasks to other healthy links. This increases system fault tolerance, minimizing the impact of link failures on overall system performance. By improving link reliability and enhancing fault tolerance and recovery capabilities, this invention solves the problem of insufficient handling of link failures and congestion, ensuring the continuity and stability of data transmission.

[0145] Disadvantage 4: High system complexity. Lambda architecture: Requires maintaining two independent systems simultaneously for batch processing and stream processing, resulting in a complex architecture, high development and maintenance costs, and a high technical barrier. Multi-level scheduling framework: Employing multi-level scheduling and resource allocation strategies increases system complexity and management difficulty, making real-time optimization and adjustment of the entire data stream challenging. To address this disadvantage 4, this invention designs a unified scheduling framework that integrates streaming and batch data processing, avoiding the use of complex multi-level architectures. Through simplified scheduling strategies and unified resource management, system simplicity and efficiency are achieved. Simplifying the system architecture and reducing system complexity solves the problem of high system complexity and reduces development and maintenance difficulties.

[0146] Disadvantage 5: Low data processing efficiency. Redundant processing: In the Lambda architecture, the stream processing layer and batch processing layer repeatedly process the same data, leading to a waste of computing and storage resources. Processing latency: Separating streaming and batch data processing can cause increased latency for real-time data under high link load, failing to meet timeliness requirements. To address this disadvantage 5, this embodiment of the invention dynamically adjusts task priority and bandwidth allocation based on system load and task type. For example, during peak link load periods, priority is given to the transmission of critical tasks, while low-priority tasks can be appropriately delayed, ensuring that system resources are prioritized for urgent tasks. Dynamic priority adjustment optimizes task scheduling to solve the problems of fixed priorities and inflexible bandwidth allocation, improving the real-time performance and efficiency of task processing.

[0147] As can be seen, the embodiments of the present invention can achieve the following objectives: reasonable initial priority allocation to ensure priority transmission of critical tasks (such as real-time monitoring and fault alarms); dynamic adjustment of task priorities and weights to ensure that the system can still prioritize urgent tasks and balance system load even under high load and link failures; and dynamic adjustment of link waiting queues to avoid excessive task backlog on a single link, ensuring efficient resource utilization and handling task migration in the event of link congestion or failure. The comprehensive optimization of these aspects can greatly improve the system's transmission efficiency, load balancing, and stability, ultimately enhancing the overall performance of the data platform.

[0148] In one applicable scenario, the State Grid's two-tiered data platform is an integrated data management platform established by the State Grid Corporation of China. It aims to integrate data from different business systems to achieve efficient data sharing and utilization. Its functions include improving data governance capabilities, supporting real-time monitoring and intelligent analysis, thereby optimizing power operations, enhancing decision support, and promoting the digital transformation of the power industry. In the current increasingly complex context of the power industry, the State Grid is facing multiple challenges in resource allocation, demand management, and data processing. With the continuous development and application of smart grid technology, the data platform plays an increasingly important role, becoming the core support for integrated streaming and batch data transmission. With the development of system informatization and electrification, the State Grid's digital construction is gradually transitioning from "data silos" to "big data platforms." In the State Grid's two-tiered data platform, data link transmission faces a series of challenges, including high transmission latency, poor service quality, and difficulty in coordinating the transmission of streaming and batch data on the same link. These problems may lead to a decrease in the efficiency of real-time data processing, affecting the responsiveness and decision support of the power system. High latency may cause delays in real-time monitoring and fault detection, reducing system reliability. Furthermore, the transmission of streaming and batch data on their respective fixed links leads to redundant link configurations, resulting in a waste of transmission resources. These challenges urgently need to be addressed by optimizing network architecture, introducing efficient data transmission protocols, and enhancing data processing capabilities to achieve efficient and stable data transmission and promote the development of smart grids. To solve these problems, this paper proposes a two-level data middleware integrated stream-batch transmission strategy based on dynamic priority. First, the grid stream-batch data is classified and initially prioritized according to its real-time nature, importance, and data type. Second, a dynamic weighted priority scheduling algorithm is designed to ensure that high-priority data transmission tasks can be processed in a timely manner. Simultaneously, by dynamically adjusting the initial priority of tasks, the weights of waiting time and task urgency, the success rate of data middleware stream-batch data transmission within the deadline, the transmission efficiency of links, and the quality of service of link transmission are effectively improved. Furthermore, the proposed dynamic adjustment strategy for the data middleware link transmission waiting queue also ensures load balancing during link transmission.

[0149] Example 2:

[0150] Based on the same inventive concept, this invention also provides a data middleware integrated batch and stream data link transmission load adjustment system, the structural diagram of which is shown below. Figure 7 As shown, it includes:

[0151] The batch data acquisition module is used to acquire various batch data that arrive at the data center within a certain period of time and request link transmission.

[0152] The dynamic priority adjustment module is used to calculate the dynamic priority of each batch of data based on its initial priority, task waiting level, and urgency; and to add the batch of data to the task waiting queue of the corresponding link based on its dynamic priority, the load of each link, and the task response status.

[0153] The two-level scheduling module is used to dynamically adjust the load of the task waiting queue of each link using a dynamic two-level scheduling mechanism; and to use the links to transmit the batch data in the adjusted task waiting queue.

[0154] Optional, dynamic priority adjustment module, specifically used for:

[0155] Based on the current waiting time and execution time of the batch data, and combined with the starvation protection mechanism, calculate the task waiting degree of the batch data;

[0156] Calculate the urgency of the batch data based on the distance between the current time and the deadline and the execution time.

[0157] The initial priority, task waiting level, and urgency of the batch data are weighted and summed to calculate the dynamic priority of the batch data at the current time.

[0158] Optionally, the task waiting level for batch data should satisfy the following formula:

[0159] Where β is the scaling factor for the starvation protection mechanism, W i (t) is the waiting time of the batch data at the current moment, B i It is the execution time of batch data;

[0160] The urgency of batch data satisfies the following formula:

[0161] The median value of urgency D i -t specifies the time from current time t to deadline D for the batch data. i The distance, B i It is the execution time of batch data;

[0162] The dynamic priority of the current batch data satisfies the following formula:

[0163] Where P i (t) represents the dynamic priority of the batch data at the current time t, P i (0) represents the initial priority of the batch data, and a is the weight of the initial priority. 'b' represents the task waiting level for batch data, and 'b' represents the weight of the task waiting level. 'c' represents the urgency level of the batch data, and 'c' represents the weight of the urgency level.

[0164] Optional, dynamic priority adjustment module, specifically used for:

[0165] Based on the dynamic priority of the batch data, the load and task response of each link, and the link load balancing strategy that minimizes the sum of transmission load and expected completion time, the task allocation link is selected in each link.

[0166] Add the batch data to the task waiting queue of the task allocation chain;

[0167] Link load balancing strategies satisfy the following formula:

[0168] The selected link satisfies the transmission load L. j and the estimated completion time T j The sum is minimized, and j represents link j.

[0169] Optional, a two-layer scheduling module, specifically used for:

[0170] Obtain the link utilization and remaining bandwidth of the link;

[0171] Based on link utilization and dynamic two-level scheduling mechanism, the optimal adjustment strategy for links is determined.

[0172] If the optimal adjustment strategy for the link is the waiting queue adjustment strategy, then based on the link quality factor and the remaining bandwidth of the link, the task queue allocation rule is used to adjust the task allocation ratio for the link.

[0173] If the optimal adjustment strategy for the link is the link bandwidth allocation strategy, then based on the link quality factor and total bandwidth resources, the bandwidth allocation ratio for the link is adjusted using the bandwidth allocation rules.

[0174] Optionally, the task allocation ratio satisfies the following formula:

[0175] Where M j W represents the task allocation ratio of link j. j W n The weights of links j and n, C j C n Q represents the remaining bandwidth of links j and n. j Q n is the link quality factor for links j and n, and N is the total number of links;

[0176] The bandwidth allocation ratio satisfies the following formula:

[0177] Among them B j B represents the bandwidth allocation ratio for link j. total This represents the total bandwidth resources.

[0178] Optionally, the initial priority of the batch data is determined based on multiple dimensions, including the urgency, data type, and real-time nature of the batch data.

[0179] Optional, also includes:

[0180] The emergency scheduling module is used to allocate emergency batch data to idle links for transmission if there is emergency batch data among multiple batch data streams.

[0181] Example 3:

[0182] like Figure 8 As shown, the present invention also provides an electronic device, which may be a computer device, a microcontroller device, a smart mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, processor, and transceiver component are connected via a bus; the memory can be used to store executable programs, and an exemplary executable program may include instructions; the processor is used to execute the instructions stored in the memory. The memory can also be used to store data, which can be accessed and / or modified when instructions are executed.

[0183] The processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, and it is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions in the storage medium to realize the corresponding method flow or corresponding function, so as to realize the steps of the link transmission load adjustment method for integrated batch and stream data in the data platform in the above embodiment.

[0184] Example 4:

[0185] Based on the same inventive concept, this invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory). This readable storage medium is a memory device within an electronic device used to store programs and data. It is understood that the storage medium here can include both built-in storage media within the electronic device and extended storage media supported by the electronic device. The storage medium provides storage space, which stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more executable programs (including program code). It should be noted that the storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. Loading and executing one or more instructions stored in the storage medium by the processor can implement the steps of the link transmission load adjustment method for integrated batch and stream data in the above embodiments.

[0186] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0187] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0188] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0189] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the scope of protection of the claims pending approval.

Claims

1. A method for adjusting the link transmission load of integrated batch and stream data in a data middleware platform, characterized in that, include: Acquire multiple batches of data that arrive at the data center within a certain period and request link transmission; Based on each type of batch data, the dynamic priority of the batch data is calculated according to its initial priority, task waiting level, and urgency level. Based on the dynamic priority of the batch data, the load of each link, and the task response status, the batch data is added to the task waiting queue of the corresponding link. Based on each of the links, a dynamic two-layer scheduling mechanism is adopted to dynamically adjust the load of the task waiting queue of the link; and the link is used to transmit the batch data in the adjusted task waiting queue.

2. The method as described in claim 1, characterized in that, The step of calculating the dynamic priority of the batch data based on its initial priority, task waiting level, and urgency level includes: Based on the waiting time and execution time of the batch data at the current moment, and combined with the starvation protection mechanism, the task waiting degree of the batch data is calculated; The urgency of the batch data is calculated based on the distance from the current time to the deadline and the execution time of the batch data. The initial priority, task waiting level, and urgency level of the batch data are weighted and summed to calculate the dynamic priority of the batch data at the current time.

3. The method as described in claim 2, characterized in that, The task waiting level of the batch data satisfies the following formula: Where β is the scaling factor for the starvation protection mechanism, W i (t) is the waiting time of the batch data at the current moment, B i This refers to the execution time of the batch data. The urgency of the batch data satisfies the following formula: The median value of urgency D i -t represents the time from the current time t to the deadline D of the batch data. i The distance, B i This refers to the execution time of the batch data. The dynamic priority of the batch data at the current moment satisfies the following formula: Where P i (t) represents the dynamic priority of the batch data at the current time t, P i (0) is the initial priority of the batch data, and a is the weight of the initial priority. is the task waiting level of the batch data, and b is the weight of the task waiting level. is the urgency level of the batch data, and c is the weight of the urgency level.

4. The method according to any one of claims 1-3, characterized in that, The step of adding the batch data to the task waiting queue of the corresponding link based on the dynamic priority of the batch data, the load of each link, and the task response status includes: Based on the dynamic priority of the batch data, the load and task response status of each link, and the link load balancing strategy that minimizes the sum of transmission load and expected completion time, a task allocation link is selected in each link. Add the batch data to the task waiting queue of the task allocation link; The link load balancing strategy satisfies the following formula: The selected link satisfies the transmission load L. j and the estimated completion time T j The sum is minimized, and j represents link j.

5. The method as described in claim 1, characterized in that, The dynamic two-layer scheduling mechanism for dynamically adjusting the load of the task waiting queue of the link includes: Obtain the link utilization and remaining bandwidth of the link; Based on the link utilization and dynamic two-level scheduling mechanism, the optimal adjustment strategy for the link is determined. If the optimal adjustment strategy for the link is the waiting queue adjustment strategy, then based on the link quality factor and remaining bandwidth of the link, the task allocation ratio for the link is adjusted using the task queue allocation rules. If the optimal adjustment strategy for the link is a link bandwidth allocation strategy, then based on the link quality factor and total bandwidth resources, the bandwidth allocation ratio for the link is adjusted using bandwidth allocation rules.

6. The method as described in claim 5, characterized in that, The task allocation ratio satisfies the following formula: Where M j W represents the task allocation ratio of link j. j W n The weights of links j and n, C j C n Q represents the remaining bandwidth of links j and n. j Q n is the link quality factor for links j and n, and N is the total number of links; The bandwidth allocation ratio satisfies the following formula: Among them B j B represents the bandwidth allocation ratio for link j. total This represents the total bandwidth resources.

7. The method as described in claim 1, characterized in that, The initial priority of the batch data is determined based on multiple dimensions, including the urgency, data type, and real-time nature of the batch data.

8. The method as described in claim 1 or 7, characterized in that, After acquiring various batches of streaming data that arrive at the data center within a certain period and request link transmission, the method further includes: If there is an urgent batch of data among the various batches of data, then the urgent batch of data will be allocated to an idle link for transmission.

9. A data middleware integrated batch and stream data link transmission load adjustment system, characterized in that, include: The batch data acquisition module is used to acquire various batch data that arrive at the data center within a certain period of time and request link transmission. The dynamic priority adjustment module is used to calculate the dynamic priority of each batch of data based on its initial priority, task waiting level, and urgency. Based on the dynamic priority of the batch data, the load of each link, and the task response status, the batch data is added to the task waiting queue of the corresponding link. The two-layer scheduling module is used to dynamically adjust the load of the task waiting queue of each link using a dynamic two-layer scheduling mechanism; and to use the links to transmit batch data in the adjusted task waiting queue.

10. An electronic device, characterized in that, include: At least one processor and memory; The memory and processor are connected via a bus; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the link transmission load adjustment method for integrated data streaming and batch processing in the data middle platform as described in any one of claims 1-8 is implemented.

Images