Batch flow integrated data processing system and method

Abstract

The invention belongs to the technical field of data processing, and particularly relates to a batch flow integrated data processing system and method. The batch flow integrated data processing process is subjected to fine management and dynamic adjustment, the corresponding task execution mode is matched according to the characteristics and requirements of the original data, the resource allocation strategy is further optimized, the high efficiency and real-time performance of data processing are ensured, and the real-time performance of the batch flow integrated data processing process is improved. And meanwhile, the batch processing data and the stream processing data in the task execution process can be intelligently allocated, and the execution resource quota can be adjusted, so that the fluency of the batch processing data and the stream processing data in parallel processing is ensured, and efficient coordination of the batch processing data and the stream processing data in resource utilization is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a batch-stream integrated data processing system and method. Background Art

[0002] With the rapid development of information technology, the demand for data processing has become increasingly complex and diverse, especially in the scenarios of big data and real-time data analysis. Traditional data processing methods often cannot meet the needs of both batch processing and stream processing at the same time. Batch processing focuses on offline processing of large amounts of data and is suitable for scenarios with low timeliness requirements but large data volumes; stream processing, on the other hand, focuses on online processing of real-time data and emphasizes the immediacy and dynamism of data. However, in many practical applications, these two data processing methods often need to work together to achieve comprehensive and efficient data processing.

[0003] In the prior art, batch-stream integrated data processing systems usually adopt independent design or simple integration strategies, which leads to low efficiency of the system when processing complex tasks, uneven resource allocation, and the simultaneous occurrence of some resources being in excess and some resources being insufficient, which will undoubtedly lead to a decline in the overall performance of the system and the inability to give full play to the advantages of data processing. In order to solve this problem, the present invention proposes an innovative batch-stream integrated data processing system and method, which realizes the mutual coordination of batch processing and stream processing through intelligent resource scheduling. Summary of the invention

[0004] The purpose of the present invention is to provide a batch-stream integrated data processing system and method, which can effectively solve the problems of uneven resource allocation and low efficiency when batch processing and stream processing work together in the field of data processing, and realize collaborative data processing through dynamic resource scheduling and task execution mode optimization.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] In a first aspect, a batch-stream integrated data processing method is provided, comprising:

[0007] According to the data volume and output delay requirement of the acquired original data, the original data is classified to obtain batch data and stream data, and task execution modes of the batch data and the stream data are synchronously matched, wherein the task execution modes include separate execution and parallel execution;

[0008] Allocate execution resource quotas to the batch processing data and the stream processing data according to the task execution mode;

[0009] When the separate execution is performed, the batch processing data is added to an independent task queue, and the batch processing data in the independent task queue is processed according to the acquisition order;

[0010] When the parallel execution is performed, the batch processing data and the stream processing data are added to a shared task queue, and the batch processing data and the stream processing data are processed concurrently in the shared task queue;

[0011] During the parallel execution process, the execution resource quota of the batch data and the stream processing data in the shared task queue is adjusted according to the output delay lag and task load of the batch data and the stream processing data, and the scheduling direction of the data to be processed in the shared task queue is adjusted.

[0012] In some embodiments, the step of classifying the original data according to the amount of the original data obtained and the output delay requirement to obtain batch data and stream data includes:

[0013] Acquire the raw data and perform preprocessing, wherein the preprocessing includes: data cleaning, deduplication and standardization; wherein the standardization is to unify the format of the raw data;

[0014] Collecting the data processing volume of the preprocessed raw data, and comparing the data processing volume with a preset data volume classification threshold;

[0015] If the data processing volume is less than or equal to the data volume classification threshold, all pre-processed raw data are output as stream processed data;

[0016] If the data processing volume is greater than the data volume classification threshold, then collecting the output delay requirements of the pre-processed original data, wherein the output delay requirements include an allowable delay requirement and an immediacy requirement;

[0017] The pre-processed original data corresponding to the allowable delay requirement is output as the batch processed data, and the pre-processed original data corresponding to the immediacy requirement is output as the stream processed data.

[0018] In some embodiments, the task execution method of matching the batch processing data and the stream processing data includes:

[0019] When only batch processing data exists in the original data, determining the task execution mode of the original data as the separate execution;

[0020] When the original data contains both batch processing data and stream processing data, or when the original data contains only stream processing data, the task execution mode of the original data is determined to be the parallel execution.

[0021] In some embodiments, the step of allocating execution resource quotas to the batch processing data and the stream processing data according to the task execution mode includes:

[0022] Get the task execution mode under the current node;

[0023] When the task execution mode is the separate execution, an independent execution resource pool is configured for the batch data, wherein the execution resource quota of the batch data is proportional to the total amount of data of the batch data;

[0024] When the task execution mode is the parallel execution, a shared execution resource pool is configured for the batch processing data and the stream processing data, and execution resource quotas are dynamically allocated according to the data volume of the batch processing data and the data demand delay of the stream processing data.

[0025] In some of the embodiments, the independent task queue includes a single thread queue and a parallel thread queue, and the determination process of the single thread queue and the parallel thread queue is as follows;

[0026] Acquire the total amount of data to be processed of the batch data and the unit processing amount of the batch data, and calculate the execution round of the batch data according to the total amount of data to be processed and the unit processing amount;

[0027] The obtained benchmark execution round is combined with the execution round of the batch data, and the combined calculation result of the benchmark execution round and the execution round of the batch data is rounded up to obtain the required thread quantity;

[0028] When the value of the required thread quantity is one, the batch processing data is added to a single thread queue for sequential processing;

[0029] When the value of the required thread quantity is greater than one, the batch processing data is divided into a plurality of parallel thread queues according to the value of the required thread quantity.

[0030] In some embodiments, the step of adjusting the execution resource quota of the batch data and the stream data in the shared task queue according to the output delay lag and task load of the batch data and the stream data during the parallel execution includes:

[0031] Acquire in real time the task load of the batch processing data and the stream processing data, and the output delay lag of the stream processing data;

[0032] Performing backtracking offset processing on the current node to obtain a backtracking period, and counting the output delay lag of the stream processing data in the backtracking period, and calculating the predicted delay lag of the stream processing data based on the output delay lag in the backtracking period;

[0033] If the value of the predicted delay lag is greater than zero, it indicates that the output of the stream processing data is abnormal, and the execution resource quota of the stream processing data is increased;

[0034] If the value of the predicted delay lag is less than or equal to zero, it indicates that the output of the stream processing data is normal, and the remaining resource quota of the stream processing data is synchronously output.

[0035] In some embodiments, the method further comprises:

[0036] When increasing the execution resource quota of the stream processing data, calling the idle resource quota in the shared resource pool;

[0037] When the idle resource quota is zero, the execution round of the batch data is reduced, the to-be-processed data under the batch data after the execution round of the batch data is reduced is added to the waiting queue, and the execution resource quota occupied by the batch data is released to obtain the allowed allocation resource quota;

[0038] reallocating the allowed allocation resource quota to the stream processing data until the output delay lag of the stream processing data is less than or equal to zero, or the execution round of the batch processing data is reduced to one and then stopped;

[0039] After the remaining resource quota is output, it is determined whether there is a waiting queue in the batch data. If there is a waiting queue in the batch data, the remaining resource quota is allocated to the batch data.

[0040] In some embodiments, after the execution round of the batch processing data is reduced to one, the method further comprises:

[0041] If the output delay lag value of the stream processing data is still positive, an alarm signal is synchronously issued, and a summary report including the current delay status of the stream processing data, the resource adjustment measures that have been taken, and the amount of data to be processed in the waiting queue after the batch data releases the execution resource quota is output.

[0042] In a second aspect, a batch-stream integrated data processing system is provided, comprising:

[0043] An initialization module, the initialization module is used to classify the original data according to the data volume and output delay requirements of the acquired original data, obtain batch data and stream data, and synchronously match the task execution mode of the batch data and the stream data, wherein the task execution mode includes separate execution and parallel execution;

[0044] A resource allocation module, the resource allocation module is used to allocate execution resource quotas to the batch processing data and the stream processing data according to the task execution mode;

[0045] A first execution module, wherein the first execution module is used to add the batch processing data to an independent task queue when the single execution is performed, and the batch processing data in the independent task queue is processed according to the acquisition order;

[0046] A second execution module, wherein the second execution module is used for adding the batch processing data and the stream processing data to a shared task queue during the parallel execution, and the batch processing data and the stream processing data are processed concurrently in the shared task queue;

[0047] A dynamic adjustment module is used to adjust the execution resource quota of the batch data and the stream processing data in the shared task queue and adjust the scheduling direction of the data to be processed in the shared task queue according to the output delay lag and task load of the batch data and the stream processing data during the parallel execution process.

[0048] According to a third aspect, an electronic device is provided, the electronic device comprising:

[0049] at least one processor;

[0050] and a memory communicatively coupled to the at least one processor;

[0051] Wherein, the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the batch-stream integrated data processing method described in any one of the first aspects.

[0052] The present invention adopts the above technical solution, and has the following technical effects compared with the prior art:

[0053] The present invention performs fine management and dynamic adjustment on the batch-stream integrated data processing process, matches the corresponding task execution mode according to the characteristics and requirements of the original data, and further optimizes the resource allocation strategy to ensure the high efficiency and real-time performance of data processing. At the same time, it can also intelligently allocate and adjust the execution resource quotas for the batch data and the stream data in the task execution process, ensure the fluency of the batch data and the stream data in parallel processing, and realize the efficient coordination of the batch data and the stream data in resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic flow chart of the method of the present invention;

[0055] Figure 2 It is a schematic diagram of the system module of the present invention;

[0056] Figure 3 It is a structural diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0057] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0058] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0059] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. The phrase "in a preferred embodiment" that appears in different places in this specification does not refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0060] See also Figure 1 As shown, the present invention provides a batch-stream integrated data processing method, comprising:

[0061] S1. According to the amount of raw data obtained and the output delay requirement, the raw data is classified to obtain batch data and stream data, and the task execution mode of the batch data and the stream data is synchronously matched, wherein the task execution mode includes separate execution and parallel execution;

[0062] S2. Allocate execution resource quotas for batch processing data and stream processing data based on the task execution mode;

[0063] S3. When executed separately, the batch data is added to the independent task queue, and the batch data in the independent task queue is processed according to the collection order;

[0064] S4. When executing in parallel, the batch processing data and the stream processing data are added to the shared task queue, and the batch processing data and the stream processing data are processed concurrently in the shared task queue;

[0065] S5. During the parallel execution process, according to the output delay lag and task load of the batch data and the stream processing data, the execution resource quota of the batch data and the stream processing data in the shared task queue is adjusted, and the scheduling direction of the data to be processed in the shared task queue is adjusted.

[0066] As described in the above steps S1-S5, with the rapid development of information technology, data processing systems are facing increasingly complex challenges. In particular, in a big data environment, how to efficiently process batch data and real-time stream data that arrive at the same time has become an urgent problem to be solved. In this embodiment, the original data will first be classified according to the data volume and output delay requirements of the acquired original data, and based on the data volume and output delay requirements of the original data, so as to divide the original data into batch data and stream data, so as to determine the task execution mode of batch data and stream data according to different types of original data. The task execution mode includes separate execution and parallel execution, so as to ensure the reasonable distribution of data processing. Then, according to the task execution mode, execution resource quotas are allocated to batch data and stream data. When executing separately, the batch data will be added to an independent task queue, and the batch data will be processed in batches in the independent task queue according to the collection order. For example, for batch data that needs to be deeply analyzed (such as historical data analysis, etc., its For tasks with large data volumes and low real-time requirements, a separate execution method can be used to ensure the accuracy of data processing and the adequacy of in-depth analysis. During parallel execution, batch data and stream data will be added to the shared task queue together, and batch data and stream data will be processed concurrently in the shared task queue. For example, for stream data that needs to be processed in real time (such as real-time monitoring, real-time analysis, etc., the data volume is relatively small, but the real-time requirements are high), a parallel execution method can be used to ensure the real-time and high efficiency of data processing. Finally, according to the output delay lag and task load of batch data and stream data under parallel execution, the execution resource quota of batch data and stream data in the shared task queue will be adjusted, as well as the scheduling direction of the data to be processed in the shared task queue. The scheduling direction includes: adjusting the execution resources of batch data to cooperate with stream data, or adjusting the execution resource quota of stream data to batch data, so as to optimize the delay and load balancing of data processing and further improve data processing efficiency.

[0067] In a preferred embodiment, the steps of classifying the original data according to the amount of the original data obtained and the output delay requirement to obtain batch data and stream data include:

[0068] S101, obtaining raw data and performing preprocessing, the preprocessing including: data cleaning, deduplication and standardization; wherein the standardization is to unify the format of the raw data;

[0069] S102, collecting the data processing volume of the pre-processed raw data, and comparing the data processing volume with a preset data volume classification threshold;

[0070] If the data processing volume is less than or equal to the data volume classification threshold, all pre-processed raw data are output as stream processing data;

[0071] If the data processing volume is greater than the data volume classification threshold, the output delay requirement of the pre-processed raw data is collected, wherein the output delay requirement includes an allowable delay requirement and an immediacy requirement;

[0072] Among them, the preprocessed raw data corresponding to the allowable delay requirement is output as batch processing data, and the preprocessed raw data corresponding to the immediacy requirement is output as stream processing data.

[0073] As described in the above steps S101-S102, when processing and analyzing a large amount of raw data, it is first necessary to perform corresponding preprocessing on the raw data, including data cleaning, removal of duplicate content, and standardization, etc., so as to ensure the quality of the data involved in data processing. Standardization is to unify the format and metric of the raw data so that different types of raw data can be effectively processed and analyzed under the same framework. Then, it is necessary to collect the data processing volume of the preprocessed raw data. The data processing volume can be understood as the scale of the data. Then, a data volume classification threshold is first introduced, and the data volume classification threshold is compared with the collected data processing volume. The data volume classification threshold is a preset value, which is used to assist in judging whether the raw data should be Whether it is classified as batch data or stream data, if the data processing volume is less than or equal to the data volume classification threshold, it means that the data is small in size and can be processed in a stream processing manner, thereby speeding up the data processing process. Therefore, the original data in this case is classified as stream data. Stream data is usually used to process real-time data scenarios, such as real-time monitoring, real-time recommendation, etc. On the contrary, if the data processing volume of the original data is greater than the data volume classification threshold, the output delay requirement of the original data will be collected. When the output delay requirement of the original data is the allowable delay requirement, the corresponding original data will be output as batch data. On the contrary, the original data under the immediacy requirement will be recorded as stream data to optimize the data processing process.

[0074] In a preferred embodiment, the task execution method of matching batch processing data and stream processing data includes:

[0075] When only batch data exists in the original data, the task execution mode of the original data is determined to be executed separately;

[0076] When both batch processing data and stream processing data exist in the original data, or when only stream processing data exists in the original data, the task execution mode of the original data is determined to be parallel execution.

[0077] When processing raw data, the task execution method of the data will be determined based on the characteristics and requirements of the data. If the raw data only contains batch data, that is, the raw data is processed at certain time intervals or batches, then a separate execution method will be chosen to process these tasks. However, if the raw data contains both batch data and stream data, or the raw data only contains stream data, then it is necessary to choose a parallel execution method to execute the data processing tasks. The parallel execution method can process multiple tasks at the same time, improving the execution efficiency of the task and the resource utilization of the system.

[0078] In a preferred embodiment, the step of allocating execution resource quotas to batch processing data and stream processing data according to the task execution mode includes:

[0079] S201, obtaining the task execution mode of the current node;

[0080] S202: When the task execution mode is the independent execution, configure an independent execution resource pool for the batch data, wherein the execution resource quota of the batch data is proportional to the total amount of the batch data;

[0081] S203: When the task execution mode is the parallel execution, a shared execution resource pool is configured for the batch processing data and the stream processing data, and execution resource quotas are dynamically allocated according to the data volume of the batch processing data and the data demand delay of the stream processing data.

[0082] As described in the above steps S201-S203, according to the task execution mode, corresponding execution resource quotas will be allocated to batch data and stream data. If the task execution mode is single execution, then it is necessary to configure an independent execution resource pool for batch data. In this case, the execution resource quota of batch data will be proportional to the total amount of data of batch data, which means that the larger the total amount of data of batch data, the larger the execution resource quota, thereby ensuring that batch data can obtain sufficient execution resources, thereby improving processing efficiency. If the task execution mode is parallel execution, a shared execution resource pool will be configured for batch data and stream data to maximize resource utilization efficiency and meet the needs of different types of data processing. Specifically, this can be achieved by increasing processing threads or by coordinating the execution resource quotas between batch data and stream data, so as to ensure that both batch data and stream data can obtain the expected output effect. In addition, when the total amount of resources is limited, the real-time requirements of stream data will be prioritized, and more resources will be allocated to data with low latency requirements to ensure that it can be processed in time.

[0083] In a preferred embodiment, the independent task queue includes a single-thread queue and a parallel thread queue, and the determination process of the single-thread queue and the parallel thread queue is as follows;

[0084] Obtaining the total amount of data to be processed and the unit processing amount of the batch data, and calculating the execution round of the batch data according to the total amount of data to be processed and the unit processing amount;

[0085] The obtained benchmark execution rounds are combined with the execution rounds of the batch data, and the combined calculation result of the benchmark execution rounds and the execution rounds of the batch data is rounded up to obtain the required thread quantity;

[0086] When the value of the required thread quantity is one, the batch data is added to the single-thread queue for sequential processing;

[0087] When the value of the required thread quantity is greater than one, the batch processing data is divided into a plurality of parallel thread queues according to the value of the required thread quantity.

[0088] In this implementation, the independent task queue mainly includes two categories, namely single-thread queue and parallel thread queue. First, the total amount of batch data to be processed and the processing amount of each unit of data need to be obtained. Then, according to the total amount of data to be processed and the unit processing amount, the rounds required to execute the batch data can be calculated. The calculation expression of the execution round is: execution round = total amount of data to be processed / unit processing amount. Then, the benchmark execution round is introduced (the default execution round when the execution resources are sufficient), and the execution round is divided by the benchmark execution round, and the result of the combined operation is rounded up to obtain the actual required thread amount (actual required thread amount = execution round / benchmark execution round). If the required number of threads is 1, then the batch data is added to the single-thread queue, and the queue processes the data in sequence. If the required number of threads is greater than 1, then the batch data will be divided into multiple parts according to the required thread amount, and each part will be used as a parallel thread queue so that multiple batches of data can be processed at the same time to improve processing efficiency.

[0089] In a preferred embodiment, during the parallel execution process, the step of adjusting the execution resource quota of the batch data and the stream data in the shared task queue according to the output delay lag and the task load of the batch data and the stream data includes:

[0090] S501, acquiring in real time the task load of batch processing data and stream processing data, and the output delay lag of stream processing data;

[0091] S502, performing backtracking offset processing on the current node to obtain a backtracking period, and counting the output delay lag of the stream processing data in the backtracking period, and calculating the predicted delay lag of the stream processing data based on the output delay lag in the backtracking period;

[0092] If the value of the predicted delay lag is greater than zero, it indicates that the output of the stream processing data is abnormal, and the execution resource quota of the stream processing data is increased;

[0093] If the value of the predicted delay lag is less than or equal to zero, it indicates that the output of the stream processing data is normal, and the remaining resource quota of the stream processing data is synchronously output.

[0094] As described in the above steps S501-S502, in the process of executing parallel operations, in order to ensure the output efficiency of batch data and stream data, it is necessary to dynamically adjust the execution resource quotas of batch data and stream data in the shared task queue. First, it is necessary to collect and analyze in real time the task load borne by batch data and stream data during the execution process, as well as the output delay lag of stream data, to provide corresponding pre-data support for the adjustment of resource quotas. Then, it is necessary to perform backtracking offset processing on the current node to obtain the output delay lag of stream data within the backtracking period, and then use the prediction function to predict the output delay lag of the next node, where the expression of the prediction function is: In the formula, f y represents the prediction delay lag, f d represents the output delay lag under the current node, T represents the time length of the backtracking period, t represents the time length between the predicted node and the current node, and f i Indicates the output delay lag within the backtracking period. If the value of the predicted delay lag is greater than zero, it means that there is an abnormality in the output of the stream processing data. At this time, it is necessary to increase the execution resource quota of the stream processing data to reduce the output delay lag of the stream processing data. On the contrary, if the value of the predicted delay lag is less than or equal to zero, it indicates that the output of the stream processing data is in a normal state. At this time, the remaining resource quota of the stream processing data can be synchronously output. The remaining resource quota = the current execution resource quota of the stream processing data - the demand quota of the current execution resource of the stream processing data. The remaining resource quota can be allocated to the batch data to speed up its output process, thereby ensuring that resources are fully utilized.

[0095] In a preferred embodiment, the method further comprises:

[0096] When increasing the execution resource quota for stream processing data, the idle resource quota in the shared resource pool is called;

[0097] When the idle resource quota is zero, the execution round of the batch data is reduced, the pending data under the batch data after the execution round of the batch data is reduced is added to the waiting queue, and the execution resource quota occupied by the batch data is released to obtain the allowed allocation resource quota;

[0098] The allowed allocation resource quota is reallocated to the stream processing data until the output delay lag of the stream processing data is less than or equal to zero, or the execution round of the batch processing data is reduced to one and then stopped;

[0099] After the remaining resource quota is output, it is determined whether there is a waiting queue in the batch data. If there is a waiting queue in the batch data, the remaining resource quota is allocated to the batch data.

[0100] In this implementation, when it is necessary to improve the processing capacity of stream processing data, the unoccupied resource quota (idle resource quota) in the shared resource pool will be actively searched and utilized first to increase the available execution resources. When the idle resource quota in the shared resource pool is completely occupied and there is no idle resource available for allocation, corresponding adjustment measures will be taken to ensure the effective utilization of resources and the fairness of allocation. Specifically, the demand for resources is reduced by reducing the number of execution times of batch data. After reducing the number of execution times of batch data, the batch data due to insufficient execution resource quota will be temporarily placed in the waiting queue. At the same time, the execution resource quota occupied by the batch data will also be released so that these resources can be used by other tasks. After the batch data releases the resource quota, it will be marked as the allowable allocation resource quota. The allowable allocation resource quota will be allocated to the stream processing data to ensure that the stream processing data can obtain sufficient resource support, thereby reducing the output delay. This process will continue until the output delay of the stream processing data is reduced to an acceptable range. Or the execution round of batch data is reduced to only one round. After the execution round of batch data is reduced to one, if the output delay lag value of stream data is still positive, it indicates that resource allocation is invalid. At this time, an alarm signal will be issued synchronously, and a summary report including the current delay status of stream data, the resource adjustment measures taken, and the amount of data to be processed in the waiting queue after the batch data releases the execution resource quota will be output, so as to provide a clear troubleshooting and processing guide for the system administrator or operation and maintenance personnel. Here, it should be noted that a lower limit value can be set for the execution round of batch data. The specific value of the lower limit value can be set according to actual needs. For the remaining resource quota, first check whether there is a waiting queue in the batch data. If it is found that there is indeed a waiting queue in the batch data, it means that there is still batch data that has not been processed. At this time, the remaining resource quota under the stream data will be allocated to the batch data to ensure that the tasks in the batch data can be processed in time, ensuring the operation efficiency and stability of the entire system.

[0101] See also Figure 2 , a batch-stream integrated data processing system, comprising:

[0102] An initialization module, which is used to classify the original data according to the amount of the original data obtained and the output delay requirement, obtain batch data and stream data, and synchronously match the task execution mode of the batch data and the stream data, wherein the task execution mode includes single execution and parallel execution;

[0103] Resource allocation module, which is used to allocate execution resource quotas for batch processing data and stream processing data according to the task execution mode;

[0104] A first execution module, the first execution module is used for adding batch processing data to an independent task queue when executed alone, and the batch processing data in the independent task queue is processed according to the collection order;

[0105] A second execution module, the second execution module is used for adding the batch processing data and the stream processing data to the shared task queue when executing in parallel, and the batch processing data and the stream processing data are processed concurrently in the shared task queue;

[0106] The dynamic adjustment module is used to adjust the execution resource quota of batch data and stream processing data in the shared task queue and adjust the scheduling direction of the data to be processed in the shared task queue according to the output delay lag and task load of the batch data and stream processing data during the parallel execution process.

[0107] In the above, the system includes an initialization module, an execution mode matching module, a resource allocation module, a first execution module, a second execution module and a dynamic adjustment module. The main function of the initialization module is to preliminarily classify the original data according to the data volume and output delay requirements of the acquired original data for subsequent batch processing and stream processing. It mainly includes preliminary cleaning and preprocessing of the data to obtain batch data and stream data suitable for batch processing and stream processing. At the same time, it will further determine the task execution mode of the batch data and stream processing data. The task execution mode here includes separate execution and parallel execution. In this way, the system can flexibly select the execution mode according to the actual situation to achieve the optimal data processing effect. The main task of the resource allocation module is to reasonably allocate resources to the batch data and stream processing data according to the task execution mode to ensure the system resources. The source is fully utilized while avoiding resource waste. The first execution module is to add the data to be processed into an independent task queue when single execution is selected, and the batch data is processed sequentially in the independent task queue. The second execution module is to add the batch data and the stream data into a shared task queue when parallel execution is selected, and the batch data and the stream data are processed concurrently in the shared task queue, thereby improving the data processing speed and improving the system efficiency. The main function of the dynamic adjustment module is to dynamically adjust the execution resource quota of the batch data and the stream data in the shared task queue according to the output delay lag and task load of the batch data and the stream data under parallel execution, as well as adjust the scheduling direction of the data to be processed (batch data and stream data) in the shared task queue, thereby ensuring that the system can maintain efficient and stable operation under different circumstances.

[0108] See also Figure 3 , an electronic device, the electronic device comprising:

[0109] at least one processor;

[0110] and a memory communicatively coupled to the at least one processor;

[0111] Among them, the memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the above-mentioned batch-stream integrated data processing method.

[0112] In the above, the processor of the electronic device can be a central processing unit (CPU) or a graphics processing unit (GPU), which can be flexibly configured according to different processing requirements. As the core component of the electronic device, the processor is responsible for executing the computer program stored in the memory to complete complex calculation and data processing tasks.

[0113] The memory is an indispensable part of the electronic device, which is used to store the operating system, application programs, data, and various configuration files. In this embodiment, the memory specifically stores computer programs for implementing the batch-stream integrated data processing method. When these programs are executed by the processor, they can guide the processor to work according to the preset logic and algorithm, thereby realizing efficient and fair processing of batch processing data and stream processing data.

[0114] In addition to processors and memory, electronic devices may also include a variety of other hardware components, such as input devices, output devices, arithmetic units, network interfaces, power supplies, etc. Input devices such as keyboards, mice, touch screens, etc. allow users to interact with electronic devices and input instructions and data. Output devices such as displays and printers are used to display processing results or output relevant information to users. The arithmetic unit is responsible for performing mathematical and logical operations, and the network interface enables electronic devices to communicate with other devices or networks to exchange and share data. The power supply is responsible for providing a stable power supply to the entire electronic device to ensure its continuous operation.

[0115] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0116] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. A batch-stream integrated data processing method, characterized in that: include: According to the data volume and output delay requirement of the acquired original data, the original data is classified to obtain batch data and stream data, and task execution modes of the batch data and the stream data are synchronously matched, wherein the task execution modes include separate execution and parallel execution; Allocate execution resource quotas to the batch processing data and the stream processing data according to the task execution mode; When the separate execution is performed, the batch processing data is added to an independent task queue, and the batch processing data in the independent task queue is processed according to the acquisition order; When the parallel execution is performed, the batch processing data and the stream processing data are added to a shared task queue, and the batch processing data and the stream processing data are processed concurrently in the shared task queue; During the parallel execution process, the execution resource quota of the batch data and the stream processing data in the shared task queue is adjusted according to the output delay lag and task load of the batch data and the stream processing data, and the scheduling direction of the data to be processed in the shared task queue is adjusted.

2. The method for batch-stream integrated data processing according to claim 1, characterized in that: The step of classifying the original data according to the amount of the original data obtained and the output delay requirement to obtain batch data and stream data includes: Acquire the raw data and perform preprocessing, wherein the preprocessing includes: data cleaning, deduplication and standardization; wherein the standardization is to unify the format of the raw data; Collecting the data processing volume of the preprocessed raw data, and comparing the data processing volume with a preset data volume classification threshold; If the data processing volume is less than or equal to the data volume classification threshold, all pre-processed raw data are output as stream processed data; If the data processing volume is greater than the data volume classification threshold, then collecting the output delay requirements of the pre-processed original data, wherein the output delay requirements include an allowable delay requirement and an immediacy requirement; The pre-processed original data corresponding to the allowable delay requirement is output as the batch processed data, and the pre-processed original data corresponding to the immediacy requirement is output as the stream processed data.

3. The batch-stream integrated data processing method according to claim 1, characterized in that: The task execution method of matching the batch processing data and the stream processing data includes: When only batch processing data exists in the original data, determining the task execution mode of the original data as the separate execution; When the original data contains both batch processing data and stream processing data, or when the original data contains only stream processing data, the task execution mode of the original data is determined to be the parallel execution.

4. The method for batch-stream integrated data processing according to claim 1, characterized in that: The step of allocating execution resource quotas to the batch processing data and the stream processing data according to the task execution mode includes: Get the task execution mode under the current node; When the task execution mode is the separate execution, an independent execution resource pool is configured for the batch data, wherein the execution resource quota of the batch data is proportional to the total amount of data of the batch data; When the task execution mode is the parallel execution, a shared execution resource pool is configured for the batch processing data and the stream processing data, and execution resource quotas are dynamically allocated according to the data volume of the batch processing data and the data demand delay of the stream processing data.

5. The batch-stream integrated data processing method according to claim 1, characterized in that: The independent task queue includes a single-thread queue and a parallel thread queue, and the determination process of the single-thread queue and the parallel thread queue is as follows; Acquire the total amount of data to be processed of the batch data and the unit processing amount of the batch data, and calculate the execution round of the batch data according to the total amount of data to be processed and the unit processing amount; The obtained benchmark execution round is combined with the execution round of the batch data, and the combined calculation result of the benchmark execution round and the execution round of the batch data is rounded up to obtain the required thread quantity; When the value of the required thread quantity is one, the batch processing data is added to a single thread queue for sequential processing; When the value of the required thread quantity is greater than one, the batch processing data is divided into a plurality of parallel thread queues according to the value of the required thread quantity.

6. The method for batch-stream integrated data processing according to claim 1, characterized in that: In the parallel execution process, the step of adjusting the execution resource quota of the batch data and the stream data in the shared task queue according to the output delay lag and task load of the batch data and the stream data includes: Acquire in real time the task load of the batch processing data and the stream processing data, and the output delay lag of the stream processing data; Performing backtracking offset processing on the current node to obtain a backtracking period, and counting the output delay lag of the stream processing data in the backtracking period, and calculating the predicted delay lag of the stream processing data based on the output delay lag in the backtracking period; If the value of the predicted delay lag is greater than zero, it indicates that the output of the stream processing data is abnormal, and the execution resource quota of the stream processing data is increased; If the value of the predicted delay lag is less than or equal to zero, it indicates that the output of the stream processing data is normal, and the remaining resource quota of the stream processing data is synchronously output.

7. The method for batch-stream integrated data processing according to claim 6, characterized in that: The method further comprises: When increasing the execution resource quota of the stream processing data, calling the idle resource quota in the shared resource pool; When the idle resource quota is zero, the execution round of the batch data is reduced, the to-be-processed data under the batch data after the execution round of the batch data is reduced is added to the waiting queue, and the execution resource quota occupied by the batch data is released to obtain the allowed allocation resource quota; reallocating the allowed allocation resource quota to the stream processing data until the output delay lag of the stream processing data is less than or equal to zero, or the execution round of the batch processing data is reduced to one and then stopped; After the remaining resource quota is output, it is determined whether there is a waiting queue in the batch data. If there is a waiting queue in the batch data, the remaining resource quota is allocated to the batch data.

8. The batch-stream integrated data processing method according to claim 7, characterized in that: After the execution round of the batch processing data is reduced to one, the method further includes: If the output delay lag value of the stream processing data is still positive, an alarm signal is synchronously issued, and a summary report including the current delay status of the stream processing data, the resource adjustment measures that have been taken, and the amount of data to be processed in the waiting queue after the batch data releases the execution resource quota is output.

9. A batch-stream integrated data processing system, characterized by: include: An initialization module, the initialization module is used to classify the original data according to the data volume and output delay requirements of the acquired original data, obtain batch data and stream data, and synchronously match the task execution mode of the batch data and the stream data, wherein the task execution mode includes separate execution and parallel execution; A resource allocation module, the resource allocation module is used to allocate execution resource quotas to the batch processing data and the stream processing data according to the task execution mode; A first execution module, wherein the first execution module is used to add the batch processing data to an independent task queue when the single execution is performed, and the batch processing data in the independent task queue is processed according to the acquisition order; A second execution module, wherein the second execution module is used for adding the batch processing data and the stream processing data to a shared task queue during the parallel execution, and the batch processing data and the stream processing data are processed concurrently in the shared task queue; A dynamic adjustment module is used to adjust the execution resource quota of the batch data and the stream processing data in the shared task queue and adjust the scheduling direction of the data to be processed in the shared task queue according to the output delay lag and task load of the batch data and the stream processing data during the parallel execution process.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the batch-stream integrated data processing method described in any one of claims 1 to 8.

Images