A method and apparatus for real-time data processing
By generating a task sub-scheduling list and updating the task status in real time, the problem of insufficient time extension or short latency and inaccuracy in data scheduling is solved, achieving accurate and low-latency data processing with high resource utilization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING BOCO COMM TECH
- Filing Date
- 2021-12-30
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, data scheduling methods suffer from problems such as long data duration or short latency and insufficient accuracy, making it difficult to simultaneously meet the requirements of data accuracy and short latency, resulting in insufficient resource utilization.
By executing real-time data collection tasks based on task configuration information, generating a task sub-scheduling list, updating task status in real time, scanning the task sub-scheduling list, and summarizing data when conditions are met, the accuracy and low latency of the data are ensured.
This approach ensures data accuracy while minimizing data processing latency, fully utilizing system resources, and improving data processing efficiency and resource utilization.
Smart Images
Figure CN114489974B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the computer software industry, and more specifically, to a real-time data processing technology. Background Technology
[0002] With the rise of big data, the demand for data visualization and aggregation services is increasing across various industries. The real-time nature of data processing and visualization has become a focal point. The industry is gradually shifting from simple delayed visualization to real-time data monitoring systems, with increasingly higher requirements for data timeliness and accuracy. Delays have progressed from days to hours, to minutes, and finally to near real-time—the requirement is to provide and display data precisely when it is available. Traditional task scheduling methods are overly simplistic, often relying on timed methods. However, timed methods have the following problems:
[0003] Firstly, early data scheduling results in short data latency, but data accuracy is questionable.
[0004] Secondly, data scheduling is late, resulting in longer data processing time. Data accuracy is not an issue.
[0005] It's difficult to find a point in time that satisfies both the need for accurate data and low latency. Even if a reasonable time is found, the expected results may still not be obtained due to data interface latency issues.
[0006] Currently, scheduled processing and single-threaded scheduling are commonly used. Scheduled processing employs Linux's CONTAB scheduler, where pre-written processing scripts are written and scheduled to process the data. After scheduling, the source data is calculated and inserted into the target table. During processing, there are no dependencies between data items; each process operates independently.
[0007] Single-threaded scheduling uses a serial task approach. While this is simple to configure, tasks are scheduled one by one, completing one task before moving on to the next. The aggregation task follows the data collection task. However, this approach leads to insufficient resource utilization. Furthermore, a delay in one data collection task often causes multiple delays in subsequent aggregation tasks, severely impacting presentation.
[0008] Therefore, how to perform real-time and accurate data acquisition and processing has become an urgent problem to be solved. Summary of the Invention
[0009] This invention provides a real-time data processing method, the method comprising:
[0010] Execute real-time data acquisition tasks based on task configuration information and store the acquired data;
[0011] Based on the requirements for summarizing the time dimension, granularity, and data volume of the data collection tasks, the task scheduling attribute information is determined, and a corresponding task sub-scheduling list is generated.
[0012] The data collection tasks are scheduled according to the task sub-scheduling list, and the task sub-scheduling list is updated in real time according to the execution status of the data collection tasks.
[0013] The task sub-scheduling list is scanned in real time. When all tasks in the task sub-scheduling list are completed or the delay time set in the task scheduling attribute information is reached, the collected data is summarized according to the summary requirements.
[0014] Furthermore, the method for determining task scheduling attribute information and generating a corresponding task sub-scheduling list based on the aggregation requirements of the time dimension, granularity, and data volume of the data collection tasks is as follows:
[0015] Based on the aforementioned requirements for the time dimension of data collection task aggregation, determine the task scheduling cycle and maximum delay;
[0016] Based on the required granularity of data collection task aggregation, determine the number of task scheduling times within the period;
[0017] A corresponding task sub-scheduling list is generated based on the number of task schedulings within the specified period.
[0018] Furthermore, the method for updating the task sub-scheduling list in real time based on the execution status of the data collection task is as follows:
[0019] When the current acquisition task in the task sub-scheduling list is completed, the current task sub-scheduling completion flag is updated;
[0020] Once all data acquisition tasks in the task sub-scheduling list have been completed, the task completion flag is updated.
[0021] Furthermore, the method for real-time scanning of the task sub-scheduling list, and summarizing the collected data according to the summarization requirements when all tasks on the task sub-scheduling list are completed or the delay time set in the task scheduling attribute information is as follows:
[0022] Real-time scanning of the task sub-scheduling list;
[0023] When the task completion flag in the task sub-scheduling list is complete, the collected data is summarized according to the data volume requirement in the summary requirement;
[0024] When the delay time set in the task scheduling attribute information is reached, the collected data is summarized according to the data volume requirement in the summary requirement.
[0025] Preferred:
[0026] Update the task sub-scheduling list after the summary is completed.
[0027] The present invention also discloses a real-time data processing apparatus, the apparatus comprising:
[0028] The task management unit is used to execute real-time data acquisition tasks according to task configuration information and store the acquired data.
[0029] The task splitting unit is used to determine the task scheduling attribute information and generate the corresponding task sub-scheduling list based on the aggregation requirements of the time dimension, granularity and data volume of the collection task.
[0030] The task progress update unit is used to schedule the collection task according to the task sub-scheduling list and update the task sub-scheduling list in real time according to the execution status of the collection task.
[0031] The aggregation unit scans the task sub-scheduling list in real time. When all tasks in the task sub-scheduling list are completed or the delay time set in the task scheduling attribute information is reached, the collected data is aggregated according to the aggregation requirements.
[0032] Specifically, the task splitting unit further includes:
[0033] The parameter setting module is used to determine the scheduling cycle and maximum delay of the real-time acquisition tasks in the task management module according to the requirements for the time dimension of the acquisition task aggregation; and to determine the number of task scheduling times within the cycle according to the requirements for the granularity of the acquisition task aggregation.
[0034] The task sub-scheduling list determination module is used to generate a corresponding task sub-scheduling list based on the number of task scheduling times within the period set by the parameter setting module.
[0035] Specifically, the task progress update unit further includes:
[0036] The verification module is used to verify the execution status of the current data acquisition task and all data acquisition tasks.
[0037] The flag update module is used to update the task flag according to the task execution status verified by the verification module.
[0038] When the current acquisition task in the task sub-scheduling list is completed, the current task sub-scheduling completion flag is updated;
[0039] Once all data acquisition tasks in the task sub-scheduling list have been completed, the task completion flag is updated.
[0040] Specifically, the summarizing unit further includes:
[0041] The real-time scanning module is used to scan the task sub-scheduling list in real time;
[0042] The aggregation module is used to aggregate the collected data according to the data volume requirement in the aggregation requirement when the task completion flag of the task sub-scheduling list scanned by the real-time scanning module is completed; and to aggregate the collected data according to the data volume requirement in the aggregation requirement when the delay time set in the task scheduling attribute information is reached.
[0043] Preferably, the device further includes:
[0044] An update unit is used to update the task subtask scheduling list after the summary unit completes the summary.
[0045] As can be seen from the above technical solution, the real-time data processing technology disclosed in this invention finely divides real-time acquisition tasks according to the time dimension, granularity, and data volume aggregation requirements; during the execution of acquisition tasks, the sub-scheduling list of tasks is updated in real time; by scanning the sub-scheduling list, the progress of acquisition tasks can be known without monitoring massive amounts of data; when all sub-scheduling lists are completed, data aggregation can be performed, ensuring data integrity and accuracy; data aggregation is performed when a set delay time is reached, ensuring that data is aggregated within the set delay time. The technical solution of this invention maximizes both data accuracy and low data latency, making good use of system resources. Attached Figure Description
[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0047] Figure 1 This is a schematic flowchart of a real-time data processing method provided in an embodiment of the present invention;
[0048] Figure 2 This is a flowchart of a method for generating a task sub-scheduling list provided in Embodiment 2 of the present invention;
[0049] Figure 3 This is a flowchart of a method for updating a task sub-scheduling list provided in Embodiment 3 of the present invention;
[0050] Figure 4This is a flowchart of a method for summarizing collected data provided in Embodiment 4 of the present invention;
[0051] Figure 5 This is a structural diagram of a real-time data processing device provided in Embodiment 5 of the present invention;
[0052] Figure 6 This is a detailed structural diagram of the device provided in Embodiment Six of the present invention. Detailed Implementation
[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0054] Please see Figure 1 The present invention provides an embodiment one, such as... Figure 1 As shown, a real-time data processing method of the present invention is described in detail, including the following steps:
[0055] Step S1: Execute the real-time data acquisition task according to the task configuration information and store the acquired data.
[0056] Task configuration information refers to relevant information for executing the data collection task, such as the source system, task type, data source name, collection period, collection time, and task lock identifier. The task lock identifier can use different numbers or characters to indicate whether it is locked or unlocked. The task lock is opened when the data collection task starts to prevent the same task from being scheduled multiple times, and the task lock is released when the task is completed.
[0057] After the data collection task is executed, the collected data is stored. The execution status of the task is monitored through a task sub-scheduling list that further subdivides the task. When the conditions for aggregation are met, the stored data is aggregated according to the aggregation requirements.
[0058] Step S2: Based on the requirements for summarizing the time dimension, granularity, and data volume of the data collection tasks, determine the task scheduling attribute information and generate the corresponding task sub-scheduling list.
[0059] In the summary requirements, the time dimension of the data collection task refers to the data collected in time units. For example, if the data is collected in 15 minutes, 1 hour, or 2 days, then the time dimension is 15 minutes, 1 hour, and 2 days, respectively.
[0060] In the summary requirements, the granularity of the data collection task refers to the summary granularity in the time dimension. For example, data for 15 minutes is summarized in 3-minute increments, data for 1 hour is summarized in 15-minute increments, and data for 2 days is summarized in 2-hour increments.
[0061] The data volume in the summary requirements refers to the requirements for the number of data items and data attributes, such as how many data items to summarize and the target fields in the summarized data.
[0062] By summarizing the time dimension, granularity, and data volume in the requirements, we can determine the task scheduling attribute information and generate a list of task sub-schedules. The task scheduling attribute information also includes latency information.
[0063] Therefore, based on the aggregation requirements, the data collection task scheduling is decomposed into task sub-schedulings of different granularities within a certain time unit, and each task sub-scheduling has a flag bit that records the completion status of the task sub-scheduling.
[0064] Step S3: Schedule the data collection tasks according to the task sub-scheduling list, and update the task sub-scheduling list in real time according to the execution status of the data collection tasks.
[0065] The data collection tasks are scheduled, and the sub-scheduling list is updated based on the completion status of the sub-scheduling tasks.
[0066] Step S4: Scan the task sub-scheduling list in real time. When all tasks in the task sub-scheduling list are completed or the delay time set in the task scheduling attribute information is reached, summarize the collected data according to the summary requirements.
[0067] Step S5: Update the task sub-scheduling list after the summary is completed.
[0068] After the aggregation is complete, the subtask schedule list is updated to prepare for the next aggregation data that depends on this task list. This update involves setting all task flags to "incomplete".
[0069] The data acquisition task maximizes system resource utilization through fine task division. By monitoring the task sub-scheduling list, the execution status of the acquisition tasks can be obtained in real time, and the decision to perform data aggregation can be made based on the execution status and set delays. Therefore, it makes full use of system resources, ensuring data accuracy while guaranteeing minimal data processing latency. This is the preferred technical solution for real-time data processing.
[0070] For a second embodiment of the present invention, please refer to the appendix. Figure 2The detailed explanation of step S2, which involves determining task scheduling attribute information and generating a corresponding task sub-scheduling list based on the time dimension, granularity, and data volume aggregation requirements of the data collection task, is as follows:
[0071] Step S21: Based on the requirements for the time dimension of the data collection task summary, determine the task scheduling cycle and maximum delay.
[0072] The task scheduling cycle is determined here based on time, and time cycles such as minutes, hours, days, and months can be set according to actual needs.
[0073] The maximum delay indicates that if this delay is exceeded, data will be aggregated regardless of whether all data collection is complete, thus ensuring the real-time nature of the data.
[0074] Step S22: Determine the number of task scheduling times within the period based on the required granularity of the data collection task summary.
[0075] Granularity represents the number of task sub-schedulings within a cycle. The finer the granularity, the more task sub-schedulings occur; conversely, the coarser the granularity, the fewer task sub-schedulings occur.
[0076] If data is aggregated in one-hour cycles with a granularity of 15 minutes, then the number of task scheduling events within the cycle is 4. The granularity of the aggregation can be represented by time markers, such as a 15-minute interval for a one-hour data set, with time markers set to 00, 15, 30, and 45 minutes.
[0077] Step S23: Generate a corresponding task sub-scheduling list based on the number of task schedulings within the cycle.
[0078] Recording the data collection tasks and generating task sub-schedules based on time and granularity forms a task sub-schedule list.
[0079] By subdividing the data collection tasks, the scheduling frequency can be freely controlled, and data can be aggregated according to needs. This allows for adjustment of the aggregation latency while ensuring the accuracy and completeness of the data.
[0080] For the third embodiment of the present invention, please refer to the appendix. Figure 3 The detailed explanation of step S3, the method for updating the task sub-scheduling list in real time based on the execution status of the acquisition tasks, is as follows: when the current acquisition task in the task sub-scheduling list is completed, the current task sub-scheduling completion flag is updated. When all acquisition tasks in the task sub-scheduling list are completed, the task completion flag is updated.
[0081] Step S31: Determine whether the current task in the task sub-scheduling list has been completed; if yes, proceed to step S32, otherwise repeat step S31.
[0082] Step S32: Update the current task sub-scheduling completion flag.
[0083] Step S33: Determine whether all data collection tasks in the task sub-scheduling list have been completed; if yes, proceed to step S34; otherwise, return to step S31.
[0084] Step S34: Update the task completion flag.
[0085] The task completion flag can be a specific number or character, such as 1 to indicate scheduling completion and 0 to indicate no scheduling.
[0086] Once all data collection tasks in the task sub-scheduling list have been completed, the data collection tasks are considered complete, and data can be aggregated according to the aggregation requirements.
[0087] By monitoring the completion status of data collection tasks through a monitoring task sub-scheduling list and triggering aggregation, the efficiency of data collection and resource utilization are greatly improved.
[0088] For the fourth embodiment of the present invention, please refer to the appendix. Figure 4 The detailed explanation of step S4, which involves real-time scanning of the task sub-scheduling list, and summarizing the collected data according to the summary requirements when all tasks on the task sub-scheduling list are completed or the delay time set in the task scheduling attribute information is as follows:
[0089] Step S41: Scan the task sub-scheduling list in real time.
[0090] Step S42: Determine whether the task completion flag in the task sub-scheduling list is complete; if yes, proceed to step S44; otherwise, return to step S41.
[0091] Step S43: Determine whether the delay time set in the task scheduling attribute information has been reached. If yes, proceed to step S44; otherwise, return to step S41.
[0092] Step S44: Summarize the collected data according to the data volume requirements in the summary requirements.
[0093] Triggering aggregation based on task completion flags yields complete data; triggering aggregation based on set delay times yields data with minimal latency. Therefore, different aggregation triggering conditions ensure the accuracy, completeness, and real-time performance of data processing.
[0094] The present invention also discloses a real-time data processing apparatus, of which Embodiment Six is given below, in conjunction with the appendix. Figure 6The structure of the device is described in detail. The device includes a task management unit 1, a task splitting unit 2, a task progress updating unit 3, and a summarizing unit 4, wherein:
[0095] Task management unit 1 is used to execute real-time data acquisition tasks according to task configuration information and store the acquired data;
[0096] Task splitting unit 2 is used to determine task scheduling attribute information and generate a corresponding task sub-scheduling list based on the requirements for summarizing the time dimension, granularity and data volume of the collection tasks.
[0097] Task progress update unit 3 is used to schedule the collection task according to the task sub-scheduling list and update the task sub-scheduling list in real time according to the execution status of the collection task.
[0098] The aggregation unit 4 scans the task sub-scheduling list in real time. When all the task sub-scheduling lists are completed or the delay time set in the task scheduling attribute information is reached, the collected data is aggregated according to the aggregation requirements.
[0099] Update unit 5 is used to update the task subtask scheduling list after the aggregation unit completes the aggregation.
[0100] To better illustrate the structure and working principle of each unit, Embodiment Six of this application is given below, in conjunction with the appendix. Figure 6 Detailed explanation:
[0101] Task management unit 1 is used to execute real-time data acquisition tasks and store the acquired data according to the task configuration information.
[0102] Task splitting unit 2 further includes:
[0103] The parameter setting module 21 is used to determine the scheduling cycle and maximum delay of the real-time acquisition task in the task management module according to the requirements for the time dimension of the acquisition task summary; and to determine the number of task scheduling within the cycle according to the requirements for the granularity of the acquisition task summary.
[0104] The task sub-scheduling list determination module 22 is used to generate a corresponding task sub-scheduling list based on the number of task scheduling times within the period set by the parameter setting module.
[0105] Task progress update unit 3 further includes:
[0106] Verification module 31 is used to verify the execution status of the current acquisition task and all acquisition tasks;
[0107] Flag update module 32 is used to update the task flag bit according to the task execution status verified by the verification module;
[0108] When the current acquisition task in the task sub-scheduling list is completed, the current task sub-scheduling completion flag is updated;
[0109] Once all data acquisition tasks in the task sub-scheduling list have been completed, the task completion flag is updated.
[0110] Summary Unit 4 further includes:
[0111] Real-time scanning module 41 is used to scan the task sub-scheduling list in real time;
[0112] The aggregation module 42 is used to aggregate the collected data according to the data volume requirement in the aggregation requirement when the task completion flag of the task sub-scheduling list scanned by the real-time scanning module is completed; and to aggregate the collected data according to the data volume requirement in the aggregation requirement when the delay time set in the task scheduling attribute information is reached.
[0113] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the corresponding processes in the above-described device embodiments can be referred to the specific working process of the aforementioned method, and will not be repeated here.
[0114] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0115] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0116] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings are used to distinguish similar parts and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated herein.
[0117] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for real-time data processing, characterized in that, The method includes: Execute real-time data acquisition tasks based on task configuration information and store the acquired data; Based on the requirements for summarizing the time dimension, granularity, and data volume of the data collection tasks, the task scheduling attribute information is determined, and a corresponding task sub-scheduling list is generated. Granularity indicates the number of task sub-schedulings within a period; the finer the granularity, the more task sub-schedulings occur, and vice versa. Furthermore, if data is summarized in one-hour periods with a 15-minute summarization granularity, the number of task schedulings within a period is 4. The granularity of the summarization is indicated by time markers, with one hour's data summarized in 15-minute intervals, and time markers set to 00, 15, 30, and 45 minutes. The data collection tasks are scheduled according to the task sub-scheduling list, and the task sub-scheduling list is updated in real time according to the execution status of the data collection tasks. The task sub-scheduling list is scanned in real time. When all tasks in the task sub-scheduling list are completed or the delay time set in the task scheduling attribute information is reached, the collected data is summarized according to the summary requirements. Update the task sub-scheduling list after the summary is complete. Specifically, the method for determining task scheduling attribute information and generating a corresponding task sub-scheduling list based on the aggregation requirements of the time dimension, granularity, and data volume of the data collection task is as follows: Based on the aforementioned requirements for summarizing the time dimension of the data collection tasks, the task scheduling cycle and maximum delay are determined. Based on the aforementioned requirements for summarizing the granularity of data collection tasks, determine the number of task scheduling operations within the period. A corresponding task sub-scheduling list is generated based on the number of task schedulings within the specified period; The method for updating the task sub-scheduling list in real time based on the execution status of the data collection task is as follows: When the current acquisition task in the task sub-scheduling list is completed, the current task sub-scheduling completion flag is updated; Once all data acquisition tasks in the task sub-scheduling list have been completed, the task completion flag is updated.
2. The method according to claim 1, characterized in that, The method of real-time scanning of the task sub-scheduling list and summarizing the collected data according to the summarization requirements when all tasks in the task sub-scheduling list are completed or the delay time set in the task scheduling attribute information is as follows: real-time scanning of the task sub-scheduling list. When the task completion flag in the task sub-scheduling list is complete, the collected data is summarized according to the data volume requirement in the summary requirement; When the delay time set in the task scheduling attribute information is reached, the collected data is summarized according to the data volume requirement in the summary requirement.
3. A real-time data processing apparatus, characterized in that, The device includes: The task management unit is used to execute real-time data acquisition tasks according to task configuration information and store the acquired data. The task splitting unit is used to determine task scheduling attribute information and generate a corresponding task sub-scheduling list based on the aggregation requirements of the time dimension, granularity, and data volume of the data collection tasks. Granularity indicates the number of task sub-schedulings within a period; the finer the granularity, the more task sub-schedulings there are, and vice versa. For example, if data is aggregated in one-hour periods with a granularity of 15 minutes, the number of task schedulings within a period is 4. The granularity of aggregation is indicated by time markers, with one hour's data aggregated in 15-minute intervals, and time markers set to 00, 15, 30, and 45 minutes. The task progress update unit is used to schedule the collection task according to the task sub-scheduling list and update the task sub-scheduling list in real time according to the execution status of the collection task. The aggregation unit scans the task sub-scheduling list in real time. When all the task sub-scheduling lists are completed or the delay time set in the task scheduling attribute information is reached, the collected data is aggregated according to the aggregation requirements. The update unit is used to update the task sub-scheduling list after the aggregation unit has completed the aggregation. The task splitting unit further includes: The parameter setting module is used to determine the scheduling cycle and maximum delay of the real-time acquisition tasks in the task management unit according to the summary requirements of the time dimension of the acquisition tasks; and to determine the number of task scheduling within the cycle according to the summary requirements of the granularity of the acquisition tasks. The task sub-scheduling list determination module is used to generate a corresponding task sub-scheduling list based on the number of task scheduling times within the period set by the parameter setting module. The task progress update unit further includes: The verification module is used to verify the execution status of the current data acquisition task and all data acquisition tasks. The flag update module is used to update the task flag according to the task execution status verified by the verification module. When the current acquisition task in the task sub-scheduling list is completed, the current task sub-scheduling completion flag is updated; Once all data acquisition tasks in the task sub-scheduling list have been completed, the task completion flag is updated.
4. The apparatus according to claim 3, characterized in that, The aggregation unit further includes: The real-time scanning module is used to scan the task sub-scheduling list in real time; The aggregation module is used to aggregate the collected data according to the data volume requirement in the aggregation requirement when the task completion flag of the task sub-scheduling list scanned by the real-time scanning module is completed; and to aggregate the collected data according to the data volume requirement in the aggregation requirement when the delay time set in the task scheduling attribute information is reached.