Business big data processing system and method based on double timed tasks

Abstract

The invention relates to a business big data processing system and method based on double timed tasks, electronic equipment, a computer readable medium and a computer program product. The method comprises the steps of obtaining business big data from a database through a first timed task; creating a thread pool according to a time window of the first timed task, wherein the thread pool comprises a plurality of threads; processing the business big data through threads in the thread pool; the processing result is updated to the database; obtaining business big data meeting a preset condition from the database through the second timed task; and processing the business big data according to the time window of the second timed task. The real-time performance of data processing and the utilization efficiency of system resources can be remarkably improved, meanwhile, load balancing is achieved through task dispersion, stable operation of the system in a high-load scene is ensured, the system safety is ensured, and the system is suitable for various service scene requirements.

Description

Technical Field

[0001] This application relates to the field of computer information processing. Specifically, it relates to a business big data processing system, method, electronic device, computer-readable medium, and computer program product based on dual timing tasks. Background Art

[0002] In current big data processing scenarios, many systems need to periodically pull a large amount of data from data sources for processing. For example: real-time log analysis, user behavior data statistics, transaction record processing, and device status monitoring, etc. Traditional solutions usually use timing tasks executed at fixed time intervals to batch pull unprocessed data from databases or other data storages, and process and update their statuses in sequence. This method can work properly when the data volume is moderate, but it exposes the following problems in large-scale data scenarios:

[0003] Processing delay: As the data volume continues to grow, the amount of data pulled and processed by a single timing task gradually increases. When the processing capacity is insufficient, data will accumulate in the system, resulting in business response delays and a decline in data timeliness.

[0004] System resource pressure: When timing tasks are executed in a single-threaded or low-concurrency manner, the processing efficiency is low. A large number of query and update operations may cause database lock contention, exacerbating the system load and even affecting the normal operation of other business modules.

[0005] Lack of dynamic adjustment ability: Fixed task scheduling and processing logic cannot flexibly adjust the processing capacity according to fluctuations in data volume and business requirements, resulting in difficulty in coping during peak periods and possible resource waste during off-peak periods.

[0006] Therefore, a new business big data processing system, method, electronic device, computer-readable medium, and computer program product based on dual timing tasks are needed.

[0007] The above information disclosed in the background art section is only used to enhance the understanding of the background of this application. Therefore, it may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0008] In view of this, this application provides a business big data processing system, method, electronic device, computer-readable medium, and computer program product based on dual timing tasks, which can significantly improve the real-time performance of data processing and the utilization efficiency of system resources. At the same time, load balancing is achieved through task dispersion to ensure the stable operation of the system in high-load scenarios, guarantee system security, and adapt to the requirements of various business scenarios.

[0009] Other features and advantages of the present application will become apparent from the following detailed description, or will be learned in part from the practice of the present application.

[0010] According to one aspect of the present application, a method for processing business big data based on dual timing tasks is proposed. The method includes: obtaining business big data from a database through a first timing task; creating a thread pool according to the time window of the first timing task, the thread pool including multiple threads; processing the business big data through the threads in the thread pool; updating the processing result to the database; obtaining business big data that meets preset conditions from the database through a second timing task; and processing the business big data according to the time window of the second timing task.

[0011] Optionally, obtaining business big data from a database through a first timing task; creating a thread pool according to the time window of the first timing task, the thread pool including multiple threads; processing the business big data through the threads in the thread pool; updating the processing result to the database; obtaining business big data that meets preset conditions from the database through a second timing task; and processing the business big data according to the time window of the second timing task.

[0012] Optionally, obtaining business big data from a database through a first timing task includes: setting a first timing task based on the Spring TaskScheduler framework; and periodically pulling the business big data from the database through the first timing task.

[0013] Optionally, creating a thread pool according to the time window of the first timing task, the thread pool including multiple threads, includes: dividing the time window of the first timing task into multiple sub-time windows; determining the number of threads according to the number of the sub-time windows; and establishing the thread pool according to the number of threads.

[0014] Optionally, dividing the time window of the first timing task into multiple sub-time windows includes: dividing the time window of the first timing task into multiple sub-time windows according to the data volume of the business big data.

[0015] Optionally, processing the business big data through the threads in the thread pool includes: setting corresponding sub-time windows for each thread; and each thread querying and processing the corresponding business big data within its sub-time window.

[0016] Optionally, processing the business big data through the threads in the thread pool further includes: monitoring the processing tasks of each thread through a CountDownLatch synchronization tool; and continuing to execute or ending the first timing task after the task processing of each thread is completed.

[0017] Optionally, updating the processing result to the database includes: marking the processed business big data as processed and updating it to the database.

[0018] Optionally, obtaining business big data that meets preset conditions from the database through a second scheduled task includes: obtaining unprocessed business big data whose last update time is greater than a time threshold from the database through a second scheduled task.

[0019] Optionally, processing the business big data according to the time window of the second scheduled task includes: processing the business big data within the time window of the second scheduled task; marking the processed business big data as processed and updating it to the database.

[0020] According to one aspect of the present application, a business big data processing system based on dual scheduled tasks is proposed. The system includes: a first task module for obtaining business big data from a database through a first scheduled task; a thread pool module for creating a thread pool according to the time window of the first scheduled task, where the thread pool includes multiple threads; a thread module for processing the business big data through the threads in the thread pool; an update module for updating the processing result to the database; a second task module for obtaining business big data that meets preset conditions from the database through a second scheduled task; and a processing module for processing the business big data according to the time window of the second scheduled task.

[0021] According to one aspect of the present application, an electronic device is proposed. The electronic device includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0022] According to one aspect of the present application, a computer-readable medium is proposed, on which a computer program is stored. When the program is executed by a processor, the method as described above is implemented.

[0023] According to one aspect of the present application, a computer program product is proposed, including: a computer program / instructions, when the computer program / instructions are executed by a processor, the method as described above is implemented.

[0024] A business big data processing system, method, electronic device, computer-readable medium, and computer program product based on dual timing tasks according to the present application obtain business big data from a database through a first timing task; create a thread pool according to the time window of the first timing task, the thread pool including multiple threads; process the business big data through the threads in the thread pool; update the processing result to the database; obtain business big data meeting preset conditions from the database through a second timing task; and process the business big data according to the time window of the second timing task, which can significantly improve the real-time performance of data processing and the utilization efficiency of system resources, and at the same time achieve load balancing through task dispersion, ensure the stable operation of the system in high-load scenarios, ensure system security, and can adapt to the requirements of various business scenarios.

[0025] It should be understood that the above general description and the following detailed description are only exemplary and do not limit the present application. Brief Description of the Drawings

[0026] By referring to the accompanying drawings and describing its exemplary embodiments in detail, the above and other objectives, features, and advantages of the present application will become more obvious. The following described drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0027] Figure 1 is a flowchart of a method for processing business big data based on dual timing tasks shown according to an exemplary embodiment.

[0028] Figure 2 is a flowchart of a method for processing business big data based on dual timing tasks shown according to another exemplary embodiment.

[0029] Figure 3 is a flowchart of a method for processing business big data based on dual timing tasks shown according to another exemplary embodiment.

[0030] Figure 4 is a block diagram of a system for processing business big data based on dual timing tasks shown according to another exemplary embodiment.

[0031] Figure 5 is a block diagram of an electronic device shown according to an exemplary embodiment. Detailed Description of the Embodiments

[0032] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. Like reference numerals in the figures denote like or similar parts, and thus their repeated description will be omitted.

[0033] Figure 1 is a flowchart of a business big data processing method based on dual timing tasks shown according to an exemplary embodiment. The business big data processing method 10 based on dual timing tasks at least includes steps S102 to S112.

[0034] As Figure 1 shown, in S102, business big data is obtained from the database by a first timing task. For example, the first timing task can be set based on the Spring Task Scheduler framework; the business big data is pulled from the database periodically by the first timing task.

[0035] More specifically, by setting a periodic first timing task, business big data to be processed is pulled from the database at a fixed time interval (such as every minute). The task retrieves data according to preset query conditions (such as "Business A" or "audit task") and adds this data to the queue to be processed, preparing for subsequent processing steps.

[0036] In a practical embodiment, for example, in an e-commerce business, the timing task pulls order data with the status of "paid" from the database every minute to ensure that the latest orders can be obtained and processed by the system in a timely manner.

[0037] In S104, a thread pool is created according to the time window of the first timing task, and the thread pool includes multiple threads. For example, the time window of the first timing task is divided into multiple sub-time windows; the number of threads is determined according to the number of sub-time windows; the thread pool is established according to the number of threads. More specifically, the time window of the first timing task can be divided into multiple sub-time windows according to the data volume of the business big data.

[0038] More specifically, to improve data processing efficiency, a thread pool is dynamically created according to the time window of the first timing task. The time window refers to the time range for the first task to process (such as 1 minute), which can be further divided into multiple sub-time windows (such as one sub-time window every 5 seconds). The number of threads in the thread pool is determined by the number of sub-time windows, and each thread is responsible for processing the data within one sub-time window.

[0039] Suppose the time window of the first scheduled task is 60 seconds, which is divided into 12 sub-time windows (each sub-time window is 5 seconds). Create a thread pool containing 12 threads, with each thread corresponding to a sub-time window to process the business data within that time period. This can significantly reduce the latency of data processing and make full use of system resources.

[0040] In S106, the business big data is processed by the threads in the thread pool. For example, corresponding sub-time windows can be set for each thread; each thread queries and processes the corresponding business big data within its sub-time window. Each thread in the thread pool independently processes the data within the allocated sub-time window. The processing tasks include querying the unprocessed data within the sub-time window from the database, executing business logic (such as updating status, calculating results, etc.), and caching or directly updating the processing results to the database. The threads run independently of each other, improving the parallel processing ability and significantly shortening the task completion time.

[0041] In one embodiment, it further includes: monitoring the processing tasks of each thread through the CountDownLatch synchronization tool; after the task processing of each thread is completed, continuing to execute or ending the first scheduled task.

[0042] In S108, the processing results are updated to the database. For example, the processed business big data can be marked as processed and updated to the database.

[0043] In the order processing scenario, each thread processes the order records within its responsible time period. For example, thread A processes the orders in the time period from 00:00:00 to 00:00:05, and thread B processes the orders in the time period from 00:00:05 to 00:00:10. After each thread completes its sub-task, the data status is marked as "processed".

[0044] In S110, the business big data that meets the preset conditions is obtained from the database through the second scheduled task. For example, the business big data that is unprocessed and whose last update time is greater than the time threshold is obtained from the database through the second scheduled task.

[0045] The second scheduled task is used to supplement the processing of data that was not completed in time. By setting a longer time interval (such as every 3 minutes), the system periodically queries the data that meets the preset conditions from the database, such as data that is unprocessed and whose last update time exceeds a certain threshold, and incorporates it back into the processing flow. This mechanism can make up for the omission problems caused by system load or abnormal conditions.

[0046] In a more specific application scenario, for example, in an order processing scenario, some orders may not be processed in the first scheduled task due to network anomalies or high load. The second scheduled task will periodically check these "unprocessed" orders and re - incorporate them into the processing queue to ensure eventual consistency.

[0047] In S112, process the business big data according to the time window of the second scheduled task. For example, process the business big data within the time window of the second scheduled task; mark the processed business big data as processed and update it to the database.

[0048] Similar to the first scheduled task, the second scheduled task processes business data according to its time window, but usually processes incomplete data with a longer time span. After the data processing is completed, it is marked as "processed" and updated to the database to ensure that the data does not enter the processing queue repeatedly.

[0049] In the order processing scenario, the second scheduled task can be set, for example, to process uncompleted orders whose last update time exceeds 5 minutes and update their status to "completed". This supplementary task ensures that all order data can be processed timely and efficiently.

[0050] Through the hierarchical design of scheduled tasks, combined with the parallel processing and dynamic adjustment mechanism of the thread pool, this method realizes an efficient big data processing process. The first scheduled task processes real - time data, and the second scheduled task supplements the missing data. The cooperation of the two ensures the timeliness, consistency, and high reliability of data processing. This design is applicable to big data processing scenarios with high concurrency and high load, such as order management, log analysis, and device monitoring.

[0051] According to the business big data processing method based on dual scheduled tasks of the present application, obtain business big data from the database through the first scheduled task; create a thread pool according to the time window of the first scheduled task, and the thread pool includes multiple threads; process the business big data through the threads in the thread pool; update the processing result to the database; obtain business big data that meets the preset conditions from the database through the second scheduled task; process the business big data according to the time window of the second scheduled task. In this way, it can significantly improve the real - time performance of data processing and the utilization efficiency of system resources. At the same time, through task dispersion, load balancing is achieved to ensure the stable operation of the system in high - load scenarios, guarantee system security, and can adapt to the requirements of various business scenarios.

[0052] It should be clearly understood that this application describes how to form and use specific examples, but the principles of this application are not limited to any details of these examples. On the contrary, based on the teachings of the content disclosed in this application, these principles can be applied to many other embodiments. <s

[0053] Figure 2 It is a flowchart of a business big data processing method based on dual timing tasks shown according to another exemplary embodiment. Figure 2 The shown process 20 is a detailed description of the first timing task process. The goal of this timing task is to achieve efficient data processing through parallel processing and resource optimization. The task is executed once per minute, and the one-minute time window is divided into multiple sub-time windows. Multiple threads in the thread pool are used to process the data within their respective time windows respectively, ultimately improving the real-time performance of data processing and reducing latency.

[0054] As Figure 2 shown, in S202, the first timing task is executed. A periodic task can be set using a task scheduling framework (such as Spring TaskScheduler) to trigger it once per minute. The trigger frequency of the timing task can be adjusted according to business requirements, but usually a shorter interval (such as 1 minute) is selected to meet the real-time requirements.

[0055] In S204, a thread pool is created. A periodic task is set using a task scheduling framework (such as Spring Task Scheduler) to trigger it once per minute. The trigger frequency of the timing task can be adjusted according to business requirements, but usually a shorter interval (such as 1 minute) is selected to meet the real-time requirements.

[0056] In S206, each thread in the thread pool queries and processes business data respectively. Specifically, it can include:

[0057] Query unprocessed data: Each thread queries the unprocessed data records within its assigned time window in the database according to the time window assigned to it. The query logic should be filtered based on clear time conditions and business status to ensure accurate acquisition of the data that needs to be processed.

[0058] Execute the processing logic: Each thread processes the queried data according to business requirements, such as calculation, status update, result saving, etc. During the data processing process, logical consistency needs to be ensured to avoid competition problems caused by parallel processing.

[0059] Update the processing status: After the data processing is completed, the thread updates the status of the corresponding data to "processed" to prevent repeated processing in the next round of tasks. This operation is usually completed through a database update statement, and logs should be recorded for auditing at the same time.

[0060] In S208, the processing results are summarized. A synchronization tool (such as CountDownLatch) can be used to monitor the execution progress of the threads.

[0061] The main thread will wait for all threads to complete before proceeding with subsequent logic or ending the current task. After all threads have completed their tasks, the main thread is responsible for recording processing logs, counting the amount of processed data, and handling exceptions. It releases the thread pool resources and prepares to enter the next task cycle.

[0062] In S210, it is determined whether the amount of processing is less than the total amount.

[0063] In S212, mark and end the current task.

[0064] In S214, continue to process the task.

[0065] Figure 3 It is a flowchart of a business big data processing method based on dual timing tasks shown according to another exemplary embodiment. Figure 3 The shown process 30 is a detailed description of the second timing task process. The design of this task is used to complement the deficiencies of the first timing task, ensuring that all data can be processed in a timely manner without missing key business data.

[0066] As Figure 3 shown, in S302, execute the second timing task. Use a task scheduling framework (such as Spring TaskScheduler) to set up a supplementary timing task, and the running frequency can be, for example, once every 3 minutes.

[0067] In S304, query the data for a preset time period. When the timing task starts, query the historical data that meets specific conditions (such as the status being "unprocessed" and the last update time being greater than 5 minutes) from the database, and re - add this data to the processing flow.

[0068] In S306, traverse and process the data.

[0069] More specifically, data that meets the following requirements can be queried from the database according to preset conditions:

[0070] The status is "unprocessed";

[0071] The last update time exceeds 5 minutes (or other configurable time thresholds).

[0072] This filtering condition can be dynamically adjusted to adapt to different business requirements and load conditions.

[0073] For the queried data, similar processing logic to the first timing task can be executed, or different processing logic can be executed. This application is not limited thereto.

[0074] After the data processing is completed, update the corresponding data status to "processed". Mark these data in the database to avoid duplicate processing and ensure that the processing results of the data can be traced.

[0075] In S308, it is determined whether there is still unprocessed data.

[0076] In one embodiment, an exception capture mechanism can also be added to each link of data processing to ensure that even if some data is skipped due to processing failure, the overall task can still be completed normally. Record the data with processing failures for subsequent analysis and manual intervention.

[0077] In one embodiment, the operation result of each task can also be recorded, including: the total amount of processed data; the number of successfully and failed processed data; the specific information and reasons of abnormal data.

[0078] In one embodiment, the log content can be used to generate reports or track the historical task status. If the system detects that the amount of unprocessed data continues to increase, it may mean that the load is too high or there is a system failure. The task can automatically trigger an alarm (such as sending an email or message to notify relevant personnel) to remind that further intervention measures need to be taken.

[0079] In S310, mark and end this task.

[0080] In S312, continue processing.

[0081] In this application, through dual timed tasks and combined with the parallel processing logic of the thread pool, this design achieves the following goals:

[0082] 1. Enhanced real-time performance:

[0083] By executing once per minute and using thread orchestration to disperse the processing process into multiple 5-second time periods, this timed task can significantly improve the real-time performance of data processing. It can quickly respond to newly generated unprocessed data and complete the processing in a very short time, reducing the waiting time of data in the system.

[0084] 2. Resource optimization:

[0085] Through parallel processing (i.e., using multiple threads), this task can more effectively utilize system resources such as CPU and IO. At the same time, since each thread only processes data within a specific time period, this helps to reduce the lock contention for database queries and updates, thereby improving the overall performance.

[0086] 3. Load balancing:

[0087] Dispersing the data processing task into multiple threads helps to balance the system load. This can prevent a single thread from becoming a performance bottleneck and ensure that the system can run smoothly, maintaining good responsiveness even under high load conditions.

[0088] 4. Data consistency:

[0089] By promptly querying and processing unprocessed data, this task helps maintain data consistency and accuracy. It reduces the risk of errors or inconsistencies that may occur due to long-term unprocessed data.

[0090] 5. Flexibility:

[0091] This design also provides flexibility to adjust the number of threads, time intervals, or processing logic as needed to adapt to different business requirements or system environments.

[0092] Those skilled in the art can understand that all or part of the steps for implementing the above embodiments are realized as a computer program executed by a CPU. When the computer program is executed by the CPU, the above functions defined by the above method provided in this application are executed. The program can be stored in a computer-readable storage medium, which can be a read-only memory, a magnetic disk, an optical disc, etc.

[0093] In addition, it should be noted that the above drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of this application, rather than for limiting purposes. It is easy to understand that the processes shown in the above drawings do not indicate or limit the time sequence of these processes. Additionally, it is also easy to understand that these processes can be executed synchronously or asynchronously in, for example, multiple modules.

[0094] The following is an embodiment of the apparatus of this application, which can be used to execute the embodiment of the method of this application. For details not disclosed in the embodiment of the apparatus of this application, please refer to the embodiment of the method of this application.

[0095] Figure 4 is a block diagram of a business big data processing system based on dual timing tasks shown according to an exemplary embodiment. As Figure 4 shown, the business big data processing system 40 based on dual timing tasks includes: a first task module 402, a thread pool module 404, a thread module 406, an update module 408, a second task module 410, and a processing module 412.

[0096] The first task module 402 is used to obtain business big data from the database through the first timing task; the first task module 402 is also used to set the first timing task based on the Spring Task Scheduler framework; and pull the business big data from the database periodically through the first timing task.

[0097] The thread pool module 404 is used to create a thread pool according to the time window of the first scheduled task. The thread pool includes multiple threads. The thread pool module 404 is also used to divide the time window of the first scheduled task into multiple sub-time windows, determine the number of threads according to the number of the sub-time windows, and establish the thread pool according to the number of threads.

[0098] The thread module 406 is used to process the business big data through the threads in the thread pool. The thread module 406 is also used to set corresponding sub-time windows for each thread. Each thread queries and processes the corresponding business big data within its sub-time window.

[0099] The update module 408 is used to update the processing result to the database. The update module 408 is also used to mark the processed business big data as processed and update it to the database.

[0100] The second task module 410 is used to obtain the business big data that meets the preset conditions from the database through the second scheduled task. The second task module 410 is also used to obtain the unprocessed business big data whose last update time is greater than the time threshold from the database through the second scheduled task.

[0101] The processing module 412 is used to process the business big data according to the time window of the second scheduled task. The processing module 412 is also used to process the business big data within the time window of the second scheduled task, mark the processed business big data as processed, and update it to the database.

[0102] According to the business big data processing system based on dual scheduled tasks of the present application, business big data is obtained from the database through the first scheduled task, a thread pool is created according to the time window of the first scheduled task, the thread pool includes multiple threads, the business big data is processed through the threads in the thread pool, the processing result is updated to the database, the business big data that meets the preset conditions is obtained from the database through the second scheduled task, and the business big data is processed according to the time window of the second scheduled task. In this way, the real-time performance of data processing and the utilization efficiency of system resources can be significantly improved. At the same time, load balancing is achieved through task dispersion to ensure the stable operation of the system in high-load scenarios, ensure system security, and adapt to the requirements of various business scenarios.

[0103] As Figure 5 shown, an embodiment of the present application provides an electronic device, including a processor 510, a memory 520, and a bus. Among them, the processor 510 and the memory 520 complete communication with each other through the bus 540;

[0104] The memory 520 is used to store a computer program;

[0105] A processor 510, when executing a program stored in a memory 520, implements the method for processing business big data based on dual timing tasks according to any of the above embodiments.

[0106] A communication interface 520 is used for communication between the above electronic device and other devices.

[0107] The memory 520 may include a random access memory 520 (Random Access Memory, abbreviated as RAM), and may also include a non-volatile memory 520 (non-volatile memory), such as at least one disk memory 520. Optionally, the memory 520 may also be at least one storage device located far from the aforementioned processor 510.

[0108] If the above method in this application is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above embodiment methods of this application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc.

[0109] The embodiments of this application provide a computer-readable storage medium. The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method for processing business big data based on dual timing tasks according to any of the above embodiments. For example, business big data can be obtained from a database through a first timing task; a thread pool including multiple threads is created according to the time window of the first timing task; the business big data is processed through the threads in the thread pool; the processing result is updated to the database; business big data meeting preset conditions is obtained from the database through a second timing task; and the business big data is processed according to the time window of the second timing task.

[0110] The above specifically shows and describes the exemplary embodiments of this application. It should be understood that this application is not limited to the detailed structures, setting manners or implementation methods described here; on the contrary, this application intends to cover various modifications and equivalent settings included within the spirit and scope of the appended claims.

Claims

1. A business big data processing system based on dual timing tasks, characterized in that, including: A first task module, configured to obtain business big data from a database through a first scheduled task; A thread pool module, configured to create a thread pool according to the time window of the first scheduled task, where the thread pool includes multiple threads; A thread module, configured to process the business big data through the threads in the thread pool; An update module, configured to update the processing result to the database; A second task module, configured to obtain business big data that meets preset conditions from the database through a second scheduled task; A processing module, configured to process the business big data according to the time window of the second scheduled task.

2. A business big data processing method based on dual timing tasks, characterized in that, including: Obtain business big data from a database through a first scheduled task; Create a thread pool according to the time window of the first scheduled task, where the thread pool includes multiple threads; Process the business big data through the threads in the thread pool; Update the processing result to the database; Obtain business big data that meets preset conditions from the database through a second scheduled task; Process the business big data according to the time window of the second scheduled task.

3. The method according to claim 2, wherein Obtaining business big data from a database through a first scheduled task includes: Setting a first scheduled task based on the Spring Task Scheduler framework; Periodically pulling the business big data from the database through the first scheduled task.

4. The method according to claim 2, wherein Creating a thread pool according to the time window of the first scheduled task, where the thread pool includes multiple threads, includes: Dividing the time window of the first scheduled task into multiple sub-time windows; Determining the number of threads according to the number of the sub-time windows; Establishing the thread pool according to the number of threads.

5. The method according to claim 4, wherein Dividing the time window of the first scheduled task into multiple sub-time windows includes: Dividing the time window of the first scheduled task into multiple sub-time windows according to the data volume of the business big data.

6. The method according to claim 2, wherein Processing the business big data through the threads in the thread pool includes: Setting corresponding sub-time windows for each thread; Each thread queries and processes the corresponding business big data within its sub-time window.

7. The method according to claim 2, wherein Processing the business big data through the threads in the thread pool further includes: Monitoring the processing tasks of each thread through the CountDownLatch synchronization tool; After the task processing of each thread is completed, continuing to execute or ending the first scheduled task.

8. The method according to claim 2, wherein Updating the processing result to the database includes: Marking the processed business big data as processed and updating it to the database.

9. The method according to claim 2, wherein Obtaining business big data that meets preset conditions from the database through a second scheduled task includes: Obtaining unprocessed business big data whose last update time is greater than a time threshold from the database through a second scheduled task.

10. The method according to claim 2, wherein Processing the business big data according to the time window of the second scheduled task includes: Processing the business big data within the time window of the second scheduled task; Marking the processed business big data as processed and updating it to the database.

11. An electronic device, characterized in that, including: One or more processors; A storage device, configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 2 to 10.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, the method according to any one of claims 2 to 10 is implemented.

13. A computer program product, characterized in that, Comprising a computer program / instructions which, when executed by a processor, implement the steps of the method according to any one of claims 2 to 10.

Images