Multi-priority multi-task data processing method and system based on database change capture triggering

Abstract

The invention discloses a multi-priority multi-task data processing method and system based on database change capture triggering, and relates to the field of data processing. The problems that an existing database change capture technology lacks task priority management, resource contention, insufficient task management functions and lacks support for multi-task parallel processing are solved. The method comprises the following steps: performing registration change monitoring, configuration task scheduling and task management through a task configuration module; monitoring a database auditing column and a transaction log or receiving a database change notification message sent by the database through a database change capturing module, capturing insertion, updating and deletion operations in the database, and generating a change event; the task scheduling module is used for performing priority marking and arrangement on the tasks based on a priority rule configured in the task configuration module, and scheduling in sequence; and the task execution module provides resources for the tasks, executes task types and provides a task management function. The method can be applied to data calculation systems in the fields of medical treatment, finance, logistics and the like.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and particularly to a multi-priority multi-task data processing method triggered by database change capture. Background Art

[0002] With the continuous development of database technology, the quantity and complexity of enterprise data have been increasing. How to achieve real-time computing and efficient processing of large-scale data has become an important technical challenge. In modern data processing systems, the database, as the core component, carries a large amount of business data. As the business scale expands and the data volume grows, database changes (such as insert, update, and delete operations) become increasingly frequent. To respond to these change events in real time, many systems adopt database change capture technology, which captures data changes and triggers corresponding data processing tasks by monitoring the audit logs, transaction logs of the database, or receiving change notification messages sent by the database.

[0003] However, the existing database change capture (CDC) technology has the following problems in practical applications:

[0004] 1. Lack of task priority management:

[0005] In the existing technical solutions, all captured database change events are usually treated equally, and the system processes them in the order of event occurrence. However, in actual business scenarios, the importance and urgency of different tasks may vary significantly. For example, some tasks may involve core business logic and need to be processed immediately, while other tasks may be non-critical and can be processed later. The existing technology lacks a task priority management mechanism, resulting in key tasks being possibly delayed, affecting the system's response speed and business continuity.

[0006] 2. Resource contention problem:

[0007] In high-concurrency scenarios, multiple tasks may be triggered simultaneously and compete for system resources (such as CPU, memory, network bandwidth, etc.). The existing technology usually adopts a simple task scheduling strategy and cannot be dynamically adjusted according to the task priority and resource usage. This may lead to uneven resource allocation, with high-priority tasks being delayed due to insufficient resources, while low-priority tasks occupy a large amount of resources, thus reducing the overall performance and efficiency of the system.

[0008] 3. Insufficient task management functions:

[0009] The management function of the prior art for the task execution process is relatively limited, and usually lacks a flexible task control mechanism. For example, when an exception occurs during the task execution process or the system resources are insufficient, operations such as pausing, stopping, resuming, or retrying the task cannot be performed. This may lead to task execution failure or resource waste, reducing the stability and reliability of the system.

[0010] 4. Lack of support for multi-task parallel processing:

[0011] When dealing with a large number of database change events, the prior art usually adopts a serial processing method, that is, only one task is processed at a time. This method has obvious bottlenecks in processing efficiency and cannot fully utilize multi-core processors and distributed computing resources. Although some technical solutions attempt to introduce a parallel processing mechanism, due to the lack of effective task scheduling and resource management strategies, problems such as resource contention and task conflicts often occur. Summary of the Invention

[0012] In view of this, the present invention aims to propose a method and system for multi-priority multi-task data processing triggered by database change capture to solve the problems existing in the prior database change capture technology, such as lack of task priority management, resource contention problems, insufficient task management functions, and lack of support for multi-task parallel processing.

[0013] To achieve the above object, the present invention adopts the following technical solutions:

[0014] A method for multi-priority multi-task data processing triggered by database change capture, the method includes:

[0015] Task configuration step: Register change listening, configure task scheduling and task management through the task configuration module. The register change listening function issues a listening task to the database change capture module. The configure task scheduling function marks each task as high-priority, medium-priority, or low-priority according to its priority. The task management function manages the tasks being executed;

[0016] Database change capture step: Monitor the database audit column, transaction log, or receive the database change notification message sent by the database through the database change capture module, capture the insert, update, and delete operations in the database, and generate change events;

[0017] Task scheduling step: Based on the priority rules configured in the task configuration module, mark and arrange the tasks according to their priorities, and schedule them in sequence;

[0018] Task execution step: Provide resources for the task through the task execution module, execute the task type, and provide the task management function.

[0019] Further, a preferred method is also proposed. The task configuration module performs database change monitoring configuration through an interface or a graphical operation interface, formats the configuration content, encapsulates the configuration information, and transmits it to the database change capture module by means of function call or message notification.

[0020] Further, a preferred method is also proposed. The database change capture module receives the configuration information sent by the task configuration module, parses the configuration information, generates corresponding database change monitoring tasks, monitors the corresponding database. If data changes occur in the source database, database change events are generated, and the monitoring tasks generate data processing tasks for the change events.

[0021] Further, a preferred method is also proposed. The task scheduling module receives the configuration information sent by the task configuration module, parses the configuration information, stores the correspondence between priorities and tasks, and schedules tasks according to priority rules. Among them, high-priority tasks are scheduled first, medium-priority tasks are scheduled next, and low-priority tasks are scheduled last.

[0022] Further, a preferred method is also proposed. The task execution module receives the configuration information sent by the task configuration module, parses the configuration information, starts, stops, and retries corresponding tasks according to the configuration, and dynamically adjusts resource allocation during the task execution process.

[0023] Based on the same inventive concept, the present invention also proposes a multi-priority multi-task data processing system triggered by database change capture. The system includes:

[0024] A task configuration module, used for registering change monitoring, configuring task scheduling, and task management;

[0025] A database change capture module, used for monitoring database audit columns, transaction logs, or receiving database change notification messages sent by the database, capturing insert, update, and delete operations in the database, and generating change events;

[0026] A task scheduling module, used for performing priority marking and scheduling of tasks based on the priority rules configured in the task configuration module, and scheduling them in sequence;

[0027] A task execution module, used for providing resources for tasks, executing task types, and providing task management functions.

[0028] Further, a preferred method is also proposed. The task configuration module performs database change monitoring configuration through an interface or a graphical operation interface, formats the configuration content, encapsulates it into configuration information, and transmits it to the database change capture module by means of function call or message notification.

[0029] Furthermore, a preferred method is proposed. The database change capture module receives the configuration information sent by the task configuration module, parses the configuration information to generate corresponding database change monitoring tasks, monitors the corresponding databases. If data changes occur in the source database, database change events are generated, and the monitoring tasks generate data processing tasks for the change events.

[0030] Based on the same inventive concept, the present invention also provides a computer device, including a memory and a processor. A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a method for triggering multi-priority multi-task data processing based on database change capture as described in any one of the above.

[0031] Based on the same inventive concept, the present invention also provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium. When the computer program is run by a processor, it executes the steps of a method for triggering multi-priority multi-task data processing based on database change capture as described in any one of the above.

[0032] The advantages of the present invention are as follows:

[0033] The method proposed by the present invention can mark and schedule tasks according to the importance and urgency of tasks by introducing a task priority management mechanism, ensuring that critical tasks are processed first. It significantly improves the system's response speed and business continuity, avoids the problem of critical tasks being delayed, and enhances the system's reliability and user experience.

[0034] During the task scheduling process of the method proposed by the present invention, the system can dynamically adjust the task execution order according to the resource usage situation to avoid resource contention. It optimizes the system's resource allocation, improves resource utilization, ensures that high-priority tasks can obtain resources first, and enhances the overall performance and efficiency of the system. The task execution module provides flexible task management functions, including pause, stop, resume, and retry operations. It enhances the flexibility and stability of the system, can effectively handle abnormal situations during task execution, reduces task failures and resource waste, and improves the system's reliability.

[0035] The method proposed by the present invention supports multi-task parallel processing. Through the collaborative work of the task scheduling module and the task execution module, it ensures that tasks can be executed in parallel efficiently. It significantly improves the processing efficiency of the system, makes full use of multi-core processors and distributed computing resources, avoids resource contention and task conflicts, and enhances the overall performance of the system. Through the flexible configuration of the task configuration module and the task scheduling module, it supports complex task dependencies and cross-system collaborative processing. It meets the actual needs of complex business scenarios, improves the applicability and practicality of the technology, can handle various complex business logics and data interactions, and enhances the adaptability of the system.

[0036] The method proposed by the present invention uses the database change capture module to monitor database change events in real time and generate corresponding data processing tasks, ensuring that the system can respond to data changes in a timely manner. It significantly improves the real-time performance and business continuity of the system, ensures that data processing tasks can be triggered in a timely manner, and guarantees the real-time performance and reliability of the business. The task execution module provides resources for tasks, specifically executes tasks, and dynamically adjusts resource allocation during task execution to optimize execution efficiency. It significantly improves the task execution efficiency and the overall performance of the system, ensures that tasks can be completed efficiently, and enhances the processing capacity and response speed of the system.

[0037] The present invention can be applied to data computing systems in fields such as healthcare, finance, and logistics. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Flowchart of a method for triggering multi-priority multi-task data processing based on database change capture according to Embodiment 1;

[0039] Figure 2 Schematic diagram of the data processing system according to Embodiment 11. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention.

[0041] Embodiment 1. Refer to Figure 1 This embodiment will be described. A method for triggering multi-priority multi-task data processing based on database change capture according to this embodiment, the method includes:

[0042] Task configuration steps: Register change monitoring, configure task scheduling, and manage tasks through the task configuration module. The register change monitoring function issues monitoring tasks to the database change capture module. The configure task scheduling function marks each task as high-priority, medium-priority, or low-priority according to its priority. The task management function manages the tasks that are being executed;

[0043] Database change capture steps: Monitor the database audit columns, transaction logs, or receive database change notification messages sent by the database through the database change capture module, capture insert, update, and delete operations in the database, and generate change events;

[0044] Task scheduling steps: Based on the priority rules configured in the task configuration module, mark and arrange the priorities of tasks, and schedule them in sequence;

[0045] Task execution steps: Provide resources for tasks through the task execution module, execute the task type, and provide task management functions.

[0046] In this embodiment, by introducing a task priority management mechanism, tasks can be marked and scheduled according to their importance and urgency, ensuring that critical tasks are processed first. Significantly improves the system's response speed and business continuity, avoids the problem of critical tasks being delayed, and enhances the system's reliability and user experience.

[0047] Furthermore, during the task scheduling process, the system can dynamically adjust the task execution order according to the resource usage situation to avoid resource occupation. Optimizes the system's resource allocation, improves resource utilization, ensures that high-priority tasks can obtain resources first, and enhances the overall performance and efficiency of the system. The task execution module provides flexible task management functions, including pause, stop, resume, and retry operations. Enhances the flexibility and stability of the system, can effectively handle abnormal situations during task execution, reduces task failures and resource waste, and improves the system's reliability.

[0048] Furthermore, the method proposed in this embodiment supports multi-task parallel processing, and through the collaborative work of the task scheduling module and the task execution module, ensures that tasks can be executed in parallel efficiently. Significantly improves the system's processing efficiency, makes full use of multi-core processors and distributed computing resources, avoids resource contention and task conflicts, and enhances the overall performance of the system. Through the flexible configuration of the task configuration module and the task scheduling module, it supports complex task dependencies and cross-system collaborative processing. Meets the actual needs of complex business scenarios, enhances the applicability and practicality of the technology, can handle various complex business logics and data interactions, and enhances the adaptability of the system.

[0049] Furthermore, the database change capture module monitors database change events in real time and generates corresponding data processing tasks to ensure that the system can respond to data changes in a timely manner. This significantly improves the real-time performance and business continuity of the system, ensures that data processing tasks can be triggered in a timely manner, and guarantees the real-time performance and reliability of the business. The task execution module provides resources for tasks, specifically executes tasks, and dynamically adjusts resource allocation during task execution to optimize execution efficiency. This significantly improves task execution efficiency and the overall performance of the system, ensures that tasks can be completed efficiently, and enhances the processing capacity and response speed of the system.

[0050] Embodiment 2: This embodiment further limits a multi-priority and multi-task data processing method triggered by database change capture described in Embodiment 1. The task configuration module configures database change monitoring through an interface or a graphical operation interface, formats the configuration content, encapsulates the configuration information, and transmits the configuration information to the database change capture module through function calls or message notifications.

[0051] Specifically, in this embodiment, database change monitoring is configured through an interface or a graphical operation interface, enabling users to set and manage the change monitoring function more intuitively and conveniently. Users do not need to deeply understand the underlying code or complex configuration files and can configure through the graphical interface, improving the operability and usability of the system and reducing the usage threshold.

[0052] The task configuration module can be flexibly configured through the interface. Users can customize the parameters of database change monitoring and task scheduling rules according to actual needs. This flexibility enables the system to quickly adapt to changes in different business scenarios, enhancing the adaptability and scalability of the system. The task configuration module formats the configuration content and encapsulates the configuration information, making the transmission of configuration information more standardized and unified. Through formatting and encapsulation, format errors in configuration files or data can be avoided, improving the stability and fault tolerance of the system. The encapsulated configuration information can also be more conveniently transmitted and managed. In this embodiment, the configuration information is also transmitted to the database change capture module through function calls or message notifications. This efficient transmission mechanism can ensure that the configuration information is transmitted to the change capture module in a timely and accurate manner, reducing delays and errors in the data transmission process, and improving the real-time performance and reliability of the system.

[0053] Embodiment 3. This embodiment further limits a method for triggering multi-priority multi-task data processing based on database change capture described in Embodiment 1. The database change capture module receives the configuration information sent by the task configuration module, parses the configuration information, generates corresponding database change monitoring tasks, monitors the corresponding databases. If data changes occur in the source database, a database change event is generated, and the monitoring task generates a data processing task for the change event.

[0054] In this embodiment, the database change capture (CDC) module can capture data change events in the source database in real time. When the data is changed, the system can immediately respond and trigger corresponding data processing tasks. Compared with the traditional batch processing method, CDC greatly improves the real-time performance of data processing, significantly improving the timeliness of data flow. The task configuration module supports dynamic configuration of tasks, enabling users to flexibly adjust task priorities and task types according to actual needs. For example, users can adjust the priorities of data processing tasks according to different business scenarios to ensure that critical tasks are processed in a timely manner. The parsing of configuration information and the generation of database change monitoring tasks mean that the system can flexibly adjust monitoring and processing strategies according to different data sources and change contents, thus supporting the requirements of different data sources and application scenarios. Multi-priority task scheduling can reasonably arrange tasks according to the urgency and importance of different tasks, giving priority to high-priority tasks to ensure that critical tasks can be completed in a timely manner. This method avoids low-priority tasks occupying too many resources, thus ensuring the effective utilization of system resources and reducing the system burden.

[0055] The method proposed in this embodiment supports multi-task parallel processing. By generating multiple independent data processing tasks for data change events, the parallel computing capabilities of multi-core processors or distributed systems can be fully utilized, effectively improving processing capabilities and reducing the response time of individual tasks. Data processing tasks can be executed in parallel on different processing threads or servers, thus improving the overall throughput of the system in high-concurrency scenarios.

[0056] Embodiment 4. This embodiment further limits a method for triggering multi-priority multi-task data processing based on database change capture described in Embodiment 1. The task scheduling module receives the configuration information sent by the task configuration module, parses the configuration information, stores the correspondence between priorities and tasks, and schedules tasks according to priority rules, where high-priority tasks are scheduled first, medium-priority tasks are scheduled second, and low-priority tasks are scheduled last.

[0057] In this embodiment, tasks are scheduled according to the priority rule, and high-priority tasks are processed preferentially, which can ensure that critical tasks or important data are completed in a short time, improving the system's response speed and processing efficiency. This priority scheduling strategy can effectively avoid the occupation of system resources by low-priority tasks and ensure that important tasks can still be executed under high load.

[0058] The configuration information received by the task scheduling module is parsed and stores the corresponding relationship between priorities and tasks, enabling the system to flexibly manage the relationships between different tasks. By dynamically adjusting the priorities of tasks, optimization can be carried out according to actual requirements. For example, when the importance of certain tasks changes, the system can quickly adjust the priorities of tasks without the need to redesign the scheduling logic, thus improving the flexibility and scalability of the system.

[0059] In a multi-task environment, resource scheduling and allocation are a key issue. By scheduling according to task priorities, high-priority tasks can be executed quickly, thus enabling the effective utilization of system resources. Although low-priority tasks do not preempt system resources, they will still be executed after high-priority tasks are completed, thus maximizing the utilization of resources and avoiding the idle or ineffective use of system resources.

[0060] Task scheduling based on the priority rule can give the system a clearer expectation of the execution time of tasks. Since high-priority tasks will be scheduled first, when the system faces sudden tasks or large-scale data processing, it can ensure that critical tasks are completed within the specified time through the predefined priority rule, thus improving the stability and reliability of the system.

[0061] In the context of database change capture, when the system receives configuration information from the task configuration module, the scheduling module can process high-priority tasks as soon as possible according to the priority rule, thus reducing the delay in task processing. Especially in scenarios that require quick response, this method can effectively improve the real-time performance of the system and reduce the response time.

[0062] Embodiment 5: This embodiment further limits a multi-priority multi-task data processing method triggered by database change capture described in Embodiment 1. The task execution module receives the configuration information sent by the task configuration module, parses the configuration information, starts, stops, and retries the corresponding tasks according to the configuration, and dynamically adjusts the resource allocation during the task execution process.

[0063] In this embodiment, based on the configuration information sent by the task configuration module, the task execution module can start, stop, and retry tasks according to real-time requirements, enabling the system to more precisely control the task execution process, dynamically adjust tasks according to actual business requirements or external conditions, and avoid blind or excessive execution of tasks.

[0064] During the task execution process, the resource allocation can be dynamically adjusted according to the running status and priority of the tasks. For example, tasks with high priority can obtain more resources, while tasks with low priority can be appropriately postponed or suspended when resources are limited. This dynamic nature of resource scheduling enables the system to more efficiently utilize limited computing and storage resources, avoiding resource waste and bottleneck problems.

[0065] The task execution module not only starts tasks according to the configuration but also can dynamically adjust the strategy according to the execution status of the tasks. This flexible scheduling method can reasonably allocate and schedule task execution based on various factors such as system load, task priority, and urgency, ensuring the maximization of the overall system performance.

[0066] By supporting the retry operation of tasks, when a certain task fails to execute due to certain reasons, the system can automatically retry according to the configuration requirements, improving the stability and reliability of the system. This fault tolerance mechanism avoids the situation where the entire process is interrupted due to the failure of a single task, thus ensuring the continuity of system processing and data integrity.

[0067] Embodiment Six. A multi-priority multi-task data processing system based on database change capture trigger described in this embodiment, the system includes:

[0068] A task configuration module for registering change monitoring, configuring task scheduling, and task management;

[0069] A database change capture module for monitoring database audit columns, transaction logs, or receiving database change notification messages sent by the database, capturing insert, update, and delete operations in the database, and generating change events;

[0070] A task scheduling module for marking and arranging the priorities of tasks based on the priority rules configured in the task configuration module, and scheduling them in sequence;

[0071] A task execution module for providing resources for tasks, executing task types, and providing task management functions.

[0072] Embodiment 7. This embodiment further defines a multi-priority multi-task data processing system triggered by database change capture described in Embodiment 6. The task configuration module configures database change monitoring through an interface or a graphical operation interface, formats the configuration content, encapsulates it into configuration information, and transfers it to the database change capture module by function call or message notification.

[0073] Embodiment 8. This embodiment further defines a multi-priority multi-task data processing system triggered by database change capture described in Embodiment 6. The database change capture module receives the configuration information sent by the task configuration module, parses the configuration information, generates corresponding database change monitoring tasks, monitors the corresponding databases. If data changes occur in the source database, database change events are generated, and the monitoring tasks generate data processing tasks for the change events.

[0074] Embodiment 9. A computer device described in this embodiment includes a memory and a processor. A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a multi-priority multi-task data processing method described in any one of Embodiments 1 to 5.

[0075] Embodiment 10. A computer-readable storage medium described in this embodiment has a computer program stored thereon. When the computer program is run by a processor, it executes the steps of a multi-priority multi-task data processing method described in any one of Embodiments 1 to 5.

[0076] Embodiment 11. Refer to Figure 2 Describe this embodiment. This embodiment provides a specific example for a multi-priority multi-task data processing method triggered by database change capture described in Embodiment 1, and is also used to explain Embodiments 2 to 5. Specifically:

[0077] Take Figure 2 as an example. The system has three source databases: Source Database 1, Source Database 2, and Source Database 3, and three target databases: Target Database 1, Target Database 2, and Target Database 3. Now it is required to record the data changes of Source Data 3 into Target Database 1, the changes of Source Database 2 into Target Database 2, and the changes of Source Data 1 into Target Database 3. And the database change processing priority of Source Database 3 is the highest, the database change processing priority of Source Database 2 is the second, and the database change processing priority of Source Database 3 is the lowest. The specific process is as follows:

[0078] (1) Task configuration process:

[0079] Change monitoring configuration:

[0080] Perform database change monitoring configuration through the interfaces or graphical operation interfaces provided by the task configuration module: monitoring configuration for change event 1 issued by source database 1, monitoring configuration for change event 2 issued by source database 2, monitoring configuration for change event 3 issued by source database 3;

[0081] The task configuration module formats the configuration content, encapsulates it into configuration information, and passes the configuration information to the database change capture module through function calls, message notifications, etc.;

[0082] The database change capture module receives the configuration information sent by the task configuration module, parses the configuration information, generates corresponding database change monitoring tasks for the monitoring configuration of change event 1 issued by source database 1, the monitoring configuration of change event 2 issued by source database 2, and the monitoring configuration of change event 3 issued by source database 3, monitors the corresponding databases. If data changes occur in the source database, generating database change events, the monitoring tasks will generate data processing tasks 1, data processing tasks 2, and data processing tasks 3 for change event 1, change event 2, and change event 3 respectively.

[0083] (2) Task priority configuration:

[0084] Perform task priority configuration through the interfaces or graphical operation interfaces provided by the task configuration module: set the priority of data processing task 3 generated by database change event 3 to 1, set the priority of data processing task 2 generated by database change event 2 to 2, and set the priority of data processing task 1 generated by database change event 1 to 3;

[0085] The task configuration module formats the configuration content, encapsulates it into configuration information, and passes the configuration information to the task scheduling module through function calls, message notifications, etc.;

[0086] The task scheduling module receives the configuration information sent by the task configuration module, parses the configuration information, and stores the corresponding relationship between priorities and tasks.

[0087] Task management:

[0088] Perform task management operations through the interfaces or graphical operation interfaces provided by the task configuration module, generating task management events such as task start, task stop, and task retry;

[0089] The task configuration module formats the task management events, encapsulates them into task management information, and passes the configuration information to the task execution module through function calls, message notifications, etc.;

[0090] The task execution module receives the configuration information sent by the task configuration module, parses the configuration information, and starts, stops, or retries the corresponding tasks according to the configuration.

[0091] (3) Task execution process:

[0092] Suppose data changes occur in data sources 1, 2, and 3, generating database change events 1, 2, and 3 respectively;

[0093] The database change capture module receives database change events 1, 2, and 3 according to the configuration and generates data processing tasks 1, 2, and 3 respectively;

[0094] The task scheduling module marks the priorities of data processing task 3, data processing task 2, and data processing task 1 as 1, 2, and 3 respectively according to the configuration;

[0095] If data processing tasks 3, 2, and 1 arrive simultaneously, the task scheduling module first sends data processing task 3 to the task execution module for execution, then sends data processing task 2 to the task execution module for execution, and finally sends data processing task 1 to the task execution module for execution;

[0096] If data processing task 1 arrives first, the task scheduling module first sends data processing task 1 to the task execution module for execution. If data processing task 2 or data processing task 3 arrives during the execution of data processing task 1, and if resources are sufficient at this time, the task scheduling module will send data processing task 3 and data processing task 2 to the task execution module for execution in sequence; if resources are insufficient at this time and only one task can be executed, the task scheduling module will suspend the execution of data processing task 1 and send data processing task 3 and data processing task 2 to the task execution module for execution in sequence, and resume the execution of data processing task 1 after data processing tasks 3 and 2 are executed.

[0097] Although the preferred embodiments of the present disclosure have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the present disclosure.

[0098] Obviously, those skilled in the art can make various changes and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalent technologies, the present disclosure also intends to include these modifications and variations.

[0099] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] The present disclosure is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks. These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means realizes the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, so that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable devices provide steps for realizing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure rather than to limit the scope of its protection. Although the present disclosure has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that after reading the present disclosure, various changes, modifications, or equivalent replacements can still be made to the specific implementation manners of the invention, but these changes, modifications, or equivalent replacements are all within the scope of the protection of the pending claims of the disclosure.

Claims

1. A method for multi-priority and multi-task data processing based on database change capture triggering, characterized in that: The method comprises: Task configuration step: register change monitoring, configure task scheduling and task management through the task configuration module. The registration change monitoring function sends monitoring tasks to the database change capture module, configures the task scheduling function to mark each task as high priority, medium priority or low priority according to the priority, and the task management function manages the tasks being executed; Database change capture step: Monitor database audit columns, transaction logs, or receive database change notification messages from the database through the database change capture module, capture insert, update, and delete operations in the database, and generate change events; Task scheduling steps: based on the priority rules configured in the task configuration module, the tasks are marked and arranged with priority, and scheduled in order; Task execution steps: The task execution module provides resources for the task, executes the task type, and provides task management functions.

2. The method for multi-priority and multi-task data processing based on database change capture triggering according to claim 1 is characterized in that: The task configuration module performs database change monitoring configuration through an interface or a graphical operation interface, formats the configuration content, encapsulates the configuration information, and transmits it to the database change capture module through a function call or a message notification.

3. The method for multi-priority and multi-task data processing based on database change capture triggering according to claim 1 is characterized in that: The database change capture module receives the configuration information sent by the task configuration module, parses the configuration information, generates a corresponding database change monitoring task, monitors the corresponding database, and if data changes occur in the source database, a database change event is generated, and the monitoring task generates a data processing task for the change event.

4. The method for multi-priority and multi-task data processing based on database change capture triggering according to claim 1 is characterized in that: The task scheduling module receives the configuration information sent by the task configuration module, parses the configuration information, stores the correspondence between priority and task, and schedules tasks according to priority rules, wherein high priority tasks are scheduled first, medium priority tasks are scheduled second, and low priority tasks are scheduled last.

5. The method for multi-priority and multi-task data processing based on database change capture triggering according to claim 1 is characterized in that: The task execution module receives the configuration information sent by the task configuration module, parses the configuration information, starts, stops, and retries the corresponding tasks according to the configuration, and dynamically adjusts resource allocation during task execution.

6. A multi-priority and multi-task data processing system based on database change capture triggering, characterized in that: The system comprises: Task configuration module, used to register change monitoring, configure task scheduling and task management; The database change capture module is used to monitor the database audit column, transaction log or receive the database change notification message sent by the database, capture the insert, update and delete operations in the database, and generate change events; The task scheduling module is used to mark and arrange the tasks according to the priority rules configured in the task configuration module, and schedule them in order; The task execution module is used to provide resources for tasks, execute task types, and provide task management functions.

7. A multi-priority and multi-task data processing system based on database change capture triggering according to claim 6, characterized in that: The task configuration module performs database change monitoring configuration through an interface or a graphical operation interface, formats the configuration content, encapsulates it into configuration information, and transmits it to the database change capture module through a function call or a message notification.

8. The multi-priority and multi-task data processing system based on database change capture triggering according to claim 6 is characterized in that: The database change capture module receives the configuration information sent by the task configuration module, parses the configuration information, generates a corresponding database change monitoring task, monitors the corresponding database, and if data changes occur in the source database, a database change event is generated, and the monitoring task generates a data processing task for the change event.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes a multi-priority and multi-task data processing method triggered by database change capture according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of a method for triggering multi-priority and multi-task data processing based on database change capture as claimed in any one of claims 1 to 5.

Images