Database change event dynamic routing processing method, system, equipment and medium

Abstract

The invention provides a database change event dynamic routing processing method, system and device and a medium, and belongs to the technical field of data processing and business logic management. The method comprises the following steps: configuring a Debezium connector to monitor a change event of a specific table of a database; converting the change event into a standard JSON format file by using a JSON converter; a Kafka message queue is configured, and a change event is routed to a dynamically generated Kafka theme through a RegexRouter; the method comprises the following steps of: establishing an independent SpringCloud Function project to consume a Kafka theme; the method comprises the following steps of: managing and calling a predefined function chain through Function Catalog; a function chain is used for connecting a plurality of independent data processing functions in series, and sequence processing of the changed data is achieved. The invention provides a database change event processing scheme capable of decoupling service logic and supporting dynamic routing and flexible function management.

Description

Technical Field

[0001] This invention belongs to the field of data processing and business logic management technology, and more specifically relates to a method, system, device and medium for dynamic routing processing of database change events. Background Technology

[0002] In public cloud service provider scenarios involving user orders and billing, the timely and accurate delivery of data to users or backend monitoring platforms has become a critical requirement due to increasingly complex business logic and rising monitoring and auditing requirements. However, traditional technologies rely on stacked code to call interfaces, leading to an exponential increase in system complexity. Taking order and billing services as examples, these businesses need to implement user notification functions while also integrating with backend overselling monitoring systems. The coupling of multiple logics results in a chaotic code structure that is difficult to separate and maintain later. Furthermore, the lack of flexibility in interface calling methods prevents independent management of interfaces or reuse by other systems, severely restricting the scalability and adaptability of the business.

[0003] In scenarios involving data changes to core business tables, existing solutions also face significant shortcomings. Core tables (such as billing data tables in government cloud scenarios) frequently undergo insert and update operations, while the business scenarios triggering these operations (such as order review, cancellation, and renewal) are diverse. Directly embedding supporting logic such as bill update notifications and reporting to regulatory platforms into the core business code further exacerbates code complexity, leading to a high degree of coupling between core business and auxiliary functions. When business requirements change, this coupling structure is difficult to adjust quickly, and the interface call method cannot achieve dynamic routing and classification processing for different operation types, resulting in a lack of flexibility in the data processing flow and making it difficult to meet real-time and customized business needs.

[0004] In summary, existing technologies for handling database change events suffer from two major shortcomings: first, the business logic is highly coupled, leading to complex and difficult-to-maintain code; second, the lack of dynamic routing and flexible management mechanisms prevents the classification and independent function invocation of different change events. These issues directly impact the timeliness of data delivery, system scalability, and adaptability to complex business scenarios, necessitating a database change event handling solution that can decouple business logic and support dynamic routing and flexible function management. Summary of the Invention

[0005] To address the above issues, the present invention aims to provide a method, system, device, and medium for dynamic routing of database change events. It utilizes Debezium to capture database change events, forwards them using a Kafka message queue, and implements flexible management of consumer functions through Spring Cloud Functions.

[0006] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, embodiments of this application provide a method for dynamic routing of database change events, including: Configure the Debezium connector to monitor change events for specific tables in the database; Use a JSON converter to convert change events into standard JSON format files; Configure the Kafka message queue and route change events to dynamically generated Kafka topics via RegexRouter; Create a separate Spring Cloud Function project to consume Kafka topics; Manage and invoke predefined function chains through FunctionCatalog; Multiple independent data processing functions are chained together using function chains to achieve sequential processing of changed data.

[0007] In an optional implementation, configuring the Debezium connector to monitor change events for specific tables in the database includes: Write a DebeziumMySQLConnector configuration file to specify the database connection information in order to establish a connection with the MySQL database.

[0008] In the DebeziumMySQLConnector configuration file, the core business tables to be monitored are set using the table.include.list parameter, and the database operation types to be monitored are configured using the include.operations parameter. These database operation types include, but are not limited to, insert and update.

[0009] In an optional implementation, the step of using a JSON converter to convert the change event into a standard JSON format file includes: By setting key.converter and value.converter to org.apache.kafka.connect.json.JsonConverter, captured change events are converted into standard JSON format files; The standard JSON format file records the data before the change, the data after the change, database-related information, operation type identifier, and timestamp.

[0010] In an optional implementation, configuring the Kafka message queue to route change events to dynamically generated Kafka topics via RegexRouter includes: In the DebeziumMySQLConnector configuration file, set the address of the Kafka service and set the Kafka historical operation record topic parameter to account_changes, which is used to store the database historical operation information recorded by Debezium; Configure RegexRouter as the routing configuration tool for Kafka and define corresponding rules. Use placeholders to dynamically replace table names and forward change events of different tables to Kafka topics with corresponding naming formats for subsequent classification and processing.

[0011] In an optional implementation, establishing a separate Spring Cloud Function project to consume a Kafka topic includes: Based on the SpringInitializr tool, create a new SpringCloudFunction project, bind it to a Kafka topic using annotations, and enable listening and consuming of change events.

[0012] In an optional implementation, the management and invocation of predefined function chains via FunctionCatalog includes: Inject the FunctionCatalog component into the consumer project as a function chain management tool; By calling the lookup method, a predefined chain of functions, accountPipeline, can be obtained.

[0013] In an optional implementation, the use of function chains to concatenate multiple independent data processing functions to achieve sequential processing of changed data includes: In the Spring Cloud Function project, the function chain accountPipeline is defined using annotations. It adopts the method of returning a Function from a consumer function and uses the andThen method to connect the user notification function, the data transfer to the monitoring dashboard function, and the data entry function. The connected functions are called sequentially according to the business process to build a change data processing flow.

[0014] Secondly, embodiments of this application also provide a dynamic routing processing system for database change events, including: The monitoring configuration module is used to configure the Debezium connector to monitor change events of specific tables in the database; The data conversion module is used to convert change events into standard JSON format files using a JSON converter; The dynamic routing rule configuration module is used to configure Kafka message queues and route change events to dynamically generated Kafka topics via RegexRouter; The consumer project component module is used to create independent Spring Cloud Function projects to consume Kafka topics; The function chain management module is used to manage and invoke predefined function chains through FunctionCatalog; The function chain definition module is used to chain multiple independent data processing functions together to achieve sequential processing of changed data.

[0015] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the database change event dynamic routing processing method described in any of the above descriptions.

[0016] Fourthly, embodiments of this application also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the database change event dynamic routing processing method described in any of the above claims.

[0017] As can be seen from the above technical solutions, the present invention has the following advantages: The database change event dynamic routing method provided in this application uses Debezium to collect data changes and triggers function services via message queues. This allows developers to write corresponding code functions to respond to database change events, significantly reducing code coupling. It also allows for relatively flexible configuration of insert or update events for different tables or fields, automatically routing them to the corresponding functions to execute the appropriate logic. This enables independent management of different functions. Combined with the Serverless Function approach, this approach makes the core business logic clearer and less affected by other management functions. Secondly, it allows for timely data push to relevant platforms by configuring change event monitoring for core business tables. Thirdly, the independent Spring Cloud Function service allows for independent or orchestrated use of functional logic, making it more flexible and easier to manage.

[0018] This application uses Debezium to collect insert or update change events from core or critical business tables, then forwards these messages time-sensitively through message queues across different Kafka topics. It then integrates with a separate Spring Cloud Functions consumer project and the FunctionCatalog component to locate and invoke functions within the application context. This application effectively monitors changes to core business table data and sends them to consumer queues on different topics, enabling real-time tracking of core data changes and the separation of core data changes with business logic. Furthermore, the independent Spring Cloud Functions project handles data processing, allowing for hot-swappable consumer functions and easy expansion.

[0019] This application uses DebeziumMysqlConnector to monitor MySQL change events, and can also use RegexRouter to forward data change times of different core tables to different message queues, ensuring timely monitoring and processing of changes to core business tables.

[0020] This application configures the Kafka path in DebeziumMySQLConnector, which can send messages to Kafka, ensuring timely message consumption under high-concurrency data updates.

[0021] This application uses KafkaListener to listen for Kafka messages and establishes an independent Spring Cloud Function project. Within the consumer function project, FunctionCatalog serves as the function chain management class, managing function chains or functions. Functions originally integrated with core business logic can be separated into independent functions, allowing for chained function calls and scaling of the consumer function. This enables rapid scaling when handling hundreds of thousands or even millions of order invoices. Furthermore, decoupling from the original core business logic allows for hot-swapping of message functions, enabling independent upgrades of core and supporting business logic without affecting each other. Attached Figure Description

[0022] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating the dynamic routing method for database change events provided in this application.

[0024] Figure 2 A schematic diagram of the structure of the database change event dynamic routing processing system provided in this application.

[0025] Figure 3 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0026] The various embodiments of this disclosure will be described more fully in the detailed steps of the dynamic routing processing method for database change events described below. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0027] In the following, the terms “comprising” or “may include”, which may be used in various embodiments of this disclosure, indicate the presence of the disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in various embodiments of this disclosure, the terms “comprising,” “having,” and their cognates are intended only to indicate a particular feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more combinations of the foregoing.

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Please see Figure 1 The diagram shown is a flowchart of a method for dynamic routing of database change events in a specific embodiment. The method includes: S1: Configure the Debezium connector to monitor change events for specific tables in the database.

[0030] In this implementation, the DebeziumMySQLConnector configuration file is first written to specify database connection information, including the database address, port, username, and password, to establish a connection with the MySQL database. Then, the `table.include.list` parameter in the configuration file is used to specify the core business tables to be monitored, such as the billing table `cloud_account`. This ensures that only changes to critical business tables are monitored, filtering out events from irrelevant tables and improving processing efficiency. Simultaneously, the `include.operations` parameter is used to configure the types of database operations to be monitored, such as "insert" and "update," ensuring that only change events triggered by critical operations such as inserting data into the billing table are captured, further precisely focusing on important data changes.

[0031] For example, in the MySQLConnector configuration file, add two event changes: statement generation (taking this service as an example, because this service involves a large number of related operations such as reporting to the regulatory platform). The main change is inserting into the statement table. Use the table.include.list parameter to configure changes to the two tables, such as the statement table cloud_account. You can also configure changes to operations, such as using include.operations to configure "insert".

[0032] S2: Use a JSON converter to convert change events into standard JSON format files.

[0033] In a specific implementation, by configuring both "key.converter" and "value.converter" as "org.apache.kafka.connect.json.JsonConverter", the captured insert events are converted into JSON format files, which facilitates subsequent parsing and processing by the consumer.

[0034] The converted JSON file contains fields such as before (data before the change, null for insertion), after (data after the change), source (database-related information), op (operation type identifier, such as "c" for insertion), and ts_ms (timestamp).

[0035] For example, org.apache.kafka.connect.json.JsonConverter is used as the message converter.

[0036] "key.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter": "org.apache.kafka.connect.json.JsonConverter" This converter can be used to convert insert events into JSON files.

[0037] Here is an example of a JSON file: { "before": null, "after": { "id": "account_789", "user_id": "user_456", "instance_id": "instance1", "amount": 1000.00, "xxxxx": "2023-10-01T12:05:00Z" }, "source": { / / Specific database information such as version, table names, etc. }, "op": "c", "ts_ms": 1678902345678 } S3: Configure the Kafka message queue and route change events to dynamically generated Kafka topics via RegexRouter.

[0038] In a specific implementation, the "database.history.kafka.bootstrap.servers" parameter is set in the DebeziumMySQLConnector configuration file to specify the address of the Kafka service, such as "10.110.xx:9092", to ensure that Debezium can communicate normally with Kafka and send the captured change events to Kafka.

[0039] Configure RegexRouter as the routing configuration tool for Kafka, define the rule "transforms.route.replacement": "$1_events", and use the placeholder $1 to dynamically replace the table name. This will forward change events of different tables to the corresponding Kafkatopic with the corresponding naming format, such as cloud account single_events, which will facilitate subsequent classification and processing.

[0040] For example, Kafka is used as a message queue, with different topics used to send messages about changes to the Debezium MySQL Connector. The Kafka address is configured in the Debezium MySQL Connector as follows: "database.history.kafka.bootstrap.servers": "10.110.xx:9092", "database.history.kafka.topic": "account_changes", configures metadata storage.

[0041] On the other hand, RegexRouter is used as the routing configuration for Kafka, and the placeholder $ in "transforms.route.replacement":"$1_events" is used to dynamically replace the event name, thereby forwarding it to different topics. For example, after replacing the topic with the placeholder, it becomes cloud_account_events.

[0042] Because RegexRouter is used in the configuration file and placeholders are used for replacement, cloud_account_events is created as a queue for the topic in Kafka. When there is an insert event, it will be sent to this topic.

[0043] S4: Create an independent Spring Cloud Function project to consume Kafka topics.

[0044] In a specific implementation, a new SpringCloudFunction project is created based on Spring Initializr or other project build tools, and necessary dependencies such as Spring Cloud Function and SpringKafka are introduced to prepare for consuming Kafka messages and executing business logic later.

[0045] In a Spring Cloud Function project, the @KafkaListener annotation is used to specify the Kafka topic to listen to, such as "cloud_account_events", to establish a consumption connection with the Kafka message queue and receive billing table change event messages sent by Debezium in real time.

[0046] S5: Manage and call predefined function chains through FunctionCatalog.

[0047] In a specific implementation, the FunctionCatalog component is introduced as a function chain management tool within the consumption method. By calling the FunctionCatalog.lookup(“accountPipeline”) method, a predefined function chain “accountPipeline” is obtained. This function chain integrates multiple consumption functions related to bill change events, which are executed sequentially according to a predetermined order to ensure that the changed data is processed comprehensively and in an orderly manner.

[0048] S6: Use function chains to chain multiple independent data processing functions to achieve sequential processing of changed data. In a specific implementation, in the Spring Cloud Function project, the function chain accountPipeline is defined using annotations. The function returns a Function, and the user notification function, the data transfer to the monitoring dashboard function, and the data entry function are linked together using the andThen method. The linked functions are called sequentially according to the business process to build a change data processing flow.

[0049] Specifically, in the Spring Cloud Function project, the function chain "accountPipeline" is defined using the @Bean annotation. In its method implementation, the return function of the consumer function is adopted, and multiple specific consumer functions are chained together using the .andThen() method, such as the user notification function notify(), the data transfer to the monitoring dashboard function dataDisplay(), and the data entry into the lake function dataLake(). These functions are called sequentially according to the business process, thus building a complete and coherent data processing flow.

[0050] Define the three consumer functions mentioned above, all annotated with @Bean. Each function receives a Map containing the changed data.<String,Object> The type parameter, after being processed by the corresponding business logic, returns the processing result or the data required for subsequent processing, realizing specific response operations for bill change events, such as sending notifications based on bill information, converting data formats and pushing them to the monitoring dashboard for display, and storing data in the data lake for subsequent data analysis.

[0051] It should be noted that in this method, steps S4 to S6 aim to create a Spring Cloud Function project to implement independent consumer functions. For example, the specific process is as follows: First, create a Spring Cloud Function project. After creating the project, create consumer functions based on the business logic. Here, we'll take an order renewal service as an example. In the consumer function, we'll put some processing that is independent of the main business (order billing time extension, instance billing time extension, etc.) here. For example, we'll send notifications to users and transfer billing order data to the data lake.

[0052] In SpringCloud projects, you can use `@Bean` to define custom functions or custom function chains. This allows message sending functions to be isolated and used by other projects. Here, we'll take user notifications, message forwarding to a monitoring dashboard, and forwarding billing and order data to a data lake as an example, defining some custom functions and their chains.

[0053] First, use `@KafkaListener(topics = "cloud_account_events")` to bind a message queue to consume messages. In the consumption method, use `FunctionCatalog` as the function chain management class, and obtain the function chain through the `lookup` method. In this case, we take `accountPipeline` as an example. `FunctionCatalog.lookup("accountPipeline")` can obtain the function chain.

[0054] Next, we continue using the `@Bean` annotation to define the function `accountPipeline`. This function chain contains two methods, which can be called sequentially using either the `return Function` method or the `.andThen` method. For example, the `accountPipeline` method contains three functions: user notification, forwarding to the dashboard, and data entry into the lake.

[0055] These three functions each use the @Bean custom function: @Bean Function <Map<String, Object> , String<String> notify() @Bean Function <Map<String, Object> , String> dataDisplay() @Bean Function <Map<String, Object> , String<String> dataLake() These three functions can be called sequentially in the accountPipeline using notify().andThen(dataDisplay()).andThen(dataLake()) to complete the subsequent coordination logic after the core business logic is finished. In this embodiment, through end-to-end decoupling and component-based design, automated processing of change data and flexible orchestration of business logic are achieved. At the data acquisition layer, the Debezium connector precisely monitors insert and update operations on core business tables, and, in conjunction with a JSON converter, generates standardized formats to ensure complete capture and unified representation of change information, providing standardized input for subsequent processing. At the message transmission layer, the Kafka message queue, combined with the RegexRouter dynamic routing mechanism, intelligently distributes change events from different tables to corresponding topics, achieving data classification and isolation, and improving system throughput and processing efficiency. At the consumption processing layer, the independently deployed Spring Cloud Function project manages predefined function chains through FunctionCatalog, encapsulating user notifications, regulatory reporting, and data lake entry into independent functions. Sequential calls are implemented using the andThen method, forming a customizable data processing pipeline. This architecture offers three advantages: First, by replacing traditional API calls with an event-driven model, it decouples core business functions from auxiliary functions, supporting independent development and iteration. Second, the dynamic routing mechanism allows the system to flexibly adjust event processing paths according to business needs without modifying the underlying code. Third, the function chain orchestration pattern provides extremely high scalability, enabling rapid response to business changes by adding or replacing function nodes, while also supporting function-level monitoring and maintenance, significantly reducing system complexity and maintenance costs.

[0056] like Figure 2As shown, the following are embodiments of the database change event dynamic routing processing system provided in this disclosure. This system and the database change event dynamic routing processing method in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the database change event dynamic routing processing system, please refer to the embodiments of the database change event dynamic routing processing method described above.

[0057] A database change event dynamic routing processing system includes: a monitoring configuration module, a data transformation module, a dynamic routing rule configuration module, a consumer item construction module, a function chain management module, and a function chain definition module.

[0058] The monitoring configuration module is used to configure the Debezium connector to monitor change events of specific tables in the database.

[0059] The data conversion module is used to convert change events into standard JSON format files using a JSON converter.

[0060] The dynamic routing rule configuration module is used to configure Kafka message queues and route change events to dynamically generated Kafka topics via RegexRouter.

[0061] The Consumption Project Component Module is used to create independent Spring Cloud Function projects that consume Kafka topics.

[0062] The function chain management module is used to manage and invoke predefined function chains through FunctionCatalog.

[0063] The function chain definition module is used to chain multiple independent data processing functions together to achieve sequential processing of changed data.

[0064] The database change event dynamic routing system provided in this embodiment achieves automated processing and flexible response to database change events through a layered design. It utilizes Debezium to accurately capture and standardize change data, dynamically routes it to the corresponding topic using Kafka, consumes the data through the Spring Cloud Function project, and orchestrates function chains via FunctionCatalog. This architecture decouples core business logic from auxiliary logic, supports dynamic adjustment of processing paths, and enables functional expansion through flexible combination of function chains, significantly improving system maintainability and response speed.

[0065] Figure 3 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0066] The database change event dynamic routing processing method provided in this application embodiment can be applied to electronic devices. Those skilled in the art will understand that the electronic device structure involved in the embodiments of this invention does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the electronic device includes, but is not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0067] Electronic devices may include processors, external memory interfaces, internal memory, universal serial bus (USB) interfaces, charging management modules, power management modules, batteries, wireless communication modules, audio modules, speakers, microphones, sensor modules, buttons, cameras, displays, and SIM card interfaces, etc.

[0068] A processor may include one or more processing units, such as: a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0069] The processor can serve as the nerve center and command center of an electronic device. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0070] The processor may also include memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or that are used repeatedly. If the processor needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0071] An external storage interface (ESI) can be used to connect external memory cards, such as microSD cards, to expand the storage capacity of electronic devices. The external memory card communicates with the processor through the ESI to perform data storage functions, such as saving music and video files on the external memory card.

[0072] Internal memory can be used to store computer executable program code, which includes instructions. The processor executes various functional applications and data processing of electronic devices by running the instructions stored in internal memory. Internal memory can include a program storage area and a data storage area. Internal memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0073] Wireless communication functionality in electronic devices can be achieved through antennas, wireless communication modules, modem processors, and baseband processors.

[0074] Wireless communication modules can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0075] Electronic devices can implement audio functions through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0076] Electronic devices can achieve shooting functions through ISPs, cameras, video codecs, GPUs, displays, and application processors.

[0077] Electronic devices can achieve display functions through GPUs, displays, and application processors.

[0078] A GPU is a microprocessor for image processing, connected to the display screen and application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information.

[0079] A display screen is used to display images, videos, etc. A display screen includes a display panel.

[0080] The aforementioned electronic device implements the database change event dynamic routing processing method of this application, which uses Debezium to accurately monitor changes to specific tables in the database, combines Kafka message queues with RegexRouter to implement dynamic topic routing, and uses Spring Cloud Functions to build an orchestratable function chain processing pipeline. This achieves the beneficial effects of decoupling business logic and data processing, automating change event classification, and significantly improving system scalability and maintainability.

[0081] The storage medium provided in this application stores a program product capable of implementing a dynamic routing processing method for database change events.

[0082] Dynamic routing methods for database change events include: Configure the Debezium connector to monitor change events for specific tables in the database; Use a JSON converter to convert change events into standard JSON format files; Configure the Kafka message queue and route change events to dynamically generated Kafka topics via RegexRouter; Create a separate Spring Cloud Function project to consume Kafka topics; Manage and invoke predefined function chains through FunctionCatalog; Multiple independent data processing functions are chained together using function chains to achieve sequential processing of changed data.

[0083] In some possible implementations, the database change event dynamic routing processing method of this disclosure can be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this disclosure.

[0084] The storage medium disclosed herein may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0085] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for dynamic routing of database change events, characterized in that, include: Configure the Debezium connector to monitor change events for specific tables in the database; Use a JSON converter to convert change events into standard JSON format files; Configure the Kafka message queue and route change events to dynamically generated Kafka topics via RegexRouter; Create a separate Spring Cloud Function project to consume Kafka topics; Manage and invoke predefined function chains through FunctionCatalog; Multiple independent data processing functions are chained together using function chains to achieve sequential processing of changed data.

2. The database change event dynamic routing processing method according to claim 1, characterized in that, The configuration of the Debezium connector to monitor change events of specific tables in the database includes: Write a DebeziumMySQLConnector configuration file to specify the database connection information in order to establish a connection with the MySQL database. In the DebeziumMySQLConnector configuration file, the core business tables to be monitored are set using the table.include.list parameter, and the database operation types to be monitored are configured using the include.operations parameter. These database operation types include, but are not limited to, insert and update.

3. The database change event dynamic routing processing method according to claim 2, characterized in that, The process of using a JSON converter to convert change events into standard JSON format files includes: By setting key.converter and value.converter to org.apache.kafka.connect.json.JsonConverter, captured change events are converted into standard JSON format files; The standard JSON format file records the data before the change, the data after the change, database-related information, operation type identifier, and timestamp.

4. The database change event dynamic routing processing method according to claim 3, characterized in that, The configuration of the Kafka message queue, which routes change events to dynamically generated Kafka topics via RegexRouter, includes: In the DebeziumMySQLConnector configuration file, set the address of the Kafka service and set the Kafka historical operation record topic parameter to account_changes, which is used to store the database historical operation information recorded by Debezium; Configure RegexRouter as the routing configuration tool for Kafka and define corresponding rules. Use placeholders to dynamically replace table names and forward change events of different tables to Kafka topics with corresponding naming formats for subsequent classification and processing.

5. The database change event dynamic routing processing method according to claim 4, characterized in that, The process of establishing an independent Spring Cloud Function project to consume Kafka topics includes: Based on the SpringInitializr tool, create a new SpringCloudFunction project, bind it to a Kafka topic using annotations, and enable listening and consuming of change events.

6. The database change event dynamic routing processing method according to claim 5, characterized in that, The management and invocation of predefined function chains through FunctionCatalog includes: Inject the FunctionCatalog component into the consumer project as a function chain management tool; By calling the lookup method, a predefined chain of functions, accountPipeline, can be obtained.

7. The database change event dynamic routing processing method according to claim 6, characterized in that, The method of using function chains to chain together multiple independent data processing functions to achieve sequential processing of changed data includes: In the Spring Cloud Function project, the function chain accountPipeline is defined using annotations. It adopts the method of returning a Function from a consumer function and uses the andThen method to connect the user notification function, the data transfer to the monitoring dashboard function, and the data entry function. The connected functions are called sequentially according to the business process to build a change data processing flow.

8. A dynamic routing processing system for database change events, characterized in that, The system employs the database change event dynamic routing processing method as described in any one of claims 1 to 7; The system includes: The monitoring configuration module is used to configure the Debezium connector to monitor change events of specific tables in the database; The data conversion module is used to convert change events into standard JSON format files using a JSON converter; The dynamic routing rule configuration module is used to configure Kafka message queues and route change events to dynamically generated Kafka topics via RegexRouter; The consumer project component module is used to create independent Spring Cloud Function projects to consume Kafka topics; The function chain management module is used to manage and invoke predefined function chains through FunctionCatalog; The function chain definition module is used to chain multiple independent data processing functions together to achieve sequential processing of changed data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the database change event dynamic routing processing method as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the database change event dynamic routing processing method as described in any one of claims 1 to 7.

Images