Canal-based intelligent service adaptation method

Abstract

The invention discloses an intelligent service adaptation method based on a canal, and relates to the technical field of digital information transmission. The method comprises the following steps: step 1, configuring and filtering a data source to ensure that a Canal can be accurately connected with and monitor a binlog of a source database, and screening related data according to service requirements to reduce unnecessary data transmission; 2, data analysis and processing: converting data in a binlog into a format which can be identified by a system, and filtering and assembling the data according to a service rule so as to construct an SQL (Structured Query Language) statement and a cache Key which are required by the service, and 3, data synchronization and recording: pushing the processed data to a downstream system through multiple protocols, recording a synchronization result, and sending the synchronization result to the system. Therefore, the synchronization process can be monitored and backtracked. The method has remarkable advantages in the aspects of solving the service synchronization problem, guaranteeing the service stability, reducing the enterprise cost and realizing efficient synchronization of the system, and is an important support for realizing high-quality, high-availability and high-stability services for enterprises.

Description

Technical Field

[0001] The present invention relates to the technical field of digital information transmission, and in particular to a canal-based business intelligent adaptation method. Background Art

[0002] With the development of Internet technology, enterprise-level application systems usually involve data synchronization issues for multiple database instances. In order to ensure data consistency and timeliness as well as efficient operation of business systems, traditional synchronization methods have gradually failed to meet the needs.

[0003] Canal is an open source tool developed by Alibaba for MySQL incremental log parsing. It can parse MySQL binlog logs and provide them to other systems for data consumption. However, when using Canal directly for data synchronization and adaptation to different business scenarios, complex development and configuration work is often required. In particular, when synchronizing and adapting data between multiple business systems, a series of technical challenges such as high concurrency, large data volume, data consistency, and low latency are faced.

[0004] like Figure 4 , the solution of the prior art:

[0005] Step 1:

[0006] 1. Data source configuration: Connect to the source database by configuring Canal and listen to binlog logs. Make sure the binlog function is enabled in the database configuration and set the correct log format (ROW format is recommended).

[0007] 2. Data filtering and selection: According to business needs, configure Canal to only monitor and push change data of specific databases or tables to avoid unnecessary data transmission.

[0008] Step 2:

[0009] 1. Data format conversion: After Canal parses the binlog log, the data will be synchronized in the form of an adapter executed by Canal

[0010] Step 3:

[0011] 1. Data synchronization method: CanalAdapter supports pushing parsed data to downstream systems via TCP, HTTP, etc. Select the appropriate synchronization method and push frequency based on the real-time requirements of the system.

[0012] 2. High availability design: To ensure high availability of the system, you can deploy Canal's cluster mode to achieve failover and load balancing through multiple instances. At the same time, combined with the persistence capabilities of message queues (such as Kafka), ensure that critical data will not be lost when Canal crashes or restarts.

[0013] The prior art has the following disadvantages:

[0014] 1. It is not flexible to be compatible with cross-cloud cache and MySQL synchronization at the same time, and ensure the order and effectiveness of MySQL transactions and cache synchronization;

[0015] 2. Unable to adapt to business transformation during synchronization;

[0016] 3. Records during the synchronization process cannot be monitored or traced back.

[0017] Therefore, a business intelligent adaptation method based on canal is proposed. Summary of the invention

[0018] The purpose of the present invention is to solve the problems raised in the above background technology, and provide a business intelligent adaptation method based on canal.

[0019] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0020] A canal-based business intelligent adaptation method includes the following steps:

[0021] Step 1: Data source configuration and filtering to ensure that Canal can accurately connect to and monitor the binlog logs of the source database, and filter relevant data according to business needs to reduce unnecessary data transmission;

[0022] Step 2: Data parsing and processing: converting the data in the binlog log into a format that the system can recognize, and filtering and assembling the data according to business rules to build the SQL statements and cache keys required by the business;

[0023] Step 3: Data synchronization and recording: push the processed data to the downstream system through various protocols and record the synchronization results to monitor and trace the synchronization process;

[0024] Step 4: High availability and exception management: ensure high availability of services through multi-instance deployment and liveness detection mechanism, and issue alarms and automatically recover when exceptions occur to maintain the stability of data synchronization;

[0025] Step 5: Data compensation and remediation. When synchronization errors or data inconsistencies are found, data compensation and remediation operations are performed to ensure the ultimate consistency and integrity of the data.

[0026] Furthermore, the step 1 of data source configuration and filtering specifically includes the following steps:

[0027] Step 11: Data source configuration, connect to the source database by configuring Canal, and monitor the binlog log. Make sure the binlog function is enabled in the database configuration and set the correct log format;

[0028] Step 12: Data filtering and selection. According to business needs, configure Canal to only monitor and push change data of specific databases or tables to avoid unnecessary data transmission.

[0029] Furthermore, the data analysis and processing in step 2 specifically includes the following steps:

[0030] Step 21: Data format conversion. After the agent-broker parses the binlog log, the data will be deserialized into object data in the system to ensure that the data can be used by the current system;

[0031] Step 22: Data grouping and filtering. Agent-broker aggregates the data changes under the same transaction into the same array based on the acquired binlog, and then filters them through the data synchronization configuration table. There is no data filtering in the data synchronization table.

[0032] Step 23: Business data assembly: assemble data according to the data synchronization configuration table, and build SQL statements and cache keys that meet business needs.

[0033] Furthermore, the step 3 of data synchronization and recording specifically includes the following steps:

[0034] Step 31: Data synchronization method, agent-broker supports synchronizing the assembled data to the downstream synchronization system through TCP, HTTP, etc.;

[0035] Step 32: Synchronization record preservation. After synchronization, the agent-broker will record the results of the synchronized data to ensure that the synchronization process can be directly viewed by the administrator;

[0036] Step 33: Synchronization exception alarm: If an exception occurs between the program or the network during the synchronization process of the agent-broker, the synchronization will fail. After the failure, the error message of the synchronization failure will be recorded, persisted in the database, and notified to the admin.

[0037] Furthermore, the step 4 high availability and exception management specifically includes the following steps:

[0038] Step 41: High availability design. To ensure high availability of the system, the agent-broker service adopts high availability deployment, and implements fault switching and load balancing through multiple instances. At the same time, the agent-broker also uses an elegant shutdown method to shut down the service, ensuring that data synchronization is not lost.

[0039] Step 42: Abnormal detection and recovery. Agent-broker will continue to detect the upstream and downstream services to ensure that the communication between services is smooth. When there is a disconnection between services or other abnormal situations that cause data synchronization failure, it will send an alarm to the administrator and automatically try to recover.

[0040] Step 43: Data consistency check. During the synchronization of heterogeneous data sources, data inconsistencies may occur. Regular data reconciliation and consistency checks can be performed to ensure the accuracy of synchronized data.

[0041] Furthermore, the data compensation and remediation in step 5 specifically includes the following steps:

[0042] Step 51: Data compensation. After developers and managers proactively discover synchronization errors in the background or receive synchronization error notifications, they can compensate for synchronization error logs after ensuring that services and networks are restored to ensure data synchronization consistency.

[0043] Step 52: Data remediation. When an unnoticeable abnormality occurs during the data consistency check, the administrator or developer needs to perform data remediation through the data remediation of the operation management background to ensure the final consistency of data synchronization.

[0044] Furthermore, the data synchronization configuration table is a core configuration table of the agent-broker service, which records the mapping relationship between data changes and cache changes.

[0045] Furthermore, the activity detection generally means that services send messages to each other even when there is no business interaction. If the return information can be received normally, it means that the services are active with each other.

[0046] Furthermore, the imperceptible exception generally refers to an unexpected abnormality in the program, and when this happens, the program cannot perceive the occurrence of the abnormality.

[0047] The beneficial effects of the present invention are as follows:

[0048] The present invention solves the problem of simultaneous synchronization of MySQL and cache for data synchronization across cloud services, ensures the transactionality of MySQL data synchronization, and the sequentiality of MySQL and cache synchronization, and can ensure the accuracy and effectiveness of data synchronization in high-concurrency scenarios;

[0049] The present invention has great advantages in adapting to different business scenarios. By configuring the mapping table, it realizes the business change of synchronized data without stopping the system, which can save repeated development and repeated online to a greater extent, and has a significant effect on reducing costs and increasing efficiency for enterprises.

[0050] The present invention can visualize the entire data synchronization process through a large number of synchronization logs. Managers and developers can directly observe the real-time synchronization status through the visualization interface, and can also query previous synchronization records and status based on the records, which greatly reduces the time for developers to troubleshoot problems and also provides data support for subsequent system optimization.

[0051] The business adaptation strategy based on canal has shown significant advantages in solving business synchronization problems, ensuring service stability, reducing enterprise costs, and achieving efficient system synchronization. It is an important support for enterprises to achieve high-quality, high-availability, and high-stability services. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the overall architecture of the present invention;

[0053] Figure 2 is a flow chart of the agent-broker of the present invention;

[0054] Figure 3 It is the logic diagram of the Agent of the present invention

[0055] Figure 4 It is a schematic diagram of the prior art of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0057] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. In addition, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0059] In the description of the embodiments of the present invention, it should be noted that the terms "inside", "outside", "upper", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or are the orientations or positional relationships in which the inventive product is usually placed when used. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0060] like Figures 1 to 3 As shown, a canal-based business intelligent adaptation method includes the following steps:

[0061] Step 1: Data source configuration and filtering:

[0062] The specific steps include:

[0063] Step 11: Data source configuration, connect to the source database by configuring Canal, and monitor the binlog log. Make sure the binlog function is enabled in the database configuration and set the correct log format;

[0064] Step 12: Data filtering and selection. According to business needs, configure Canal to only monitor and push change data of specific databases or tables to avoid unnecessary data transmission.

[0065] Step 2: Data analysis and processing:

[0066] The specific steps include:

[0067] Step 21: Data format conversion. After the agent-broker parses the binlog log, the data will be deserialized into object data in the system to ensure that the data can be used by the current system;

[0068] Step 22: Data grouping and filtering. Agent-broker aggregates the data changes under the same transaction into the same array based on the acquired binlog, and then filters them through the data synchronization configuration table. There is no data filtering in the data synchronization table.

[0069] Step 23: Business data assembly: assemble data according to the data synchronization configuration table, and build SQL statements and cache keys that meet business needs.

[0070] Step 3: Data synchronization and recording:

[0071] The specific steps include:

[0072] Step 31: Data synchronization method, agent-broker supports synchronizing the assembled data to the downstream synchronization system through TCP, HTTP, etc.;

[0073] Step 32: Synchronization record preservation. After synchronization, the agent-broker will record the results of the synchronized data to ensure that the synchronization process can be directly viewed by the administrator;

[0074] Step 33: Synchronization exception alarm: If an exception occurs between the program or the network during the synchronization process of the agent-broker, the synchronization will fail. After the failure, the error message of the synchronization failure will be recorded, persisted in the database, and notified to the admin.

[0075] Step 4: High availability and exception management:

[0076] 5. The specific steps include:

[0077] Step 41: High availability design. To ensure high availability of the system, the agent-broker service adopts high availability deployment, and implements fault switching and load balancing through multiple instances. At the same time, the agent-broker also uses an elegant shutdown method to shut down the service, ensuring that data synchronization is not lost.

[0078] Step 42: Abnormal detection and recovery. Agent-broker will continue to detect the upstream and downstream services to ensure that the communication between services is smooth. When there is a disconnection between services or other abnormal situations that cause data synchronization failure, it will send an alarm to the administrator and automatically try to recover.

[0079] Step 43: Data consistency check. During the synchronization of heterogeneous data sources, data inconsistencies may occur. Regular data reconciliation and consistency checks can be performed to ensure the accuracy of synchronized data.

[0080] Step 5: Data Compensation and Remediation:

[0081] The specific steps include:

[0082] Step 51: Data compensation. After developers and managers proactively discover synchronization errors in the background or receive synchronization error notifications, they can compensate for synchronization error logs after ensuring that services and networks are restored to ensure data synchronization consistency.

[0083] Step 52: Data remediation. When an unnoticeable abnormality occurs during the data consistency check, the administrator or developer needs to perform data remediation through the data remediation of the operation management background to ensure the final consistency of data synchronization.

[0084] The data synchronization configuration table is the core configuration table of the agent-broker service, which records the mapping relationship between data changes and cache changes.

[0085] More specifically, it can effectively adapt to business changes. Its main function is to filter synchronized data and map business caches, providing a basis for building related cache keys.

[0086] Activity detection usually means that services send messages to each other even when there is no business interaction. If the return information can be received normally, it means that the services are active with each other.

[0087] Unperceived exceptions usually refer to unexpected exceptions in the program, and when this happens, the program cannot perceive the occurrence of the exception.

[0088] In summary: This strategy increases the flexibility of business transformation during data synchronization, while ensuring the transactional and sequential nature of cache synchronization during MySQL synchronization, improving the scalability and stability of the system, and ensuring the real-time and consistency of data synchronization.

[0089] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A canal-based intelligent business adaptation method, characterized in that: The following steps are involved: Step 1: Data source configuration and filtering to ensure that Canal can accurately connect to and monitor the binlog logs of the source database, and filter relevant data according to business needs to reduce unnecessary data transmission; Step 2: Data parsing and processing: converting the data in the binlog log into a format that the system can recognize, and filtering and assembling the data according to business rules to build the SQL statements and cache keys required by the business. Step 3: Data synchronization and recording: push the processed data to the downstream system through various protocols and record the synchronization results to monitor and trace the synchronization process; Step 4: High availability and exception management: ensure high availability of services through multi-instance deployment and liveness detection mechanism, and issue alarms and automatically recover when exceptions occur to maintain the stability of data synchronization; Step 5: Data compensation and remediation. When synchronization errors or data inconsistencies are found, data compensation and remediation operations are performed to ensure the ultimate consistency and integrity of the data.

2. According to claim 1, a canal-based business intelligent adaptation method is characterized in that: The step 1 of data source configuration and filtering specifically includes the following steps: Step 11: Data source configuration, connect to the source database by configuring Canal, and monitor the binlog log. Make sure the binlog function is enabled in the database configuration and set the correct log format; Step 12: Data filtering and selection. According to business needs, configure Canal to only monitor and push change data of specific databases or tables to avoid unnecessary data transmission.

3. The canal-based intelligent business adaptation method according to claim 1, characterized in that: The data analysis and processing of step 2 specifically includes the following steps: Step 21: Data format conversion. After the agent-broker parses the binlog log, the data will be deserialized into object data in the system to ensure that the data can be used by the current system; Step 22: Data grouping and filtering. Agent-broker aggregates the data changes under the same transaction into the same array based on the acquired binlog, and then filters them through the data synchronization configuration table. There is no data filtering in the data synchronization table. Step 23: Business data assembly: assemble data according to the data synchronization configuration table, and build SQL statements and cache keys that meet business needs.

4. The canal-based intelligent business adaptation method according to claim 1, characterized in that: The step 3 data synchronization and recording specifically includes the following steps: Step 31: Data synchronization method, agent-broker supports synchronizing the assembled data to the downstream synchronization system through TCP, HTTP, etc.; Step 32: Synchronization record preservation. After synchronization, the agent-broker will record the results of the synchronized data to ensure that the synchronization process can be directly viewed by the administrator; Step 33: Synchronization exception alarm: If an exception occurs between the program or the network during the synchronization process of the agent-broker, the synchronization will fail. After the failure, the error message of the synchronization failure will be recorded, persisted in the database, and notified to the admin.

5. The canal-based intelligent business adaptation method according to claim 1, characterized in that: The step 4 high availability and exception management specifically includes the following steps: Step 41: High availability design. To ensure high availability of the system, the agent-broker service adopts high availability deployment, and implements fault switching and load balancing through multiple instances. At the same time, the agent-broker also uses an elegant shutdown method to shut down the service, ensuring that data synchronization is not lost. Step 42: Abnormal detection and recovery. Agent-broker will continue to detect the upstream and downstream services to ensure that the communication between services is smooth. When there is a disconnection between services or other abnormal situations that cause data synchronization failure, it will send an alarm to the administrator and automatically try to recover. Step 43: Data consistency check. During the synchronization of heterogeneous data sources, data inconsistencies may occur. Regular data reconciliation and consistency checks can be performed to ensure the accuracy of synchronized data.

6. The canal-based intelligent business adaptation method according to claim 1, characterized in that: The step 5 data compensation and remediation specifically includes the following steps: Step 51: Data compensation. After developers and managers proactively discover synchronization errors in the background or receive synchronization error notifications, they can compensate for synchronization error logs after ensuring that services and networks are restored to ensure data synchronization consistency. Step 52: Data remediation. When an unnoticeable abnormality occurs during the data consistency check, the administrator or developer needs to perform data remediation through the data remediation of the operation management background to ensure the final consistency of data synchronization.

7. The canal-based intelligent business adaptation method according to claim 3, characterized in that: The data synchronization configuration table is the core configuration table of the agent-broker service, which records the mapping relationship between data changes and cache changes.

8. The canal-based intelligent business adaptation method according to claim 5, characterized in that: The detection of activity usually means that services send messages to each other even when there is no business interaction. If the return information can be received normally, it means that the services are active with each other.

9. The canal-based intelligent business adaptation method according to claim 6, characterized in that: The imperceptible exception generally refers to an unexpected abnormal situation in the program, and when this happens, the program cannot perceive the occurrence of the exception.

Images