A network topology synchronization method based on Canal

Abstract

The present invention provides a canal-based network topology synchronization method, comprising: in response to the normal operation of a canal cluster, starting a CVK configuration distribution layer agent service, linking to the canal cluster, and establishing a connection with the canal cluster; the agent service subscribing to database table information from the canal cluster; in response to the agent service's subscription request, when the database table information changes, the CVK cluster transmits the changed database table information back to the CVK; and the CVK distributes the changed database table information to the agent service. The canal-based network topology synchronization method employs an incremental approach for data synchronization, reducing the amount of data required for full topology synchronization, alleviating the computational burden on the agent process on the CVK, and improving topology synchronization efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud services, and in particular relates to a network topology synchronization method based on Canal. Background Art

[0002] In cloud computing systems, cross-cloud network management, interconnection between traditional network architectures and infrastructure, and support for complex application scenarios such as edge cluster deployment, overcome the limitations of hardware communication equipment. Most communication solutions are implemented using virtual network topologies. The speed of network topology synchronization and the convergence of network protocols determine the communication of resources such as virtual machines (VMs). With the increasing number of VMs used by users and the rapid increase in the number of CVKs managed by CVK clusters, the ability to quickly and instantly synchronize network topologies has impacted network traffic. Therefore, a method for network topology synchronization that combines security, maintainability, manageability, and performance is required. Summary of the Invention

[0003] In view of this, in order to overcome the above-mentioned defects, the present invention aims to propose a network topology synchronization method based on Canal.

[0004] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0005] In a first aspect, the present invention provides a network topology synchronization method based on Canal, comprising:

[0006] In response to the normal operation of the Canal cluster, the CVK configuration distribution layer Agent service starts, links to the Canal cluster, and establishes a connection with the Canal cluster;

[0007] The Agent service subscribes to the database table information from the Canal cluster;

[0008] In response to the subscription request of the Agent service, when the database table information changes, the CVK cluster returns the changed database table information to CVK;

[0009] CVK sends the changed database table information to the Agent service.

[0010] Furthermore, in response to the normal operation of the Canal cluster, the CVK configuration distribution layer Agent service is started, links to the Canal cluster, and establishes a connection with the Canal cluster, which also includes:

[0011] After the Agent service is connected to the Canal cluster, it needs to cache the connection information and perform periodic checks to detect whether the connection is abnormal.

[0012] Furthermore, in response to the subscription request of the Agent service, when the database table information changes, the CVK cluster returns the changed database table information to CVK, including:

[0013] Get the relevant information of the online virtual network card and write it to the specified MySQL data table;

[0014] Canal cluster filters out changed data based on the MySQL data table;

[0015] Convert the changed data into data of the specified format and protocol EntryProtocol.proto, send a notification, and synchronize the operation and maintenance data to Redis;

[0016] CVK filters out relevant data from the subscribed database table information, converts it, and sends it to the Agent service.

[0017] Furthermore, the CVK cluster sends the changed database table information back to CVK, including:

[0018] The control layer configured on CVK detects whether the link between the cached Agent service and the Canal cluster is available. If it is available, it obtains the Binlog message, filters out the data of the subscribed database table information, filters the relevant data of the relevant VPC on this CVK, converts it into configuration-related data and sends it to the Agent service. Otherwise, it deletes the cached link information and replaces it with the newly established Canal cluster service link.

[0019] Furthermore, the control layer on CVK is configured to periodically compare the information on CVK with the Redis data of the operation and maintenance data to see if they are the same. If they are different, the difference data is reported and an alarm is issued.

[0020] In a second aspect, the present invention provides an electronic device / server comprising a processor and a memory communicatively connected to the processor and used to store instructions executable by the processor. When the processor executes the instructions, it implements the Canal-based network topology synchronization method described in the first aspect above.

[0021] In a third aspect, the present invention provides a readable storage medium for storing a computer program, wherein when the computer program is executed by a processor, the canal-based network topology synchronization method described in the first aspect is implemented.

[0022] Compared with the existing technology, the network topology synchronization method based on Canal described in the present invention has the following advantages:

[0023] 1. Use incremental data synchronization to reduce the amount of data required for full topology synchronization, alleviate the computing pressure on the Agent process on CVK, and improve topology synchronization efficiency.

[0024] 2. Use Binlog subscription to distribute configurations, reducing the interaction between the data layer and the configuration layer and improving the performance of distributing configuration data.

[0025] 3. Directly subscribing to MySQL data files can ensure that the data sources of the data layer and the configuration layer are consistent, without the need for data comparison, ensuring the reliability of the configuration, and eliminating the need to compare related data.

[0026] 4. Through the alarm email of abnormal operation, the alarm is reported and sent by email immediately, and the operation and maintenance personnel are notified in advance to deal with the fault.

[0027] 5. Synchronize data to Redis regularly, compare data every day, find out the difference data, and send emails to issue alerts. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which constitute part of the present invention, are provided to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are provided to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0029] Figure 1 This is a flow chart of the Canal-based network topology synchronization method described in the present invention. DETAILED DESCRIPTION

[0030] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0031] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0032] This embodiment provides a network topology synchronization method based on Canal, and the specific steps include:

[0033] 1. Deploy and build the Canal cluster and ensure its normal operation.

[0034] 2. The CVK configuration distribution layer Agent service is started, connects to the Canal cluster, establishes a connection with the Canal cluster, caches the connection information, and starts a 1-minute periodic detection task for link anomalies.

[0035] 3. The Agent service on CVK subscribes to the configuration-related table structure information and synchronizes it to the Canal cluster.

[0036] 4. The data control layer virtual network card is online, and relevant information (such as L3vni of the VPC to which the network card belongs, the network card MAC address, the subnet L2vni, CVK related information, etc.) is assembled and written to the specified Mysql data table. According to the virtual network card information of the online virtual machine, the Canal cluster receives the data of the database table structure change, converts it into data in the specified format and specified protocol EntryProtocol.proto, sends a notification, and synchronizes the operation and maintenance data to Redis.

[0037] The relevant content of EntryProtocol.proto data is as follows:

[0038]

[0039]

[0040] 5. Configure the control layer service agent on CVK to detect whether the cached link is available. If it is available, obtain the Binlog message, filter out the data of the subscribed table structure, filter out the relevant data of the relevant VPC on this CVK, convert it into configuration-related data and send it to the Agent service. Otherwise, delete the cached link information and replace it with the newly established Canal cluster service link.

[0041] 6. Send a callback to notify the data control layer that the network card information data synchronization is successful.

[0042] 7. Configure the control layer service agent on CVK to periodically compare the information on CVK with the Redis data of the operation and maintenance data to see if they are the same. If they are different, report the difference data and issue an alarm.

[0043] Compared with the prior art, the present invention has the following technical innovations:

[0044] 1. Use incremental data synchronization

[0045] Data synchronization is performed through incremental topology to reduce the computing time of the Agent process on CVK.

[0046] 2. Use Binlog subscription to distribute configuration

[0047] Based on Canal's Binlog data subscription method, the data of the specified configuration is directly synchronized, reducing the interaction between the data control layer and the configuration control layer, and improving data performance.

[0048] 3. Orderly subscription to ensure reliable configuration arrival

[0049] Since Binlog data is final data, data is written in the order in which the data layer writes data. This ensures that the business order of the configuration layer and the data layer on CVK is consistent, and ensures the sequential distribution of configurations.

[0050] 4. Improve data reliability

[0051] Directly subscribing to MySQL data files can ensure that the data sources of the data layer and the configuration layer are consistent, eliminating the need for data comparison and ensuring the reliability of the configuration without comparing related data.

[0052] 5. Improve operation and maintenance capabilities

[0053] Synchronize data to Redis regularly, compare data every day, find out the difference, and send emails as alerts.

[0054] This embodiment provides a server, including a processor and a memory that is communicatively connected to the processor and is used to store instructions executable by the processor. When the processor executes the instructions, it implements the Canal-based network topology synchronization method described in the above embodiment.

[0055] This embodiment provides a readable storage medium for storing a computer program. When the computer program is executed by a processor, the canal-based network topology synchronization method described in the above embodiment is implemented.

[0056] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0057] In the several embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the division of the units described above is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The above-mentioned units may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present invention.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and description of the present invention.

[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A network topology synchronization method based on Canal, characterized in that: include: In response to the normal operation of the Canal cluster, the CVK configuration distribution layer Agent service is started and a connection is established with the Canal cluster; The Agent service subscribes to the database table information from the Canal cluster; In response to the subscription request of the Agent service, when the database table information changes, the CVK cluster returns the changed database table information to CVK; CVK sends the changed database table information to the Agent service.

2. The network topology synchronization method based on Canal according to claim 1, characterized in that: In response to the normal operation of the Canal cluster, the CVK configuration distribution layer Agent service starts and establishes a connection with the Canal cluster, which also includes: After the Agent service is connected to the Canal cluster, it needs to cache the connection information and perform periodic checks to check whether the connection is abnormal.

3. The network topology synchronization method based on Canal according to claim 1, characterized in that: In response to the subscription request from the Agent service, when the database table information changes, the CVK cluster returns the changed database table information to CVK, including: Get the relevant information of the online virtual network card and write it to the specified MySQL data table; Canal cluster filters out changed data based on the MySQL data table; Convert the changed data into data of the specified format and protocol EntryProtocol.proto, send a notification, and synchronize the operation and maintenance data to Redis; CVK filters out relevant data from the subscribed database table information, converts it, and sends it to the Agent service.

4. The network topology synchronization method based on Canal according to claim 3, characterized in that: The CVK cluster returns the changed database table information to CVK, including: The control layer configured on CVK detects whether the link between the cached Agent service and the Canal cluster is available. If it is available, it obtains the Binlog message, filters out the data of the subscribed database table information, filters the relevant data of the relevant VPC on this CVK, converts it into configuration-related data and sends it to the Agent service. Otherwise, it deletes the cached link information and replaces it with the newly established Canal cluster service link.

5. The network topology synchronization method based on Canal according to claim 3, characterized in that: The control layer configured on CVK periodically compares the information on CVK with the Redis data of the operation and maintenance data to see if they are the same. If they are different, the difference data is reported and an alarm is issued.

6. A server comprising a processor and a memory connected to the processor for storing instructions executable by the processor, characterized in that: When executing the instructions, the processor implements the Canal-based network topology synchronization method described in any one of claims 1-5.

7. A readable storage medium for storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the Canal-based network topology synchronization method according to any one of claims 1 to 5.

Images