A VPC configuration delivery method based on KAFKA

Abstract

The present invention provides a VPC configuration delivery method based on KAFKA, comprising reading and parsing a virtual machine VPC configuration to obtain virtual machine VPC online information and configuration parameters of a device to be assigned; obtaining configuration information under a specified topic based on a KAFKA cluster processing platform; calculating corresponding HashCode values according to the ID of the device to be assigned and the ID of the virtual machine VPC; performing module calculation on the KAFKA cluster processing platform configuration information and the HashCode value, calculating the configuration parameters of the device to be assigned and sending them to a specified partition of the KAFKA cluster processing platform, and then delivering them asynchronously; the KAFKA cluster processing platform performs asynchronous monitoring and records the delivery operation log. By utilizing the multi-partition feature of KAFKA, while improving the throughput, multiple partitions can also be specified under the same topic, and multiple partitions can then specify multiple consumers for parallel consumption. Finally, by utilizing the multi-consumer feature, the throughput requirement for delivering VPC configuration can be achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of VPC configuration orderly delivery equipment, and in particular to a VPC configuration delivery method based on KAFKA. Background Art

[0002] As the foundation of cloud computing networks, VPC provides users with customized private networks, with logical isolation between different VPCs. As the foundation of cloud computing communications, VPC configurations must be accurately, efficiently, orderly, and persistently delivered to designated devices. VPC configurations on devices fall into three categories:

[0003] Specifically, the configurations of firewalls, leaf switches, and border switches are interdependent and must be distributed to each device in an orderly manner. Because the VPC configuration distribution process requires order, efficiency, low latency, and persistence, and must ensure that configurations are distributed to designated devices in an orderly manner even under extreme conditions such as network fluctuations, it is necessary to leverage KAFKA's high throughput, data persistence, and high availability. This requires improvements to the orderliness, efficiency, persistence, and stability of the VPC configuration distribution process to meet product requirements for cloud computing resource creation, network communication, and tenant data isolation.

[0004] Based on this, the VPC configuration delivery method based on KAFKA proposed in this application relies on the characteristics of KAFKA consumer groups, partitions, asynchronous data flushing, etc., which greatly avoids the instability of HTTP requests and the disadvantages of data persistence, improves the success rate of configuration delivery, and at the same time improves the disadvantage that multiple devices and multiple operation configurations cannot reach related devices in an orderly manner, so as to achieve the advantage of quickly responding to related industries under distributed high-concurrency business needs. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a VPC configuration delivery method based on KAFKA to solve the problems raised in the above background technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a VPC configuration delivery method based on KAFKA, comprising the following steps:

[0007] The first step is to read and parse the configured VM VPC to obtain the VM VPC online information and the configuration parameters of the device to be allocated;

[0008] The second step is to obtain the configuration information of the specified topic based on the KAFKA cluster processing platform. The configuration information of the KAFKA cluster processing platform includes the number of partitions and partition-related parameters.

[0009] Step 3: Calculate the corresponding HashCode values based on the ID of the device to be assigned and the ID of the virtual machine VPC.

[0010] The fourth step is to calculate the configuration parameters of the device to be allocated by taking the modulus of the KAFKA cluster processing platform configuration information and the HashCode value, and then send them to the designated partition of the KAFKA cluster processing platform for asynchronous distribution.

[0011] In the fifth step, the KAFKA cluster processing platform performs asynchronous monitoring and records the operation log.

[0012] As an improvement to the VPC configuration delivery method based on KAFKA described in the present invention, after obtaining the virtual machine VPC online information and the configuration parameters of the device to be allocated, and before obtaining the configuration information of the specified topic of the KAFKA cluster processing platform, it also includes:

[0013] The configuration parameters of the device to be allocated need to be assembled into the message format required by the device to be allocated.

[0014] As an improvement to the VPC configuration delivery method based on KAFKA described in the present invention, in the first step, if the acquisition of the virtual machine VPC online information and the configuration parameters of the device to be allocated is unsuccessful, or

[0015] If the configuration parameters of the device to be allocated are not assembled into the message format required by the device to be allocated, the data needs to be rolled back until it succeeds.

[0016] As an improvement to the KAFKA-based VPC configuration delivery method described in the present invention, in the second step, if the KAFKA cluster processing platform fails to obtain the configuration information under its specified topic, it also includes rolling back the unsuccessful configuration information and retracing the configuration information under the specified topic until it is successfully obtained.

[0017] As an improvement to the KAFKA-based VPC configuration delivery method described in the present invention, in the fourth step, the asynchronous delivery method is firewall configuration delivery, Leaf switch configuration delivery, or Border switch configuration delivery.

[0018] As an improvement to the KAFKA-based VPC configuration delivery method described in the present invention, if the asynchronous delivery fails, data rollback is performed, and the virtual machine VPC configuration is reread and parsed based on the first step until the delivery is successful.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] 1. The present invention first utilizes the characteristic that the KAFKA consumer can only consume a single partition, and distributes the data of the same device to a unified partition to ensure the orderliness of the configuration and distribution of devices;

[0021] 2. Secondly, we leverage the multi-partition feature of Kafka to improve throughput. Kafka allows you to specify multiple partitions for the same topic, and multiple partitions can be assigned multiple consumers for parallel consumption. This multi-consumer feature helps achieve the throughput requirements set by the VPC configuration.

[0022] 3. Thirdly, due to the multi-node distribution characteristics of distributed services, data needs to be backed up on multiple nodes. When the VPC configuration is sent to KAKFA, multiple copies need to save the data. Therefore, the present invention uses KAFKA partition replicas to ensure partition fault tolerance of the VPC configuration. This ensures that when some nodes are abnormal, the VPC configuration can still be effectively sent to the designated device.

[0023] 4. By establishing the ACK confirmation mechanism of KAKFA messages, the impact of network fluctuations on configuration delivery is eliminated. Furthermore, when network fluctuations prevent confirmation, KAFKA can repeatedly send messages, thereby ensuring that the VPC configuration delivery service can receive unprocessed VPC configuration information and re-deliver it, ensuring that the configuration can be effectively delivered to the designated device after network fluctuations. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The disclosure of the present invention is described with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. In the drawings, the same reference numerals are used to refer to the same components. Among them:

[0025] Figure 1 This is a specific implementation flowchart of the KAFKA-based VPC configuration delivery method proposed in one embodiment of the present invention;

[0026] Figure 2 The figure is a flowchart of a KAFKA-based VPC configuration delivery method proposed in one embodiment of the present invention. DETAILED DESCRIPTION

[0027] It is easy to understand that according to the technical solution of the present invention, without changing the essential spirit of the present invention, a person skilled in the art can propose a variety of interchangeable structural modes and implementation modes. Therefore, the following specific embodiments and drawings are only exemplary descriptions of the technical solution of the present invention and should not be regarded as the entire invention or as a limitation or restriction of the technical solution of the present invention.

[0028] like Figure 1-2As shown, the present invention provides a VPC configuration delivery method based on KAFKA, comprising the following steps:

[0029] The first step is to read and parse the configured VM VPC to obtain the VM VPC online information and the configuration parameters of the device to be allocated;

[0030] The second step is to obtain the configuration information of the specified topic based on the KAFKA cluster processing platform. The configuration information of the KAFKA cluster processing platform includes the number of partitions and partition-related parameters. The purpose is to utilize the multi-copy feature of the partitions to support VPC parallel delivery, improve throughput, and eliminate the disadvantage of low concurrency of HTTP synchronous response. It should be noted that after obtaining the virtual machine VPC online information and the configuration parameters of the device to be allocated, and before obtaining the configuration information of the specified topic of the KAFKA cluster processing platform, it is also required to assemble the configuration parameters of the device to be allocated into the message format required by the device to be allocated, where the message format is the agent basic message.

[0031] Step 3: Calculate the corresponding HashCode values based on the ID of the device to be assigned and the ID of the virtual machine VPC.

[0032] The fourth step is to calculate the configuration parameters of the device to be allocated by taking the modulus of the KAFKA cluster processing platform configuration information and the HashCode value, and then send them to the designated partition of the KAFKA cluster processing platform for asynchronous delivery. The asynchronous delivery method is firewall configuration delivery, Leaf switch configuration delivery, or Border switch configuration delivery. It should be noted that the HashCode modulus and the number of KAFKA topic partitions are used to ensure that messages can be delivered to the designated devices in an orderly manner.

[0033] The fifth step is to asynchronously send the device configuration message to the KAFKA server and perform asynchronous monitoring. When the KAFKA cluster processing platform performs asynchronous monitoring, after receiving the message sending callback, it records the operation log of this delivery and completes the VPC configuration delivery instruction. It can be understood that if the asynchronous delivery is unsuccessful, the data will be rolled back. Based on the first step, the virtual machine VPC configuration is re-read and parsed until the delivery is successful. By recording persistent logs, the configuration that failed to be delivered can be traced back and the configuration can be re-delivered to ensure the persistence of the VPC configuration. The specific implementation is: KFAKA data is persisted to the server's disk in the form of log files and split into multiple small files.

[0034] In one embodiment of the present invention, if the acquisition of the virtual machine VPC online information and the configuration parameters of the device to be allocated is unsuccessful, or

[0035] If the configuration parameters of the device to be allocated are not assembled into the message format required by the device to be allocated, the data needs to be rolled back until it succeeds.

[0036] At the same time, if the configuration information under the specified topic is unsuccessful based on the KAFKA cluster processing platform, the unsuccessful configuration information will be rolled back and the configuration information under the specified topic will be backtracked again until it is successfully obtained.

[0037] Based on the above technical concept, it should be noted that the KFAKA cluster processing platform proposed in the present invention uses OFFSET as a consumption identifier, and can consume data with specified related OFFSETs in automatic and manual modes. At the same time, VPC configuration can use this feature to perform repeated consumption, repeated confirmation, and configuration backtracking for problematic configurations. The VPC configuration delivery method based on KAFKA can quickly backtrack VPC failed configurations, improve the throughput of VPC configuration delivery, and ensure that VPC configurations can be delivered to various devices in an orderly manner.

[0038] The technical scope of the present invention is not limited to the contents of the above description. Those skilled in the art can make various deformations and modifications to the above embodiments without departing from the technical concept of the present invention, and these deformations and modifications should all fall within the protection scope of the present invention.

Claims

1. A VPC configuration delivery method based on KAFKA, characterized by: The following steps are involved: The first step is to read and parse the configured VM VPC to obtain the VM VPC online information and the configuration parameters of the device to be allocated; The second step is to obtain the configuration information of the specified topic based on the KAFKA cluster processing platform. The configuration information of the KAFKA cluster processing platform includes the number of partitions and partition-related parameters. Step 3: Calculate the corresponding HashCode values based on the ID of the device to be assigned and the ID of the virtual machine VPC. The fourth step is to perform a modulo calculation on the number of partitions of the KAFKA cluster processing platform based on the HashCode value of the device ID to be assigned, and determine the target partition. The configuration parameters of the device to be assigned are sent to the target partition and then asynchronously distributed. In the fifth step, the KAFKA cluster processing platform performs asynchronous monitoring and records the operation log.

2. The method for issuing VPC configuration based on KAFKA according to claim 1, characterized in that: After obtaining the virtual machine VPC online information and the configuration parameters of the device to be allocated, and before obtaining the configuration information of the specified topic of the Kafka cluster processing platform, it also includes: The configuration parameters of the device to be allocated need to be assembled into the message format required by the device to be allocated.

3. A KAFKA-based VPC configuration delivery method according to claim 1 or 2, characterized in that: In the first step, if the acquisition of the virtual machine VPC online information and the configuration parameters of the device to be allocated is unsuccessful, or If the configuration parameters of the device to be allocated are not assembled into the message format required by the device to be allocated, the data needs to be rolled back until it succeeds.

4. The method for issuing VPC configuration based on KAFKA according to claim 1, characterized in that: In the second step, if the configuration information under the specified topic is not successfully obtained based on the KAFKA cluster processing platform, the unsuccessful configuration information is rolled back and the configuration information under the specified topic is re-traced until it is successfully obtained.

5. The method for issuing VPC configuration based on KAFKA according to claim 1, characterized in that: In the fourth step, the asynchronous sending mode is sending the firewall configuration, sending the Leaf switch configuration, or sending the Border switch configuration.

6. A KAFKA-based VPC configuration delivery method according to claim 1 or 5, characterized in that: If the asynchronous delivery fails, the data is rolled back and the virtual machine VPC configuration is re-read and parsed based on the first step until the delivery is successful.

Images