Kafka cluster replica migration processing method and related device

Abstract

The application provides a Kafka cluster replica migration processing method and related equipment, and relates to the technical field of computers. The method comprises the following steps: in response to a migration processing instruction of a Kafka cluster replica, determining the target type of the migration processing instruction; determining one or more parameters associated with the target type, wherein the parameters comprise the starting migration position and the ending migration position of each replica corresponding to each partition of an associated topic (Topic) in the Kafka cluster, the input flow of each replica corresponding to each partition of the associated topic (Topic), and the size of each replica, wherein the one or more parameters are collected and output by a collection component according to a preset collection index; and outputting the migration processing condition of the Kafka cluster replica according to the associated one or more parameters. The application can accurately estimate the remaining time consumption of migration and accurately determine whether the migration is stagnant.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for processing Kafka cluster replica migration and related equipment. Background Technology

[0002] Kafka, as a high-throughput distributed message middleware, has become a core component for building real-time data stream processing architectures and is widely used in finance, telecommunications, government affairs and other fields.

[0003] During operation and maintenance, due to hardware failures, data center relocation, scaling up or down, or node maintenance, it is often necessary to migrate replicas of corresponding topics in a Kafka cluster. Existing Kafka replica migration solutions, such as native Kafka replica migration scripts, have problems such as the inability of operations and maintenance personnel to obtain fine-grained data on migration progress in real time after the migration task is submitted and executed, the inability to estimate the remaining migration time, and the difficulty in determining whether the migration has stalled.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a method, apparatus, device, storage medium and product for processing Kafka cluster replica migration, which aims to solve the problems in the current technology where, after the migration task is submitted for execution, the operation and maintenance personnel cannot obtain fine-grained data on the migration progress in real time, cannot estimate the remaining migration time, and find it difficult to determine whether the migration has stalled.

[0006] To achieve the above objectives, this application proposes a method for migrating Kafka cluster replicas, which includes: In response to a migration processing instruction for a Kafka cluster replica, determine the target type of the migration processing instruction; Determine one or more parameters associated with the target type, wherein the parameters include the start migration position and end migration position of each replica corresponding to each partition of the Topic associated with the migration processing instruction in the Kafka cluster, the input traffic of each replica corresponding to each partition of the associated Topic and the size of each replica, wherein the one or more parameters are collected and output by the collection component according to the preset collection indicators; Based on one or more of the associated parameters, output the migration processing status of the Kafka cluster replicas.

[0007] In one embodiment, the Topic associated with the migration processing instruction includes the Topic to be migrated, and prior to the step of determining the target type of the migration processing instruction in response to the migration processing instruction of the Kafka cluster replica, the method includes: Identify the topic to be migrated and the target topic; Based on the partition information of the Topic to be migrated and the target Topic, the rack information of each Broker node in the cluster, the replica distribution information of each Broker node, and the migration rate limiting configuration, a migration plan is generated. Based on the migration scheme, the preset collection indicators are determined.

[0008] In one embodiment, after the step of outputting the migration processing status of the Kafka cluster replicas based on one or more parameters associated with the migration query type, the method includes: Determine the migration progress of the Topic to be migrated based on the start and end migration positions of each replica. If the migration progress is greater than the preset progress, the preset migration verification process is invoked to automatically check the migration results and obtain the check results. The preset migration verification process includes a traffic verification process and a consumer group verification process. If the inspection results are satisfactory, the migration is considered complete.

[0009] In one embodiment, after the step of invoking a preset migration verification process to automatically check the migration results and obtain the check results, the following steps are included: If traffic verification or consumer group verification fails, proceed with any of the following steps: Based on the current partition distribution of the Topic and the previous migration plan, a migration rollback plan is generated, and an automatic rollback is performed based on the migration rollback plan, generating a rollback migration record. An alarm message is generated, which includes the migration plan, migration verification results, input and output traffic, consumer group-related indicators, and responsible person information.

[0010] In one embodiment, the Kafka cluster replica migration processing method further includes: Based on the real-time processing metrics of the Kafka cluster, the processing metrics include input / output traffic metrics, CPU utilization of each Broker node, memory utilization, network thread idle rate, and processing thread idle rate. The migration rate limiting configuration is dynamically adjusted based on the processing metrics and preset dynamic adjustment rules.

[0011] In one embodiment, after the step of outputting the migration processing status of the Kafka cluster replicas based on one or more of the associated parameters, the method includes: Receive visualization instructions for Kafka cluster replica migration processing, and visualize the target type according to the visualization instructions. The visualization page is associated with the target type of the migration processing instructions.

[0012] Furthermore, to achieve the above objectives, this application also proposes a Kafka cluster replica migration processing apparatus, which includes: The response module is used to respond to migration processing instructions for Kafka cluster replicas and determine the target type of the migration processing instructions; The first determining module is used to determine one or more parameters associated with the target type. The parameters include the start migration position and end migration position of each replica corresponding to each partition of the Topic associated with the migration processing instruction in the Kafka cluster, the input traffic of each replica corresponding to each partition of the associated Topic, and the size of each replica. The one or more parameters are collected and output by the collection component according to the preset collection indicators. The second determining module is used to output the migration processing status of the Kafka cluster replicas based on one or more of the associated parameters.

[0013] In addition, to achieve the above objectives, this application also proposes a Kafka cluster replica migration processing device, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the Kafka cluster replica migration processing method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the Kafka cluster replica migration processing method described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the Kafka cluster replica migration processing method described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: Compared to related technologies, which suffer from issues such as the inability of operations and maintenance personnel to obtain fine-grained data on migration progress in real time after a migration task is submitted and executed, the inability to estimate the remaining migration time, and the difficulty in determining whether the migration has stalled, this application addresses the problem of determining the target type of a migration processing instruction for a Kafka cluster replica in response to the migration processing instruction; determining one or more parameters associated with the target type, wherein the parameters include the start and end migration positions of each replica corresponding to each partition of the topic associated with the migration processing instruction in the Kafka cluster, the input traffic of each replica corresponding to each partition of the associated topic, and the size of each replica; wherein the one or more parameters are collected and output by a collection component according to preset collection indicators; and outputting the migration processing status of the Kafka cluster replica based on the one or more associated parameters. It is understood that in this application, once the migration processing instruction is obtained, the corresponding migration processing status can be output based on one or more parameters associated with the target type of the migration processing instruction (including the start and end migration positions of each replica corresponding to each partition of the associated Topic, the input traffic of each replica corresponding to each partition of the associated Topic, and the size of each replica). That is, in this application, because the collection indicators are pre-set, one or more parameters can be obtained in a timely manner. Therefore, based on these obtained parameters, the processing status corresponding to the instruction can be output, allowing operations and maintenance personnel to obtain fine-grained data on migration progress and remaining migration time in real time, to determine whether the migration has stalled. Thus, this solves the problems existing in current technologies where, after a migration task is submitted for execution, operations and maintenance personnel cannot obtain fine-grained data on migration progress in real time, cannot estimate remaining migration time, and find it difficult to determine whether the migration has stalled. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the Kafka cluster replica migration processing method in this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the Kafka cluster replica migration processing method in this application. Figure 3 This is a flowchart illustrating Embodiment 3 of the Kafka cluster replica migration processing method in this application; Figure 4 This is a schematic diagram of the module structure of the Kafka cluster replica migration processing device according to an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the Kafka cluster replica migration processing method in the embodiments of this application; Figure 6 A schematic diagram of a first scenario provided for an embodiment of the Kafka cluster replica migration processing method of this application; Figure 7 This is a schematic diagram of a second scenario provided for an embodiment of the Kafka cluster replica migration processing method in this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of this application embodiment is as follows: In response to a migration processing instruction for a Kafka cluster replica, determine the target type of the migration processing instruction; determine one or more parameters associated with the target type, wherein the parameters include the start and end migration positions of each replica corresponding to each partition of the Topic associated with the migration processing instruction in the Kafka cluster, the input traffic of each replica corresponding to each partition of the associated Topic, and the size of each replica, wherein the one or more parameters are collected and output by a collection component corresponding to a preset collection index; and output the migration processing status of the Kafka cluster replica based on the one or more associated parameters.

[0024] It is understood that in this application, once the migration processing instruction is obtained, the corresponding migration processing status can be output based on one or more parameters associated with the target type of the migration processing instruction (including the start and end migration positions of each replica corresponding to each partition of the associated Topic, the input traffic of each replica corresponding to each partition of the associated Topic, and the size of each replica). That is, in this application, because the collection indicators are pre-set, one or more parameters can be obtained in a timely manner. Therefore, based on these obtained parameters, the processing status corresponding to the instruction can be output, allowing operations and maintenance personnel to obtain fine-grained data on migration progress and remaining migration time in real time, to determine whether the migration has stalled. Thus, this solves the problems existing in current technologies where, after a migration task is submitted for execution, operations and maintenance personnel cannot obtain fine-grained data on migration progress in real time, cannot estimate remaining migration time, and find it difficult to determine whether the migration has stalled.

[0025] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as a Kafka cluster replica migration processing device. The following description uses a Kafka cluster replica migration processing device as an example to illustrate this embodiment and the subsequent embodiments.

[0026] Based on this, this application provides a method for migrating replicas in a Kafka cluster, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the Kafka cluster replica migration processing method of this application.

[0027] In this embodiment, the Kafka cluster replica migration processing method includes steps S10~S30: Step S10: In response to the migration processing instruction of the Kafka cluster replica, determine the target type of the migration processing instruction; It should be noted that the execution entity in this embodiment is a Kafka cluster replica migration processing device. This Kafka cluster replica migration processing device can be set within Kafka (message middleware), or it can be set outside of the message middleware. There is no specific limitation.

[0028] In this embodiment, a specific application scenario may be: During operation and maintenance, due to reasons such as hardware failure, data center relocation, scaling up or down, or node maintenance, it is often necessary to migrate replicas in the Kafka cluster.

[0029] In this embodiment, it could be in a financial scenario where a hardware failure necessitates the migration of replicas in a Kafka cluster.

[0030] In this embodiment, specifically, a replica from one node in a Kafka cluster is migrated to another node.

[0031] In this embodiment, in response to a migration processing instruction for a Kafka cluster replica, the target type of the migration processing instruction is determined; In this context, the migration instructions for Kafka cluster replicas can be either operation instructions or query instructions.

[0032] In this embodiment, the operation instructions may specifically be: execute migration operation, execute pause migration operation, etc., and the query instructions may specifically be migration progress query, migration time query, etc., without any specific limitation.

[0033] In this embodiment, determining the target type of the migration processing instruction specifically refers to determining whether the migration processing instruction is to perform a migration operation or a query operation such as a migration progress query or a migration time query.

[0034] The migration processing instruction is associated with a topic including the topic to be migrated. Prior to the step of determining the target type of the migration processing instruction in response to a Kafka cluster replica, the following steps are included: Step S01: Determine the Topic to be migrated and the target Topic; In this embodiment, before performing the Kafka cluster replica migration process, it is necessary to first determine the Topic to be migrated and the target Topic. It should be noted that a Kafka cluster includes multiple Broker nodes, each Broker node includes multiple Topics, and each Topic includes multiple partitions. Therefore, before migration, it is necessary to determine the Topic to be migrated and the target Topic.

[0035] Step S02: Generate a migration plan based on the partition information of the Topic to be migrated and the target Topic, the rack information of each Broker node in the cluster, the replica distribution information of each Broker node, and the set migration rate limiting configuration. In this embodiment, it should be noted that when allocating partition replicas, Kafka tries to distribute multiple replicas of the same partition across different racks. This ensures that even if the entire rack fails (such as a power outage or network interruption), not all replicas will become unavailable, thus guaranteeing high data availability. Therefore, a rack identifier needs to be specified for each Broker node.

[0036] In other words, in this embodiment, the rack information (RackID) of the Broker node in the Kafka cluster is an optional configuration item, mainly used to implement the rack-awareness replica allocation strategy to improve the cluster's disaster recovery capability and data reliability.

[0037] In this embodiment, a migration plan is generated based on the partition information of the Topic to be migrated and the target Topic, the rack information (RackID) of each Broker node in the cluster, the replica distribution information of each Broker node, and the migration rate limiting configuration.

[0038] Specifically, in this embodiment, during the migration scheme generation process, based on the relevant data of the Topic to be migrated, the relevant data of the target Topic, the overall migration inflow rate limiting configuration constraints, and the migration outflow rate limiting configuration constraints, a planning and scheduling mathematical model is constructed with the goal of optimizing the migration turnover rate; the migration scheme is obtained through the planning and scheduling mathematical model.

[0039] In this embodiment, the migration scheme can be visualized and compared with the original distribution of replicas, while also supporting manual adjustments.

[0040] In this embodiment, the migration rate limiting configuration can be dynamically adjusted to meet the rate limiting requirements in different scenarios.

[0041] Step S03: Determine the preset collection indicators according to the migration scheme.

[0042] In this embodiment, after determining the migration plan, preset data collection metrics are also determined, and corresponding data collection components (metric collection Java programs) are set up. Specifically, the developed metric collection Java program collects various real-time metrics of the cluster from the Kafka server's JMX and outputs the metrics to the Topic. Then, the Flink application consumes the collected metric data in real time and stores it in the OLAP database. Therefore, the input traffic and Topic size of the Topic can be queried in real time from the OLAP database, and the startOffset and endOffset of each replica of each partition of the Topic can be queried in real time.

[0043] Step S20: Determine one or more parameters associated with the target type, wherein the parameters include the start migration position and end migration position of each replica corresponding to each partition of the Topic associated with the migration processing instruction in the Kafka cluster, the input traffic of each replica corresponding to each partition of the associated Topic and the size of each replica, wherein the one or more parameters are collected and output by the collection component according to the preset collection indicators; In this embodiment, since preset acquisition indicators are collected and output in advance by the acquisition component, one or more parameters associated with the target type can be directly obtained after the target type is obtained.

[0044] In this embodiment, when the migration processing instruction is a migration time estimation instruction, the input traffic and size of the Topic are queried in real time from the OLAP database, and the migration time is estimated by combining the migration rate limiting configuration with the number of iterations of each target replica. The calculation logic for the migration time estimate is as follows: Calculate the original leader replica output rate limit: Node rate limit / Original number of leader replicas; Calculate the target follower replica input rate limit: Node rate limit / Number of target node follower replicas; Calculate the final replica rate limit: min(original leader replica output rate limit, target follower replica input rate limit); Estimated partition migration time = (Current size of original leader replica - Current size of target follower replica) / (Final replica rate limit - Leader replica input flow rate); In this embodiment, a Topic includes multiple partitions; therefore, the estimated migration time of a Topic is equal to the estimated migration time of a partition.

[0045] In this embodiment, when the migration processing instruction is a migration progress instruction, the startOffset and endOffset of each replica of each partition of the Topic are queried in real time, the migration progress of each partition of the Topic is calculated, the migration progress of the Topic is obtained based on the migration progress of each partition, and the migration progress in the corresponding migration task record is updated.

[0046] The migration progress calculation logic is as follows: Partition migration progress = min(target replica endOffset - target replica startOffset) / max(original replica endOffset - original replica startOffset); In this embodiment, a Topic includes multiple partitions; therefore, the Topic migration progress = min(the migration progress of each partition of the Topic). After the step of outputting the migration processing status of the Kafka cluster replicas based on one or more of the associated parameters, the following steps are included: Step S40: Receive a visualization instruction for Kafka cluster replica migration processing, and visualize the target type according to the visualization instruction. The visualization page is associated with the target type of the migration processing instruction.

[0047] In this embodiment, it also supports visual viewing of migration progress and estimated migration time. For example... Figure 6 As shown in this embodiment, during the visualization process, the target type of the migration processing instruction currently being queried is also highlighted so that the user can focus on it (the focus or highlight may be different for each visualization).

[0048] Compared to related technologies, which suffer from issues such as the inability of operations and maintenance personnel to obtain fine-grained data on migration progress in real time after a migration task is submitted and executed, the inability to estimate the remaining migration time, and the difficulty in determining whether the migration has stalled, this application addresses the problem of determining the target type of a migration processing instruction for a Kafka cluster replica in response to the migration processing instruction; determining one or more parameters associated with the target type, wherein the parameters include the start and end migration positions of each replica corresponding to each partition of the topic associated with the migration processing instruction in the Kafka cluster, the input traffic of each replica corresponding to each partition of the associated topic, and the size of each replica; wherein the one or more parameters are collected and output by a collection component according to preset collection indicators; and outputting the migration processing status of the Kafka cluster replica based on the one or more associated parameters. It is understood that in this application, once the migration processing instruction is obtained, the corresponding migration processing status can be output based on one or more parameters associated with the target type of the migration processing instruction (including the start and end migration positions of each replica corresponding to each partition of the associated Topic, the input traffic of each replica corresponding to each partition of the associated Topic, and the size of each replica). That is, in this application, because the collection indicators are pre-set, one or more parameters can be obtained in a timely manner. Therefore, based on these obtained parameters, the processing status corresponding to the instruction can be output, allowing operations and maintenance personnel to obtain fine-grained data on migration progress and remaining migration time in real time, to determine whether the migration has stalled. Thus, this solves the problems existing in current technologies where, after a migration task is submitted for execution, operations and maintenance personnel cannot obtain fine-grained data on migration progress in real time, cannot estimate remaining migration time, and find it difficult to determine whether the migration has stalled.

[0049] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2After the step of outputting the migration processing status of the Kafka cluster replicas based on one or more parameters associated with the migration query type, the following steps are included: Step A1: Determine the migration progress of the Topic to be migrated based on the start and end migration positions of each replica. Step A2: If the migration progress is greater than the preset progress, then the preset migration verification process is invoked to automatically check the migration results and obtain the check results. The preset migration verification process includes a traffic verification process and a consumer group verification process. Step A3: If the inspection result is satisfactory, then the migration is considered complete.

[0050] In this embodiment, the migration progress is first obtained (the method of obtaining the progress is as described above and will not be repeated). For topics with a migration progress greater than 90%, KafkaAdminClient is periodically called to automatically check the migration results. The call interval is configurable (to meet the needs of different scenarios and avoid affecting normal business operations). If all replicas of a topic have been migrated, the status of the corresponding migration task record is updated, and automated verification of the migration results is started. Figure 7 As shown, automated verification includes traffic self-check (traffic verification process) and consumer group self-check (consumer group verification process). The core logic of traffic self-check is to compare the input and output traffic within 5 minutes before and after the Topic migration is completed to determine whether there is a traffic interruption or sudden drop (traffic change greater than a preset change threshold). If so, the traffic self-check fails. The core logic of consumer group self-check is to determine whether the consumer group has stopped submitting offsets or has a consumption delay greater than 5 minutes based on the consumer group's metrics within 5 minutes before and after the migration (example provided, can be modified). If the consumer group stops submitting offsets or has a consumption delay greater than 5 minutes, the consumer group self-check fails. In this embodiment, it should be noted that this embodiment also supports visual viewing of the Topic's input and output traffic metrics and the consumer group's consumption delay-related metrics.

[0051] The Kafka cluster replica migration processing method also includes: Step B1: Based on the real-time processing metrics of the Kafka cluster, the processing metrics include input / output traffic metrics, CPU utilization of each Broker node, memory utilization, network thread idle rate, and processing thread idle rate. Step B2: Dynamically adjust the migration rate limiting configuration according to the processing indicators and the preset dynamic adjustment rules.

[0052] In this embodiment, based on real-time input / output traffic metrics of the cluster, as well as metrics such as CPU utilization, memory utilization, network thread idle rate, and processing thread idle rate of each Broker node, and preset dynamic adjustment rules, if any relevant metric is detected to be below a set threshold, the migration rate limiting configuration can be automatically increased according to the configuration; if any relevant metric is detected to be above the set threshold, the migration rate limiting configuration can be automatically decreased according to the configuration. This configuration ensures the stable operation of the cluster and also improves migration efficiency.

[0053] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 After the step of calling the preset migration verification process to automatically check the migration results and obtain the check results, the following steps are included: Step C1: If traffic verification or consumer group verification fails, proceed with any of the following steps: Step C2: Generate a migration rollback plan based on the current partition distribution of the Topic and the previous migration plan, and automatically roll back based on the migration rollback plan, and generate a rollback migration record. Step C3: Generate alarm information, which includes migration plan, migration verification results, input and output traffic, consumer group related indicators, and responsible person information.

[0054] In this embodiment, if traffic verification fails or consumer group verification fails, one or more of the following processes will be performed: First: If the traffic self-check or the consumer group self-check fails, an alarm will be issued. Second: Generate a migration rollback plan based on the current partition distribution of the Topic and the previous migration plan, and automatically roll back based on the migration rollback plan, generating a rollback migration record. Specifically, automatic rollback can be performed according to the configuration. The core of automatic rollback is to generate a rollback migration record for the corresponding Topic, generate a migration rollback plan based on the current partition distribution of the Topic and the previous persistent migration plan, and then automatically roll back. If automatic rollback is not configured, after receiving an alarm, the operations and maintenance personnel can set up a post-processing plan based on the details of the Topic migration. The post-processing plan includes the migration plan, migration verification results, input and output traffic, consumer group related indicators, upstream and downstream and their responsible persons information, etc. In the post-processing plan, a group can be quickly created for problem localization and investigation, and operations and maintenance personnel can quickly roll back with one click when needed.

[0055] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the Kafka cluster replica migration processing method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0056] This application also provides a Kafka cluster replica migration processing device; please refer to... Figure 4 The Kafka cluster replica migration processing device includes: Response module 10 is used to respond to migration processing instructions for Kafka cluster replicas and determine the target type of the migration processing instructions; The first determining module 20 is used to determine one or more parameters associated with the target type. The parameters include the start migration position and end migration position of each replica corresponding to each partition of the Topic associated with the migration processing instruction in the Kafka cluster, the input traffic of each replica corresponding to each partition of the associated Topic, and the size of each replica. The one or more parameters are collected and output by the collection component according to the preset collection indicators. The second determining module 30 is used to output the migration processing status of the Kafka cluster replicas based on one or more of the associated parameters.

[0057] In one embodiment, the Topic associated with the migration processing instruction includes the Topic to be migrated, and the Kafka cluster replica migration processing device is used to implement: Identify the topic to be migrated and the target topic; Based on the partition information of the Topic to be migrated and the target Topic, the rack information of each Broker node in the cluster, the replica distribution information of each Broker node, and the migration rate limiting configuration, a migration plan is generated. Based on the migration scheme, the preset collection indicators are determined.

[0058] In one embodiment, the Kafka cluster replica migration processing device is used to implement: Determine the migration progress of the Topic to be migrated based on the start and end migration positions of each replica. If the migration progress is greater than the preset progress, the preset migration verification process is invoked to automatically check the migration results and obtain the check results. The preset migration verification process includes a traffic verification process and a consumer group verification process. If the inspection results are satisfactory, the migration is considered complete.

[0059] In one embodiment, the Kafka cluster replica migration processing device is used to implement: If traffic verification or consumer group verification fails, proceed with any of the following steps: Based on the current partition distribution of the Topic and the previous migration plan, a migration rollback plan is generated, and an automatic rollback is performed based on the migration rollback plan, generating a rollback migration record. An alarm message is generated, which includes the migration plan, migration verification results, input and output traffic, consumer group-related indicators, and responsible person information.

[0060] In one embodiment, the Kafka cluster replica migration processing device is used to implement: Based on the real-time processing metrics of the Kafka cluster, the processing metrics include input / output traffic metrics, CPU utilization of each Broker node, memory utilization, network thread idle rate, and processing thread idle rate. The migration rate limiting configuration is dynamically adjusted based on the processing metrics and preset dynamic adjustment rules.

[0061] In one embodiment, the Kafka cluster replica migration processing device is used to implement: Receive visualization instructions for Kafka cluster replica migration processing, and visualize the target type according to the visualization instructions. The visualization page is associated with the target type of the migration processing instructions.

[0062] The Kafka cluster replica migration processing apparatus provided in this application, employing the Kafka cluster replica migration processing method in the above embodiments, can solve the technical problem of Kafka cluster replica migration processing. Compared with the prior art, the beneficial effects of the Kafka cluster replica migration processing apparatus provided in this application are the same as those of the Kafka cluster replica migration processing method provided in the above embodiments, and other technical features in the Kafka cluster replica migration processing apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0063] This application provides a Kafka cluster replica migration processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the Kafka cluster replica migration processing method in Embodiment 1 above.

[0064] The following is for reference. Figure 5This document illustrates a structural diagram of a Kafka cluster replica migration processing device suitable for implementing embodiments of this application. The Kafka cluster replica migration processing device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, tablets, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The Kafka cluster replica migration processing device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments in this application.

[0065] like Figure 5 As shown, the Kafka cluster replica migration processing device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the Kafka cluster replica migration processing device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the Kafka cluster replica migration processing appliance to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a Kafka cluster replica migration processing appliance with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented alternatively.

[0066] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0067] The Kafka cluster replica migration processing device provided in this application, employing the Kafka cluster replica migration processing method described in the above embodiments, can solve the technical problem of Kafka cluster replica migration processing. Compared with the prior art, the beneficial effects of the Kafka cluster replica migration processing device provided in this application are the same as those of the Kafka cluster replica migration processing method provided in the above embodiments, and other technical features in this Kafka cluster replica migration processing device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0068] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0069] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0070] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the Kafka cluster replica migration processing method in the above embodiments.

[0071] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0072] The aforementioned computer-readable storage medium may be included in the Kafka cluster replica migration processing device; or it may exist independently and not be assembled into the Kafka cluster replica migration processing device.

[0073] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the Kafka cluster replica migration processing device, the Kafka cluster replica migration processing device performs the following actions: acquires single-frame raw data and point cloud data output by the LiDAR; locates high-inflection points based on the single-frame raw data and determines high-inflection regions based on the high-inflection points; combines the point cloud data and the overflow distance threshold corresponding to the high-inflection points to determine high-inflection expansion points located in the high-inflection regions from the single-frame raw data; and performs a filtering operation to remove the high-inflection expansion points.

[0074] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0075] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0076] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0077] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described Kafka cluster replica migration processing method, thereby solving the technical problem of Kafka cluster replica migration processing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the Kafka cluster replica migration processing method provided in the above embodiments, and will not be repeated here.

[0078] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the Kafka cluster replica migration processing method described above.

[0079] The computer program product provided in this application can solve the technical problem of Kafka cluster replica migration processing. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the Kafka cluster replica migration processing method provided in the above embodiments, and will not be repeated here.

[0080] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A method for handling replica migration in a Kafka cluster, characterized in that, The method includes: In response to a migration processing instruction for a Kafka cluster replica, determine the target type of the migration processing instruction; Determine one or more parameters associated with the target type, wherein the parameters include the start migration position and end migration position of each replica corresponding to each partition of the Topic associated with the migration processing instruction in the Kafka cluster, the input traffic of each replica corresponding to each partition of the associated Topic and the size of each replica, wherein the one or more parameters are collected and output by the collection component according to the preset collection indicators; Based on one or more of the associated parameters, output the migration processing status of the Kafka cluster replicas.

2. The Kafka cluster replica migration processing method according to claim 1, characterized in that, The migration processing instruction is associated with a topic including the topic to be migrated. Prior to the step of determining the target type of the migration processing instruction in response to a Kafka cluster replica, the following steps are included: Identify the topic to be migrated and the target topic; Based on the partition information of the Topic to be migrated and the target Topic, the rack information of each Broker node in the cluster, the replica distribution information of each Broker node, and the migration rate limiting configuration, a migration plan is generated. Based on the migration scheme, the preset collection indicators are determined.

3. The Kafka cluster replica migration processing method according to claim 2, characterized in that, After the step of outputting the migration processing status of the Kafka cluster replicas based on one or more parameters associated with the migration query type, the following steps are included: Determine the migration progress of the Topic to be migrated based on the start and end migration positions of each replica. If the migration progress is greater than the preset progress, the preset migration verification process is invoked to automatically check the migration results and obtain the check results. The preset migration verification process includes a traffic verification process and a consumer group verification process. If the inspection results are satisfactory, the migration is considered complete.

4. The Kafka cluster replica migration processing method according to claim 3, characterized in that, After the step of calling the preset migration verification process to automatically check the migration results and obtain the check results, the following steps are included: If traffic verification or consumer group verification fails, proceed with any of the following steps: Based on the current partition distribution of the Topic and the previous migration plan, a migration rollback plan is generated, and an automatic rollback is performed based on the migration rollback plan, generating a rollback migration record. An alarm message is generated, which includes the migration plan, migration verification results, input and output traffic, consumer group related indicators, and responsible person information.

5. The Kafka cluster replica migration processing method according to claim 1, characterized in that, The Kafka cluster replica migration processing method also includes: Based on the real-time processing metrics of the Kafka cluster, the processing metrics include input / output traffic metrics, CPU utilization of each Broker node, memory utilization, network thread idle rate, and processing thread idle rate. The migration rate limiting configuration is dynamically adjusted based on the processing metrics and preset dynamic adjustment rules.

6. The Kafka cluster replica migration processing method according to claim 1, characterized in that, After the step of outputting the migration processing status of the Kafka cluster replicas based on one or more of the associated parameters, the following steps are included: Receive visualization instructions for Kafka cluster replica migration processing, and visualize the target type according to the visualization instructions. The visualization page is associated with the target type of the migration processing instructions.

7. A Kafka cluster replica migration processing device, characterized in that, The Kafka cluster replica migration processing device includes: The response module is used to respond to migration processing instructions for Kafka cluster replicas and determine the target type of the migration processing instructions; The first determining module is used to determine one or more parameters associated with the target type. The parameters include the start migration position and end migration position of each replica corresponding to each partition of the Topic associated with the migration processing instruction in the Kafka cluster, the input traffic of each replica corresponding to each partition of the associated Topic, and the size of each replica. The one or more parameters are collected and output by the collection component according to the preset collection indicators. The second determining module is used to output the migration processing status of the Kafka cluster replicas based on one or more of the associated parameters.

8. A Kafka cluster replica migration processing device, characterized in that, The Kafka cluster replica migration processing device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the Kafka cluster replica migration processing method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the Kafka cluster replica migration processing method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the Kafka cluster replica migration processing method as described in any one of claims 1 to 6.

Images