Data consumption method and device using kafka, terminal equipment and medium

Abstract

The application relates to the technical field of financial technology, and discloses a data consumption method and device using KAFKA, a terminal device and a computer storage medium. The data consumption method using KAFKA registers a data consumption instance according to an instance registration rule through a KAFKA consumption end, wherein the instance registration rule comprises that a single partition of a single KAFKA topic corresponds to a consumption thread of a data consumption instance; load balancing operation is performed according to the weight of the data consumption instance and the partition of the KAFKA topic corresponding to the data consumption instance; and a load balancing state is maintained for data consumption of the data consumption instance. The application realizes a lightweight and high-availability data consumption scheme using KAFKA.

Description

Technical Field

[0001] This invention relates to the field of financial technology (Fintech), and more particularly to a data consumption method, apparatus, terminal device, and computer storage medium using KAFKA. Background Technology

[0002] With the development of computer technology, more and more technologies are being applied in the financial field. The traditional financial industry is gradually transforming into financial technology. However, due to the requirements of security, real-time performance and stability in the financial industry, higher demands are also being placed on technology.

[0003] Currently, in the big data field, the mainstream approach is to use Kafka (an open-source stream processing platform, a high-throughput distributed publish-subscribe messaging system that can process all action stream data of consumers on a website) for real-time or near-real-time data consumption. This is mainly achieved through real-time stream computing frameworks based on big data components such as Spark Streaming (a stream processing system that performs high-throughput, fault-tolerant processing of real-time data streams) or Apache Flink (an open-source stream processing framework whose core is a distributed stream data stream engine written in Java and Scala).

[0004] However, real-time stream computing frameworks based on big data components not only have a high barrier to entry, but also require significant machine costs and manpower for operation and maintenance, resulting in high costs for using Kafka for data consumption. Summary of the Invention

[0005] The main objective of this invention is to provide a data consumption method, apparatus, terminal device, and computer storage medium using Kafka, aiming to solve the technical problem of high data consumption costs caused by the real-time streaming computing framework based on big data components when using Kafka for data consumption.

[0006] To achieve the above objectives, the present invention provides a data consumption method using Kafka, which is applied to a Kafka consumer and includes the following steps:

[0007] Register data consumption instances according to the instance registration rules, wherein the instance registration rules include: a single partition of a single KAFKA topic corresponds to a consumption thread of a data consumption instance;

[0008] Load balancing is performed based on the weight of the data consumption instance and the partition of the KAFKA topic corresponding to the data consumption instance.

[0009] Maintain a load-balanced state for the data consumption instances to consume data.

[0010] Furthermore, to achieve the above objectives, the present invention also provides a data consumption device using KAFKA, wherein the data consumption device using KAFKA is applied to a KAFKA consumer terminal, and the data consumption device using KAFKA includes:

[0011] The instance registration module is used to register data consumption instances according to the instance registration rules, wherein the instance registration rules include: a single partition of a single KAFKA topic corresponds to a consumption thread of a data consumption instance;

[0012] The load balancing module is used to perform load balancing operations based on the weight of the data consumption instance and the partition of the KAFKA topic corresponding to the data consumption instance.

[0013] The data consumption module is used to maintain a load-balanced state for the data consumption instances to consume data.

[0014] In this invention, each functional module of the KAFKA data consumption device implements the steps of the KAFKA data consumption method described above during operation.

[0015] In addition, to achieve the above objectives, the present invention also provides a terminal device, the terminal device comprising: a memory, a processor, and a data consumption program using KAFKA stored in the memory and executable on the processor, wherein when the data consumption program using KAFKA is executed by the processor, it implements the steps of the data consumption method using KAFKA as described above.

[0016] Furthermore, to achieve the above objectives, the present invention also provides a computer storage medium storing a data consumption program using KAFKA, wherein when the data consumption program using KAFKA is executed by a processor, it implements the steps of the data consumption method using KAFKA as described above.

[0017] In addition, to achieve the above objectives, the present invention also provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the data consumption method using KAFKA as described above.

[0018] This invention provides a data consumption method, apparatus, terminal device, computer storage medium, and computer program product using Kafka. The method involves registering data consumption instances through a Kafka consumer according to instance registration rules. These rules include: a single partition of a single Kafka topic corresponds to a consumption thread of one data consumption instance; load balancing is performed based on the weight of the data consumption instance and the partition of the corresponding Kafka topic; and a load-balanced state is maintained for the data consumption instances to consume data.

[0019] When using Kafka for data consumption, this invention first registers data consumption instances on the Kafka consumer according to the instance registration rule that a single partition of a single Kafka topic can only correspond to one data consumption instance and one consumption thread. Then, the Kafka consumer performs load balancing based on the weights of all registered data consumption instances and the partition of a specific Kafka topic on the Kafka consumer corresponding to each data consumption instance. Finally, the Kafka consumer maintains a load-balanced state so that all registered data consumption instances can consume data through their corresponding partitions.

[0020] Compared to traditional methods that rely on big data components like Kafka for data consumption, this invention only requires Kafka consumer instances to register. Load balancing is then performed based on the weights of each consumer and the corresponding Kafka topic partitions, maintaining a balanced load for registered consumer instances. This achieves a lightweight, highly available data consumption solution using Kafka, eliminating the need for real-time stream processing frameworks like Spark Streaming or Apache Flink, thus reducing data consumption costs. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the device structure of the terminal device hardware operating environment involved in the embodiments of the present invention;

[0022] Figure 2 This is a flowchart illustrating an embodiment of the data consumption method using KAFKA according to the present invention;

[0023] Figure 3 This is a functional module diagram of an embodiment of the data consumption device using KAFKA according to the present invention.

[0024] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0025] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0026] Reference Figure 1 , Figure 1 This is a schematic diagram of the hardware operating environment of the terminal device involved in the embodiments of the present invention.

[0027] In this embodiment of the invention, the terminal device can be a KAFKA consumer terminal configured as a data consumption instance for data consumption. The terminal device can be a server, smartphone, PC (Personal Computer), tablet computer, portable computer, etc.

[0028] like Figure 1 As shown, the terminal device may include: a processor 1001, such as a CPU; a communication bus 1002; a user interface 1003; a network interface 1004; and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0029] Those skilled in the art will understand that Figure 1 The terminal device structure shown does not constitute a limitation on the terminal device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0030] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a data consumption program using KAFKA.

[0031] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; and the processor 1001 can be used to call the data consumption program using KAFKA stored in the memory 1005 and execute the following embodiments of the data consumption method using KAFKA.

[0032] Based on the above hardware structure, various embodiments of the data consumption method using KAFKA of the present invention are proposed.

[0033] It should be noted that currently, in the big data field, the mainstream approach to consuming data in real-time or near real-time is to use Kafka, which is mainly based on real-time stream computing frameworks such as Spark Streaming or Apache Flink.

[0034] However, real-time stream computing frameworks based on big data components not only have a high barrier to entry, but also require significant machine costs and manpower for operation and maintenance, resulting in high costs for using Kafka for data consumption.

[0035] To address the above issues, this invention provides a data consumption method using Kafka. Please refer to... Figure 2 , Figure 2 This is a flowchart illustrating a first embodiment of the data consumption method using Kafka according to the present invention. In this embodiment, the data consumption method using Kafka is applied to the terminal device configured as a Kafka consumer. This terminal device acts as a message intermediary between the data consumption instance and the data to be consumed. The data consumption method using Kafka according to the present invention includes:

[0036] Step S10: Register a data consumption instance according to the instance registration rules, wherein the instance registration rules include: a single partition of a single KAFKA topic corresponds to a consumption thread of a data consumption instance;

[0037] A terminal device configured as a KAFKA consumer can only correspond to one data consumption instance and one consumption thread for a single partition of a single KAFKA theme. However, a single data consumption instance can be registered with the registry center corresponding to the current KAFKA consumer by creating multiple consumption threads to correspond to multiple partition instances.

[0038] It should be noted that, in this embodiment, a partition is a data transfer interface used to interface with a data consumption instance, thereby providing the data consumption instance with the data to be consumed. The instance registration rules are pre-defined to limit the association between each data consumption instance and the partition of the KAFKA topic configured on the KAFKA consumer when registering a data consumption instance on the KAFKA consumer. That is, a single partition of a KAFKA topic on the KAFKA consumer can correspond to one and only one consumption thread of a data consumption instance, while a data consumption instance can correspond to multiple partitions simultaneously by creating multiple consumption threads. These multiple partitions can belong to the same KAFKA topic, or they can belong to different KAFKA topics.

[0039] Specifically, for example, a terminal device configured as a KAFKA consumer obtains the pre-defined instance registration rules and registers one or more data consumption instances in the registry center corresponding to the current KAFKA consumer according to the instance registration rules. Each data consumption instance registered by the terminal device creates a consumption thread corresponding to a single partition of a unique KAFKA topic on the current KAFKA consumer to obtain and process data to be consumed. Alternatively, the same data consumption instance simultaneously creates multiple consumption threads corresponding to all partitions (or all partitions of multiple KAFKA topics) on the current KAFKA consumer to obtain and process data to be consumed. Or, multiple data consumption instances registered by the terminal device each create one or more consumption threads corresponding to multiple partitions of multiple KAFKA topics on the current KAFKA consumer to obtain and process data to be consumed.

[0040] It should be noted that in this embodiment, the data to be consumed provided by a partition of a certain KAFKA topic configured on the KAFKA consumer terminal cannot be consumed and processed by two data consumption instances simultaneously or asynchronously, thus avoiding duplication. Therefore, it must be ensured that a partition of a certain KAFKA topic can only correspond to one data consumption instance.

[0041] Specifically, for example, in this embodiment, the terminal device configured as a KAFKA consumer uses a distributed lock to control that only one application instance can consume data from a single partition of a single KAFKA topic configured on the current KAFKA consumer. A single data consumption instance can simultaneously acquire distributed locks for multiple partitions to consume data.

[0042] Furthermore, in a feasible embodiment, when a data consumption instance registered by a terminal device configured as a KAFKA consumer becomes abnormal or should be taken offline, the terminal device will release the distributed locks of one or more partitions acquired by that data consumption instance, so that other data consumption instances can preempt and take over the data of that one or more partitions.

[0043] Step S20: Perform load balancing operation based on the weight of the data consumption instance and the partition of the KAFKA topic corresponding to the data consumption instance;

[0044] Step S30: Maintain a load-balanced state for the data consumption instance to consume data.

[0045] After the terminal device configured as a KAFKA consumer completes the registration process for the data consumption instance, it performs load balancing based on the weight of all currently online data consumption instances and the partition of a KAFKA topic on the KAFKA consumer corresponding to each data consumption instance. This ensures that all data consumption instances consume and process data from their respective connected partitions under load balancing conditions.

[0046] It should be noted that in this embodiment, the terminal device configured as a Kafka consumer has a built-in or external registry center. Each data consumption instance registered by the terminal device according to the instance registration rules needs to register with the registry center upon startup to indicate that the data consumption instance has started and can begin consuming Kafka data. The registry center then sends notifications to all other data consumption instances that are currently online and consuming data, informing them that a new instance has come online and can share the burden of processing Kafka data. At this point, the terminal device begins load balancing operations for all data consumption instances based on their respective weights and the partition of a Kafka topic on the Kafka consumer corresponding to each data consumption instance.

[0047] Furthermore, in a feasible embodiment, step S20 above may include:

[0048] Step S201: Detect all currently online data consumption instances and calculate the total weight value based on the weight of each data consumption instance.

[0049] It should be noted that, in this embodiment, the weight of the data consumption instance is used to identify the performance of the data consumption instance in connecting to the partition for data consumption processing. The higher the weight, the higher the performance of the data consumption instance in data consumption processing, that is, it can connect to more partitions for data consumption processing.

[0050] When a terminal device configured as a KAFKA consumer consumes data from a registered data consumption instance, it detects all data consumption instances that are currently online and ready to consume data. It then obtains the weight of each data consumption instance among all these instances and calculates a total weight value.

[0051] Furthermore, in a feasible embodiment, when registering a data consumption instance, the terminal device configured as a KAFKA consumer can set the weight of the data consumption instance to be the same for all data consumption instances. However, based on the differences in the machine performance of each registered data consumption terminal, the weights of each data consumption terminal can also be different. In this regard, the step of "calculating the total weight value based on the weight of each data consumption instance" can include:

[0052] Detect whether the weights of each of the data consumption instances are the same;

[0053] If so, the total weight value is calculated by multiplying the total number of all data consumption instances by the weight.

[0054] If not, the total weight value is calculated by summing the weights of all the data consumption instances.

[0055] After obtaining the weights of all data consumption instances that have started and gone online to consume data at the current moment, the terminal device configured as a Kafka consumer further checks whether these weights are the same. Therefore, if the terminal device detects that all data consumption instances have the same weight, it calculates the total weight by multiplying the number of detected data consumption instances by the same weight. Alternatively, if the terminal device detects that any weight of any data consumption instance is different from the others, it calculates the total weight by adding up the weights of each detected data consumption instance.

[0056] Specifically, for example, assuming that a terminal device configured as a KAFKA consumer sets the weight of the data consumption instance to O (O=1) when registering the data consumption instance, then when the terminal device retrieves A total number of registered data consumption instances through the registration center (the terminal device registers the relevant information of the data consumption instance in the registration center for each data consumption instance it registers), the terminal device can calculate the total weight value as Q=A*O.

[0057] Alternatively, assuming the terminal device configured as a Kafka consumer detects that the weights O of the registered data consumption instances are not the same, the terminal device obtains the weights O of the A data consumption instances and calculates the total weight value by summing them.

[0058] Step S202: Compare the total weight value, the number of partitions of the KAFKA topic corresponding to the data consumption instance, and the number of threads of the data consumption instance that process the partitions of the KAFKA topic to obtain the data comparison result;

[0059] Step S203: Perform load balancing operation based on the data comparison results.

[0060] It should be noted that, in this embodiment, the load balancing operation is to adjust the threads used by each data consumption instance to connect to the partition for data consumption processing, so that the performance of each partition connected to by all data consumption instances matches that of their data consumption processing.

[0061] After calculating the total weight of all data consumption instances that are currently online and ready for data consumption, the terminal device configured as a KAFKA consumer calculates the total weight of each data consumption instance's corresponding KAFKA topic partitions. Then, it divides this total weight by the number of partitions for each data consumption instance and rounds it down. The rounded value is then compared with the number of threads currently used by that data consumption instance to process data through the KAFKA topic partitions. If the comparison result indicates that the rounded value is greater than the number of threads, the terminal device pauses each thread used by that data consumption instance to process data through the partitions, until the rounded value equals the number of threads. Conversely, if the comparison result indicates that the rounded value is less than the number of threads, the terminal device controls the data consumption instance to create new threads to process data through the KAFKA topic partitions, until the rounded value equals the number of threads.

[0062] Specifically, for example, assume that the number of partitions of a certain KAFKA topic configured by a terminal device configured as a KAFKA consumer is B, and the number of threads currently created by a data consumption instance to process the partition data of the KAFKA topic is C. Thus, after calculating the total weight Q, the terminal device compares E, the integer value obtained by dividing B by Q, with C. If E>C, the terminal device starts to suspend the threads created by the data consumption instance for processing the partition data of the KAFKA topic, releases the lock in the distributed lock after suspending one thread each time, and decreases the locally recorded thread count C by 1, suspending threads one by one until C=E. If E<C, the terminal device instructs the data consumption instance to continue creating new threads for consuming the partition data of the KAFKA topic, and increases the locally recorded thread count C by 1, adding threads one by one until C=E.

[0063] It should be noted that, in this embodiment, if E obtained by comparing the integer value E of B / Q with C satisfies E=C, no processing is required, and the load balancing state is already achieved at this time. In addition, when the terminal device instructs the data consumption instance to continue creating new threads to continue consuming partition data of the KAFKA topic, the data consumption instance further needs to use the created new thread to acquire the lock in the distributed lock. If the lock is acquired, consumption continues; if the lock is not acquired, the new thread will be destroyed.

[0064] Further, in a feasible embodiment, the rule for triggering load balancing operation by a terminal device configured as a KAFKA consumer is: every time a new data consumption instance goes online, or an originally online data consumption instance stops, or some data consumption instances actively modify their weights to the registration center when they have performance bottlenecks. It should be understood that, based on different design requirements of practical applications, in other different feasible implementation manners, the rule for triggering the load balancing operation can of course also be other rules different from those listed herein. The data consumption method using KAFKA of the present invention does not limit the specific content of the rule for triggering the load balancing operation.

[0065] In addition, the process of performing load balancing operation on the configured data consumption instances by the terminal device configured as the KAFKA consumer is continuous. The above step S203 may include:

[0066] Step S2031: performing a preset number of load balancing operations according to the comparison result at a preset first period, and detecting whether a load balancing state is reached after each load balancing operation;

[0067] Step S2032: If the target is not reached, repeat the load balancing operation until the number of load balancing operations reaches the preset number.

[0068] Step S2033: If a load balancing state is not achieved after performing a preset number of load balancing operations, the load balancing operation is performed according to a preset second cycle, wherein the preset second cycle is equal to the preset first cycle multiplied by the preset number of operations.

[0069] Specifically, for example, in this embodiment, assuming the preset first period is 2 minutes and the preset number of times is 5, then the preset second period is 2 multiplied by 5 equals 10 minutes. Then, during the load balancing operation of the configured data consumption instance, the terminal device configured as a KAFKA consumer compares the integer value E of B / Q with C for the first time and finds E>C, thus pausing one of the threads created by that data consumption instance (decreasing C by 1). After a 2-minute interval, E is compared with C again. If E>C again, one of the threads created by that data consumption instance is paused again, and this cycle repeats 5 times.

[0070] If, after 5 iterations, comparing E and C still results in E>C, the terminal device will perform a load balancing check every 10 minutes to determine if the load is balanced. If E>C is found (due to state loss or special circumstances such as a new KAFKA topic being launched), the terminal device will continue to compare E and C every 2 minutes within the current 10-minute cycle to continue pausing one of the threads created by the data consumption instance. This process will repeat 5 times.

[0071] It should be noted that in this embodiment, when the terminal device configured as the KAFKA consumer finds E=C during any loop, it stops continuing the loop to perform load balancing operation.

[0072] This invention provides a data consumption method using Kafka. The method involves a terminal device configured as a Kafka consumer registering data consumption instances on the current Kafka consumer, following the rule that each partition of a single Kafka topic can correspond to only one data consumption instance, while a single data consumption instance can correspond to multiple partitions simultaneously. After completing the registration process, the terminal device, configured as a Kafka consumer, performs load balancing based on the weights of all currently online data consumption instances and the partition of a specific Kafka topic on the Kafka consumer corresponding to each data consumption instance. This ensures that all data consumption instances consume data from their respective connected partitions under load balancing conditions.

[0073] Compared to traditional methods that rely on big data components like Kafka for data consumption, this invention only requires Kafka consumer instances to register. Load balancing is then performed based on the weights of each consumer and the corresponding Kafka topic partitions, maintaining a balanced load for registered consumer instances. This achieves a lightweight, highly available data consumption solution using Kafka, eliminating the need for real-time stream processing frameworks like Spark Streaming or Apache Flink, thus reducing data consumption costs.

[0074] Furthermore, based on the first embodiment described above, a second embodiment of the data consumption method using KAFKA of the present invention is proposed. The main difference between this embodiment and the first embodiment is that the data consumption method using KAFKA of the present invention may further include:

[0075] Step S40: Modify the weight of the data consumption instance according to the data consumption status of the data consumption instance, and perform load balancing operation based on the weight of the data consumption instance and the partition of the KAFKA topic corresponding to the data consumption instance.

[0076] It should be noted that, in this embodiment, the data consumption status of the data consumption instance includes, but is not limited to: CPU utilization, memory consumption, data backlog, and consumption thread status.

[0077] During the process of maintaining a load-balanced state for data consumption instances to process data, the terminal devices configured as KAFKA consumers can each collect data consumption status locally and report it to the registry center. After receiving the data consumption status, the registry center instructs each data consumption instance to reset its weight. Subsequently, the terminal devices further perform load balancing operations based on the weight of each data consumption instance and the partition of a KAFKA topic on the KAFKA consumer corresponding to each data consumption instance, thereby ensuring that each data consumption instance processes data from its respective connected partition under a load-balanced state.

[0078] Specifically, for example, each data consumption instance registered by the terminal device configured with the KAFKA consumer starts a persistent thread. This persistent thread monitors the data consumption status of the data consumption instance during its consumption of partition data, including CPU usage, memory consumption, data backlog, and consumption thread status. The monitored data consumption status is then reported to the terminal device's built-in or external registry center. Upon receiving the data consumption status, the registry center distributes it to the other data consumption instances currently processing data. Each data consumption instance then resets its weights based on machine load. These reset weights are then sent back to the registry center, which distributes them to each data consumption instance for further processing of partition data. Furthermore, while each data consumption instance is using these new weights to process partition data, the terminal device further executes step S20 to ensure that all data consumption instances process data under load balancing.

[0079] It should be noted that, in this embodiment, when the terminal device configured with the KAFKA consumer instructs each data consumption instance to re-set its weight through a built-in or external registry center, if the data consumption instance monitors that the CPU utilization has been below 50% for one day, the memory consumption also shows that the memory has been occupied below 50% for one day, and the consumption thread status shows that the data accumulation of each thread is below 0 for one day, then the data consumption instance can double its weight, i.e., O = O * 2, to re-set the weight. However, if the data consumption instance monitors that the CPU utilization is greater than 90% for 5 minutes, or the memory consumption shows that the memory has been occupied above 90% for 30 minutes, or the consumption thread status shows that the thread has accumulated data for more than 5 minutes, and the data accumulation shows that the data accumulation amount is greater than 10,000, then the data consumption instance can halve its weight, i.e., O = O * 1 / 2, to re-set the weight.

[0080] In this embodiment, while the terminal device configured as a KAFKA consumer is maintaining a load-balanced state for data consumption instances to process data, each data consumption instance can collect its data consumption status locally and report it to the registry center. Upon receiving the data consumption status, the registry center instructs each data consumption instance to reset its weights. Subsequently, the terminal device further performs load balancing based on the weights of each data consumption instance and the partition of a KAFKA topic on the KAFKA consumer corresponding to each data consumption instance, thereby ensuring that each data consumption instance processes data from its respective connected partition under a load-balanced state.

[0081] Thus, the present invention uses the KAFKA data consumption method to automatically adjust the weights based on the data consumption status of the data consumption instance to ensure load-balanced data consumption processing, thereby ensuring high availability of data consumption using KAFKA.

[0082] Furthermore, based on the first and second embodiments described above, a third embodiment of the data consumption method using KAFKA of the present invention is proposed. The main difference between this embodiment and the first and second embodiments is that, after performing load balancing operations according to the weight of the data consumption instance and the partition of the KAFKA topic corresponding to the data consumption instance in step S20, the data consumption method using KAFKA of the present invention may further include:

[0083] Step S50: Detect whether all the data consumption instances have reached a load balancing state;

[0084] Step S60: If the load is not achieved, then perform the corresponding expansion operation on the partition of the KAFKA topic and / or the data consumption instance so that all data consumption instances after expansion reach the load balancing state.

[0085] It should be noted that in this embodiment, if performance bottlenecks persist for any data consumption instance after multiple load balancing operations, capacity expansion needs to be evaluated. Furthermore, if any data consumption instance exhibits abnormalities, the terminal device must immediately trigger an alarm (such as an IMS (IP Multimedia Subsystem) alarm) to notify operations and development teams to assess whether capacity expansion is necessary.

[0086] After each load balancing operation performed by the terminal device configured as a Kafka consumer, based on the weights of all currently online data consumption instances and the partition of a Kafka topic on the Kafka consumer corresponding to each data consumption instance, the terminal device checks whether all data consumption instances have reached a load-balanced state. Therefore, if the terminal device detects that all data consumption instances have not reached a load-balanced state after multiple load balancing operations, the terminal device starts to perform corresponding expansion operations on the partition of the Kafka topic and / or the data consumption instances used for data consumption processing, so that all data consumption instances can reach a load-balanced state for data consumption processing after the expansion operation.

[0087] It should be noted that, in this embodiment, when a terminal device configured as a KAFKA consumer rapidly expands the capacity of a KAFKA topic partition, it can first expand the partition of the KAFKA topic. After the number of partitions increases, the number of corresponding consumption threads for that KAFKA topic will also increase. Thus, by adding a corresponding number of data consumption instances, the overall data consumption processing capacity of the terminal device can be improved. Alternatively, one or more partitions of the KAFKA topic can be split to increase the number of connectable threads, and then a corresponding number of data consumption instances can be added to take over the data consumption processing from the increased threads.

[0088] In addition, the pressure on data consumption instances that are experiencing performance bottlenecks can be reduced by manually increasing the weight of some high-performance data consumption instances, allowing those instances to take over more threads.

[0089] In addition, for Kafka used by multiple business products, the business products with large volumes can be split, that is, the business product can be split from multiple Kafka topics originally configured into a single Kafka topic to improve overall performance.

[0090] In this embodiment, after each load balancing operation performed by the terminal device configured with a KAFKA consumer, based on the weights of all currently online data consumption instances and the partition of a KAFKA topic on the KAFKA consumer corresponding to each data consumption instance, the terminal device checks whether all data consumption instances have reached a load-balanced state. If the terminal device detects that all data consumption instances have not reached a load-balanced state after multiple load balancing operations, it begins to perform corresponding scaling operations on the partition of the KAFKA topic and / or the data consumption instances used for data consumption processing. This ensures that all data consumption instances, after scaling operations, can reach a load-balanced state for data consumption processing, further improving data consumption efficiency.

[0091] Furthermore, based on the first, second, and third embodiments described above, a fourth embodiment of the data consumption method using KAFKA of the present invention is proposed. The main difference between this embodiment and the first, second, and third embodiments is that the data consumption method using KAFKA of the present invention may further include:

[0092] Step S70: Record the consumed data identifier of the data consumption instance in the load balancing state, and modify the consumed data identifier according to the actual data consumption needs, so that the data consumption instance can continue to consume data in the load balancing state according to the modified consumed data identifier.

[0093] When a terminal device configured with a KAFKA consumer provides data for all currently online data consumption instances under load balancing, each data consumption instance records a consumed data identifier in the built-in or external registry center of the terminal device after each successful data consumption process. Then, the terminal device can modify the recorded consumed data identifier according to the actual data consumption needs, so that the data consumption instance can continue to consume data under load balancing in the next data consumption process according to the modified consumed data identifier.

[0094] It should be noted that, in this embodiment, the actual data consumption needs include, but are not limited to: losing some data, processing some data repeatedly, transaction rollback, skipping some data without processing, and data backtracking (also known as replay).

[0095] Specifically, for example, after a data consumption instance acquires the distributed lock corresponding to a partition, it starts a consumption thread to process data for that partition once all currently online data consumption instances have achieved load balancing. Each consumption reads 1MB of data (if the current partition data is less than 1MB, it reads all available partition data), and performs processing, distribution, and database write operations on the read data. After each successful data consumption, the data consumption instance records the consumed data offset in the Kafka registry in the format: "topic, partition, offset". This indicates that a partition of a Kafka topic has been successfully processed up to that offset. If the data consumption instance fails to process data, it does not record the corresponding offset in the registry. Thus, when the fault is recovered, the data consumption instance can continue data consumption from that offset, ensuring no data loss.

[0096] If a data consumption instance experiences partial failures and partial successes in processing partition data, the terminal device can choose, based on the actual data consumption requirement: lost data; to register the offset with the registry center, thus preventing the data consumption instance from processing the lost or failed data; or, based on the actual data consumption requirement: repeated processing of some data; to choose not to register the offset with the registry center, allowing the data consumption instance to repeat processing of the successfully processed data; or, based on the actual data consumption requirement: transaction rollback, to choose to perform a transaction rollback, allowing the data consumption instance to roll back the successfully processed data to ensure data consistency.

[0097] When a terminal device fails to process partition data or partially fails to do so, it will modify the offset already recorded in the registry center based on the actual data consumption needs: skipping some data (due to certain data compatibility issues, it is necessary to skip and not process it). That is, it will find the corresponding record in the registry center: "topic, partition, offset" and replace "offset" with "offset+N" to update the record in the registry center: "topic, partition, offset" to "topic, partition, offset+N". In this way, the partition data of the current KAFKA topic will skip these N data entries and not process them.

[0098] When a terminal device fails to process partition data or partially fails, it performs data backtracking based on actual data consumption needs. For certain reprocessed data, it modifies the offset of the corresponding record in the registry. Specifically, it finds the record corresponding to the data in the registry: "topic, partition, offset", and replaces "offset" with "offset-N". This updates the record in the registry from "topic, partition, offset" to "topic, partition, offset-N". In this way, the partition data of the KAFKA topic skips these N data entries and is reprocessed (the maximum length of N in the backtracking is the amount of partition data of the KAFKA topic cached on the current KAFKA consumer client).

[0099] Furthermore, in one feasible embodiment, the terminal device configured with a KAFKA consumer can obtain the offset value currently consumed by any data consumption instance recorded by the registry center. Based on the latest offset value of the data consumption instance being processed by the terminal device, the difference between the two offset values ​​can be calculated to determine the data backlog of the partition of the KAFKA topic currently being processed by the data consumption instance. In this way, the corresponding data backlog can be calculated for each KAFKA topic, and the health status of the thread currently used to process data for the current data consumption instance can be calculated based on the data backlog. Then, the corresponding data processing method can be performed according to different strategies.

[0100] Furthermore, the data backlog status of the KAFKA topic partition currently being processed by the data consumption instance can also be used to configure different production alarm levels to notify operations and maintenance personnel to intervene promptly. For example, when the data backlog is less than or equal to 5000, an IMS alarm is output to notify the data consumption instance that its data processing speed is slow; when the data backlog is less than or equal to 10000, an IMS alarm is output to notify the data consumption instance that its weight needs to be adjusted; and when the data backlog is less than or equal to 20000, an IMS alarm is output to notify the data consumption instance of an anomaly requiring timely operations and maintenance personnel intervention.

[0101] In this embodiment, when the terminal device configured as a Kafka consumer provides data consumption to the currently online data consumption instances under load balancing, each data consumption instance records a consumed data identifier in the built-in or external registry center of the terminal device after each successful data consumption process. The terminal device can then modify the recorded consumed data identifier according to actual data consumption needs, so that the data consumption instance can continue data consumption under load balancing in the next data consumption process based on the modified consumed data identifier. This achieves flexible control over the consumption offset to enable data skipping and backtracking functions, further improving the high availability of data consumption using Kafka.

[0102] Furthermore, the present invention also provides a data consumption device using Kafka, which is applied to a Kafka consumer terminal. Please refer to... Figure 3 , Figure 3 This is a functional module diagram of an embodiment of the data consumption device using KAFKA according to the present invention. Figure 3 As shown, the data consumption device using KAFKA of the present invention includes:

[0103] The instance registration module 10 is used to register data consumption instances according to the instance registration rules, wherein the instance registration rules include: a single partition of a single KAFKA topic corresponds to a consumption thread of a data consumption instance;

[0104] The load balancing module 20 is used to perform load balancing operations based on the weight of the data consumption instance and the partition of the KAFKA topic corresponding to the data consumption instance.

[0105] The data consumption module 30 is used to maintain a load-balanced state for the data consumption instance to consume data.

[0106] Furthermore, the load balancing module 20 includes:

[0107] The detection unit is used to detect all the data consumption instances currently online, and calculate the total weight value according to the weight of each data consumption instance.

[0108] The comparison unit is used to compare the total weight value, the number of partitions of the KAFKA topic corresponding to the data consumption instance, and the number of threads of the data consumption instance that process the partitions of the KAFKA topic to obtain the data comparison result.

[0109] The load balancing unit is used to perform load balancing operations based on the data comparison results.

[0110] Furthermore, the load balancing unit includes:

[0111] The first load balancing subunit is used to perform a preset number of load balancing operations based on the comparison results according to a preset first cycle, and to detect whether a load balancing state has been reached after each load balancing operation.

[0112] The second load balancing subunit is used to repeatedly perform the load balancing operation until the number of times the load balancing operation is performed reaches the preset number.

[0113] The third load balancing subunit is used to perform the load balancing operation according to a preset second period if the load balancing state is not achieved after performing a preset number of load balancing operations, wherein the preset second period is equal to the preset first period multiplied by the preset number of times.

[0114] Furthermore, the detection unit is also configured to detect whether the weights of each of the data consumption instances are the same; and if so, to calculate the total weight value by multiplying the total number of all the data consumption instances by the weights; if not, to calculate the total weight value by summing the weights of all the data consumption instances.

[0115] Furthermore, the load balancing module 20 of the KAFKA data consumption device in this invention is also used to modify the weight of the data consumption instance according to the data consumption status of the data consumption instance, and to perform a load balancing operation based on the weight of the data consumption instance and the partition of the KAFKA topic corresponding to the data consumption instance.

[0116] Furthermore, the present invention uses a data consumption device based on KAFKA, which further includes:

[0117] The expansion module is used to detect whether all the data consumption instances have reached a load balancing state; and if not, to perform corresponding expansion operations on the partition of the KAFKA topic and / or the data consumption instances so that all data consumption instances after expansion reach the load balancing state.

[0118] Furthermore, the present invention uses a data consumption device based on KAFKA, which further includes:

[0119] The recording module is used to record the consumed data identifiers of the data consumption instance in the load balancing state, and to modify the consumed data identifiers according to the actual data consumption needs, so that the data consumption instance can continue to consume data in the load balancing state according to the modified consumed data identifiers.

[0120] The functions of each module in the data consumption device using KAFKA described above correspond to the steps in the data consumption method embodiment using KAFKA described above, and their functions and implementation processes will not be described in detail here.

[0121] The present invention also provides a computer storage medium storing a data consumption program using KAFKA, wherein when the data consumption program using KAFKA is executed by a processor, it implements the steps of the data consumption method using KAFKA as described in any of the above embodiments.

[0122] The specific embodiments of the computer storage medium of the present invention are basically the same as the embodiments of the data consumption method using KAFKA described above, and will not be repeated here.

[0123] The present invention also provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the data consumption method using KAFKA as described in any of the above embodiments.

[0124] The specific embodiments of the computer program product of the present invention are basically the same as the embodiments of the data consumption method using KAFKA described above, and will not be repeated here.

[0125] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A data consumption method using Kafka, characterized in that, The data consumption method using KAFKA is applied to the KAFKA consumer end, and the data consumption method using KAFKA includes the following steps: Register data consumption instances according to the instance registration rules, wherein the instance registration rules include: a single partition of a single KAFKA topic corresponds to a consumption thread of a data consumption instance; Load balancing is performed based on the weight of the data consumption instance and the partition of the KAFKA topic corresponding to the data consumption instance. Maintain a load-balanced state for the data consumption instances to consume data; The load balancing operation, based on the weight of the data consumption instance and the partition of the KAFKA topic corresponding to the data consumption instance, includes: Detect all currently online data consumption instances and calculate the total weight value based on the weight of each data consumption instance. For each data consumption instance: calculate the ratio between the number of partitions of the KAFKA topic corresponding to the data consumption instance and the total weight value, and round it down; compare the rounded value with the number of threads of the data consumption instance processing the partitions of the KAFKA topic; when the rounded value is less than the number of threads, pause the threads of the data consumption instance processing the partitions of the KAFKA topic one by one until the number of threads is equal to the rounded value; when the rounded value is greater than the number of threads, control the data consumption instance to create new threads to connect to the partitions of the KAFKA topic until the number of threads is equal to the rounded value.

2. The data consumption method using Kafka as described in claim 1, characterized in that, The steps for performing load balancing based on the data comparison results, according to the weight of the data consumption instance and the partition of the KAFKA topic corresponding to the data consumption instance, include: According to a preset first cycle, a preset number of load balancing operations are performed based on the comparison result between the rounded value and the number of threads, and after each load balancing operation, it is checked whether a load balancing state has been reached. If the target is not reached, the load balancing operation is repeated until the number of load balancing operations reaches the preset number. If a load balancing state is not achieved after a preset number of load balancing operations, the load balancing operation is performed according to a preset second cycle, wherein the preset second cycle is equal to the preset first cycle multiplied by the preset number of operations.

3. The data consumption method using Kafka as described in claim 1, characterized in that, The step of calculating the total weight value based on the respective weight of each data consumption instance includes: Detect whether the weights of each of the data consumption instances are the same; If so, the total weight value is calculated by multiplying the total number of all data consumption instances by the weight. If not, the total weight value is calculated by summing the weights of all the data consumption instances.

4. The data consumption method using Kafka as described in claim 1, characterized in that, The method further includes: The steps involve modifying the weight of the data consumption instance based on its data consumption status, and performing load balancing based on the weight of the data consumption instance and the partition of the KAFKA topic corresponding to the data consumption instance.

5. The data consumption method using KAFKA as described in any one of claims 1 to 4, characterized in that, After the step of performing load balancing based on the weight of the data consumption instance and the partition of the KAFKA topic corresponding to the data consumption instance, the method further includes: Check whether all the data consumption instances have reached a load-balanced state; If the load is not achieved, then the corresponding expansion operation is performed on the partition of the KAFKA topic and / or the data consumption instance so that all data consumption instances after expansion reach the load balancing state.

6. The data consumption method using Kafka as described in claim 5, characterized in that, The method further includes: Record the consumed data identifiers of the data consumption instance in the load balancing state, and modify the consumed data identifiers according to the actual data consumption needs, so that the data consumption instance can continue to consume data in the load balancing state according to the modified consumed data identifiers.

7. A data consumption device using Kafka, characterized in that, The data consumption device using KAFKA is applied to a KAFKA consumer terminal, and the data consumption device using KAFKA includes: The instance registration module is used to register data consumption instances according to the instance registration rules, wherein the instance registration rules include: a single partition of a single KAFKA topic corresponds to a consumption thread of a data consumption instance; The load balancing module is used to perform load balancing operations based on the weight of the data consumption instance and the partition of the KAFKA topic corresponding to the data consumption instance. The data consumption module is used to maintain a load-balanced state for the data consumption instances to consume data. The load balancing operation, based on the weight of the data consumption instance and the partition of the KAFKA topic corresponding to the data consumption instance, includes: Detect all currently online data consumption instances and calculate the total weight value based on the weight of each data consumption instance. For each data consumption instance: calculate the ratio between the number of partitions of the KAFKA topic corresponding to the data consumption instance and the total weight value, and round it down; compare the rounded value with the number of threads of the data consumption instance processing the partitions of the KAFKA topic; when the rounded value is less than the number of threads, pause the threads of the data consumption instance processing the partitions of the KAFKA topic one by one until the number of threads is equal to the rounded value; when the rounded value is greater than the number of threads, control the data consumption instance to create new threads to connect to the partitions of the KAFKA topic until the number of threads is equal to the rounded value.

8. A terminal device, characterized in that, The terminal device includes: a memory, a processor, and a data consumption program using KAFKA stored in the memory and executable on the processor, wherein when the data consumption program using KAFKA is executed by the processor, it implements the steps of the data consumption method using KAFKA as described in any one of claims 1 to 6.

9. A computer storage medium, characterized in that, The computer storage medium stores a data consumption program using KAFKA, which, when executed by a processor, implements the steps of the data consumption method using KAFKA as described in any one of claims 1 to 6.

Images