Method and device for adaptively adjusting concurrency of consumer threads

Abstract

The invention provides a method and a device for adaptively adjusting concurrency of consumer threads. The method comprises the following steps: acquiring consumer offset and log end offset of each partition of a consumer, calculating message accumulation amount of each partition, and summarizing to obtain current message accumulation amount of the consumer; based on the current message accumulation amount, a target thread count is calculated in combination with a preset rule or a dynamic index, and the dynamic index at least comprises a change rate of the accumulation amount within a preset time and a single-thread average message processing throughput; and under the condition of determining that the preset adjustment condition is met, adjusting the core thread count and the maximum thread count of the thread pool of the consumer to the target thread count through the interface of the Kafka consumption task executor. Based on the method, the processing capacity of the consumer can be matched with the real-time message load, the resource utilization rate is improved while the processing delay is reduced, and the stability of the system adjustment process is guaranteed.

Description

Technical Field

[0001] This application relates to the field of distributed message middleware technology, and in particular to a method and apparatus for adaptively adjusting the concurrency of consumer threads. Background Technology

[0002] In modern backend system architectures, Apache Kafka, as a distributed message middleware, is widely used for scenarios such as asynchronous decoupling between systems, traffic shaping, and log aggregation. Typically, Kafka consumer applications use a fixed-size thread pool to process subscribed messages.

[0003] However, this fixed thread pool configuration has inherent drawbacks. On the one hand, when the upstream message production rate surges due to business peaks (such as promotional activities or batch data imports), the fixed number of consumer threads may be insufficient to quickly process the influx of messages, leading to a large backlog of messages in partitions (i.e., lag), which in turn significantly increases data processing latency and affects the real-time performance of the business. On the other hand, during off-peak periods, when message volume is low, many pre-configured threads may be idle, but they still occupy valuable CPU and memory resources, resulting in resource waste and increased operating costs. Furthermore, although external container orchestration tools such as Kubernetes HPA can be used for instance-level scaling, these solutions are usually based on macro-level metrics such as CPU or memory, resulting in significant response latency and an inability to accurately and quickly respond to the real load pressure driven by changes in message backlog.

[0004] Therefore, existing technologies fail to provide an effective mechanism for quickly and safely adjusting the number of consumer threads dynamically based on real-time message backlog within the application, making it difficult to balance system processing performance and resource utilization when dealing with traffic fluctuations. Summary of the Invention

[0005] The purpose of this application is to provide a method and apparatus for adaptively adjusting the concurrency of consumer threads, so as to solve the problem that the number of consumer threads is fixed in the prior art and cannot be dynamically adjusted according to the real-time message backlog, resulting in resource waste or processing delay.

[0006] To achieve the above objectives, this application provides a method for adaptively adjusting the concurrency of consumer threads, which is applied to a Kafka-based message processing system, and includes the following steps:

[0007] Obtain the consumer offset and log end offset of each partition of the consumer, calculate the message backlog of each partition, and summarize to obtain the current message backlog of the consumer. Based on the current message backlog, the target number of threads is calculated by combining preset rules or dynamic indicators. The dynamic indicators include at least the rate of change of the backlog within a preset time and the average message processing throughput per thread. If the preset adjustment conditions are met, the core thread count and maximum thread count of the consumer's thread pool are adjusted to the target thread count through the interface of the Kafka consumer task executor. The adjustment conditions include: if it is necessary to increase the core thread count, the JVM heap memory utilization rate must be less than a first threshold and the CPU utilization rate must be less than a second threshold.

[0008] Optionally, the target number of threads is calculated based on preset rules, including: Compare the current message backlog with the preset low watermark threshold and high watermark threshold. If the current message backlog is greater than the high watermark threshold, the target number of threads is the smaller of a preset multiple of the current number of threads and the maximum allowed number of threads; if the current message backlog is less than the low watermark threshold and the current number of threads is higher than the minimum allowed number of threads, the target number of threads is the larger of a preset ratio of the current number of threads and the minimum allowed number of threads.

[0009] Optionally, the formula for calculating the target number of threads, combined with dynamic metrics, is as follows:

[0010] in, For the target number of threads, This represents the current message backlog. Preset time Changes in the amount of internal message backlog Clear the preset target time window. This represents the average message processing throughput for a single thread.

[0011] Optionally, the average message processing throughput per thread is calculated by the total number of messages processed within a statistical sliding time window and the total processing time, and the duration of the sliding time window is a configurable parameter.

[0012] Optionally, the adjustment conditions further include: The difference between the target number of threads and the current number of threads reaches a preset threshold; The preset cooldown time has been reached since the last adjustment of the core thread count.

[0013] Optionally, dynamically adjusting the core thread count and maximum thread count of the consumer's thread pool to the target thread count via the Kafka consumer task executor interface includes: If the target number of threads is greater than the current number of threads, call the setCorePoolSize and setMaximumPoolSize methods of the thread pool to adjust the core number of threads and the maximum number of threads to the target number of threads. If the target number of threads is less than the current number of threads, first call the thread pool's allowCoreThreadTimeOut method to allow core threads to time out and exit, then set the idle thread survival time using the setKeepAliveTime method, and finally call the setCorePoolSize method to reduce the number of core threads.

[0014] To achieve the above objectives, this application also provides an apparatus for adaptively adjusting the concurrency of consumer threads, applied to a Kafka-based message processing system, the apparatus comprising: The message backlog status awareness module is used to obtain the consumer offset and log end offset of each partition of the consumer, calculate the message backlog of each partition, and summarize to obtain the current message backlog of the consumer. The target thread count calculation module is used to calculate the target thread count based on the current message backlog and in combination with preset rules or dynamic indicators. The dynamic indicators include at least the rate of change of the backlog within a preset time and the average message processing throughput per thread. The thread pool dynamic refactoring module is used to adjust the core thread count and maximum thread count of the consumer's thread pool to the target thread count through the interface of the Kafka consumer task executor, provided that preset adjustment conditions are met. The adjustment conditions include: if it is necessary to increase the core thread count, the JVM heap memory utilization rate must be less than a first threshold and the CPU utilization rate must be less than a second threshold.

[0015] Optionally, the device can be integrated with a Java microservice system via a Starter or annotation to be deployed on a Kafka message processing system.

[0016] Compared with the prior art, this application has the following beneficial effects: 1. By acquiring the current message backlog of consumers in real time and calculating the target number of threads based on this in conjunction with preset rules or dynamic metrics, dynamic and precise adjustment of consumer thread concurrency is achieved. This solution enables the consumer's processing capacity to match the real-time message load, rapidly increasing processing capacity to digest the backlog when message volume surges and proactively releasing excess resources when message volume decreases, avoiding the rigidity problem of traditional fixed thread pools when dealing with traffic fluctuations.

[0017] 2. By setting adjustment conditions, such as checking JVM heap memory usage and CPU usage before increasing the number of core threads, it is ensured that the expansion operation is carried out within the safe boundary of sufficient system resources, effectively avoiding stability risks such as memory overflow (OOM) or performance degradation caused by excessive CPU contention due to excessive thread expansion. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Furthermore, these drawings and textual descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments.

[0019] Figure 1 A flowchart illustrating a method for adaptively adjusting the concurrency of consumer threads provided in an embodiment of this application; Figure 2 This is a schematic diagram of a device for adaptively adjusting the concurrency of consumer threads, provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the embodiments of this application. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0021] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0022] (1) Kafka: refers to an open-source distributed event streaming platform that is widely used to build real-time, high-throughput data pipelines and stream processing applications. It allows multiple producers to publish event streams to it and allows multiple consumers to subscribe to these event streams. In the context of this application, it constitutes the infrastructure environment for messaging.

[0023] (2) Producer: refers to the client application that generates and publishes events (messages) in the Kafka architecture.

[0024] (3) Consumer: refers to a client application in the Kafka architecture that subscribes to one or more topics and processes the published events (messages). The technical solution in this application mainly works on the consumer side, improving processing performance by optimizing its internal resource configuration.

[0025] (4) Message Lag: Also known as consumption latency, it refers to the difference between the offset of the latest message written by the producer (log end offset) and the offset of the latest message processed by the consumer (consumer offset) in a specific consumer partition at a given time. It is a key indicator for measuring whether the consumer's processing capacity is keeping up with the producer's production speed. The current message lag of a consumer is the sum of the message lag of all partitions allocated to the consumer.

[0026] (5) Consumer Offset: This refers to the position of the last message that a consumer has successfully consumed and acknowledged in a certain partition. Kafka records this offset to ensure ordered message consumption and fault recovery.

[0027] (6) Log End Offset (LEO): This refers to the position of the latest message written by the producer in a certain partition. This value represents the latest data point available for consumption in the current partition.

[0028] (7) Thread Pool: In computer programs, a mechanism for managing and reusing threads can be used. By pre-creating a set of threads, the performance overhead caused by frequently creating and destroying threads can be avoided. The core of this application lies in dynamically adjusting the size of the thread pool used to perform message consumption tasks.

[0029] (8) JVM (Java Virtual Machine): The Java Virtual Machine is an abstract computer that runs all Java applications. The JVM heap memory utilization and CPU utilization mentioned in this application are key indicators for measuring the resource load of the application's runtime environment and are used as protective constraints for thread expansion.

[0030] like Figure 1 As shown in the illustration, this application provides a method for adaptively adjusting the concurrency of consumer threads. This method is applied to a Kafka-based message processing system and aims to solve the problems of high processing latency during traffic surges and resource waste during traffic slumps caused by a fixed number of consumer threads. The method establishes a closed-loop monitoring, decision-making, and execution system within the consumer application to respond in real-time to changes in message load and dynamically adjust the number of processing threads, thereby optimizing system resource utilization while ensuring processing timeliness.

[0031] like Figure 1 As shown, this method includes several core steps: S101: Obtain the consumer offset and log end offset of each partition of the consumer, calculate the message backlog of each partition, and summarize to obtain the current message backlog of the consumer.

[0032] This step corresponds to the monitoring phase. In this phase, the system needs to perceive the current message processing pressure in real time, which is the basis for all subsequent decisions. Specifically, this step obtains the consumer offset and log end offset for each partition of the consumer, calculates the message backlog for each partition based on these two values, and then summarizes the message backlog for all partitions to obtain the total message backlog currently faced by the consumer, i.e., the current message backlog L. This process provides the system with a quantitative, real-time load metric.

[0033] More specifically, obtaining the consumer offset can typically be done by calling the client interface provided by Kafka. For example, in a Java environment, the `AdminClient.listConsumerGroupOffsets()` method can be used to get the consumption progress of each partition in a specified consumer group. Similarly, obtaining the log end offset can be done using the `AdminClient.listOffsets()` method to query the end position of the log for each partition. Subtracting these two offsets, i.e., `logEndOffset - consumerOffset`, gives the message backlog for a single partition. Summing the backlogs of all partitions handled by the consumer yields the global current message backlog.

[0034] In some embodiments, a sliding window (such as averaging the last 3 samples) can be used to calculate the smoothed current message backlog to avoid instantaneous jitter.

[0035] Specifically, the system can maintain a historical queue L_hist[] containing the message backlog samples from the past M times (e.g., M=3), and use a weighted moving average or a simple arithmetic average to calculate a smoothed message backlog, which is then used as the current message backlog for subsequent decisions.

[0036] After obtaining the current message backlog, the system proceeds to the next step.

[0037] S102: Calculate the target number of threads based on the current message backlog and in conjunction with preset rules or dynamic metrics.

[0038] This step corresponds to the decision-making phase. In this phase, based on the current message backlog L and a set of preset rules or dynamically changing performance metrics, an ideal target number of threads is calculated. This target number of threads represents the system's reasonable concurrent processing capacity under the current load. The design of dynamic metrics is a key aspect of this solution; it considers factors such as the rate of change of the backlog over a preset time period and the average message processing throughput of a single thread. These dynamic metrics make the decision-making process more forward-looking and adaptable.

[0039] After calculating the target number of threads, the adjustment is not executed immediately. The system first needs to determine whether the preset adjustment conditions are met. Figure 1 Step S103. The adjustment conditions include at least the following: if it is necessary to increase the number of core threads, the JVM heap memory utilization must be less than the first threshold and the CPU utilization must be less than the second threshold.

[0040] When the decision is to increase the number of threads (the target number of threads is greater than the current number of threads, i.e., expansion is required), the system will check the JVM's heap memory usage and CPU usage to determine whether system resources are sufficient.

[0041] This step rigorously checks the resource status of the current runtime environment. Specifically, the adjustment conditions include a mandatory resource constraint: if the number of core threads needs to be increased, the Java Virtual Machine (JVM) heap memory utilization must be less than a preset first threshold (e.g., 80%), and the host CPU utilization must be less than a preset second threshold (e.g., 70%). This mechanism is a crucial safeguard against uncontrolled growth in the number of threads leading to system resource exhaustion (such as OutOfMemoryError or excessive CPU contention). If resources are sufficient, the process continues; if resources are scarce, the expansion operation will be suppressed, maintaining the current number of threads, and an alarm may be triggered.

[0042] Once all adjustment conditions are met, the system enters the execution phase, i.e., step S104. This method adjusts the core thread count (corePoolSize) and maximum thread count (maximumPoolSize) of the consumer's thread pool to the target thread count through the interface of Kafka's consumer task executor (ThreadPoolExecutor). In other words, it makes the core thread count and maximum thread count equal to the previously calculated target thread count. In the Java technology stack, this typically means interacting with a ThreadPoolExecutor instance and calling its corresponding methods to change the thread pool configuration in real time. This process is smooth and does not interrupt threads processing messages, thus ensuring business continuity.

[0043] Through the closed-loop process described above—obtaining the message backlog, calculating the target number of threads, and determining and executing adjustments—this method enables the consumer's processing capacity to automatically and quickly match the message production rate. When upstream traffic surges, the message backlog rapidly increases. Upon detection, the system automatically increases the number of threads, provided resources allow, to enhance concurrent processing capabilities, thereby quickly "shaving off peaks" and effectively reducing processing latency. When traffic drops to a low point, the message backlog remains at a low level or is cleared. The system then correspondingly reduces the number of threads, releasing unnecessary CPU and memory resources to save costs. The entire process is completed within the application, with a response time in the second range, far faster than the minute-level scaling based on external monitoring (such as Kubernetes HPA), and it does not depend on a specific deployment environment.

[0044] In a preferred embodiment, the logic for calculating the target number of threads based on preset rules in step S102 is refined. This approach provides a simple and effective strategy suitable for scenarios where load variation patterns are relatively predictable. Specifically, the system pre-sets two key water level thresholds: a low water level threshold (e.g., LOW_THRESHOLD = 10,000) and a high water level threshold (e.g., HIGH_THRESHOLD = 100,000). Simultaneously, a minimum allowed number of threads (minThreads) and a maximum allowed number of threads (maxThreads) are defined to limit the upper and lower bounds of the thread pool size.

[0045] The adjustment logic for this rule is as follows: The system periodically compares the current message backlog with two pre-set thresholds (low watermark threshold and high watermark threshold). If the current message backlog exceeds the preset high watermark threshold, it indicates that the current processing capacity is severely insufficient and immediate expansion is required. In this case, the target number of threads will be calculated as the smaller of a preset multiple (e.g., 2 times) of the current number of threads and the maximum allowed number of threads, maxThreads. The purpose of taking the smaller value is to prevent the number of threads from growing indefinitely and to ensure that it does not exceed the system's set upper limit. For example, if there are currently 8 threads and the maximum allowed is 32, the target number of threads will be set to min(8×2, 32) = 16.

[0046] On the other hand, if the system detects that the current message backlog is less than the preset low-water mark threshold, it indicates that the current processing capacity is excessive and resources are being wasted, and scaling down can be implemented. In this case, and only when the current number of threads is higher than the minimum allowed number of threads (minThreads), the target number of threads will be calculated as the larger of a preset ratio of the current number of threads (e.g., 0.5 times, i.e., divided by 2) and the minimum allowed number of threads (minThreads). The purpose of taking the larger value is to ensure that a basic processing capacity is maintained even when there are no messages, and to prevent the number of threads from being reduced to zero. For example, if there are currently 16 threads and the minimum allowed number is 4, then the target number of threads will be set to max(16 / 2, 4) = 8.

[0047] If the system detects that the current message backlog is between the low watermark threshold and the high watermark threshold, it indicates that the number of threads is reasonable and no adjustment is needed.

[0048] This dual-threshold-based rule achieves a tiered, rapidly responsive scaling mechanism. Its logic is simple, computationally inexpensive, and it can quickly respond to significant load changes. By setting a "buffer" between high and low thresholds, frequent adjustments caused by minor fluctuations near the critical point can be effectively avoided. Furthermore, the settings for minThreads and maxThreads provide robust boundary guarantees for stable system operation.

[0049] In another, more refined and forward-looking preferred embodiment, the calculation of the target thread count no longer relies solely on the absolute value of the current message backlog, but rather incorporates a series of dynamic metrics to form a predictive model. These dynamic metrics include the current message backlog, the rate of change of the message backlog over a preset time period, and the average message processing throughput per thread. The specific formula for calculating the target thread count is as follows:

[0050] in, The calculated target number of threads; This represents the current message backlog. To be in the preset time The change in the amount of internal message backlog, therefore This represents the rate of change of the message backlog. Set a preset target time window for clearing messages, which represents how long we expect to process all backlogged messages (e.g., 30 seconds). The average message processing throughput for a single thread (in messages per second). The numerator in the formula... This can be understood as "predicting backlog," meaning that if we maintain our current processing capacity, what will the backlog be in the future? What will the total number of messages pending processing be after the specified time? (Denominator) This means that a thread is in The total number of messages that can be processed within a given time period. The physical meaning of the entire formula is: in order to clear the target time window... How many threads are theoretically needed to process the predicted backlog internally? The final rounding up sign (...) This ensures that the number of threads is the smallest integer sufficient to complete the task.

[0051] This method, based on dynamic indicators and predictive models, achieves more intelligent and precise resource scheduling. It not only considers the current "existing" backlog (…). It also considered the "incremental" trend ( This allows the system to predict upcoming load peaks or troughs. For example, even if there is a backlog of messages... Not big, but if If the message volume is very high, the system will expand its capacity in advance to cope with the upcoming surge. Conversely, even if the message backlog is low... It's still quite high, but if If the value is negative and very large in absolute terms (indicating that the consumption rate far exceeds the production rate), the system can also predict that the backlog will be cleared soon, thus preparing to reduce capacity in advance. This makes the adjustment of the number of threads smoother and avoids the "overshoot" and "lag" phenomena that may occur when based on static thresholds.

[0052] Furthermore, to make the above dynamic calculation model more accurate, the average message processing throughput of a single thread was further analyzed. The calculation method has been optimized. In a preferred embodiment, this average message processing throughput... It is no longer a fixed preset value, but is dynamically calculated through real-time statistics. Specifically, the system maintains a sliding time window (e.g., the most recent 60 seconds). Within this window, the system continuously accumulates the total number of messages processed by all consumer threads (totalProcessed) and the total processing time they spend (totalProcessingTimeMs).

[0053] Average message processing throughput per thread The calculation is as follows: divide the total number of processed messages recorded within the sliding window by the total processing time in seconds, and then divide by the number of threads in the current consumer thread pool. For example, =(totalProcessed / (totalProcessingTimeMs / 1000.0)) / currentThreadCount. The duration of the sliding time window can be designed as a configurable parameter, allowing developers to adjust it according to business characteristics. For example, for businesses with large fluctuations in message processing time, the window can be appropriately lengthened to obtain a smoother and more representative throughput estimate.

[0054] This dynamic calculation of average message processing throughput The effect is that it endows the entire adaptive adjustment system with "learning" capabilities. A consumer's actual processing capacity is affected by various factors, such as the complexity of the message itself, the response time of downstream services, and network conditions. Through real-time statistics, the system can accurately grasp the true single-threaded processing efficiency under the current environment, rather than relying on a potentially outdated static assumption. When business logic changes or the performance of external dependencies changes, the calculated efficiency... It updates automatically, thus ensuring the target number of threads. The calculations are always based on the latest performance data, further improving the accuracy and adaptability of the adjustments.

[0055] To further enhance system stability and prevent unnecessary resource jitter, in another preferred embodiment, additional constraints are added to the triggering conditions for performing adjustments. In addition to the aforementioned resource protection checks, two constraints are introduced into the adjustment conditions: a change threshold and a cooldown time. This means that even if a new target number of threads is calculated, adjustments will not be performed immediately under all circumstances.

[0056] The first constraint is that the difference between the target number of threads and the current number of threads must reach a preset threshold. For example, this preset threshold can be set to 2, meaning that adjustment is only triggered when the difference between the target number of threads and the current number of threads is greater than or equal to 2. The purpose of this design is to filter out meaningless fine-tuning caused by minor fluctuations or transient noise in the computational model (e.g., adjusting from 10 threads to 11). By ignoring such minor changes, the number of changes to the thread pool configuration can be effectively reduced, minimizing disturbances to the system.

[0057] The second constraint is that a preset cooldown time (e.g., 60 seconds) must elapse since the last successful adjustment of the core thread count. This mechanism, also known as a "cooldown period" or "quiet period," aims to prevent the system from rapidly and repeatedly scaling up and down near its load threshold, a phenomenon known as "jitter" or "bumping." For example, without a cooldown, the system might decide to scale up from 8 threads to 10 threads in the first second, and then scale back down to 8 threads in the second second due to a slight decrease in backlog, repeating this process repeatedly, causing significant resource overhead and instability. With the introduction of a cooldown, after an adjustment is completed, the system enters a cooldown period. During this period, even if a new target thread count is calculated, the adjustment will not be executed, ensuring that the effects of each adjustment have sufficient time to manifest and stabilize.

[0058] The technical effect of these two additional conditions is that they together constitute a "debouncing" mechanism, greatly improving the stability and robustness of the entire adaptive adjustment mechanism. The change threshold focuses on the "amplitude" of the adjustment, avoiding ineffective fine-tuning; the cooldown time focuses on the "frequency" of the adjustment, avoiding destructive high-frequency oscillations. Combined, these two ensure that every adjustment to the thread pool is prudent, effective, and meaningful, thus maximizing system stability while pursuing rapid response.

[0059] At the specific thread pool adjustment execution level, a preferred embodiment of this application details how to achieve smooth and lossless thread count changes through the existing interface of ThreadPoolExecutor in Java. This process employs different strategies depending on the adjustment direction (expansion or contraction) to ensure the continuity of task processing and effective resource management.

[0060] Specifically, the core thread count and maximum thread count of the consumer's thread pool are dynamically adjusted to the target thread count through the interface of Kafka's consumer task executor, including: When the number of threads needs to be increased, i.e., the target number of threads is greater than the current number of threads (requiring expansion), the process is relatively straightforward. The system will call the `setCorePoolSize()` and `setMaximumPoolSize()` methods of the thread pool instance to synchronize the core thread count and the maximum thread count to the calculated target number. For example, through `executor.setCorePoolSize()`... Adjust the core thread count to via executor.setMaximumPoolSize( Adjust the maximum number of threads to When ThreadPoolExecutor receives a new, larger core thread count, it automatically creates new threads to execute tasks until the target thread count is reached. This method can quickly increase concurrent processing capacity to cope with growing loads.

[0061] When the number of threads needs to be reduced—that is, when the target number of threads is less than the current number (requiring scaling down)—a smooth strategy is needed to avoid abruptly terminating working threads and causing message loss or processing interruption. First, the system calls the thread pool's `allowCoreThreadTimeOut(true)` method. This call allows core threads to be recycled like non-core threads when idle, meaning core threads are allowed to time out. Next, a reasonable idle thread survival time (e.g., 30 seconds) is set using the `setKeepAliveTime()` method. Finally, the `setCorePoolSize()` method is called to set the core thread count to the smaller target number. In this way, those "old" core threads that currently exceed the new core thread count will be automatically terminated and recycled by the thread pool if they remain idle for longer than the set survival time after completing their tasks, thus smoothly reducing the total number of threads to the target number.

[0062] This fine-grained execution strategy, which distinguishes between scaling up and scaling down, achieves the technical advantage of dynamic, safe, and lossless adjustment of the thread pool size. The directness of scaling up ensures a rapid response to load increases. Meanwhile, the "graceful exit" mechanism for scaling down, by allowing core threads to time out, ensures that no message processing tasks are forcibly interrupted due to thread pool reduction, guaranteeing the integrity and consistency of data processing—a crucial feature in production environments.

[0063] This application also provides a device for adaptively adjusting the concurrency of consumer threads, which serves as the physical or logical carrier of the aforementioned method. This device is also applied to the Kafka message processing system, and its internal structure closely corresponds to the method flow. Figure 2 As shown, the device includes: a message backlog status awareness module 21, a target thread count calculation module 22, and a thread pool dynamic reconfiguration module 23.

[0064] The message backlog awareness module 21 corresponds to step S101 of the aforementioned method. It is configured to periodically interact with the Kafka cluster to obtain the consumer offset and log end offset of each partition subscribed by the consumer. After obtaining this basic data, the module performs calculations to determine the message backlog for each partition and finally aggregates them to output a "current message backlog" value representing the current overall load.

[0065] The target thread count calculation module 22 receives the current message backlog amount from the message backlog status awareness module. This module internally implements decision-making logic, and its function corresponds to step S102 of the aforementioned method. It is configured to calculate an ideal target thread count based on the input backlog amount, combined with preset rules (such as high and low water level thresholds) or a set of dynamic indicators. This module is the "brain" of the entire device, responsible for formulating concurrency strategies.

[0066] The thread pool dynamic refactoring module 23 is the final executor of the strategy, and its function corresponds to step S104 of the aforementioned method. This module receives the target thread count output from the target thread count calculation module. It is configured to interact with the consumer task executor (usually a thread pool) inside the consumer, provided that all adjustment conditions (including resource protection conditions) are met. Specifically, it calls the interface provided by the executor to dynamically adjust the core thread count and maximum thread count of the thread pool to the target thread count. A key built-in logic is that before performing the operation to increase the core thread count, it must first check whether the JVM heap memory usage is less than a first threshold and whether the CPU usage is less than a second threshold, as a "safety gate" for performing the expansion operation.

[0067] These three modules work together to form a complete adaptive control loop. The message backlog awareness module acts as the "eyes," continuously monitoring the system load; the target thread count calculation module acts as the "brain," making decisions based on the observations; and the thread pool dynamic reconfiguration module acts as the "hands and feet," responsible for executing decisions and adjusting the actual processing capacity. The entire system runs within the consumer application process, enabling real-time, automatic, and intelligent control of Kafka consumption capacity.

[0068] In a specific implementation example, to facilitate integration into existing microservice systems, the device can be implemented as a modular component. For instance, it can be encapsulated as a component that can be enabled via dependency injection or declarative configuration (such as annotations). More specifically, it can be integrated with Java microservice systems via Starters or annotations for deployment within a Kafka message processing system. When enabled, the component automatically associates with Kafka consumer instances within the application and manages their thread pools, achieving low-intrusion integration with business logic. This modular design allows the technology to be quickly and widely applied to various microservice architectures, significantly improving the practicality and versatility of the solution.

[0069] To more specifically demonstrate a complete and optimized implementation of this application, an embodiment incorporating several preferred features will be described below. This embodiment represents a preferred configuration. In this embodiment, a Kafka consumer running in a Java environment has its thread concurrency adaptively adjusted by a built-in, highly intelligent controller.

[0070] First, the consumer integrates a monitoring agent that uses Kafka's AdminClient to perform a state snapshot at a configurable period (e.g., every 10 seconds). In each snapshot, it retrieves the current consumer offset and the latest log end offset for all relevant partitions within the consumer group. The global current message backlog is obtained by calculating the difference between these two and summing them. Simultaneously, to smooth out instantaneous fluctuations, it maintains a historical queue L_hist[] containing message backlog samples from the past M times (e.g., M=3), and calculates a smoothed message backlog using a weighted moving average or a simple arithmetic average, serving as the primary basis for subsequent decisions. Furthermore, it calculates the rate of change of the message backlog based on L_hist[]. .

[0071] Next, the target number of threads was calculated using a predictive model based on dynamic metrics. In this embodiment, the average message processing throughput per thread was... It's dynamically learned. After processing each batch of messages, the consumer records the number of messages and the processing time for that batch. A separate statistics module maintains a sliding time window (e.g., the most recent 1 minute) and calculates the current effective single-threaded average message processing throughput in real time based on the data within the window. Then, the controller will smooth out the current message backlog and the rate of change of the message backlog. And the dynamically calculated average message processing throughput T is substituted into the core formula for calculation:

[0072] Among them, the target clearing time window It is set to 30 seconds, which is a policy parameter representing the system's tolerance for latency.

[0073] After calculating the original target number of threads, it is not used immediately. The controller first performs smoothing filtering on it, for example, using exponential smoothing:

[0074] in, The target number of threads obtained after smoothing. This represents the original target thread count, which is the target thread count calculated in the previous step. The target number of threads obtained in the previous calculation. The smoothing factor is set to 0.7, which further prevents output jitter caused by noise in the input parameters. Then, the smoothed target number of threads is compared with the preset lower limit of the number of threads, minThreads (e.g., 2), and the upper limit, maxThreads (e.g., 32), to ensure that the target value falls within the safe range.

[0075] Before executing a decision, the controller performs a double "debouncing" check. First, it checks whether the absolute difference between the calculated final target thread count and the current thread count is greater than or equal to a minimum adjustment step size (e.g., 2). Second, it checks whether the set cooldown time (e.g., 60 seconds) has elapsed since the last successful thread pool adjustment. Only when both conditions are met is the adjustment request considered valid and necessary.

[0076] If the adjustment request is for scaling up (target thread count > current thread count), the controller will trigger a final "safety brake": checking if the JVM heap memory utilization is below 80% and the overall CPU utilization is below 70%. Only if this resource check passes will the scaling up instruction be actually issued. If resources are scarce, scaling up will be suppressed, and a warning log will be logged.

[0077] Finally, during the adjustment process, the controller interacts with the underlying ThreadPoolExecutor. If it's expanding, it directly calls `setCorePoolSize()` and `setMaximumPoolSize()` to set both the core count and maximum thread count to the new target values. If it's shrinking, it first calls `allowCoreThreadTimeOut(true)` and `setKeepAliveTime()` (e.g., setting it to 60 seconds), and then calls `setCorePoolSize()` to reduce the core thread count, ensuring that excess threads can gracefully exit.

[0078] An example of the entire workflow can be described as follows: At the start of a major e-commerce promotion, a large number of order messages flood into Kafka. The monitoring broker detects a message backlog within 10 seconds. It surged rapidly from near 0 to 50,000, and the accumulation rate... The result is positive. The controller calculates that 15 threads are needed to clear the expected backlog within 30 seconds based on the formula. After smoothing and limiting, the target number of threads is determined to be 15. At this time, the current number of threads is the default minThreads, which is 2. Since |15-2|=13 is greater than the minimum step size of 2, and the cooldown time has passed, the adjustment is triggered. Resource checks show that memory and CPU usage are very low, thus meeting the safety conditions. The controller then calls setCorePoolSize(15) and setMaximumPoolSize(15). The thread pool quickly creates 13 new threads, greatly improving the consumption capacity and reducing the message backlog. The growth momentum was curbed and gradually absorbed within minutes, returning to normal levels. After the peak of the promotion, the volume of messages decreased sharply. If the number of threads remains below the low water level, the controller calculates that the target thread count should return to 2. After the debouncing condition is met, the controller executes the scaling-down logic, allowing core threads to time out, and the thread pool size eventually smoothly drops back to 2, freeing up valuable system resources.

[0079] This embodiment organically combines multiple technologies, including dynamic prediction models, self-learning throughput, multiple debouncing mechanisms, and resource protection circuit breakers, to achieve efficient, stable, and secure adaptive adjustment of consumer thread concurrency. Compared to solutions that only implement some features, its overall technical effect is more significant: it can not only respond to load changes at the fastest speed and minimize message processing latency, but also maintain a high degree of system stability throughout the process, avoiding resource abuse and potential crash risks, achieving a good balance between performance, stability, and cost.

[0080] The adaptive adjustment method and apparatus provided in this application have a wide range of applications, and are especially suitable for modern distributed systems with huge fluctuations in business traffic, high requirements for real-time message processing, and a focus on resource cost-effectiveness.

[0081] A typical application scenario is a large e-commerce platform. During major promotional events, the production rate of messages such as order creation, payment success, and inventory deduction can surge hundreds or even thousands of times in a short period. With this application, the consumer service requires no manual intervention; its internal thread pool can automatically expand from a few threads normally to dozens or even hundreds of threads within seconds to handle the surge in order messages, ensuring a smooth ordering experience for users and avoiding serious problems such as order processing delays and overselling of inventory caused by message backlog. After the promotion ends at midnight, when traffic drops sharply, the system automatically reduces the number of threads, freeing up server resources for other services, significantly saving computing costs.

[0082] Another important application scenario is in the financial services sector, particularly in real-time transaction processing, risk control, and clearing and settlement systems. Messages in these systems typically have extremely high time value; any delay can lead to financial losses or compliance risks. For example, a real-time anti-fraud system needs to analyze relevant data streams the instant a user's transaction occurs. This application ensures that even during peak trading periods (such as when the stock market opens), the consumer analyzing the messages maintains sufficient processing power, controlling fraud detection latency to the millisecond level, thereby effectively intercepting risky transactions. During off-peak trading periods, resource consumption is automatically reduced.

[0083] In the field of the Internet of Things (IoT), this application is also applicable. A smart city project may connect millions of sensors (such as traffic cameras, environmental monitors, and smart meters), which intermittently or centrally report massive amounts of data during specific events (such as traffic congestion or pollution incidents). The backend consumers processing this data exhibit typical pulse-like load characteristics. By applying this application, the system can rapidly scale up when data peaks are detected, promptly completing data analysis, storage, and alerts, while operating with extremely low resource consumption during periods of sparse data. This makes it suitable for deployment on resource-constrained edge computing nodes or cost-sensitive cloud environments.

[0084] Furthermore, this application demonstrates its core advantages in numerous scenarios, including log aggregation and analysis, real-time data synchronization, and social network information stream distribution. It intelligently decentralizes resource scheduling within the application, freeing it from reliance on external heavyweight scheduling systems (such as Kubernetes HPA), or serving as a supplement to HPA to provide more granular and faster responses. Whether deployed on traditional virtual machines, private clouds, modern containerized platforms, or even serverless architectures, this application delivers significant performance improvements and cost optimizations to Kafka consumers in a lightweight and non-intrusive manner, enhancing the resilience and robustness of the entire system.

[0085] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0086] It should be noted that in the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means at least two.

[0087] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0088] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0089] Those skilled in the art will understand that all or part of the steps of the methods implementing the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0090] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.

[0091] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0092] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for adaptively adjusting the concurrency of consumer threads, characterized in that, Applied to a Kafka-based message processing system, the method includes the following steps: Obtain the consumer offset and log end offset of each partition of the consumer, calculate the message backlog of each partition, and summarize to obtain the current message backlog of the consumer. Based on the current message backlog, the target number of threads is calculated by combining preset rules or dynamic indicators. The dynamic indicators include at least the rate of change of the backlog within a preset time and the average message processing throughput per thread. If the preset adjustment conditions are met, the core thread count and maximum thread count of the consumer's thread pool are adjusted to the target thread count through the interface of the Kafka consumer task executor. The adjustment conditions include: if it is necessary to increase the core thread count, the JVM heap memory utilization rate must be less than a first threshold and the CPU utilization rate must be less than a second threshold.

2. The method according to claim 1, characterized in that, The target number of threads is calculated based on preset rules, including: Compare the current message backlog with the preset low watermark threshold and high watermark threshold. If the current message backlog is greater than the high watermark threshold, the target number of threads is the smaller of a preset multiple of the current number of threads and the maximum allowed number of threads; if the current message backlog is less than the low watermark threshold and the current number of threads is higher than the minimum allowed number of threads, the target number of threads is the larger of a preset ratio of the current number of threads and the minimum allowed number of threads.

3. The method according to claim 1, characterized in that, The formula for calculating the target number of threads using dynamic metrics is as follows: in, For the target number of threads, This represents the current message backlog. preset time Changes in the amount of internal message backlog That is, the rate of change. Clear the preset target time window. This represents the average message processing throughput for a single thread.

4. The method according to claim 3, characterized in that, The average message processing throughput per thread is calculated by the total number of messages processed within a sliding time window and the total processing time. The duration of the sliding time window is a configurable parameter.

5. The method according to claim 1, characterized in that, The adjustment conditions also include: The difference between the target number of threads and the current number of threads reaches a preset threshold; The preset cooldown time has been reached since the last adjustment of the core thread count.

6. The method according to claim 1, characterized in that, The method of dynamically adjusting the core thread count and maximum thread count of the consumer's thread pool to the target thread count through the Kafka consumer task executor interface includes: If the target number of threads is greater than the current number of threads, call the setCorePoolSize and setMaximumPoolSize methods of the thread pool to adjust the core number of threads and the maximum number of threads to the target number of threads. If the target number of threads is less than the current number of threads, first call the thread pool's allowCoreThreadTimeOut method to allow core threads to time out and exit, then set the idle thread survival time using the setKeepAliveTime method, and finally call the setCorePoolSize method to reduce the number of core threads.

7. A device for adaptively adjusting the concurrency of consumer threads, characterized in that, The apparatus, used in a Kafka-based message processing system, includes: The message backlog status awareness module is used to obtain the consumer offset and log end offset of each partition of the consumer, calculate the message backlog of each partition, and summarize to obtain the current message backlog of the consumer. The target thread count calculation module is used to calculate the target thread count based on the current message backlog and in combination with preset rules or dynamic indicators. The dynamic indicators include at least the rate of change of the backlog within a preset time and the average message processing throughput per thread. The thread pool dynamic refactoring module is used to adjust the core thread count and maximum thread count of the consumer's thread pool to the target thread count through the interface of the Kafka consumer task executor, provided that preset adjustment conditions are met. The adjustment conditions include: if it is necessary to increase the core thread count, the JVM heap memory utilization rate must be less than a first threshold and the CPU utilization rate must be less than a second threshold.

8. The apparatus according to claim 7, characterized in that, Integrate with Java microservice systems via Starters or annotations to deploy on a Kafka message processing system.

Images