Optimization strategy determination method and apparatus, and electronic device

Abstract

The invention discloses an optimization strategy determination method and apparatus, and an electronic device. The method comprises the steps that multiple performance indexes of a target cluster are acquired, the target cluster is used for processing ticket data of multiple service types, and different service types correspond to different service priorities; target scores are determined according to the service priorities and the performance indexes, and the target scores at least comprise one of partition scores used for reflecting the performance state of partitions in the target cluster, theme scores used for reflecting the performance state of themes in the target cluster and cluster scores used for reflecting the overall performance state of the target cluster; and determining an optimization strategy corresponding to the target score, the optimization strategy being used for adjusting resource configuration of the target cluster. The technical problem that the overall performance and reliability of a system are affected due to the fact that cluster resource configuration cannot be dynamically adjusted according to requirements of different services in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, and electronic device for determining an optimization strategy. Background Art

[0002] In carrier billing systems, using multiple Kafka clusters to carry call record data streams has become an industry standard to meet the real-time collection and processing needs of massive amounts of data. However, with the continuous expansion of business scale and the explosive growth of data volume, related technologies are unable to dynamically adjust cluster resource allocation according to the needs of different businesses. As a result, when critical businesses are under high load, additional resources are not released in a timely manner, affecting the overall performance and reliability of the system.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, and electronic device for determining an optimization strategy to at least solve the technical problem that related technologies cannot dynamically adjust cluster resource configuration according to the needs of different businesses, affecting the overall performance and reliability of the system.

[0005] According to one aspect of an embodiment of the present application, a method for determining an optimization strategy is provided, including: obtaining multiple performance indicators of a target cluster, wherein the target cluster is used to process call record data of various business types, and different business types correspond to different business priorities; determining a target score based on the business priority and the performance indicator, wherein the target score includes at least one of the following: a partition score for reflecting the performance status of a partition in the target cluster, a topic score for reflecting the performance status of a topic in the target cluster, and a cluster score for reflecting the overall performance status of the target cluster; determining an optimization strategy corresponding to the target score, wherein the optimization strategy is used to adjust the resource configuration of the target cluster.

[0006] In some embodiments of the present application, the target cluster includes multiple topics, each topic includes multiple partitions, wherein each topic corresponds to a business type; determining the target score based on the business priority and performance indicators, including: obtaining a first performance indicator corresponding to each partition in the target cluster, and a performance indicator threshold corresponding to the first performance indicator, wherein the performance indicator threshold is used to represent the performance target of the first performance indicator; determining the target topic corresponding to each partition, and determining a first weight of the first performance indicator based on the business priority corresponding to the target topic; determining a ratio between the first performance indicator and the performance indicator threshold, wherein the ratio is used to reflect the degree of deviation between the first performance indicator and the ideal state; determining the partition score in the target score based on the ratio and the first weight.

[0007] In some embodiments of the present application, the topic score is determined in the following manner: determining a target partition score from all partition scores of the target topic, wherein the target partition score includes the worst partition score; obtaining a second performance indicator and a third performance indicator corresponding to the target topic, wherein the second performance indicator is used to reflect the message processing capability of consumers subscribed to the target topic, and the third performance indicator is used to reflect the health status of data synchronization between multiple partitions within the target topic and server nodes; determining a first penalty coefficient and a second penalty coefficient corresponding to the second performance indicator and the third performance indicator respectively, wherein the first penalty coefficient and the second penalty coefficient are determined based on the business type corresponding to the target topic; determining the topic score based on the target partition score, the second performance indicator, the third performance indicator, the first penalty coefficient and the second penalty coefficient.

[0008] In some embodiments of the present application, the cluster score is determined in the following manner: obtaining a topic score set for all topics in the target cluster, and determining a second weight corresponding to each topic score, wherein the second weight is determined based on the business priority corresponding to the topic; obtaining a fourth performance indicator corresponding to the target cluster, wherein the fourth performance indicator is used to reflect the load status of the server nodes in the target cluster; determining a third penalty coefficient corresponding to the fourth performance indicator, wherein the third penalty coefficient is determined based on the degree of resource tension of the target cluster; and determining the cluster score based on the topic score set, the second weight, the fourth performance indicator, and the third penalty coefficient.

[0009] In some embodiments of the present application, when the target score is a partition score, an optimization strategy corresponding to the target score is determined, including: when the partition score of the target partition is continuously greater than a first threshold within a preset time period, and the resource utilization rate of the server node where the target partition is located is greater than the utilization rate threshold, a first optimization strategy is generated, wherein the first optimization strategy is used to adjust the partition resource configuration within the target cluster.

[0010] In some embodiments of the present application, the first optimization strategy includes: increasing the number of partitions within the target cluster; or, obtaining a first server node within the target cluster whose load is less than a load threshold, and determining the first server node as the server node for processing write requests for the target partition.

[0011] In some embodiments of the present application, when the target score is a topic score, an optimization strategy corresponding to the target score is determined, including: when the topic score of the target topic is greater than a second threshold, the third performance indicator is greater than a time threshold, and the business priority corresponding to the target topic is greater than or equal to the priority threshold, a second optimization strategy is generated, wherein the second optimization strategy is used to adjust the topic resource configuration within the target cluster.

[0012] In some embodiments of the present application, the second optimization strategy includes: obtaining a second server node in the target cluster whose network delay is less than a delay threshold, and processing message read and write operations of the target topic on the second server node; or, increasing the number of threads for processing replica data synchronization of each server node in the target cluster.

[0013] In some embodiments of the present application, when the target score is a topic score, an optimization strategy corresponding to the target score is determined, including: when the second performance indicator is greater than the message backlog threshold, the continuous growth duration of the second performance indicator is greater than the duration threshold, and the difference between the message consumption rate and the message production rate of the target topic is greater than the difference threshold, a third optimization strategy is generated, wherein the third optimization strategy is used to adjust the topic resource configuration within the target cluster.

[0014] In some embodiments of the present application, the third optimization strategy includes: increasing the number of consumers in the consumer group corresponding to the target topic, wherein the consumer group includes multiple consumers and each consumer group jointly consumes messages from at least one topic; or, adjusting the consumer parameters corresponding to the target topic, wherein the consumer parameters include the maximum number of bytes of data requested by the consumer from the server node or the upper limit of the time the consumer waits when there is not enough data; or, reducing the message production rate of the target topic.

[0015] In some embodiments of the present application, the method further includes: obtaining cluster scores corresponding to multiple target clusters, and determining the health level corresponding to each target cluster based on the cluster scores, wherein the health level is used to quantitatively represent the performance status of the target cluster when processing call record data; sorting the multiple target clusters based on the health level to obtain a sorting result, wherein the sorting result is used to determine the optimization order of the target clusters.

[0016] In some embodiments of the present application, after determining the optimization strategy corresponding to the target score, it also includes: determining the execution script corresponding to the optimization strategy, and simulating the execution of the execution script in a sandbox cluster to obtain a simulated execution result, wherein the sandbox cluster is used to test the feasibility of the execution script; when the simulated execution result meets the preset conditions, the execution script is executed in the target cluster.

[0017] According to another aspect of an embodiment of the present application, a device for determining an optimization strategy is also provided, including: an acquisition module for acquiring multiple performance indicators of a target cluster, wherein the target cluster is used to process call record data of various business types, and different business types correspond to different business priorities; a determination module for determining a target score based on the business priority and performance indicators, wherein the target score includes at least one of the following: a partition score for reflecting the performance status of a partition in the target cluster, a topic score for reflecting the performance status of a topic in the target cluster, and a cluster score for reflecting the overall performance status of the target cluster; an optimization module for determining an optimization strategy corresponding to the target score, wherein the optimization strategy is used to adjust the resource configuration of the target cluster.

[0018] According to another aspect of the embodiments of the present application, an electronic device is provided, including: a memory and a processor, the memory being used to store program instructions; the processor being connected to the memory and being used to execute a determination method for implementing the above-mentioned optimization strategy.

[0019] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned method for determining the optimization strategy by running the computer program.

[0020] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned method for determining the optimization strategy when executed by a processor.

[0021] In the embodiment of the present application, a multi-level performance evaluation and intelligent resource scheduling method is adopted. By real-time monitoring and quantitative analysis of the performance indicators of the target cluster in processing call data of various business types, and combining business priorities, the purpose of dynamically optimizing the configuration of the Kafka cluster is achieved, thereby achieving the technical effect of dynamically adjusting the resource configuration of the Kafka cluster according to the real-time and importance of the business, and thus solving the technical problem that the relevant technology cannot dynamically adjust the cluster resource configuration according to the needs of different businesses, affecting the overall performance and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 This is a hardware structure block diagram of a computer terminal according to a method for determining an optimization strategy in an embodiment of the present application;

[0024] Figure 2is a flow chart of a method for determining an optimization strategy according to an embodiment of the present application;

[0025] Figure 3 is a system architecture diagram of a method for determining an optimization strategy according to an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of an optimization process of a method for determining an optimization strategy according to an embodiment of the present application;

[0027] Figure 5 It is a structural diagram of a device for determining an optimization strategy according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0031] Call Detail Record (CDR): In telecommunications operations, it refers to data records that record detailed information about communication services such as telephone calls, data usage, and text messages. In the embodiments of this application, call detail record data is the core object processed by the system and is used to support real-time billing and bill generation.

[0032] Kafka Cluster: A distributed messaging system built on Apache Kafka, used for high-throughput data publishing and subscription services. In the embodiment of this application, the Kafka cluster is the infrastructure that carries the call record data stream and is responsible for the real-time collection, storage, and transmission of call record data.

[0033] In carrier billing systems, the real-time collection and processing of call record data (such as call logs and traffic usage records) is a core business. With the increasing number of users and the rapid growth of data volume, the system needs to process billions of call record data daily, with a peak write rate of millions per second. However, the solutions used in related technologies have the following shortcomings in handling high-concurrency, low-latency scenarios, affecting the performance and stability of the overall billing system:

[0034] (1) Difficulty in locating performance bottlenecks: Across Kafka clusters, the load distribution of topics and partitions is uneven, causing some Kafka brokers to become performance bottlenecks and making it difficult to achieve load balancing. The performance data provided by related technical monitoring tools (such as Prometheus) is mostly based on basic indicators, lacking in-depth analysis of the performance of different topics and partitions in the Kafka cluster. This makes it impossible for the system to identify performance bottlenecks in a timely manner, resulting in resource waste or poor performance, further affecting billing processing efficiency.

[0035] (2) Low resource utilization: Since most existing systems use static configuration, they cannot automatically adjust resource configuration according to real-time business fluctuations. Especially during business peak periods (such as holidays or specific time periods), Kafka cluster resource configuration cannot adapt to the sudden increase in business load, resulting in reduced throughput, increased latency, and even possible service interruption.

[0036] (3) Lack of dynamic optimization capabilities based on business priorities: Related technologies cannot dynamically adapt to business needs and lack intelligent analysis and real-time optimization of Kafka cluster performance. For example, most tools provide resource monitoring such as throughput and latency, and cannot dynamically adjust cluster configurations based on the real-time requirements of different businesses (such as voice call records and traffic call records). They fail to effectively combine business priorities with dynamic adjustments of system resources, resulting in failure to optimize in a timely manner when key businesses are under high load, affecting the overall performance and reliability of the system.

[0037] In order to solve the above technical problems, the embodiments of the present application provide corresponding solutions, which are described in detail below.

[0038] The method for determining the optimization strategy provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal for implementing a method for determining an optimization strategy. Figure 1 As shown, the computer terminal 10 may include one or more (illustrated by 102a, 102b, ..., 102n in the figure) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0039] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for determining the optimization strategy in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned method for determining the optimization strategy. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0041] The transmission module 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission module 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission module 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0042] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0043] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0044] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a method for determining an optimization strategy. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] Figure 2 is a flow chart of a method for determining an optimization strategy according to an embodiment of the present application, such as Figure 2 As shown, the method includes the following steps:

[0046] Step S202 : obtaining multiple performance indicators of a target cluster, wherein the target cluster is used to process call record data of various service types, and different service types correspond to different service priorities.

[0047] In the above step S202, the target cluster includes a set of Kafka clusters that process call record data of various business types. The performance indicators refer to performance parameters used to evaluate the target cluster when processing data, including but not limited to throughput, latency, resource load, data synchronization status, etc.

[0048] The service type refers to the service category to which the call bill data belongs, such as real-time voice call bills, near-real-time traffic call bills, and offline monthly bills. Each service type has different processing time requirements and priority weights.

[0049] In some embodiments of the present application, various key performance indicators can be collected in real time through monitoring tools deployed on the target cluster (such as Kafka internal indicators, Prometheus, Grafana, etc.), including but not limited to:

[0050] (1) Throughput: reflects the writing and processing rate of call record data and is a basic indicator for evaluating the cluster's carrying capacity.

[0051] (2) Latency: This includes end-to-end latency (i.e., processing time from producer to consumer) and ISR synchronization latency, which are used to measure the timeliness of data processing and the reliability of data.

[0052] (3) Resource load: Monitor the CPU, memory, disk IO utilization, and network bandwidth of the Broker (i.e., the server node of the target cluster) to help determine whether the system bottleneck is caused by insufficient resources.

[0053] (4) Load balancing: Analyze the distribution status of partitions to determine whether there are hotspot partitions that affect the overall processing efficiency of the cluster.

[0054] To ensure the priority of different service types in performance optimization, service weight configuration can also be introduced (such as a voice call weight of 0.9, a traffic call weight of 0.7, and a monthly bill weight of 0.3). This configuration combines the real-time nature and importance of the service while collecting performance indicators, providing key parameters for the subsequent construction of the performance scoring model. For example, when calculating the partition-level performance index (i.e., the partition score below), the service priority weight directly affects the weight distribution of the indicator, ensuring that performance issues of high-time-efficiency services are prioritized for identification and resolution. This solves the problem in related technologies that cannot optimize resources based on differences in service requirements and enables service-aware performance bottleneck identification.

[0055] In some embodiments of the present application, for billing service metadata collection and weight configuration, in order to support the business perception capabilities of Kafka multi-cluster performance optimization analysis, a data input module can be used to collect and maintain billing-related metadata information. This module provides basic parameter support required by the optimization algorithm and has dynamic adjustment and persistence capabilities. Its core functions can be divided into the following four aspects:

[0056] (1) Kafka cluster connection information: stores the Kafka cluster’s Broker address, port, authentication parameters (SASL / SSL), cluster identifier, and business domain (e.g., “a certain city’s billing cluster”).

[0057] (2) Business mapping relationship: Associate Kafka topics with billing business types to ensure that optimization strategies can be adjusted based on business attributes. For example, map "voice_call" to "voice call bill", map "data_usage" to "traffic call bill", and map "monthly_bill" to "monthly bill".

[0058] (3) Service Priority Weight: Set weights for different service types to guide optimization decisions. For example, the weight of voice call records (real-time billing) is 0.9, the weight of traffic call records (quasi-real-time billing) is 0.7, and the weight of monthly bills (offline processing) is 0.3. The weights can be used for subsequent throughput optimization, partition balancing, and resource allocation strategies.

[0059] (4) Dynamic weight adjustment mechanism: Monitor business load fluctuations and dynamically adjust business priorities based on real-time traffic. For example, during peak business periods (such as holidays), the weight of voice calls is increased to 1.0 and the weight of traffic calls is reduced to 0.5. During low business periods, the default weight configuration is restored to optimize resource utilization.

[0060] In some embodiments of the present application, after billing service metadata collection and weight configuration, the following steps may also be performed: The performance evaluation module is responsible for real-time performance monitoring and benchmark capability testing of the Kafka multi-cluster environment, with the aim of providing a quantitative basis for performance bottleneck identification, indicator threshold setting, and subsequent optimization strategies. This module includes two core sub-functions, namely real-time performance indicator collection and benchmark performance capability testing, the specific contents of which are as follows:

[0061] (1) Real-time performance indicator collection, including but not limited to:

[0062] 1) Throughput: The message production rate (Records / s) reflects the call record writing capability and affects the real-time billing.

[0063] 2) Latency: End-to-end latency (Producer to Consumer) directly determines billing processing time.

[0064] 3) Data reliability: ISR replica synchronization delay (Replica Lag). Excessive delay may lead to data loss.

[0065] 4) Resource load: Broker CPU or memory utilization, disk I / O. Resource overload can trigger message accumulation.

[0066] 5) Load balancing: Partition distribution must be uniform. Uneven distribution can cause some brokers to become performance bottlenecks.

[0067] It should be noted that all the above indicators support minute-level or second-level sampling frequencies and are processed in a unified performance analysis module through a streaming data pipeline.

[0068] (2) Benchmark performance capability testing (executed in a test environment). To accurately set thresholds and verify system performance limits, this application may also include a benchmark testing process for the Kafka cluster, which can be executed in a pre-production or sandbox environment.

[0069] 1) High-load write simulation: Use the official Kafka performance testing tool to simulate peak business scenarios (such as writing millions of call records per second). Test the maximum write throughput and minimum achievable latency of the Kafka cluster under different partitioning, replica, and compression configurations. Record the following data as a performance baseline: maximum carryable throughput (records / s), optimal replication factor configuration, minimum end-to-end latency, and identification and comparative evaluation of system bottlenecks.

[0070] 2) Benchmarking can also be used to identify the source of resource bottlenecks, such as whether disk IO is limiting, whether the broker CPU bottleneck is obvious, whether partition distribution is skewed, and whether ISR replicas fall behind during peak write times. Test results are used to guide resource scheduling strategy setting and cluster scale planning.

[0071] 3) Threshold setting references: This involves combining real-time monitoring data with benchmark test results to set key performance thresholds. For example, end-to-end latency thresholds: ≤1 second for real-time services, ≤3 seconds for near-real-time services; ISR synchronization latency thresholds: ≤5 seconds for high-reliability services; and consumer backlog thresholds: triggering an alert if the backlog exceeds 100,000 and continues to grow. These thresholds can serve as a basis for triggering subsequent optimization processes.

[0072] It should be noted that the target cluster may include multiple topics, each topic includes multiple partitions, and each topic corresponds to a business type.

[0073] The target cluster refers to the Kafka cluster that processes call record data for different business types. It consists of multiple Broker nodes and is responsible for data storage, transmission, and access. Topics are the mechanism used in the Kafka cluster to logically categorize messages. Each topic can carry different types of call record data (such as voice call records and traffic call records), and different topics correspond to different business types and performance requirements. Partitions are physical data segments in Kafka, used to achieve parallel data processing and load balancing. Each topic can contain multiple partitions. Partitions are the smallest processing unit for performance optimization and analysis, and their performance directly affects the overall processing capacity of the topic.

[0074] Step S204: Determine a target score based on the business priority and performance indicators, wherein the target score includes at least one of the following: a partition score for reflecting the performance status of a partition in the target cluster, a topic score for reflecting the performance status of a topic in the target cluster, and a cluster score for reflecting the overall performance status of the target cluster.

[0075] In the above step S204, the target score is a score generated for the partitions, topics and overall cluster status in the cluster through quantitative evaluation, which is used to identify performance bottlenecks and decide on optimization directions. The target score may include the partition score (Score Partition ), Score Topic ) and cluster score (Score Cluster ).

[0076] In some embodiments of the present application, a multi-level scoring model may be constructed, including:

[0077] (1) Partition-level scoring: Based on a single partition, the throughput, latency, and resource consumption are analyzed, and combined with the business priority weight, a score is derived that reflects the performance status of the partition. Partition-level scoring can identify hotspot partitions with high load or high latency, providing an entry point for resource balancing.

[0078] (2) Topic-level scoring: This is calculated based on the scores of all partitions within a topic, as well as the topic's own characteristics (such as message backlog and replica synchronization delay), to reflect the overall performance of the entire topic. Topic-level scoring provides insights into the overall processing capacity and potential bottlenecks of specific business types, facilitating optimization at the business level.

[0079] (3) Cluster-level scoring: All topic scores are aggregated and combined with the resource load of the Broker nodes to calculate the performance score of the entire cluster. The cluster-level score is used to evaluate and compare the health of clusters, helping to identify clusters that need priority optimization and achieve reasonable allocation of global resources.

[0080] In order to solve the problem of matching resource allocation with business needs and ensure that the billing system can automatically identify and prioritize the performance of high-priority services when processing various business types, the indicator weights in the scoring model can be dynamically adjusted according to the real-time and importance of the business. For example, for services with extremely high real-time requirements (such as voice call records), the indicator weights of delay and backlog will be set higher so that the system can respond more sensitively to performance fluctuations of such services.

[0081] In some embodiments of the present application, the partition score can be determined in the following manner: obtain the first performance indicator corresponding to each partition in the target cluster, and the performance indicator threshold corresponding to the first performance indicator, wherein the performance indicator threshold is used to represent the performance target of the first performance indicator; determine the target topic corresponding to each partition, and determine the first weight of the first performance indicator based on the business priority corresponding to the target topic; determine the ratio between the first performance indicator and the performance indicator threshold, wherein the ratio is used to reflect the degree of deviation between the first performance indicator and the ideal state; determine the partition score in the target score based on the ratio and the first weight.

[0082] The first performance indicator refers to one of the key indicators that affect partition performance, such as end-to-end latency, message backlog, ISR replica synchronization latency, etc. The performance indicator threshold is the ideal performance target set for the first performance indicator, representing the maximum or optimal performance acceptable to the business.

[0083] Service priority is a priority level set based on different service types. For example, real-time voice call records have a higher priority than offline monthly bills. This is used to adjust the weight of the first performance indicator to ensure that high-priority services are given priority. The first weight is the weight given to the first performance indicator based on the service priority, reflecting the relative importance of this performance indicator in a specific service.

[0084] For example, for each topic's single partition, the performance index (i.e., partition score) is calculated as follows:

[0085]

[0086] Among them, W i Metric is the indicator weight, which is determined based on the business priority (e.g. if delay is important, it is set to 0.5); i is the current value of the i-th performance indicator (such as current delay); Threshold i The maximum threshold allowed by the service (for example, the end-to-end delay threshold is 1 second).

[0087] It should be noted that the higher the partition score, the greater the current performance load pressure of the partition. PartitionNormalized to [0,∞), for example, you can set an early warning threshold such as Score Partition A value >1 indicates a performance anomaly. In addition, performance indicators that can be covered include but are not limited to latency, ISR synchronization delay, backlog, and throughput variation.

[0088] In some embodiments of the present application, the topic score is determined in the following manner: determining a target partition score from all partition scores of the target topic, wherein the target partition score includes the worst partition score; obtaining a second performance indicator and a third performance indicator corresponding to the target topic, wherein the second performance indicator is used to reflect the message processing capability of consumers subscribed to the target topic, and the third performance indicator is used to reflect the health status of data synchronization between multiple partitions within the target topic and server nodes; determining a first penalty coefficient and a second penalty coefficient corresponding to the second performance indicator and the third performance indicator respectively, wherein the first penalty coefficient and the second penalty coefficient are determined based on the business type corresponding to the target topic; determining the topic score based on the target partition score, the second performance indicator, the third performance indicator, the first penalty coefficient and the second penalty coefficient.

[0089] The target topic refers to a topic that carries call record data of a specific service type, such as a voice call record topic or a traffic call record topic. The target partition score is a representative or critical score selected from all partition scores of the target topic, including but not limited to the worst partition score, which is used to evaluate the lowest point and hotspot of the overall performance of the topic.

[0090] The second performance indicator refers to an indicator that reflects the consumer's message processing capabilities, such as consumer throughput or message backlog size. It is used to evaluate whether the processing capabilities of the consumer side match those of the producer side, and directly affects the processing efficiency and response speed of the topic.

[0091] The third performance indicator is an indicator that measures the health of data synchronization between multiple partitions and server nodes within a topic, such as ISR replica delay (In-Sync Replica Lag). It reflects the reliability of data and the synchronization efficiency within the system. It is an important indicator for ensuring zero data loss and high reliability.

[0092] The first and second penalty coefficients are set based on the target topic's business type and real-time requirements. They are used to amplify the impact of the second and third performance indicators when calculating the topic score. A larger penalty coefficient indicates a higher weight for that performance indicator in the score, and accordingly, the topic score is more sensitive to deviations from that indicator.

[0093] For example, the Topic-level performance index Score TopicThe topic score is based on the partition and further evaluates the overall status of the entire topic under business traffic pressure. The calculation formula is:

[0094] Score Topic =max(Score Partition )+α·Backlog Size +γ·ISR Lag (Formula 2)

[0095] Among them, Score Partition The worst partition score in the current topic, Backlog Size The backlog of unconsumed messages (unit: 10,000), reflecting the processing capacity of the consumer end; ISR Lag is the ISR replica synchronization delay (unit: second), reflecting the healthy status of data synchronization; α is the backlog penalty coefficient (i.e., the first penalty coefficient, default α = 0.2). The larger the backlog, the higher the index. For high-real-time services (such as voice call records), α can be increased (e.g., 0.3-0.5) to ensure that backlog issues are handled first. For offline batch services (such as monthly bills), α can be appropriately reduced (e.g., 0.1-0.2); γ is the ISR delay penalty coefficient (i.e., the second penalty coefficient, default γ = 0.1). The higher the delay, the higher the index.

[0096] It should be noted that the introduction of topic-level global factors can avoid relying solely on local partition information, and the penalty coefficient can be adjusted according to the business type to achieve strategy differentiation. Topic It is highly sensitive to three types of problems: "hot partition + consumption accumulation + replica delay".

[0097] In some embodiments of the present application, the cluster score is determined in the following manner: obtaining a topic score set for all topics in the target cluster, and determining a second weight corresponding to each topic score, wherein the second weight is determined based on the business priority corresponding to the topic; obtaining a fourth performance indicator corresponding to the target cluster, wherein the fourth performance indicator is used to reflect the load status of the server nodes in the target cluster; determining a third penalty coefficient corresponding to the fourth performance indicator, wherein the third penalty coefficient is determined based on the degree of resource tension of the target cluster; and determining the cluster score based on the topic score set, the second weight, the fourth performance indicator, and the third penalty coefficient.

[0098] The topic score set is the collection of topic scores for all topics within the target cluster, reflecting the overall performance of different business types within the cluster. The second weight corresponds to the business priority of each topic, reflecting its importance in the overall cluster performance assessment. This weight is used to weight topic scores when calculating the cluster score, ensuring that performance issues in high-priority topics are identified and addressed first.

[0099] The fourth performance indicator reflects the load status of server nodes, such as CPU utilization, memory usage, and disk I / O, and is used to assess cluster resource usage and health. The third penalty factor is set based on the resource shortage level of the target cluster. It is used to amplify the impact of the fourth performance indicator when calculating the cluster score. When cluster resources are scarce, the third penalty factor increases, prompting the scoring system to place greater emphasis on resource utilization.

[0100] For example, the cluster-level performance index Score cluster The calculation formula for cluster score is:

[0101] Score cluster =∑(Score Topic Topic Weight )+β·Broker Load (Formula 3)

[0102] Among them, Score Topic The performance score of the Topic (i.e., the topic score); Weight The service priority weight corresponding to the Topic (such as 0.8 for voice call records); Broker Load is the average Broker load (such as the comprehensive score of CPU + memory + disk IO); β is the load penalty coefficient (i.e. the third penalty coefficient, the default β = 0.1). The higher the load, the higher the index. In a resource-constrained environment (such as an insufficient number of Brokers), it is recommended to increase β (such as 0.2-0.3) and give priority to optimizing high-load Brokers. In an environment with sufficient resources, β can be appropriately reduced (such as 0.05-0.1).

[0103] It should be noted that cluster scores can be weighted and integrated according to the importance of different businesses, and system load can be included in the calculation to prevent over-reliance on business-side indicators. Cluster scores can be used to sort the health of multiple Kafka clusters, giving priority to optimizing the cluster with the highest score.

[0104] All target scoring indices (i.e., partition scoring, topic scoring, and cluster scoring) support real-time or quasi-real-time updates (e.g., once per minute), and the scoring results will be used to trigger optimization strategies. For example, if the Score of a Partition Partition>1.5, enter the hotspot partition optimization candidate queue, if Score cluster If the value is higher than the preset threshold for 5 consecutive minutes, optimization suggestions will be automatically issued.

[0105] The index system in this application has good scalability and can support the introduction of new indicators (such as network bandwidth, cross-cluster synchronization delay, etc.) to calculate the target score.

[0106] Step S206 : determining an optimization strategy corresponding to the target score, wherein the optimization strategy is used to adjust resource configuration of the target cluster.

[0107] In step S206 above, the optimization strategy is a series of resource allocation adjustment measures formulated based on the performance bottlenecks and deviation points identified by the target score. For example:

[0108] (1) When the system identifies that the target score for a specific partition is too high, it means that the partition is experiencing a performance bottleneck. At this point, optimization strategies may include increasing the number of partitions under the topic to distribute the load, or adjusting the replica distribution of the partition to improve data reliability.

[0109] (2) The topic score reflects the overall performance of all partitions under the topic, as well as the consumer processing capacity and data synchronization status. When the topic score is higher than the threshold, the system automatically analyzes the processing capacity and data synchronization status of the consumer side, adjusts the concurrency of the consumer group, or optimizes the replica synchronization configuration on the broker side.

[0110] (3) If the overall cluster score of the target cluster is higher than the predetermined threshold, it indicates that the cluster is facing resource constraints and performance challenges. Optimization strategies may include scaling cluster resources horizontally or vertically, such as increasing the number of broker nodes or upgrading server hardware, while adjusting cluster parameter configurations to optimize resource utilization efficiency.

[0111] When the target score is a partition score, the optimization strategy corresponding to the target score can be determined in the following manner: if the partition score of the target partition is continuously greater than a first threshold for a preset time period, and the resource utilization rate of the server node where the target partition resides is greater than a utilization rate threshold, a first optimization strategy is generated, wherein the first optimization strategy is used to adjust the partition resource configuration within the target cluster. The first optimization strategy may include: increasing the number of partitions within the target cluster; or obtaining a first server node within the target cluster whose load is less than a load threshold, and determining the first server node as the server node for processing write requests for the target partition.

[0112] The first threshold is the set performance score limit, which is used to determine whether the partition needs to be optimized. When the score of the target partition continues to exceed the first threshold, it indicates that the performance of the partition is insufficient to meet business needs and optimization measures need to be taken.

[0113] Resource utilization refers to the resource consumption of the server node where the target partition resides, such as CPU, memory, and network bandwidth. Excessive resource utilization can reduce the partition's processing capacity, necessitating adjustments to resource allocation or load distribution. The utilization threshold is a warning line for resource utilization, used to determine whether a server node is overloaded. When utilization exceeds this threshold, server resources are limited, potentially becoming a performance bottleneck, and optimization strategies are needed.

[0114] The first optimization strategy is a solution for optimizing resource allocation based on the resource status of the target partition and the server node where it is located. It aims to disperse the load and improve performance by increasing the number of partitions or adjusting the partition write request processing nodes.

[0115] The load threshold refers to the set server node load limit, which is used to screen server nodes suitable for receiving more partition write requests. Nodes with lower load thresholds mean that they have sufficient processing power and can take on more partition write tasks.

[0116] Specifically, for the optimization strategy for the partition hotspot problem, the triggering condition can be: the score of a partition Partition >1.5 and lasts for more than three monitoring cycles and / or the CPU or disk IO usage of the broker where the partition is located is higher than 80%. The corresponding optimization strategies include:

[0117] (1) Dynamically increase the number of partitions (e.g., from 12 to 24) to enhance parallelism;

[0118] (2) Use the Kafka Admin API to reassign the Partition Leader to a less-loaded Broker. It should be noted that the Partition Leader is the Broker instance responsible for receiving and processing write requests in each Partition. Each Partition has a specific Broker designated as the Leader, while other Brokers serve as the Followers or Replicas of that Partition for redundant data storage and fault recovery.

[0119] (3) Check the standard deviation of the partition distribution to ensure that the load balance is improved.

[0120] When the target score is a topic score, the optimization strategy corresponding to the target score can be determined in the following manner: when the topic score of the target topic is greater than a second threshold, the third performance indicator is greater than a time threshold, and the business priority corresponding to the target topic is greater than or equal to the priority threshold, a second optimization strategy is generated, wherein the second optimization strategy is used to adjust the topic resource configuration within the target cluster. The second optimization strategy includes: obtaining a second server node within the target cluster whose network latency is less than the latency threshold, and processing message read and write operations for the target topic on the second server node; or increasing the number of threads for processing replica data synchronization on each server node within the target cluster.

[0121] The second threshold is the warning line for the topic score, which is used to determine whether optimization measures are needed for the topic. The purpose of setting the second threshold is to automatically identify topics with abnormal performance based on business needs and trigger the optimization mechanism.

[0122] The third performance indicator (see the previous examples) measures the health of data synchronization, such as ISR replica latency, reflecting the efficiency of data synchronization between multiple partitions within the target topic and the server nodes. When the third performance indicator exceeds a time threshold, data synchronization anomalies are present and require optimization. The time threshold is used to determine whether the third performance indicator exceeds the normal range. Setting the time threshold is intended to promptly identify data synchronization anomalies.

[0123] Business priorities are set based on business type and real-time requirements. They determine the order and intensity of optimization strategy execution. Topics with high business priorities receive greater weight in resource allocation and performance optimization, ensuring efficient and stable processing of critical business tasks. Priority thresholds are used to prioritize optimization for topics. Topics with business priorities exceeding this threshold receive priority in optimization strategy execution to meet the performance requirements of high-priority businesses.

[0124] The second optimization strategy refers to the optimization strategy automatically formulated by the system based on the performance and business priority of the target topic. It aims to improve data processing efficiency and system stability and reduce latency by adjusting the topic resource configuration within the target cluster.

[0125] The second server node is a server node whose network delay in the target cluster is lower than the delay threshold. It is used to process message read and write operations on the target topic. Selecting the second server node can reduce data transmission delay and improve processing speed.

[0126] Latency Threshold: A threshold used to determine whether network latency is within an acceptable range. Server nodes with network latency below the latency threshold are prioritized for processing message read and write operations to optimize data transmission efficiency. The number of threads used to process replica data synchronization is the number of threads on each server node dedicated to data synchronization. Increasing the number of threads can speed up replica data synchronization, improve data processing parallelism, and enhance overall performance.

[0127] Specifically, the optimization strategy for abnormal ISR synchronization delay can be triggered by: Lag >5 seconds and Score Topic >1.2; and the topic is a high-reliability service (e.g., weight ≥0.8). The corresponding optimization strategies include:

[0128] (1) Optimize Broker parameters (such as num.replica.fetchers) to improve thread synchronization capabilities;

[0129] (2) Prioritize scheduling the topic to the broker with lower network latency.

[0130] In the case where the target score is a topic score, the optimization strategy corresponding to the target score can also be determined in the following manner: when the second performance indicator is greater than the message backlog threshold, the duration of the continuous growth of the second performance indicator is greater than the duration threshold, and the difference between the message consumption rate and the message production rate of the target topic is greater than the difference threshold, a third optimization strategy is generated, wherein the third optimization strategy is used to adjust the topic resource configuration within the target cluster. The third optimization strategy includes: increasing the number of consumers in the consumer group corresponding to the target topic, wherein the consumer group includes multiple consumers, and each consumer group jointly consumes messages from at least one topic; or, adjusting the consumer parameters corresponding to the target topic, wherein the consumer parameters include the maximum number of bytes of data requested by the consumer from the server node or the upper limit of the time the consumer waits when there is not enough data; or, reducing the message production rate of the target topic.

[0131] The second performance indicator (refer to the above embodiment) includes the message backlog, that is, the number of messages in the target topic that have not been fully processed by consumers. The message backlog threshold is a set message backlog warning line, which is used to determine whether the data processing speed of the target topic cannot keep up with the production speed. When the message backlog exceeds the threshold, it indicates that the system may face a performance bottleneck or resource allocation problem.

[0132] The duration threshold refers to the length of time when the second performance indicator continues to be higher than the message backlog threshold. It is used to assess the severity and stability of the backlog problem. Setting the duration threshold helps identify long-term performance problems rather than temporary fluctuations.

[0133] The message consumption rate and message production rate reflect the speed of message consumption and production, respectively, for the target topic. The difference between these two rates is an important indicator of system processing capacity and load balancing. The difference threshold is a set warning line for the difference between the production rate and the consumption rate. If the difference threshold is too high, it may indicate insufficient processing capacity on the consumer side, requiring measures to improve the processing efficiency of the consumer group to avoid data backlogs.

[0134] The third optimization strategy is an optimization strategy automatically formulated by the system when an abnormality occurs in the data processing status of the target topic. It aims to improve data consumption efficiency and ensure the real-time and accuracy of data processing by adjusting the configuration of the consumer group.

[0135] A consumer group is a group of consumers that coordinates the consumption of one or more topics. This concept allows multiple consumers to process messages from a shared topic in parallel, improving data consumption throughput and system performance. Consumer parameters include the maximum number of bytes of data a consumer can request from a server node and the maximum time a consumer will wait if insufficient data is available.

[0136] Reducing the message production rate means temporarily reducing the message production rate as a short-term remedial measure when the processing capacity on the consumer side is insufficient, preventing further increase in message backlog and ensuring stable system operation.

[0137] Specifically, for optimization strategies with serious backlogs on the consumer side, the triggering conditions can be: Backlog Size >100,000 and the number of entries increases continuously for more than 5 minutes and / or the difference between the topic consumption rate and the production rate exceeds 30%. The corresponding optimization strategies include:

[0138] (1) Dynamic expansion corresponds to the number of Consumer Group instances;

[0139] (2) Adjust Consumer parameters, such as increasing fetch.max.bytes and reducing fetch.wait.max.ms;

[0140] (3) Configure a short-term traffic restriction policy for the topic to avoid continuous accumulation.

[0141] In some embodiments of the present application, the following steps can also be performed: obtaining cluster scores corresponding to multiple target clusters, and determining the health level corresponding to each target cluster based on the cluster scores, wherein the health level is used to quantitatively represent the performance status of the target cluster when processing call record data; sorting the multiple target clusters according to the health level to obtain a sorting result, wherein the sorting result is used to determine the optimization order of the target clusters.

[0142] Health can be calculated by integrating cluster scores with multiple factors such as business weights and resource loads. For example, cluster scores can be combined with business priority weights while considering the overall resource load of the cluster to form a comprehensive health evaluation system.

[0143] The system sorts multiple target clusters based on the calculated health, with clusters with lower health being optimized first. The sorting algorithm can be simple ascending or descending, or a more complex sorting strategy can be adopted based on business priority and resource availability.

[0144] After determining the optimization strategy corresponding to the target score, the following steps may be performed: determining the execution script corresponding to the optimization strategy, and simulating the execution of the execution script in a sandbox cluster to obtain a simulated execution result, wherein the sandbox cluster is used to test the feasibility of the execution script; and executing the execution script in the target cluster when the simulated execution result meets the preset conditions.

[0145] Specifically, once a policy is approved or meets the automatic triggering conditions, the system can implement optimization actions according to the following process:

[0146] (1) Generate optimization scripts: Automatically call the Kafka Admin API, configuration center, or script platform. The script content includes: adding / reallocating partitions, modifying topic parameters (such as retention.ms, replication factor), and adjusting broker-side parameters (such as sending cache and number of pull threads).

[0147] (2) Sandbox environment verification: First, perform simulation execution in the sandbox Kafka cluster (the structure is consistent with production), that is, inject simulated call record data, monitor the improvement trend of indicators, check partition distribution, ISR status, consumer compatibility, etc., and perform online deployment after verifying that there are no abnormalities to reduce risks.

[0148] (3) Formal change execution: Generate a complete operation log before execution, support visual monitoring during execution (real-time tracking of performance indicators), and retain a change snapshot (including the old configuration) after execution.

[0149] (4) Rollback mechanism and exception handling: To prevent potential risks brought by optimization operations, the system supports a one-click rollback strategy to ensure business continuity and data integrity.

[0150] Through the above steps S202 to S206, a multi-level performance evaluation and intelligent resource scheduling method is adopted. By real-time monitoring and quantitative analysis of the performance indicators of the target cluster in processing call data of various business types, and combining business priorities, the purpose of dynamically optimizing the configuration of the Kafka cluster is achieved, thereby achieving the technical effect of dynamically adjusting the resource configuration of the Kafka cluster according to the real-time and importance of the business, and thus solving the technical problem that the relevant technology cannot dynamically adjust the cluster resource configuration according to the needs of different businesses, affecting the overall performance and reliability of the system.

[0151] Figure 3 is a system architecture diagram of a method for determining an optimization strategy according to an embodiment of the present application, such as Figure 3 As shown, in some embodiments of the present application, the system may include:

[0152] Data input and weight configuration module 302: This module is responsible for business metadata management, dynamic weight adjustment, and business-Topic mapping, providing basic data support and policy guidance for the entire optimization process. By maintaining the connection information of the Kafka cluster, the relationship between business types and TOPICs, and adjusting the weight configuration according to real-time business fluctuations, it ensures that the optimization strategy can not only reflect the priority of business needs, but also adapt to real-time changes in business traffic, providing customized parameter support for subsequent performance evaluation and optimization.

[0153] Kafka Cluster System 304: This module, comprised of multiple target clusters (Kafka Clusters 1 through N), serves as the data processing core of the entire system architecture. Each cluster is responsible for handling call log data streams for different service types. Leveraging Kafka's high throughput and low latency, it supports large-scale, real-time data collection, transmission, and billing processing. Inter-cluster collaboration and resource allocation are key to ensuring the efficient and stable operation of the billing system.

[0154] Performance monitoring module 306: includes an indicator collector and an indicator database. The former obtains key performance data such as throughput, latency, resource utilization, etc. from the Kafka cluster system and business traffic in real time, while the latter is responsible for storing and managing this data. The efficient operation of this module ensures real-time monitoring of the Kafka cluster and provides accurate and timely data basis for subsequent performance analysis and optimization strategy formulation.

[0155] Performance analysis and scoring model module 308: Based on the collected performance indicators, a multi-level performance evaluation system is constructed through partition-level scoring, topic-level scoring, and cluster-level scoring. This module quantitatively evaluates the performance status of each component in the Kafka cluster, identifies performance bottlenecks and abnormal hotspots, and provides accurate scoring indexes and decision-making basis for the generation of optimization strategies.

[0156] Optimization Strategy and Control Module 310: This module is responsible for automatically generating strategy recommendations based on the output of the scoring model through the optimization recommendation generator. These recommendations cover various aspects, such as increasing the number of partitions, adjusting consumer parameters, and optimizing broker node configurations. It also provides strategy script generation and manual review / automatic execution, ensuring that optimization strategies can be generated and executed automatically while also receiving manual review and adjustment, balancing the needs of automation and human intervention.

[0157] Execution Environment 312: This module includes sandbox execution, validation, production execution, and automatic rollback control. This ensures the pre-implementation and verification of optimization strategies. By simulating execution and monitoring results in the sandbox environment, the risks associated with direct implementation in production are mitigated. Once an optimization strategy passes validation, it is securely deployed to the production environment. An automatic rollback mechanism quickly restores the system to its pre-optimization state if any issues arise during execution, ensuring business continuity and data integrity.

[0158] Figure 4 FIG. 1 is a schematic diagram of an optimization process of a method for determining an optimization strategy according to an embodiment of the present application, such as Figure 4 As shown, in some embodiments of the present application, the optimization process may include the following steps:

[0159] S402: Start, the starting point of the process, marks the start of the optimization strategy determination method.

[0160] S404: Initialize business metadata. The data input and weight configuration module loads and maintains the connection information of the Kafka cluster, the mapping relationship between business types and topics, and the business priority weights. The weights can be dynamically adjusted according to the real-time requirements of the business, providing business awareness capabilities for performance optimization strategies.

[0161] S406: Start performance monitoring and benchmarking. The performance monitoring module begins to collect performance indicators of the Kafka cluster in real time, such as throughput, latency, resource load, etc. At the same time, benchmarking is performed in a preset test environment to set performance thresholds, providing a quantitative basis for subsequent performance analysis and optimization strategies.

[0162] S408: Multi-level performance score calculation. The performance analysis and scoring model module calculates partition-level, topic-level, and cluster-level performance scores based on real-time monitoring data, builds a multi-level performance evaluation system, and identifies performance bottlenecks and abnormal hotspots in a quantitative manner.

[0163] S410: Triggering the optimization strategy. When the performance score exceeds the preset threshold, the optimization strategy and control module automatically generates an optimization strategy, such as increasing the number of partitions, adjusting consumer parameters, optimizing Broker load, etc., to deal with performance anomalies.

[0164] S412: Generate optimization suggestions. The optimization suggestion generator converts the optimization strategy into specific suggestions, including resource configuration adjustment, parameter optimization, etc., expressed in the form of scripts, and supports automated execution.

[0165] S414: Administrator review or automatic confirmation: Optimization suggestions are manually reviewed or automatically confirmed according to preset rules to ensure the security and applicability of the strategy and avoid the risks that may arise from blind execution.

[0166] S416: Execute the optimization script in the sandbox environment. The optimization strategy is first simulated and executed in the sandbox environment to pre-test the strategy effect and reduce the risk of direct implementation in the production environment.

[0167] S418: Verify the optimization effect, monitor the improvement of performance indicators in the sandbox environment, verify whether the optimization strategy is effective, and ensure that the strategy can achieve the expected performance improvement goals before implementation in the production environment.

[0168] S420: Determine whether the verification is successful. Based on the optimization verification results, determine whether the expected goal has been achieved and whether the verification pass conditions have been met. If the verification is successful, the process jumps to S422; if the verification fails, the process jumps to S434.

[0169] S422: Push to the production environment. After the optimization strategy is verified to be effective, it is ready to be pushed to the production environment to start the actual optimization execution process.

[0170] S424: Execute production optimization operations, implement optimization strategies in the production environment, adjust resource allocation, optimize system performance, and improve call record data processing efficiency.

[0171] S426: Determine whether an exception has occurred. After the optimization operation is executed, the system monitors the operating status of the production environment to check for any anomalies, ensuring that the execution of the optimization strategy does not negatively impact the production environment. If no anomalies have occurred, the process jumps to S432; if an anomaly has occurred, the process jumps to S428.

[0172] S428: Trigger automatic rollback. When a production optimization operation causes an exception, the rollback mechanism is automatically triggered to ensure that the system can quickly recover to the state before optimization, ensuring business continuity and data integrity.

[0173] S430: Restore the production configuration, perform a rollback operation, and restore the original production environment configuration to prevent the abnormal state from continuously affecting business processing.

[0174] S432: Record a success log. When the production optimization operation does not cause an exception, the system records the successful optimization operation log to provide historical data reference for subsequent performance analysis and optimization.

[0175] S434: Abort the launch. If the optimization effect verification in the sandbox environment fails, the process will be aborted to avoid deploying invalid or high-risk strategies to the production environment.

[0176] S436: Record the reasons for verification failure. For optimization strategies for verification failure, record the specific reasons for failure to analyze the problem and optimize subsequent strategy formulation.

[0177] S438: End, the final end point of the process. Whether the optimization is successful, the rollback is completed, or the launch is aborted, it will end here.

[0178] In order to facilitate understanding of the above-mentioned method for determining the optimization strategy, it is explained below with reference to some specific embodiments.

[0179] For example, the billing system has two Kafka clusters (Cluster A and B), which process real-time call records and offline billing data respectively. The configuration is as follows:

[0180] Cluster Topic Business Type Weight Number of Partitions Replication Factor Cluster A voice_call Real-time voice call records 0.9 6 3 Cluster A data_usage Near real-time traffic bills 0.7 4 2 Cluster B monthly_bill Offline monthly bill 0.3 8 2

[0181] Step 1: Performance indicator collection and threshold setting.

[0182] Data collection (real-time monitoring indicators) includes:

[0183] (1) Partition level: end-to-end latency, ISR latency, and message backlog size.

[0184] (2) Topic level: maximum partition score, consumption rate, and production rate.

[0185] (3) Cluster level: Broker load (comprehensive score of CPU, memory, and disk IO).

[0186] The threshold settings corresponding to the performance indicators (based on benchmark tests) can be:

[0187] End-to-end latency: real-time services (≤1 second), offline services (≤5 seconds); ISR latency: real-time services (≤5 seconds), offline services (≤10 seconds); message backlog: real-time services (≤100,000 messages), offline services (≤200,000 messages); Broker load threshold: comprehensive score ≤70.

[0188] Step 2: Partition-level performance index calculation, taking Topic voice_call Partition 3 of Cluster A as an example.

[0189] Index value:

[0190] (1) End-to-end delay: 1.5 seconds (threshold 1 second) → normalized value: 1.5 / 1 = 1.5.

[0191] (2) ISR delay: 7 seconds (threshold 5 seconds) → normalized value: 7 / 5 = 1.4.

[0192] (3) Message backlog: 80,000 (does not exceed the threshold and does not trigger a penalty).

[0193] Weight configuration (real-time business):

[0194] The end-to-end delay weight w1=0.6.

[0195] ISR delay weight w2=0.4.

[0196] Calculation formula: Score Partition =(0.6×1.5)+(0.4×1.4)=0.9+0.56=1.46 Conclusion: Score 1.46>1, marked as "hotspot partition".

[0197] Step 3: Calculate the topic-level performance index, taking the topic voice_call in Cluster A as an example.

[0198] (1) The highest score of Partition is max(1.46,1.2,0.9,1.1,0.8,1.3)=1.46.

[0199] (2) Message backlog: 120,000 (threshold 100,000) → Penalty term: α×12 / 10=0.3×1.2=0.36.

[0200] (3) ISR delay: 7 seconds (threshold 5 seconds) → penalty term: γ×7 / 5=0.2×1.4=0.28.

[0201] Calculation formula: ScoreTopic = 1.46 + 0.36 + 0.28 = 2.1 Conclusion: Score 2.1 > 1.5, triggering Topic-level optimization.

[0202] Step 4: Calculate the cluster-level performance index, taking Cluster A as an example.

[0203] Topic score and weight:

[0204] The voice_call score is 2.1, and the weight is 0.9 → 2.1 × 0.9 = 1.89.

[0205] data_usage score is 1.2, weight is 0.7 → 1.2×0.7=0.84.

[0206] Broker load: Maximum load 75 (threshold 70) → Penalty term: β×75 / 70=0.1×1.07=0.107.

[0207] Calculation formula: ScoreCluster = (1.89 + 0.84) + 0.107 = 2.73 + 0.107 = 2.837 Conclusion: The score of 2.837 is greater than 2, triggering cluster-level optimization.

[0208] Step 5: Optimize strategy triggering and execution.

[0209] (1) Partition-level optimization:

[0210] Problem: Partition 3 has an end-to-end latency of 1.5 seconds and an ISR latency of 7 seconds.

[0211] Strategy: Increase the number of partitions (from 6 to 12) to improve parallelism; migrate the leader to a low-load broker (load score 65).

[0212] (2) Topic-level optimization:

[0213] Problem: There is a backlog of 120,000 messages, and the consumption rate is lagging behind.

[0214] Strategy: Expand the number of Consumer Group instances (from 4 to 8).

[0215] (3) Cluster-level optimization:

[0216] Problem: Broker load is 75, and resources are tight.

[0217] Strategy: Horizontally expand the number of broker nodes (from 3 to 5); adjust num.replica.fetchers to 4 (originally 2) to improve replica synchronization efficiency.

[0218] Through real-time monitoring of multi-dimensional performance indicators and the construction of a multi-level performance scoring model, this application can quickly identify and locate performance bottlenecks that occur in large-scale call record data processing, including overloaded topics, partitions, and high-pressure cluster nodes. This precise positioning capability helps operation and maintenance personnel or automated systems take timely measures to eliminate performance obstacles, avoid delays and backlogs in call record data processing, and ensure the real-time and accuracy of billing data.

[0219] In addition, this application can intelligently analyze the real-time fluctuations of business load and dynamically adjust the resource configuration of the Kafka cluster based on the priorities of different business types. It allows the system to automatically increase the number of partitions and adjust the consumption concurrency when the amount of call data surges or the business priority changes, ensuring the efficient use of resources, avoiding resource waste and overload, and improving the flexibility and response speed of resource utilization.

[0220] In terms of improving billing processing efficiency and stability, this application is based on a performance score-driven policy recommendation mechanism, which can dynamically execute a series of optimization operations, automatically trigger the optimization process according to the performance index, and dynamically generate corresponding resource adjustment suggestions, including Partition expansion, Consumer concurrency adjustment, etc. It supports automated or semi-automated execution to ensure that the system can respond quickly and achieve adaptive optimization when performance bottlenecks occur. These operations directly target performance bottlenecks, effectively reduce end-to-end processing delays, and reduce the risk of backlog of billing data. Especially in high-concurrency, low-latency data processing scenarios, it can significantly improve the efficiency of call data collection, transmission and billing processing, and ensure the timely generation and accuracy of user bills.

[0221] In addition, this application deeply integrates Kafka's underlying performance indicators with business types, priorities, and real-time requirements to build a business-aware optimization decision support system. This allows the optimization strategy to not only focus on technical performance improvements, but also fully consider business characteristics and needs, ensuring the positive impact of optimization measures on the business and improving the accuracy and efficiency of optimization decisions.

[0222] Finally, this application associates different Kafka topics (such as voice call records, traffic call records, and monthly bills) with their corresponding processing timeliness levels based on a weight configuration model based on the real-time characteristics of billing services. By setting service weight parameters, resource scheduling priorities are dynamically adjusted to achieve differentiated configuration and refined management of Kafka cluster resources. By constructing a three-layer performance evaluation model at the Partition level, Topic level, and cluster level, and combining real-time performance indicators (throughput, latency, ISR synchronization, etc.), service weights, and resource load conditions, a multi-dimensional, quantifiable performance scoring system is formed. This model can identify system bottlenecks step by step from the micro to the macro level, and serve as a decision-making basis for intelligent optimization strategies.

[0223] Figure 5 is a structural diagram of a device for determining an optimization strategy according to an embodiment of the present application, such as Figure 5 As shown, the device includes:

[0224] An acquisition module 502 is configured to acquire multiple performance indicators of a target cluster, wherein the target cluster is configured to process call record data of various service types, and different service types correspond to different service priorities;

[0225] a determination module 504 configured to determine a target score based on the service priority and the performance indicator, wherein the target score includes at least one of the following: a partition score reflecting the performance status of a partition in the target cluster, a topic score reflecting the performance status of a topic in the target cluster, and a cluster score reflecting the overall performance status of the target cluster;

[0226] The optimization module 506 is configured to determine an optimization strategy corresponding to the target score, wherein the optimization strategy is used to adjust resource configuration of the target cluster.

[0227] It should be noted that Figure 5 The optimization strategy determination device shown is used to perform Figure 2 The optimization strategy is determined by Figure 2 The explanations of the optimization strategy determination method in the Figure 5 The determination device of the optimization strategy shown is not described here in detail.

[0228] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the steps of the method for determining the optimization strategy in each embodiment of the present application.

[0229] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the steps of the method for determining the optimization strategy in each embodiment of the present application by running the computer program.

[0230] An embodiment of the present application further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method for determining the optimization strategy in each embodiment of the present application.

[0231] An embodiment of the present application further provides a computer program, which, when executed by a processor, implements the steps of the method for determining the optimization strategy in each embodiment of the present application.

[0232] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0233] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0234] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0235] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0236] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0237] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0238] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for determining an optimization strategy, characterized in that: include: Acquire multiple performance indicators of a target cluster, where the target cluster is used to process call record data of multiple service types, and different service types correspond to different service priorities; Determining a target score based on the business priority and the performance indicator, wherein the target score includes at least one of the following: a partition score for reflecting the performance status of a partition in the target cluster, a topic score for reflecting the performance status of a topic in the target cluster, and a cluster score for reflecting the overall performance status of the target cluster; An optimization strategy corresponding to the target score is determined, wherein the optimization strategy is used to adjust resource configuration of the target cluster.

2. The method according to claim 1, characterized in that The target cluster includes multiple topics, each of the topics includes multiple partitions, wherein each of the topics corresponds to a business type; determining a target score based on the business priority and the performance indicator includes: Obtaining a first performance indicator corresponding to each partition in the target cluster and a performance indicator threshold corresponding to the first performance indicator, wherein the performance indicator threshold is used to represent a performance target of the first performance indicator; Determine a target topic corresponding to each partition, and determine a first weight of the first performance indicator according to a business priority corresponding to the target topic; determining a ratio between the first performance indicator and the performance indicator threshold, wherein the ratio is used to reflect a degree of deviation between the first performance indicator and an ideal state; A partition score in the target score is determined according to the ratio and the first weight.

3. The method according to claim 2, characterized in that The topic score is determined as follows: determining a target partition score from all partition scores of the target topic, wherein the target partition score includes a worst partition score; Obtain a second performance indicator and a third performance indicator corresponding to the target topic, wherein the second performance indicator is used to reflect the message processing capability of consumers subscribed to the target topic, and the third performance indicator is used to reflect the health status of data synchronization between multiple partitions in the target topic and the server nodes; Determining a first penalty coefficient and a second penalty coefficient corresponding to the second performance indicator and the third performance indicator, respectively, wherein the first penalty coefficient and the second penalty coefficient are determined according to a business type corresponding to the target topic; The topic score is determined based on the target partition score, the second performance indicator, the third performance indicator, the first penalty coefficient, and the second penalty coefficient.

4. The method according to claim 3, characterized in that The cluster score is determined as follows: Obtain a topic score set for all topics in the target cluster, and determine a second weight corresponding to each topic score, wherein the second weight is determined according to a business priority corresponding to the topic; Acquire a fourth performance indicator corresponding to the target cluster, wherein the fourth performance indicator is used to reflect the load status of the server nodes in the target cluster; Determining a third penalty coefficient corresponding to the fourth performance indicator, wherein the third penalty coefficient is determined according to a degree of resource scarcity of the target cluster; The cluster score is determined according to the topic score set, the second weight, the fourth performance indicator, and the third penalty coefficient.

5. The method according to claim 1, characterized in that In the case where the target score is a partition score, an optimization strategy corresponding to the target score is determined, including: when the partition score of the target partition is continuously greater than a first threshold within a preset time period and the resource utilization rate of the server node where the target partition is located is greater than the utilization rate threshold, a first optimization strategy is generated, wherein the first optimization strategy is used to adjust the partition resource configuration within the target cluster.

6. The method according to claim 5, characterized in that The first optimization strategy includes: Increase the number of partitions in the target cluster; or, A first server node in the target cluster whose load is less than a load threshold is obtained, and the first server node is determined to be a server node for processing write requests of the target partition.

7. The method according to claim 3, characterized in that In the case where the target score is a topic score, an optimization strategy corresponding to the target score is determined, including: when the topic score of the target topic is greater than a second threshold, the third performance indicator is greater than a time threshold, and the business priority corresponding to the target topic is greater than or equal to the priority threshold, a second optimization strategy is generated, wherein the second optimization strategy is used to adjust the topic resource configuration within the target cluster.

8. The method according to claim 7, characterized in that The second optimization strategy includes: Acquire a second server node in the target cluster whose network delay is less than a delay threshold, and process the message read and write operations of the target topic on the second server node; or, Increase the number of threads processing replica data synchronization on each server node in the target cluster.

9. The method according to claim 3, characterized in that In the case where the target score is a topic score, an optimization strategy corresponding to the target score is determined, including: when the second performance indicator is greater than the message backlog threshold, the continuous growth duration of the second performance indicator is greater than the duration threshold, and the difference between the message consumption rate and the message production rate of the target topic is greater than the difference threshold, a third optimization strategy is generated, wherein the third optimization strategy is used to adjust the topic resource configuration within the target cluster.

10. The method according to claim 9, characterized in that The third optimization strategy includes: Increase the number of consumers in the consumer group corresponding to the target topic, wherein the consumer group includes multiple consumers, and each consumer group jointly consumes messages from at least one topic; or Adjust the consumer parameters corresponding to the target topic, wherein the consumer parameters include the maximum number of bytes of data that the consumer requests from the server node or the upper limit of the time the consumer waits when there is not enough data; or Reduce the message production rate of the target topic.

11. The method according to claim 4, characterized in that The method further comprises: Obtain cluster scores corresponding to multiple target clusters respectively, and determine the health of each target cluster according to the cluster scores, wherein the health is used to quantitatively represent the performance status of the target cluster when processing the call record data; The multiple target clusters are sorted according to the health levels to obtain a sorting result, wherein the sorting result is used to determine an optimization order of the target clusters.

12. The method according to claim 1, characterized in that After determining the optimization strategy corresponding to the target score, the method further includes: Determining an execution script corresponding to the optimization strategy, and simulating the execution of the execution script in a sandbox cluster to obtain a simulated execution result, wherein the sandbox cluster is used to test the feasibility of the execution script; When the simulation execution result meets a preset condition, the execution script is executed in the target cluster.

13. A device for determining an optimization strategy, characterized in that: include: an acquisition module, configured to acquire multiple performance indicators of a target cluster, wherein the target cluster is configured to process call record data of multiple service types, and different service types correspond to different service priorities; a determination module, configured to determine a target score based on the service priority and the performance indicator, wherein the target score includes at least one of the following: a partition score for reflecting the performance status of a partition in the target cluster, a topic score for reflecting the performance status of a topic in the target cluster, and a cluster score for reflecting the overall performance status of the target cluster; An optimization module is used to determine an optimization strategy corresponding to the target score, wherein the optimization strategy is used to adjust the resource configuration of the target cluster.

14. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the method for determining the optimization strategy described in any one of claims 1 to 12.

15. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the method for determining the optimization strategy according to any one of claims 1 to 12 by running the computer program.

16. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method for determining the optimization strategy according to any one of claims 1 to 12 is implemented.

Images