Message consumption delay optimization method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides a message consumption delay optimization method and device, equipment, a storage medium and a product. According to the technical scheme provided by the embodiment of the invention, the first consumption delay information of the preset consumption group is determined, whether the theme expansion condition is met or not is determined according to the effective threshold value and the first consumption delay information, and when the theme expansion condition is met, the temporary data theme corresponding to the preset data theme is created; configuring a preset number of temporary partitions for the temporary data theme, and after the configuration of the preset number of temporary partitions for the temporary data theme is completed, redistributing cache messages consumed by a preset consumption group between the temporary partitions of the temporary data theme and the preset partitions of the preset data theme, the preset consumption group can consume the data in the temporary partition and the preset partition, so that the message processing efficiency is improved, the partition number of the theme and the limitation of the service capacity are effectively reduced, the change of a high-concurrency scene can be quickly responded, and the consumption delay processing effect is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technologies, and in particular, to a method, apparatus, device, storage medium, and product for optimizing message consumption latency. Background Art

[0002] With the rapid development of mobile communication technologies, the efficient transmission and processing of data have become indispensable core elements, and message queues are increasingly widely used in various services. In the field of big data processing, message queues (such as Kafka message queues) have been widely applied to real-time data consumption scenarios. However, when business data explodes or traffic surges, message queue systems often experience consumption latency due to the processing load exceeding the limit, resulting in message backlog problems.

[0003] Traditional latency processing solutions mainly rely on manual static scaling or adjusting the number of threads to solve the consumption latency problem. However, this method lacks flexibility and is limited by the number of partitions of a single topic and the capacity of the service itself. The amount of data backlogged in the topic quickly reaches the bottleneck of the service's ability to process messages, making it difficult to quickly respond to changes in high-concurrency scenarios. It is necessary to manually scale the topic to improve message processing efficiency. Otherwise, the queued messages need to wait for the previous messages to be processed before they can be processed, resulting in low message consumption efficiency, large message consumption latency, and poor consumption latency processing effects. Summary of the Invention

[0004] The embodiments of the present application provide a method, apparatus, device, storage medium, and product for optimizing message consumption latency to solve the technical problems in the related art that the latency processing solution is limited by the number of partitions of a single topic and the capacity of the service itself, resulting in low message consumption efficiency, difficulty in quickly responding to changes in high-concurrency scenarios, and poor consumption latency processing effects. It can effectively reduce the limitations of the number of partitions of the topic and the capacity of the service itself, adaptively adjust the message consumption efficiency, quickly respond to changes in high-concurrency scenarios, and improve the consumption latency processing effect.

[0005] In a first aspect, the embodiments of the present application provide a method for optimizing message consumption latency, including:

[0006] Determine the first consumption latency information of a preset consumption group, and determine whether the topic expansion condition is satisfied according to the effective threshold and the first consumption latency information. The effective threshold is determined according to the dynamic threshold and the preset reference threshold, and the dynamic threshold is determined according to the consumption rate at which the preset consumption group successfully consumes messages corresponding to a preset data topic and the preset tolerance time window;

[0007] In the case where the topic expansion condition is satisfied, create a temporary data topic corresponding to the preset data topic, and configure a preset number of temporary partitions for the temporary data topic;

[0008] When the preset number of temporary partitions are configured for the temporary data topic, re - allocate the cached messages consumed by the preset consumer group between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic for the preset consumer group to consume in the temporary partitions and the preset partitions.

[0009] In a second aspect, an embodiment of the present application provides a message consumption delay optimization device, including a delay monitoring module, an expansion processing module, and a data allocation module, where:

[0010] The delay monitoring module is configured to determine the first consumption delay information of a preset consumer group, and determine whether the topic expansion condition is met according to the effective threshold and the first consumption delay information. The effective threshold is determined according to the dynamic threshold and the preset reference threshold, and the dynamic threshold is determined according to the consumption rate of the preset consumer group successfully consuming the messages corresponding to the preset data topic and the preset tolerance time window;

[0011] The expansion processing module is configured to create a temporary data topic corresponding to the preset data topic and configure a preset number of temporary partitions for the temporary data topic when the topic expansion condition is met;

[0012] The data allocation module is configured to re - allocate the cached messages consumed by the preset consumer group between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic for the preset consumer group to consume in the temporary partitions and the preset partitions when the preset number of temporary partitions are configured for the temporary data topic.

[0013] In a third aspect, an embodiment of the present application provides a message consumption delay optimization device, including: a memory and one or more processors;

[0014] The memory is used to store one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the message consumption delay optimization method as described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a non - volatile storage medium storing computer - executable instructions, and the computer - executable instructions are used to execute the message consumption delay optimization method as described in the first aspect when executed by a computer processor.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device executes the message consumption delay optimization method as described in the first aspect.

[0018] In the embodiment of the present application, by determining the first consumption delay information of a preset consumption group, and determining whether the topic expansion condition is satisfied according to the effective threshold and the first consumption delay information. When the topic expansion condition is satisfied, a temporary data topic corresponding to the preset data topic is created, a preset number of temporary partitions are configured for the temporary data topic, and after completing the configuration of the preset number of temporary partitions for the temporary data topic, the cached messages consumed by the preset consumption group are redistributed between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic. The preset consumption group can consume data in the temporary partitions and the preset partitions, improving the processing efficiency of messages, effectively reducing the number of partitions of the topic and the limitation of the service's own capacity, adaptively adjusting the message consumption efficiency, quickly responding to changes in high-concurrency scenarios, and improving the consumption delay processing effect. Description of the Drawings

[0019] Figure 1 is a flowchart of a message consumption delay optimization method provided by an embodiment of the present application;

[0020] Figure 2 is a flowchart of another message consumption delay optimization method provided by an embodiment of the present application;

[0021] Figure 3 is a schematic structural diagram of a message consumption delay optimization device provided by an embodiment of the present application;

[0022] Figure 4 is a schematic structural diagram of a message consumption delay optimization device provided by an embodiment of the present application. Detailed Embodiments

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following further describes specific embodiments of this application in conjunction with the accompanying drawings. It can be understood that the specific embodiments described herein are only used to explain this application, rather than limiting this application. Additionally, it should be noted that for ease of description, only parts related to this application are shown in the drawings, rather than all content. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can be rearranged. When the operations are completed, the above process can be terminated, but there can also be additional steps not included in the drawings. The above process can correspond to a method, function, procedure, subroutine, subprogram, and so on.

[0024] The message consumption delay optimization method provided by this application can be applied to high-concurrency data consumption scenarios for message consumption delay optimization and resource management based on message queues such as Kafka, for example, distributed systems that have certain requirements for consumption speed and need to consume and process massive amounts of data in real time, such as financial trading systems, social media analysis platforms, live streaming platforms, and online advertising monitoring systems. The aim is to automatically divert data and manage resources when the consumption delay meets the topic expansion condition, improving the system's response efficiency and resource utilization rate.

[0025] Existing message consumption delay processing solutions mainly rely on manual static expansion or adjusting the number of threads to solve the consumption delay problem. However, this method lacks flexibility and is limited by the number of partitions of a single topic and the capacity of the service itself. It quickly reaches the bottleneck of the message queue's ability to process messages and is difficult to quickly respond to changes in high-concurrency scenarios. Based on this, a message consumption delay optimization method for the embodiments of this application is provided to solve the technical problems that existing delay processing solutions are limited by the number of partitions of a single topic and the capacity of the service itself, resulting in low message consumption efficiency, difficulty in quickly responding to changes in high-concurrency scenarios, and poor consumption delay processing effects.

[0026] Figure 1 A flowchart of a message consumption delay optimization method provided by an embodiment of this application is given. The message consumption delay optimization method provided by the embodiments of this application can be executed by a message consumption delay optimization device, which can be implemented in hardware and / or software and integrated in a message consumption delay optimization device.

[0027] The following describes the message consumption delay optimization method executed by the message consumption delay optimization device as an example.

[0028] Refer to Figure 1 , the message consumption delay optimization method includes:

[0029] S110: Determine the first consumption delay information of a preset consumption group, and determine whether the topic expansion condition is satisfied according to the effective threshold and the first consumption delay information. The effective threshold is determined according to the dynamic threshold and the preset reference threshold, and the dynamic threshold is determined according to the consumption rate at which the preset consumption group successfully consumes messages corresponding to the preset data topic and the preset tolerance time window.

[0030] The message queue provided in this application is a container for storing messages during the message transmission process, which can provide efficient routing and ensure message delivery. The message queue is configured with corresponding topics, and each topic can be configured with one or more partitions. Producers can save the produced messages to the partitions of the topic, and consumers can subscribe to the topic and consume messages from the partitions of the topic. Optionally, the message consumption delay optimization method provided in this application can be configured in a preset configuration center (such as Apollo configuration center), the monitoring function of the preset configuration center can be flexibly enabled, and the configuration of the preset configuration center can be dynamically adjusted according to actual needs. Different business parties (such as consumers) can also configure corresponding preset reference thresholds and preset tolerance time windows through the preset configuration center. After enabling the preset configuration center, the preset configuration center will automatically monitor the consumption delay situation of each business module (such as consumption group group) (such as including the consumption delay information of each consumption group) to achieve message shunting and dynamic resource expansion, ensure the stable operation of the business in high-concurrency scenarios, endow the system with almost unlimited expansion capabilities, adapt to the rapid growth of business needs, and effectively improve data processing efficiency and resource utilization.

[0031] In one embodiment, for each preset consumption group (group), the dynamic threshold can be determined according to the consumption rate at which the preset consumption group successfully consumes messages corresponding to the preset data topic and the preset tolerance time window, and the effective threshold can be determined according to the dynamic threshold and the preset reference threshold. Among them, the preset data topic is the topic subscribed by the preset consumption group, and the preset consumption group obtains messages from the preset partition corresponding to the preset data topic for consumption.

[0032] Optionally, the consumption rate query interface of the message queue can be used to determine and record the consumption rate at which the preset consumption group successfully consumes messages corresponding to the preset data topic. For example, the corresponding consumption rate can be obtained by using the Metrics interface of Kafka Consumer (such as records-consumed-rate), that is, the number of messages successfully consumed by the system per second. Through continuous collection and analysis of the consumption rate, the average consumption rate of a preset time period can be calculated. Optionally, the data related to the message consumption delay optimization process can be saved to a preset database (such as Codis database) for use as a judgment basis for triggering alarms and scaling operations in subsequent message consumption delay optimization logic.

[0033] In a possible embodiment, the message consumption delay optimization method provided by this application determines a dynamic threshold according to the consumption rate at which a preset consumption group successfully consumes messages corresponding to a preset data topic and a preset tolerance time window. It can be: determining the consumption rate at which a preset consumption group successfully consumes messages corresponding to a preset data topic, multiplying the consumption rate, the preset tolerance time window (Duration), and a preset safety factor (such as 1.2), and determining the multiplication result as the dynamic threshold. Among them, the consumption rate can be the average consumption rate of the preset consumption group within a preset time period (such as the previous day), that is, the current consumption rate = the average consumption rate of the preset time period × the preset tolerance time window × the preset safety factor. This application accurately determines the current consumption rate according to the consumption rate, the preset tolerance time window, and the preset safety factor, dynamically calculates the threshold configuration suitable for the current situation, triggers an alarm in a timely manner, and automatically performs scaling operations as needed to ensure that the system is always in an optimal state.

[0034] In a possible embodiment, the message consumption delay optimization method provided by this application determines an effective threshold according to the dynamic threshold and a preset reference threshold. It can be to determine the minimum value of the dynamic threshold and the preset reference threshold as the effective threshold, that is, the effective threshold = min(dynamic threshold, preset reference threshold). By dynamically determining the effective threshold, the message consumption delay is flexibly optimized, and the consumption delay processing effect is improved.

[0035] Exemplarily, for each preset consumption group, determine the first consumption delay information of the preset consumption group, and determine whether the topic expansion condition is met according to the effective threshold and the first consumption delay information. Optionally, meeting the topic expansion condition can be that the first consumption delay information reaches the effective threshold, or that within the preset tolerance time window, the proportion of the first consumption delay information reaching the dynamic threshold reaches a preset proportion threshold, or that within the preset tolerance time window, the first consumption delay information continuously reaches the dynamic threshold. This application accurately determines whether the topic expansion condition is met according to the comparison result between the first consumption delay information and the dynamic threshold, flexibly determines the timing of expanding the topic, adaptively adjusts the message consumption efficiency, and effectively improves the consumption delay processing effect.

[0036] S120: When the topic expansion condition is met, create a temporary data topic corresponding to the preset data topic, and configure a preset number of temporary partitions for the temporary data topic.

[0037] Exemplarily, when the condition for expanding the topic is met, a temporary data topic corresponding to the preset data topic is created, and a preset number of temporary partitions are configured for the newly created temporary data topic. Herein, the preset number can be the same as the preset number of partitions of the preset data topic, or can be less than the preset number of partitions of the preset data topic. For example, the number of partitions for each topic in the kafka message queue can be limited within 6. Six temporary partitions can be configured for the newly created temporary data topic, or six temporary partitions can be configured for the newly created temporary data topic. Subsequently, it can be determined whether to continue adding new temporary partitions to the newly created temporary data topic according to the consumption delay information.

[0038] S130: When the configuration of the preset number of temporary partitions for the temporary data topic is completed, the cached messages consumed by the preset consumer group are redistributed between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic for the preset consumer group to consume in the temporary partitions and the preset partitions.

[0039] Exemplarily, after the configuration of the preset number of temporary partitions for the temporary data topic is completed, the cached messages consumed by the preset consumer group can be redistributed between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic. Among them, part of the cached messages will be retained in the original preset partitions and processed in the original logic, and part of the cached messages will be redistributed to the temporary partitions to achieve horizontal expansion of the consumption capacity.

[0040] In one embodiment, after the redistribution of the cached messages is completed, the preset consumer group can obtain messages in the temporary partitions and the preset partitions for consumption. Optionally, the consumption delay information of the preset consumer group can be continuously monitored, and it can be continuously determined whether the condition for expanding the topic is met according to the effective threshold and the consumption delay information. When the condition for expanding the topic is met again, a temporary data topic corresponding to the preset data topic is continued to be created, a preset number of temporary partitions are configured for the temporary data topic, and the cached messages consumed by the preset consumer group are redistributed again between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic to achieve flexible and efficient dynamic expansion of the topic and continuously cope with and alleviate the delay problem.

[0041] As described above, by determining the first consumption delay information of a preset consumption group and determining whether the topic expansion condition is met based on the effective threshold and the first consumption delay information, when the topic expansion condition is met, a temporary data topic corresponding to the preset data topic is created, a preset number of temporary partitions are configured for the temporary data topic, and after the preset number of temporary partitions are configured for the temporary data topic, the cached messages consumed by the preset consumption group are redistributed between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic. The preset consumption group can consume data in the temporary partitions and the preset partitions, improving the message processing efficiency, effectively reducing the number of partitions of the topic and the limitation of the service's own capacity, adaptively adjusting the message consumption efficiency, quickly responding to changes in high-concurrency scenarios, and improving the consumption delay processing effect.

[0042] Based on the above embodiments, Figure 2 The flowchart of another message consumption delay optimization method provided by the embodiment of the present application is given. This message consumption delay optimization method is a specific implementation of the above message consumption delay optimization method.

[0043] Refer to Figure 2 , this message consumption delay optimization method includes:

[0044] S210: Determine the first consumption delay information of a preset consumption group, and determine whether the topic expansion condition is met based on the effective threshold and the first consumption delay information. The effective threshold is determined based on the dynamic threshold and the preset reference threshold, and the dynamic threshold is determined based on the consumption rate at which the preset consumption group successfully consumes the messages corresponding to the preset data topic and the preset tolerance time window.

[0045] S220: When the topic expansion condition is met, create a temporary data topic corresponding to the preset data topic, and configure a first number of temporary partitions for the temporary data topic. The first number is determined based on the preset partition number of the preset data topic corresponding to a preset ratio.

[0046] Exemplarily, when the first consumption delay information meets the topic expansion condition, a temporary data topic corresponding to the preset data topic is created, and a first number of temporary partitions are configured for the temporary data topic. Among them, the first number can be determined based on the preset partition number of the preset data topic (such as a preset percentage of the preset partition number), or the upper limit of the preset partition number corresponding to the preset data topic (such as a preset percentage of the upper limit of the preset partition number).

[0047] For example, when the upper limit of the number of preset partitions corresponding to the preset data topic is 6, the first quantity can be set to 3. That is, after creating the temporary data topic, 3 temporary partitions are configured for the temporary data topic. When the first consumption delay information meets the topic expansion condition, this application expands a new temporary data topic and gradually configures temporary partitions for the temporary data topic in the range of the first quantity, alleviating the data processing pressure by gradually diverting, and improving the data processing ability of the message queue.

[0048] S230: In the case where the preset number of temporary partitions is configured for the temporary data topic, re - allocate the cached messages consumed by the preset consumer group between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic for the preset consumer group to consume in the temporary partitions and the preset partitions.

[0049] In a possible embodiment, for the message consumption delay optimization method provided by this application to re - allocate the cached messages consumed by the preset consumer group between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic, it may be: determine the data allocation ratio according to the number of temporary partitions of the temporary data topic and the number of preset partitions of the preset data topic; re - allocate the cached messages consumed by the preset consumer group between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic according to the data allocation ratio.

[0050] Exemplarily, after configuring the preset number of temporary partitions for the temporary data topic, determine the number of temporary partitions of the temporary data topic and the number of preset partitions of the preset data topic, and determine the data allocation ratio according to the proportions of the number of temporary partitions and the number of preset partitions in the total number of partitions, and re - allocate the cached messages consumed by the preset consumer group between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic according to the data allocation ratio. The preset consumer group can consume in the temporary partitions and the preset partitions. Among them, the data allocation ratio can be understood as the ratio of re - allocating information to the temporary partitions corresponding to the temporary data topic.

[0051] For example, when it is detected that the first consumption delay information of a certain preset consumption group (denoted as the original group) exceeds the effective threshold within the preset tolerance time window for a continuous period, it is considered that the topic expansion condition is met, and message consumption delay optimization is automatically performed. That is, a new temporary data topic is dynamically created according to the currently backlogged preset data topic (denoted as the original topic) to relieve the consumption pressure. The naming of the new temporary data topic can follow the format of "{original topic}_{original group}_tmp_sequence number", such as "{original topic}_{original group}_tmp_1", and six temporary partitions can be configured to facilitate parallel processing and load balancing. Subsequently, the cached messages consumed by the original consumption group (original group) from the preset data topic can be redistributed. A part of the cached messages will be pushed to this newly created temporary data topic, and the remaining cached messages will still be processed in the original logic. The data distribution ratio of the redistributed cached messages can be calculated according to the following formula: the number of preset partitions of the preset data topic / (the total number of temporary partitions of the temporary data topic + the number of preset partitions of the preset data topic). Taking the example of creating a temporary data topic with 6 temporary partitions, according to the above formula, approximately 50% (i.e., 6 / (6 + 6) = 1 / 2) of the cached messages will be redirected to the new temporary data topic to achieve horizontal expansion of the consumption capacity.

[0052] In a possible embodiment, the redistribution of the cached messages consumed by the preset consumption group between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic can be performed by using the round-robin distribution method and / or the hash partitioning strategy. For example, for the round-robin distribution method, the temporary partitions of the temporary data topic and the preset partitions of the preset data topic are polled in sequence to redistribute the cached messages consumed by the preset consumption group among multiple temporary partitions and preset partitions, ensuring that the messages can be evenly distributed into each partition and maximizing the parallel processing ability of the message queue. When the cached messages consumed by the preset consumption group include a preset key (preset key value), the hash partitioning strategy is used to redistribute the cached messages consumed by the preset consumption group between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic, that is, the final landing point of the cached message is determined according to the hash value of the preset key, which can optimize the distribution balance of the data while maintaining the order of the messages.

[0053] S240: Determine the second consumption delay information of the preset consumption group, and determine whether the partition expansion condition is met according to the effective threshold and the second consumption delay information.

[0054] S250: When the partition expansion condition is met, add temporary partitions to the temporary data topic, and redistribute the cached messages consumed by the preset consumption group among the temporary partitions of multiple temporary data topics and the preset partitions of the preset data topic.

[0055] Exemplarily, after configuring a preset number of temporary partitions for the temporary data topic, continue to monitor the consumption latency of the preset consumer group to determine the second consumption latency information of the preset consumer group, and determine whether the partition expansion condition is met according to the effective threshold and the second consumption latency information.

[0056] Optionally, meeting the partition expansion condition can be that the second consumption latency information reaches the effective threshold, or that within a preset tolerance time window, the proportion of the second consumption latency information reaching the dynamic threshold reaches a preset proportion threshold, or that within a preset tolerance time window, the second consumption latency information continuously reaches the dynamic threshold.

[0057] In one embodiment, when the second consumption latency information meets the partition expansion condition, new temporary partitions are added to the temporary data topic, and the cached messages consumed by the preset consumer group are reallocated between the temporary partitions of multiple temporary data topics and the preset partitions of the preset data topic. Among them, some cached messages will be retained in the original preset partitions and original temporary partitions and processed in the original logic, and some cached messages will be reallocated to the newly configured temporary partitions. The preset consumer group can consume in each temporary partition and preset partition, realizing the horizontal expansion of consumption capacity.

[0058] In one embodiment, when the number of temporary partitions of the temporary data topic does not reach the upper limit of the number of partitions of the topic, the second consumption latency information of the preset consumer group can continue to be determined, and when the second consumption latency information meets the partition expansion condition, new temporary partitions are continuously added to the temporary data topic until the number of temporary partitions of the temporary data topic reaches the upper limit of the number of partitions of the topic. Subsequently, when the topic expansion condition is met, a new temporary data topic is expanded. For example, when creating a new temporary data topic, new nodes equivalent to half of the number of partitions of the temporary data topic are default-expanded to accelerate the data processing capacity. If the consumption latency alarm persists within a preset tolerance time window period after the creation of the temporary data topic, it can be further expanded, and new nodes equivalent to half of the current number of partitions are added again until the expansion upper limit is reached. This upper limit value is set to the sum of the total number of partitions of the temporary data topic and the number of partitions of the preset data topic, ensuring that system resources are maximally utilized within a reasonable range, realizing almost infinite dynamic expansion of topics and partitions, and continuously coping with and alleviating the latency problem.

[0059] S260: When the cached messages of the temporary data topic reach the preset capacity upper limit, create a temporary data topic corresponding to the preset data topic, and reallocate the cached messages consumed by the preset consumer group between the temporary partitions of multiple temporary data topics and the preset partitions of the preset data topic.

[0060] In a possible embodiment, after configuring a preset number of temporary partitions for the temporary data topic, the message consumption delay optimization method provided by this application can also monitor in real time whether the cached messages of the temporary data topic reach the preset capacity upper limit of the temporary data topic.

[0061] When the cached messages of the temporary data topic reach the preset capacity upper limit, continue to create a corresponding temporary data topic for the preset data topic, and re - allocate the cached messages consumed by the preset consumer group between the temporary partitions of multiple temporary data topics and the preset partitions of the preset data topic. At this time, some cached messages will be retained in the original preset partitions and temporary partitions and processed in the original logic, and some cached messages will be re - allocated to the newly created temporary partitions. The preset consumer group can consume in each temporary partition and preset partition, realizing the horizontal expansion of consumption capacity. This application continuously monitors the delay situation of the cached messages of the temporary data topic. Once it is detected that the consumption delay meets the topic expansion condition, a new temporary data topic will be automatically created immediately for data shunting, which can reduce the manual intervention in message consumption delay optimization and greatly improve the automation level and response speed of message consumption delay optimization.

[0062] In a possible embodiment, after re - allocating the cached messages consumed by the preset consumer group between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic, the message consumption delay optimization method provided by this application further includes: determining the third consumption delay information of the preset consumer group, and determining whether the shrinkage condition is met according to the effective threshold and the third consumption delay information; in the case where the shrinkage condition is met, mark a preset number of temporary data topics as to - be - deleted status, and stop writing data to the temporary data topics marked as to - be - deleted status; continuously monitor the temporary data topics marked as to - be - deleted status, and when the internal data consumption of the temporary data topics marked as to - be - deleted status is completed, perform deletion processing on the corresponding temporary partitions of the temporary data topics marked as to - be - deleted status; after completing the deletion processing on each temporary partition corresponding to the temporary data topics marked as to - be - deleted status and in the case where no consumer listens to the temporary data topics marked as to - be - deleted status, delete the temporary data topics marked as to - be - deleted status.

[0063] Exemplarily, after reallocating cached messages between the temporary partition and the preset partition, continuously determine the third consumption delay information of the preset consumption group, and determine whether the capacity reduction condition is met according to the effective threshold and the third consumption delay information. Optionally, meeting the capacity reduction condition may be that the third consumption delay information is less than the effective threshold, or that within a preset tolerance time window or within a preset number of preset tolerance time windows, the proportion of the third consumption delay information less than the dynamic threshold reaches a preset proportion threshold, or that within a preset tolerance time window or within a preset number of preset tolerance time windows, the third consumption delay information is continuously less than the dynamic threshold.

[0064] In one embodiment, when the capacity reduction condition is met, mark a preset number of temporary data topics as to-be-deleted status, and stop writing data to the temporary data topics marked as to-be-deleted status. For example, mark the newly created temporary data topic as to-be-deleted status, and stop writing data to the temporary data topic marked as to-be-deleted status. Subsequently, continuously monitor the temporary data topics marked as to-be-deleted status, and wait for the internal data consumption of the temporary data topics marked as to-be-deleted status to be completed.

[0065] In one embodiment, when the internal data consumption of the temporary data topics marked as to-be-deleted status is completed, perform deletion processing on some or all of the corresponding temporary partitions of the temporary data topics marked as to-be-deleted status. Optionally, after performing deletion processing on some of the corresponding temporary partitions of the temporary data topics marked as to-be-deleted status, continue to monitor whether the third consumption delay information of the preset consumption group meets the capacity reduction condition. If the capacity reduction condition is met, continue to mark the corresponding temporary data topic as to-be-deleted status, and when the internal data consumption of the temporary data topic marked as to-be-deleted status is completed, continue to perform deletion processing on the corresponding temporary partitions of the temporary data topic marked as to-be-deleted status until all the corresponding temporary partitions of the temporary data topic are deleted. If the consumption delay information of the preset consumption group monitored in real time meets the topic expansion condition or the partition expansion condition, continue to re-execute the creation of the temporary data topic or the temporary partition to achieve dynamic expansion and contraction of resources.

[0066] After completing the deletion processing of each temporary partition corresponding to the temporary data topic marked as to-be-deleted status and there is no consumer listening to the temporary data topic marked as to-be-deleted status, the temporary data topic marked as to-be-deleted status can be deleted to achieve resource recovery after the consumption pressure is relieved.

[0067] For example, continuously monitor the third consumption delay information of a preset consumption group. When the third consumption delay information is determined to meet the condition for capacity reduction (for example, no delay alarm is triggered within two consecutive preset tolerance time window cycles), resource optimization measures can be taken, such as pausing writing new data to the newly created temporary data topic and marking it as a to-be-deleted state. Subsequently, continuously listen and wait for the internal data consumption of the temporary data topic to complete. After confirming that all data has been properly processed, automatically reduce the service nodes in batches until the final number of nodes = (the total number of temporary partitions of the surviving temporary data topics + the preset number of partitions of the preset data topic). After the capacity reduction is completed, the corresponding preset consumption group can be removed. If it is confirmed that there are no consumers listening to the temporary data topic, the temporary data topic can be removed from the message queue. This process will be periodically repeated. Every time a preset number of preset tolerance time window cycles pass, unneeded temporary data topics will be re-evaluated and removed, and corresponding service node capacity reduction will be performed simultaneously to achieve efficient management and dynamic optimization of resources. This application realizes the management of message queue resources by continuously monitoring the status of all temporary data topics and service nodes. When the delay problem has not been solved, the topic or partition can be expanded according to actual needs. When the delay problem is solved, the service nodes can be automatically reduced, and useless temporary data topics and consumption groups can be cleaned up to accurately release redundant resources, thereby achieving efficient utilization of system resources and effective control of costs.

[0068] As described above, by determining the first consumption delay information of a preset consumption group, and determining whether the topic expansion condition is met according to the effective threshold and the first consumption delay information, when the topic expansion condition is met, a temporary data topic corresponding to the preset data topic is created, a preset number of temporary partitions are configured for the temporary data topic, and after completing the configuration of the preset number of temporary partitions for the temporary data topic, the cached messages consumed by the preset consumption group are redistributed between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic. The preset consumption group can consume data in the temporary partitions and the preset partitions, improving the message processing efficiency, effectively reducing the number of partitions of the topic and the limitation of the service's own capacity, adaptively adjusting the message consumption efficiency, quickly responding to changes in high-concurrency scenarios, and improving the consumption delay processing effect. At the same time, a message dynamic shunting mechanism is also introduced. When it is detected that the consumption delay exceeds the effective threshold, multiple temporary data topics will be automatically generated and activated in sequence. The original data will be evenly and efficiently dispersed to these temporary data topics in the original service node, and the original data distribution strategy (including hash distribution and polling push) will be retained to ensure the diversity and balance of data processing. It is also possible to comprehensively manage and intelligently schedule all temporary data topics and their derived service nodes. During the duration of the delay alarm, it can autonomously make decisions and execute expansion operations, including creating new temporary data topics to further disperse the load. When the delay returns to normal, it will automatically execute the shrinkage process, clean up the consumption groups and temporary data topics that are no longer needed, effectively recycle and reallocate resources, not only quickly respond when the consumption delay is too high and relieve the pressure through data shunting, but also automatically optimize the resource configuration when the consumption pressure is reduced, significantly improving the system's response speed, resource utilization efficiency, and overall performance.

[0069] Figure 3 It is a schematic structural diagram of a message consumption delay optimization device provided by an embodiment of the present application. Refer to Figure 3 , the message consumption delay optimization device includes a delay monitoring module 31, an expansion processing module 32, and a data distribution module 33. Among them, the delay monitoring module 31 is configured to determine the first consumption delay information of a preset consumption group, and determine whether the topic expansion condition is met according to the effective threshold and the first consumption delay information. The effective threshold is determined according to the dynamic threshold and the preset reference threshold, and the dynamic threshold is determined according to the consumption rate of the preset consumption group successfully consuming the messages corresponding to the preset data topic and the preset tolerance time window; the expansion processing module 32 is configured to create a temporary data topic corresponding to the preset data topic and configure a preset number of temporary partitions for the temporary data topic when the topic expansion condition is met; the data distribution module 33 is configured to redistribute the cached messages consumed by the preset consumption group between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic when the configuration of the preset number of temporary partitions for the temporary data topic is completed, for the preset consumption group to consume in the temporary partitions and the preset partitions.

[0070] As described above, by determining the first consumption delay information of the preset consumption group and determining whether the theme expansion condition is satisfied according to the effective threshold and the first consumption delay information, when the theme expansion condition is satisfied, a temporary data theme corresponding to the preset data theme is created, a preset number of temporary partitions are configured for the temporary data theme, and after completing the configuration of the preset number of temporary partitions for the temporary data theme, the cached messages consumed by the preset consumption group are redistributed between the temporary partitions of the temporary data theme and the preset partitions of the preset data theme. The preset consumption group can consume data in the temporary partitions and the preset partitions, improving the message processing efficiency, effectively reducing the number of partitions of the theme and the limitation of the service's own capacity, adaptively adjusting the message consumption efficiency, quickly responding to changes in high-concurrency scenarios, and improving the consumption delay processing effect.

[0071] In a possible embodiment, satisfying the theme expansion condition includes: within a preset tolerance time window, the first consumption delay information continuously reaches the dynamic threshold.

[0072] In a possible embodiment, the data distribution module 33 redistributes the cached messages consumed by the preset consumption group between the temporary partitions of the temporary data theme and the preset partitions of the preset data theme, configured as:

[0073] Determine the data distribution ratio according to the number of temporary partitions of the temporary data theme and the number of preset partitions of the preset data theme;

[0074] Redistribute the cached messages consumed by the preset consumption group between the temporary partitions of the temporary data theme and the preset partitions of the preset data theme according to the data distribution ratio.

[0075] In a possible embodiment, the data distribution module 33 redistributes the cached messages consumed by the preset consumption group between the temporary partitions of the temporary data theme and the preset partitions of the preset data theme, configured as:

[0076] Adopt a polling distribution method to redistribute the cached messages consumed by the preset consumption group between the temporary partitions of the temporary data theme and the preset partitions of the preset data theme; and / or

[0077] When the cached messages consumed by the preset consumption group include a preset key, adopt a hash partitioning strategy to redistribute the cached messages consumed by the preset consumption group between the temporary partitions of the temporary data theme and the preset partitions of the preset data theme.

[0078] In a possible embodiment, the expansion processing module 32 configures a preset number of temporary partitions for the temporary data theme, configured as:

[0079] Configure a first number of temporary partitions for the temporary data topic, where the first number is determined according to the preset number of partitions corresponding to the preset data topic in a preset ratio;

[0080] The expansion processing module 32 is further configured to: determine the second consumption delay information of the preset consumer group, and determine whether the partition expansion condition is met according to the effective threshold and the second consumption delay information;

[0081] When the partition expansion condition is met, add temporary partitions to the temporary data topic, and re - allocate the cached messages consumed by the preset consumer group between the temporary partitions of multiple temporary data topics and the preset partitions of the preset data topic.

[0082] In a possible embodiment, the expansion processing module 32 is further configured to: when the cached messages of the temporary data topic reach the preset capacity upper limit, create a temporary data topic corresponding to the preset data topic, and re - allocate the cached messages consumed by the preset consumer group between the temporary partitions of multiple temporary data topics and the preset partitions of the preset data topic.

[0083] In a possible embodiment, the message consumption delay optimization device further includes a shrinkage processing module, and the shrinkage processing module is configured to:

[0084] Determine the third consumption delay information of the preset consumer group, and determine whether the shrinkage condition is met according to the effective threshold and the third consumption delay information;

[0085] When the shrinkage condition is met, mark a preset number of temporary data topics as to - be - deleted status, and stop writing data to the temporary data topics marked as to - be - deleted status;

[0086] Continuously monitor the temporary data topics marked as to - be - deleted status. When the internal data consumption of the temporary data topics marked as to - be - deleted status is completed, perform deletion processing on the temporary partitions corresponding to the temporary data topics marked as to - be - deleted status;

[0087] After completing the deletion processing of each temporary partition corresponding to the temporary data topic marked as to - be - deleted status, and when there is no consumer listening to the temporary data topic marked as to - be - deleted status, delete the temporary data topic marked as to - be - deleted status.

[0088] In a possible embodiment, the message consumption delay optimization device further includes a threshold determination module, and the threshold determination module is configured to determine the effective threshold according to the dynamic threshold and the preset reference threshold.

[0089] In a possible embodiment, the threshold determination module is configured to determine the effective threshold according to the dynamic threshold and the preset reference threshold, and is configured to: determine the minimum value between the dynamic threshold and the preset reference threshold as the effective threshold.

[0090] In a possible embodiment, the threshold determination module is further configured to determine a dynamic threshold according to the consumption rate at which a preset consumption group successfully consumes messages corresponding to a preset data topic and a preset tolerance time window.

[0091] In a possible embodiment, the threshold determination module determines a dynamic threshold according to the consumption rate at which a preset consumption group successfully consumes messages corresponding to a preset data topic and a preset tolerance time window, and is configured as follows:

[0092] Determine the consumption rate at which a preset consumption group successfully consumes messages corresponding to a preset data topic;

[0093] Multiply the consumption rate, the preset tolerance time window, and a preset safety factor, and determine the multiplication result as the dynamic threshold.

[0094] It should be noted that in the embodiments of the above message consumption delay optimization device, the included units and modules are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the embodiments of the present application.

[0095] The embodiments of the present application further provide a message consumption delay optimization device, and this message consumption delay optimization device can integrate the message consumption delay optimization device provided in the embodiments of the present application. Figure 4 It is a schematic structural diagram of a message consumption delay optimization device provided by the embodiments of the present application. Refer to Figure 4 , this message consumption delay optimization device includes: an input device 43, an output device 44, a memory 42, and one or more processors 41; the memory 42 is used to store one or more programs; when the one or more programs are executed by the one or more processors 41, the one or more processors 41 implement the message consumption delay optimization method provided in the above embodiments. The above-provided message consumption delay optimization device, device, and computer can be used to execute the message consumption delay optimization method provided in any of the above embodiments, and have the corresponding functions and beneficial effects.

[0096] The embodiments of the present application further provide a non-volatile storage medium storing computer-executable instructions, and the computer-executable instructions are used to execute the message consumption delay optimization method provided in the above embodiments when being executed by a computer processor. Of course, for the non-volatile storage medium storing computer-executable instructions provided by the embodiments of the present application, the computer-executable instructions are not limited to the message consumption delay optimization method provided above, and can also execute related operations in the message consumption delay optimization method provided by any embodiment of the present application. The message consumption delay optimization device, equipment and storage medium provided in the above embodiments can execute the message consumption delay optimization method provided by any embodiment of the present application. For technical details not described in detail in the above embodiments, reference can be made to the message consumption delay optimization method provided by any embodiment of the present application.

[0097] Based on the above embodiments, the embodiments of the present application further provide a computer program product. Essentially, or the part that contributes to the prior art, or all or part of the technical solution of the present application can be embodied in the form of a software product. The computer program product is stored in a storage medium and includes several instructions for causing a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the message consumption delay optimization method provided by each embodiment of the present application.

Claims

1. A message consumption delay optimization method, characterized in that: include: Determine the first consumption delay information of the preset consumer group, and determine whether the topic expansion condition is met according to the effective threshold and the first consumption delay information, wherein the effective threshold is determined according to the dynamic threshold and the preset reference threshold, and the dynamic threshold is determined according to the consumption rate of the message corresponding to the preset data topic successfully consumed by the preset consumer group and the preset tolerance time window; When the topic expansion condition is met, a temporary data topic corresponding to the preset data topic is created, and a preset number of temporary partitions are configured for the temporary data topic; When a preset number of temporary partitions are configured for the temporary data topic, cached messages consumed by the preset consumer group are redistributed between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic, so that the preset consumer group can consume the temporary partitions and the preset partitions.

2. The message consumption delay optimization method according to claim 1, characterized in that: The subject expansion condition is satisfied, including: Within the preset tolerance time window, the first consumption delay information continues to reach the dynamic threshold.

3. The message consumption delay optimization method according to claim 1, characterized in that: The reallocating the cached messages consumed by the preset consumer group between the temporary partition of the temporary data topic and the preset partition of the preset data topic includes: Determining a data allocation ratio according to the number of temporary partitions of the temporary data topic and the number of preset partitions of the preset data topic; The cached messages consumed by the preset consumer group are reallocated between the temporary partition of the temporary data topic and the preset partition of the preset data topic according to the data distribution ratio.

4. The message consumption delay optimization method according to claim 1, characterized in that: The reallocating the cached messages consumed by the preset consumer group between the temporary partition of the temporary data topic and the preset partition of the preset data topic includes: Redistributing cached messages consumed by the preset consumer group between the temporary partition of the temporary data topic and the preset partition of the preset data topic in a round-robin manner; and / or In the case where the cached messages consumed by the preset consumer group include a preset key, a hash partitioning strategy is adopted to redistribute the cached messages consumed by the preset consumer group between the temporary partition of the temporary data topic and the preset partition of the preset data topic.

5. The message consumption delay optimization method according to claim 1, characterized in that: The configuring a preset number of temporary partitions for the temporary data topic includes: Configuring a first number of temporary partitions for the temporary data topic, where the first number is determined according to a preset number of partitions corresponding to the preset data topic in a preset ratio; After configuring a preset number of temporary partitions for the temporary data topic, the method further includes: Determine the second consumption delay information of the preset consumer group, and determine whether the partition expansion condition is met according to the effective threshold and the second consumption delay information; When the partition expansion conditions are met, a temporary partition is added to the temporary data topic, and the cached messages consumed by the preset consumer group are redistributed between the temporary partitions of multiple temporary data topics and the preset partitions of the preset data topic.

6. The message consumption delay optimization method according to claim 1, characterized in that: After configuring a preset number of temporary partitions for the temporary data topic, the method further includes: When the cached messages of the temporary data topic reaches the preset capacity upper limit, a new temporary data topic corresponding to the preset data topic is created, and the cached messages consumed by the preset consumer group are redistributed between temporary partitions of multiple temporary data topics and preset partitions of the preset data topic.

7. The message consumption delay optimization method according to claim 1, characterized in that: After redistributing the cached messages consumed by the preset consumer group between the temporary partition of the temporary data topic and the preset partition of the preset data topic, the method further includes: Determine the third consumption delay information of the preset consumer group, and determine whether the reduction condition is met according to the effective threshold and the third consumption delay information; When the shrinking condition is met, a preset number of temporary data topics are marked as being in a to-be-deleted state, and data writing to the temporary data topics marked as being in a to-be-deleted state is stopped; Continuously monitor temporary data topics marked as pending deletion, and delete the temporary partitions corresponding to the temporary data topics marked as pending deletion when the internal data consumption of the temporary data topics marked as pending deletion is completed; After completing the deletion processing of each temporary partition corresponding to the temporary data topic marked as to be deleted, and there is no consumer listening to the temporary data topic marked as to be deleted, delete the temporary data topic marked as to be deleted.

8. The message consumption delay optimization method according to claim 1, characterized in that: The effective threshold is determined based on the dynamic threshold and the preset baseline threshold, including: The minimum value between the dynamic threshold and the preset reference threshold is determined as the effective threshold.

9. The message consumption delay optimization method according to claim 1, characterized in that: Determining a dynamic threshold according to a consumption rate of messages corresponding to a preset data topic successfully consumed by the preset consumer group and a preset tolerance time window includes: Determine the consumption rate of messages corresponding to the preset data topic that the preset consumer group successfully consumes; The consumption rate, the preset tolerance time window and the preset safety factor are multiplied, and the multiplication result is determined as the dynamic threshold.

10. A message consumption delay optimization device, characterized in that: It includes a delay monitoring module, a capacity expansion processing module and a data distribution module, among which: The delay monitoring module is configured to determine the first consumption delay information of the preset consumer group, and determine whether the topic expansion condition is met according to the effective threshold and the first consumption delay information, wherein the effective threshold is determined according to the dynamic threshold and the preset reference threshold, and the dynamic threshold is determined according to the consumption rate of the message corresponding to the preset data topic successfully consumed by the preset consumer group and the preset tolerance time window; The expansion processing module is configured to create a temporary data topic corresponding to the preset data topic and configure a preset number of temporary partitions for the temporary data topic when the topic expansion condition is met; The data allocation module is configured to reallocate the cached messages consumed by the preset consumer group between the temporary partitions of the temporary data topic and the preset partitions of the preset data topic after completing the configuration of a preset number of temporary partitions for the temporary data topic, so that the preset consumer group can consume in the temporary partitions and the preset partitions.

11. A message consumption delay optimization device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the message consumption delay optimization method as described in any one of claims 1-9.

12. A non-volatile storage medium storing computer executable instructions, characterized in that: The computer executable instructions are used to execute the message consumption delay optimization method as described in any one of claims 1-9 when executed by a computer processor.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the message consumption delay optimization method described in any one of claims 1-9 is implemented.

Images