A dynamic sharding optimization method and system based on a message queue

Abstract

This application discloses a dynamic fragmentation optimization method and system based on message queues, belonging to the technical field of distributed message queue transmission scheduling. The method includes: classifying stable message types and fluctuating message types; constructing a time-series fragmentation threshold range based on the number of messages and message queue operation indicators under the transmission time sequence; obtaining a dynamic fragmentation ratio based on the proportion of stable and fluctuating message types under the same transmission time sequence in historical transmission cases; outputting a preset number of stable fragments and a preset number of fluctuating fragments under the corresponding transmission time sequence using the time-series fragmentation threshold range and the dynamic fragmentation ratio; performing initial message transmission using the preset number of stable fragments and the preset number of fluctuating fragments in the initial time sequence; and adjusting the number of fluctuating fragments in the next time sequence based on the proportion of real-time fragmentation redundancy and the preset number of fluctuating fragments in the next time sequence. The beneficial effect of this application is that it balances resource utilization and transmission stability.

Description

Technical Field

[0001] This application relates to the field of distributed message queue transmission scheduling technology, specifically to a dynamic fragmentation optimization method and system based on message queues. Background Technology

[0002] In distributed message queue transmission scenarios, message sharding is a core means to ensure message transmission efficiency and system load balancing. Most existing message queue sharding scheduling schemes adopt a fixed number of shards configuration. Stable transmission messages and messages with sudden fluctuations share shard resources. The fixed number of shards is difficult to adapt to load changes under different transmission sequences. When message traffic is low, it is easy to cause idle and wasteful shard resources. During traffic fluctuations or sudden peaks, insufficient shards may lead to message backlog and increased transmission latency. The overall sharding scheduling flexibility, rationality and system stability are poor, and it cannot meet the needs of distributed message queues to operate efficiently, stably and evenly in complex transmission scenarios. Summary of the Invention

[0003] This application addresses the problem in related technologies where message queue sharding scheduling struggles to balance resource utilization and transmission stability. It provides a dynamic sharding optimization method and system based on message queues. By classifying messages into stable and fluctuating message types based on their stability, the method dynamically adjusts the number of shards for fluctuating message types in conjunction with a time-series sharding threshold range, while maintaining a stable number of shards for stable message types. This improves system throughput and transmission stability while reducing resource waste and response latency.

[0004] To achieve the above technical objectives, this application provides a technical solution: a dynamic fragmentation optimization method based on a message queue, comprising the following steps: classifying stable message types and fluctuating message types according to the probability of occurrence of each message type under the same transmission sequence in historical transmission data; constructing a time-series fragmentation threshold range based on the number of messages under the transmission sequence and message queue operation indicators; obtaining a dynamic fragmentation ratio based on the proportion of stable message types and fluctuating message types under the same transmission sequence in historical transmission data; outputting a preset number of stable fragments and a preset number of fluctuating fragments under the corresponding transmission sequence using the time-series fragmentation threshold range and the dynamic fragmentation ratio; performing initial message transmission using the preset number of stable fragments and the preset number of fluctuating fragments in the initial sequence to obtain real-time fragmentation redundancy for each message type; and adjusting the number of fluctuating fragments in the next time sequence based on the proportion of real-time fragmentation redundancy and the preset number of fluctuating fragments in the next time sequence.

[0005] Furthermore, the step of classifying stable message types and fluctuating message types based on the probability of occurrence of each message type under the same transmission sequence in historical transmission situations includes: statistically analyzing the probability of occurrence of the same message type under the same transmission sequence in multiple consecutive historical transmission periods; classifying message types with a probability of occurrence greater than a first preset threshold as stable message types, and classifying message types with a probability of occurrence less than or equal to the first preset threshold as fluctuating message types.

[0006] Furthermore, the step of constructing the time-series fragmentation threshold range based on the number of messages and message queue operation indicators under the transmission time sequence includes: constructing the transmission load time sequence relationship based on the relationship between the number of messages and message queue operation indicators and the transmission time sequence in historical transmission cases; and constructing the time-series fragmentation threshold range corresponding to the transmission load time sequence relationship based on the minimum and maximum number of fragments corresponding to the same transmission load in historical transmission cases.

[0007] Furthermore, the step of obtaining the dynamic fragmentation ratio based on the proportion of stable message types and fluctuating message types under the same transmission time sequence in historical transmission situations includes: calculating the first proportion of stable message types and fluctuating message types under each transmission time sequence in historical transmission situations; assigning a time sequence attenuation weight based on the time sequence proximity in the same transmission time sequence; and calculating the dynamic fragmentation ratio using the time sequence attenuation weight and the first proportion.

[0008] Furthermore, the step of outputting the preset stable fragment number and preset fluctuating fragment number under the corresponding transmission timing based on the timing fragmentation threshold range and dynamic fragmentation ratio includes: retrieving the timing fragmentation threshold range and dynamic fragmentation ratio for the same transmission timing; using the minimum value of the timing fragmentation threshold range as the preset stable fragment number; calculating a reference value for the preset fluctuating fragment number using the minimum value of the timing fragmentation threshold range and the dynamic fragmentation ratio; if the sum of the reference values ​​of the preset stable fragment number and the preset fluctuating fragment number is greater than the maximum value of the timing fragmentation threshold range, then the difference between the maximum value of the timing fragmentation threshold range and the preset stable fragment number is used as the preset fluctuating fragment number.

[0009] Furthermore, the step of performing initial message transmission with a preset stable number of shards and a preset fluctuating number of shards in the initial time sequence to obtain real-time shard redundancy for each message type includes: configuring shard resources for the message queue according to the preset stable number of shards and the preset fluctuating number of shards corresponding to the initial time sequence, and performing initial message transmission; collecting the real-time shard usage number for each message type, calculating the actual total number of shards used for the fluctuating message type, and calculating the real-time shard redundancy for the fluctuating message type based on the difference between the actual total number of shards used and the preset fluctuating number of shards.

[0010] Furthermore, adjusting the number of fluctuating fragments in the next time series based on the proportion of real-time fragment redundancy and the preset number of fluctuating fragments in the next time series includes: obtaining the proportion of real-time fragment redundancy by the ratio of real-time fragment redundancy to the preset number of fluctuating fragments in the initial time series; and calculating the adjustment value of the number of fluctuating fragments in the next time series based on the preset number of fluctuating fragments in the next time series and the proportion of real-time fragment redundancy.

[0011] Furthermore, the step of performing initial message transmission with a preset stable number of shards and a preset fluctuating number of shards for the initial time sequence, and obtaining real-time shard redundancy for each message type, includes: configuring shard resources for the message queue according to the preset stable number of shards and the preset fluctuating number of shards corresponding to the initial time sequence, and performing initial message transmission; collecting the real-time shard usage for each message type, and calculating the first actual shard usage total for the stable message type and the second actual shard usage total for the fluctuating message type respectively; calculating the first real-time shard redundancy for the stable message type based on the difference between the first actual shard usage total and the preset stable number of shards; and calculating the second real-time shard redundancy for the fluctuating message type based on the difference between the second actual shard usage total and the preset fluctuating number of shards.

[0012] Furthermore, adjusting the number of fluctuating fragments in the next time series based on the proportion of real-time fragment redundancy and the preset number of fluctuating fragments in the next time series includes: calculating a first proportion based on the first real-time fragment redundancy and the preset number of stable fragments in the initial time series; calculating a second proportion based on the second real-time fragment redundancy and the preset number of fluctuating fragments in the initial time series; adjusting the minimum value of the time series fragment threshold range in the next time series according to the first proportion; recalculating the preset number of fluctuating fragments in the next time series based on the adjusted minimum value of the time series fragment threshold range; and outputting the number of fluctuating fragments in the next time series based on the second real-time fragment redundancy and the recalculated preset number of fluctuating fragments.

[0013] Another technical solution provided in this application is a dynamic fragmentation optimization system based on a message queue, used to implement the method described above, including: a message classification module, used to classify stable message types and fluctuating message types according to the probability of occurrence of each message type under the same transmission time sequence in historical transmission cases; a threshold construction module, used to construct a time-series fragmentation threshold range based on the number of messages under the transmission time sequence and message queue operation indicators; a ratio calculation module, used to obtain a dynamic fragmentation ratio based on the proportion of stable message types and fluctuating message types under the same transmission time sequence in historical transmission cases; a fragmentation preset module, used to output a preset number of stable fragments and a preset number of fluctuating fragments under the corresponding transmission time sequence based on the time-series fragmentation threshold range and the dynamic fragmentation ratio; a redundancy monitoring module, used to perform initial message transmission with the preset number of stable fragments and the preset number of fluctuating fragments in the initial time sequence to obtain the real-time fragmentation redundancy of each message type; and a dynamic adjustment module, used to adjust the number of fluctuating fragments in the next time sequence based on the proportion of real-time fragmentation redundancy and the preset number of fluctuating fragments in the next time sequence.

[0014] The beneficial effects of this application are as follows: 1. By classifying messages into stable message types and fluctuating message types based on the stability of message types in historical transmission, and constructing a time-series fragmentation threshold range based on the message traffic situation of the transmission time sequence, the application avoids too many or too few fragments. Combining the time-series fragmentation threshold range and the pre-divided message types, the application allocates a stable number of fragments to stable message types and a fluctuating number of fragments to fluctuating message types. This improves system throughput and transmission stability while reducing resource waste and response latency.

[0015] 2. By assigning time-series attenuation weights based on the proximity of time sequences, the impact of recent historical transmission cycles on the dynamic fragmentation ratio becomes greater, while the impact of historical transmission cycles further away from the current transmission sequence becomes smaller. This makes the fragmentation ratio more in line with the latest transmission trends, improving resource utilization and scheduling rationality. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a dynamic sharding optimization method based on message queues according to this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely one preferred embodiment of this application and are only used to explain this application. They do not limit the scope of protection of this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] like Figure 1As shown, a dynamic sharding optimization method based on message queues includes the following steps: Based on the probability of each message type appearing under the same transmission sequence in historical transmission data, stable message types and fluctuating message types are classified. Construct a time-series fragmentation threshold range based on the number of messages under the transmission time sequence and message queue operation indicators; The dynamic fragmentation ratio is obtained based on the proportion of stable message types and fluctuating message types under the same transmission sequence in historical transmission data. Output the preset stable number of fragments and the preset fluctuating number of fragments under the corresponding transmission timing based on the timing fragmentation threshold range and the dynamic fragmentation ratio; Initial message transmission is performed using a preset stable number of shards and a preset fluctuating number of shards in the initial timing sequence to obtain real-time shard redundancy for each message type. The number of fluctuation fragments in the next time series is adjusted based on the proportion of real-time fragment redundancy and the preset number of fluctuation fragments in the next time series.

[0019] In this embodiment, messages are divided into stable message types and fluctuating message types based on the stability of message types in historical transmission. A time-series fragmentation threshold range is constructed based on the message traffic situation during transmission to avoid too many or too few fragments. Combining the time-series fragmentation threshold range and the pre-divided message types, a stable number of fragments is allocated to stable message types, and a fluctuating number of fragments is allocated to fluctuating message types. This improves system throughput and transmission stability while reducing resource waste and response latency.

[0020] Specifically, based on the probability of each message type occurring under the same transmission sequence in historical transmission data, stable message types and fluctuating message types are categorized as follows: The probability of the same message type occurring under the same transmission sequence within multiple consecutive historical transmission periods is statistically analyzed. Message types with a probability of occurrence greater than a first preset threshold are classified as stable message types, while message types with a probability of occurrence less than or equal to the first preset threshold are classified as fluctuating message types.

[0021] By using message stability-based segmentation scheduling, stable message types and fluctuating message types are separated. This allows for initial segmentation based on stable message types and adaptive segmentation based on fluctuating message types when dynamically adjusting the number of segments. By performing initial segmentation based on stable message types in advance, it is ensured that the segments can meet the traffic transmission requirements of stable message types, while reducing the magnitude of segmentation adjustments, lowering the response latency caused by segmentation scheduling, and improving the transmission efficiency and stability of the message queue.

[0022] In this embodiment, message types are statistically analyzed over at least 15 consecutive historical transmission cycles. The probability of occurrence of each message type over these 15 cycles is obtained using a sliding window. A first preset threshold is 80%. If the probability of occurrence of a message type is less than or equal to 80%, the message type is considered to be a fluctuating message type in the current transmission sequence. If the probability of occurrence of a message type is greater than 80%, the message type is considered to be a stable message type in the current transmission sequence.

[0023] The time-series fragmentation threshold range is constructed based on the number of messages under transmission timing and message queue operation metrics, including: Construct a transmission load timing relationship based on the changes in message quantity and message queue operation indicators with transmission timing in historical transmission data; Based on the minimum and maximum number of fragments corresponding to the same transmission load in historical transmission scenarios, a time-series fragmentation threshold range corresponding to the timing relationship of transmission load is constructed.

[0024] Message queue performance metrics include one or a combination of queue backlog length, message transmission latency, bandwidth utilization, and CPU utilization. The transmission load for each transmission sequence is determined by the number of messages arriving and the message queue's operational status. Higher message counts, queue congestion, latency, and resource scarcity result in a higher transmission load, while fewer messages, an idle queue, and lower latency lead to a lower transmission load. By establishing a temporal relationship between message count, message queue performance metrics, and transmission load, the system reflects how transmission load changes over time. Furthermore, by defining the minimum and maximum number of fragments corresponding to the same transmission load, a temporal fragmentation threshold range matching the transmission load temporal relationship is constructed. This sets appropriate upper and lower limits for the number of fragments for different transmission sequences, ensuring that low-load sequences are not allocated too many fragments, resulting in resource waste, while high-load sequences are not blocked or delayed due to insufficient fragments, thus improving the overall resource utilization and transmission stability of the message queue.

[0025] In one embodiment, obtaining the dynamic fragmentation ratio based on the proportion of stable message types and fluctuating message types under the same transmission sequence in historical transmission cases includes: Obtain the number of first messages of stable message type and the number of second messages of fluctuating message type under the same transmission sequence in historical transmission data; The ratio of the number of first messages to the number of second messages is used as the dynamic fragmentation ratio.

[0026] In this scenario, all stable message types and all fluctuating message types under the same transmission time sequence are directly integrated, and the ratio of stable message types to fluctuating message types is calculated based on the total quantity as the dynamic fragmentation ratio. It's understandable that before calculating the dynamic fragmentation ratio, all stable message types and all fluctuating message types under the same transmission time sequence are preprocessed to filter out transmission time sequences that deviate significantly. It's also understandable that if the historical transmission period is one day, then the 24 hours of a day are divided according to a preset time dimension, such as dividing it by minute to obtain the transmission time sequence, and the same minutes from different days are considered to have the same transmission time sequence.

[0027] In the second scenario of this embodiment, obtaining the dynamic fragmentation ratio based on the proportion of stable message types and fluctuating message types under the same transmission sequence in historical transmission scenarios includes: Calculate the first proportion of stable message types and fluctuating message types under each transmission time sequence in historical transmission data; Timing attenuation weights are assigned based on the proximity of timings within the same transmission timing sequence, and the dynamic fragmentation ratio is calculated using the timing attenuation weights and the first proportion.

[0028] The number of stable and fluctuating message types under each transmission time sequence during historical transmission is statistically analyzed, and the initial proportion of stable and fluctuating message types under each transmission time sequence is calculated. Based on the temporal proximity of the same transmission time sequence in historical transmission scenarios, corresponding time sequence attenuation weights are assigned. Within the same transmission time sequence, the transmission time sequence corresponding to a historical transmission period closer to the current time sequence has a higher time sequence attenuation weight, and the transmission time sequence corresponding to a historical transmission period farther from the current time sequence has a lower time sequence attenuation weight. A weighted calculation is performed based on the time sequence attenuation weights and the initial proportion of the corresponding transmission time sequence to obtain the dynamic fragmentation ratio of stable and fluctuating message types under the current transmission time sequence.

[0029] By assigning time-series decay weights based on the proximity of time sequences, the impact of recent historical transmission cycles on the dynamic fragmentation ratio becomes greater, while the impact of historical transmission cycles further away from the current transmission sequence becomes smaller. This makes the fragmentation ratio more in line with the latest transmission trends, improving resource utilization and scheduling rationality.

[0030] The preset stable fragment number and preset fluctuating fragment number are output according to the timing fragmentation threshold range and dynamic fragmentation ratio, respectively, under the corresponding transmission timing. Retrieve the time-series fragmentation threshold range and dynamic fragmentation ratio for transmissions with the same timing sequence; The minimum value within the time-series fragmentation threshold range is used as the preset stable fragmentation number; The reference value for the preset number of fluctuating fragments is calculated based on the minimum value of the time-series fragmentation threshold range and the dynamic fragmentation ratio; If the sum of the reference values ​​of the preset stable shard number and the preset volatile shard number is greater than the maximum value of the time series shard threshold range, then the difference between the maximum value of the time series shard threshold range and the preset stable shard number is used as the preset volatile shard number.

[0031] A single transmission timing corresponds to a timing fragmentation threshold range and a dynamic fragmentation ratio. Since the transmission load timing relationship and the corresponding timing fragmentation threshold range are pre-built, the transmission load of the next timing can be obtained in advance based on the transmission load timing relationship, and then the timing fragmentation threshold range of the next timing can be obtained. This allows for pre-calculation of the number of fragments, improving fragmentation efficiency.

[0032] In this embodiment, the time-series fragmentation threshold range is used as a constraint on dynamic fragmentation. Message transmission stability and real-time performance are given priority, and resource utilization is maximized on this basis. Therefore, the minimum value of the time-series fragmentation threshold range is first used as the preset stable fragmentation number, and then the reference value of the preset fluctuating fragmentation number is calculated according to the dynamic fragmentation ratio.

[0033] Understandably, the time-series fragmentation threshold range is determined based on the transmission load. The difference in the number of fragments under the same transmission load primarily stems from traffic variations in fluctuating message types, while the fragmentation requirements for stable message types remain relatively fixed. Therefore, the minimum value of the time-series fragmentation threshold range is used as the preset stable fragmentation number to ensure reliable transmission of stable message types. Simultaneously, a reference value for the preset fluctuating fragmentation number is calculated using a dynamic fragmentation ratio to match fragmentation allocation with the actual message structure. When the total number of fragments exceeds the threshold limit, only the fluctuating fragmentation number is constrained and adjusted. This ensures that stable message transmission is not affected while avoiding system resource overload, and simultaneously meets the transmission needs of fluctuating message types as much as possible, achieving a balance between transmission stability, real-time performance, and resource utilization.

[0034] Initial message transmission is performed using a preset stable number of shards and a preset fluctuating number of shards for the initial timing, and real-time shard redundancy for each message type is obtained, including: Based on the preset stable shard count and preset fluctuating shard count corresponding to the initial timing sequence, the message queue is configured with sharding resources, and the initial message transmission is executed. Collect the real-time usage of fragments for each message type, calculate the actual total number of fragments used for fluctuating message types, and calculate the real-time fragment redundancy of fluctuating message types based on the difference between the actual total number of fragments used and the preset number of fluctuating fragments.

[0035] By configuring message queue sharding resources according to the preset stable sharding quantity and preset fluctuating sharding quantity corresponding to the initial timing, the initial message transmission is carried out according to the pre-allocated resources, ensuring the stability and reliability of the initial transmission phase. By collecting the real-time sharding quantity used for each message type and calculating the real-time sharding redundancy based on the difference between the actual total number of shardings used for fluctuating message types and the preset fluctuating sharding quantity, the sufficiency and usage status of sharding resources for fluctuating message types are reflected, thereby facilitating further adjustments to sharding in the future.

[0036] Adjusting the number of fluctuation fragments in the next time series based on the proportion of real-time fragment redundancy and the preset number of fluctuation fragments in the next time series includes: The proportion of real-time fragment redundancy is obtained by the ratio of real-time fragment redundancy to the preset number of fragments with fluctuation in the initial timing. The adjustment value for the number of fluctuation fragments in the next time series is calculated based on the preset number of fluctuation fragments and the proportion of real-time fragment redundancy in the next time series.

[0037] The real-time fragmentation redundancy ratio is obtained by comparing the real-time fragmentation redundancy with the preset fluctuation fragmentation number for the current transmission time sequence. This quantifies the actual usage and allocation matching degree of fluctuating message fragmentation resources. An adjustment value is calculated by combining the preset fluctuation fragmentation number for the next time sequence with the real-time fragmentation redundancy ratio. Based on the actual operating state of the previous time sequence, the fragmentation allocation for subsequent time sequences is adaptively corrected, making the fragmentation allocation more closely match actual transmission needs and improving the accuracy and rationality of dynamic scheduling. It can be understood that if the total number of fragments actually used is greater than the preset fluctuation fragmentation number, the real-time fragmentation redundancy is positive, the ratio is positive, the fluctuation fragmentation number adjustment value is positive, and the fluctuation fragmentation number increases. Conversely, if the total number of fragments actually used is less than the preset fluctuation fragmentation number, the real-time fragmentation redundancy is negative, the ratio is negative, the fluctuation fragmentation number adjustment value is negative, and the fluctuation fragmentation number decreases.

[0038] As a second embodiment of this application, initial message transmission is performed using a preset stable number of shards and a preset fluctuating number of shards in the initial timing sequence, and the real-time shard redundancy for each message type is obtained as follows: Based on the preset stable shard count and preset fluctuating shard count corresponding to the initial timing sequence, the message queue is configured with sharding resources, and the initial message transmission is executed. Collect the real-time usage of shards for each message type, and calculate the first actual usage total number of shards for stable message types and the second actual usage total number of shards for fluctuating message types respectively; The first real-time fragment redundancy of a stable message type is calculated based on the difference between the first actual total number of fragments used and the preset stable number of fragments. The second real-time shard redundancy of the fluctuation message type is calculated based on the difference between the total number of shards actually used and the preset number of fluctuation shards.

[0039] In this embodiment, the real-time sharding redundancy of stable messages and fluctuating messages is calculated simultaneously, which facilitates the subsequent synchronous adjustment of the number of stable shards and the number of fluctuating shards.

[0040] Adjusting the number of fluctuation fragments in the next time series based on the proportion of real-time fragment redundancy and the preset number of fluctuation fragments in the next time series includes: The first proportion is calculated based on the first real-time fragment redundancy and the preset stable fragment number of the initial timing; The second proportion is calculated based on the second real-time shard redundancy and the preset fluctuation shard number of the initial timing; The minimum value of the time series segmentation threshold range is adjusted according to the first proportion; The preset fluctuation fragment number for the next time series is recalculated based on the minimum value of the adjusted time series fragment threshold range. The number of fluctuation fragments in the next time series is output based on the second real-time fragment redundancy and the recalculated preset number of fluctuation fragments.

[0041] By calculating the real-time fragment redundancy ratios for stable and fluctuating message types separately, the actual surplus or shortage of fragment resources for these two types of messages is reflected. Then, based on the first ratio corresponding to the stable message type, the minimum value of the subsequent time-series fragmentation threshold range is adjusted to improve the adaptability of the threshold range. The preset number of fluctuating fragments is recalculated using the adjusted minimum value, ensuring that the fragmentation allocation of fluctuating messages is consistent with the updated constraints. Combined with the second real-time fragment redundancy, the number of fluctuating fragments in the subsequent time series is output, achieving closed-loop adaptive adjustment of fragment resources. This improves the accuracy of fluctuating message fragmentation allocation and transmission efficiency while ensuring stable message transmission.

[0042] Understandably, in this scheme, the preset stable fragment count for the next time sequence remains unchanged. Instead, the minimum value of the time sequence fragmentation threshold range is adjusted, and the preset fluctuating fragment count is recalculated. Changes in the preset stable fragment count are adjusted through the preset fluctuating fragment count. When the total number of fragments actually used in the previous time sequence is less than the preset stable fragment count, the minimum value of the time sequence fragmentation threshold range decreases, and the preset fluctuating fragment count decreases under the adjustment of the first proportion. When the total number of fragments actually used in the previous time sequence is greater than the preset stable fragment count, the minimum value of the time sequence fragmentation threshold range increases, and the preset fluctuating fragment count increases under the adjustment of the first proportion. This ensures stable transmission of stable message types. Simultaneously, the second proportion is used to further adjust the fluctuating fragment count for the next time sequence, avoiding unnecessary fragment increases due to redundancy between the two message types and improving resource utilization. Understandably, when entering the next time sequence, the next time sequence is updated to the initial time sequence.

[0043] As a third embodiment of this application, a dynamic sharding optimization system based on a message queue includes: The message classification module is used to classify stable message types and fluctuating message types based on the probability of occurrence of each message type under the same transmission sequence in historical transmission data. The threshold construction module is used to construct the time-series fragmentation threshold range based on the number of messages under the transmission time sequence and message queue operation indicators. The ratio calculation module is used to obtain the dynamic fragmentation ratio based on the proportion of stable message types and fluctuating message types under the same transmission sequence in historical transmission data. The fragmentation preset module is used to output the preset stable fragmentation number and preset fluctuating fragmentation number under the corresponding transmission timing based on the timing fragmentation threshold range and dynamic fragmentation ratio. The redundancy monitoring module is used to perform initial message transmission with a preset number of stable shards and a preset number of fluctuating shards in the initial timing, and to obtain the real-time shard redundancy of each message type. The dynamic adjustment module is used to adjust the number of fluctuation fragments in the next time series based on the proportion of real-time fragment redundancy and the preset number of fluctuation fragments in the next time series.

[0044] In this embodiment, the message classification module, threshold construction module, sharding preset module, redundancy monitoring module, and dynamic adjustment module are connected to the distributed message queue server. The message classification module is also connected to the ratio calculation module. The threshold construction module and the ratio calculation module are connected to the sharding preset module. The sharding preset module is connected to the redundancy monitoring module and the dynamic adjustment module. The redundancy monitoring module is connected to the dynamic adjustment module.

[0045] The specific embodiments described above are preferred embodiments of a dynamic sharding optimization method and system based on message queues in this application, and are not intended to limit the specific implementation scope of this application. The scope of this application includes but is not limited to the specific embodiments described above. All equivalent changes made in accordance with the shape and structure of this application are within the protection scope of this application.

Claims

1. A dynamic sharding optimization method based on message queues, characterized in that: Includes the following steps: Based on the probability of each message type appearing under the same transmission sequence in historical transmission data, stable message types and fluctuating message types are classified. Construct a time-series fragmentation threshold range based on the number of messages under the transmission time sequence and message queue operation indicators; The dynamic fragmentation ratio is obtained based on the proportion of stable message types and fluctuating message types under the same transmission sequence in historical transmission data. Output the preset stable number of fragments and the preset fluctuating number of fragments under the corresponding transmission timing based on the timing fragmentation threshold range and the dynamic fragmentation ratio; Initial message transmission is performed using a preset stable number of shards and a preset fluctuating number of shards in the initial timing sequence to obtain real-time shard redundancy for each message type. The number of fluctuation fragments in the next time series is adjusted based on the proportion of real-time fragment redundancy and the preset number of fluctuation fragments in the next time series.

2. The dynamic sharding optimization method based on message queues as described in claim 1, characterized in that: The method of classifying stable message types and fluctuating message types based on the probability of occurrence of each message type under the same transmission sequence in historical transmission data includes: The probability of the same message type occurring under the same transmission sequence within multiple consecutive historical transmission periods is statistically analyzed. Message types with a probability of occurrence greater than a first preset threshold are classified as stable message types, while message types with a probability of occurrence less than or equal to the first preset threshold are classified as fluctuating message types.

3. The dynamic sharding optimization method based on message queues as described in claim 1, characterized in that: The step of constructing the time-series fragmentation threshold range based on the number of messages under transmission timing and message queue operation indicators includes: Construct a transmission load timing relationship based on the changes in message quantity and message queue operation indicators with transmission timing in historical transmission data; Based on the minimum and maximum number of fragments corresponding to the same transmission load in historical transmission scenarios, a time-series fragmentation threshold range corresponding to the timing relationship of transmission load is constructed.

4. The dynamic sharding optimization method based on message queues as described in claim 1, characterized in that: The process of obtaining the dynamic fragmentation ratio based on the ratio of stable message types to fluctuating message types under the same transmission sequence in historical transmission data includes: Calculate the first proportion of stable message types and fluctuating message types under each transmission time sequence in historical transmission data; Timing attenuation weights are assigned based on the proximity of timings within the same transmission timing sequence, and the dynamic fragmentation ratio is calculated using the timing attenuation weights and the first proportion.

5. The dynamic sharding optimization method based on message queues as described in claim 1, characterized in that: The step of outputting the preset stable number of fragments and the preset fluctuating number of fragments under the corresponding transmission timing based on the timing fragmentation threshold range and dynamic fragmentation ratio includes: Retrieve the time-series fragmentation threshold range and dynamic fragmentation ratio for transmissions with the same timing sequence; The minimum value within the time-series fragmentation threshold range is used as the preset stable fragmentation number; The reference value for the preset number of fluctuating fragments is calculated based on the minimum value of the time-series fragmentation threshold range and the dynamic fragmentation ratio; If the sum of the reference values ​​of the preset stable shard number and the preset volatile shard number is greater than the maximum value of the time series shard threshold range, then the difference between the maximum value of the time series shard threshold range and the preset stable shard number is used as the preset volatile shard number.

6. The dynamic sharding optimization method based on message queues as described in claim 1, characterized in that: The initial message transmission based on a preset stable number of shards and a preset fluctuating number of shards in the initial time sequence, and the acquisition of real-time shard redundancy for each message type, includes: Based on the preset stable shard count and preset fluctuating shard count corresponding to the initial timing sequence, the message queue is configured with sharding resources, and the initial message transmission is executed. Collect the real-time usage of fragments for each message type, calculate the actual total number of fragments used for fluctuating message types, and calculate the real-time fragment redundancy of fluctuating message types based on the difference between the actual total number of fragments used and the preset number of fluctuating fragments.

7. The dynamic sharding optimization method based on message queues as described in claim 6, characterized in that: The adjustment of the number of fluctuation fragments in the next time series based on the proportion of real-time fragment redundancy and the preset number of fluctuation fragments in the next time series includes: The proportion of real-time fragment redundancy is obtained by the ratio of real-time fragment redundancy to the preset number of fragments with fluctuation in the initial timing. The adjustment value for the number of fluctuation fragments in the next time series is calculated based on the preset number of fluctuation fragments and the proportion of real-time fragment redundancy in the next time series.

8. The dynamic sharding optimization method based on message queues as described in claim 5, characterized in that: The initial message transmission based on a preset stable number of shards and a preset fluctuating number of shards in the initial time sequence, and the acquisition of real-time shard redundancy for each message type, includes: Based on the preset stable shard count and preset fluctuating shard count corresponding to the initial timing sequence, the message queue is configured with sharding resources, and the initial message transmission is executed. Collect the real-time usage of shards for each message type, and calculate the first actual usage total number of shards for stable message types and the second actual usage total number of shards for fluctuating message types respectively; The first real-time fragment redundancy of a stable message type is calculated based on the difference between the first actual total number of fragments used and the preset stable number of fragments. The second real-time shard redundancy of the fluctuation message type is calculated based on the difference between the total number of shards actually used and the preset number of fluctuation shards.

9. The dynamic sharding optimization method based on message queues as described in claim 8, characterized in that: The adjustment of the number of fluctuation fragments in the next time series based on the proportion of real-time fragment redundancy and the preset number of fluctuation fragments in the next time series includes: The first proportion is calculated based on the first real-time fragment redundancy and the preset stable fragment number of the initial timing; The second proportion is calculated based on the second real-time shard redundancy and the preset fluctuation shard number of the initial timing; The minimum value of the time series segmentation threshold range is adjusted according to the first proportion; The preset fluctuation fragment number for the next time series is recalculated based on the minimum value of the adjusted time series fragment threshold range. The number of fluctuation fragments in the next time series is output based on the second real-time fragment redundancy and the recalculated preset number of fluctuation fragments.

10. A message queue-based dynamic sharding optimization system, used to implement the method as described in any one of claims 1 to 9, characterized in that: include: The message classification module is used to classify stable message types and fluctuating message types based on the probability of occurrence of each message type under the same transmission sequence in historical transmission data. The threshold construction module is used to construct the time-series fragmentation threshold range based on the number of messages under the transmission time sequence and message queue operation indicators. The ratio calculation module is used to obtain the dynamic fragmentation ratio based on the proportion of stable message types and fluctuating message types under the same transmission sequence in historical transmission data. The fragmentation preset module is used to output the preset stable fragmentation number and preset fluctuating fragmentation number under the corresponding transmission timing based on the timing fragmentation threshold range and dynamic fragmentation ratio. The redundancy monitoring module is used to perform initial message transmission with a preset number of stable shards and a preset number of fluctuating shards in the initial timing, and to obtain the real-time shard redundancy of each message type. The dynamic adjustment module is used to adjust the number of fluctuation fragments in the next time series based on the proportion of real-time fragment redundancy and the preset number of fluctuation fragments in the next time series.

Images