Dynamic partitioning method and device for message queue system, equipment, medium and product
By acquiring target indicator data and adjusting the number of partitions using a partition prediction model, the resource waste problem of the message queue system when the data volume changes dynamically is solved, achieving stable operation and efficient utilization.
Patent Information
- Application Number
- CN202511326290.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-17
- Publication Date
- 2026-01-13
AI Technical Summary
When the amount of data changes dynamically, the static partitioning method of the existing message queue system cannot effectively adapt, resulting in wasted or insufficient partitions and low resource utilization.
By acquiring target indicator data from business databases and message queue systems, using partition prediction models to predict production rates, and combining the processing capacity, utilization, and security factor of a single partition, the number of partitions is dynamically adjusted to achieve scaling up and down.
It improves resource utilization, ensures the stable operation of the message queue system, and adapts to dynamic changes in data volume.
Smart Images
Figure CN121326599A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of big data and machine learning, and can be used in the field of financial technology, and particularly relates to a dynamic partitioning method and device for a message queue system, equipment, medium and product. BACKGROUND
[0002] Real-time lake entry is a technology for synchronizing a business database to a data lake. The technology is achieved by binding a data replication service to a business database, so that the data change log in the database can be read and transmitted to a message queue system. The message queue system forwards the data from the producer to the data lake consumer for consumption, thereby achieving the purpose of synchronizing the data source.
[0003] In this process, for each table in the source database, there is usually a corresponding topic, and multiple message queue systems are partitioned to ensure the speed of message queue system production and consumption. However, this static setting often cannot well adapt to the dynamic changes in data volume over time, resulting in partition waste or deficiency. Therefore, an efficient partitioning method is needed to adapt to the dynamic changes in data volume over time. SUMMARY
[0004] The present application provides a dynamic partitioning method, device, equipment, medium and product for a message queue system to improve resource utilization.
[0005] According to an aspect of the present application, a dynamic partitioning method for a message queue system is provided, which comprises:
[0006] Obtaining target indicator data from a business database and a message queue system; wherein the target indicator data comprises at least one of message production rate, message production rate, message lag amount and message size distribution;
[0007] Performing production rate prediction on the target indicator data based on a partition prediction model to obtain a target predicted production rate of the message queue system;
[0008] Determining the required number of partitions according to the target predicted production rate, the single-partition processing capacity, utilization rate and safety factor corresponding to the message queue system;
[0009] According to the current number of partitions of the message queue system and the required number of partitions, the message queue system is scaled to obtain a target number of partitions.
[0010] According to another aspect of the present application, a dynamic partitioning device for a message queue system is provided, which comprises:
[0011] An index data acquisition module is configured to acquire target index data from a business database and a message queue system, wherein the target index data comprises at least one of a message production rate, a message production rate, a message lag amount, and a message size distribution.
[0012] A production rate prediction module is configured to perform production rate prediction on the target index data based on a partition prediction model, to obtain a target predicted production rate of the message queue system.
[0013] A demand partition number determination module is configured to determine a demand partition number according to the target predicted production rate, a single-partition processable capacity corresponding to the message queue system, a utilization rate, and a safety coefficient.
[0014] A target partition number determination module is configured to perform expansion or contraction of the message queue system according to a current partition number of the message queue system and the demand partition number, to obtain a target partition number.
[0015] According to another aspect of the present application, an electronic device is provided, which comprises:
[0016] at least one processor; and
[0017] a memory connected to the at least one processor in communication; wherein
[0018] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the dynamic partition method of the message queue system according to any one of the embodiments of the present application.
[0019] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to perform the dynamic partition method of the message queue system according to any one of the embodiments of the present application when executed by the processor.
[0020] According to another aspect of the present application, a computer program product is provided, which comprises a computer program for enabling a processor to perform the dynamic partition method of the message queue system according to any one of the embodiments of the present application when executed by the processor.
[0021] The technical scheme of the embodiment of the application comprises the following steps: obtaining target index data from a service database and a message queue system; wherein the target index data comprises at least one of a message production rate, a message production rate, a message lag amount and a message size distribution; performing production rate prediction on the target index data based on a partition prediction model to obtain a target predicted production rate of the message queue system; determining a required partition number according to the target predicted production rate, a single-partition processable capacity corresponding to the message queue system, a utilization rate and a safety factor; and performing expansion or contraction of the message queue system according to a current partition number of the message queue system and the required partition number to obtain a target partition number. The above technical scheme can predict the message production rate at a hourly level through the partition prediction model, thereby effectively determining the required partition number of the message queue system, dynamically adjusting the partition, improving the resource utilization rate and ensuring stable operation of the message queue system.
[0022] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the application, nor is it used to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS
[0023] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.
[0024] Figure 1 is a flow chart of a dynamic partition method of a message queue system according to an embodiment of the application;
[0025] Figure 2 is a flow chart of a dynamic partition method of a message queue system according to an embodiment of the application;
[0026] Figure 3 is a structural schematic diagram of a dynamic partition device of a message queue system according to an embodiment of the application;
[0027] Figure 4 is a structural schematic diagram of an electronic device for implementing a dynamic partition method of a message queue system according to an embodiment of the application. DETAILED DESCRIPTION
[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0030] Furthermore, it should be noted that the information collected in the technical solution of this invention is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data all comply with the relevant laws, regulations and standards of relevant countries and regions, necessary confidentiality measures have been taken, and public order and good morals are not violated. Corresponding operation entry points are provided for users to choose to authorize or refuse.
[0031] Figure 1 This is a flowchart illustrating a dynamic partitioning method for a message queue system according to an embodiment of the present invention. This embodiment is applicable to scenarios involving real-time inbound traffic and high-throughput stream processing, addressing how to dynamically partition message queue writes. This method can be executed by a dynamic partitioning device within the message queue system. This device can be implemented in hardware and / or software and can be configured in an electronic device that carries the dynamic partitioning function of the message queue system, such as a server. Figure 1 As shown, the method includes:
[0032] S110. Obtain target indicator data from the business database and message queue system.
[0033] Among them, the target indicator data refers to the indicator data used to obtain in real time from the business database and message queue system; optionally, the target indicator data includes at least one of message production rate, message production rate, message latency, and message size distribution.
[0034] Specifically, initial indicator data is obtained from the business database and message queue system. The initial indicator data is preprocessed, including but not limited to resampling, missing data completion, anomaly detection and removal, etc., to obtain intermediate indicator data. Then, the intermediate indicator data is processed in a time series manner, such as time window processing, holiday marking, historical lag processing, rolling mean or variance processing, to obtain target indicator data. This can ensure data continuity and reliability.
[0035] S120. Based on the partition prediction model, the production rate of the target indicator data is predicted to obtain the target predicted production rate of the message queue system.
[0036] In this embodiment, the partition prediction model is used to predict the message production rate of the message queue system, and it can be obtained by pre-training a neural network model based on historical indicator data. The target predicted production rate refers to the predicted message production rate for the next 24 hours.
[0037] Specifically, the target indicator data is input into the partition prediction model, and the target predicted production rate of the message queue system is obtained through model prediction.
[0038] S130. Determine the number of demand partitions based on the target predicted production rate, the single partition processing capacity of the message queue system, utilization rate, and security factor.
[0039] In this embodiment, the number of required partitions refers to the number of partitions currently required by the message queue system.
[0040] One alternative approach is to obtain the processing capacity, utilization rate, and security factor of a single partition corresponding to the messaging system, and then, based on a preset formula, determine the required number of partitions according to the target predicted production rate, the processing capacity, utilization rate, and security factor of a single partition corresponding to the message queue system.
[0041] Another option is to determine the message production volume based on the target predicted production rate and safety factor; determine the effective processing efficiency based on the processing capacity and utilization of a single partition; and determine the number of demand partitions based on the message production volume and effective processing efficiency.
[0042] Specifically, the message production volume is obtained by multiplying the target predicted production rate λ and the safety factor s, and the single partition's processing capacity C is calculated. p Multiply the utilization rate U to get the effective processing efficiency. Divide the message production volume and the effective processing efficiency, and round up the result to get the number of demand partitions.
[0043] Understandably, the number of partitions based on demand facilitates dynamic scaling up and down in the future, thereby improving resource utilization.
[0044] S140. Based on the current number of partitions and the required number of partitions in the message queue system, scale the message queue system up or down to obtain the target number of partitions.
[0045] In this embodiment, the current number of partitions refers to the current number of partitions in the message queue system. The target number of partitions refers to the number of partitions after the message queue system is scaled up or down, i.e., updated.
[0046] An optional approach involves scaling up or down the message queue system based on the current number of partitions and the required number of partitions to obtain the target number of partitions. This includes: determining the absolute value of the difference between the current number of partitions and the required number of partitions, using this as the partition difference; if the current number of partitions is less than the required number of partitions, and the partition difference is greater than a first threshold, then scaling up the message queue system to obtain the target number of partitions; if the current number of partitions is greater than the required number of partitions, and the partition difference is greater than a second threshold, then scaling down the message queue system to obtain the target number of partitions. It should be noted that the first and second thresholds can be determined based on...
[0047] Specifically, the absolute value of the difference between the current number of partitions and the required number of partitions is calculated as the partition difference. If the current number of partitions is less than the required number of partitions and the partition difference is greater than the first threshold, the message queue system is expanded. For example, the message queue system management interface can be called to add partitions to obtain the target number of partitions. If the current number of partitions is greater than the required number of partitions and the partition difference is greater than the second threshold, the message queue system is scaled down. For example, soft scaling down can be performed without changing the number of partitions, reducing consumer concurrency and rate limiting to obtain the target number of partitions. Alternatively, hard scaling down can be performed, creating fewer new partitions to obtain the target number of partitions, and then data can be migrated from the current partitions to the new partitions using streaming migration tools.
[0048] Understandably, by comparing the current number of partitions with the required number of partitions, it is possible to dynamically monitor in real time whether the message queue system needs to be expanded due to high load or scaled down due to low load, thereby achieving dynamic scaling up and down and improving resource utilization.
[0049] The technical solution of this invention involves obtaining target indicator data from a business database and a message queue system. The target indicator data includes at least one of message production rate, message latency, and message size distribution. A partition prediction model is used to predict the production rate of the target indicator data, resulting in the target predicted production rate of the message queue system. Based on the target predicted production rate, the processing capacity of a single partition of the message queue system, its utilization rate, and security factor, the required number of partitions is determined. The message queue system is then scaled up or down based on its current number of partitions and the required number of partitions to obtain the target number of partitions. This technical solution, through the partition prediction model, can predict the message production rate at the hourly level, thereby effectively determining the required number of partitions for the message queue system. This allows for better dynamic adjustment of partitions, improves resource utilization, and ensures stable operation of the message queue system.
[0050] Figure 2 This is a flowchart of a dynamic partitioning method for a message queue system according to an embodiment of the present invention. Based on the above embodiments, this embodiment further elaborates on the training process of the partitioning prediction model. Figure 2 As shown, the method includes:
[0051] S210. Obtain target indicator data from the business database and message queue system.
[0052] The target metrics include at least one of the following: message production rate, message production speed, message latency, and message size distribution.
[0053] S220. Based on the partition prediction model, the production rate of the target indicator data is predicted to obtain the target predicted production rate of the message queue system.
[0054] S230. Determine the number of demand partitions based on the target predicted production rate, the single partition processing capacity of the message queue system, utilization rate, and security factor.
[0055] S240. Based on the current number of partitions and the required number of partitions in the message queue system, scale the message queue system up or down to obtain the target number of partitions.
[0056] In one optional approach, the partition prediction model is trained as follows: Sample indicator data is acquired; wherein the sample indicator data includes at least one of message production rate, message production speed, message lag, and message size distribution; the sample indicator data is input into an initial prediction model to obtain the sample predicted production rate corresponding to multiple quantiles; wherein the initial prediction model includes an encoder, at least two layers of long short-term memory networks, and a multi-head fully connected layer; the training loss is determined based on the sample predicted production rate corresponding to multiple quantiles and the label data corresponding to the sample indicator data; the initial prediction model is iteratively trained using the training loss to obtain the partition prediction model.
[0057] Among them, the sample predicted production rate refers to the message production rate predicted based on sample indicator data.
[0058] Specifically, historical indicator data is acquired and preprocessed to obtain sample indicator data. It should be noted that the preprocessing method is the same as that used for the initial indicator data, and will not be elaborated here. The sample indicator data is then input into the initial prediction model to obtain the sample predicted production rate corresponding to multiple quantiles. Preferably, this invention sets the sample predicted production rate corresponding to three quantiles, and then determines the training loss based on the sample predicted production rates corresponding to multiple quantiles and the label data corresponding to the sample indicator data. Using the training loss, the initial prediction model is iteratively trained until the training stopping condition is met. The model corresponding to the stopping point is used as the partition prediction model. The training stopping condition is that the training loss stabilizes within a set range, or the number of iterations reaches a set number. The set range and the set number of iterations can be set by those skilled in the art according to actual conditions.
[0059] Understandably, by training an initial prediction model that includes an encoder, a long short-term memory network, and a multi-head fully connected layer, a partitioned prediction model can be obtained, which can accurately predict production rates.
[0060] For example, inputting sample index data into an initial prediction model to obtain sample prediction production rates corresponding to multiple quantiles includes: inputting sample index data into the encoder of the initial prediction model to obtain encoded features; inputting the encoded features into at least two long short-term memory networks to obtain temporal features; and inputting the temporal features into multi-head fully connected layers to obtain sample prediction production rates corresponding to multiple quantiles.
[0061] Encoded features refer to the features extracted by the encoder, which can be represented in matrix or vector form. Temporal features refer to the features extracted by the Long Short-Term Memory network, which can also be represented in matrix or vector form.
[0062] Specifically, the sample index data is input into the encoder of the initial prediction model to obtain the encoded features; the encoded features are input into at least two long short-term memory networks to obtain the temporal features; the temporal features are then input into multi-head fully connected layers to obtain the sample predicted production rate corresponding to multiple quantiles.
[0063] Understandably, introducing long short-term memory networks and multi-head fully connected layers can effectively perform time series prediction and simultaneously obtain samples from multiple quantiles to predict production rates.
[0064] For example, the training loss is determined based on the production rate prediction data of samples corresponding to multiple quantiles and the label data corresponding to the sample index data. This includes: for each quantile, determining the quantile loss based on the quantile loss function, the production rate prediction data of samples corresponding to that quantile, and the label data; and weighted summing of multiple quantile losses to obtain the training loss.
[0065] Specifically, for each quantile, based on the quantile loss function, the production rate and label data predicted from the samples corresponding to that quantile are used to determine the quantile loss, for example:
[0066] in, The label data represents the true value, y represents the sample predicted production rate; τ can be set; when e>0, that is, the sample predicted production rate is greater than the true value, the quantile loss is τe, otherwise, the quantile loss is (τ-1)e.
[0067] Then, the multiple quantile losses are weighted and summed to obtain the training loss. It should be noted that the higher the quantile, the larger the loss weight and the greater the corresponding penalty, which is more accurate during backpropagation training and results in better model performance.
[0068] Understandably, introducing quantile loss can improve model performance and make it more accurate in predicting production rates.
[0069] It should also be noted that during training, sample metrics data can be used for validation with data from the past 2-4 weeks and for testing with data from the last 1-2 weeks.
[0070] Furthermore, we continuously monitor indicator data and update and adjust the parameters of the regional prediction model online.
[0071] The technical solution of this invention involves obtaining target indicator data from a business database and a message queue system. The target indicator data includes at least one of message production rate, message latency, and message size distribution. A partition prediction model is used to predict the production rate of the target indicator data, resulting in the target predicted production rate of the message queue system. Based on the target predicted production rate, the processing capacity of a single partition of the message queue system, its utilization rate, and security factor, the required number of partitions is determined. The message queue system is then scaled up or down based on its current number of partitions and the required number of partitions to obtain the target number of partitions. This technical solution, through the partition prediction model, can predict the message production rate at the hourly level, thereby effectively determining the required number of partitions for the message queue system. This allows for better dynamic adjustment of partitions, improves resource utilization, and ensures stable operation of the message queue system.
[0072] Figure 3 This is a schematic diagram of a dynamic partitioning device for a message queue system according to an embodiment of the present invention. This embodiment is applicable to scenarios involving real-time inbound traffic and high-throughput stream processing, specifically how to dynamically partition message queue writes. This method can be executed by the dynamic partitioning device of the message queue system, which can be implemented in hardware and / or software. This device can be configured in an electronic device that carries the dynamic partitioning function of the message queue system, such as a server. Figure 3 As shown, the device includes:
[0073] The indicator data acquisition module 310 is used to acquire target indicator data from the business database and message queue system; wherein, the target indicator data includes at least one of message production rate, message production speed, message latency, and message size distribution;
[0074] The production rate prediction module 320 is used to predict the production rate of target indicator data based on the partition prediction model, so as to obtain the target predicted production rate of the message queue system.
[0075] The demand partition number determination module 330 is used to determine the demand partition number based on the target predicted production rate, the single partition processing capacity of the message queue system, utilization rate and security factor.
[0076] The target partition number determination module 340 is used to scale the message queue system up or down based on the current number of partitions and the required number of partitions to obtain the target number of partitions.
[0077] The technical solution of this invention involves obtaining target indicator data from a business database and a message queue system. The target indicator data includes at least one of message production rate, message latency, and message size distribution. A partition prediction model is used to predict the production rate of the target indicator data, resulting in the target predicted production rate of the message queue system. Based on the target predicted production rate, the processing capacity of a single partition of the message queue system, its utilization rate, and security factor, the required number of partitions is determined. The message queue system is then scaled up or down based on its current number of partitions and the required number of partitions to obtain the target number of partitions. This technical solution, through the partition prediction model, can predict the message production rate at the hourly level, thereby effectively determining the required number of partitions for the message queue system. This allows for better dynamic adjustment of partitions, improves resource utilization, and ensures stable operation of the message queue system.
[0078] Optionally, the device also includes a model training module for:
[0079] Obtain sample indicator data; wherein, the sample indicator data includes at least one of message production rate, message production rate, message latency, and message size distribution;
[0080] The sample index data is input into the initial prediction model to obtain the sample predicted production rate corresponding to multiple quantiles; the initial prediction model includes an encoder, at least two layers of long short-term memory network, and a multi-head fully connected layer;
[0081] The training loss is determined by predicting the production rate based on samples corresponding to multiple quantiles and the label data corresponding to the sample indicator data.
[0082] The initial prediction model is iteratively trained using training loss to obtain the partition prediction model.
[0083] Optionally, the model training module is specifically used for:
[0084] The sample index data is input into the encoder of the initial prediction model to obtain the encoded features;
[0085] Input the encoded features into at least two long short-term memory networks to obtain temporal features;
[0086] The temporal features are input into a multi-head fully connected layer to obtain the production rate prediction of samples corresponding to multiple quantiles.
[0087] Optionally, the model training module is specifically used for:
[0088] For each quantile, based on the quantile loss function, the production rate and label data are predicted according to the sample corresponding to that quantile to determine the quantile loss;
[0089] The training loss is obtained by weighted summation of the losses at multiple quantiles.
[0090] Optionally, the module 330 for determining the number of required partitions is specifically used for:
[0091] The message production volume is determined based on the target predicted production rate and safety factor;
[0092] Determine the effective processing efficiency based on the processing capacity and utilization rate of a single partition;
[0093] The number of demand partitions is determined based on message production volume and effective processing efficiency.
[0094] Optionally, the target partition number determination module 340 is specifically used for:
[0095] Determine the absolute value of the difference between the current number of partitions and the required number of partitions, and use this as the partition difference.
[0096] If the current number of partitions is less than the required number of partitions, and the partition difference is greater than the first threshold, then the message queue system is expanded to obtain the target number of partitions;
[0097] If the current number of partitions is greater than the required number of partitions, and the partition difference is greater than the second threshold, then the message queue system is scaled down to obtain the target number of partitions.
[0098] The dynamic partitioning device for the message queue system provided in this embodiment of the invention can execute the dynamic partitioning method for the message queue system provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.
[0099] According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.
[0100] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the dynamic partitioning method of the message queue system according to an embodiment of the present invention. Figure 4 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0101] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0102] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0103] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the dynamic partitioning method of a message queue system.
[0104] In some embodiments, the dynamic partitioning method of the message queuing system may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the dynamic partitioning method of the message queuing system described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the dynamic partitioning method of the message queuing system by any other suitable means (e.g., by means of firmware).
[0105] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0106] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0107] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0108] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0109] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0110] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product within the cloud computing service system to address the shortcomings of traditional physical hosts and virtual private servers, such as high management difficulty and weak business scalability.
[0111] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.
[0112] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A dynamic partitioning method for a message queue system, characterized in that, include: Obtain target indicator data from the business database and message queue system; wherein, the target indicator data includes at least one of message production rate, message production speed, message latency, and message size distribution; Based on the partition prediction model, the target indicator data is used to predict the production rate, and the target predicted production rate of the message queue system is obtained. The number of required partitions is determined based on the target predicted production rate, the single partition processing capacity of the message queue system, the utilization rate, and the security factor. Based on the current number of partitions in the message queue system and the required number of partitions, the message queue system is scaled up or down to obtain the target number of partitions.
2. The method according to claim 1, characterized in that, The partition prediction model is trained in the following manner: Obtain sample indicator data; wherein, the sample indicator data includes at least one of message production rate, message production rate, message latency, and message size distribution; The sample index data is input into the initial prediction model to obtain the sample predicted production rate corresponding to multiple quantiles; wherein, the initial prediction model includes an encoder, at least two layers of long short-term memory network, and a multi-head fully connected layer; The training loss is determined based on the production rate predicted by the samples corresponding to the multiple quantiles and the label data corresponding to the sample indicator data. The initial prediction model is iteratively trained using the training loss to obtain a partition prediction model.
3. The method according to claim 2, characterized in that, The sample index data is input into the initial prediction model to obtain the sample predicted production rate corresponding to multiple quantiles, including: The sample index data is input into the encoder of the initial prediction model to obtain the encoded features; The encoded features are input into at least two long short-term memory networks to obtain temporal features; The time-series features are input into a multi-head fully connected layer to obtain the production rate prediction of samples corresponding to multiple quantiles.
4. The method according to claim 1, characterized in that, Based on the production rate prediction data corresponding to the samples corresponding to the multiple quantiles and the label data corresponding to the sample indicator data, the training loss is determined, including: For each quantile, based on the quantile loss function, the production rate and label data are predicted according to the sample corresponding to that quantile to determine the quantile loss; The training loss is obtained by weighted summation of the losses at multiple quantiles.
5. The method according to claim 1, characterized in that, The step of determining the number of demand partitions based on the target predicted production rate, the single partition processing capacity of the message queue system, utilization rate, and security factor includes: The message production volume is determined based on the target predicted production rate and the security factor; The effective processing efficiency is determined based on the processing capacity of the single partition and the utilization rate. The number of demand partitions is determined based on the message production volume and the effective processing efficiency.
6. The method according to claim 1, characterized in that, The step of scaling up or down the message queue system based on the current number of partitions and the required number of partitions to obtain the target number of partitions includes: Determine the absolute value of the difference between the current number of partitions and the required number of partitions, and use it as the partition difference; If the current number of partitions is less than the required number of partitions, and the partition difference is greater than the first threshold, then the message queue system is expanded to obtain the target number of partitions; If the current number of partitions is greater than the required number of partitions, and the partition difference is greater than the second threshold, then the message queue system is scaled down to obtain the target number of partitions.
7. A dynamic partitioning device for a message queue system, characterized in that, include: The indicator data acquisition module is used to acquire target indicator data from the business database and the message queue system; wherein, the target indicator data includes at least one of message production rate, message production speed, message latency, and message size distribution; The production rate prediction module is used to predict the production rate of the target indicator data based on the partition prediction model, so as to obtain the target predicted production rate of the message queue system. The demand partition number determination module is used to determine the demand partition number based on the target predicted production rate, the single partition processing capacity of the message queue system, utilization rate, and security factor. The target partition number determination module is used to scale the message queue system up or down based on the current number of partitions and the required number of partitions to obtain the target number of partitions.
8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the dynamic partitioning method of the message queue system according to any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the dynamic partitioning method of the message queue system according to any one of claims 1-6.
10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the dynamic partitioning method of the message queue system according to any one of claims 1-6.