Message queue dynamic balancing method and device, electronic equipment and storage medium

Abstract

The invention provides a message queue dynamic balancing method and device, electronic equipment and a storage medium. The method comprises the following steps: receiving real-time data collected from a specified business page; acquiring a first data volume received by each data processing task node in unit time and a processed second data volume; determining a distribution coefficient of a message queue consumed by the data processing task node according to the first data volume and the second data volume; and distributing the real-time data to the message queues consumed by the data processing task nodes according to the distribution coefficients, so that the data processing task nodes consume the real-time data distributed to the message queues, the data distribution amount can be dynamically adjusted without adjusting the number of the message queues, and the operation stability and the operation data processing effect are improved.

Description

Technical Field

[0001] The present invention relates to the field of dynamic balancing technology, and in particular to a message queue dynamic balancing method, device, electronic equipment and storage medium. Background Art

[0002] When processing bank operational data (such as credit card business data), performance bottlenecks often occur when different data processing processes process message queues, which leads to uneven data consumption and incorrect indicator calculation results.

[0003] Currently, the above problem is usually solved by increasing the number of message queues. However, increasing the number of message queues requires restarting and stopping the job, which will cause data processing to be temporarily stopped. There is also a certain probability of failure when re-releasing the job, resulting in low job stability and poor job data processing effect. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a message queue dynamic balancing method, device, electronic device and storage medium to solve the problems of low operation stability and poor operation data processing effect existing in the method of expanding the number of message queues.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of an embodiment of the present invention discloses a message queue dynamic balancing method, the method comprising:

[0007] Receive real-time data collected from designated business pages;

[0008] Obtaining the first data volume received and the second data volume processed by each data processing task node within a unit time;

[0009] Determining a distribution coefficient of a message queue consumed by the data processing task node according to the first data volume and the second data volume;

[0010] The real-time data is distributed to the message queue consumed by each of the data processing task nodes according to the distribution coefficient, so that the data processing task node consumes the real-time data distributed to the message queue.

[0011] Preferably, determining the distribution coefficient of the message queue consumed by the data processing task node according to the first data volume and the second data volume includes:

[0012] Calculating a ratio between the second data volume and the first data volume to obtain a back pressure value of the data processing task node;

[0013] A coefficient corresponding to a preset range in which the back pressure value is located is determined to obtain a distribution coefficient of a message queue consumed by the data processing task node.

[0014] Preferably, obtaining the first data volume received and the second data volume processed by each data processing task node within a unit time includes:

[0015] For each data processing task node, the amount of data flowing into the input buffer area of ​​the data processing task node within a unit time is obtained from the memory database to obtain a first amount of data received by the data processing task node within a unit time;

[0016] The amount of data flowing into the output buffer area of ​​the data processing task node within a unit time is obtained from the memory database to obtain a second amount of data processed by the data processing task node within a unit time.

[0017] Preferably, distributing the real-time data to the message queue consumed by each of the data processing task nodes according to the distribution coefficient so that the data processing task node consumes the real-time data distributed to the message queue includes:

[0018] Determining the total amount of the real-time data to be distributed per unit time;

[0019] Calculate the amount of data received by the message queue consumed by each of the data processing task nodes per unit time according to the distribution coefficient and the total amount of the real-time data to be distributed per unit time;

[0020] The real-time data matching the data reception amount of the message queue is randomly distributed to the message queue, so that the data processing task node consumes the real-time data distributed to the message queue.

[0021] A second aspect of an embodiment of the present invention discloses a message queue dynamic balancing device, the device comprising:

[0022] A receiving unit, used to receive real-time data collected from a specified business page;

[0023] An acquisition unit, used for acquiring the first data volume received and the second data volume processed by each data processing task node within a unit time;

[0024] a determining unit, configured to determine a distribution coefficient of a message queue consumed by the data processing task node according to the first data volume and the second data volume;

[0025] A distribution unit is used to distribute the real-time data to the message queue consumed by each data processing task node according to the distribution coefficient, so that the data processing task node consumes the real-time data distributed to the message queue.

[0026] Preferably, the determining unit includes:

[0027] a calculation subunit, configured to calculate a ratio between the second data volume and the first data volume to obtain a back pressure value of the data processing task node;

[0028] The determination subunit is used to determine a coefficient corresponding to a preset range of the back pressure value to obtain a distribution coefficient of the message queue consumed by the data processing task node.

[0029] Preferably, the acquisition unit is specifically used to: for each data processing task node, acquire from the memory database the amount of data flowing into the input buffer area of ​​the data processing task node within a unit time, so as to obtain the first amount of data received by the data processing task node within a unit time;

[0030] The amount of data flowing into the output buffer area of ​​the data processing task node within a unit time is obtained from the memory database to obtain a second amount of data processed by the data processing task node within a unit time.

[0031] Preferably, the distribution unit comprises:

[0032] A determination subunit, used to determine the total amount of the real-time data to be distributed per unit time;

[0033] A calculation subunit, configured to calculate the amount of data received by the message queue consumed by each of the data processing task nodes per unit time according to the distribution coefficient and the total amount of the real-time data to be distributed per unit time;

[0034] The distribution subunit is used to randomly distribute the real-time data matching the data reception amount of the message queue to the message queue, so that the data processing task node consumes the real-time data distributed to the message queue.

[0035] A third aspect of an embodiment of the present invention discloses an electronic device, comprising: a processor and a memory, wherein the processor and the memory are connected via a communication bus; wherein the processor is used to call and execute a program stored in the memory; and the memory is used to store a program, wherein the program is used to implement the message queue dynamic balancing method disclosed in the first aspect of the embodiment of the present invention.

[0036] A fourth aspect of an embodiment of the present invention discloses a computer-readable storage medium, in which computer-executable instructions are stored. The computer-executable instructions are used to execute the message queue dynamic balancing method disclosed in the first aspect of the embodiment of the present invention.

[0037] Based on the above-mentioned embodiment of the present invention, a message queue dynamic balancing method, device, electronic device and storage medium are provided. The method is: receiving real-time data collected from a specified business page; obtaining the first data volume received and the second data volume processed by each data processing task node in a unit time; determining the distribution coefficient of the message queue consumed by the data processing task node according to the first data volume and the second data volume; distributing the real-time data to the message queue consumed by each data processing task node according to the distribution coefficient, so that the data processing task node consumes the real-time data distributed to the message queue. In this scheme, according to the first data volume received and the second data volume processed by the data processing task node in a unit time, the distribution coefficient of the message queue consumed by the data processing task node is calculated, and then the real-time data is distributed to the message queue according to the distribution coefficient. The data distribution volume can be dynamically adjusted without adjusting the number of message queues, thereby improving the operation stability and operation data processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0039] Figure 1 A flowchart of a message queue dynamic balancing method provided by an embodiment of the present invention;

[0040] Figure 2 An example diagram of the processing flow of credit card business data provided by an embodiment of the present invention;

[0041] Figure 3 An example diagram of the processing flow of a dynamic load balancer provided in an embodiment of the present invention;

[0042] Figure 4 A structural block diagram of a message queue dynamic balancing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0045] In order to better understand the following content, some terms are explained here:

[0046] Stream computing: Analyze large-scale flow data in real time during its ever-changing motion process, capture potentially useful information, and send the results to the next computing node.

[0047] Message queue (also known as message service queue): is a communication method between processes or different threads of the same process. The software queue is used to process a series of inputs. The message queue provides an asynchronous communication protocol. The sender and receiver of the message do not need to interact with the message queue at the same time. The message will be saved in the message queue until the receiver retrieves it.

[0048] Back pressure: Back pressure is a very common problem in real-time computing application development, especially in streaming computing. Back pressure means that a node in the data pipeline becomes a bottleneck and the processing rate cannot keep up with the rate at which data is sent upstream, so the upstream needs to be speed-limited.

[0049] Data skew: For cluster systems, cache is generally distributed, that is, different nodes are responsible for a certain range of cache data; insufficient dispersion of cache data results in a large amount of cache data being concentrated on one or several service nodes, which is called data skew.

[0050] When using stream computing technology to process bank operation data (such as credit card business data), performance bottlenecks often occur when different data processing processes process message queues, resulting in uneven data consumption and incorrect calculation results of real-time indicators. For example, when processing real-time data on the credit card application step page, each piece of data needs to be checked online with the marketing target database to determine whether it is the target population, which is used to generate a secondary marketing list. Due to the uneven consumption data capacity and the uncertainty of the time required for online query of the marketing target database, the consumption speed between various consumption tasks will be uneven. Once a consumption node reaches the consumption bottleneck, it will also cause data back pressure, affecting the timeliness of the marketing list.

[0051] More specifically, the current steps for processing a bank's credit card business data are as follows:

[0052] 1. Collect the data from the user on the credit card application page and send it to the message queue;

[0053] 2. Start a stream computing job in the stream computing platform. The job generates multiple consumer instances to consume the message queue. One consumer instance corresponds to one message queue.

[0054] 3. Use the credit card application page information to access the marketing target database online to check whether it is on the marketing list;

[0055] 4. Generate marketing lists and push them to the marketing team for real-time marketing.

[0056] However, in step 3 of processing credit card business data, the data processing process often requires online access to external data or involves complex logic calls, which will lead to uncertain processing time for a single piece of data, and even uncertain number of data processed per unit time. When a large amount of data takes a long time to process, performance bottlenecks are likely to occur, the timeliness of data processing becomes low, and marketers cannot obtain the latest target customer list.

[0057] In response to the above problems, two existing processing methods are currently used to alleviate the above problems. The first existing processing method is: to expand the number of consumption queues by expanding resource capacity, and at the same time expand the number of data consumption tasks, so that each data consumption task processes less data.

[0058] The second existing processing method is to break up the data to make the load more balanced, thereby avoiding data skew and hot spots.

[0059] However, the first existing processing method involves stopping the original job and re-issuing a new job, which causes a short-term processing interruption to the job data processing. At the same time, since the performance bottleneck only exists in some consumption tasks, blindly expanding the number of tasks will cause a certain waste of resources; restarting and stopping the job will also cause data processing to be temporarily stopped, and there is a certain probability of failure when the job is re-published, resulting in reduced job stability and reduced business service capabilities, which does not meet the business requirements for indicator processing stability and data accuracy.

[0060] The second existing processing method, data fragmentation, will increase network overhead to a certain extent.

[0061] Therefore, this scheme proposes a message queue dynamic balancing method, device, electronic device and storage medium. According to the first data amount received by the data processing task node in unit time and the second data amount processed, the distribution coefficient of the message queue consumed by the data processing task node is calculated, and then the real-time data is distributed to the message queue according to the distribution coefficient. The data distribution volume can be dynamically adjusted without adjusting the number of message queues, thereby improving the operation stability and operation data processing effect, and there is no need to break up the data to avoid increasing network overhead.

[0062] See also Figure 1 , shows a flow chart of a message queue dynamic balancing method provided by an embodiment of the present invention, the message queue dynamic balancing method is applicable to a dynamic load balancing device in a message middleware, and the message queue dynamic balancing method includes:

[0063] Step S101: receiving real-time data collected from a designated business page.

[0064] In the specific implementation of step S101, real-time data collected from a designated business page is received, for example, real-time data collected from a credit card application page and sent by the credit card application page is received.

[0065] Step S102: Obtain the first data volume received and the second data volume processed by each data processing task node within a unit time.

[0066] It should be noted that a corresponding number of data processing task nodes are created in advance according to the number of distributed message queues, and the number of data processing task nodes corresponds to the number of message queues; the data processing task nodes correspond to the message queues, and the data processing task nodes act as consumers to consume the data in the corresponding message queues.

[0067] It should be further explained that the data processing task node includes at least three parts: input buffer area (also called input buffer queue), data consumption processing, and output buffer area (also called output buffer queue); the way in which the data processing task node consumes data from the message queue is as follows: the data processing task node consumes the data to be processed from the message queue, and the data to be processed flows into the input buffer area of ​​the data processing task node, and then obtains the data from the input buffer area for data processing (data consumption processing part), and after the data processing is completed, the processing result flows into the output buffer area.

[0068] In the specific implementation of step S102, for each data processing task node, the amount of data flowing into the input buffer area of ​​the data processing task node per unit time is obtained from the memory database to obtain the first amount of data received by the data processing task node per unit time (recorded as L1 i ), i represents data processing task node i;

[0069] The amount of data flowing into the output buffer of the data processing task node per unit time is obtained from the memory database to obtain the second amount of data processed by the data processing task node per unit time (recorded as L2 i ).

[0070] It should be noted that for each data processing task node, the amount of data flowing into the input buffer area of ​​the data processing task node per unit time is collected in advance, and the amount of data flowing into the output buffer area of ​​the data processing task node per unit time is collected in advance, and the collected data amounts are stored in (for example, cached in) a memory database.

[0071] It should be further explained that, for each data processing task node, the amount of data (first data amount) flowing into the input buffer area of ​​the data processing task node per unit time represents: the amount of data consumed by the data processing task node from the corresponding message queue (the amount of data to be processed), or in other words, the amount of data received by the message queue corresponding to the data processing task node from upstream distribution.

[0072] The amount of data (second data amount) flowing into the output buffer of the data processing task node within a unit time represents: the amount of data processed by the data processing task node.

[0073] Step S103: Determine the distribution coefficient of the message queue consumed by the data processing task node according to the first data volume and the second data volume.

[0074] In the process of implementing step S103, for each data processing task node, after obtaining the first data volume and the second data volume of the data processing task node, the distribution coefficient of the message queue corresponding to the data processing task node is determined according to the first data volume and the second data volume of the data task processing node.

[0075] In some embodiments, the specific method of determining the distribution coefficient of the message queue is: for each data processing task node, calculate the second data volume L2 of the data processing task node i With the first data volume L1 i The ratio between them is used to obtain the back pressure value of the data processing task node (denoted as R i ), that is, R i =L2 i / L1 i ;

[0076] The back pressure value of a data processing task node can be used to reflect whether the data processing task node has reached a performance bottleneck. Specifically, for a data processing task node, if the back pressure value R i If it is greater than or equal to 1, it indicates that the current data processing speed of the data processing task node is greater than or equal to "the amount of data distributed upstream received by the message queue corresponding to the data processing task node", that is, the data processing speed is greater than the data receiving speed, and the data processing task node has sufficient performance;

[0077] On the contrary, if the back pressure value R of the data processing task node i If it is less than 1, it indicates that the current data processing speed of the data processing task node is less than "the amount of data distributed upstream and received by the message queue corresponding to the data processing task node", that is, the data processing speed cannot keep up with the data receiving speed, and the data processing task node has reached a performance bottleneck.

[0078] After calculating the back pressure value of each data processing task node, for each data processing task node, determine the coefficient corresponding to the preset range of the back pressure value of the data processing task node to obtain the distribution coefficient of the message queue consumed by the data processing task node (denoted as A i ), A i is the distribution coefficient of the message queue corresponding to the data processing task node i.

[0079] That is, the distribution coefficient of the message queue corresponding to a certain data processing task node is: the coefficient corresponding to the preset range of the back pressure value of the data processing task node.

[0080] Specifically, multiple preset ranges related to the back pressure value are preset, and a corresponding coefficient is set for each preset range. The coefficient corresponding to the preset range can be used as a distribution coefficient of the message queue;

[0081] After calculating the back pressure value of the data processing task node, first determine in which preset range the back pressure value is (or which preset range covers the calculated back pressure value). The coefficient corresponding to the preset range in which the back pressure value is located is used as the "distribution coefficient of the message queue corresponding to the data processing task node".

[0082] For example, for a data processing task node i, the back pressure value R of the data processing task node i is calculated by the above method. i , then the distribution coefficient A of the message queue corresponding to the consumption of data processing task node i i Including the following situations: When R i ≧1, A i =1; when 0.5≦R i <1, A i =0.5; when 0.1≦R i <0.5, A i =0.1; when R i <0.1, A i =0.

[0083] Step S104: Distribute the real-time data to the message queue consumed by each data processing task node according to the distribution coefficient, so that the data processing task node consumes the real-time data distributed to the message queue.

[0084] In the process of implementing step S104, after calculating the distribution coefficient of the message queue corresponding to each data processing task node, the real-time data received in step S101 is randomly distributed to each message queue according to the distribution coefficient of each message queue, so that each data processing task node consumes the real-time data in the corresponding message queue respectively.

[0085] In some embodiments, the specific implementation method of distributing real-time data to each message queue is: determining the total amount of real-time data to be distributed per unit time; calculating the data reception amount of the message queue consumed by each data processing task node per unit time based on the distribution coefficient and the total amount of real-time data to be distributed per unit time; randomly distributing real-time data that matches the data reception amount of the message queue to the message queue, so that the data processing task node consumes the real-time data distributed to the message queue.

[0086] Specifically, first determine the total amount of real-time data to be distributed per unit time. For the message queue corresponding to each data processing task node, calculate the data reception amount (the amount of real-time data received) of the message queue per unit time according to the "total amount of real-time data to be distributed per unit time" and the distribution coefficient of the message queue, and distribute the real-time data that matches the data reception amount of the message queue to the message queue.

[0087] For example, assuming there are n data processing task nodes in total, then the distribution coefficients of the message queues corresponding to the n data processing task nodes are A 1 To A n , the total amount of real-time data to be distributed per unit time is S; for data processing task node i (i is greater than or equal to 1 and less than or equal to n), the data receiving amount S of the message queue corresponding to the data processing task node i i =A i (S / (A 1 +A 2 +…+A n )).

[0088] From the above content about back pressure value and distribution coefficient, it can be seen that when the performance pressure of the data processing task node is large, the distribution coefficient of the message queue corresponding to the data processing task node is small, so that less real-time data can be sent to the message queue corresponding to the data processing task node;

[0089] On the contrary, when the performance of the data processing task node is sufficient, the distribution coefficient of the message queue corresponding to the data processing task node is larger, so that more real-time data can be sent to the message queue corresponding to the data processing task node.

[0090] In an embodiment of the present invention, the distribution coefficient of the message queue consumed by the data processing task node is calculated according to the first data volume received by the data processing task node in unit time and the second data volume processed, and then the real-time data is distributed to the message queue according to the distribution coefficient. The data distribution volume can be dynamically adjusted without adjusting the number of message queues, thereby improving the operation stability and the operation data processing effect.

[0091] To better understand the above embodiments of the present invention Figure 1 The following is an example of the complete credit card business data processing process. Figure 2 and Figure 3 Let's take an example:

[0092] See also Figure 2 , shows an example diagram of the processing flow of credit card business data provided by an embodiment of the present invention, Figure 2It includes the following seven parts: "1. Credit card application page", "2. Dynamic load balancer", "3. Distributed message service queue", "4. Data processing task node", "5. Index calculation task", "6. Collected data volume", and "7. Result output". The contents of these seven parts are explained below.

[0093] 1. Credit card application page: The credit card application page collects real-time data from the credit card application page and sends the collected real-time data to the dynamic load balancer of the message middleware.

[0094] 2. Dynamic load balancer: such as Figure 3 As can be seen from the example diagram of the processing flow of the dynamic load balancer, the dynamic load balancer includes the following four parts: "2.1. Data receiver", "2.2. Calculation of back pressure value Ri", "2.3. Adjustment of distribution coefficients Ai of each message queue", and "2.4. Data distribution".

[0095] 2.1. Data receiver: Receives real-time data sent from the credit card application page.

[0096] 2.2. Calculate the back pressure value Ri: Obtain the first amount of data (L1 i , Figure 3 The second data volume (L2 i , Figure 3 L2 value in Chinese); using R i =L2 i / L1 i Calculate the back pressure value Ri of the data processing task node.

[0097] 2.3. Adjust the distribution coefficient Ai of each message queue: Use the back pressure value R of the data processing task node i , respectively determine the distribution coefficient A of the message queue corresponding to each data processing task node i ; Among them, when R i ≧1, A i =1; when 0.5≦R i <1, A i =0.5; when 0.1≦R i <0.5, A i =0.1; when R i <0.1, A i =0.

[0098] 2.4. Data distribution: According to the distribution coefficient Ai of the determined message queue, the data receiving amount S of each message queue is calculated i=A i (S / (A 1 +A 2 +…+A n )); The dynamic load balancer randomly sends real-time data matching the data reception amount to each message queue according to the distribution coefficient Ai of each message queue.

[0099] 3. Distributed message service queue: The distributed message service queue (that is, the message queue mentioned above) receives real-time data sent by the dynamic load balancer.

[0100] 4. Data processing task nodes: such as Figure 2 As shown, the data processing task node includes the following three parts: "4.1, input buffer area", "4.2, data consumption processing", and "4.3, output buffer area".

[0101] 4.1. Input buffer: Get real-time data from the corresponding message queue into the input buffer.

[0102] 4.2. Data consumption and processing: Obtain real-time data from the input buffer for processing.

[0103] 4.3. Output buffer: The results of real-time data processing are stored in the output buffer.

[0104] 5. Index calculation task: obtain the "results after real-time data processing" from the output buffer area, and further process the "results after real-time data processing" to generate index results.

[0105] 6. Collect data volume: Collect the amount of data flowing into the input buffer and the output buffer per unit time (that is, collect L1 value and L2 value), and store the collected data volume in the memory database.

[0106] 7. Result output: Output the indicator results to the data repository.

[0107] It should be noted that the above Figure 2 and Figure 3 The execution principle of the present invention can be found in the above-mentioned embodiment of the present invention. Figure 1 The contents of each step will not be repeated here.

[0108] From the contents of the above embodiments, it can be seen that the present solution places a dynamic load balancer in front of the message queue. The dynamic load balancer receives real-time data sent from the producer and calculates the back pressure value of the data processing task node as a consumer. The back pressure value represents the degree of back pressure of the data processing task node. The dynamic load balancer dynamically adjusts the amount of data sent to each message queue according to the back pressure value of the data processing task node. If the back pressure of the data processing task node is severe, less data is sent. If the back pressure of the data processing task node is not severe, more data is sent. In this way, the data distribution volume can be adjusted dynamically and in real time, thereby solving the performance bottleneck problem of the data processing task node from the source and ensuring the stability of job operation.

[0109] Corresponding to the message queue dynamic balancing method provided by the above embodiment of the present invention, see Figure 4 , the embodiment of the present invention also provides a structural block diagram of a message queue dynamic balancing device, the message queue dynamic balancing device is suitable for a dynamic load balancing device in a message middleware, the message queue dynamic balancing device includes: a receiving unit 100, an acquiring unit 200, a determining unit 300 and a distributing unit 400;

[0110] The receiving unit 100 is used to receive real-time data collected from a specified business page.

[0111] The acquisition unit 200 is used to acquire the first data volume received and the second data volume processed by each data processing task node within a unit time.

[0112] In a specific implementation, the acquisition unit 200 is specifically used to: for each data processing task node, acquire the amount of data flowing into the input buffer area of ​​the data processing task node within a unit time from the memory database to obtain the first amount of data received by the data processing task node within a unit time;

[0113] The amount of data flowing into the output buffer area of ​​the data processing task node within a unit time is obtained from the memory database to obtain a second amount of data processed by the data processing task node within a unit time.

[0114] The determination unit 300 is used to determine the distribution coefficient of the message queue consumed by the data processing task node according to the first data volume and the second data volume.

[0115] The distribution unit 400 is used to distribute the real-time data to the message queue consumed by each data processing task node according to the distribution coefficient, so that the data processing task node consumes the real-time data distributed to the message queue.

[0116] In an embodiment of the present invention, the distribution coefficient of the message queue consumed by the data processing task node is calculated according to the first data volume received by the data processing task node in unit time and the second data volume processed, and then the real-time data is distributed to the message queue according to the distribution coefficient. The data distribution volume can be dynamically adjusted without adjusting the number of message queues, thereby improving the operation stability and the operation data processing effect.

[0117] Preferably, combined Figure 4 The content shown, the determination unit 300 includes a calculation subunit and a determination subunit, and the execution principle of each subunit is as follows:

[0118] The calculation subunit is used to calculate the ratio between the second data volume and the first data volume to obtain the back pressure value of the data processing task node.

[0119] The determination subunit is used to determine a coefficient corresponding to a preset range of the back pressure value to obtain a distribution coefficient of a message queue consumed by a data processing task node.

[0120] Preferably, combined Figure 4 As shown in the figure, the distribution unit 400 includes a determination subunit, a calculation subunit and a distribution subunit, and the execution principle of each subunit is as follows:

[0121] The determination subunit is used to determine the total amount of real-time data to be distributed per unit time.

[0122] The calculation subunit is used to calculate the data reception amount of the message queue consumed by each data processing task node per unit time according to the distribution coefficient and the total amount of real-time data to be distributed per unit time.

[0123] The distribution subunit is used to randomly distribute the real-time data matching the data receiving amount of the message queue to the message queue, so that the data processing task node consumes the real-time data distributed to the message queue.

[0124] Preferably, the present invention also provides an electronic device, including: a processor and a memory, the processor and the memory are connected via a communication bus; wherein the processor is used to call and execute a program stored in the memory; the memory is used to store the program, and the program is used to implement the message queue dynamic balancing method provided in the above method embodiment.

[0125] Preferably, the present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored, and the computer-executable instructions are used to execute the message queue dynamic balancing method provided by the above method embodiment.

[0126] In summary, the embodiments of the present invention provide a message queue dynamic balancing method, device, electronic device and storage medium. According to the first data amount received by the data processing task node in unit time and the second data amount processed, the distribution coefficient of the message queue consumed by the data processing task node is calculated, and then the real-time data is distributed to the message queue according to the distribution coefficient. The data distribution amount can be dynamically adjusted without adjusting the number of message queues, thereby improving the operation stability and the operation data processing effect.

[0127] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0128] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0129] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A message queue dynamic balancing method, characterized in that: The method comprises: Receive real-time data collected from designated business pages; Obtaining the first data volume received and the second data volume processed by each data processing task node within a unit time; Determining a distribution coefficient of a message queue consumed by the data processing task node according to the first data volume and the second data volume; The real-time data is distributed to the message queue consumed by each of the data processing task nodes according to the distribution coefficient, so that the data processing task node consumes the real-time data distributed to the message queue.

2. The method according to claim 1, characterized in that Determining a distribution coefficient of a message queue consumed by the data processing task node according to the first data volume and the second data volume includes: Calculating a ratio between the second data volume and the first data volume to obtain a back pressure value of the data processing task node; A coefficient corresponding to a preset range in which the back pressure value is located is determined to obtain a distribution coefficient of a message queue consumed by the data processing task node.

3. The method according to claim 1, characterized in that: Obtaining the first data volume received and the second data volume processed by each data processing task node within a unit time, including: For each data processing task node, the amount of data flowing into the input buffer area of ​​the data processing task node within a unit time is obtained from the memory database to obtain a first amount of data received by the data processing task node within a unit time; The amount of data flowing into the output buffer area of ​​the data processing task node within a unit time is obtained from the memory database to obtain a second amount of data processed by the data processing task node within a unit time.

4. The method according to claim 1, characterized in that: Distributing the real-time data to the message queues consumed by the data processing task nodes according to the distribution coefficient, so that the data processing task nodes consume the real-time data distributed to the message queues, includes: Determining the total amount of the real-time data to be distributed per unit time; Calculate the amount of data received by the message queue consumed by each of the data processing task nodes per unit time according to the distribution coefficient and the total amount of the real-time data to be distributed per unit time; The real-time data matching the data reception amount of the message queue is randomly distributed to the message queue, so that the data processing task node consumes the real-time data distributed to the message queue.

5. A message queue dynamic balancing device, characterized in that: The device comprises: A receiving unit, used to receive real-time data collected from a specified business page; An acquisition unit, used for acquiring the first data volume received and the second data volume processed by each data processing task node within a unit time; a determining unit, configured to determine a distribution coefficient of a message queue consumed by the data processing task node according to the first data volume and the second data volume; A distribution unit is used to distribute the real-time data to the message queue consumed by each data processing task node according to the distribution coefficient, so that the data processing task node consumes the real-time data distributed to the message queue.

6. The device according to claim 5, characterized in that The determining unit comprises: a calculation subunit, configured to calculate a ratio between the second data volume and the first data volume to obtain a back pressure value of the data processing task node; The determination subunit is used to determine a coefficient corresponding to a preset range of the back pressure value to obtain a distribution coefficient of the message queue consumed by the data processing task node.

7. The device according to claim 5, characterized in that The acquisition unit is specifically used to: for each data processing task node, acquire from the memory database the amount of data flowing into the input buffer area of ​​the data processing task node within a unit time, so as to obtain the first amount of data received by the data processing task node within a unit time; The amount of data flowing into the output buffer area of ​​the data processing task node within a unit time is obtained from the memory database to obtain a second amount of data processed by the data processing task node within a unit time.

8. The device according to claim 5, characterized in that The distribution unit comprises: A determination subunit, used to determine the total amount of the real-time data to be distributed per unit time; A calculation subunit, configured to calculate the amount of data received by the message queue consumed by each of the data processing task nodes per unit time according to the distribution coefficient and the total amount of the real-time data to be distributed per unit time; The distribution subunit is used to randomly distribute the real-time data matching the data reception amount of the message queue to the message queue, so that the data processing task node consumes the real-time data distributed to the message queue.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the processor and the memory are connected via a communication bus; wherein the processor is used to call and execute a program stored in the memory; The memory is used to store a program, and the program is used to implement the message queue dynamic balancing method as described in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the message queue dynamic balancing method described in any one of claims 1-4.

Images