Sentinel-based high-performance mass data cluster throttling method and system

Abstract

The patent provides a high-performance mass data cluster throttling method and system based on sentinel, which comprises the following steps: (1) separate the data of multiple platforms separately, which realizes concurrent operation, and schedules the producers of each platform at regular intervals; (2) when calling the producer, each platform has its own producer, and when the producer queries the data, the data queried from the database is different, and the concurrent multiple platform query operation is controlled; (3) after the platform produces data, the data is sent to the message middleware, the message middleware forwards the data from the producer to the corresponding queue binding consumer, and realizes concurrent consumption of the data produced by the same platform; (4) after the consumer receives the data from the message middleware, the request is encapsulated according to the data to call the third-party interface, and the cluster throttling control mode is adopted to process the data. It has intelligence and high performance, and can accurately and flexibly manage mass data.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of intelligent data processing of Internet of Things, and particularly relates to a high-performance mass data cluster flow limiting method and system based on sentinel. BACKGROUND

[0002] The existing Internet of Things data synchronization and flow limiting strategy is basically time polling, which queries the database to obtain existing data needing synchronization without interruption. However, the following situations may occur.

[0003] 1. Data synchronization delay problem. The existing technical solution is that there is only one producer for the data of all platforms, although there are multiple consumers to obtain, but there is only one consumer entry, and then the data is distributed to call the third party. If this method is used, it is impossible to synchronize the data back in a day or even several days, and the number of calls is very limited according to the flow limiting rules of the third party, so the data obtained is very delayed, and the delay brings the problem of inaccurate data.

[0004] 2. System pressure and performance problem. The existing system adopts time polling, and basically executes when it is started, and the internal specific execution is executed through a dead loop. This synchronization method is very wasteful of computer CPU resources, because it is executed without interruption, which is equivalent to not resting, but the resource utilization rate is not high, because it sleeps for a few seconds when it encounters flow limiting. This method locks resources and does not release them, which brings great pressure and performance problems to the system.

[0005] 3. Uncontrollable flow limiting strategy problem. The existing technology cannot separately limit the flow of a certain platform, or even a certain account or interface, that is, the existing data flow limiting is uncontrollable, and is passively adjusted according to the information returned by the third party. SUMMARY

[0006] In view of the above problems of the prior art, the present application provides a high-performance mass data cluster flow limiting method and system based on sentinel, which has intelligence and high performance, and can accurately and flexibly manage mass data.

[0007] The present application provides a high-performance mass data cluster flow limiting method based on sentinel, which comprises the following steps:

[0008] (1) Separate the data of multiple platforms, so as to realize concurrent operation, and schedule the producers of each platform at a time;

[0009] (2) When calling the producer, each platform has its own producer, which queries the data, and the data queried from the database is different, and multiple platform query operations are controlled and concurrent;

[0010] (3) When the platform produces data, it is then sent to the message middleware rabbitmq, which forwards the data from the producer to the corresponding queue-bound consumer, realizing concurrent consumption of the data produced by the same platform;

[0011] (4) After the consumer receives the data from the rabbitmq, it sends an encapsulation request to call the third-party interface according to the data, and uses a cluster flow control method to process the data.

[0012] Further, the step (1) is to separate the data of each platform, and each platform is independently and concurrently run, and then a distributed task scheduling platform xxl-job is used to schedule the producers of each platform.

[0013] Further, in the step (3), the rabbitmq message middleware forwards the data from the producer to the corresponding queue-bound consumer, and the same platform data can be consumed concurrently through the rabbitmq, which uses at least two hosts to consume data.

[0014] Further, in the step (4), a cluster flow control method is used to process data, specifically based on sentinel technology, multiple hosts are deployed to realize a cluster, and each host has multiple consumers for each account of each platform, and each account is set to multiple consumers to consume data;

[0015] Then set the flow control rules, each account can set its own flow control rules, and these cluster flow control rules are saved in the redis memory and can be modified and controlled at any time;

[0016] Then perform cluster flow control, each consumer will be intercepted by the cluster flow control system with the set flow control rules when calling the third-party data, and it is judged whether it can pass the flow control rules, if it passes, it can continue to call the third-party interface, and a confirmation message is sent to the rabbitmq to tell that the data has been consumed; if it is rejected and cannot pass, it is directly returned and cannot call the third-party interface.

[0017] Further, each account can set its own flow control rules, and the flow control rules of multiple consumers of the same account can be flexibly set to be the same, and the flow control rules of different accounts can be set differently.

[0018] The application also provides a high-performance mass data cluster flow limiting system based on sentinel, which comprises the following:

[0019] The data concurrent operation module is used for separating the data of multiple platforms separately, thereby realizing concurrent operation and timing scheduling of the producers of each platform.

[0020] The concurrent platform query control module is used for having a producer for each platform when the producer is called, and the data queried from the database is different when the producer queries data, thereby controlling concurrent multiple platform query operations.

[0021] The data concurrent consumption control module is used for sending the data generated by the platform to the message middleware rabbitmq, forwarding the data from the producer to the corresponding queue binding consumer, and realizing concurrent consumption of the data generated by the same platform.

[0022] The cluster flow limiting control module is used for encapsulating the request according to the data from the rabbitmq to call the third-party interface and processing the data by using the cluster flow limiting control mode when the consumer receives the data from the rabbitmq.

[0023] Further, the cluster flow limiting control module is also used for realizing the cluster by deploying multiple hosts based on the sentinel technology, and there can be multiple consumers for each account of each platform on each host, and each account is set to multiple consumers to consume data.

[0024] Then, the flow limiting rule is set, each account can set its own flow limiting rule, and the cluster flow limiting rule is saved in the redis memory and can be modified and controlled at any time.

[0025] The cluster flow limiting control is further performed, each consumer is intercepted by the cluster flow limiting control system with the set flow limiting rule when calling the third-party data, it is judged whether the flow limiting rule can be passed, if yes, the third-party interface can be continuously called, and a confirmation message is sent to the rabbitmq to tell that the data is consumed, if no, the third-party interface cannot be called.

[0026] The method and system of the application have the following advantages:

[0027] 1. The cluster flow limiting control method of the application can accurately and conveniently control the flow of large data, the data flow is controllable, and passive operation is not relied on the return information of the third party, and the flow limiting method and system of the application can monitor the data in real time and adjust the flow limiting rule at any time.

[0028] 2. The system performance is greatly improved and the system pressure is reduced. The application uses multiple host clusters, each platform, and each account of each platform is not affected, and the data flow is executed in parallel, fully utilizing the system resources. The timing task reduces the resource waste caused by the program dead loop, and the data flow control is controlled by the cluster flow control strategy, so the performance is very high.

[0029] 3. The data delay problem is greatly reduced. The method and system of the application realize data synchronization by implementing the cluster flow control strategy and parallel running strategy, reduce the system pressure and provide system performance, solve the data delay problem, and improve the data processing speed by 12%, and greatly improve the accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The application provides a high-performance massive data cluster flow control method based on sentinel.

[0031] Figure 2 The application provides a cluster flow control method sub-step control method flowchart.

[0032] Figure 3 The application provides a cluster flow control method monitoring data processing result page diagram. DETAILED DESCRIPTION

[0033] The application provides a high-performance massive data cluster flow control method based on sentinel.

[0034] As shown in the drawings, the application provides a high-performance massive data cluster flow control method based on sentinel, which includes the following steps: Figures 1-3 (1) Separate the data of multiple platforms, which realizes concurrent running and timing scheduling of the producers of each platform.

[0035] For different operator platforms, such as

[0036] Figure 1 ​The data of each platform is separated, and the data between platforms is not affected, and each platform runs independently, so that concurrent operation is realized. A distributed task scheduling platform xxl-job is adopted to schedule the producers of each platform at regular intervals. The so-called regular interval refers to setting a few seconds or minutes for scheduling (the time can be flexibly set long or short), and the program is not in a loop, so the scheduling process will not fall into a dead loop and will not consume system cpu resources. For example, the xxl-job timer is set to trigger scheduling of platform A to query data every 5 seconds. After the current time triggers the scheduling once, the timer will continue to trigger the next scheduling of platform A to query data every 5 seconds, and the trigger scheduling of each platform is not affected.

[0037] (2) When the xxl-job calls the producer, each platform has its own producer. When the producer queries the data, because the platforms are different, the data queried from the database is different. Therefore, multiple platforms query respectively, which can improve the efficiency of querying data, and is equivalent to controlling concurrent multiple platform query operations.

[0038] (3) When each platform generates data, the data is sent to the message middleware rabbitmq. The rabbitmq forwards the data from the producer to the corresponding queue binding consumer to realize concurrent consumption of the data produced by the same platform.

[0039] Rabbitmq is an open source message broker software (also known as message-oriented middleware) that implements the Advanced Message Queuing Protocol (AMQP). The RabbitMQ server is written in Erlang language, and clustering and failover are built on the Open Telecom Platform framework. There are client library interfaces for all major programming languages. This step introduces rabbitmq to realize concurrent consumption of data produced by the same platform. If the data volume is large, the production data will be produced quickly, and if only one host is used to run, it will take a long time to consume. Therefore, at least two hosts are used to consume data, and the number of hosts can be expanded horizontally according to the data volume.

[0040] (4) After the consumer receives the data from the rabbitmq, the request is encapsulated according to the data to call the third-party interface, and the cluster flow control method is used to process the data.

[0041] As Figure 2As shown in the cluster flow control mode process diagram of the sub-step of the method of the present application, in order to solve the uncontrollable problem of flow limiting, the scheme adopts a cluster flow limiting strategy, specifically based on the sentinel technology, which can very flexibly and conveniently control flow limiting, at any time increase or decrease the rate of flow, and also can control the number of concurrents.

[0042] Sub-step (I) is to realize the cluster by deploying multiple hosts, such as deploying two machines, which are cluster machine 1 and cluster machine 2, and on each host, there can be multiple consumers for each account of each platform (each platform has multiple accounts because there are different cities, so each platform has multiple accounts, which call the interface according to the account), and each account is set to multiple consumers to consume data, such as Figure 2 is cluster machine 1-consumer 1 and cluster machine 1-consumer 2.

[0043] Sub-step (II) is to set the flow limiting rules, because each platform and each account are separated and have their own consumers, so each account can set its own flow limiting rules, and the flow limiting rules of multiple consumers of the same account are the same, while the flow limiting rules of different accounts can be set differently, which can be flexibly controlled and set, and these flow limiting rules are saved in the redis memory and can be modified and controlled at any time; if the third-party interface flow limiting rules are changed, only the targeted matching change is needed, without the need to modify the system code as a whole, which is very convenient and time-saving, and the method is flexible and has a wide range of adaptability.

[0044] For example, the third-party interface A originally limits the system call rate to 30 times per second, but now the system call rate is increased to 60 times per second. At this time, the method of the present application only needs to change the system limiting rate from the original 30 to 60, which can be flexibly modified and take effect.

[0045] Sub-step (III) is to perform cluster flow control, each consumer will be intercepted by the cluster flow control system with the set flow limiting rules when calling the third-party data, multiple machines and multiple consumers form a cluster control, and it is judged whether it can pass the flow limiting rules interception, and the consumers of different machines of the same account of the same platform will pass through the same flow limiting rules, if it passes, it can continue to call the third-party interface and send a confirmation message to rabbitmq to tell that the data has been consumed; if it is rejected and cannot pass, it directly returns and cannot call the third-party interface, and the consumer also sends a message to rabbitmq to tell it that the consumption is unsuccessful. In this way, rabbitmq will not lose data when it pushes data again next time. The cluster flow control is realized in the following way: the account flow limiting rules are described in the above sub-step (II), and the cluster flow limiting also has rules, for example as follows:

[0046] [{"clientSet":["10.20.11.161@8720","10.20.11.161@8721","10.20.11.161@8722"],"ip":"10.20.11.161","machineId":"10.20.11.161@8720","port":17631}].

[0047] The above is the cluster rule, which is stored in redis. The clientSet contains the IP and port of all cluster machines, and the external IP is the master IP of the cluster machine. When each machine starts, it needs to determine whether it is already in the cluster. If it is in the cluster, it does not need to join. If it is not in the cluster, it needs to be added to the cluster, that is, it needs to be added to the cluster control rule. At the same time, messages are sent through the redis channel, and each machine needs to listen to this channel. When a message is received, the cluster control rule in the memory of each machine is updated, so that it knows that other machines have joined the cluster. At the same time, the cluster control rules of all clusters are consistent, so that the cluster control is realized. The cluster flow control is combined with the flow control rule in sub-step (ii) to count the number of passes of multiple consumers on multiple machines to determine whether to pass or reject. If it exceeds the threshold set in the flow control rule, it is rejected. Otherwise, it is passed.

[0048] The above cluster rule mainly limits the total number of concurrent calls to third-party interfaces on a single platform. The main machine IP is set to collect and count the number of concurrent calls on other machines. For example, there are three machines, machine A, machine B, and machine C, and the total flow rate is 30 times per second. Because it is a concurrent call to the third-party interface, the performance of each machine is different. Some machines may be faster, and some machines may be slower. At this time, set a main machine A to count the rate of the three machines, and then add up the rates of the three machines to calculate the total number. The total number cannot exceed the total flow rate of 30 times per second.

[0049] As shown in Figure 3 , the effect of the cluster flow control strategy and the advantages it brings. Through the system monitoring page, you can observe and see the number of passes through the flow control rule and the number of rejections per second. The waveform on the left side of the upper part of the graph is the number of rejections, and the waveform on the left side of the lower part of the graph is the number of passes. The horizontal axis represents time, and the vertical axis represents the number. Because the maximum number of passes per second is set to 10, that is, the threshold is 10, if the number of messages pushed by the rabbitmq middleware is greater than 10, the remaining messages are rejected.

[0050] The application also provides a high-performance mass data cluster flow limiting system based on sentinel, which comprises the following steps:

[0051] The data concurrent running module is used for separating the data of multiple platforms separately, so that concurrent running is realized, and the producers of the platforms are scheduled in a time manner;

[0052] The concurrent platform query control module is used for when the producers are called, each platform has its own producer, and when the producers query data, the data queried from the database is different, and the concurrent multiple platform query operations are controlled;

[0053] The data concurrent consumption control module is used for when the platform produces data, the data is sent to the message middleware rabbitmq, the rabbitmq forwards the data from the producers to the corresponding queue binding consumers, and the concurrent consumption of the data produced by the same platform is realized.

[0054] The cluster flow limiting control module is used for when the consumers receive the data from the rabbitmq, the request is encapsulated according to the data to call the third-party interface, and the cluster flow limiting control mode is used to process the data.

[0055] Further, the cluster flow limiting control module is also used for realizing the cluster by deploying multiple hosts based on the sentinel technology, and each account of each platform on each host can have multiple consumers, and each account is set to multiple consumers to consume data.

[0056] Then, the flow limiting rules are set, each account can set its own flow limiting rules, and the cluster flow limiting rules are saved in the redis memory and can be modified and controlled at any time.

[0057] The cluster flow limiting control is further performed, each consumer is intercepted by the cluster flow limiting control system with the set flow limiting rules when calling the third-party data, it is judged whether the flow limiting rules can be passed, if yes, the third-party interface can be continuously called, and a confirmation message is sent to the rabbitmq to tell that the data is consumed, and if not, the third-party interface cannot be called.

[0058] The method and system of the application realize data synchronization by using the cluster flow limiting strategy and the parallel running strategy, reduce system pressure, provide system performance, and solve the data delay problem.

[0059] By joining the cluster of multiple computers, a certain strategy is needed, the first one becomes the master server, and the others join according to the strategy to judge whether the message needs to be sent through the redis channel function to inform other clusters, so as to realize the machine cluster. Different platforms and different accounts are run in parallel to improve efficiency and system performance. In terms of data flow limiting, the limiting rules stored in redis and the machine cluster are used to intercept and judge the strategy. If it is passed, it will continue to call; if it is rejected, it will be intercepted and returned, greatly saving system resources and improving data processing efficiency.

[0060] The above is only the preferred specific implementation of the patent, but the protection scope of the patent is not limited to this. Any skilled person in the art can easily think of changes or replacements within the technical scope disclosed by the patent, which should be covered within the protection scope of the patent.

Claims

1. A sentinel-based high-performance mass data cluster throttling method, characterized by comprising: (1) for different operator platforms, separate the data of each platform, independently and concurrently run, and then use the distributed task scheduling platform xxl-job to schedule the producers of each platform at regular intervals; (2) when the producer is called, each platform has its own producer, and when the producer queries data, the data queried from the database is different, and the concurrent multiple platform query operation is controlled; (3) when the platform generates data, it is sent to the message middleware rabbitmq, the message middleware rabbitmq forwards the data from the producer to the corresponding queue-bound consumer, and the data produced by the same platform is consumed concurrently through the rabbitmq, which uses at least two hosts to consume data; (4) after the consumer receives the data from the rabbitmq, it sends an encapsulation request to call the third-party interface according to the data, and uses a cluster throttling control method to process data; In step (4), the cluster throttling control method is used to process data, which is based on sentinel technology and uses multiple hosts to implement a cluster, and each host has multiple consumers for each account of each platform, and each account is set to consume data by the multiple consumers; Then set the throttling rules, set the throttling rules for each account, and save these cluster throttling rules in the redis memory for real-time modification and control; Then perform cluster throttling control, each consumer will be intercepted by the cluster throttling control system with set throttling rules when calling third-party data, and determine whether it can pass the throttling rules, if it can, it will be released, continue to call the third-party interface, and send a confirmation message to the rabbitmq to tell that the data has been consumed; if it is rejected, it cannot call the third-party interface. Each account is set with its own throttling rules, and the throttling rules of multiple consumers of the same account are the same, while the throttling rules of different accounts are different.

2. The method of claim 1, wherein the method is based on sentinel for high performance mass data cluster throttling. 3.A sentinel-based high-performance mass data cluster throttling system, characterized by comprising: a data concurrent running module for separating the data of multiple platforms, which realizes concurrent running and schedules the producers of each platform at regular intervals; a concurrent platform query control module for controlling concurrent multiple platform query operations when the producer is called, each platform has its own producer, and the data queried from the producer is different; a data concurrent consumption control module for forwarding the data from the platform to the message middleware rabbitmq, and forwarding the data from the producer to the corresponding queue-bound consumer to realize concurrent consumption of data produced by the same platform. ​ ​ The cluster flow control module is used for encapsulating a request to call a third-party interface according to the data from the rabbitmq when the consumer receives the data from the rabbitmq, and processing the data by using a cluster flow control mode. The cluster flow control module is further used for implementing a cluster by deploying multiple hosts based on a sentinel technology, and there are multiple consumers for each account of each platform on each host, and each account is set to consume data by using multiple consumers. Then, a flow control rule is set, each account is set with its own flow control rule, and the cluster flow control rules are stored in a redis memory, which can be modified and controlled at any time. The cluster flow control is further performed, each consumer is intercepted by the cluster flow control system with the set flow control rule when calling third-party data, it is judged whether the flow control rule can be passed, if yes, the third-party interface is continuously called, and a confirmation message is sent to the rabbitmq to tell that the data is consumed, if not, the third-party interface is directly returned and cannot be called.

Images