Cross-machine-room scene flow control method, device and system and storage medium

Abstract

The invention discloses a cross-machine-room scene flow control method. The method comprises the following steps: detecting whether a flow control switch is turned on or not; if the flow control switch is turned on, detecting whether the target DN is in the machine room or not; if the target DN is not in the machine room, the residual flow in the local cache of the machine room is read; judging whether the residual flow in the local cache of the machine room is greater than 0 or not; if yes, pulling the data from the DN; and if not, waiting for reaching the current limiting threshold value, returning to the step, and if the target DN is not in the machine room, reading the residual flow in the local cache of the machine room. In the distributed database, the flow among the local, the same city and the different places is cross-machine-room, the method is suitable for the flow control of a cross-machine-room scene, the flow limiting processing can be flexibly carried out, and the problem of wrong flow limiting is not easy to occur.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a distributed database cross-computer room scenario flow control method, device, system and storage medium. Background Art

[0002] In a distributed database, traffic between local, intra-city, and remote locations is across computer rooms and passes through the operator's router, which often has bandwidth limitations and certain delays. With the continuous development of databases, the types of distributed database services are diversified. For example, there are a large number of complex OLAP (online analytical processing) queries, and there are more and more scenarios based on OLTP (online transaction processing) queries. In big data products, there is a trend to include computing engines (CN) that are good at processing OLTP-based services, and massively parallel processing (MPP) computing engines that are good at processing OLAP services. The traffic of the business engine reading DB data may also cross computer rooms. When the OLAP business statements are complex, it will cause too much cross-computer room business traffic, occupying the bandwidth across computer rooms, thereby affecting execution efficiency. In extreme cases, it may even affect more important control messages or other application communications, such as OLTP business requests with high real-time requirements, causing unpredictable risks.

[0003] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, an object of the present invention is to provide a method, device, system and storage medium for traffic control in a distributed database cross-computer room scenario.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides a distributed database cross-computer room scenario flow control method, comprising the following steps:

[0007] Check whether the flow control switch is turned on;

[0008] If the flow control switch is turned on, then detecting whether the target DN is in the computer room;

[0009] If the target DN is not in the computer room, the remaining traffic in the local cache of the computer room is read;

[0010] Determine whether the remaining traffic in the local cache of the computer room is greater than 0;

[0011] If yes, then pull the data from the DN; if no, then wait until the flow limit threshold is reached and return to step If the target DN is not in the computer room, then read the remaining flow in the local cache of the computer room.

[0012] Furthermore, the distributed database cross-computer room scenario flow control method further includes, after the step of detecting whether the target DN is in the computer room if the flow control switch is turned on:

[0013] If the target DN is not in the computer room, the packet size of this request is calculated asynchronously;

[0014] Update the local cache of the accumulated value of the pulled data;

[0015] Determine whether the local cache exceeds the traffic threshold;

[0016] If yes, update the accumulated value in the local cache to the IP node of ZK.

[0017] Furthermore, the distributed database cross-computer room scenario flow control method further includes, before the step of detecting whether the flow control switch is turned on:

[0018] The flow control switch is configured with a plurality of continuous time periods, and a corresponding flow threshold is set for each time period.

[0019] Furthermore, the distributed database cross-computer room scenario flow control method further includes, before the step of detecting whether the flow control switch is turned on:

[0020] Setting a flow limiting statement template, wherein the flow limiting statement template sets a corresponding flow threshold;

[0021] The statement in the requested data packet is matched according to the current limiting statement template.

[0022] Furthermore, the distributed database cross-computer room scenario flow control method, after the step of detecting whether the target DN is in the computer room if the flow control switch is turned on, further comprises the steps of:

[0023] If the target DN is in the computer room, data is pulled from the DN.

[0024] Furthermore, the distributed database cross-computer room scenario flow control method, after the flow control switch configures a plurality of continuous time periods and sets a corresponding flow threshold for each time period, further includes:

[0025] In the current time period, if the flow threshold is greater than zero, the flow control switch is turned on.

[0026] Furthermore, the distributed database cross-computer room scenario flow control method, after the step of matching the statement in the requested data packet according to the flow limiting statement template, further includes:

[0027] If the number of matched statements is greater than zero, the flow control switch is turned on.

[0028] The present invention provides a distributed database cross-computer room scenario flow control device, the device comprising:

[0029] A switch detection module is used to detect whether the flow control switch is turned on;

[0030] The computer room detection module is used to detect whether the target DN is in the computer room;

[0031] The traffic reading module is used to read the remaining traffic in the local cache of the computer room;

[0032] The flow determination module is used to determine whether the remaining flow in the local cache of the computer room is greater than 0;

[0033] The data pulling module is used to pull data from DN.

[0034] The present invention provides a distributed database cross-computer room scenario flow control system, the system comprising at least one processor; and,

[0035] a memory communicatively connected to the at least one processor; wherein,

[0036] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the distributed database cross-computer room scenario traffic control method as described above.

[0037] The present invention also provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the distributed database cross-computer room scenario traffic control method as described above.

[0038] Compared with the prior art, the present invention provides a distributed database cross-computer room scenario flow control method, including the following steps: detecting whether the flow control switch is turned on; if the flow control switch is turned on, detecting whether the target DN is in the computer room; if the target DN is not in the computer room, reading the remaining flow in the local cache of the computer room; judging whether the remaining flow in the local cache of the computer room is greater than 0; if so, pulling data from the DN; if not, waiting to reach the flow limit threshold, returning to the step if the target DN is not in the computer room, reading the remaining flow in the local cache of the computer room. In a distributed database, the flow between local, same-city, and different locations is cross-computer room. The present invention is suitable for flow control in cross-computer room scenarios. The present invention can flexibly perform flow limit processing, and is not prone to flow limit errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The process of the distributed database cross-computer room scenario flow control method provided by the present invention Figure 1 .

[0040] Figure 2 A schematic diagram of the functional modules of the distributed database cross-computer room scenario traffic control device provided by the present invention.

[0041] Figure 3 A schematic diagram of the hardware structure of the distributed database cross-computer room scenario traffic control system provided by the present invention. DETAILED DESCRIPTION

[0042] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a distributed database cross-computer room scenario traffic control method, application server, system and storage medium.

[0043] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0044] The present invention can be applied in a computer device, and one or more applications can be installed in the computer device to process relevant data. The computer device can be a terminal or a server, wherein the terminal can be an electronic device with communication function such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant and a wearable device. The computer device includes a processor, a non-volatile storage medium, an internal memory and a network interface connected through a system bus. Among them, the non-volatile storage medium of the computer device can store an operating system and a computer-readable program, and when the computer-readable program is executed, the processor can execute a method for predicting the difficulty of underwriting. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The internal memory can store a computer-readable program, and when the computer-readable program is executed by the processor, the processor can execute a method for predicting the difficulty of underwriting. The network interface of the computer device is used for network communication, such as sending assigned tasks.

[0045] like Figure 1 As shown, the present invention provides a distributed database cross-computer room scenario flow control method, comprising the following steps:

[0046] S10, check whether the flow control switch is turned on;

[0047] S20, if the flow control switch is turned on, detecting whether the target DN is in the computer room;

[0048] S30. If the target DN is not in the computer room, read the remaining traffic in the local cache of the computer room;

[0049] S40, determining whether the remaining flow in the local cache of the computer room is greater than 0;

[0050] S50. If yes, pull data from DN; if no, wait until the flow limit threshold is reached and return to step S30. If the target DN is not in the computer room, read the remaining flow in the local cache of the computer room.

[0051] A distributed database DN cluster is a cluster consisting of multiple data node (DN) instances that work together to provide data storage, query, and processing services. In the architecture of distributed databases, DN clusters play a vital role. DN clusters are usually composed of multiple DN instances that are scattered in different physical locations and connected to each other through the network. Each DN instance is an independent data storage and processing unit that can independently complete operations such as data storage, query, and modification. At the same time, DN instances will communicate and collaborate with each other to ensure data consistency and integrity.

[0052] After the step of detecting whether the flow control switch is turned on in S10, if the flow control switch is not turned on, the subsequent steps are not executed. If the flow control switch is turned on, it is determined whether the DN for pulling data is in the same computer room as the current computing engine. If so, it continues to obtain data; if not, it reads the computer room remaining flow and statement granularity remaining flow in the local cache, and determines whether the remaining flow is greater than 0.

[0053] Furthermore, the distributed database cross-computer room scenario flow control method provided by the present invention further includes the following steps after the step of detecting whether the target DN is in the computer room if the flow control switch is turned on in S20:

[0054] If the target DN is in the computer room, data is pulled from the DN.

[0055] After determining whether the remaining flow is greater than 0, the present invention continues to obtain data if so; if not, it prints the log, and waits for the waiting time of the data operator to reach the current limiting threshold, and then executes the above determination in a loop.

[0056] The remaining flow is zero, and waiting to reach the flow limiting threshold refers to waiting for the next cycle (at the beginning of the next time period, the local cache is cleared, and the remaining flow is updated to the flow threshold), and then returning to execute the judgment.

[0057] In the present invention, the consumed traffic can be added to the local cumulative cache of the computing engine. The local cache also records the consumed traffic at the SQL (Structured Query Language) granularity, including the bandwidth traffic consumed by the SQL in the current computing cycle, and some other information, such as SQL statements, submission time, update time, client IP, etc. If the sum of the accumulated traffic in the local cache exceeds the predetermined traffic threshold, the value in the cache is written to the node node (data node (DataNode) in the distributed database, which is responsible for storing actual data. They receive data read and write requests from clients or computing nodes and perform corresponding operations. Data nodes usually adopt redundancy and replication strategies to improve data reliability and fault tolerance) of the current computing engine corresponding to ZK (ZooKeeper, essentially a distributed small file storage system, provides data storage based on a directory tree similar to a file system, and can effectively manage the nodes in the tree, so as to maintain and monitor the state changes of the stored data. By monitoring the changes in these data states, data-based cluster management can be achieved). If the node node does not exist, it is newly created, and the name ensures global uniqueness and readability.

[0058] Furthermore, the distributed database cross-computer room scenario flow control method provided by the present invention further includes the following steps after the step of detecting whether the target DN is in the computer room if the flow control switch is turned on in S20:

[0059] If the target DN is not in the computer room, the packet size of this request is calculated asynchronously;

[0060] Update the local cache of the accumulated value of the pulled data;

[0061] Determine whether the local cache exceeds the traffic threshold;

[0062] If yes, update the accumulated value in the local cache to the IP node of ZK.

[0063] The present invention is executed asynchronously. Specifically, it traverses each SQL record in the cache and searches for the corresponding SQL statistics under the SQL traffic statistics node of ZK. If not, the newly created node is locked. If yes, the updated node data is locked, mainly the usedFlow and updateTime fields. When updating, the value of usedFlow is added to the value corresponding to the bandwidth traffic consumed by the SQL in the current calculation cycle in the current local cache, and updateTime takes the current timestamp;

[0064] Traverse the information of other SQL statements under the SQL flow statistics node and calculate the average flow of the statement using the formula: usedFlow / (updateTime-commitTime). If the average flow is not in TopN, lock and delete the current record.

[0065] Clear the local traffic accumulation cache of the Worker (the unit that executes the query plan);

[0066] Monitor changes in the content of the sub-nodes under the computer room and update the remaining traffic of the computer room node.

[0067] Furthermore, in order to perform flow limiting more flexibly, the distributed database cross-computer room scenario flow control method provided by the present invention further includes, before the step of S10, detecting whether the flow control switch is turned on:

[0068] The flow control switch is configured with a plurality of continuous time periods, and a corresponding flow threshold is set for each time period.

[0069] Furthermore, the distributed database cross-computer room scenario flow control method provided by the present invention further includes, before the step of detecting whether the flow control switch is turned on in S10:

[0070] The flow control switch is configured with a plurality of continuous time periods;

[0071] Setting a flow limiting statement template, wherein the flow limiting statement template sets a corresponding flow threshold within a corresponding time period;

[0072] The statement in the requested data packet is matched according to the current limiting statement template.

[0073] Furthermore, the distributed database cross-computer room scenario flow control method provided by the present invention, after the step of configuring a plurality of continuous time periods in the flow control switch and setting a corresponding flow threshold for each time period, further includes:

[0074] In the current time period, if the flow threshold is greater than zero, the flow control switch is turned on.

[0075] Furthermore, the distributed database cross-computer room scenario flow control method provided by the present invention further includes, after the step of matching the statement in the requested data packet according to the flow limiting statement template:

[0076] If the number of matched statements is greater than zero, the flow control switch is turned on.

[0077] The present invention configures the traffic threshold according to the specific business, and each computing engine can be configured separately. The switch controls whether each business starts the flow limit, sets the flow for a certain type of statement, and sets different flow strategies for different time periods. The present invention uses SQL statements as units for flow statistics, and the Top SQL is used to record the flow in ZK.

[0078] The main application scenario of the present invention is distributed database. The traffic between local, same-city and remote locations is a cross-computer room scenario. The traffic threshold can be configured according to the business granularity, time period and statement granularity, and the traffic can be recorded using the Top statement. Different business computing engines are configured with a traffic switch, which can be configured to be on or off, and traffic control will be performed when it is turned on.

[0079] After opening, the time period and statement granularity settings will be read.

[0080] The time period configuration includes: start time, end time and traffic threshold. Different thresholds can be configured for multiple time periods.

[0081] The statement granularity configuration content is: set the current limiting statement template, the template content is the statement id and the corresponding statement, such as the statement is "select*from tablename where col1=x and col2=y", ignore the case when parsing and perform fuzzy matching on the values ​​in the condition (x, y).

[0082] like Figure 2 As shown, the present invention provides a distributed database cross-computer room scenario flow control device, the device comprising:

[0083] The switch detection module 11 is used for S10 to detect whether the flow control switch is turned on;

[0084] The computer room detection module 12 is used to detect whether the target DN is in the computer room;

[0085] The traffic reading module 13 is used to read the remaining traffic in the local cache of the computer room;

[0086] The flow determination module 14 is used for S40 to determine whether the remaining flow in the local cache of the computer room is greater than 0;

[0087] The data pulling module 15 is used to pull data from the DN.

[0088] The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which is more suitable for the execution process of the distributed database cross-computer room scenario traffic control method than the program. For the specific implementation method of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0089] The device may further include a waiting module, configured to wait for a current limiting threshold to be reached.

[0090] Another embodiment of the present invention also provides a distributed database cross-computer room scenario flow control system, such as Figure 3 As shown, the system 10 includes:

[0091] One or more processors 110 and memory 120, Figure 3 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 2 The example of connecting through bus is taken in the following.

[0092] The processor 110 is used to complete various control logics of the system 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0093] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the distributed database cross-computer room scenario flow control method in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the system 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the distributed database cross-computer room scenario flow control method in the above method embodiment.

[0094] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the system 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the system 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0095] One or more units are stored in the memory 120, and when executed by one or more processors 110, the water supply proportional valve fault detection method in any of the above method embodiments is executed, for example, the above described Figure 1 The method comprises steps S101 to S103.

[0096] An embodiment of the present invention provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example, to execute the above-described Figure 1 The method comprises steps S101 to S103.

[0097] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0098] In summary, the present invention configures the traffic threshold according to the specific business, and each computing engine can be configured separately. The switch controls whether each business starts the flow limit, sets the flow for a certain type of statement, and sets different flow strategies for different time periods. The main application scenario of the present invention is a distributed database. The flow between local, same-city, and remote locations is a cross-computer room scenario. The flow threshold can be configured according to the business granularity, time period, and statement granularity, and the flow can be recorded using the Top statement. Different business computing engines are configured with flow switches, which can be configured to be on or off, and flow control will be performed when turned on.

[0099] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0100] What has been described in this specification and the accompanying drawings includes examples that can provide water proportional valve fault detection methods, devices, systems and media. Of course, it is not possible to describe every conceivable combination of elements and / or methods for the purpose of describing the various features of the present disclosure, but it can be recognized that many other combinations and permutations of the disclosed features are possible. Therefore, it is obvious that various modifications can be made to the present disclosure without departing from the scope or spirit of the present disclosure. In addition, or in an alternative, other embodiments of the present disclosure may be obvious from consideration of this specification and the accompanying drawings and from the practice of the present disclosure as presented herein. It is intended that the examples proposed in this specification and the accompanying drawings are considered to be illustrative and not restrictive in all respects. Although specific terms are used in this article, they are used in a general and descriptive sense and are not used for the purpose of limitation.

Claims

1. A distributed database cross-computer room scenario flow control method, characterized in that: The steps include: Check whether the flow control switch is turned on; If the flow control switch is turned on, then detecting whether the target DN is in the computer room; If the target DN is not in the computer room, the remaining traffic in the local cache of the computer room is read; Determine whether the remaining traffic in the local cache of the computer room is greater than 0; If yes, then pull the data from the DN; if no, then wait until the flow limit threshold is reached and return to step If the target DN is not in the computer room, then read the remaining flow in the local cache of the computer room.

2. The distributed database cross-computer room scenario flow control method according to claim 1 is characterized in that: If the flow control switch is turned on, the step of detecting whether the target DN is in the computer room also includes: If the target DN is not in the computer room, the packet size of this request is calculated asynchronously; Update the local cache of the accumulated value of the pulled data; Determine whether the local cache exceeds the traffic threshold; If yes, update the accumulated value in the local cache to the IP node of ZK.

3. The distributed database cross-computer room scenario flow control method according to claim 1 is characterized in that: Before the step of detecting whether the flow control switch is turned on, the method further includes: The flow control switch is configured with a plurality of continuous time periods, and a corresponding flow threshold is set for each time period.

4. The distributed database cross-computer room scenario flow control method according to claim 1 is characterized in that: Before the step of detecting whether the flow control switch is turned on, the method further includes: Setting a flow limiting statement template, wherein the flow limiting statement template sets a corresponding flow threshold; The statement in the requested data packet is matched according to the current limiting statement template.

5. The distributed database cross-computer room scenario flow control method according to claim 1 is characterized in that: After the step of detecting whether the target DN is in the computer room if the flow control switch is turned on, the step further includes: If the target DN is in the computer room, data is pulled from the DN.

6. The distributed database cross-computer room scenario flow control method according to claim 3 is characterized in that: After the flow control switch is configured with a plurality of continuous time periods and a corresponding flow threshold is set for each time period, the method further includes: In the current time period, if the flow threshold is greater than zero, the flow control switch is turned on.

7. The distributed database cross-computer room scenario flow control method according to claim 4 is characterized in that: After the step of matching the statement in the requested data packet according to the current limiting statement template, the method further includes: If the number of matched statements is greater than zero, the flow control switch is turned on.

8. A distributed database cross-computer room scenario flow control device, characterized in that: The device comprises: A switch detection module is used to detect whether the flow control switch is turned on; The computer room detection module is used to detect whether the target DN is in the computer room; The traffic reading module is used to read the remaining traffic in the local cache of the computer room; The flow determination module is used to determine whether the remaining flow in the local cache of the computer room is greater than 0; The data pulling module is used to pull data from DN.

9. A distributed database cross-computer room scenario traffic control system, characterized in that: The system includes at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the distributed database cross-computer room scenario traffic control method described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the distributed database cross-computer room scenario traffic control method described in any one of claims 1-7.

Images