Method, device and equipment for limiting query rate of distributed database and storage medium

By reading and judging network traffic values ​​in a distributed database system, controlling the rate at which the work nodes acquire data, solving the problem of network communication pressure caused by AP-type query, realizing the granularity of the computer room or SQL statements, and meeting multiple solutions Speed ​​limit requirements for business scenarios.

CN119088838APending Publication Date: 2024-12-06JINZHUAN INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411179701.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

In distributed database systems, network communication pressure caused by AP-type query results in full network card bandwidth, causing network communication to be blocked, affecting query execution and service operation.

Method used

By reading the network traffic value in the current time window in the cache, we judge whether it exceeds the preset value. If it exceeds the value, we pause the acquisition of the next line of data from the data node, perform this step in a loop, and clear and archive the traffic value in the timing task. Provides query interfaces for statistics and control.

Benefits of technology

The flow control of different computer rooms or SQL statements is realized, which avoids network communication pressure, ensures that the query rate is suspended or suspended under preset conditions, and meets the speed limit requirements of massive data parallel processing systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure FT_1
    Figure FT_1
  • Figure FT_2
    Figure FT_2
  • Figure FT_3
    Figure FT_3
Patent Text Reader

Abstract

The invention discloses a method, device and equipment for limiting the query rate of a distributed database and a storage medium, and relates to the technical field of databases, and the method comprises the following steps: reading a network traffic value used in a current time window in a cache; judging whether the used network flow value in the current time window exceeds a preset value or not, and if not, acquiring the next row of data from the data node by the working node; if yes, the working node stops acquiring the next row of data from the data node; updating the obtained row data volume to a cache; and circularly executing the steps. According to the method and the device, the flow of data interaction among the nodes in different machine rooms can be counted in unit time in the SQL statement execution process, and data communication is postponed or paused after the preset triggering condition is reached, so that the aim of limiting the query rate is fulfilled. The preset condition can be machine room granularity and can also be SQL statement level. Speed control of machine room or SQL statement granularity is supported, and various service application scenes can be met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a method, device, equipment and storage medium for limiting the query rate of a distributed database. Background Art

[0002] A network card is a piece of computer hardware designed to allow computers to communicate on a computer network. Each network card has an indicator to measure its data transmission rate - the network card bandwidth, that is, the data transmission rate of the network interface controller, which is one of the important indicators to measure the performance of the network card. When this computer and other computers perform network data transmission, the more data is transmitted per unit time, the greater the instantaneous bandwidth value. When it approaches the rated value of the network card, data transmission will be blocked and network communication with other computers will no longer be possible until the instantaneous bandwidth value returns to below the rated value of the network card, and network communication with other computers can resume normal.

[0003] In a distributed database system, the process from sending SQL statements from the client to the server and then receiving the final execution result from the server by the client generally requires lexical parsing, semantic analysis, generating execution plans, optimizing execution plans, and executing. Among these steps, the execution step generally takes up the highest proportion of time consumption for massive data parallel processing scenarios. The amount of data in the tables involved in the statement is generally above TB or even PB level, and these data are usually distributed on data nodes in many different local, same-city or remote computer rooms according to the table partitioning rules and system topology.

[0004] During the execution process, the work node obtains the data that the current operator needs to process from the corresponding data node through the network communication protocol according to the execution plan, and there is also data exchange between the upstream and downstream tasks of each work node through the HTTP (Hypertext Transfer Protocol) protocol. In the MPP system, these two parts of data usually need to interact through the network, and for AP-type queries, the amount of these two parts of data is generally very large. If there is no effective control method, it is very likely to bring great pressure to the network card of the machine for a period of time, causing a "traffic jam" phenomenon in network communication.

[0005] Once a single query fills up the bandwidth of a node's network card, the node will be unable to communicate with external nodes through this network card, which will directly affect the normal execution of subsequent queries and even other applications on the computer that need to use this network card for network communication. This impact may manifest as a short-term "stuttering" phenomenon or a long-term "pause", depending on the amount of data in the query that currently fills up the network card. This situation will not only bring a bad user experience, but also affect normal business operations, resulting in more serious consequences.

[0006] Based on this background, some AP (Analytical Processing, referred to as AP) database vendors will provide some methods to limit the data transmission rate during SQL execution, including operating system-level control tools (such as Traffic Control, Cgroup, etc.), or integrate some third-party applications (such as YARN, etc.). However, these methods are difficult to achieve precise control at the specific business level, let alone provide additional extended functions. Summary of the invention

[0007] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a method, device, equipment and storage medium for limiting the query rate of a distributed database.

[0008] In order to solve the above technical problems, the technical solution of the present invention is as follows: A method for limiting a distributed database query rate, comprising: Read the network traffic value used in the current time window in the cache; Determine whether the network traffic value used in the current time window exceeds the preset value. If not, the working node obtains the next row of data from the data node; if so, the working node stops obtaining the next row of data from the data node; Update the acquired row data into the cache; Execute the above steps repeatedly.

[0009] As a preferred solution of the method for limiting the query rate of a distributed database according to the present invention, the network traffic value used in the current time window in the read cache includes: Read the network traffic value used by the current computer room in the current time window or the network traffic value used when the current SQL statement is executed in the current time window in the cache.

[0010] As a preferred solution of the method for limiting the query rate of a distributed database according to the present invention, after the working node stops acquiring the next row of data from the data node, the method further includes: After time T has passed, it is determined again whether the network traffic value used in the current time window exceeds the preset value.

[0011] As a preferred solution of the method for limiting the query rate of a distributed database according to the present invention, the method for updating the network traffic value used in the current time window in the cache includes: When the master node is running, a scheduled task is started, and the scheduled task is used to clear the network traffic value used by each computer room or each SQL statement in the current time window during operation.

[0012] As a preferred solution of the method for limiting the query rate of a distributed database described in the present invention, the time interval for running the scheduled task is a preset time window size.

[0013] As a preferred solution of the method for limiting the query rate of a distributed database described in the present invention, the master node records the network traffic value used in the current time window and writes it into the storage system for archiving before clearing the network traffic value used by each computer room or each SQL statement in the current time window.

[0014] As a preferred solution of the method for limiting the query rate of a distributed database according to the present invention, after recording the network traffic value used in the current time window and writing it into the storage system for archiving, it also includes: Set the external query interface for obtaining the used network traffic value in the corresponding time window in the archive.

[0015] The present invention also provides a device for limiting the query rate of a distributed database, comprising: A reading module is used to read the network traffic value used in the current time window in the cache; A judgment module is used to judge whether the network traffic value used in the current time window exceeds a preset value, and to control the working node to obtain the next row of data from the data node if it does not exceed the preset value, and to control the working node to stop obtaining the next row of data from the data node if it exceeds the preset value; Update module: used to update the acquired row data into the cache; The loop module is used to execute the above steps in a loop.

[0016] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any of the above methods for limiting the query rate of a distributed database is implemented.

[0017] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the method described in any of the above methods for limiting the query rate of a distributed database is implemented.

[0018] The beneficial effects of the present invention are: (1) During the SQL statement execution process, the present invention can count the data exchange traffic between nodes in different computer rooms in a unit time, and suspend or pause data communication after reaching a preset trigger condition, thereby achieving the purpose of limiting the query rate. The preset condition can be at the computer room granularity or at the SQL statement level. Supporting speed control at the computer room or SQL statement granularity can meet a variety of business application scenarios.

[0019] (2) The present invention can be applied to a node in a massive data parallel processing system to perform speed limit control.

[0020] (3) The present invention provides an SQL statement and a statistical information query interface at the computer room granularity to facilitate user decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0022] Figure 1 A schematic diagram of a flow chart of a method for limiting a distributed database query rate provided by the present invention; Figure 2 Another schematic diagram of a flow chart of a method for limiting the query rate of a distributed database provided by the present invention; Figure 3 A schematic diagram of a device for limiting the query rate of a distributed database provided by the present invention; Figure 4 A schematic diagram of a computer device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific implementation modes and in combination with the accompanying drawings.

[0024] Figure 1 A flow chart of a method for limiting the query rate of a distributed database provided in an embodiment of the present application, the method comprising the following steps: Step S101: Read the network traffic value used in the current time window in the cache.

[0025] Specifically, the worker node will obtain data from the data node row by row. Before the worker node obtains the next row of data from the data node, it needs to first read the network traffic value used by the current computer room in the current time window or the network traffic value used when the current SQL statement is executed in the current time window in the cache.

[0026] It should be noted that the size of the time window is pre-set.

[0027] Step S102: determine whether the network traffic value used in the current time window exceeds the preset value. If not, the working node obtains the next row of data from the data node; if so, the working node stops obtaining the next row of data from the data node.

[0028] Specifically, after reading the network traffic value used in the current time window in the cache, determine whether the network traffic value used in the current time window exceeds the preset value. If the network traffic value used in the current time window does not exceed the preset value, the working node obtains the next row of data from the data node (ie, DN); if the network traffic value used in the current time window exceeds the preset value, the working node stops obtaining the next row of data from the data node, and determines again after a time T whether the network traffic value used in the current time window exceeds the preset value, until the network traffic value used in the current time window does not exceed the preset value, the working node can obtain the next row of data from the data node.

[0029] The time T can be set according to actual conditions.

[0030] Step S103: Update the acquired row data volume into the cache.

[0031] Step S104: Execute the above steps in a loop.

[0032] Specifically, the above steps S101 to S103 are executed cyclically. Specifically, after completing one cycle, the network traffic value used in the current time window in the cache is read again. Afterwards, it is determined whether the network traffic value used in the current time window exceeds the preset value. If not, the working node obtains the next row of data from the data node; if so, the working node stops obtaining the next row of data from the data node. Finally, the amount of row data obtained is updated to the cache.

[0033] It is understandable that the network traffic value used in the current time window in the cache needs to be updated. The update method is as follows: The master node starts a scheduled task when it is started. The interval of the positioning task is the preset time window size. The responsibility of this scheduled task is to clear the network traffic value used by each computer room or each SQL statement in the current time window, that is, to start the next time window.

[0034] It should be noted that before the master node prepares to clear the network traffic values ​​used by each computer room or each SQL statement in the current time window, it will record the network traffic values ​​used in the current time window, such as persisting them locally or writing them to a distributed storage system for archiving.

[0035] In addition, an external query interface or other viewing method is also provided. Through the external query interface or other viewing method, the network traffic value used in the corresponding time window in the archive can be obtained.

[0036] In addition, this embodiment also provides a processing method for querying detailed information of network traffic resource consumption, including the following steps: Step S201: Record the execution details of each SQL in the cache (distributed), including the cumulative size of the consumed data, submission time, update time, etc.

[0037] Step S202: Every time new data is obtained from a data node, the statistical data of the SQL in the cache is updated (or batch updated).

[0038] Step S203: Calculate the average size of the network bandwidth of the current query based on the cumulative size of data consumed by the SQL statement, submission time, update time and other information.

[0039] Step S204: setting an external query interface, through which the statistical detailed information of the SQL in the cache can be obtained.

[0040] Therefore, through the above scheme, during the execution of SQL statements, the traffic of data interaction between nodes in different computer rooms can be counted within a unit time, and data communication can be suspended or stopped after reaching the preset trigger condition, so as to achieve the purpose of limiting the query rate. The preset conditions can be at the computer room granularity or at the SQL statement level. At the same time, the traffic data consumed by different statements can be counted within a unit time, and the details of the occupied network resources can be recorded to facilitate user inquiries. In addition, by counting the traffic data consumed by different computer rooms within a unit time and recording the details of the occupied network resources, a basis is provided for the preset value.

[0041] Figure 3 A schematic diagram of a device for limiting the query rate of a distributed database provided in an embodiment of the present application. The device includes a reading module 301, a judging module 302, an updating module 303 and a circulation module 304.

[0042] The reading module is used to read the network traffic value used in the current time window in the cache. Specifically, the reading module is used to read the network traffic value used in the current computer room in the current time window or the network traffic value used when the current SQL statement is executed in the current time window in the cache.

[0043] The judgment module is used to judge whether the network traffic value used in the current time window exceeds the preset value, and control the working node to obtain the next row of data from the data node when it does not exceed the preset value. When it exceeds the preset value, the working node is controlled to stop obtaining the next row of data from the data node, and after time T, it is judged again whether the network traffic value used in the current time window exceeds the preset value, until the network traffic value used in the current time window does not exceed the preset value, the working node can obtain the next row of data from the data node.

[0044] The update module is used to update the acquired row data into the cache.

[0045] The loop module is used to loop the operations performed by the above-mentioned reading module, judgment module and update module. Specifically, after completing one loop, the network traffic value used in the current time window in the cache is read again. Afterwards, it is determined whether the network traffic value used in the current time window exceeds the preset value. If not, the working node obtains the next row of data from the data node; if so, the working node stops obtaining the next row of data from the data node. Finally, the amount of row data obtained is updated to the cache.

[0046] See also Figure 4 This embodiment also provides a computer device, and the components of the computer device may include but are not limited to: one or more processors or processing units, a system memory, and a bus connecting different system components (including the system memory and the processing unit).

[0047] The term "bus" refers to one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0048] The computer system / server typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer system / server, including volatile and non-volatile media, removable and non-removable media.

[0049] The system memory may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The computer device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system may be used to read and write non-removable, non-volatile magnetic media. A disk drive for reading and writing removable non-volatile disks (such as "floppy disks") and an optical disk drive for reading and writing removable non-volatile optical disks (such as CD-ROMs, DVD-ROMs or other optical media) may be provided. In these cases, each drive may be connected to the bus via one or more data medium interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0050] A program / utility having a set (at least one) of program modules may be stored, for example, in a memory, such program modules including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules generally perform the functions and / or methods of the embodiments described herein.

[0051] The computer device may also communicate with one or more external devices such as a keyboard, a pointing device, a display, etc. Such communication may be performed through an input / output (I / O) interface. Furthermore, the computer device may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter.

[0052] The processing unit executes the functions and / or methods described in the embodiments of the present invention by running the programs stored in the system memory.

[0053] The above-mentioned computer program can be set in a computer storage medium, that is, the computer storage medium is encoded with a computer program, and when the program is executed by one or more computers, it enables one or more computers to execute the method flow and / or device operation shown in the above-mentioned embodiments of the present invention.

[0054] With the development of time and technology, the meaning of medium is becoming more and more extensive, and the propagation path of computer programs is no longer limited to tangible media, but can also be downloaded directly from the network, etc. Any combination of one or more computer-readable media can be used. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, - but not limited to - electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media (non-exhaustive list) include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing programs, which can be used by or in combination with instruction execution systems, devices or devices.

[0055] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0056] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0057] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0058] In addition to the above embodiments, the present invention may also have other implementation modes; any technical solutions formed by equivalent replacement or equivalent transformation shall fall within the protection scope required by the present invention.

Claims

1. A method for limiting the query rate of a distributed database, characterized in that: include: Read the network traffic value used in the current time window in the cache; Determine whether the network traffic value used in the current time window exceeds the preset value. If not, the working node obtains the next row of data from the data node; if so, the working node stops obtaining the next row of data from the data node; Update the acquired row data into the cache; Execute the above steps repeatedly.

2. The method for limiting the query rate of a distributed database according to claim 1, characterized in that: The network traffic value used in the current time window in the read cache includes: Read the network traffic value used by the current computer room in the current time window or the network traffic value used when the current SQL statement is executed in the current time window in the cache.

3. The method for limiting the query rate of a distributed database according to claim 1, characterized in that: After the working node stops acquiring the next row of data from the data node, the method further includes: After time T has passed, it is determined again whether the network traffic value used in the current time window exceeds the preset value.

4. The method for limiting the query rate of a distributed database according to claim 2, characterized in that: The method for updating the network traffic value used in the current time window in the cache includes: When the master node is running, a scheduled task is started, and the scheduled task is used to clear the network traffic value used by each computer room or each SQL statement in the current time window during operation.

5. The method for limiting the query rate of a distributed database according to claim 4, characterized in that: The time interval for the scheduled task to run is a preset time window size.

6. The method for limiting the query rate of a distributed database according to claim 4, characterized in that: Before clearing the network traffic value used by each computer room or each SQL statement in the current time window, the master node records the network traffic value used in the current time window and writes it into the storage system for archiving.

7. The method for limiting the query rate of a distributed database according to claim 6, characterized in that: After recording the network traffic value used in the current time window and writing it into the storage system for archiving, the method further includes: Set the external query interface for obtaining the used network traffic value in the corresponding time window in the archive.

8. A device for limiting the query rate of a distributed database, characterized in that: include: A reading module is used to read the network traffic value used in the current time window in the cache; A judgment module is used to judge whether the network traffic value used in the current time window exceeds a preset value, and to control the working node to obtain the next row of data from the data node if it does not exceed the preset value, and to control the working node to stop obtaining the next row of data from the data node if it exceeds the preset value; Update module: used to update the acquired row data into the cache; The loop module is used to execute the above steps in a loop.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.