Query statement processing method, device and equipment and computer medium

Abstract

The present disclosure discloses a query statement processing method, device and equipment and a computer medium. The method comprises: obtaining a database query statement; determining a target traffic type to which the database query statement belongs, the target traffic type being any one of the following multiple traffic types: webpage-related traffic, open application interface-related traffic, cloud function service-related traffic, and traffic related to a business process corresponding to an automatic triggering mechanism; determining whether the target traffic type is a limited traffic type of a first database, and if so, obtaining limit information corresponding to the target traffic type, wherein the limited traffic type is determined based on a plurality of first SQL cost differences corresponding to a plurality of historical traffic data groups in adjacent time periods when the first database is overloaded; and processing the database query statement based on the limit information, which can improve the efficiency of solving the DB overload problem.

Description

Technical Field

[0001] This disclosure belongs to the field of database technology, and in particular relates to a query statement processing method, apparatus, device and computer medium. Background Technology

[0002] In the context of multi-tenant converged deployment of aPaaS (Application Platform as a Service), an increasing number of tenants and application packages share a single database (DB). This database frequently experiences overload incidents that last for extended periods, exhibiting poor stability and slow query efficiency in identifying the cause of the overload.

[0003] In related technologies, the database overload problem is generally solved by limiting the QPS (Queries-per-second) of tenant application packets through the gateway layer. However, this method cannot flexibly handle traffic data of different traffic types, and the way to deal with the database overload problem is relatively crude, with poor effect and low efficiency in solving the database overload problem. Summary of the Invention

[0004] This disclosure provides an implementation scheme that differs from related technologies, in order to solve the technical problem of low efficiency in solving DB overload problems in related technologies.

[0005] Firstly, this disclosure provides a query statement processing method, including:

[0006] Retrieve the database query statement;

[0007] Determine the target traffic type to which the database query statement belongs. The target traffic type is any one of the following traffic types: traffic related to web pages, traffic related to open application programming interfaces, traffic related to cloud function services, and traffic related to business processes corresponding to automatic triggering mechanisms.

[0008] Determine whether the target traffic type is a restricted traffic type of the first database. If so, obtain the restriction information corresponding to the target traffic type. The restricted traffic type is determined based on the first SQL cost difference corresponding to multiple historical traffic data groups of the multiple traffic types in adjacent time periods when the first database is detected to be overloaded.

[0009] The database query statement is processed based on the aforementioned restriction information.

[0010] Secondly, this disclosure provides a query statement processing apparatus, including:

[0011] The retrieval unit is used to retrieve database query statements;

[0012] The determining unit is used to determine the target traffic type to which the database query statement belongs. The target traffic type is any one of the following traffic types: traffic related to web pages, traffic related to open application programming interfaces, traffic related to cloud function services, and traffic related to business processes corresponding to automatic triggering mechanisms.

[0013] The determining unit is used to determine whether the target traffic type is a restricted traffic type of the first database. If so, it obtains the restriction information corresponding to the target traffic type. The restricted traffic type is determined based on the first SQL cost difference corresponding to multiple historical traffic data groups of the multiple traffic types in adjacent time periods when the first database is detected to be overloaded.

[0014] The processing unit is used to process the database query statement based on the restriction information.

[0015] Thirdly, this disclosure provides an electronic device, including:

[0016] Processor; and

[0017] Memory for storing the executable instructions of the processor;

[0018] The processor is configured to execute the methods described above by executing the executable instructions.

[0019] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned methods.

[0020] This disclosure provides a scheme for obtaining database query statements; determining the target traffic type to which the database query statement belongs, wherein the target traffic type is any one of the following traffic types: traffic related to web pages, traffic related to open application programming interfaces, traffic related to cloud function services, and traffic related to business processes corresponding to automatic triggering mechanisms; determining whether the target traffic type is a restricted traffic type of the first database; if so, obtaining the restriction information corresponding to the target traffic type, wherein the restricted traffic type is determined based on the cost differences of multiple first SQL statements corresponding to multiple historical traffic data groups in adjacent time periods when the first database is detected to be overloaded; and processing the database query statement based on the restriction information. This scheme can analyze the traffic type to which the database query statement belongs when the database query statement is obtained, and perform rate limiting processing on the database query statement when it is determined that the traffic type of the database query statement is a restricted traffic type of the first database. This scheme can process database query statements accessing the database at the granularity of the traffic type to which the database query statement belongs, considering finer granularity and higher flexibility, thereby improving the efficiency of solving the DB overload problem. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0022] Figure 1a This is a schematic diagram illustrating the configuration of a DB platform for an embodiment of this disclosure;

[0023] Figure 1b A schematic diagram of a microservice call chain provided in an embodiment of this disclosure;

[0024] Figure 2 This is a schematic diagram of the structure of a system provided in one embodiment of the present disclosure;

[0025] Figure 3a A flowchart illustrating a query statement processing method provided in an embodiment of this disclosure;

[0026] Figure 3b A schematic diagram illustrating a method for determining the type of traffic restriction provided in an embodiment of this disclosure;

[0027] Figure 3c A schematic diagram illustrating another method for determining the type of traffic restriction provided in an embodiment of this disclosure;

[0028] Figure 3d A scenario diagram illustrating a query statement processing method provided in an embodiment of this disclosure;

[0029] Figure 4 This is a schematic diagram of the structure of a query statement processing device provided in an embodiment of the present disclosure;

[0030] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0031] Embodiments of this disclosure are described in detail below, with examples of these embodiments illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting it.

[0032] The terms "first" and "second," etc., used in the specification, claims, and drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the present disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] First, some terms used in the embodiments of this disclosure will be explained below to facilitate understanding by those skilled in the art.

[0034] aPaaS, short for Application Platform as a Service, is a cloud computing service model that provides users with a complete platform for developing, running, and managing software applications.

[0035] FaaS (Function as a Service) is a cloud computing service model that provides event-driven computing services. Users can write and upload function code, and when an event is triggered, the cloud service provider will automatically execute this function code and return the result.

[0036] Metadata, also known as intermediary data or relay data, is data about data. It mainly describes the properties of data and is used to support functions such as indicating storage location, historical data, resource lookup, and file records.

[0037] MySQL is a relational database management system.

[0038] SQL (Structured Query Language) is a database language with multiple functions such as data manipulation and data definition. This language has interactive features and can provide great convenience to users. Database management systems should make full use of SQL to improve the working quality and efficiency of computer application systems.

[0039] The null value rate of a table refers to the proportion of null values ​​in a database table. A null value is a field that does not store any data or value. The null value rate can be used to assess the quality and integrity of data, as an excessive number of null values ​​may indicate missing or incomplete data.

[0040] In the context of multi-tenant converged deployment of aPaaS (Application Platform as a Service), an increasing number of tenants and application packages share a single database. Databases frequently experience overload incidents that last for extended periods, exhibiting poor stability. Furthermore, related technologies are slow to determine the cause of overload, resulting in low query efficiency.

[0041] The inventors discovered through research that, in some cases, [the following can be seen] Figure 1a As shown, different databases in aPaaS have different hardware configurations, and the number of tenants and application packages on different databases varies, making it impossible to quantify the rate limiting threshold. aPaaS employs a multi-tenant, multi-application package integrated deployment approach, where some application packages consume fewer resources, allowing a portion of application packages to exceed the safety threshold to achieve the maximum overall system benefit.

[0042] Further, see Figure 1b As shown, in a microservice call chain, from the perspective of service C, it's easy to know that the upstream call (up_stream) is service B, but the source of the traffic (origin_stream) is unknown. Identifying the traffic source is crucial in aPaaS detection and overload protection or rate limiting scenarios, as it allows us to pinpoint which type of traffic is causing significant pressure on the database.

[0043] This application provides a solution that, based on the context of multi-tenant converged deployment of aPaaS, serves as many application packages as possible when the database is available, or allows a small number of application packages to elastically exceed the database access limit. When the database exceeds the limit, it can be identified immediately and overload protection measures can be taken, such as targeted rate limiting (dropping SQL requests) to avoid overall service collapse. High-priority requests are guaranteed, and as the overload situation eases, no manual intervention is required to ensure that the database quickly returns to normal load.

[0044] In the proposed solution, a traffic tracing protocol can be established first to label the type of traffic source and transmit it transparently across the entire link, thereby achieving the goal of accurately distinguishing the traffic source of DB access data.

[0045] In this application, when it comes to measuring the pressure on the database, user operations on data, once they reach the data server, become read and write operations on aPaaS platform objects. Objects are mapped to MySQL tables, and object fields are mapped to MySQL table columns. In other words, data operations from traffic sources are translated into SQL after reaching the data server. Quantifying the pressure on the database caused by each SQL statement allows for comparison of the pressure on the database by traffic on the same order of magnitude, referred to as SQL cost or SQL cost.

[0046] The technical solutions of this disclosure and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0047] Figure 2 A schematic diagram of the structure of a system provided for an exemplary embodiment of this disclosure, the structure including:

[0048] This application relates to a system comprising: an aPaaS platform and user equipment, wherein the aPaaS platform includes: at least one business server, at least one data server, and at least one database; wherein:

[0049] The business server is used to: obtain access requests for accessing the database, which may include database query statements; optionally, the access request may be sent by the user device or generated based on an automatic triggering mechanism.

[0050] The data server is used to: obtain access requests from the business server and determine the database query statement; determine the target traffic type to which the database query statement belongs, wherein the target traffic type is any one of the following traffic types: traffic related to web pages, traffic related to open application programming interfaces, traffic related to cloud function services, and traffic related to business processes corresponding to automatic triggering mechanisms; determine whether the target traffic type is a restricted traffic type of the first database, and if so, obtain the restriction information corresponding to the target traffic type, wherein the restricted traffic type is determined based on the cost difference of multiple first SQL statements corresponding to multiple historical traffic data groups in adjacent time periods when the first database is detected to be overloaded; and process the database query statement based on the restriction information.

[0051] The first database is used for access by database query statements.

[0052] The execution principles and interaction processes of each component unit in this system embodiment can be found in the descriptions of the following method embodiments.

[0053] Figure 3a This is a flowchart illustrating a query statement processing method provided in an exemplary embodiment of the present disclosure. The method is applicable to data servers in an aPaaS platform and includes at least the following steps S201-S204:

[0054] S201. Obtain the database query statement;

[0055] Database query statements are commands used to retrieve information from a database. Database queries mainly include basic queries, conditional queries, deduplication queries, sorting queries, and grouping queries.

[0056] S202. Determine the target traffic type to which the database query statement belongs. The target traffic type is any one of the following traffic types: traffic related to web pages, traffic related to open application programming interfaces, traffic related to cloud function services, and traffic related to business processes corresponding to automatic triggering mechanisms.

[0057] Specifically, the target traffic type of a database query statement refers to the traffic type of the traffic to which the database query statement belongs.

[0058] Specifically, the aforementioned traffic related to web pages refers to traffic generated through triggering web pages; traffic related to open application programming interfaces refers to traffic generated through triggering OpenAPI; traffic related to cloud function services refers to traffic generated through FaaS; and traffic related to business processes corresponding to automatic triggering mechanisms refers to traffic generated through Automation.

[0059] In some alternative embodiments, the aforementioned traffic types may also include other types, which are not limited in this application.

[0060] In some optional embodiments of this application, the data server may first obtain an access request for the first database and parse the access request to obtain a database query statement.

[0061] Optionally, the target traffic type of the database query statement can be determined based on the traffic type identifier contained in the corresponding access request.

[0062] S203. Determine whether the target traffic type is a restricted traffic type of the first database. If so, obtain the restriction information corresponding to the target traffic type. The restricted traffic type is determined based on the first SQL cost difference corresponding to multiple historical traffic data groups of the multiple traffic types in adjacent time periods when the first database is detected to be overloaded.

[0063] In some optional embodiments of this application, the restricted traffic type of the first database may be stored in the data server, such as in the cache of the data server.

[0064] Optionally, the restriction information corresponding to the target traffic type can also be stored in the data server, such as in the data server's cache.

[0065] In some optional embodiments of this application, the restriction information corresponding to the target traffic type may include the number of times the database query statement corresponding to the target traffic type is allowed to access the first database within a first preset time period.

[0066] Optionally, if the number of times the statement corresponding to the target traffic type accesses the first database within the first preset time period does not reach the preset threshold, the database query statement is considered to comply with the restriction information, and in this case, the database query statement is allowed to access the first database without rate limiting. If the number of times the statement corresponding to the target traffic type accesses the first database within the first preset time period has reached or exceeded the preset threshold, the database query statement is considered to not comply with the restriction information, and in this case, rate limiting is applied to the database query statement.

[0067] S204. Process the database query statement based on the restriction information.

[0068] In some optional embodiments of this application, in the aforementioned S204, processing the database query statement based on the restriction information includes: in response to the restriction information indicating that the number of times the statement of the target traffic type accesses the first database within a first preset time period has reached a preset threshold, limiting the flow of the database query statement.

[0069] Optionally, the preset thresholds for different target traffic types can be different or the same.

[0070] In some optional embodiments of this application, the rate limiting of the database query statement includes the following steps S041-S042:

[0071] S041. In response to the database query statement being a read data statement, the database query statement is allowed to read data from the second database;

[0072] Optionally, the aPaaS platform in this application may include multiple first databases, each first database may have multiple slave databases, and the second database may be any one of the multiple slave databases of the first database.

[0073] Optionally, the data stored in the second database is the same as the data stored in the first database.

[0074] S042. If the database query statement is a write data statement, no processing is performed.

[0075] On aPaaS, most scenarios involve read-heavy and write-light operations, which can cause database overload. Therefore, in this application, we can choose to first assign rate limiting to read operations, and only assign rate limiting rules when the QPS of write operations exceeds a certain number, such as 1000.

[0076] In some optional embodiments of this application, the method further includes: determining that the first database is overloaded in response to detecting that the CPU utilization rate of the processor in the first database exceeds a preset threshold.

[0077] After analyzing the reasons for database overload and unavailability multiple times, the inventors found that three indicators—Load averages, Active Threads, and CPU usage—can provide a more intuitive reflection of the database's load status.

[0078] Load averages and active thread count are relatively sensitive metrics; slow queries and concurrent updates can cause significant increases in these metrics. Furthermore, the reference values ​​for load and active thread count differ depending on the database configuration; databases with more cores can withstand higher loads. CPU usage provides a more reliable indicator of database load; a CPU usage rate exceeding 60% indicates a high database load, and exceeding 80% will impact almost all business requests. Therefore, CPU usage is chosen as the primary metric for measuring database load.

[0079] In some optional embodiments of this application, the adjacent time periods are the first time period and the time period preceding the first time period, and each of the historical traffic data groups includes at least one historical access data. Regarding the aforementioned method for determining the restricted traffic type, the method further includes the following S01-S03:

[0080] S01. In response to detecting that the first database is overloaded, obtain the historical traffic data group corresponding to each of the multiple traffic types to obtain multiple historical traffic data groups corresponding to the multiple traffic types.

[0081] In some optional embodiments of this application, each traffic type corresponds to a historical traffic data group.

[0082] S02. For each historical access data in the multiple historical access data included in the multiple historical traffic data groups, calculate the difference between the first structured query language (SQL) cost corresponding to the historical access data in the first time period and the second SQL cost corresponding to the historical access data in the time period before the first time period, to obtain the first SQL cost difference corresponding to the historical access data, and then obtain the multiple first SQL cost differences corresponding to the multiple historical access data.

[0083] Optionally, the multiple first SQL cost differences corresponding to the multiple historical access data are the multiple first SQL cost differences corresponding to multiple historical traffic data groups in adjacent time periods.

[0084] In some optional embodiments, each historical traffic data group includes at least one historical access data item, and each historical traffic data item includes traffic data corresponding to an access unit. Specifically, when the traffic type is webpage-related traffic, the traffic data corresponding to at least one access unit in the historical traffic data group for that traffic type is the historical access data of each page within at least one web page, with each web page constituting one access unit.

[0085] When the traffic type is related to the Open Application Programming Interface, the traffic data corresponding to at least one access unit in the historical traffic data group corresponding to the traffic type is the historical access data of each client in at least one client, and each client is an access unit.

[0086] When the traffic type is related to cloud function services, the traffic data corresponding to at least one access unit in the historical traffic data group corresponding to the traffic type is the historical traffic data related to each cloud function server in at least one cloud function service, and each cloud function service is an access unit.

[0087] When the traffic type is related to the business process corresponding to the automatic triggering mechanism, the traffic data corresponding to at least one access unit in the historical traffic data group corresponding to the traffic type is the historical traffic data of each business process in at least one business process, and each business process is an access unit.

[0088] Optionally, the access unit refers to the sending source or triggering source that triggers the transmission of traffic data, wherein the triggering source can refer to a cloud function service or a business process.

[0089] Optionally, within the same historical traffic data group, different access units are distinguished by their corresponding identifiers; different web pages are distinguished by different web page identifiers; different clients are distinguished by different client identifiers; different cloud function services are distinguished by different function names; and different business processes are distinguished by different process identifiers.

[0090] It should be noted that the first SQL cost corresponding to the historical access data in the first time period refers to the sum of the SQL costs of all query statements in the historical access data in the first time period. Specifically, each query statement corresponds to one SQL cost.

[0091] Optionally, the aforementioned first time period may refer to a time period that ends at the current time and begins at a time that is a third preset time interval from the current time.

[0092] Optionally, the preceding time period of the aforementioned first time period is a time period ending at a time when the duration from the current time is a third preset duration, and starting at a time when the duration from the current time is twice the third preset duration.

[0093] S03. Determine the restricted traffic type based on the multiple first SQL cost differences.

[0094] In some optional embodiments of this application, the aforementioned S03, determining the restricted traffic type based on the plurality of first SQL cost differences, includes the following S0221-S0224:

[0095] S0221. For each of the plurality of first SQL cost differences, obtain the preset coefficient corresponding to the traffic type of the first SQL cost difference;

[0096] In some optional embodiments of this application, when the traffic type is traffic related to web pages and traffic related to business processes corresponding to automatic triggering mechanisms, the preset coefficient corresponding to the traffic type is not greater than the preset coefficient corresponding to the traffic type when the traffic type is traffic related to cloud function services.

[0097] When the traffic type is related to cloud function services, the preset coefficient corresponding to the traffic type is no greater than the preset coefficient corresponding to the traffic type when the traffic type is related to open application programming interfaces.

[0098] S0222. Calculate the product of the first SQL cost difference and the preset coefficient to obtain the first cost information corresponding to the first SQL cost difference, and then obtain the multiple first cost information corresponding to the multiple first SQL cost differences;

[0099] S0223. Sort the plurality of first cost information to obtain a first sorting result;

[0100] S0224. Determine the traffic restriction type based on the first sorting result.

[0101] Optionally, multiple first cost information can be sorted in descending order to obtain a first sorting result. In this case, determining the restricted traffic type based on the first sorting result includes: taking the traffic type corresponding to the preset number of first cost information at the top of the first sorting result as the restricted traffic type.

[0102] Specifically, the aforementioned method for determining the type of traffic restriction can also be found in [reference needed]. Figure 3b As shown.

[0103] By introducing preset coefficients, restricted traffic types can be prioritized. The smaller the preset coefficient, the lower the priority; the higher the priority, the easier it is to be restricted. This scheme can ensure that traffic types with low preset coefficients are processed normally as much as possible, improving the experience for relevant personnel.

[0104] In some optional embodiments of this application, after determining the restricted traffic type based on the first sorting result, the method further includes the following steps S1-S2:

[0105] S1. After the second preset time period, detect whether the first database is still in an overloaded state. If so, calculate the third SQL cost corresponding to the historical access data in the second time period for each historical access data, and then obtain the multiple third SQL costs corresponding to the multiple historical access data.

[0106] Optionally, the aforementioned second preset duration is equal to the third preset duration. The starting point of the second time period is the aforementioned current moment, and the ending point is the moment that is the third preset duration away from the current moment.

[0107] Optionally, the aforementioned second preset duration can be 30 seconds.

[0108] S2. Determine the restricted traffic type based on the multiple third SQL costs.

[0109] In some optional embodiments of this application, determining the restricted traffic type based on the plurality of third SQL costs in S2 above includes the following S21-S24:

[0110] S21. For each of the plurality of third SQL costs, obtain the preset coefficient corresponding to the traffic type of the third SQL cost;

[0111] In some optional embodiments of this application, when the traffic type is traffic related to web pages and traffic related to business processes corresponding to automatic triggering mechanisms, the preset coefficient corresponding to the traffic type is not greater than the preset coefficient corresponding to the traffic type when the traffic type is traffic related to cloud function services.

[0112] When the traffic type is related to cloud function services, the preset coefficient corresponding to the traffic type is no greater than the preset coefficient corresponding to the traffic type when the traffic type is related to open application programming interfaces.

[0113] S22. Calculate the product of the third SQL cost and the preset coefficient to obtain the second cost information corresponding to the third SQL cost, and then obtain the multiple second cost information corresponding to the multiple third SQL costs;

[0114] S23. Sort the plurality of second cost information to obtain a second sorting result;

[0115] S24. Determine the restricted traffic type based on the second sorting result.

[0116] Optionally, multiple second cost information can be sorted in descending order to obtain a second sorting result. In this case, determining the restricted traffic type based on the second sorting result includes: taking the traffic type corresponding to the preset number of second cost information at the top of the second sorting result as the restricted traffic type.

[0117] Specifically, the aforementioned method for determining the type of traffic restriction can also be found in [reference needed]. Figure 3c As shown.

[0118] In some optional embodiments of this application, S02-S03 are Diff strategies, and S1-S2 are Top strategies. The Diff strategy can be executed twice first, followed by the Top strategy twice.

[0119] In some optional embodiments of this application, the number of times and the order in which the Diff strategy and the Top strategy are executed are not limited.

[0120] In this application, for the first database, all record IDs and the -isDeleted field can be queried page by page, and the number of record IDs and the number of -isDeleted field entries on each page can be counted. Up to 20,000 record IDs can be counted to obtain the total number of object records for each field in the target first database and the number of non-deleted object records. In addition, this application also includes determining the following database information corresponding to each field in the first database: null value rate, number of distinct values, high-frequency word information, and histogram information corresponding to the target first database. The aforementioned database information can be used to determine the SQL cost corresponding to each query statement.

[0121] Experimental data shows that the solution based on this application can identify DB overload within 31 seconds and abnormal traffic sources within 200ms, and issue rate limiting decisions; DB recovers to normal within 2 minutes; after DB overload protection is put into operation, sudden traffic is quickly and accurately identified and protection measures are issued after DB exceeds the limit, aPaaSDataDB overload incidents are reduced to 0, and DB overload protection is enabled for all DataDBs.

[0122] This solution combines three data points—traffic type, SQL cost-based query pressure on the database, and actual database load feedback—to identify abnormal traffic during database overload and resolve database overload issues. It offers fast response, accurate identification and calibration, and rapid database recovery, resulting in significant effectiveness.

[0123] To provide a more comprehensive description of the solution in this application, a scenario diagram is provided below. Figure 3d Further explanation of this plan:

[0124] In this application, the traffic type of the traffic to which the database query statement belongs can be any of a variety of traffic types, which may include: traffic related to web pages, traffic related to open application programming interfaces, traffic related to cloud function services, and traffic related to business processes corresponding to automatic triggering mechanisms.

[0125] Specifically, the aforementioned traffic related to web pages refers to traffic generated through triggering web pages; traffic related to open application programming interfaces refers to traffic generated through triggering OpenAPI; traffic related to cloud function services refers to traffic generated through FaaS; and traffic related to business processes corresponding to automatic triggering mechanisms refers to traffic generated through Automation.

[0126] After receiving a database query statement, the data server can determine the traffic type of the query statement and whether that traffic type corresponds to any restrictions. If so, it checks whether the query statement conforms to the restrictions. If not, it determines whether the query statement is for reading or writing data. If it's for reading, the query statement is allowed to access the slave database of the first database, i.e., the second database; if it's for writing, no action is taken. If the query statement conforms to the restrictions, it is allowed to access the first database. If no restrictions are associated with the aforementioned traffic type, the query statement is allowed to access the first database.

[0127] Furthermore, this solution also includes detecting whether the first database is overloaded. When the first database is overloaded, the traffic type of the surge is calculated, the traffic type of the surge is used as the restricted traffic type, and the restriction information corresponding to the restricted traffic type is generated.

[0128] In some optional embodiments of this application, the restriction information corresponding to the restricted traffic type can be set by relevant personnel.

[0129] Optionally, the aforementioned restriction information and the aforementioned restricted traffic types may be stored in the cache of the data server.

[0130] Optionally, the aforementioned cache also stores access data for the aforementioned database query statements, as well as historical access data for the first database within a certain time period. When calculating the traffic type of a sudden surge in traffic, this historical access data needs to be retrieved from the cache. When a database query statement accesses the first database, the corresponding access data can be asynchronously stored in the cache. When determining whether a traffic type corresponds to restriction information, pre-stored restriction information needs to be retrieved in advance.

[0131] Optionally, the restriction information can be pre-stored in a cache.

[0132] It should be noted that, Figure 3d This diagram illustrates only the query processing method for a single database, namely the first database. When aPaaS includes multiple databases, the corresponding query processing method is the same as that for the first database, and will not be elaborated here.

[0133] This disclosure provides a scheme for obtaining database query statements; determining the target traffic type to which the database query statement belongs, wherein the target traffic type is any one of the following traffic types: traffic related to web pages, traffic related to open application programming interfaces, traffic related to cloud function services, and traffic related to business processes corresponding to automatic triggering mechanisms; determining whether the target traffic type is a restricted traffic type of the first database; if so, obtaining the restriction information corresponding to the target traffic type, wherein the restricted traffic type is determined based on the cost differences of multiple first SQL statements corresponding to multiple historical traffic data groups in adjacent time periods when the first database is detected to be overloaded; and processing the database query statement based on the restriction information. This scheme can analyze the traffic type to which the database query statement belongs when the database query statement is obtained, and perform rate limiting processing on the database query statement when it is determined that the traffic type of the database query statement is a restricted traffic type of the first database. This scheme can process database query statements accessing the database at the granularity of the traffic type to which the database query statement belongs, considering finer granularity and higher flexibility, thereby improving the efficiency of solving the DB overload problem.

[0134] Figure 4 A schematic diagram of the structure of a query statement processing apparatus provided as an exemplary embodiment of this disclosure;

[0135] The device includes:

[0136] Unit 41 is used to retrieve database query statements;

[0137] The determining unit 42 is used to determine the target traffic type to which the database query statement belongs. The target traffic type is any one of the following traffic types: traffic related to web pages, traffic related to open application programming interfaces, traffic related to cloud function services, and traffic related to business processes corresponding to automatic triggering mechanisms.

[0138] The determining unit 42 is used to determine whether the target traffic type is a restricted traffic type of the first database. If so, it obtains the restriction information corresponding to the target traffic type. The restricted traffic type is determined based on the first SQL cost difference corresponding to multiple historical traffic data groups of the multiple traffic types in adjacent time periods when the first database is detected to be overloaded.

[0139] Processing unit 43 is used to process the database query statement based on the restriction information.

[0140] According to one or more embodiments of this disclosure, when the aforementioned apparatus is used to process the database query statement based on the restriction information, it is specifically used for:

[0141] In response to the restriction information indicating that the number of times the target traffic type of statement accesses the first database within a first preset time period has reached a preset threshold, the database query statement is subject to rate limiting.

[0142] According to one or more embodiments of this disclosure, when the aforementioned apparatus is used to rate limit the database query statement, it is specifically used for:

[0143] In response to the database query statement being a read data statement, the database query statement is allowed to read data from the second database;

[0144] If the database query statement is a write statement, no processing is performed.

[0145] According to one or more embodiments of this disclosure, the aforementioned apparatus is further used for:

[0146] In response to detecting that the CPU utilization rate of the processor in the first database exceeds a preset threshold, it is determined that the first database is overloaded.

[0147] According to one or more embodiments of this disclosure, the adjacent time periods are the first time period and the time period preceding the first time period, each of the historical traffic data groups includes at least one piece of historical access data, and the aforementioned apparatus is further configured to:

[0148] In response to detecting that the first database is overloaded, historical traffic data groups corresponding to each of the multiple traffic types are obtained to obtain multiple historical traffic data groups corresponding to the multiple traffic types.

[0149] For each historical access data in the multiple historical access data included in the multiple historical traffic data groups, calculate the difference between the first structured query language SQL cost corresponding to the historical access data in the first time period and the second SQL cost corresponding to the historical access data in the time period before the first time period, to obtain the first SQL cost difference corresponding to the historical access data, and then obtain multiple first SQL cost differences corresponding to the multiple historical access data.

[0150] The restricted traffic type is determined based on the cost difference of the multiple first SQL statements.

[0151] According to one or more embodiments of this disclosure, when the aforementioned apparatus is used to determine the restricted traffic type based on the plurality of first SQL cost differences, it is specifically used for:

[0152] For each of the multiple first SQL cost differences, obtain the preset coefficient corresponding to the traffic type of the first SQL cost difference;

[0153] Calculate the product of the first SQL cost difference and the preset coefficient to obtain the first cost information corresponding to the first SQL cost difference, and then obtain the multiple first cost information corresponding to the multiple first SQL cost differences;

[0154] The plurality of first cost information are sorted to obtain a first sorting result;

[0155] The traffic restriction type is determined based on the first sorting result.

[0156] According to one or more embodiments of this disclosure, after determining the restricted flow type based on the first sorting result, the aforementioned apparatus is further configured to:

[0157] After a second preset time period, it is detected whether the first database is still in an overloaded state. If so, for each historical access data, the third SQL cost corresponding to the historical access data in the second time period is calculated, and then multiple third SQL costs corresponding to the multiple historical access data are obtained.

[0158] The restricted traffic type is determined based on the multiple third-party SQL costs.

[0159] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, they will not be repeated here. Specifically, the device can execute the above method embodiments, and the foregoing and other operations and / or functions of each module in the device correspond to the corresponding processes in the various methods in the above method embodiments, which will not be repeated here for the sake of brevity.

[0160] The apparatus of this disclosure embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this disclosure can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this disclosure embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0161] Figure 5 This is a schematic block diagram of an electronic device provided in an embodiment of this disclosure. The electronic device may include:

[0162] The system includes a memory 501 and a processor 502. The memory 501 stores computer programs and transfers the program code to the processor 502. In other words, the processor 502 can retrieve and run the computer programs from the memory 501 to implement the methods described in this embodiment.

[0163] For example, the processor 502 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0164] In some embodiments of this disclosure, the processor 502 may include, but is not limited to:

[0165] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0166] In some embodiments of this disclosure, the memory 501 includes, but is not limited to:

[0167] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0168] In some embodiments of this disclosure, the computer program may be divided into one or more modules, which are stored in the memory 501 and executed by the processor 502 to perform the method provided in this disclosure. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0169] like Figure 5 As shown, the electronic device may also include:

[0170] Transceiver 503, which may be connected to processor 502 or memory 501.

[0171] The processor 502 can control the transceiver 503 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 503 may include a transmitter and a receiver. The transceiver 503 may further include antennas, and the number of antennas may be one or more.

[0172] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0173] This disclosure also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this disclosure also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0174] When implemented using software, it can be implemented wholly or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0175] According to one or more embodiments of this disclosure, a query statement processing method is provided, including:

[0176] Retrieve the database query statement;

[0177] Determine the target traffic type to which the database query statement belongs. The target traffic type is any one of the following traffic types: traffic related to web pages, traffic related to open application programming interfaces, traffic related to cloud function services, and traffic related to business processes corresponding to automatic triggering mechanisms.

[0178] Determine whether the target traffic type is a restricted traffic type of the first database. If so, obtain the restriction information corresponding to the target traffic type. The restricted traffic type is determined based on the first SQL cost difference corresponding to multiple historical traffic data groups of the multiple traffic types in adjacent time periods when the first database is detected to be overloaded.

[0179] The database query statement is processed based on the aforementioned restriction information.

[0180] According to one or more embodiments of this disclosure, processing the database query statement based on the restriction information includes:

[0181] In response to the restriction information indicating that the number of times the target traffic type of statement accesses the first database within a first preset time period has reached a preset threshold, the database query statement is subject to rate limiting.

[0182] According to one or more embodiments of this disclosure, the rate limiting of the database query statement includes:

[0183] In response to the database query statement being a read data statement, the database query statement is allowed to read data from the second database;

[0184] If the database query statement is a write statement, no processing is performed.

[0185] According to one or more embodiments of this disclosure, the method further includes:

[0186] In response to detecting that the CPU utilization rate of the processor in the first database exceeds a preset threshold, it is determined that the first database is overloaded.

[0187] According to one or more embodiments of this disclosure, the adjacent time periods are a first time period and the time period preceding the first time period, each of the historical traffic data groups includes at least one piece of historical access data, and the method further includes:

[0188] In response to detecting that the first database is overloaded, historical traffic data groups corresponding to each of the multiple traffic types are obtained to obtain multiple historical traffic data groups corresponding to the multiple traffic types.

[0189] For each historical access data in the multiple historical access data included in the multiple historical traffic data groups, calculate the difference between the first structured query language SQL cost corresponding to the historical access data in the first time period and the second SQL cost corresponding to the historical access data in the time period before the first time period, to obtain the first SQL cost difference corresponding to the historical access data, and then obtain multiple first SQL cost differences corresponding to the multiple historical access data.

[0190] The restricted traffic type is determined based on the cost difference of the multiple first SQL statements.

[0191] According to one or more embodiments of this disclosure, determining the restricted traffic type based on the plurality of first SQL cost differences includes:

[0192] For each of the multiple first SQL cost differences, obtain the preset coefficient corresponding to the traffic type of the first SQL cost difference;

[0193] Calculate the product of the first SQL cost difference and the preset coefficient to obtain the first cost information corresponding to the first SQL cost difference, and then obtain the multiple first cost information corresponding to the multiple first SQL cost differences;

[0194] The plurality of first cost information are sorted to obtain a first sorting result;

[0195] The traffic restriction type is determined based on the first sorting result.

[0196] According to one or more embodiments of this disclosure, after determining the restricted traffic type based on the first sorting result, the method further includes:

[0197] After a second preset time period, it is detected whether the first database is still in an overloaded state. If so, for each historical access data, the third SQL cost corresponding to the historical access data in the second time period is calculated, and then multiple third SQL costs corresponding to the multiple historical access data are obtained.

[0198] The restricted traffic type is determined based on the multiple third-party SQL costs.

[0199] According to one or more embodiments of the present disclosure, a data processing apparatus is provided, wherein the apparatus includes:

[0200] The retrieval unit is used to retrieve database query statements;

[0201] The determining unit is used to determine the target traffic type to which the database query statement belongs. The target traffic type is any one of the following traffic types: traffic related to web pages, traffic related to open application programming interfaces, traffic related to cloud function services, and traffic related to business processes corresponding to automatic triggering mechanisms.

[0202] The determining unit is used to determine whether the target traffic type is a restricted traffic type of the first database. If so, it obtains the restriction information corresponding to the target traffic type. The restricted traffic type is determined based on the first SQL cost difference corresponding to multiple historical traffic data groups of the multiple traffic types in adjacent time periods when the first database is detected to be overloaded.

[0203] The processing unit is used to process the database query statement based on the restriction information.

[0204] According to one or more embodiments of this disclosure, when the aforementioned apparatus is used to process the database query statement based on the restriction information, it is specifically used for:

[0205] In response to the restriction information indicating that the number of times the target traffic type of statement accesses the first database within a first preset time period has reached a preset threshold, the database query statement is subject to rate limiting.

[0206] According to one or more embodiments of this disclosure, when the aforementioned apparatus is used to rate limit the database query statement, it is specifically used for:

[0207] In response to the database query statement being a read data statement, the database query statement is allowed to read data from the second database;

[0208] If the database query statement is a write statement, no processing is performed.

[0209] According to one or more embodiments of this disclosure, the aforementioned apparatus is further used for:

[0210] In response to detecting that the CPU utilization rate of the processor in the first database exceeds a preset threshold, it is determined that the first database is overloaded.

[0211] According to one or more embodiments of this disclosure, the adjacent time periods are the first time period and the time period preceding the first time period, each of the historical traffic data groups includes at least one piece of historical access data, and the aforementioned apparatus is further configured to:

[0212] In response to detecting that the first database is overloaded, historical traffic data groups corresponding to each of the multiple traffic types are obtained to obtain multiple historical traffic data groups corresponding to the multiple traffic types.

[0213] For each historical access data in the multiple historical access data included in the multiple historical traffic data groups, calculate the difference between the first structured query language SQL cost corresponding to the historical access data in the first time period and the second SQL cost corresponding to the historical access data in the time period before the first time period, to obtain the first SQL cost difference corresponding to the historical access data, and then obtain multiple first SQL cost differences corresponding to the multiple historical access data.

[0214] The restricted traffic type is determined based on the cost difference of the multiple first SQL statements.

[0215] According to one or more embodiments of this disclosure, when the aforementioned apparatus is used to determine the restricted traffic type based on the plurality of first SQL cost differences, it is specifically used for:

[0216] For each of the multiple first SQL cost differences, obtain the preset coefficient corresponding to the traffic type of the first SQL cost difference;

[0217] Calculate the product of the first SQL cost difference and the preset coefficient to obtain the first cost information corresponding to the first SQL cost difference, and then obtain the multiple first cost information corresponding to the multiple first SQL cost differences;

[0218] The plurality of first cost information are sorted to obtain a first sorting result;

[0219] The traffic restriction type is determined based on the first sorting result.

[0220] According to one or more embodiments of this disclosure, after determining the restricted flow type based on the first sorting result, the aforementioned apparatus is further configured to:

[0221] After a second preset time period, it is detected whether the first database is still in an overloaded state. If so, for each historical access data, the third SQL cost corresponding to the historical access data in the second time period is calculated, and then multiple third SQL costs corresponding to the multiple historical access data are obtained.

[0222] The restricted traffic type is determined based on the multiple third-party SQL costs.

[0223] According to one or more embodiments of this disclosure, this disclosure provides an electronic device, including:

[0224] Processor; and

[0225] Memory for storing the executable instructions of the processor;

[0226] The processor is configured to execute the methods described above by executing the executable instructions.

[0227] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the aforementioned methods.

[0228] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0229] In the embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0230] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this disclosure may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0231] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A query statement processing method, characterized in that, include: Retrieve the database query statement; Determine the target traffic type to which the database query statement belongs. The target traffic type is any one of the following traffic types: traffic related to web pages, traffic related to open application programming interfaces, traffic related to cloud function services, and traffic related to business processes corresponding to automatic triggering mechanisms. Determine whether the target traffic type is a restricted traffic type of the first database. If so, obtain the restriction information corresponding to the target traffic type. The restricted traffic type is determined based on the first SQL cost difference corresponding to multiple historical traffic data groups of the multiple traffic types in adjacent time periods when the first database is detected to be overloaded. The database query statement is processed based on the aforementioned restriction information; The adjacent time periods are the first time period and the time period preceding the first time period, and each of the historical traffic data groups includes at least one historical access data item. The method further includes: In response to detecting that the first database is overloaded, historical traffic data groups corresponding to each of the multiple traffic types are obtained to obtain multiple historical traffic data groups corresponding to the multiple traffic types. For each historical access data in the multiple historical access data included in the multiple historical traffic data groups, calculate the difference between the first structured query language SQL cost corresponding to the historical access data in the first time period and the second SQL cost corresponding to the historical access data in the previous time period of the first time period, to obtain the first SQL cost difference corresponding to the historical access data, and then obtain the multiple first SQL cost differences corresponding to the multiple historical access data. The restricted traffic type is determined based on the cost difference of the multiple first SQL statements.

2. The method according to claim 1, characterized in that, The process of processing the database query statement based on the restriction information includes: In response to the restriction information indicating that the number of times the target traffic type of statement accesses the first database within a first preset time period has reached a preset threshold, the database query statement is subject to rate limiting.

3. The method according to claim 2, characterized in that, The rate limiting of the database query statements includes: In response to the database query statement being a read data statement, the database query statement is allowed to read data from the second database.

4. The method according to claim 1, characterized in that, The method further includes: In response to detecting that the CPU utilization rate of the processor in the first database exceeds a preset threshold, it is determined that the first database is overloaded.

5. The method according to claim 1, characterized in that, The determination of the restricted traffic type based on the plurality of first SQL cost differences includes: For each of the multiple first SQL cost differences, obtain the preset coefficient corresponding to the traffic type of the first SQL cost difference; Calculate the product of the first SQL cost difference and the preset coefficient to obtain the first cost information corresponding to the first SQL cost difference, and then obtain the multiple first cost information corresponding to the multiple first SQL cost differences; The plurality of first cost information are sorted to obtain a first sorting result; The traffic restriction type is determined based on the first sorting result.

6. The method according to claim 5, characterized in that, After determining the restricted traffic type based on the first sorting result, the method further includes: After a second preset time period, it is detected whether the first database is still in an overloaded state. If so, for each historical access data, the third SQL cost corresponding to the historical access data in the second time period is calculated, and then multiple third SQL costs corresponding to the multiple historical access data are obtained. The restricted traffic type is determined based on the multiple third-party SQL costs.

7. A query statement processing device, characterized in that, include: The retrieval unit is used to retrieve database query statements; The determining unit is used to determine the target traffic type to which the database query statement belongs. The target traffic type is any one of the following traffic types: traffic related to web pages, traffic related to open application programming interfaces, traffic related to cloud function services, and traffic related to business processes corresponding to automatic triggering mechanisms. The determining unit is used to determine whether the target traffic type is a restricted traffic type of the first database. If so, it obtains the restriction information corresponding to the target traffic type. The restricted traffic type is determined based on the first SQL cost difference corresponding to multiple historical traffic data groups of the multiple traffic types in adjacent time periods when the first database is detected to be overloaded. The processing unit is used to process the database query statement based on the restriction information; The adjacent time periods are the first time period and the time period preceding the first time period. Each historical traffic data group includes at least one historical access data item. The determining unit is further configured to, in response to detecting the first database overload, acquire the historical traffic data group corresponding to each of the multiple traffic types to obtain multiple historical traffic data groups corresponding to the multiple traffic types; for each historical access data item included in the multiple historical access data items in the multiple historical traffic data groups, calculate the difference between the first structured query language (SQL) cost corresponding to the historical access data in the first time period and the second SQL cost corresponding to the historical access data in the time period preceding the first time period to obtain the first SQL cost difference corresponding to the historical access data, and then obtain the multiple first SQL cost differences corresponding to the multiple historical access data items; determine the restricted traffic type based on the multiple first SQL cost differences.

8. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.

Images