Data query method, apparatus and device
By caching query requests and datasets in external memory, the method addresses the high concurrency issues in analytical databases, reducing resource overload and preventing query timeouts, thus enhancing user experience.
Patent Information
- Application Number
- JP2020539715
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2018-01-19
- Filing Date
- 2019-01-11
- Publication Date
- 2024-11-14
- Estimated Expiration
- 2039-01-11
AI Technical Summary
Analytical databases face issues with high concurrency of query requests, leading to CPU, memory, and network bandwidth overload, resulting in client query timeouts and failures due to the large volume of data being returned within a short period.
A data query method and device for a front-end server that caches query requests and datasets in external memory, managing resource occupancy to reduce the load on CPU, memory, and network resources, thereby avoiding timeouts and improving user experience.
The caching of query requests and datasets in external memory effectively manages resource usage, reducing CPU, memory, and network bandwidth occupancy, preventing query timeouts and improving user experience by handling high concurrency without failure.
Smart Images

Figure 0007583501000006 
Figure 0007583501000007 
Figure 0007583501000008
Abstract
Description
[Technical field]
[0001] This application claims priority to Chinese Patent Application No. 201810053977.8, filed on January 19, 2018, and entitled “DATA QUERY METHOD, APPARATUS AND DEVICE,” the entirety of which is incorporated herein by reference.
[0002] The present application relates to the field of Internet technology, and more particularly to a data query method, apparatus, and device. [Background technology]
[0003] An analytical database is a real-time computing engine that can perform analytical and statistical tasks on large amounts of data of any dimension, supporting high concurrency, low latency (millisecond response), real-time online analysis, large amounts of data query, and many other features. Also, an analytical database can store large amounts of data. After receiving a query request sent by a client, it can query the data corresponding to the query request and return the queried data to the client.
[0004] However, in certain scenarios (such as map data queries, data profiling queries, etc.), a large number of query requests may be received within a short period of time (i.e., the number of concurrent requests is very large), and the amount of data corresponding to each query request is also very large. In such cases, a large number of query requests need to be processed in a very short period of time, and a large amount of data is returned for each query request, causing abnormalities in central processing unit (CPU) resources, memory resources, network bandwidth, etc. Such situations may result in client queries timeout or even failure, which affects the user experience. Summary of the Invention
[0005] The present application provides a data query method, the method being applied to a front-end server, the method comprising: receiving a query request sent by a client; determining resource occupancy information for a dataset corresponding to the query request; In response to determining to cache the query request according to the resource occupancy information, obtaining a dataset corresponding to the query request and storing the dataset in an external memory; reading the data set from the external memory and transmitting the data set to the client.
[0006] The present application further provides a data query device, the device being applied to a front-end server, the device comprising: a receiving module used to receive a query request sent by a client; a determination module used to determine resource occupancy information of a dataset corresponding to a query request; an acquisition module used to acquire a dataset corresponding to the query request in response to determining to cache the query request according to the resource occupancy information; a storage module used for storing the dataset in an external memory; a transmission module used to read the data set from the external memory and transmit the data set to the client.
[0007] The present application further provides a front-end server, the front-end server comprising: A receiver used to receive a query request sent by a client; a processor used to determine resource occupancy information of a dataset corresponding to the query request, and in response to determining to cache the query request according to the resource occupancy information, obtain the dataset corresponding to the query request, store the dataset in an external memory, and read the dataset from the external memory; a transmitter used to transmit the data set to the client.
[0008] Based on the technical solutions provided above, in an exemplary embodiment of the present application, after a query request is received, instead of directly sending the dataset to the client, the query request and the dataset are first stored in an external memory. In this way, the dataset corresponding to the query request is locally cached, thereby avoiding processing a large number of query requests within a short period of time. That is, by avoiding returning a large number of datasets to the client within a short period of time, the occupation of CPU resources, memory resources, and network bandwidth is reduced, the timeout or possible failure of client queries is avoided, and the user experience is improved. [Brief description of the drawings]
[0009] In order to more clearly describe the exemplary embodiments of the present application or the technical solutions of the prior art, the drawings of the exemplary embodiments of the present application or the prior art are briefly described. Obviously, the drawings in the following description are only some exemplary embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings of the exemplary embodiments of the present application. [Figure 1] FIG. 1 is a schematic diagram of a system structure in an exemplary embodiment of the present application. [Figure 2A] FIG. 2 is a schematic diagram of a query request process in an exemplary embodiment of the present application. [Figure 2B] FIG. 13 is a schematic diagram of a query request process in another exemplary embodiment of the present application; [Diagram 3] 1 is a flowchart of a data query method in an exemplary embodiment of the present application. [Figure 4A] 4 is a flowchart of a data query method in another exemplary embodiment of the present application. [Figure 4B] 4 is a flowchart of a data query method in another exemplary embodiment of the present application. [Figure 4C] 4 is a flowchart of a data query method in yet another exemplary embodiment of the present application. [Figure 5A]FIG. 2 is a schematic diagram of an effect in an exemplary embodiment of the present application. [Figure 5B] FIG. 13 is a schematic diagram of an effect in another exemplary embodiment of the present application. [Figure 5C] FIG. 13 is a schematic diagram of an effect in another exemplary embodiment of the present application. [Figure 5D] 1 is a schematic diagram of an effect in another exemplary embodiment of the present application. [Figure 5E] FIG. 13 is a schematic diagram of an effect in yet another exemplary embodiment of the present application. [Figure 6] FIG. 2 is a structural diagram of a data query device in an exemplary embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] The terminology used in this application is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting of the application. As used in this application and in the claims, the singular forms "a," "said," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. The term "and / or" as used herein is also to be understood to refer to any or all possible combinations including one or more of the listed items.
[0011] In this specification, terms such as first, second, third, etc. may be used to describe various information in the present application, but it should be understood that the information should not be limited to these terms. These terms are used only to distinguish the same type of information from each other. For example, the first information may also be called the second information without departing from the scope of the present application. Similarly, the second information may also be called the first information depending on the context. Also, the word "if" may be interpreted as "when" or "during" or "in response to a decision."
[0012] The exemplary embodiment of the present application proposes a data query method that can be applied to a system including a client, a front-end server (also called a front node), a computing server (also called a computing node), and a database such as a system used to implement an analytical database (such as an analytical DB called ADS). Of course, other servers, such as a resource scheduling server, which are not limited in the present application, may also be included.
[0013] Referring to Fig. 1, which is a schematic diagram of an application scenario according to an exemplary embodiment of the present application, the number of clients, front-end servers, computing servers, and databases can be one or more. In Fig. 1, three clients, one front-end server, three computing servers, and three databases are used in the example. Of course, the number of clients, front-end servers, computing servers, and databases can be other values and are not limited in the present application.
[0014] Among them, analytical databases are real-time computation engines that can perform analytical and statistical tasks using large amounts of data of all dimensions, and can support high concurrency, low latency, real-time online analysis, large-scale data queries, and other capabilities.
[0015] The client is used to query data from a database, such as an APP (application) on a terminal device (such as a PC (personal computer), a notebook computer, a mobile terminal, etc.) or a browser on a terminal device. There is no limitation to the specific type of the client.
[0016] The database is used to store various types of data, and the data stored in the database can be provided to the client. Regarding the specific type of data stored in the database, there is no limitation in the exemplary embodiment of the present application, and it can be user data, data, map data, video data, image data, audio data, etc.
[0017] The front-end server is used to receive the query request sent by the client, perform SQL (Structured Query Language) parsing on the query request, use the SQL parsing result to generate a scheduling request, and then send the scheduling request to the computing server. The scheduling request is used to request data corresponding to the query request. It also receives the data returned by the computing server, and then sends the data to the client.
[0018] The computing server receives the scheduling request sent by the front-end server, uses the scheduling request to read data corresponding to the scheduling request from the database, and then sends the data to the front-end server.
[0019] As shown in FIG. 2A, the front-end server can start multiple threads, and each thread is used to process a query request. For example, after receiving query request 1, the front-end server starts thread 1 for query request 1, and thread 1 processes query request 1, i.e., performs operations such as sending a scheduling request to the computing server and sending data returned by the computing server to the client. After sending the data to the client, thread 1 is released. When thread 1 is working, if the front-end server receives query request 2, thread 2 is started for query request 2, and thread 2 processes query request 2. When both thread 1 and thread 2 are working, if the front-end server receives query request 3, thread 3 is started for query request 3, and thus thread 3 processes query request 3, and so on.
[0020] In summary, when a front-end server receives a large number of query requests within a short period of time (i.e., high concurrency), the front-end server may start a large number of threads, and then the large number of threads process a large number of query requests. In this case, if the amount of data corresponding to each query request is large, a large amount of data needs to be returned for each query request, which will cause abnormalities in CPU resources, memory resources, network bandwidth, etc., resulting in a decrease in the processing performance of the front-end server, and further causing problems of client query timeout or failure in some cases, which will have a negative impact on the user experience. For example, when a large amount of data needs to be returned for each query request, these data will occupy a large amount of memory resources. As a result, problems such as memory overflow or frequent memory recycling may occur, resulting in query timeout or failure.
[0021] To address the aforementioned problem, as shown in FIG. 2B, in an exemplary embodiment of the present application, the front-end server may start multiple query threads, each of which is used to process a query request. The front-end server may also start a scheduling thread, which is used to sequentially process the query requests in the external memory. For example, after receiving query request 1, the front-end server starts query thread 1 for query request 1, and query thread 1 processes query request 1. When query thread 1 is working, if the front-end server receives query request 2, query thread 2 is started for query request 2, and thus query thread 2 processes query request 2. When both query thread 1 and query thread 2 are working, if the front-end server receives query request 3, query request 4, query request 5, etc., no further query threads are started for these query requests, but these query requests are stored in the external memory.
[0022] The front-end server can start a scheduling thread. The scheduling thread first processes query request 3 in the external memory. When the processing of query request 3 is completed, the scheduling thread processes query request 4 in the external memory. After the processing of query request 4 is completed, the scheduling thread then processes query request 5 in the external memory, and so on.
[0023] In summary, when a front-end server receives a large number of query requests within a short period of time (i.e., high concurrency), the number of query threads to be started is controlled to avoid processing a large number of query requests within a short period of time. In this way, the present application can avoid returning a large amount of data to a client within a short period of time, thereby reducing the occupation of CPU resources, memory resources, and network bandwidth, improving processing performance, and avoiding the timeout or failure of client queries, thereby improving user experience.
[0024] In the above application scenario, referring to FIG. 3, which is a flowchart of the data query method proposed in an exemplary embodiment of the present application, the method can be applied to a front-end server, and the method can include the following steps: Step 301 includes receiving a query request sent by a client; Step 302 includes determining resource occupancy information for a dataset corresponding to the query request; Step 303 includes, in response to determining to cache the query request according to the resource occupancy information, obtaining a dataset corresponding to the query request and storing the dataset in an external memory; Step 304 includes reading the data set from the external memory and transmitting the data set to the client.
[0025] In one example, the above execution order is only an example given for convenience of description. In actual application, the execution order of steps may also be changed, and there is no limitation on the execution order in this application. Furthermore, in other exemplary embodiments, the steps of the corresponding method do not necessarily need to be performed in the order shown and described in this description. The method may include more or less steps than those described in this description. Also, each step described in this description may be divided into multiple steps in the description of other exemplary embodiments. Multiple steps described in this description may be combined into a single step in the description of other exemplary embodiments.
[0026] Regarding step 301, in one example, when a client requests data, a query request may be sent, and the front-end server receives the query request sent by the client, and then the query request is used to request data in the database.
[0027] Regarding step 302, after determining the resource occupancy information of the dataset corresponding to the query request, if it is determined that the query request is not to be cached according to the resource occupancy information, the dataset corresponding to the query request can be obtained, and then the dataset is sent to the client without caching the dataset.
[0028] For step 303, the process of obtaining a dataset corresponding to the query request and storing the dataset in an external memory may include, but is not limited to, storing context information corresponding to the query request in an external memory, then reading the context information from the external memory, using the context information to obtain a dataset corresponding to the query request, and then storing the dataset in the external memory.
[0029] Regarding step 303, in one example, after receiving a query request, the front-end server may directly store the context information corresponding to the query request in an external memory, or the front-end server may store the context information corresponding to the query request in an external memory only when a specific specific condition is met.
[0030] The storage medium for storing the context information and the dataset corresponding to the query request (for convenience of explanation, the data finally returned to the client is called the dataset, and the dataset will be described in more detail in the following section of this application) is called the external memory, not the internal memory (or memory), to avoid occupying the resources of the internal memory. The external memory refers to a storage device other than the internal memory and the CPU cache. The external memory may include, but is not limited to, a hard disk, a floppy disk, an optical disk, a U disk, etc. There is no restriction on the type of the external memory, as long as it is different from the internal memory.
[0031] The context information corresponding to the query request may include, but is not limited to, a query identifier corresponding to the query request, a receiving time of the query request, user information (such as a client's IP address) corresponding to the query request, an expected data volume corresponding to the query request, or one or any combination of the query request. For an example of the context information corresponding to the query request, see Table 1, which is not limited in this application.
[0032] [Table 1]
[0033] The query request can correspond to a query identifier, which can be unique, i.e., different query requests have different query identifiers. Through the query identifier (such as the query identifier 101 shown in Table 1), the front-end server can obtain the contents of the query request, such as the reception time, user information, expected data volume, and so on.
[0034] The receiving time refers to the time when the front-end server receives the query request, that is, when the front-end server receives the query request, it may record the receiving time of the query request, such as 2017.11.6.14.16.10.
[0035] The user information may include, but is not limited to, the client's IP address (the source IP address of the query request), user identification information (username, password, etc. that can be obtained from the query request), and there are no limitations thereto in this application.
[0036] The estimated data amount refers to the determined data amount of the dataset corresponding to the query request (i.e., the data size of the dataset). For example, after receiving the query request 1, the front-end server has not actually obtained the dataset corresponding to the query request 1 yet, so it can be determined for the data amount of the dataset corresponding to the query request 1 first. The data amount in this specification is the estimated data amount, and the determination process will be described in the following section.
[0037] The query request is a query request received by the front-end server, including all content carried by the query request.
[0038] In one example, the process of "a front-end server storing context information corresponding to a query request in an external memory when a certain condition is met" may include, but is not limited to, the following situations: Case 1 as shown in Figure 4A is another flowchart of a data query method. The method includes the following steps: Step 411 includes determining resource occupancy information for a data set corresponding to the query request.
[0039] Step 412 includes determining whether to cache the query request according to the resource occupancy information.
[0040] If yes, step 413 is performed; if no, step 414 is performed.
[0041] Step 413 includes storing the context information corresponding to the query request in an external memory.
[0042] Further, a step of reading the context information from the external memory is performed, the context information is used to obtain a dataset corresponding to the query request, and the dataset is then stored in the external memory.
[0043] Step 414 includes obtaining a data set corresponding to the query request and sending the data set to the client.
[0044] For step 302 or step 411, in one example, the process of "determining resource occupancy information of a dataset corresponding to a query request" may include, but is not limited to, querying a mapping table via a data identifier carried by the query request to obtain resource occupancy information of a dataset corresponding to the data identifier, in which case the mapping table is used to record the correspondence between the data identifier and the resource occupancy information. Further, to generate the mapping table, after obtaining the dataset corresponding to the query request, the front-end server may record the correspondence between the data identifier carried by the query request and the resource occupancy information of the dataset in the mapping table.
[0045] For example, after receiving a query request for dataset A for the first time, the front-end server queries the mapping table via a data identifier carried by the query request (such as data identifier 1 corresponding to dataset A). Because no resource occupancy information corresponding to data identifier 1 exists in the mapping table, the front-end server determines that the resource occupancy information is default resource occupancy information 1 (resource occupancy information 1 is pre-configured based on experience, and when no resource occupancy information corresponding to a data identifier exists in the mapping table, resource occupancy information 1 is determined).
[0046] Furthermore, after the front-end server obtains the dataset corresponding to the query request (in a later section, a method for obtaining the dataset corresponding to the query request will be described), the resource occupancy information of the dataset can be obtained. If the resource occupancy information is resource occupancy information A, the correspondence between the data identifier 1 and the resource occupancy information A can be recorded in a mapping table. As shown in Table 2, this is an example of a mapping table.
[0047] [Table 2]
[0048] After the front-end server receives a query request to request the data set A again (for the second time, the third time, etc.), the resource occupancy information A corresponding to the data identifier 1 can be obtained by querying the mapping table using the data identifier 1 carried by the query request. That is, the determination result is the resource occupancy information A. After obtaining the data set corresponding to the query request, the front-end server can aggregate the resource occupancy information of the data set. If the current aggregation result is the same as the resource occupancy information A, the resource occupancy information A is left unchanged; if the current aggregation result is different from the resource occupancy information A, the resource occupancy information A is replaced with the current aggregation result.
[0049] In one example, the above resource occupancy information may include, but is not limited to, one or any combination of memory occupancy, CPU occupancy, and amount of data in the dataset (i.e., size of the dataset).
[0050] The process of "aggregating resource occupancy information of datasets" may include: after a dataset corresponding to a query request is obtained, a memory occupancy rate corresponding to the dataset can be aggregated, i.e., how much memory is occupied by the dataset can be aggregated, such as 5%, which means that this dataset occupies 5% of the total memory; after a dataset corresponding to a query request is obtained, a CPU usage rate corresponding to the dataset can also be aggregated, i.e., how much CPU is occupied by the dataset can be aggregated, such as 5%, which indicates that the dataset occupies 5% of the total CPU; after a dataset corresponding to a query request is obtained, a data amount corresponding to the dataset can be aggregated, i.e., the size of the dataset can be aggregated.
[0051] Regarding step 412, in one example, the process of “determining whether to cache the query request according to the resource occupancy information” may include, but is not limited to, that if the sum of the resource occupancy information and the currently occupied resources of the front-end server is greater than or equal to a resource threshold, it may be determined that the query request is cached; or, if the sum of the resource occupancy information and the currently occupied resources of the front-end server is less than the resource threshold, it may be determined that the query request is not cached.
[0052] Assuming that the resource occupancy information includes memory occupancy, CPU occupancy, and data amount of a dataset, the resource threshold may include a memory threshold (e.g., 85%), a CPU threshold (e.g., 90%), and a data threshold amount (e.g., 200M). The front-end server can aggregate the current memory usage (e.g., 60%, indicating that 60% of the memory is currently used), the current CPU usage (e.g., 60%, indicating that 60% of the CPU is currently used), and the current bandwidth usage (e.g., 100M, indicating that 100M of bandwidth is currently occupied).
[0053] Next, if the sum of the memory occupancy and the current memory usage is less than the memory threshold, the sum of the CPU occupancy and the current CPU usage is less than the CPU threshold, and the sum of the data volume of the dataset and the current bandwidth usage is less than the data volume threshold, it is determined that the query request is not cached. If one or more of the following three conditions are met: the sum of the memory occupancy and the current memory usage is equal to or greater than the memory threshold, the sum of the CPU occupancy and the current CPU usage is equal to or greater than the CPU threshold, and the sum of the data volume of the dataset and the current bandwidth usage is equal to or greater than the data volume threshold, it is determined that the query request is cached.
[0054] In one example, the resource thresholds (memory thresholds, CPU thresholds, data volume thresholds, etc.) may be fixed values configured based on experience, i.e., the resource thresholds do not change. In another example, the resource thresholds can be dynamically adjusted, i.e., after an initial value of the resource thresholds is configured according to experience, the resource thresholds can then be dynamically adjusted. The dynamic adjustment process of the resource thresholds is described below in connection with certain specific exemplary embodiments.
[0055] For query requests in the external memory, after transmitting the dataset of the query request to the client, the resource threshold can be dynamically adjusted, i.e., each query request in the external memory triggers an adjustment of the resource threshold.
[0056] For example, after receiving query request 1, resource threshold 1 is used to determine whether to cache query request 1, and if cached, context information corresponding to query request 1 is stored in external memory, and after sending a dataset corresponding to query request 1 to the client, resource threshold 1 is adjusted to be resource threshold 2. After receiving query request 2, resource threshold 2 is used to determine whether to cache query request 2, and if not cached, context information corresponding to query request 2 is not stored in external memory. After sending a dataset corresponding to query request 2 to the client, there is no need to adjust resource threshold 2.
[0057] In one example, the process of dynamically adjusting the resource threshold may include obtaining a processing time corresponding to the query request, where the processing time may be a time from receiving the query request to sending the data set to the client. Then, the resource threshold may be adjusted according to the processing time corresponding to the query request. Furthermore, the process of "adjusting the resource threshold according to the processing time corresponding to the query request" may include, but is not limited to, if the processing time corresponding to the query request is longer than a pre-set first time threshold, the value of the resource threshold may be increased, where the increased resource threshold is not greater than an upper limit of the resource threshold, and if the processing time corresponding to the query request is shorter than a pre-set second time threshold, the value of the resource threshold may be decreased, where the decreased resource threshold is not less than a lower limit of the resource threshold.
[0058] The receiving time of the query request can be obtained from the context information, and the time when the data set is sent to the client is the current time for which the resource threshold is adjusted. In summary, the front-end server may determine the difference between the current time and the receiving time as the processing time corresponding to the query request.
[0059] Both the above-mentioned first preset time threshold and the second preset time threshold can be configured according to experience, and there is no limitation thereto, and the first preset time threshold can be greater than the second preset time threshold.
[0060] The upper limit of the resource threshold can be configured according to experience, and there is no limit to the upper limit of the resource threshold. The lower limit of the resource threshold can be configured according to experience, and there is no limit to the lower limit of the resource threshold. Of course, when configuring the upper limit of the resource threshold and the lower limit of the resource threshold, the upper limit of the resource threshold can be greater than the lower limit of the resource threshold.
[0061] Increasing the value of the resource threshold may include calculating the sum of the current resource threshold and the first resource adjustment value, and if the sum of the two is not greater than the upper resource threshold limit, then the increased resource threshold is the sum of the two, whereas if the sum of the two is greater than the upper resource threshold limit, then the increased resource threshold is the upper resource threshold limit.
[0062] Decreasing the value of the resource threshold may include calculating the difference between the current resource threshold and the second resource adjustment value, and if the difference between the two is not less than the lower resource threshold, then the decreased resource threshold is the difference between the two, whereas if the difference between the two is less than the lower resource threshold, then the decreased resource threshold is the lower resource threshold.
[0063] Both the first resource adjustment value and the second resource adjustment value may be configured based on experience, and the first resource adjustment value and the second resource adjustment value may be the same or different.
[0064] In the above exemplary embodiment, if the processing time corresponding to the query request is longer than the preset first time threshold, it means that the processing time of the query request is relatively long when the query request is stored in the cache. Therefore, the number of query requests that need to be cached should be minimized. Based on this, the value of the resource threshold can be increased, so that the sum of the resource occupancy information and the currently occupied resource of the front-end server is less than the resource threshold. As a result, the query request does not need to be cached, and the problem of the processing time of the query request being too long is avoided.
[0065] In the above exemplary embodiment, if the processing time corresponding to the query request is less than the preset second time threshold, it means that the processing time of the query request can be shortened when the query request is stored in the cache. Therefore, the number of query requests that need to be cached can be increased. Based on this, the value of the resource threshold can be decreased. In this way, the sum of the resource occupancy information and the currently occupied resource of the front-end server should be equal to or greater than the resource threshold. As a result, the query request can be cached, and the processing resource of the front-end server can be saved.
[0066] Case 2: Referring to Figure 4B, another flowchart of a data query method is shown. The method includes: Step 421 involves parsing parameter information carried by the query request. Step 422 includes determining whether the parameter information is present in a parameter table, which is used to record the parameter information for the cache, and if the parameter information is present, performing step 423; if the parameter information is not present, performing step 424.
[0067] Step 423 includes storing the context information corresponding to the query request in an external memory.
[0068] Further steps are then performed including reading the context information from the external memory, obtaining a dataset corresponding to the query request using the context information, and storing the dataset in the external memory.
[0069] Step 424 includes obtaining a data set corresponding to the query request and sending the data set to the client.
[0070] In one example, the front-end server can obtain parameter information for caching and storing the parameter information in a parameter table. Based on this, after receiving a query request, the parameter information carried by the query request is analyzed, and then it is determined whether the parameter information exists in the parameter table. If the answer is "yes", the query request needs to be cached, while if the answer is "no", the query request does not need to be cached.
[0071] In one example, the process of "obtaining parameter information for caching from a front-end server and storing the parameter information in a parameter table" may include, but is not limited to, the following methods. Method 1: The front-end server receives a database creation request, the database creation request carries parameter information for caching, and records the parameter information carried by the database creation request in a parameter table.
[0072] If some parameter information needs to be cached, those parameter information for caching may be specified. Specifically, the database creation request including the parameter information may be sent to the front-end server to inform the front-end server of the parameter information for caching. Then, the front-end server records the parameter information carried by the database creation request in a parameter table.
[0073] For example, a database creation request may be create database test options(resource_type='ecu' ecu_type='c1' ecu_count=2 with_result_cache='true'), where create database test options indicates that the current message is a database creation request, resource_type='ecu' is the type of resource model of the target database, ecu_type='c1' is the resource model of the target database, ecu_count=2 is quantity information of the resource model of the target database, and with_result_cache='true' is parameter information for the cache.
[0074] Moreover, the above parameter information is valid for all tables in the current database, which means that for all tables in the current database, if a query request for these tables is received and the query request carries the above parameter information for caching, the query request needs to be cached.
[0075] Method 2: The front-end server receives a table group creation request carrying parameter information for a cache, and records the parameter information carried by the table group creation request in a parameter table.
[0076] If some parameter information needs to be cached, the parameter information for caching may be specified when creating a database table group. Specifically, a table group creation request carrying the parameter information for caching can be sent to the front-end server to inform the front-end server of the parameter information for caching. The front-end server records the parameter information carried by the table group creation request in a parameter table.
[0077] For example, an example of a table group create request may include create tablegroup test options(with_result_cache='true'), where create tablegroup test options indicates that the current message is a table group create request, and with_result_cache='true' is parameter information for caching.
[0078] Moreover, the above parameter information is valid for all tables in the current table group, i.e., for all tables in the current table group, when a query request for these tables is received and the query request carries the above parameter information for caching, the query request indicates that the query request needs to be cached.
[0079] Method 3: A front-end server receives a table creation request, and the table creation request carries parameter information for caching, and then the parameter information carried by the table creation request is recorded in a parameter table.
[0080] If some parameter information needs to be cached, the parameter information for caching can be specified when creating a database table. Specifically, a table creation request carrying the parameter information can be sent to a front-end server, so that the front-end server can be notified of the parameter information for caching. The front-end server then records the parameter information carried by the table creation request in a parameter table.
[0081] For example, an example of a create table request may be CREATE TABLE test (col1 varchar, col2 varchar, PRIMARY KEY(col1)) PARTITION BY HASH KEY(col1) PARTITION NUM 10 SUBPARTITION BY LIST KEY(col2) TABLEGROUP test_group OPTIONS(UPDATETYPE='realtime' with_result_cache='true'), where CREATE TABLE test indicates that the current message is a create table request, (col1 varchar, col2 varchar, PRIMARY KEY(col1)) PARTITION BY HASH KEY(col1) PARTITION NUM 10 SUBPARTITION BY LIST KEY(col2) TABLEGROUP test_group OPTIONS is related to the table, UPDATETYPE='realtime' is table type information, and with_result_cache='true' is parameter information for caching.
[0082] The above parameter information is only valid for the current table, which means that if a query request for the current table is received and the query request carries the parameter information, the query request needs to be cached.
[0083] In the above exemplary embodiment, all tables refer to data tables in a database, which is not a limitation in this application.
[0084] Of course, the above exemplary embodiments only provide some examples of "whether to cache a query request", and there is no limitation thereto in the present application. For example, if a query request needs to be cached, a hint parameter can be carried by the query request. For example, the query request is: / *+withResultCache=true* / select id,name from test limit 100. Because the query request carries the hint parameter, the front-end server decides to cache the query request after receiving the query request. In another example, the front-end server may use the cache through the specific situations of the global switch, the database switch, the table group switch, and the table switch. Specifically, when the global switch is on, all query requests are cached. When the global switch is off and the database switch is on, all query requests to the database are cached. When the global switch and the database switch are off and the table group switch is on, all query requests to the table group are cached. When the global switch, the database switch, and the table group switch are off and the table switch is on, all query requests to the table are cached.
[0085] Case 3, as shown in Figure 4C, another flowchart of a data query method is provided. The method includes: Step 431 involves parsing parameter information carried by the query request. Step 432 involves determining whether the parameter information is present in the parameter table.
[0086] If the answer is yes, then step 433 is executed, if the answer is no, then step 436 is executed.
[0087] Step 433 includes determining resource occupancy information for the data set corresponding to the query request.
[0088] Step 434 includes determining whether to cache the query request according to the resource occupancy information.
[0089] If the answer is yes, then step 435 is executed; if the answer is no, then step 436 is executed.
[0090] Step 435 includes storing the context information corresponding to the query request in an external memory.
[0091] Step 436 includes obtaining a data set corresponding to the query request and sending the data set to the client.
[0092] In the above-mentioned cases 1, 2, and 3, for the process of "obtaining a dataset corresponding to a query request and sending the dataset to a client", the front-end server may perform SQL parsing on the query request, may use the SQL parsing result to generate a scheduling request, and may send the scheduling request to the computing server, in which case the scheduling request is used to request data corresponding to the query request, and then the front-end server may receive the dataset returned by the computing server, and may send the dataset to the client.
[0093] Regarding step 303, in one example, the process of “reading context information from external memory” may include, but is not limited to, determining a priority corresponding to the context information using the context information in the external memory, then sorting the context information in the external memory using the priority corresponding to the context information, and reading the context information with the highest priority from the external memory according to the sorting result.
[0094] The external memory may store a number of context information, and for all the context information, the front-end server can determine a priority corresponding to each context information, and can sort all the context information in the external memory according to a decreasing priority, and then read the context information with the highest priority, i.e., the first context information, or can sort all the context information in the external memory according to a decreasing priority, and then read the context information with the highest priority, i.e., the last context information.
[0095] In one example, the context information may include, but is not limited to, the receiving time of the query request and the predicted data amount corresponding to the query request, as shown in Table 1. Based on this, the process of "using the context information in the external memory to determine the priority corresponding to the context information" includes, but is not limited to, the front-end server determining the waiting time corresponding to the context information using the difference between the current time and the receiving time, and then determining the priority corresponding to the context information according to the waiting time and the predicted data amount.
[0096] Furthermore, the process of “determining a priority corresponding to the context information according to the waiting time and the predicted data amount” may include, but is not limited to, performing a normalization process on the waiting time to obtain a first sub-weight, performing a normalization process on the predicted data amount to obtain a second sub-weight, then obtaining a first weight value according to the first sub-weight and the second sub-weight, and then obtaining a priority corresponding to the context information according to the first weight value, in which the larger the first weight value, the higher the priority corresponding to the context information, and the smaller the first weight value, the lower the priority corresponding to the context information.
[0097] For example, if the context information in the external memory is shown in Table 3, the front-end server can determine a priority A corresponding to query ID 101, a priority B corresponding to query ID 102, and a priority C corresponding to query ID 103. Specifically, if the current time is 2017.11.6.14.16.18 (i.e., 14:16:18 on November 6, 2017), the waiting time corresponding to query identifier 101 is 8 seconds, the waiting time corresponding to query identifier 102 is 3 seconds, and the waiting time corresponding to query identifier 103 is 1 second.
[0098] [Table 3]
[0099] Then, a normalization process is performed on the waiting time, i.e., the waiting time can be converted into a value between 0 and 1. In the conversion process, the longer the waiting time, the larger the normalized first sub-weight. For example, the waiting time corresponding to the query identifier is divided by the sum of the waiting times of all the query identifiers. Therefore, the first sub-weight corresponding to the query identifier 101 is 8 / 12, i.e., 0.667. The first sub-weight corresponding to the query identifier 102 is 3 / 12, i.e., 0.25. The first sub-weight corresponding to the query identifier 103 is 1 / 12, i.e., 0.083. Of course, the above method is only one example of the waiting time normalization process, and there is no limitation on the processing method.
[0100] A normalization process may be performed on the predicted data amount, that is, the predicted data amount can be converted into a value between 0 and 1. In the conversion process, the larger the predicted data amount, the smaller the normalized second sub-weight, and the specific calculation is, for example, subtracting the predicted data amount corresponding to this query identifier from the sum of the predicted data amounts of all query identifiers, and then dividing the result by the sum of the predicted data amounts of all query identifiers. Therefore, the second sub-weight corresponding to the query identifier 101 is (37.6-25.6) / 37.6, that is, 0.319. The second sub-weight corresponding to the query identifier 102 is (37.6-10) / 37.6, that is, 0.734. The second sub-weight corresponding to the query identifier 103 is (37.6-2) / 37.6, that is, 0.947. Of course, the above is only one example of the normalization process, and there is no limitation on the processing method.
[0101] Then, the product of the first sub-weight and the second sub-weight is used as the first weight value. For example, the first weight value corresponding to the query identifier 101 is 0.667*0.319, or 0.213, the first weight value corresponding to the query identifier 102 is 0.25*0.734, or 0.183, and the first weight value corresponding to the query identifier 103 is 0.083*0.947, or 0.079.
[0102] Then, a priority corresponding to the query identifier is obtained according to the first weight value. There is no restriction on this matter as long as the condition that the larger the first weight value is, the higher the priority is, and the smaller the first weight value is, the lower the priority is satisfied. For example, the priority corresponding to the first weight value of 0.213 is 213 (or 21), the priority corresponding to the first weight value of 0.183 is 183 (or 18), and the priority corresponding to the first weight value of 0.079 is 79 (or 8).
[0103] In summary, the priority corresponding to the query identifier 101 may be higher than the priority corresponding to the query identifier 102, and the priority corresponding to the query identifier 102 may be higher than the priority corresponding to the query identifier 103. That is, the sorting result of all the context information in the external memory is the context information corresponding to the query identifier 101, the context information corresponding to the query identifier 102, and the context information corresponding to the query identifier 103. Therefore, the front-end server can read the context information corresponding to the query identifier 101 from the external memory.
[0104] Next, the front-end server analyzes the query request 1 from the context information corresponding to the query identifier 101, and obtains a dataset corresponding to the query request 1. The obtained dataset is dataset A. Then, dataset A is stored in an external memory. Please refer to Table 4, which shows an example of dataset A after it is stored.
[0105] [Table 4]
[0106] Regarding step 303, in one example, the process of "using the context information to obtain a dataset corresponding to the query information" may include, but is not limited to, the steps of: a front-end server parses the query request from the context information, generates a scheduling request corresponding to the query request, and then sends the scheduling request to a computing server, and then the front-end server receives a dataset returned by the computing server corresponding to the scheduling request, where the dataset is also a dataset corresponding to the above-mentioned query request. For convenience of description, the data returned by the computing server to the front-end server may be referred to as a dataset.
[0107] The above query request may be an SQL query request. Thus, the front-end server performs SQL analysis on the query request, and learns that the query request is used to request datasets, such as dataset 1 and dataset 2. Then, the front-end server performs an analysis to find a computing server corresponding to dataset 1, and then sends a scheduling request to the computing server to request dataset 1, and after receiving the scheduling request, the computing server sends dataset 1 to the front-end server. Also, the front-end server performs an analysis to find another computing server corresponding to dataset 2, and then sends a scheduling request to the computing server to request dataset 2, and after receiving the scheduling request, the computing server sends dataset 2 to the front-end server. After receiving dataset 1 and dataset 2, the front-end server can form dataset 1 and dataset 2 into a dataset, i.e., a dataset corresponding to the query request.
[0108] In step 303, when the front-end server reads the context information from the external memory, even if there are many context information in the external memory, the front-end server can only read one context information in one reading process. After using the context information to obtain a data set corresponding to the query request and storing the data set in the external memory, another context information may be read from the external memory (the reading process, the steps of determining priority, sorting and other operations are performed again, but are not repeated here), and so on.
[0109] As shown in FIG. 2B, the front-end server can start a scheduling thread, which first processes query request 3 in the external memory, obtains a dataset corresponding to query request 3, and after storing the dataset in the external memory, the scheduling thread processes query request 4 in the external memory, and so on.
[0110] Regarding step 304, in one example, the process of “reading datasets from external memory” may include, but is not limited to, steps of: using context information in the external memory to determine a dataset corresponding to the context information, then sorting the datasets in the external memory using priorities of the datasets, and then reading the datasets from the external memory according to the sorting results of the datasets.
[0111] A number of datasets may be stored in the external memory, with each dataset having corresponding context information, and for every dataset, the front-end server may use the context information corresponding to each dataset to determine a priority of the dataset and sort all the datasets in the external memory in descending order of priority, or may sort all the datasets in the external memory in descending order of priority.
[0112] In one example, after obtaining the dataset corresponding to the query request, the front-end server may aggregate the data volume of the dataset (i.e., the actual data volume, which may be different from the predicted data volume), and then store the data volume of the dataset in the context information of the dataset. As shown in Table 5, this is an example of context information.
[0113] [Table 5]
[0114] In summary, the context information may include, but is not limited to, the receiving time of the query request and the data amount of the dataset corresponding to the query request. Based on the above, the process of "using the context information in the external memory to determine the priority of the dataset corresponding to the context information" may include, but is not limited to, the front-end server uses the difference between the current time and the receiving time to determine the waiting time corresponding to the context information, and then the priority of the dataset corresponding to the context information is determined according to the waiting time and the data amount of the dataset.
[0115] Furthermore, the process of "determining a priority of a dataset corresponding to context information according to a waiting time and a data amount of the dataset" includes, but is not limited to, performing a normalization process on the waiting time to obtain a third sub-weight, performing a normalization process on the data amount to obtain a fourth sub-weight, obtaining a second weight value according to the third sub-weight and the fourth sub-weight, and obtaining a priority of the dataset according to the second weight value, in which a larger second weight value indicates a higher priority, and a smaller second weight value indicates a lower priority.
[0116] Through a normalization process on the waiting time, the waiting time can be converted into a value from 0 to 1. In the conversion process, the longer the waiting time, the larger the normalized third sub-weight. After the data amount is normalized, the data amount can be converted into a value from 0 to 1. In the conversion process, the larger the data amount, the smaller the normalized fourth sub-weight. When the second weight value is obtained based on the third sub-weight and the fourth sub-weight, the product of the third sub-weight and the fourth sub-weight is used as the second weight value.
[0117] The process of "determining a priority of datasets according to the wait time and the data volume of the datasets" is similar to the process of "determining a priority corresponding to context information according to the wait time and the predicted data volume", the difference between them is that the data volume of the datasets used here is different from the predicted data volume, and this process is not repeated here.
[0118] In summary, if the priority of dataset A is higher than the priority of dataset B, and the priority of dataset B is higher than the priority of dataset C, the sorting result will be dataset A, dataset B, and dataset C.
[0119] In one example, the process of "reading datasets from external memory according to the sorting results of the datasets" may include, but is not limited to, if sorted in order of higher priority, the front-end server may read the top N datasets from the external memory according to the sorting results of the datasets, or if sorted in order of lower priority, the front-end server may read the bottom N datasets from the external memory according to the sorting results of the datasets.
[0120] N is a positive integer equal to or greater than 1. In addition, the sum of the resource occupancy information corresponding to the N data sets and the currently occupied resources of the front-end server is less than the current resource threshold.
[0121] When the resource occupancy information includes a memory occupancy rate, a CPU occupancy rate, and a data volume of a data set, the resource threshold may include a memory threshold, a CPU threshold, and a data volume threshold. In addition, the front-end server can also aggregate a current memory usage, a current CPU usage, and a current bandwidth usage.
[0122] Based on this, the front-end server determines the memory occupancy (see step 411), the CPU usage (see step 411), and the amount of data of the dataset corresponding to the dataset with the highest priority (obtained from Table 5).
[0123] If the sum of the memory occupancy and the current memory usage is less than the memory threshold, while the sum of the CPU usage and the current CPU usage is less than the CPU threshold, and the sum of the data volume of the dataset and the current bandwidth usage is less than the data volume threshold, the dataset with the highest priority is read from the external memory. If one or more of the above three conditions are not met, the dataset is not read from the external memory. After waiting for a preset time, it is determined again whether the dataset with the highest priority can be read from the external memory.
[0124] After reading the data set with the highest priority from the external memory, the front-end server determines the memory occupancy, CPU usage, and data amount of the data set corresponding to the data set with the second highest priority. If the sum of the memory occupancy corresponding to the data set with the highest priority, the memory occupancy corresponding to the data set with the second highest priority, and the current memory usage is less than the memory threshold, while the sum of the CPU usage corresponding to the data set with the highest priority, the CPU usage corresponding to the data set with the second highest priority, and the current CPU usage is less than the CPU threshold, and the sum of the data amount corresponding to the data set with the highest priority, the data amount corresponding to the data set with the second highest priority, and the current bandwidth usage is less than the data amount threshold, the data set with the second highest priority is read from the external memory, and so on. If one or more of the above three conditions are not met, the data set with the second highest priority is not read from the external memory, and the reading process is terminated.
[0125] In summary, the front-end server can read N data sets from the external memory, and the sum of the resource occupancy information corresponding to the N data sets and the currently occupied resources of the front-end server is less than the resource threshold.
[0126] In one example, the front-end server reads the N data sets from the external memory, and then sends the read N data sets to the client. For example, the user information in the context information is used to send the data sets to the corresponding client, and the sending process is not repeated here. Then, the front-end server may delete the context information corresponding to the data sets from the external memory, thus completing the whole process of sending the data sets.
[0127] When the front-end server reads the data sets from the external memory in step 304, even if there are many data sets in the external memory, the front-end server can only read N data sets in a single read process. After the N data sets are sent to the client, the front-end server again reads another data set from the external memory, and so on.
[0128] Based on the above technical solution, in an exemplary embodiment of the present application, after receiving a query request, instead of directly sending the dataset to the client, the query request and the dataset are actually stored in an external memory first. In this way, the dataset corresponding to the query request can be cached locally, thereby avoiding the problem of processing a large number of query requests in a short time, and avoiding returning a large number of datasets to the client in a short time, thereby reducing the occupation of CPU resources, memory resources, and network bandwidth, avoiding the problem of timeout or failure of client queries, and improving user experience. Furthermore, in the above method, a query request containing a large amount of data can be cached in the local storage, but a query request containing a small amount of data is not cached in the local storage. In this way, a dataset containing a small amount of data can be directly sent to the client without affecting the query request containing a small amount of data, and thus the frequency of full gc is reduced as much as possible.
[0129] FIG. 5A is a schematic diagram of the test result. A query is performed on a table with a large amount of data, in which / *+withResultCache=true* / select id,name,....,region from test where id=xxx limit xxx, limit 10, limit 100, limit 500, limit 1000, limit 2000, limit 5000, limit 8000, limit10000 are used to limit the number of rows in the dataset, respectively. In this case, the more rows in the dataset, the more data the dataset has. In FIG. 5A, the relationship between the data amount of the dataset and the average query time is provided. In order to simulate memory usage, another program is run before the test to keep the memory usage at about 75%, and then the above query operation is performed asynchronously. As can be seen in FIG. 5A, by implementing the technical solution of the present application, when the dataset is large, the query time can be significantly reduced, and the frequency of full gc can also be reduced.
[0130] 5B is a schematic diagram of the relationship between memory usage and query time. In this case, an SQL query request is tested by running other programs to control memory usage. The test shows that when memory usage exceeds a certain level, the query time can be significantly reduced if the method of the present application is used.
[0131] As shown in FIG. 5C, this is a schematic diagram of querying user detail data. For example, / *+withResultCache=true * / select id,name,...region from test where id=xxx limit 5000, a total of 100 columns of data are queried, in this case, aggregation is performed according to different concurrency and query timeout rate. As seen in FIG. 5C, the more concurrent queries there are, the higher the timeout rate. Moreover, when a certain number of concurrency is exceeded, the query timeout rate increases significantly. Obviously, after using the method of the present application, the increase in the query timeout rate can be slowed down.
[0132] As shown in FIG. 5D, this is a schematic diagram of a concurrency test that executes a query only on a relatively small data set, for example, / *+withResultCache=true* / select id,name from test where id=xxx limit 50. The relationship between query concurrency and query timeout rate is shown in FIG. 5D. Similar to FIG. 5C, when a certain number of concurrency is exceeded, the query timeout rate begins to increase continuously. Obviously, after using the method of the present application, the critical value of concurrency at which the query timeout rate suddenly increases can be made higher. Therefore, in the same scenario, the number of user concurrency can be increased.
[0133] FIG. 5E is a schematic diagram of a mixed scenario (i.e., there are query operations using large data sets and query operations using small data sets), showing the impact of query operations using large data sets on the timeout rate of query operations using small data sets. As can be seen in FIG. 5E, when the concurrent execution number of query operations using large data sets exceeds a certain critical value, the query timeout rate of query operations using small data sets begins to increase continuously. Obviously, after using the method of the present application, the query timeout rate of query operations using small data sets can be improved.
[0134] Based on the same application concept as the above method, the exemplary embodiment of the present application further provides a data query device that can be applied to a front-end server. As shown in Figure 6, this is a structural diagram of the device, and the device includes: A receiving module 601 used to receive a query request sent by a client; A determination module 602 used to determine resource occupancy information of a dataset corresponding to a query request; an acquisition module 603 used to acquire a dataset corresponding to the query request in response to determining to cache the query request according to the resource occupancy information; a storage module 604 used to store the dataset in an external memory; and a transmission module 605 that is used to read the dataset from the external memory and transmit the dataset to the client.
[0135] The acquisition module 603 is specifically used in the process of acquiring a dataset corresponding to the query request, storing context information corresponding to the query request in an external memory, reading the context information from the external memory, and using the context information to acquire a dataset corresponding to the query request.
[0136] The acquisition module 603 is also specifically used in the process of reading context information from the external memory to determine a priority corresponding to the context information using the context information in the external memory, sort the context information in the external memory using the priority corresponding to the context information, and read the context information with the highest priority from the external memory according to the sorting result.
[0137] In addition, the process of obtaining a dataset corresponding to the query request using the context information includes generating a scheduling request corresponding to the query request, sending the scheduling request to a computing server, and receiving the dataset corresponding to the scheduling request returned by the computing server.
[0138] The sending module 605 is specifically used in the process of reading a dataset from an external memory for determining a priority of a dataset corresponding to the context information using the context information in the external memory, sorting the datasets in the external memory using the priority of the dataset, and reading the datasets from the external memory according to the sorting result of the datasets.
[0139] Based on the same application concept as the above method, an exemplary embodiment of the present application also provides a front-end server, which may include a receiver, a processor, and a transmitter. The receiver is used to receive a query request sent by a client. The processor is used to determine resource occupancy information of a dataset corresponding to the query request, and in response to determining to cache the query request according to the resource occupancy information, the processor is used to obtain a dataset corresponding to the query request, store the dataset in an external memory, and read the dataset from the external memory. The transmitter is used to send the dataset to the client.
[0140] Based on the same application concept as the above method, an exemplary embodiment of the present application also provides a machine-readable storage medium that can be applied to a front-end server, which stores a number of computer instructions, and when the computer instructions are executed, a process is performed to receive a query request sent by a client, determine resource occupancy information of a dataset corresponding to the query request, and in response to determining to cache the query request according to the resource occupancy information, obtain a dataset corresponding to the query request, store the dataset in an external memory, read the dataset from the external memory, and send the dataset to the client.
[0141] For the systems, devices, modules or units described in the above exemplary embodiments, they may be specifically implemented by computer chips or entities, or by products with specific functions. A typical implementation device is a computer, and the specific form of the computer may be a personal computer, a laptop computer, a mobile phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an e-mail sending and receiving device, a game console, a tablet computer, a wearable device, or any combination of the aforementioned devices.
[0142] For convenience of description, when the above devices are described, they are divided into various units based on their functions and described separately. Of course, when implementing the present application, the functions of each unit can be implemented in one or more software and / or hardware.
[0143] Those skilled in the art will appreciate that the exemplary embodiments of the present application may be provided as a method, a system, or a computer program product. Thus, the present application may take the form of an entirely hardware exemplary embodiment, an entirely software exemplary embodiment, or an exemplary embodiment combining software and hardware. Furthermore, the exemplary embodiments of the present application may take the form of a computer program product embodied on one or more computer usable storage mediums (including, but not limited to, disk storage devices, CD-ROMs, optical storage devices, etc.) that contain computer usable program code.
[0144] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to exemplary embodiments of the present application. It should be understood that each step and / or block in the flowcharts and / or block diagrams, and combinations of steps and / or blocks in the flowcharts and / or block diagrams, may be implemented by computer program instructions. These computer program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine that enables the generation of a device for implementing the function specified in the flowcharts and / or in one or more steps in one or more blocks by using the instructions executed by the processor of the computer or other programmable data processing device.
[0145] Further, these computer program instructions may be stored in a computer readable memory that can instruct a computer or other programmable data processing device to operate in a specific manner, causing the instructions stored in the computer readable memory to generate an article of manufacture that includes an instruction device, and the instruction device may implement a function specified in a step or multiple steps of a flowchart and / or a block or multiple block diagrams.
[0146] These computer program instructions may also be loaded onto a computer or other programmable data processing device and cause the computer or programmable data processing device to execute a series of operational steps to generate a computer-implemented process, such that the instructions executing on the computer or programmable data processing device provide steps for implementing a function specified in a step or multiple steps of the flowchart and / or in a block or multiple blocks of the block diagram.
[0147] The above only includes some examples of the present application, and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. 1. A data query method applied to a front-end server, the data query method comprising: receiving a query request sent by a client; determining the resource occupancy information of the dataset corresponding to the query request by querying a mapping table that records a correspondence between the data identifier and resource occupancy information via a data identifier in the query request to obtain the resource occupancy information of the dataset corresponding to the data identifier, wherein the resource occupancy information includes at least a memory usage amount; in response to determining to cache the query request according to the resource occupancy information, obtaining the dataset corresponding to the query request and storing the dataset in an external memory; reading the data set from the external memory and transmitting the data set to the client.
2. After determining the resource occupancy information of the data set corresponding to the query request, the data query method includes:
2. The data query method of claim 1, further comprising: in response to determining not to cache the query request according to the resource occupancy information, obtaining the data set corresponding to the query request and sending the data set to the client.
3. After determining the resource occupancy information of the data set corresponding to the query request, the data query method includes: determining to cache the query request if a sum of the resource occupancy information and the currently occupied resource of the front-end server is equal to or greater than a resource threshold; or The data query method according to claim 1 or 2, further comprising: determining not to cache the query request if the sum of the resource occupancy information and the currently occupied resource of the front-end server is less than the resource threshold.
4. After transmitting the data set to the client, the data query method includes: obtaining a processing time corresponding to the query request, the processing time being a time from when the query request is received to when the data set is sent to the client; and The data query method of claim 3 , further comprising: adjusting the resource threshold according to the processing time corresponding to the query request.
5. adjusting the resource threshold according to the processing time corresponding to the query request, increasing a value of the resource threshold when the processing time corresponding to the query request is longer than a first preset time threshold, where the increased resource threshold is not greater than an upper limit of the resource threshold; or decreasing the value of the resource threshold when the processing time corresponding to the query request is shorter than a second preset time threshold, wherein the decreased resource threshold is not smaller than a lower limit of the resource threshold; Including, The data query method of claim 4 , wherein the first time threshold is greater than the second time threshold.
6. The resource occupancy information is Memory occupancy, CPU usage, and The data query method of claim 1 or 2, further comprising one or any combination of the data amounts of the data sets.
7. A data query device applied to a front-end server, the data query device comprising: a receiving module for receiving a query request sent by a client; a determination module for determining the resource occupancy information of the dataset corresponding to the query request by querying a mapping table that records a correspondence between the data identifier and resource occupancy information via a data identifier in the query request to obtain the resource occupancy information of the dataset corresponding to the data identifier, the resource occupancy information including at least a memory usage; an acquisition module that acquires the data set corresponding to the query request in response to determining to cache the query request according to the resource occupancy information; a storage module for storing the data set in an external memory; a transmission module that reads the data set from the external memory and transmits the data set to the client.
8. a receiver for receiving a query request sent by a client; a processor: determining, via a data identifier in the query request, resource occupancy information of the dataset corresponding to the query request by querying a mapping table that records a correspondence between the data identifier and resource occupancy information to obtain the resource occupancy information of the dataset corresponding to the data identifier; and in response to determining to cache the query request according to the resource occupancy information, obtaining the dataset corresponding to the query request, storing the dataset in an external memory, and reading the dataset from the external memory; a transmitter for transmitting the data set to the client; The resource occupancy information includes at least memory usage.
Citation Information
Patent Citations
Caching method, query method, caching apparatus and query apparatus for database data
CN105354193A
Caching dynamic web documents
US20030101231A1
