Data caching method and device

Abstract

The invention discloses a data caching method and device, and the method comprises the steps: obtaining a plurality of first logs in a first time period from a query log of a database; wherein one log is used for recording query operation of one SQL statement on the database; determining at least one first high-frequency query item through SQL statements corresponding to the plurality of first logs; wherein the preset field in the same SQL statement corresponds to one query item; the high-frequency query item is determined at least based on the occurrence frequency of the query item in the plurality of logs; generating an aggregation table according to a query result corresponding to the at least one first high-frequency query item in the database; and caching the aggregation table into a memory. By adopting the method, the accuracy of generating the aggregation table can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of databases, and in particular to a data caching method and device. BACKGROUND

[0002] With the advent of the big data era, multi-dimensional data analysis plays an increasingly important role in enterprise decision-making, business intelligence and other fields. StarRocks, as a high-performance analytical database, has an advantage in processing multi-dimensional data queries. However, the existing aggregation strategy of StarRocks is determined based on the experience of staff, which cannot generate effective and accurate cached data and consumes a large amount of human resources. SUMMARY

[0003] The present application provides a data caching method and device for improving the accuracy of generating an aggregated table.

[0004] In a first aspect, the embodiments of the present application provide a data caching method, which can be executed by a data caching device. The method comprises: obtaining a plurality of first logs in a first time period from a query log of a database; wherein a log is used to record a query operation of a SQL statement on the database; determining at least one first high-frequency query item through the SQL statement corresponding to the plurality of first logs; wherein a preset field in the same SQL statement corresponds to a query item; the high-frequency query item is determined based on at least the number of occurrences of the query item in the plurality of logs; generating an aggregated table according to the query result corresponding to the at least one first high-frequency query item in the database; and caching the aggregated table in the memory.

[0005] By using the above method, the first high-frequency query item is determined according to the SQL statement corresponding to the plurality of first logs, and then the aggregated table is generated according to the query result corresponding to the first high-frequency query item, so that the accuracy of the generated aggregated table is higher.

[0006] In a possible implementation, after the aggregated table is cached in the memory, the method further comprises: obtaining a plurality of second logs in a second time period from the query log of the database; the second time period is an adjacent time period after the first time period; determining at least one second high-frequency query item through the SQL statement corresponding to the plurality of second logs; the second high-frequency query item is different from the first high-frequency query item; updating the aggregated table according to the query result corresponding to the at least one second high-frequency query item in the database, and caching the updated aggregated table in the memory.

[0007] By using the above method, according to the obtained updated query log, the high-frequency query item can be dynamically identified, and when the high-frequency query item is determined to be updated, the aggregation table is regenerated, so that when a new SQL statement is received, the latest data can be obtained according to the newly generated aggregation table, and the accuracy of the query performance is improved.

[0008] In a possible implementation, after the aggregation table is cached in the memory, the method further includes: monitoring a corresponding original table of the aggregation table in the database; and if data change in the original table meets a set condition, updating the aggregation table cached in the memory.

[0009] By using the above method, according to the corresponding original table of the aggregation table, the data change is monitored in real time, and when the data change meets the set condition, the aggregation table is updated in time, so that the timeliness of the data of the aggregation table is ensured.

[0010] In a possible implementation, the preset field includes one or more of a query dimension, a filter condition, and an aggregation function; and determining at least one first high-frequency query item according to the SQL statements corresponding to the plurality of first logs includes: determining, according to the proportion of the number of times that a query item of a same SQL statement appears in the plurality of first logs and the performance information of the same SQL statement in each query operation, whether the query item is a high-frequency query item.

[0011] By using the above method, when it is determined whether a query item is a high-frequency query item, not only the number of times that the query item appears in the plurality of first logs is considered, but also the performance information of the SQL statement corresponding to the query item in the query operation is considered, so that the accuracy that the obtained high-frequency query item meets the business requirement is improved, and the flexibility of determining the high-frequency query item is improved.

[0012] In a possible implementation, the performance information includes a query duration and a data volume of a query result; and determining, according to the proportion of the number of times that a query item of a same SQL statement appears in the plurality of first logs and the performance information of the same SQL statement in each query operation, whether the query item is a high-frequency query item includes: the probability that the query item is a high-frequency query item is positively correlated with the proportion of the number of times that the query item appears, the query duration, and the data volume of the query result.

[0013] By using the above method, the performance information is further limited to include the query duration and the data volume of the query result, the longer the query duration, the larger the data volume of the query result, and the higher the proportion of the number of times that the query item appears, the greater the probability that the query item is a high-frequency query item.

[0014] In a possible implementation, after the aggregation table is cached in the memory, the method further includes: determining corresponding data blocks of the aggregation table in the memory; each data block includes a plurality of data in the aggregation table; determining access hotness of each data in the aggregation table, thereby obtaining a hotness value of each data block; and when the memory is insufficient, deleting, from the memory, a data block whose hotness value is lower than a threshold.

[0015] By using the method, the data block with a low hotness value is determined to be preferentially evicted, and the data block with a high hotness value is retained, so that the cache state of the data block in the memory is dynamically adjusted, and the dynamic allocation and optimization of the cache resource are implemented, and the storage overhead of the aggregation table is reduced.

[0016] In a possible implementation, the method further includes: for any data in the aggregation table, obtaining an access frequency of the data and a query complexity corresponding to the data; wherein the query complexity is determined according to information of a preset field in a SQL statement corresponding to the data and a data amount of a query result; and determining the access hotness of the data according to the access frequency and the query complexity; wherein the access frequency and the query complexity are positively correlated with the access hotness.

[0017] By using the method, the access hotness of each data is further determined according to the access frequency of the data and the query complexity corresponding to the data, and the query complexity is determined according to the information of the preset field in the SQL statement corresponding to the data and the data amount of the query result.

[0018] In a second aspect, an embodiment of the present application provides a data buffering device, the device includes: an obtaining module, configured to obtain a plurality of first logs in a first time period from a query log of a database; wherein one log is used to record a query operation of one SQL statement on the database; a determining module, configured to determine at least one first high-frequency query item through a SQL statement corresponding to the plurality of first logs; wherein a preset field in a same SQL statement corresponds to one query item; and the high-frequency query item is determined based on at least a number of times that the query item appears in the plurality of logs; a processing module, configured to generate an aggregation table according to a query result corresponding to the at least one first high-frequency query item in the database; and cache the aggregation table in a memory.

[0019] In a possible implementation, the obtaining module is further configured to: obtain a plurality of second logs in a second time period from a query log of the database; the second time period is an adjacent time period after the first time period; the determining module is further configured to: determine at least one second high-frequency query item according to SQL statements corresponding to the plurality of second logs; the second high-frequency query item is different from the first high-frequency query item; and the processing module is further configured to: update the aggregation table according to a query result corresponding to the at least one second high-frequency query item in the database, and cache the updated aggregation table in the memory.

[0020] In a possible implementation, the apparatus further includes a monitoring module configured to monitor a corresponding original table of the aggregation table in the database; and the processing module is further configured to update the aggregation table cached in the memory if data in the original table changes meet a set condition.

[0021] In a possible implementation, the preset field includes one or more of a query dimension, a filter condition, and an aggregation function; and the determining module is specifically configured to determine, according to a proportion of a number of times that a query item of a same SQL statement appears in the plurality of first logs and performance information of the same SQL statement in each query operation, whether the query item is a high-frequency query item.

[0022] In a possible implementation, the performance information includes a query duration and a data amount of a query result; and a probability that the query item is a high-frequency query item is positively correlated with a proportion of a number of times that the query item appears, the query duration, and the data amount of the query result.

[0023] In a possible implementation, the determining module is further configured to: determine a data block corresponding to the aggregation table in the memory; each data block includes a plurality of data in the aggregation table; determine an access heat of each piece of data in the aggregation table, to obtain a heat value of each data block; and the processing module is further configured to: delete, when the memory is insufficient, a data block with a heat value lower than a threshold from the memory.

[0024] In a possible implementation, the determining module is further configured to: for any piece of data in the aggregation table, obtain an access frequency of the data and a query complexity corresponding to the data; the query complexity is determined according to information of a preset field in a SQL statement corresponding to the data and a data amount of a query result; and determine an access heat of the data according to the access frequency and the query complexity; the access frequency and the query complexity are positively correlated with the access heat.

[0025] In a third aspect, the embodiments of the present application further provide a data caching device, comprising a memory and a processor, wherein the memory is configured to store a computer program or instructions; and the processor is configured to invoke the computer program or instructions stored in the memory to execute the method in any possible implementation manner of the first aspect.

[0026] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, wherein the computer readable storage medium stores instructions, and when a computer reads and executes the instructions, the computer executes the method in any possible implementation manner of the first aspect.

[0027] In a fifth aspect, the embodiments of the present application provide a computer program product, wherein the computer program product stores instructions, and when a computer reads and executes the instructions, the computer executes the method in any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0029] Figure 1 A flowchart corresponding to the data caching method provided by the embodiments of the present application is shown in the figure.

[0030] Figure 2 A flowchart corresponding to another data caching method provided by the embodiments of the present application is shown in the figure.

[0031] Figure 3 An internal module diagram of a data caching device 3000 provided by the embodiments of the present application is shown in the figure.

[0032] Figure 4 A structure diagram of a data caching device 4000 provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present application more clear, the following will further describe the present application in combination with the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0034] For high-performance databases, it has an advantage in processing multi-dimensional data queries, and is usually used for querying, analyzing and processing a large amount of data. How to quickly obtain the required data by using the database is crucial. In the traditional scheme, the cache function is usually used to generate an aggregated table by using an aggregation strategy to improve the processing speed of obtaining data. However, the traditional aggregation strategy is determined by the experience of workers, which lacks accuracy and wastes a large amount of human resources.

[0035] Based on this, the embodiment of the application provides a data caching method for improving the accuracy of generating an aggregated table.

[0036] Figure 1 A flowchart of a data caching method provided by the embodiment of the application corresponds to the flowchart, which can be executed by a data caching device, as shown in the flowchart, the flowchart includes the following steps: Figure 1

[0037] Step 101, the data caching device obtains a plurality of first logs in a first time period from the query log of the database.

[0038] The data caching device can be an intelligent device such as a computer or a server, which can be installed with a database. According to the storage path of the query log of the database in the data caching device, the query log of the database is obtained. There are some invalid records in the query log, such as records authorized to users, or logs that are not a complete query statement. After removing these invalid records, a plurality of first logs in a first time period are obtained. A log is used to record the operation of a SQL statement on a database. The same SQL statement can be used by multiple users at the same time, or can generate multiple logs when used by users at different times. The duration of the first time period can be adjusted according to business needs, such as in the peak season of the company, a large amount of data needs to be processed frequently, so the duration of the first time period can be shorter, such as obtaining a plurality of first logs within 24 hours from the current time, or obtaining a plurality of first logs within two hours from the current time. On the contrary, in the off-season of the company, there are not many data-related businesses, and data does not need to be processed frequently, so the duration of the first time period can be set longer.

[0039] Step 102, the data caching device determines at least one first high-frequency query item through the SQL statement corresponding to the plurality of first logs.

[0040] By obtaining a plurality of first logs in a first time period, the SQL statement corresponding to each log is determined, and the high-frequency query item is determined according to the SQL statement corresponding to the plurality of first logs. The predetermined field in the SQL statement corresponds to a query item, and the high-frequency query item is determined based on at least the number of times the query item appears in the plurality of logs.

[0041] ​There are various methods for determining the high-frequency query item through the SQL statements corresponding to the plurality of first logs. For example, the execution of the SQL statements is monitored in real time according to a performance monitoring tool of the database system, and information such as the execution frequency and execution time of the SQL statements is provided to determine the query item in the SQL statement with a higher execution frequency.

[0042] Optionally, the preset field in the SQL statement includes one or more of a query dimension, a filter condition, and an aggregation function. For example, the query dimension can be user_id and product_category, the filter condition can be age>30 and sales>1000, and the aggregation function can be SUM(sales) and COUNT(users). The above query dimension, filter condition, and aggregation function are only examples, and the specific content is not limited in the present application.

[0043] For example, for the SQL statement SELECT department,count(*)AS employee_count,FROM employees WHERE hire_date BETWEEN'2023-01-01'AND'2023-12-31', if the preset field is the query dimension, the filter condition, and the aggregation function, then one query item corresponding to the SQL statement is {department,count(*),hire_date BETWEEN'2023-01-01'AND'2023-12-31'}.

[0044] Further, the number of times that the query item of the same SQL statement appears in the plurality of first logs and the performance information of the same SQL statement in each query operation are determined to determine whether the query item is a high-frequency query item.

[0045] For example, there are a total of 10,000 logs in the first time period, and the 10,000 logs correspond to 500 different SQL statements. Each of the 500 SQL statements corresponds to a query item. If 200 logs in the 10,000 logs correspond to the same SQL statement, then the number of times that the query item of the SQL statement appears in the plurality of first logs is 200 / 10000, i.e., 0.02. Based on the number of times that the query item of the same SQL statement appears in the plurality of first logs, the performance information of the same SQL statement in each query operation is also determined to determine whether the query item is a high-frequency query item.

[0046] Specifically, the performance information includes a query duration and a data volume of a query result, the query duration is a duration required by the SQL statement to query data, and the data volume of the query result is a data row number of the query result of the SQL statement. The probability that the query term is a high-frequency query term is positively correlated with the proportion of the number of occurrences of the query term, the query duration, and the data volume of the query result, that is, the greater the proportion of the number of occurrences of the query term, the longer the query duration, and the greater the data volume of the query result, the higher the probability that the query term is determined as a high-frequency query term. It can be understood that the query duration and the data volume of the query result can be obtained according to the query log. Each log includes the query duration and the data volume of the query result of the SQL statement corresponding to the log.

[0047] According to the proportion of the number of occurrences of the query term, the query duration, and the data volume of the query result, it is determined whether the query term is a high-frequency query term, and a corresponding weight of the three factors can also be set. According to the actual business requirement, the corresponding weight of the three factors can be adjusted. If the proportion of the number of occurrences of the query term is considered more, the corresponding weight of the proportion of the number of occurrences of the query term is greater than the corresponding weight of the query duration and the data volume of the query result. The present application does not make a specific limitation on this.

[0048] In step 103, the data caching device generates an aggregation table according to the query result corresponding to at least one first high-frequency query term in the database.

[0049] For example, if the meaning of a SQL statement is to count the number of times that the first user and the second user buy bread in March, respectively, and the query term corresponding to the SQL statement is a high-frequency query term, which is {D1, D2, times, March}, where D1 and D2 are query dimensions, times is an aggregation function, and March is a filter condition; the meaning of another SQL statement is to count the average spending of the first user and the second user on bread in the first quarter, and the query term corresponding to the SQL statement is a high-frequency query term, which is {(D1, D2), average spending, first quarter}. According to the query result corresponding to the above two high-frequency query terms in the database, an aggregation table is generated, for example, as shown in Table 1.

[0050] Table 1

[0051] D1 D2 Times Average spend D1 NULL M1 NULL NULL D2 M2 NULL D1 D2 NULL M3

[0052] In the formula, NULL represents that the dimension has no value, and M1, M2 and M3 represent corresponding values. In a possible implementation, the aggregation table does not include a filtering condition, and only includes a query dimension and an aggregation function. The high-frequency query item described above is only an example, and in actual applications, the high-frequency query item can be more complex. When a new SQL statement is received, if it is determined that the query item corresponding to the new SQL statement is a certain high-frequency query item, the data queried by the SQL statement can be directly determined from the aggregation table. Even if the aggregation table does not include a filtering condition, the data meeting the SQL statement can be found from the aggregation table, because the aggregation table is generated according to the high-frequency query item corresponding to the SQL statement, and the filtering condition in the high-frequency query item includes the filtering condition in the SQL statement. Before the data queried by the SQL statement is determined from the aggregation table, it is determined whether the filtering condition in the high-frequency query item corresponding to the data in the aggregation table is the same as the filtering condition in the SQL statement. In another possible implementation, the aggregation table includes a filtering condition, so that the data in the aggregation table can be directly matched according to the query item of the newly received SQL statement.

[0053] In step 104, the data caching device caches the aggregation table into the memory.

[0054] For example, after the aggregation table is generated, a plurality of data in the aggregation table can be generated into a data block and cached into the memory. It can be understood that each piece of data has a data id, and the data id can represent the data. The manner of generating a plurality of data into a data block can be that, according to the data id and a preset hash function, a result obtained by performing hash function operation on the data id is determined, and a plurality of pieces of data having the same result after the hash function operation are combined into a data block. There are many ways to generate a plurality of data into a data block, which are not limited in the present application.

[0055] In a possible implementation, the data block with a high hotness value higher than a certain threshold can also be cached in the memory, and the data block with a low hotness value lower than the threshold can be stored in the disk, so as to save the space of the cache. How the hotness value of the data block is calculated will be described in detail below. After the aggregation table is cached in the memory, a plurality of second logs in a second time period are obtained from the query log of the database, the second time period is an adjacent time period after the first time period, and the length of the second time period can be the same as or different from the length of the first time period. At least one second high-frequency query item is determined through the SQL statements corresponding to the plurality of second logs, and at least one second high-frequency item is determined according to the SQL statements corresponding to the plurality of second logs, which is the same as the method of determining at least one first high-frequency item according to the SQL statements corresponding to the plurality of first logs, and will not be described in detail here. Since the query log in the second time period is different from the query log in the first time period, the second high-frequency query item can also be different from the first high-frequency query item. The aggregation table is updated according to the query result corresponding to the at least one second high-frequency query item in the database, and the updated aggregation table is cached in the memory.

[0056] In a possible implementation, after the aggregation table is cached in the memory, the original table corresponding to the aggregation table in the database is also monitored, and if the data change in the original table meets a set condition, the aggregation table cached in the memory is updated. This embodiment ensures that when the high-frequency query item does not change and the query data corresponding to the high-frequency query item changes, the aggregation table can also be updated in time, so that the data in the aggregation table is always in the latest state. The set condition can be that the data change amount meets a set threshold, and if the data change amount in the original table exceeds the set threshold, the aggregation table cached in the memory is updated according to the changed data. The update of the aggregation table includes incremental update and full update, when the data amount in the aggregation table is large and the change is frequent, and the real-time requirement is high, the incremental update can be selected, and when the data amount in the aggregation table is small, the full update is selected. The incremental update is characterized in that only the newly added, modified or deleted data is synchronized, and the performance consumption is low, while the full update is characterized in that all data is synchronized, and the performance consumption is high.

[0057] In a possible implementation, after the aggregation table is cached in the memory, the data block corresponding to the aggregation table in the memory can also be determined, the access hotness of each piece of data in the aggregation table is determined, so as to obtain the hotness value of each data block, and when the memory is insufficient, the data block with a hotness value lower than a threshold is deleted from the memory.

[0058] Specifically, one data block includes a plurality of data, and the hotness value {H1, H2,..., H50} of one data block is determined according to the access hotness of 50 data included in the data block, and 50 is only an example, and the number of data included in one data block is not limited in the present application. Alternatively, the hotness value of the data block can be determined according to the average value of the access hotness of 50 data, or the sum of the access hotness of 50 data.

[0059] The access hotness of each data in the aggregation table is determined, including: for any data in the aggregation table, obtaining the access frequency of the data and the query complexity corresponding to the data, and determining the access hotness of the data according to the access frequency and the query complexity. One data can be determined by multiple SQL statements, and the access frequency of the data is determined according to the number of times the data is determined by the SQL statement in a period of time, and the query complexity is determined according to the information of the preset field in the SQL statement corresponding to the data and the data amount of the query result. Specifically, the query complexity of the data can be determined according to the filter condition, the aggregation function and the data amount of the query result in the plurality of SQL statements corresponding to the data, if more than half of the SQL statements corresponding to the data include both the filter condition and the aggregation function, and the data amount of the query result is greater than 10, then the query complexity of the data is 10 points, if more than one third of the SQL statements corresponding to the data include both the filter condition and the aggregation function, and the data amount of the query result is greater than 10, then the query complexity of the data is 7 points.

[0060] Alternatively, the query time and the data amount of the query result of the plurality of SQL statements corresponding to the data can also be used to determine, a score is given to the query time of each SQL statement corresponding to the data, the closer the query time is to the current time, the higher the score given; a score is given to the data amount of the query result of each SQL statement corresponding to the data, the larger the data amount of the query result, the higher the score. According to the score of the query time and the score of the data amount of the query result of each SQL statement corresponding to the data, the score of each SQL statement is calculated, and thus the complexity score of the data is obtained, wherein the score of the query time and the score of the data amount of the query result can also have respective weights, and the weight of the query time score and the weight of the data amount of the query result can be adjusted according to business requirements.

[0061] After the access frequency and the query complexity of the data are determined, the access hotness of the data is determined according to the access frequency and the query complexity, wherein the access frequency and the query complexity are positively correlated with the access hotness. That is, the higher the access frequency and the higher the query complexity, the greater the access hotness of the data, and specifically, the access frequency and the query complexity can also have respective weights, and the weight of the access frequency and the weight of the query complexity can be adjusted according to actual business requirements.

[0062] Figure 2 A flowchart corresponding to another data caching method provided in an embodiment of the present application. In one possible implementation, after determining the SQL statements corresponding to multiple logs from the query log, it can also be determined whether the SQL statement needs to participate in the determination of high-frequency query items. If it does not participate in the determination of high-frequency query items, an aggregate table can be directly generated. Because some SQL statements are more important, or consume more resources, or the query takes a long time, it is necessary to generate an aggregate table for such SQL statements, regardless of whether they correspond to high-frequency query items.

[0063] Figure 3 A schematic diagram of the internal modules of a data cache device 3000 provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the device may include: an acquisition module 301, a determination module 302, a processing module 303, a monitoring module 404, and optionally, a storage module, the storage module is used to store computer instructions or programs, and the processing module 303 can call the computer instructions or programs in the storage module.

[0064] An acquisition module 301 is used to obtain multiple first logs within a first time period from a query log of a database; wherein a log is used to record a query operation of an SQL statement on the database; a determination module 302 is used to determine at least one first high-frequency query item through the SQL statements corresponding to the multiple first logs; wherein a preset field in the same SQL statement corresponds to a query item; a high-frequency query item is determined based at least on the number of times the query item appears in the multiple logs; a processing module 303 is used to generate an aggregate table based on the query results corresponding to the at least one first high-frequency query item in the database; and cache the aggregate table in memory.

[0065] In one possible implementation, the acquisition module 301 is also used to obtain multiple second logs within a second time period from the query log of the database; the second time period is an adjacent period located after the first time period; the determination module 302 is also used to determine at least one second high-frequency query item through the SQL statements corresponding to the multiple second logs; the second high-frequency query item is different from the first high-frequency query item; the processing module 303 is also used to update the aggregation table according to the query results corresponding to the at least one second high-frequency query item in the database, and cache the updated aggregation table in the memory.

[0066] In a possible implementation, the apparatus further includes a monitoring module 304, configured to monitor a corresponding original table of the aggregation table in the database; and the processing module 303 is further configured to update the aggregation table stored in the memory, if data in the original table changes and meets a set condition.

[0067] In a possible implementation, the preset field includes one or more of a query dimension, a filter condition, and an aggregation function; and the determining module 302 is specifically configured to determine, according to the proportion of the number of times that a query item in a same SQL statement appears in the plurality of first logs and performance information of the same SQL statement in each query operation, whether the query item is a high-frequency query item.

[0068] In a possible implementation, the performance information includes a query duration and a data volume of a query result; and the probability that the query item is a high-frequency query item is positively correlated with the proportion of the number of times that the query item appears, the query duration, and the data volume of the query result.

[0069] In a possible implementation, the determining module 302 is further configured to determine a data block corresponding to the aggregation table in the memory; each data block includes a plurality of pieces of data in the aggregation table; determine an access heat of each piece of data in the aggregation table, to obtain a heat value of each data block; and the processing module 303 is further configured to delete, when the memory is insufficient, a data block with a heat value lower than a threshold from the memory.

[0070] In a possible implementation, the determining module 302 is further configured to, for any piece of data in the aggregation table, acquire an access frequency of the data and a query complexity corresponding to the data; wherein the query complexity is determined according to information of a preset field in a SQL statement corresponding to the data and a data volume of a query result; and the determining module 302 is further configured to determine an access heat of the data according to the access frequency and the query complexity; wherein the access frequency and the query complexity are positively correlated with the access heat.

[0071] Figure 4 A structure diagram of a data caching device 4000 provided by an embodiment of the present application is shown in FIG. 4. As shown in FIG. 4, the data caching device 4000 includes at least one processor 401 and a memory 402 connected with the at least one processor 401. In the embodiment of the present application, the specific connection medium between the processor 401 and the memory 402 is not limited, and the connection between the processor 401 and the memory 402 through a bus is taken as an example. Figure 4 Figure 4 The bus can be divided into an address bus, a data bus, a control bus, and the like.

[0072] ​In the embodiments of the present application, the memory 402 stores instructions executable by the at least one processor 401, and the at least one processor 401 can implement the steps of the data caching method by executing the instructions stored in the memory 402.

[0073] The processor 401 is the control center of the computer device, can connect various parts of the computer device by using various interfaces and lines, and perform resource setting by running or executing instructions stored in the memory 402 and calling data stored in the memory 402. Optionally, the processor 401 can include one or more processing units, and the processor 401 can integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 401. In some embodiments, the processor 401 and the memory 402 can be implemented on the same chip, and in some embodiments, they can also be implemented on independent chips respectively.

[0074] The processor 401 can be a general-purpose processor, for example, a central processing unit (CPU), a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as hardware processor execution or executed by a combination of hardware and software modules in the processor.

[0075] The memory 402, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 402 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. The memory 402 is any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory 402 in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used for storing program instructions and / or data.

[0076] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as a combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing apparatus to produce a machine, so that the instructions executed by the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0078] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 of the flow or flows and / or blocks Figure 1 of the block or blocks specified in the flow.

[0079] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that are executed on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 of the flow or flows and / or blocks Figure 1 of the block or blocks specified in the flow.

[0080] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the apparent to those skilled in the art that various modifications and changes can be made thereto without departing from the scope and spirit of the application. It is therefore desired that the present application be interpreted as not limited to the specific examples described but that it includes all modifications and changes falling within the scope of the claims and their equivalents.

Claims

1. A data caching method, characterized in that: The method comprises: Acquire multiple first logs within a first time period from a query log of a database; wherein one log is used to record a query operation of an SQL statement on the database; Determining at least one first high-frequency query item through the SQL statements corresponding to the plurality of first logs; wherein a preset field in the same SQL statement corresponds to a query item; and the high-frequency query item is determined based at least on the number of times the query item appears in the plurality of logs; Generate an aggregate table according to the query result corresponding to the at least one first high-frequency query item in the database; The aggregate table is cached in memory.

2. The method according to claim 1, characterized in that After caching the aggregate table into memory, the following steps are also included: Acquire multiple second logs within a second time period from the query log of the database; the second time period is an adjacent period after the first time period; Determining at least one second high-frequency query item through the SQL statements corresponding to the plurality of second logs; wherein the second high-frequency query item is different from the first high-frequency query item; According to the query result corresponding to the at least one second high-frequency query item in the database, the aggregation table is updated, and the updated aggregation table is cached in the memory.

3. The method according to claim 1, characterized in that After caching the aggregate table into memory, the following steps are also included: Monitor the original table corresponding to the aggregate table in the database; If the data changes in the original table meet the set conditions, the aggregate table cached in the memory is updated.

4. The method according to any one of claims 1 to 3, characterized in that The preset fields include one or more of query dimensions, filter conditions, and aggregation functions; Determining at least one first high-frequency query item through the SQL statements corresponding to the plurality of first logs includes: Based on the percentage of occurrences of a query item of the same SQL statement in the plurality of first logs and performance information of the same SQL statement in each query operation, it is determined whether the query item is a high-frequency query item.

5. The method according to claim 4, characterized in that The performance information includes query duration and query result data volume; Determining whether a query item of the same SQL statement is a high-frequency query item based on a percentage of occurrences of the query item in the plurality of first logs and performance information of the same SQL statement in each query operation includes: The probability that the query term is a high-frequency query term is positively correlated with the proportion of the number of times the query term appears, the query duration, and the data volume of the query result.

6. The method according to any one of claims 1 to 3, characterized in that After caching the aggregate table into memory, the following steps are also included: Determine a data block corresponding to the aggregate table in the memory; each data block includes multiple data in the aggregate table; Determine the access popularity of each data in the aggregate table, thereby obtaining the popularity value of each data block; When the memory is insufficient, data blocks with heat values ​​lower than a threshold are deleted from the memory.

7. The method according to claim 6, characterized in that Determining the access popularity of each piece of data in the aggregate table includes: For any piece of data in the aggregate table, obtain the access frequency of the data and the query complexity corresponding to the data; wherein the query complexity is determined based on the information of the preset fields in the SQL statement corresponding to the data and the data volume of the query result; The access popularity of the data is determined according to the access frequency and the query complexity; wherein both the access frequency and the query complexity are positively correlated with the access popularity.

8. A data cache device, characterized in that: The device comprises: An acquisition module, configured to acquire a plurality of first logs within a first time period from a query log of a database; wherein one log is used to record a query operation of an SQL statement on the database; a determination module, configured to determine at least one first high-frequency query item based on the SQL statements corresponding to the plurality of first logs; wherein a preset field in the same SQL statement corresponds to a query item; and the high-frequency query item is determined based at least on the number of times the query item appears in the plurality of logs; A processing module is configured to generate an aggregate table according to a query result corresponding to the at least one first high-frequency query item in the database; and cache the aggregate table in a memory.

9. A data cache device, characterized in that: include: Memory, used to store computer programs or instructions; A processor, configured to call the computer program or instruction stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when a computer reads and executes the instructions, the computer is caused to execute the method according to any one of claims 1 to 7.

Images