Data storage method, device, equipment and medium

Abstract

Embodiments of the present invention relate to a data storage method, apparatus, device and medium. The data storage method includes: obtaining query information of a data set in a previous calculation cycle from a query log; calculating a query priority value of the data set based on the query information of the data set; and storing the data sets in sequence in different levels of cache in descending order according to the query priority values ​​of the data set.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a data storage method, device, equipment and medium. Background Art

[0002] With the development of Internet technology, when accessing a large amount of data, it is necessary to pre-store the frequently read data in a cache structure (Cache) to avoid wasting a lot of time reading directly from the hard disk. Taking the field of Business Intelligence (BI) as an example, when accessing certain data or data sets for analysis, the frequently read data or data sets will be cached, but the size of the cache is limited, so the cache needs to be managed.

[0003] In the process of realizing the concept of the present invention, the inventors found that there are at least the following technical problems in the related technology: the existing cache management solutions, such as the more common first-in-first-out and least recently used solutions, have poor effects and low efficiency when applied to real scenarios. It is difficult to pre-store the data that needs to be cached most in the cache according to actual needs, and there is a problem of not being able to perform adaptive optimization for specific application scenarios. Summary of the invention

[0004] In a first aspect, an embodiment of the present invention provides a data storage method, comprising: obtaining query information of a data set that is a query operation object in a previous calculation cycle from a query log, wherein the calculation cycle is the time interval between two adjacent calculations of the query priority value of the data set; calculating the query priority value of the data set based on the query information of the data set; and storing the data sets in sequence in different levels of cache according to the query priority values ​​of the data set from high to low.

[0005] According to an embodiment of the present invention, the query information includes query instructions for executing query operations, data sets queried by each query instruction, and query time consumption of the data sets.

[0006] According to an embodiment of the present invention, the above-mentioned calculation of the query priority value of the above-mentioned data set based on the query information of the above-mentioned data set includes: for each data set, calculating the weighted sum of the query time corresponding to each query operation performed on the data set in the previous calculation cycle.

[0007] According to an embodiment of the present invention, for each data set, calculating the weighted sum of query times corresponding to each query operation performed on the data set in the previous calculation cycle includes: for each data set, obtaining a preset weight p corresponding to the data set, according to Calculate the query priority value S of the data set D , t kIndicates the time consumed for querying the data set for the kth time in the current computing cycle, where k ≥ 0 and k is an integer.

[0008] According to an embodiment of the present invention, the above-mentioned calculation of the query priority value of the above-mentioned data set based on the query information of the above-mentioned data set includes: obtaining the capacity information of each data set; for each data set, calculating the ratio of the sum of the query times corresponding to each query operation performed on the data set in the previous calculation cycle to the capacity of the data set.

[0009] According to an embodiment of the present invention, for each data set, calculating the ratio of the sum of query times corresponding to each query operation performed on the data set in the previous calculation cycle to the capacity of the data set includes: for each data set, obtaining the data set D m The corresponding preset weight a m and preset bias b m ,according to Calculate the query priority value S of the data set m , V m represents the capacity of the data set, t k Indicates the time consumed for querying the data set for the kth time in the current computing cycle, where k ≥ 0 and k is an integer.

[0010] According to an embodiment of the present invention, the preset weight a m and preset bias b m Set the storage priority based on the business corresponding to the data set.

[0011] According to an embodiment of the present invention, the preset weight a m and preset bias b m Set according to different data sets.

[0012] According to an embodiment of the present invention, after storing the data sets in sequence in different levels of cache according to the query priority values ​​of the data sets from high to low, the method further includes: receiving a user's query instruction, the query instruction including query operations within the current calculation cycle and data set information to be queried corresponding to each query operation; based on the data set information to be queried, querying is performed preferentially in the caches of the different levels.

[0013] According to an embodiment of the present invention, the method further includes: obtaining a query log in the current calculation cycle to provide query information for the next calculation of the query priority value of the data set.

[0014] In a second aspect, an embodiment of the present invention provides a device for data storage, comprising: an acquisition module for acquiring query information of a data set in a previous calculation cycle from a query log; a calculation module for calculating a query priority value of the data set based on the query information of the data set; and a storage module for storing the data sets in sequence into caches of different levels according to the query priority values ​​of the data sets from high to low.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor is used to implement the method described above when executing the program stored in the memory.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described above is implemented.

[0017] The above technical solution provided by the embodiment of the present invention has at least some or all of the following advantages compared with the related art:

[0018] The embodiment of the present invention adopts a multi-level cache structure, and obtains the query information of the data set of the query operation object in the previous calculation cycle from the query log, thereby calculating the query priority value of the data set, and storing it in the caches of each level in descending order according to the query priority value, so as to realize dynamic adjustment of the cache based on the analysis and calculation of the historical query situation to optimize the future query effect, thereby adaptively adjusting the data sets loaded in the caches of each level, so that under the constraint of limited cache space, the cache is efficiently managed, and the data with query priority values ​​from high to low are stored in the caches of each level with transmission speeds from high to low in sequence, so as to meet the needs of engineering practice applications, and reasonably store different data sets in caches of different levels, effectively reduce the query time, and significantly improve the query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1Schematically illustrates the system architecture of the data storage method and device applicable to the embodiments of the present invention;

[0022] Figure 2 A flowchart schematically shows a method for storing data according to an embodiment of the present invention;

[0023] Figure 3 A flowchart of a data storage method according to another embodiment of the present invention is schematically shown;

[0024] Figure 4 A flowchart schematically shows a method for data storage according to another embodiment of the present invention;

[0025] Figure 5 A schematic diagram shows a structural block diagram of a device for data storage according to an embodiment of the present invention;

[0026] Figure 6 The structural block diagram of an electronic device provided by an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0027] With the development of Internet technology, the concept of cache has been expanded. Any structure between two hardware with large speed differences (such as memory and hard disk, hard disk and network, etc.) used to coordinate the difference in data transmission speed between the two can be called cache, not just the cache between CPU and main memory in the traditional sense. Cache is widely present in Internet technology architecture. For example, in the field of Business Intelligence (BI), when accessing certain data or data sets for analysis, frequently read data or data sets will be cached. However, due to the limited cache space constraints, the cache needs to be managed.

[0028] In the related art, commonly used cache management schemes, such as LRU (Least Recently Used) and FIFO (First In First Out), are used to pre-store frequently read data in the cache to avoid directly reading from the hard disk and consuming a lot of time. However, these common schemes have poor effects and low efficiency when applied to real scenarios. If better results are to be achieved for specific scenarios, it is often necessary to optimize by manually setting key parameters (for example, the k value of LRU). It is difficult to pre-store the data that needs to be cached most in the cache according to actual needs, and there is a problem that adaptive optimization cannot be performed for specific application scenarios. In summary, under the constraints of limited cache space, how to pre-store the data that needs to be cached most in the cache, and to achieve adaptive optimization for specific application scenarios, so as to more efficiently utilize the limited cache space, has important practical value.

[0029] In view of this, an embodiment of the present invention provides a method for data storage, which includes: obtaining query information of a data set that is a query operation object in a previous computing cycle from a query log; calculating a query priority value of the data set based on the query information of the data set; and storing the data sets in sequence in different levels of cache according to the query priority values ​​of the data set from high to low.

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] Figure 1 The system architecture of the method and device for data storage applicable to the embodiments of the present invention is schematically shown.

[0032] Reference Figure 1 As shown, the system architecture 100 of the method and apparatus for data storage according to the embodiment of the present invention includes: terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0033] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The sent message can be query information, and the received message can be a query result. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0034] The terminal devices 101, 102, and 103 may be various electronic devices with display screens and supporting web browsing, such as electronic devices including but not limited to smart phones, tablet computers, laptop computers, desktop computers, smart watches, and the like.

[0035] The server 105 may be a server that provides various services, such as a background management server (only as an example) that processes data and provides service support for web pages accessed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received access request and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device.

[0036] It should be noted that the data storage method provided in the embodiment of the present invention can generally be performed by the server 105 or a terminal device with a certain computing capability. Accordingly, the data storage device provided in the embodiment of the present invention can generally be set in the server 105 or in the above-mentioned terminal device with a certain computing capability. The data storage method provided in the embodiment of the present invention can also be performed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the data storage device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0037] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0038] In an embodiment of the present invention, for example, in application scenarios such as Business Intelligence (BI), data is often frequently queried based on business needs for business analysis. Among them, the data warehouse provides data support for many application systems, such as BI systems, data mining systems, data analysis systems, etc. In the data warehouse, the data therein are usually calculated in a certain working cycle (operating cycle). This process can be called a "scheduled calculation job". The results obtained by the scheduled calculation job of each working cycle can be data sets of different scales and sizes (also called result set data), including many data sets. These data sets can be distributed in various data marts for access and query by front-end applications for business analysis.

[0039] In the embodiment of the present invention, the data in the data mart is obtained by selectively processing the original data in the data warehouse, that is, not all the original data in the data warehouse is processed, but the required data is selected for processing based on actual application needs to form multiple data marts, each of which can provide one or more application systems with data required for business, and each application system business can also obtain the data required for business from one or more applications / systems. The data for business analysis required to be queried based on business needs comes from the data mart, which can be multiple data sets in the data mart. It can be understood as obtaining one or more data sets related to specific business needs from one or more data marts, and querying and analyzing them.

[0040] In order to make frequently queried data or data sets accessed more quickly, they can be loaded into the cache. The embodiment of the present invention adopts a multi-level cache structure. By obtaining the query information of the data set of the query operation object in the previous calculation cycle from the query log, the query priority value of the data set is calculated, and the query priority value is stored in the cache at each level in descending order. The analysis and calculation based on the historical query situation is realized to dynamically adjust the cache to optimize the future query effect, so as to adaptively adjust the data sets loaded in the cache at each level, so that the cache can be efficiently managed under the constraint of limited cache space, and the data with query priority values ​​from high to low are stored in the cache at each level with transmission speed from high to low in turn, so as to meet the needs of engineering practice applications and reasonably store different data sets in caches of different levels, effectively reduce query time, and significantly improve query efficiency.

[0041] It should be noted that the multi-level cache architecture of the application scenario of the present invention is based on a broad cache, that is, any structure located between two hardware / software with a large speed difference and used to coordinate the difference in data transmission speed between the two can be called a cache. Therefore, caches can exist between hardware / software at different levels. For example, the cache between the hard disk and the network can be an Internet temporary folder or a network content cache, and another example is the cache between the CPU and the main memory, and the cache between the memory and the hard disk.

[0042] Although the multi-level cache architecture in the implementation scenario of the present invention takes memory as an example as the first storage space with the fastest cache speed and the smallest capacity, it is not limited to this. In the multi-level cache architecture, the storage space with the fastest cache speed and the smallest capacity can be used as the first storage space in a relative sense, that is, the first-level cache. Similarly, as the cache speed decreases step by step and the capacity increases step by step, the storage space that is slower and larger in capacity than the first-level cache can be called the second-level cache (second storage space), the third-level cache (third storage space), and so on.

[0043] A first exemplary embodiment of the present invention provides a data storage method.

[0044] Figure 2 The flowchart of the data storage method according to an embodiment of the present invention is schematically shown. Figure 3 The following schematically shows a flow chart of a method for storing data according to another embodiment of the present invention. Figure 4 The following schematically shows a flow chart of a method for storing data according to yet another embodiment of the present invention.

[0045] Reference Figure 2 As shown, the data storage method provided by the embodiment of the present invention includes the following operations: S202, S204 and S206. The above operations S202 to S206 can be performed by a server or a terminal device with certain computing capabilities.

[0046] In operation S202, query information of a data set in a previous calculation cycle is obtained from a query log.

[0047] Among them, the calculation cycle can be understood as a statistical cycle, that is, the time interval between the execution of the operation of the query priority value of the last calculation data set and the execution of the current calculation operation. The query information of all query operations executed within this cycle can be used for analysis and statistics. Optionally, the time interval between any two adjacent calculations of the query priority value of the data set can be the same, and can be set according to the needs of the actual application scenario, for example, the calculation cycle is set to one day, two days, one week, etc. Exemplarily, the query operation can be implemented by a large-scale computing framework such as presto.

[0048] Exemplarily, if the calculation cycle is set to one day, the operation of calculating the query priority value of the data set is performed once a day. Taking the commonly used report system in the BI field as an example, the report system can obtain the query information of the data set corresponding to the query operation executed yesterday from the query log at a specified time every day. The above query information may include the query instructions for executing the query operation, the data set queried by each query instruction, and the query time of the data set. Specifically, the query instruction may be, for example, an SQL statement for executing the corresponding query operation. The last calculation cycle may include multiple SQL statements and execute multiple query operations. The log records the execution time of each SQL statement and the specific content of the statement, for example, which data sets are queried and the query time of each data set, and other related information. Based on this, the log can be analyzed to find out which data sets are queried by all query operations executed yesterday, how much time is consumed to query each data set, and other information.

[0049] In operation S204, a query priority value of the data set is calculated according to the query information of the data set.

[0050] Optional, see Figure 3 Operation S204 may further include operation S2041a, that is, for each data set, a weighted sum of query times corresponding to each query operation performed on the data set in the previous calculation cycle may be calculated.

[0051] In operation S2041a, for each data set, the weighted sum of the query times corresponding to the query operations performed on the data set in the previous calculation cycle is calculated. If the calculation cycle is set to one day, taking the commonly used reporting system in the BI field as an example, based on the query information of the query operations performed yesterday through the above log analysis, it is possible to obtain which data sets were queried and the query time (query delay) of each data set, thereby calculating the query priority value of each data set (for example, it can be specifically implemented as scoring the data sets, and the obtained scores are used to characterize the priority value of the data sets).

[0052] Specifically, for each data set, obtain the preset weight p corresponding to the data set, and combine the query time of the data set in the previous calculation cycle obtained above, according to Calculate the query priority value S of the data set D , t k Indicates the time taken to query the data set for the kth time in the current computing cycle, k ≥ 0 and k is an integer. The query priority value S of the data set D , can be used to represent the total time consumed by each query on the data set in the previous computing cycle, that is, the weighted sum of the query time corresponding to each query operation performed on the data set in the previous computing cycle.

[0053] The preset weight p is set according to the storage priority of the business corresponding to the data set. For example, when calculating the query priority value for data sets corresponding to different businesses, the query priority value of the data set can be adjusted and optimized by changing the preset weight p corresponding to the data set. For example, for two different data sets D' and D", which correspond to two businesses with different priorities, the business priority corresponding to D' is higher than the business priority corresponding to D", and D' needs to be stored in the first-level cache first. Then, according to S D =p' Calculate S D’ and according to S D =p" Calculate S D” If S D’ and S D” The size does not meet the business needs and cannot achieve priority cache D'. You can adjust the preset weights p' and p", so that S D’ Higher than S D” This enables priority caching of D', further meeting actual business needs.

[0054] For example, when the preset weight p is 1, for each data set, Calculate the query priority value S of the data set D , can be used to represent the total time consumed by each query on the data set in the previous computing cycle, that is, the weighted sum of the query time corresponding to each query operation performed on the data set in the previous computing cycle.

[0055] Optional, see Figure 4 Operation S204 may further include operations S2041b and S2042b, that is, the capacity information of each data set may be obtained; for each data set, the ratio of the sum of the query times corresponding to each query operation performed on the data set in the previous calculation cycle to the capacity of the data set is calculated.

[0056] In operation S2041b, the capacity information of each data set is obtained. If the calculation period is set to one day, taking the commonly used reporting system in the BI field as an example, based on the query information of the query operation executed yesterday through the log analysis, it is possible to obtain which data sets are queried and the query time of each data set. This operation also includes obtaining the capacity information of each data set. Specifically, the capacity information of the data set can be obtained from the data warehouse, which contains the basic information of the data set, such as the data set capacity and other metadata that can describe the data source metadata.

[0057] In operation S2042b, for each data set, a ratio of the sum of query times corresponding to query operations performed on the data set in the previous calculation cycle to the capacity of the data set is calculated.

[0058] Specifically, for each data set, obtain the data set D m The corresponding preset weight a m and preset bias b m , and combined with the above-obtained data set capacity information and the query time of the data set in the previous calculation cycle, it can be based on Calculate the data set D m The query priority value S m , where t k Indicates the time taken to query the data set for the kth time in the current computing cycle, k ≥ 0 and k is an integer, V m Indicates the capacity of the data set, query priority value S m It can be used to represent the ratio of the total time consumed by each query on the data set in the previous computing cycle to the capacity of the data set, that is, the ratio of the sum of the query times corresponding to each query operation performed on the data set in the previous computing cycle to the capacity of the data set.

[0059] The query priority value S calculated above mIt can also be used to characterize the query priority value generated by each unit of data volume in a data set, and can measure the cost-effectiveness of caching the data set. m and preset bias b m , can be set differently according to different business types in specific application scenarios. For example, when the preset weight a m and preset bias b m When the values ​​of are all 0, calculate the query priority value S of the data set D Then according to Perform calculations.

[0060] For data sets corresponding to different businesses, when calculating the query priority value of the data set, the preset weight a can be changed m and preset bias b m To adjust and optimize the query priority value of the data set, a m a may represent a preset weight related to the business, and b may represent a preset bias related to the business, both of which may be set according to the storage priority of the business corresponding to the data set.

[0061] For example, for two different data sets D1 and D2, they correspond to two services with different priorities. The service priority corresponding to D1 is higher than the service priority corresponding to D2. D1 needs to be stored in the first-level cache first. Then, according to When calculating the query priority values ​​S1 and S2 of the two data sets respectively, if the values ​​of S1 and S2 do not meet the business needs and the priority cache D1 cannot be implemented, you can adjust the preset weight a m and / or preset bias b m , so that according to When the query priority values ​​of the data set are calculated to obtain S1 and S2, the query priority value of the data set D1 can be made higher than that of D2, thereby achieving priority caching of D1 and further meeting actual business needs.

[0062] For data sets corresponding to different businesses, when calculating the query priority value of the data set, the preset weight a can be changed m and preset bias b m To adjust and optimize the query priority value of the data set, a m a may represent a preset weight related to the business, and b may represent a preset bias related to the business, both of which may be set according to the storage priority of the business corresponding to the data set.

[0063] For example, for two different data sets D1 and D2, they correspond to two services with different priorities. The service priority corresponding to D1 is higher than the service priority corresponding to D2. D1 needs to be stored in the first-level cache first. Then, according to When calculating the query priority values ​​S1 and S2 of the two data sets respectively, if the values ​​of S1 and S2 do not meet the business needs and the priority cache D1 cannot be implemented, you can adjust the preset weight a m and / or preset bias b m , so that according to When the query priority values ​​of the data set are calculated to obtain S1 and S2, the query priority value of the data set D1 can be made higher than that of D2, thereby achieving priority caching of D1 and further meeting actual business needs.

[0064] Optional, preset weight a m and preset bias b m It can be set according to different data sets. For multiple data sets corresponding to a specific business, the query priority values ​​of different data sets can be adjusted and optimized by changing the preset weights and preset biases. For example, for two different data sets D1 and D2 involved in the same specific business, data set D1 has a higher priority than D2, and D1 needs to be stored in the first-level cache first. Then, according to When calculating the query priority values ​​S1 and S2 of the two data sets respectively, the preset weight a can be adjusted m and preset bias b m , the query priority value of data set D1 is increased, so that the query priority value S1 of D1 is higher than the query priority value S2 of D2, thereby achieving priority caching of D1 and further meeting actual business needs. m and preset bias b m , and further adjust the query priority value of the data set involved in the specific business according to actual needs, so as to adjust the query priority of different data sets.

[0065] Optionally, the above-mentioned calculation operations include but are not limited to the above-mentioned formulas, and can also be implemented by, for example, machine learning models, and can be selected and designed according to the business needs in the specific application scenario. For example, when the business is more concerned with the weighted sum of the query time of the data set, the method such as operation S2041a can be used to calculate the query priority value; when the business is more concerned with the sum of the query time per unit data volume of the data set, the method such as operations S2041b and S2042b can be used to calculate the query priority value.

[0066] Regardless of the calculation scheme used to implement the above operation S204, the data storage method provided by the present invention can be applied. In addition, when the above scheme is implemented in a specific application scenario, after a certain number of calculation cycles, the query priority values ​​of the many data sets that are the query objects may show a certain regularity, which can reflect the situation in which the relevant data sets of the corresponding business are queried, thereby further reflecting the access situation of the corresponding business (for example, the data access mode in some BI analysis scenarios). Based on this, the above implementation scheme can further adjust and optimize the calculation scheme for the query priority value of the data set, thereby affecting the order of storing the data set in multiple different levels of cache, continuously optimizing, and providing the best cache strategy, so as to realize that the data that needs to be cached most is pre-stored in the cache according to actual needs, so that adaptive optimization can be performed for specific application scenarios, so as to achieve the effect of continuously improving query efficiency, and further embody the effect-oriented and data-driven advantages of the method according to the embodiment of the present invention.

[0067] In operation S206, the data sets are stored in different levels of cache in descending order according to the query priority values ​​of the data sets. Specifically, based on the query priority values ​​of the data sets calculated above, the data sets are loaded into a multi-level cache structure such as a first-level cache space and a second-level cache space in descending order. When the first-level cache is full, the data sets are loaded into the second-level cache. Similarly, when the second-level cache is full, the data sets are loaded into the third-level cache, and so on.

[0068] Optionally, after storing the data sets in sequence in different levels of cache according to the query priority values ​​of the above data sets from high to low, the method provided by the embodiment of the present invention may further include receiving a user's query instruction, wherein the query instruction includes query operations within the current calculation cycle and the data set information to be queried corresponding to each query operation; based on the data set information to be queried corresponding to each query operation, querying is performed preferentially in the above different levels of cache. For example, after receiving the user's query instruction, according to the query instruction containing the query operation to be performed and the data set information to be queried corresponding to the query operation, data set Q is the data set to be queried. Since in the operation of the data storage method according to the embodiment of the present invention that has been executed, the query priority value S of data set Q has been calculated. Q , and the data set Q has been stored in a certain level of cache, so when querying the data set Q, it is preferentially queried in caches of different levels, such as the first-level cache, the second-level cache, and so on.

[0069] Specifically, when a user initiates a query through an application such as a reporting system, a server or device running the method provided by an embodiment of the present invention receives the user's query instruction. The query instruction may be, for example, an SQL statement. Each statement may perform a query operation on multiple different data sets. The query instruction may include relevant information such as which data sets are to be queried, basic information of each data set (e.g., attribute information), etc. The query is preferentially performed level by level in the multi-level cache. If the data set to be queried has been stored in the cache in the above operation S206, the data set can be queried at a high speed. If it is not in the cache, the query is continued in other storage devices (e.g., local storage devices), etc. The query information of the query operation will generate corresponding records in the query log.

[0070] Exemplarily, after applying the method provided by the embodiment of the present invention in an actual application scenario, without any maintenance or intervention, according to the records in the query log, after adopting the above storage method, the average query priority value of the data set has obviously dropped by 60-70%, and has remained stable. It is further verified that the above method provided by the embodiment of the present invention can adaptively and reasonably store different data sets in caches of different levels, realize the analysis and calculation based on historical query situations to dynamically adjust the cache to optimize future query effects, effectively reduce query time, and significantly improve query efficiency. The above optional multiple analysis and calculation methods also embody the idea of ​​data-driven, optimize query performance with effect as the guide, and can automatically and continuously achieve the above optimization effects.

[0071] Optionally, after the above-mentioned query operation, the query log within the current calculation cycle can also be obtained to provide query information for the query priority value of the next calculation of the data set. The query log within the current calculation cycle is the query information of the data set that needs to be obtained in operation S202 before the next calculation of the query priority value, and the operation steps S202 to S206 of any of the above-mentioned embodiments are repeated.

[0072] Based on the above operations, the embodiment of the present invention adopts a multi-level cache structure, by obtaining the query information of the data set of the query operation object in the previous calculation cycle from the query log, thereby calculating the query priority value of the data set, and storing it in the cache at each level in descending order according to the query priority value, so as to realize dynamic adjustment of the cache based on the analysis and calculation of the historical query situation to optimize the future query effect, thereby adaptively adjusting the data set loaded in the cache at each level, so that under the constraint of limited cache space, the cache is efficiently managed, and the data with query priority values ​​from high to low are stored in the cache at each level with transmission speed from high to low in sequence, so as to meet the needs of engineering practice applications, and reasonably store different data sets in caches of different levels, effectively reduce query time, and significantly improve query efficiency.

[0073] A third exemplary embodiment of the present invention provides an apparatus for data storage.

[0074] Figure 5 The structure block diagram of a device for data storage according to another embodiment of the present invention is schematically shown.

[0075] Reference Figure 5 As shown, the device 500 for data storage provided by the embodiment of the present invention includes an acquisition module 501, a calculation module 502 and a storage module 503.

[0076] The acquisition module 501 is used to acquire query information of a data set that is a query operation object in a previous calculation cycle from a query log. The calculation cycle is a time interval between two adjacent calculations of query priority values ​​of a data set.

[0077] The calculation module 502 is used to calculate the query priority value of the data set according to the query information of the data set.

[0078] The storage module 503 is used to store the data sets in sequence into multiple levels of cache according to the query priority values ​​of the data sets from high to low.

[0079] In addition to the acquisition module 501, the calculation module 502 and the storage module 503, the data storage device 500 may further include: a query module.

[0080] The query module is used to receive a user's query instruction, which includes query operations in the current calculation cycle and data set information corresponding to each query operation; based on the data set information corresponding to each query operation, the query is preferentially performed in the multi-level cache.

[0081] The query module may further include a log acquisition module, which is used to acquire query logs in the current calculation cycle to provide query information for the next calculation of the query priority value of the data set.

[0082] Any of the acquisition module 501, the calculation module 502 and the storage module 503 can be combined into one module for implementation, or any of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. At least one of the acquisition module 501, the calculation module 502 and the storage module 503 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 501, the calculation module 502 and the storage module 503 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0083] A fourth exemplary embodiment of the present invention provides an electronic device.

[0084] Figure 6 The structural block diagram of an electronic device provided by an embodiment of the present invention is schematically shown.

[0085] Reference Figure 6 As shown, the electronic device 600 provided in the embodiment of the present invention includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602 and the memory 603 communicate with each other through the communication bus 604; the memory 603 is used to store computer programs; the processor 601 is used to implement the data storage method as described above when executing the program stored in the memory.

[0086] A fifth exemplary embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the data storage method described above is implemented.

[0087] The computer-readable storage medium may be included in the device / apparatus described in the above embodiment; or it may exist independently without being assembled into the device / apparatus. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method for data storage according to the embodiment of the present invention is implemented.

[0088] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0089] In this article, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0090] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A data storage method, characterized in that: include: Obtain query information of the data set in the previous computing cycle from the query log; Calculating a query priority value of the data set according to the query information of the data set, and adjusting and optimizing the query priority value of the data set by changing a preset weight and a preset bias for data sets corresponding to different services; storing the data sets in sequence into caches of different levels according to the query priority values ​​of the data sets from high to low; The query information includes query instructions for executing query operations, data sets queried by each query instruction, and query time consumption of the data sets; The calculating the query priority value of the data set according to the query information of the data set includes: Get the capacity information of each data set; For each data set, calculate the ratio of the sum of the query times corresponding to each query operation performed on the data set in the previous calculation cycle to the capacity of the data set, including: For each data set, get the data set D m The corresponding preset weight a m and preset bias b m ,according to Calculate the query priority value S of the data set m , V m Represents the capacity of the data set, t k Indicates the time consumed for querying the data set for the kth time in the current computing cycle, where k ≥ 0 and k is an integer.

2. The method according to claim 1, characterized in that The calculating the query priority value of the data set according to the query information of the data set includes: For each data set, the weighted sum of the query time consumption corresponding to each query operation executed on the data set in the previous calculation cycle is calculated.

3. The method according to claim 2, characterized in that For each data set, calculate the weighted sum of the query time corresponding to each query operation performed on the data set in the previous calculation cycle, including: For each data set, obtain the preset weight p corresponding to the data set, according to Calculate the query priority value S of the data set D , t k Indicates the time consumed for querying the data set for the kth time in the current computing cycle, where k ≥ 0 and k is an integer.

4. The method according to claim 1, characterized in that: The preset weight a m and preset bias b m Set it according to the storage priority of the business corresponding to the data set.

5. The method according to claim 4, characterized in that The preset weight a m and preset bias b m Set according to different data sets.

6. The method according to any one of claims 1 to 5, characterized in that: After storing the data sets in sequence in different levels of cache in descending order of the query priority values ​​of the data sets, the method further includes: Receive a query instruction from a user, where the query instruction includes query operations in a current computing cycle and data set information to be queried corresponding to each query operation; Based on the data set information to be queried, the query is preferentially performed in the caches of the different levels.

7. The method according to claim 6, characterized in that The method further comprises: Get the query log in the current calculation cycle to provide query information for the next calculation of the query priority value of the data set.

8. A device for data storage, characterized in that: include: An acquisition module, used to acquire query information of a data set in a previous computing cycle from a query log, wherein the query information includes a query instruction for executing a query operation, a data set queried by each query instruction, and a query time consumption of the data set; A calculation module, used to calculate the query priority value of the data set according to the query information of the data set, and adjust and optimize the query priority value of the data set by changing the preset weight and the preset bias for the data sets corresponding to different services; The calculating the query priority value of the data set according to the query information of the data set includes: Get the capacity information of each data set; For each data set, calculate the ratio of the sum of the query times corresponding to each query operation performed on the data set in the previous calculation cycle to the capacity of the data set, including: For each data set, get the data set D m The corresponding preset weight a m and preset bias b m ,according to Calculate the query priority value S of the data set m , V m Represents the capacity of the data set, t k Indicates the time taken to query the data set for the kth time in the current computing cycle, k ≥ 0 and k is an integer; The storage module is used to store the data sets in sequence into caches of different levels according to the query priority values ​​of the data sets from high to low.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; A processor, configured to implement the method according to any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images