Data query method and related device
By updating the storage path mapping relationship in the whitelist in the big data system and using the heat record table to judge the data value, the problem of low data query efficiency in the existing technology is solved, and efficient data query and cache system optimization are achieved. .
Patent Information
- Application Number
- CN202310898205.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-20
- Publication Date
- 2025-01-21
AI Technical Summary
Existing data query methods are not very efficient in big data systems, especially for high-frequency queries. Existing methods require frequent loading of data from the underlying storage system to the cache system, and the whitelist depends on manual configuration and cannot be automatically updated. .
When loading the data to be queried in the data cache system, the storage path mapping relationship in the whitelist is updated to ensure that the storage conditions of the whitelist and the data cache system are synchronized. At the same time, the heat record table is used to make value judgments on the data to be queried, and only high-hot data is loaded into the cache system, and the whitelist and heat record table are periodically updated to adapt to changes in the underlying storage path and changes in hot and cold data.
It improves the efficiency of data query in the big data system, reduces the need to modify the file system protocol of the underlying storage system, supports the coexistence of multiple query engines, and improves the storage space utilization of the data cache system.
Smart Images
Figure CN119336788A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of database technology, and in particular to a data query method and related devices. Background Art
[0002] With the rapid development of Internet technology, the amount of data generated by humans has exploded, and we have entered the era of big data. In big data applications, the efficiency of data query directly affects the system response time and user experience.
[0003] A data query method is as follows: some high-frequency query data are loaded from the underlying storage space into the cache in advance, and a whitelist is set, in which the data are recorded to have been loaded into the cache; when a user queries data through a client, the whitelist is queried first, and if the whitelist records that the data to be queried has been loaded into the cache, the data to be queried is directly read from the cache; if the whitelist does not record that the data to be queried has been loaded into the cache, the data to be queried is loaded from the underlying storage space into the cache, and then the data to be queried is read from the cache.
[0004] However, the above data query methods are not efficient enough for data query. Summary of the invention
[0005] The present application provides a data query method and related devices, aiming to improve the data query efficiency in a big data system.
[0006] In a first aspect, the present application provides a data query method, the method comprising:
[0007] Receive a query request, the query request is used to request to read the data to be queried; determine the first storage path of the data to be queried, the first storage path is the storage path of the data to be queried in the underlying storage system; detect whether the first storage path is included in the whitelist, the whitelist is used to record the storage path of the data in the underlying storage system and the storage path of the data in the data cache system, and there is a mapping relationship between the storage path of the data in the underlying storage system and the storage path of the data in the data cache system, and the second storage path is a storage path that can be recognized by the data cache system; if the whitelist contains the first storage path, read the data at the second storage path from the data cache system as the data to be queried, the second storage path is the path recorded in the whitelist and has a mapping relationship with the first storage path; if the whitelist does not contain the first storage path, read the data to be queried from the underlying storage system according to the first storage path; load the data to be queried into the data cache system, and add the first storage path and the second storage path and the mapping relationship between the first storage path and the second storage path in the whitelist, the second storage path is the storage path of the data to be queried in the data cache system.
[0008] In this method, when the data to be queried is loaded into the data cache system, the first storage path, the second storage path, and the mapping relationship between the first storage path and the second storage path are updated in the whitelist to ensure that the whitelist accurately indicates the storage status in the data cache system, so that the data cache system can give full play to its high-speed reading performance and improve data query efficiency.
[0009] In some implementations, a storage path of data in the underlying storage system and a storage path of data in the data cache system are the same.
[0010] In some implementations, a storage path of data in an underlying storage system and a storage path of data in a data cache system are different.
[0011] Through the mapping relationship between the first storage path and the second storage path in the whitelist, it is possible to avoid modifying the file system protocol in the underlying storage system, and flexibly adapt to the coexistence of multiple query engines.
[0012] In some implementations, loading the data to be queried into a data cache system, and adding the first storage path and the second storage path and the mapping relationship between the first storage path and the second storage path in a whitelist include:
[0013] The query heat of the data to be queried is added to the heat record table, which is used to record the query heat of the data to be queried in all previous times; if the query heat of the data to be queried indicates that the data to be queried is hot data, the data to be queried is loaded from the underlying storage system to the data cache system, and the first storage path and the second storage path and the mapping relationship between the first storage path and the second storage path are added to the whitelist.
[0014] Relying on the heat record table to make value judgments on the query data can ensure that high-heat and high-value data are stored in the data cache system, thereby improving the query efficiency of hot data.
[0015] In some implementations, the method further includes:
[0016] After the storage path of the data to be queried in the underlying storage system is changed from the first storage path to the third storage path, the first storage path in the whitelist is updated to the third storage path.
[0017] In some implementations, updating the first storage path in the whitelist to the third storage path includes:
[0018] Periodically check whether there is changed data in the underlying storage system, and the changed data includes data whose storage path in the underlying storage system has changed; if changed data is detected in the underlying storage system, the storage path before the changed data recorded in the whitelist is updated to the storage path after the change, and the changed data includes the data to be queried.
[0019] By detecting changed data in the underlying storage system, it is possible to ensure that the underlying storage path recorded in the whitelist and the actual underlying storage path of the data stored in the data cache system are synchronized, reducing the possibility of reduced query efficiency due to changes in the underlying storage path.
[0020] In some implementations, the method further includes:
[0021] Periodically update the heat record table; periodically clear the data stored at the storage path mapped by the target storage path in the data cache system, as well as the target storage path recorded in the whitelist and the storage path mapped by the target storage path, where the target storage path is the storage path associated with the cold data indicated by the query heat recorded in the heat record table.
[0022] By periodically updating the heat record table and the data cache system, the utilization rate of the storage space in the data cache system can be further improved, and the data query efficiency can be improved.
[0023] In a second aspect, the present application provides a data query device, which includes various functional modules for implementing any query method mentioned in the above implementations. In some implementations, each module can be implemented by software and / or hardware.
[0024] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor.
[0025] The memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the data query method provided in the first aspect.
[0026] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, they are used to implement the data query method provided in the first aspect.
[0027] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the data query method provided in the first aspect when the computer program is executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0029] Figure 1 This is a schematic diagram of the architecture of the big data ecosystem;
[0030] Figure 2 A flowchart of a data query method provided by an embodiment of the present application;
[0031] Figure 3 A schematic diagram of a flow chart for implementing a heat record table update in a data query method provided in an embodiment of the present application;
[0032] Figure 4 A schematic diagram of the structure of a data query device provided by an embodiment of the present application;
[0033] Figure 5 A schematic diagram of the structure of a data query device provided in another embodiment of the present application.
[0034] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0035] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0036] Figure 1 This is a schematic diagram of the architecture of the big data ecosystem. Figure 1 As shown in the figure, the big data ecosystem is divided into three layers, namely the application layer, computing layer and storage layer from top to bottom.
[0037] Among them, the application layer is usually the client of big data applications, and developers can implement operational interactions in the big data ecosystem through the client.
[0038] The computing layer includes a data query and analysis engine, through which data from a variety of heterogeneous data sources can be collected to access the storage layer.
[0039] The storage layer includes a data cache system and an underlying storage system. The data cache system and the underlying storage system can use different types of file systems to store data, and accordingly, the data storage path can contain different prefixes.
[0040] The underlying data storage system can store data that needs to be persisted and locate the file location through a unified namespace "directory tree". The underlying data storage system can map the stored structured data files into data tables and process and calculate the stored files.
[0041] As an example, the underlying data storage system may be a distributed file system (Hadoop Distributed File System, HDFS) under the “Hadoop” framework.
[0042] The data cache system is used to temporarily store data that needs to be read at high speed. It can adopt a memory-centric virtual distributed storage system, unify the file system of the underlying heterogeneous data sources, and build an interactive bridge between the upper computing layer and the underlying storage system.
[0043] As an example, the data cache system may be the distributed storage system “Alluxio”.
[0044] When developers need to query data from the big data ecosystem, they first input a query request from the client. The query request is usually structured query language (SQL). The SQL query statement contains three dimensions of information. The hierarchy of these three dimensions of information is "catalog.database.table". The category of the data source can be determined based on the catalog information, for example, the data source is "Hive". The corresponding database in the data source can be determined based on the database information. The corresponding table in the database can be indexed based on the table information. The table stores the metadata information of the target data to be queried. The metadata information not only includes the storage path of the target data in the underlying storage system, but also includes field information, storage format information, compression information, and other information used to describe the target data.
[0045] The client submits the query request to the query engine, which parses the query request containing the above three-dimensional information to obtain the storage path of the data to be queried in the underlying storage system. According to the storage path, the data to be queried can be read and returned from the underlying storage system.
[0046] In order to improve the efficiency of data query, in one data query method, the data that needs to be queried frequently is loaded from the underlying data storage system to the data cache system in advance, a whitelist is set up for the data that needs to be queried frequently, and the storage path of the above data in the data cache system is recorded in the whitelist. The developer uses the query engine to parse the storage path of the data to be queried in the underlying storage system. According to the storage path, the query engine checks the whitelist. If the same storage path exists in the whitelist, the data to be queried is read directly from the data cache system. Otherwise, the data to be queried needs to be loaded from the underlying storage system to the data cache system, and then read from the data cache system.
[0047] Taking the data cache system using the "Alluxio" system as an example, the storage path prefix corresponding to the data stored in the data cache system is "Alluxio: / / ". The "Alluxio" system can only recognize the storage path with the "Alluxio: / / " prefix.
[0048] The underlying data storage system is the HDFS system, and the storage path prefix corresponding to the data stored in the underlying data storage system is "HDFS: / / ". Therefore, in the above data query method, in order to detect the data to be queried from the whitelist, before querying, it is necessary to first modify the file system protocol of each data in the underlying storage system. For the big data ecosystem, the underlying storage system stores a huge amount of data. If the above modifications are made to each data, it will take a lot of effort.
[0049] In addition, the whitelist in the above data query method relies on manual configuration. When the hot data that needs to be queried frequently increases, this method cannot automatically update the whitelist, resulting in low data query efficiency and difficulty in meeting demand.
[0050] To solve the above problems, the present application provides a new data query method and related devices, aiming to improve the data query efficiency in big data systems.
[0051] The technical concept of the present application is: when the data to be queried is loaded from the underlying storage system to the data cache system, the storage information related to the data to be queried is recorded in the whitelist to ensure that the data in the whitelist is synchronized with the data stored in the data cache system.
[0052] At the same time, the whitelist in the present application also establishes a mapping relationship between the storage path of data in the underlying storage system and the storage path of data in the data cache system, thereby avoiding changes to the file system protocol in the underlying storage system.
[0053] Figure 2 The following is a flow chart of a data query method provided by an embodiment of the present application. Figure 2As shown, the data query method proposed in this application includes the following steps:
[0054] S201, receiving a query request, where the query request is used to request to read data to be queried.
[0055] In this step, the query engine receives a query request from the client. The query request is used to request to read the data to be queried, and is usually expressed as an SQL query statement input by a developer. The information contained in the SQL statement indicates the data to be queried.
[0056] S202, determining a first storage path of the data to be queried, where the first storage path is a storage path of the data to be queried in an underlying storage system.
[0057] The query engine parses the information contained in the SQL query statement, and based on the three-dimensional information obtained from the analysis, indexes the table storing the metadata information of the data to be queried in the corresponding database in the management component "Hive" of the HDFS system. Based on the metadata information of the data to be queried, the storage path of the data to be queried in the underlying storage system can be determined, that is, the first storage path of the data to be queried.
[0058] S203, detect whether the whitelist contains the first storage path, the whitelist is used to record the storage path of the data in the underlying storage system and the storage path of the data in the data cache system, there is a mapping relationship between the storage path of the data in the underlying storage system and the storage path of the data in the data cache system, the second storage path is a storage path that can be recognized by the data cache system.
[0059] The whitelist is used to indicate whether the data cache system stores the corresponding data. When the data is loaded from the underlying storage system to the data cache system, the whitelist records the storage path of the data in the underlying storage system and the storage path of the data in the data cache system. In the whitelist, there is a mapping relationship between the storage path of the data in the underlying storage system and the storage path of the data in the data cache system, that is, the storage path of the data in the data cache system can be determined by the storage path of the data in the underlying storage system.
[0060] In some implementations, the file system protocol of each data in the underlying storage system is not changed. For example, the prefix of the storage path corresponding to the data in the HDFS system is "HDFS: / / ", and the prefix of the storage path corresponding to the data in the "Alluxio" system is "Alluxio: / / ". The storage path of the data in the underlying storage system is different from the storage path of the data in the data cache system.
[0061] In some implementations, the file system protocol of the data in the underlying storage system is changed, and the prefix of the storage path corresponding to the data in the HDFS system is changed from "HDFS: / / " to "Alluxio: / / ", and the data in the underlying storage system is loaded into the data cache system. In this case, the storage path of the data in the underlying storage system is the same as the storage path of the data in the data cache system.
[0062] The query engine compares the first storage path of the data to be queried with the storage path recorded in the white list. If the white list contains the first storage path, it indicates that the data to be queried has been stored in the data cache system, and the process jumps to step S204. Otherwise, it indicates that the data to be queried has not been stored in the data cache system, and the process jumps to step S205.
[0063] S204: Read data at a second storage path from the data cache system as data to be queried, where the second storage path is a path recorded in the whitelist and having a mapping relationship with the first storage path.
[0064] The whitelist records the storage path of the data in the underlying storage system and the storage path of the data in the data cache system respectively, and there is a mapping relationship between the above two storage paths. When the whitelist includes the first storage path, the second storage path can be determined through the mapping relationship between the first storage path and the second storage path, that is, the storage path of the data to be queried in the data cache system.
[0065] The query engine reads target data from the second storage path in the data cache system. The target data is the data to be queried. The read data to be queried can be returned to the client through the query engine.
[0066] In this step, through the mapping relationship between the first storage path and the second storage path in the whitelist, it is possible to detect from the whitelist whether the data to be queried is stored in the cache system without changing the file system protocol of the data in the underlying storage system, saving the time of modifying the storage path of the data in the underlying storage system and eliminating the energy cost required for the modification. In addition, since there is no need to modify the metadata information of the data in the underlying storage system, the data query method of the present application can also support other query scenarios that do not require access to the data cache system, and support the coexistence of multiple query engines.
[0067] S205: Read the data to be queried from the underlying storage system according to the first storage path.
[0068] In this step, the data cache system transparently transmits the query request issued by the query engine to the underlying storage system, and the underlying storage system returns the data at the first storage path as the data to be queried to the query engine according to the query request.
[0069] S206, loading the data to be queried into the data cache system, and adding the first storage path and the second storage path and the mapping relationship between the first storage path and the second storage path in the whitelist, the second storage path being the storage path of the data to be queried in the data cache system.
[0070] The underlying storage system loads the data to be queried at the first storage path into the data cache system. Since the role of the whitelist is to indicate whether the corresponding data is stored in the data cache system, the whitelist needs to be updated. Accordingly, the whitelist records the storage path of the data to be queried in the underlying storage system and the storage path of the data to be queried in the data cache system, that is, the first storage path and the second storage path are added to the whitelist. In order to ensure that the query engine can read the data to be queried from the data cache system when it detects that the whitelist contains the first storage path, the whitelist also needs to record the mapping relationship between the first storage path and the second storage path.
[0071] In some implementations, the query engine relies on a heat record table to record the query heat of all previous data to be queried. As an example, the query heat of each data to be queried is the number of times it is queried within a certain period of time. Whenever the query engine obtains the same storage path based on the query request resolution, the query heat of the data to be queried corresponding to the storage path increases by 1 time.
[0072] The query engine determines the query heat of the data to be queried. When the query heat of the data to be queried reaches a preset threshold, it indicates that the data to be queried is data that needs to be accessed frequently within a certain period of time, and the data to be queried is determined to be hot data. Only when the data to be queried is hot data, it indicates that the data to be queried is worth caching. The underlying storage system will load the data to be queried that is determined to be hot data into the data cache system. Accordingly, the query engine records the mapping relationship between the first storage path and the second storage path and the first storage path and the second storage path in the whitelist.
[0073] As an example, a remote dictionary server (remote dictionary server, redis) or other services with similar functions may be used to maintain the heat record table.
[0074] Taking redis as an example, the heat record table uses key-value pairs<key,value> The data structure is in the form of, key is the first storage path, value is the query heat of the data to be queried associated with the first storage path. For example: a data sample in the heat record table is<HDFS: / / path1,999> , the key in this data sample indicates the data to be queried in the underlying storage system with the storage path HDFS: / / path1, and the value indicates that the data has been queried 999 times within 24 hours. If the preset threshold is 1000, the query engine can determine that the data is cold data by parsing the heat record table. If the developer still requests to read the data in the next query request and the time from the first query request is no more than 24 hours, the data sample is updated to<HDFS: / / path1,1000> ,The query engine parses the data as hot data according to the preset threshold, and the underlying storage system loads the hot data into the data cache system and updates the whitelist.
[0075] In the above implementation, the value of the data to be queried can be judged by relying on the heat record table. Only when the data to be queried is hot data with cache value, the data to be queried is loaded from the underlying storage system to the data cache system, ensuring that the storage space in the data cache system is efficiently utilized and improving data query efficiency.
[0076] In this embodiment, when the data to be queried is loaded into the data cache system, the first storage path, the second storage path, and the mapping relationship between the first storage path and the second storage path are updated in the whitelist to ensure that the whitelist accurately indicates the storage situation in the data cache system, so that the data cache system can give full play to its high-speed reading performance and improve data query efficiency.
[0077] According to the above embodiments, after multiple queries, the whitelist records the storage path of each target data in the underlying storage system, the storage path of each target data in the data cache system, and the mapping relationship between the above two storage paths. The query engine detects the whitelist based on the storage path of the data to be queried in the underlying storage system in each query. If the data to be queried has been stored in the data cache system, but its storage path in the underlying storage system has changed, the first storage path in the whitelist cannot match the changed underlying storage path, which causes the query engine to fail to detect the whitelist, and has to read the data to be queried from the underlying storage system again according to step S205 in the above embodiment, which ultimately reduces the efficiency of data query.
[0078] In view of the above situation, the data query method proposed in this application also includes:
[0079] When the storage path of the to-be-queried data in the underlying storage system is changed from the first storage path to the third storage path, the first storage path in the whitelist is updated to the third storage path.
[0080] In some implementations, the method periodically detects whether there is changed data in the underlying storage system, and the changed data includes data in which the storage path in the underlying storage system has changed; if changed data is detected in the underlying storage system, the storage path before the change of the changed data recorded in the whitelist is updated to the storage path after the change, and the changed data includes the data to be queried.
[0081] Since the whitelist cannot actively sense changes in the data to be queried in the underlying storage system, the underlying storage system needs to periodically detect changes in the data stored in the data cache system and actively pass the changed storage path to the data cache system, which will then update the content recorded in the whitelist.
[0082] For example, the storage path of a piece of data in the underlying storage system is HDFS: / / path1, and the storage path of the data in the data cache system is Alluxio: / / path1. The whitelist records the above two storage paths and uses key-value pairs.<key,value> The mapping relationship between the first storage path and the second storage path is recorded in the form of<HDFS: / / path1,Alluxio: / / path1> The underlying storage system detects the underlying storage paths stored in the whitelist every hour with a detection cycle of one hour. If it is detected that the storage path of the data in the underlying storage system is changed to HDFS: / / path2, that is, the first storage path of the data is changed to HDFS: / / path2, the changed third storage path is passed to the data cache system. To ensure that the query engine can detect the third storage path in the whitelist, the data sample in the whitelist is updated to<HDFS: / / path2,Alluxio: / / path1> The above update not only modifies the first storage path, but also reconstructs the mapping relationship between the third storage path and the second storage path.
[0083] Through periodic self-checking of the underlying storage system, it can be ensured that the underlying storage path recorded in the whitelist of hot data is synchronized with the actual underlying storage path, reducing the possibility of decreased query efficiency of hot data due to changes in the underlying storage path.
[0084] like Figure 3 As shown, an embodiment of the present application provides a data query method for periodically clearing a data cache system and a heat record table, which specifically includes the following steps:
[0085] S301, periodically update the heat record table.
[0086] The heat record table records the query heat of all the data to be queried, and the query heat of each data to be queried is the number of times it is queried within a period of time. Therefore, as time goes by, the query heat of each data to be queried also changes. Considering that the heat record table does not have the ability to automatically update, the query engine needs to periodically check the heat record table and update the query heat of all the data to be queried recorded therein.
[0087] For example, the query engine updates the query heat in the heat record table in an hourly cycle. The storage path of a piece of data in the underlying storage system is HDFS: / / path1. The data is hot data stored in the data cache system. The number of queries in the past 24 hours from 0:00 to 24:00 on the same day is 1010. The query engine updates the query heat of the data at 24:00 on the same day. The data is represented in the heat record table as<HDFS: / / path1,1010> One hour later, at 1:00 a.m. the next day, the number of queries for the data in the past 24 hours from 1:00 a.m. that day to 1:00 a.m. the next day drops to 1005. The query engine updates the heat record table at 1:00 a.m. the next day, and the data is updated in the heat record table as<HDFS: / / path1,1005> , because its query popularity is 1005 times, which exceeds the preset threshold, the query engine determines that the data is still hot data. When the time reaches 2:00 the next day, the number of queries for the data in the past 24 hours from 2:00 on the current day to 2:00 on the next day drops to 990 times, so the query engine updates the heat record table at 2:00 on the next day, and the data is represented in the heat record table as<HDFS: / / path1,990> At this time, the query popularity of the data is lower than the preset threshold, and the query engine determines that the data has changed to cold data and no longer has cache value.
[0088] S302, periodically clear the data stored at the target storage path in the data cache system, as well as the target storage path recorded in the whitelist and the storage path mapped to the target storage path, where the target storage path is the storage path associated with the cold data indicated by the query heat recorded in the heat record table.
[0089] If the query engine detects that hot data has changed to cold data after periodically updating the heat record table, the data that has changed from hot data to cold data no longer has cache value, and the underlying storage path associated with the data is the target storage path. According to the target storage path recorded in the whitelist and the storage path mapped to the target storage path, the data at the storage path mapped to the target storage path in the data cache system can be cleared, and the target storage path in the whitelist and the storage path mapped to the target storage data can be cleared at the same time, and the low-value data in the whitelist and the data cache system can be cleared.
[0090] In this embodiment, by periodically updating the heat record table and the data cache system, the utilization rate of the storage space in the data cache system can be further improved, and the data query efficiency can be improved.
[0091] It can be understood that in the above embodiment, developers only need to input query requests through the client, do not need to understand the details of data tiered storage, and do not need to interact with multiple levels. The client only accesses the query engine. The above data query process realizes imperceptible transparent access, reduces the coupling between the client and the data storage level, and improves the efficiency of data query.
[0092] FIG4 is a schematic diagram of the structure of a data query device provided by an embodiment of the present application. Figure 4 As shown, the device 400 of this embodiment may include a receiving module 401 , a detecting module 402 , and a processing module 403 .
[0093] The receiving module 401 is used to receive a query request, and the query request is used to request to read the data to be queried.
[0094] The processing module 403 is used to determine a first storage path of the data to be queried, where the first storage path is a storage path of the data to be queried in the underlying storage system.
[0095] The detection module 402 is used to detect whether the whitelist includes the first storage path. The whitelist is used to record the storage path of the data in the underlying storage system and the storage path of the data in the data cache system. There is a mapping relationship between the storage path of the data in the underlying storage system and the storage path of the data in the data cache system. The second storage path is a storage path that can be recognized by the data cache system.
[0096] The processing module 403 is also used to read data at a second storage path from the data cache system as the data to be queried if the whitelist includes the first storage path, where the second storage path is a path recorded in the whitelist and having a mapping relationship with the first storage path.
[0097] The processing module 403 is further configured to read the to-be-queried data from the underlying storage system according to the first storage path if the whitelist does not include the first storage path.
[0098] The processing module 403 is used to load the data to be queried into the data cache system, and add the first storage path and the second storage path and the mapping relationship between the first storage path and the second storage path in the whitelist, where the second storage path is the storage path of the data to be queried in the data cache system.
[0099] In some implementations, a storage path of data in the underlying storage system and a storage path of data in the data cache system are the same.
[0100] In some implementations, a storage path of data in an underlying storage system and a storage path of data in a data cache system are different.
[0101] In some implementations, the processing module 403 is used to add the query heat of the data to be queried in the heat record table, and the heat record table is used to record the query heat of the data to be queried in all previous times.
[0102] Processing module 403 is used to load the data to be queried from the underlying storage system to the data cache system if the query heat of the data to be queried indicates that the data to be queried is hot data, and add the first storage path and the second storage path and the mapping relationship between the first storage path and the second storage path to the whitelist.
[0103] In some implementations, the processing module 403 is used to update the first storage path in the whitelist to the third storage path after the storage path of the to-be-queried data in the underlying storage system is changed from the first storage path to the third storage path.
[0104] The processing module 403 is used to update the first storage path in the whitelist to the third storage path, including:
[0105] The detection module 402 is used to periodically detect whether there is changed data in the underlying storage system, and the changed data includes data in which the storage path in the underlying storage system is changed.
[0106] The processing module 403 is used to update the storage path of the changed data recorded in the whitelist before the change to the storage path after the change if it is detected that there is changed data in the underlying storage system, and the changed data includes the data to be queried.
[0107] In some implementations, the processing module 403 is also used to periodically update the heat record table. The processing module 403 is used to periodically clear the data stored at the storage path mapped by the target storage path in the data cache system and the target storage path recorded in the whitelist and the storage path mapped by the target storage path, where the target storage path is the storage path associated with the cold data indicated by the query heat recorded in the heat record table.
[0108] It should be understood that the device 400 is embodied in the form of functional modules. The term "module" may refer to an application-specific integrated circuit, an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, a combined logic circuit, and / or other suitable components that support the described functionality.
[0109] The above-mentioned device 400 has the function of implementing the corresponding processes and / or steps in the above-mentioned method embodiment; the above-mentioned functions can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions.
[0110] Figure 5 A schematic diagram of the structure of a data query device provided in another embodiment of the present application. Figure 5 The device 500 shown can be used to execute any of the aforementioned methods performed by the data query device.
[0111] like Figure 5 As shown, the device 500 of this embodiment includes: a memory 501, a processor 502, a communication interface 503 and a bus 504. The memory 501, the processor 502 and the communication interface 503 are connected to each other through the bus 504.
[0112] The memory 501 may be a read-only memory, a static storage device, a dynamic storage device or a random access memory (RAM). The memory 501 may store a program, and when the program stored in the memory 501 is executed by the processor 502, the processor 502 is used to execute any of the aforementioned methods.
[0113] The processor 502 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit, or one or more integrated circuits for executing related programs.
[0114] The processor 502 may also be an integrated circuit chip with signal processing capability. In the implementation process, each relevant step in the embodiment of the present application may be completed by an integrated logic circuit of hardware in the processor 502 or by instructions in the form of software.
[0115] The processor 502 may also be a general purpose processor, a digital signal processor, an application specific integrated circuit, an off-the-shelf programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0116] The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 501, and the processor 502 reads the information in the memory 501, and completes the functions required to be performed by the units included in the device of the present application in combination with its hardware.
[0117] The communication interface 503 may use, but is not limited to, a transceiver or other transceiver device to implement communication between the apparatus 500 and other devices or apparatuses.
[0118] The bus 504 may include a path for transmitting information between various components of the device 500 (eg, the memory 501 , the processor 502 , and the communication interface 503 ).
[0119] An embodiment of the present application further provides a computer-readable storage medium, in which computer instructions are stored. When a processor executes the computer instructions, each step of the method in the above embodiment is implemented.
[0120] The embodiment of the present application also provides a computer program product, including computer instructions, which implement the various steps of the method in the above embodiment when executed by a processor.
[0121] It should be noted that the modules or components shown in the above embodiments may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits, or one or more microprocessors, or one or more field programmable gate arrays, etc. For another example, when a certain module above is implemented in the form of a processing element calling a program code, the processing element may be a general-purpose processor, such as a central processing unit or other processor that can call a program code, such as a controller. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).
[0122] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, software module or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).
[0123] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0124] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A data query method, characterized in that: The method comprises: Receiving a query request, wherein the query request is used to request to read the data to be queried; Determine a first storage path of the data to be queried, where the first storage path is a storage path of the data to be queried in an underlying storage system; Detect whether the first storage path is included in the whitelist, the whitelist is used to record the storage path of the data in the underlying storage system and the storage path of the data in the data cache system, and there is a mapping relationship between the storage path of the data in the underlying storage system and the storage path of the data in the data cache system, and the second storage path is a storage path that can be recognized by the data cache system; If the whitelist includes the first storage path, data at a second storage path is read from the data cache system as the data to be queried, where the second storage path is a path recorded in the whitelist and having a mapping relationship with the first storage path; If the whitelist does not include the first storage path, reading the data to be queried from the underlying storage system according to the first storage path; The data to be queried is loaded into the data cache system, and the first storage path and the second storage path as well as the mapping relationship between the first storage path and the second storage path are added to the whitelist, the second storage path being the storage path of the data to be queried in the data cache system.
2. The method according to claim 1, characterized in that The storage path of data in the underlying storage system and the storage path of data in the data cache system are the same.
3. The method according to claim 1, characterized in that The storage path of data in the underlying storage system and the storage path of data in the data cache system are different.
4. The method according to any one of claims 1 to 3, characterized in that The step of loading the to-be-queried data into the data cache system and adding the first storage path and the second storage path and a mapping relationship between the first storage path and the second storage path in the whitelist includes: Adding the query heat of the data to be queried in a heat record table, wherein the heat record table is used to record the query heat of the data to be queried in all previous times; If the query heat of the data to be queried indicates that the data to be queried is hot data, the data to be queried is loaded from the underlying storage system into the data cache system, and the first storage path and the second storage path and the mapping relationship between the first storage path and the second storage path are added to the whitelist.
5. The method according to claim 4, characterized in that The method further comprises: After the storage path of the to-be-queried data in the underlying storage system is changed from the first storage path to the third storage path, the first storage path in the whitelist is updated to the third storage path.
6. The method according to claim 5, characterized in that The updating the first storage path in the whitelist to the third storage path includes: Periodically detecting whether there is changed data in the underlying storage system, the changed data including data in which a storage path in the underlying storage system is changed; If it is detected that the changed data exists in the underlying storage system, the storage path of the changed data before the change recorded in the whitelist is updated to the storage path after the change, and the changed data includes the data to be queried.
7. The method according to claim 6, characterized in that The method further comprises: Periodically updating the heat record table; Periodically clear the data stored at the storage path mapped by the target storage path in the data cache system, as well as the target storage path and the storage path mapped by the target storage path recorded in the whitelist, wherein the target storage path is the storage path associated with the cold data indicated by the query heat recorded in the heat record table.
8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the data query method according to any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the data query method according to any one of claims 1 to 7.
10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the data query method according to any one of claims 1 to 7.