Data query and storage method, related equipment and storage medium

Abstract

An embodiment of the present application discloses a data query method, a storage method, related equipment and a computer storage medium, wherein the data query method includes: obtaining information to be queried, the information to be queried is used to indicate data filtering conditions and ranking information under the data filtering conditions; determining the storage address of data that meets the data filtering conditions and ranking information on the disk based on the record information in the memory; reading the data from the storage address on the disk; and using the read data as the data expected to be queried under the information to be queried.

Description

Technical Field

[0001] The present application relates to data processing technology, and in particular to a data query method, storage method, related equipment and computer storage medium. Background Art

[0002] For persistent data—data that is unlikely to change over a period of time, such as mobile phone rankings from the past week—related technologies typically use in-memory local caches, such as Caffeine local caches, or distributed caches, such as Redis distributed caches, to cache this data. Specifically, in practical applications, to query data, such as the top 10 mobile phone rankings from the past week, one first needs to obtain query criteria (Query Criteria 1: Last Week; Query Criteria 2: Mobile Phone Category; Query Criteria 3: Top 10), perform a query in the database based on the query criteria, and then read the query results into the in-memory Redis distributed cache or Caffeine local cache to present the query results, thereby completing the data query. This solution has at least the following drawbacks: Data must be read into memory for caching, and memory capacity is often limited, resulting in a limited amount of cached data. Querying data that does not exist in memory requires deleting the existing data and then reading the desired data into memory for caching. Deleted data may subsequently be cached back into memory when it is queried again, resulting in a large amount of duplicate caching operations. Duplicate caching reduces data query efficiency and may even damage memory. In practical applications, memory itself is expensive, and frequent memory replacement is not conducive to economic cost savings. Moreover, the above solution cannot adapt to the situation of parallel querying of large amounts of data. Summary of the Invention

[0003] To solve existing technical problems, the embodiments of the present application provide a data query method, a storage method, related equipment and a computer storage medium.

[0004] The technical solution of the embodiment of the present application is implemented as follows: The embodiment of the present application provides a data query method, including:

[0005] Obtaining information to be queried, the information to be queried being used to indicate a data screening condition and ranking information under the data screening condition;

[0006] Determine the storage address of the data in the disk that meets the data screening conditions and ranking information based on the record information in the memory;

[0007] Reading the data from the storage address of the disk;

[0008] The read data is used as the data expected to be queried under the information to be queried.

[0009] In the above solution, the data is stored on the disk as data at a node position in a tree, and the tree stored on the disk has a tree address on the disk;

[0010] Accordingly, the method of determining the storage address of the data that meets the data screening condition and ranking information on the disk based on the record information in the memory; reading the data from the storage address on the disk, and using the read data as the data expected to be queried under the information to be queried includes:

[0011] Determine the tree where the data that meets the data screening conditions and ranking information is located based on the record information in the memory;

[0012] Determine the tree address of the tree on disk;

[0013] Based on the tree address, the tree in the disk is traversed to determine the node position in the tree for storing the data of the ranking information; and the data stored at the node position is read;

[0014] The read data is used as the data expected to be queried under the information to be queried.

[0015] In the above solution, the method further includes:

[0016] Determining whether the memory stores record information of data that meets the data screening condition and ranking information;

[0017] If the judgment is yes, the tree where the data that meets the data screening condition and ranking information is located is determined based on the record information in the memory.

[0018] In the above solution, the method further includes:

[0019] If the judgment is no,

[0020] Obtaining data that meets the data screening conditions and the ranking information from a database;

[0021] Create a tree on disk for the obtained data;

[0022] The tree is assigned a tree address, and the data is assigned a node position in the tree.

[0023] In the above solution, the method further includes:

[0024] Obtain all data stored in the disk that meets the data screening conditions;

[0025] Obtaining an attribute of each of the data, wherein the attribute represents a ranking of each of the data under the data screening condition;

[0026] Combine all data according to the properties.

[0027] In the above solution, all data that meets the data screening conditions are located in at least two trees;

[0028] Accordingly, all data are merged according to the attributes, including:

[0029] According to attributes of all data stored in the at least two trees, the at least two trees are merged into one tree.

[0030] The present invention provides a data storage method applied to a disk, including:

[0031] Obtaining data to be stored;

[0032] Storing the data to be stored; the stored data to be stored has at least indication information, the indication information being used to indicate a data screening condition for screening out the data to be stored and a ranking of the data to be stored under the data screening condition;

[0033] The indication information and the storage location of the data to be stored in the disk are recorded to obtain record information; the record information is used to query the data to be stored.

[0034] The present invention provides a data storage method for use in a memory, including:

[0035] Obtaining record information, wherein the record information is information recording indication information and a storage location of the data stored in the disk, wherein the indication information is used to indicate a data screening condition for screening the data to be stored and a ranking of the data to be stored under the data screening condition;

[0036] The record information is stored, wherein the record information is used to query the data stored in the disk.

[0037] The present invention provides a data query device, including:

[0038] an obtaining unit, configured to obtain information to be queried, wherein the information to be queried is used to indicate a data screening condition and ranking information under the data screening condition;

[0039] A first determining unit is configured to determine, based on the record information in the memory, a storage address in the disk of the data that meets the data screening condition and the ranking information;

[0040] A reading unit, configured to read the data from the storage address of the disk;

[0041] The second determining unit is configured to use the read data as data expected to be queried under the information to be queried.

[0042] An embodiment of the present application provides a disk, including:

[0043] A storage unit for storing data to be stored; the stored data to be stored has at least indication information, the indication information being used to indicate a data screening condition for screening out the data to be stored and a ranking of the data to be stored under the data screening condition;

[0044] The recording unit is used to record the indication information and the storage location of the data to be stored in the disk to obtain recording information; the recording information is used to query the data to be stored.

[0045] An embodiment of the present application provides a memory, including:

[0046] an obtaining unit, configured to obtain record information, wherein the record information is information recording the indication information and the storage location of the data stored in the disk, wherein the indication information is used to indicate a data screening condition for screening the data to be stored and a ranking of the data to be stored under the data screening condition;

[0047] The storage unit is used to store the record information, wherein the record information is used to query the data stored in the disk.

[0048] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned data query and / or storage method.

[0049] An embodiment of the present application provides a data query method, a storage method, related equipment and a computer storage medium, wherein the data query method includes: obtaining information to be queried, the information to be queried is used to indicate data filtering conditions and ranking information under the data filtering conditions; determining the storage address of data that meets the data filtering conditions and ranking information on the disk based on the record information in the memory; reading the data from the storage address on the disk; and using the read data as the data expected to be queried under the information to be queried.

[0050] The embodiments of the present application cache or store the data to be queried on disk, thus achieving disk-level caching. The memory in the embodiments of the present application does not need to cache data, but instead records or stores information, which can reduce the burden on the memory and effectively avoid the problem of frequent memory replacement that may result from data caching in the memory. Because the disk space is larger than the memory cache space, the amount of disk cached data is large, which can support the query of large amounts of parallel data. Furthermore, the cost of disks is cheaper than that of memory, which can effectively save expenditure costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0052] Figure 1 Schematic diagram of the process of the embodiment of the data query method provided by this application Figure 1 ;

[0053] Figure 2 Schematic diagram of the process of the embodiment of the data query method provided by this application Figure 2 ;

[0054] Figure 3 A schematic diagram of a flow chart of an embodiment of a data storage method applied to a disk provided in this application;

[0055] Figure 4 A schematic diagram of a flow chart of an embodiment of a data storage method applied to a memory provided by the present application;

[0056] Figure 5 A schematic diagram of the interactions involved in the disk-based local caching solution provided in this application;

[0057] Figure 6 A schematic diagram of the specific implementation of the disk-based local caching solution provided in this application;

[0058] Figure 7 A schematic diagram of the structure of the data query device provided in this application;

[0059] Figure 8 A schematic diagram of the structure of the disk provided for this application;

[0060] Figure 9 A schematic diagram of the memory structure provided for this application;

[0061] Figure 10 Schematic diagram of the hardware structure of the data query device provided in this application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. Unless there is a conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. The steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions. In addition, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that here.

[0063] In related technologies, the technology of caching data in memory can be regarded as memory-level data caching technology. The data storage and query method provided in the embodiments of the present application can achieve disk-level data caching and querying, which can effectively avoid the problem of frequent memory replacement caused by data caching in memory (equivalent to memory-level data caching). Because the disk space is larger than the cache space of the memory, the disk can cache a large amount of data, which can effectively realize the query of a large amount of parallel data.

[0064] This application provides an embodiment of a data query method, which is applied to a data query device, such as Figure 1 As shown, the method includes:

[0065] S101: Obtain information to be queried, where the information to be queried is used to indicate a data screening condition and ranking information under the data screening condition;

[0066] In this step, the data query device receives the information to be queried input by the user, and the user wants to query the expected data. Furthermore, the user can input data screening conditions into the data query device, such as the user expects to query 3C products, clothing products, etc., and also input ranking information under the data screening conditions, such as the top 10 products in the 3C category, the top 5 products in the clothing category, etc. It can be understood that the ranking information input by the user may refer to at least two ranking information under the data screening conditions, such as ranking 1st to 10th in terms of positive reviews or ranking 1st to 5th; in addition, the ranking information in the embodiment of the present application may also be a specified or specific ranking information, such as sales ranking 1st or 3rd. It can be flexibly input according to the actual needs of the user, without specific limitation. Data

[0067] S102: Determine the storage address of the data in the disk that meets the data screening condition and ranking information based on the record information in the memory;

[0068] In this step, the memory is used to record or store record information. The data query device can determine the storage address of the data that meets the data screening conditions and the ranking information on the disk based on the record information recorded or stored in the memory. In the embodiment of the present application, the data to be queried is cached or stored on the disk. And the information of the storage address of the data to be queried on the disk is recorded or stored in the memory, so that the data stored or cached on the disk can be found based on this information stored or recorded in the memory. In one embodiment, the record information stored or recorded in the memory is used to record the storage address of a certain data on the disk and the data screening conditions and ranking information required to filter out the data. For example, the product with the highest number of positive reviews in the 3C category is stored in address A of the disk. The product with the highest number of positive reviews in the clothing category is stored in address B of the disk. It can be understood that the record information itself does not store or record the product data itself, but the data screening conditions that can filter out the data and the ranking information of the data under the data screening conditions. That is, in the embodiment of the present application, the memory does not need to cache the data itself, but only needs to store or record the data filtering conditions for filtering out the data, the ranking information of the data under the data filtering conditions, and the storage address of the data.

[0069] S103: Reading the data from the storage address of the disk;

[0070] S104: The read data is used as the data expected to be queried under the information to be queried.

[0071] In S103-S104, the data query device reads the data that the user wants to query from the storage address of the disk.

[0072] In an optional solution, after S101 to S104, the method further includes: the data query device displays the data expected to be queried under the information to be queried for display to the user. Preferably, when displaying, the expected query data can also be displayed according to the ranking information under the data filtering conditions. For example, if the expected query data is the top 5 sales volume of 3C products, when displaying, the data can be arranged and displayed in order of sales volume from 1 to 5.

[0073] In the aforementioned S101 to S104, the embodiment of the present application caches or stores the data to be queried on the disk, thereby realizing disk-level caching. The memory in the present application no longer caches the queried data, but instead records or stores the information of the storage address of the queried data on the disk, which can reduce the burden on the memory and effectively avoid the problem of frequent memory replacement that may be caused by data caching in the memory (equivalent to memory-level data caching). In addition, since the disk space is larger than the cache space of the memory, the amount of disk cached data is larger, which can support the query of a large amount of parallel data. Moreover, the cost of the disk is cheaper than the memory, which can effectively save expenditure costs.

[0074] In an embodiment of the present application, data can be cached or stored in the form of an LSM (log merge) tree on the disk. In specific implementation, the LSM tree can be implemented based on an ordered structure such as a balanced binary tree and an array. Each tree can have two or more nodes, and each node stores a corresponding data. The data stored on a certain node can be regarded as a node data. For an understanding of the concepts of tree, tree node, etc., please refer to the relevant instructions, which will not be elaborated in this article. It can be understood that when the disk stores data in the form of a tree, each data can be stored as a node data in the tree, and each tree stored on the disk has a storage address in the disk. In order to distinguish it from each tree stored on the disk, the storage address of each tree on the disk is regarded as a tree address. It can be understood that the tree address can facilitate the data query device to find the tree on the disk. Only by finding the tree can the node data stored on the tree be found. In order to distinguish between the tree address and the address of the node data on the tree, the present embodiment introduces the concept of node position, which is used to uniquely identify the position of a node in the tree. The storage address of a node data in the tree on the disk can be regarded as the position of the node where the node data is located on the tree. Based on this, the above S102 to S104 can be implemented by the following scheme, such as Figure 2 As shown: S1021: according to the record information in the memory, determine the tree where the data that meets the data screening condition and ranking information is located;

[0075] S1022: Determine the tree address of the tree in the disk;

[0076] S1021 and S1022 are equivalent to implementing a solution for confirming the tree address on disk of the tree in which the data that meets the data screening conditions and ranking information resides. Based on this, in the embodiment of the present application, the memory does not need to cache the data itself. The recorded information in the memory can be the data screening conditions that filter out the data, the ranking information of the data under the data screening conditions, and the storage address of the tree in which the data is located, such as the tree address.

[0077] S1023: Based on the tree address, traverse the tree in the disk to determine the node position in the tree for storing the data of the ranking information; and read the data stored at the node position;

[0078] S1024: The read data is used as the data expected to be queried under the information to be queried.

[0079] S1023~S1024, based on the tree address, the tree is searched from the disk, and then the node positions on it are traversed to obtain the node positions in the tree for storing the data of the ranking information, and the data stored at the node positions are read out, which can be used as the data that the user expects to query.

[0080] In S1021 to S1024, the disk stores data in a tree format, which allows for rapid data search. The memory does not need to cache data; only record information needs to be stored or recorded. Data stored on the disk can also be quickly searched based on the record information recorded or stored in the memory.

[0081] In the above solution, the default memory stores the record information of the data that meets the data screening conditions and ranking information. In actual application, after obtaining the information to be queried, the method further includes:

[0082] S105: Determine whether the memory stores record information of data that meets the data screening condition and ranking information;

[0083] If the judgment is yes, the aforementioned technical solution of determining the tree where the data that meets the data screening condition and ranking information is located is executed based on the record information in the memory;

[0084] If the judgment is no, execute S106;

[0085] S106: Obtain data that meets the data screening condition and the ranking information from the database; create a tree for the obtained data in the disk; assign a tree address to the tree, and assign a node position in the tree to the data.

[0086] It will be understood that in the embodiments of the present application, the database stores a large amount of data, and caching the database data to disk can be considered as caching the database data locally. Compared to continuously reading data from the database when a query is required, reading data from the local disk is faster. If there is no record information for the desired query data in memory, the desired query data is read from the database, a tree is created, and a tree address is assigned to the tree, as well as a node position for the data within the tree. The tree address created for the tree on disk must uniquely identify the tree and not be confused with the tree addresses of other trees on disk. The data read from the database is stored at a node position in the created tree as node data. At this point, the data filtering conditions for the data can be stored or recorded corresponding to the ranking information of the data under the data filtering conditions and the tree address of the tree where the data is located, forming record information. This record information can be stored on disk or in memory for subsequent use. It will be understood that if the data is stored in the form of a tree, the tree address + the node position of the data in the tree can be considered as the storage address or storage location of the data on disk.

[0087] In a specific implementation, the data stored on the disk can be stored in a tree format or not. It can be understood that storing data in a tree format can be considered as caching data using a certain data storage structure. Alternatively, data can be directly stored without using any data storage structure, and the data's storage location on the disk can be recorded. In this case, the method further includes: obtaining all data stored on the disk that meets the data screening criteria; obtaining attributes of each data item; and merging all data items according to the attributes. The data attributes can specifically be the product category to which the data corresponds, such as 3C products or apparel. Furthermore, they can be rankings within a certain category, such as first in the 3C category or second in the apparel category. For example, if data is stored without using any data storage structure, the data for the top-ranked 3C product is stored at address a on the disk, the data for the second-ranked 3C product is stored at address c on the disk, and the data for the third-ranked 3C product is stored at address b on the disk. The top three products in the same category are merged according to their ranking within the 3C category to form an array representing the top three 3C products. The first element in the array represents the data of the first-ranked 3C product, the second element in the array represents the data of the second-ranked 3C product, and the third element in the array represents the data of the third-ranked 3C product.

[0088] In the above scheme, the example of data storage in a disk without any data storage structure is taken. If the data is stored in a certain data storage structure in the disk, such as in the form of LSM, if the data that meets the data screening conditions in the information to be queried is located on two or more trees, that is, all data that meet the same data screening conditions are located in at least two trees, then the two or more trees can be merged according to the attributes of all data stored in the at least two trees, and merged into one tree to merge the data belonging to the same data screening conditions. Exemplarily, the disk caches data on the top 10 products in the 3C category, of which the records of the top 5 products in the 3C category are stored on tree 1, and the records of the products ranked 6th to 10th in the 3C category are stored on tree 2, then the two trees can be merged to form a new tree. Those skilled in the art should understand that the merging of trees is actually the merging of data that meets the same data screening conditions, such as the top 10 in the 3C category. For example, the merged tree is equivalent to a new tree that merges data that meets the same data screening criteria, such as the top 10 in the 3C category. The 1st to 5th node positions in the new tree are used to sequentially record the top 5 product data in the 3C category, and the 6th to 10th node positions in the new tree are used to sequentially record the 6th to 10th product data in the 3C category. A tree address is assigned to the merged new tree. Each node in the tree has its own position in the tree. The address of the tree + the position of the node where the product data is located in the tree constitutes the storage address of the product data on the disk. Whether the cached data on the disk adopts a certain data storage structure or directly stores the data can be flexibly set according to the specific situation.

[0089] In the above scheme, a disk-based local cache is implemented, which is different from the memory-based local cache or distributed cache in related technologies. It utilizes cheap disk resources to solve the data caching problem. At the same time, the disk cache data has a large capacity, which can achieve caching of large amounts of data and concurrent querying of data. In the embodiment of the present application, only record information is cached or stored in the memory, and there is no need to cache or store the data itself. Based on the record information in the memory, the storage address of the data on the disk can be quickly and accurately determined, which can effectively improve query efficiency, support concurrent queries, and improve the efficiency of concurrent queries.

[0090] The aforementioned data query method is implemented based on the data storage method described below. The data storage method in the embodiment of the present application includes a method applied to a disk and a method applied to a memory. Figure 3 Shown, including:

[0091] S301: Obtain data to be stored;

[0092] In this step, the data to be stored is read from the database. All data in the database can be used as the data to be stored that needs to be cached to disk. Alternatively, data in the database that is most efficiently queried can be used as the data to be stored that needs to be cached to disk.

[0093] S302: Storing the data to be stored; the stored data to be stored has at least indication information, the indication information being used to indicate a data screening condition for screening out the data to be stored and a ranking of the data to be stored under the data screening condition;

[0094] In this step, the data to be stored, read from the database, is stored or cached on the local disk. The data cached on the local disk includes indicative information indicating the data filtering conditions that can be used to filter out the data to be stored, as well as the ranking of the data to be stored under the data filtering conditions. This indicative information can be allocated by the disk to each cached data item based on its attributes. The storage location of each cached data item on the disk can also be allocated.

[0095] S303: Record the indication information and the storage location of the data to be stored in the disk to obtain record information; the record information is used to query the data to be stored.

[0096] In this step, the indication information of each cached data and the storage location of each cached data in the disk are recorded correspondingly to form record information. The record information can be used to query the data cached in the disk.

[0097] In S301-S303, the data to be stored is stored on disk, achieving local disk-level data storage. This also obtains information indicating the data to be stored and a record of its storage location on disk. Based on this record, the data stored on disk can be quickly queried. Unlike memory-based local caching or distributed caching in related technologies, this approach utilizes inexpensive disk resources to solve the data caching problem. Furthermore, the disk cache has a large data capacity, enabling caching of large amounts of data and concurrent data queries.

[0098] The data storage method applied to the memory provided in the embodiment of the present application is as follows: Figure 4 As shown, the method includes:

[0099] S401: Obtaining record information, wherein the record information is information recording indication information and a storage location of data stored in the disk, wherein the indication information is used to indicate a data screening condition for screening out the data to be stored and a ranking of the data to be stored under the data screening condition;

[0100] S402: Storing the record information, wherein the record information is used to query data stored in the disk.

[0101] In S401-S402, the memory can read the record information from the disk and store it in itself. Alternatively, the memory can record the corresponding indication information and storage location assigned by the disk to each data cached on the disk to form the record information and store it.

[0102] Unlike the solutions in related technologies that require caching the queried data into memory, the memory in the embodiments of the present application no longer caches the queried data, but only needs to record or store the record information. This can reduce the burden on the memory and effectively avoid the problem of frequent memory replacement due to data caching in the memory (equivalent to memory-level data caching).

[0103] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0104] In this application scenario, Figure 5 As an example of the interactive scenario shown in Figure 5 The interactive scenario shown includes an operation end, a business system, a memory, a disk, and a source data storage layer. Among them, the operation end allows users to input information to be queried; the business system can query the data that the user wants to query based on the memory record or the stored record information. The memory is used to store the record information. The disk is used to cache the data that the user wants to query. The source data storage layer can be used to store source data, and its function is similar to the aforementioned database. All data in the source data storage layer or part of the data that needs to be queried can be cached locally in the business system to achieve disk-level local caching. In one application scenario, the business system can be implemented based on the AOP (Aspect Oriented Programming) method, and the functions it performs are the same as those of the aforementioned data query device. Those skilled in the art should know that there are several notification types in AOP technology, and two notification types, pre-notification and post-notification, will be used in the embodiment of this application. Among them, pre-notification is used to execute before the target method is executed; post-notification is used to execute after the target method is executed. In this application scenario, the target method Figure 5 The encapsulated query distributed computing API (Application Programming Interface) or open source data management library (Client House) API method in Figure 5As shown, the data query method of the embodiment of the present application is executed before the target execution method, which does not affect the normal execution of the target method. In addition, the present application also provides a solution for merging the stored data in the disk. This merging solution can effectively save disk occupancy on the one hand, and improve the efficiency of concurrent queries on the other hand. The merging solution is executed after the target solution is executed, and does not affect the execution of the target method. Among them, the target method is not the focus of this solution. The above description is to describe that the data query method and data merging method provided by this application will not affect the target method. Among them, please refer to the relevant technology for details of AOP, pre-notification and post-notification, which will not be described in detail in this article.

[0105] Combine Figure 5 For the interactive scenario shown, the relevant execution process of the data query method and storage method of the embodiment of the present application can be specifically described below.

[0106] In this application scenario, the source data storage layer stores a large amount of data, from which the disk can read the data locally and store it in the form of an LSM tree. Data that falls under the same data filtering criteria can be stored in the same tree or in different trees. On each tree, data that is expected to be stored in the same tree under the same data filtering criteria can be stored according to a certain ranking. For example, the data of the top five 3C products with the best reviews in the previous quarter stored in the source data storage layer are stored in the same tree according to the ranking of the best reviews. For example, the disk can read the data of the top five 3C products with the best reviews in the previous quarter from the source data storage layer, such as brand A laptops, brand B mobile phones, brand C all-in-one computers, etc. A new tree, such as Tree 1, is created on the disk and a tree address, such as 0X01, is assigned to the tree. The new tree creates node positions with the same value as the number of data read, such as 5 node positions, and the data of the top five 3C products with the best reviews are stored or recorded in each node position in sequence. For example, node position 1 is used to store the data for the product with the highest positive reviews, such as brand A laptops, and node position 2 is used to store the data for the product with the second highest positive reviews, such as brand B mobile phones. It can be understood that in this application scenario, the tree address plus the position of the node containing the product data in the tree constitutes the storage address of the product data on disk. This achieves local storage of product data on disk in the form of an LSM tree. It can be understood that in this application scenario, data that meets or falls under the same data filtering criteria is stored in the same tree to facilitate parallel or concurrent data queries (simultaneously querying multiple product data, such as querying the top five 3C products with the highest positive reviews in the previous quarter). By reading the data for multiple products to be queried from the same tree, data query time can be greatly shortened and query efficiency can be improved. For example, querying the top five 3C products with the highest positive reviews in the previous quarter. In this application scenario, an LSM tree is used to store data that may be queried by users in the source data storage layer. During storage, data that meets or falls under the same data filtering criteria is stored according to a certain ranking, which can significantly reduce query time and improve query efficiency. Furthermore, data from the source data storage layer is cached or stored on local disk, achieving disk-level local caching. Disks have a larger storage capacity and can store more data than memory caches. Furthermore, disks are less expensive than memory, significantly reducing costs.

[0107] In a specific implementation, the tree address of Tree 1 and the ranking information of each product data stored at each node position in Tree 1 among similar products, such as 3C products, can be stored or recorded in a corresponding manner on the disk to form record information. The memory reads the recorded information from the disk. It can be understood that the record information on the disk can be flexibly updated according to the actual storage situation of the data. When the record information on the disk is updated, the memory needs to re-read the record information. In addition, considering that each tree corresponds to a data screening condition, such as the data screening condition of Tree 1 is 3C products with good reviews, the node position in the tree corresponds to the ranking information of the stored products. Based on this, a key value can be set for each tree, such as Tree 1, on a technical level. The key value represents the data screening condition corresponding to the tree. The key value, the tree address of Tree 1, and the ranking information of the product data stored at each node position in Tree 1 are stored or recorded in a corresponding manner to form record information. Among them, since the data is stored in the form of a tree on the disk, and the data is stored in a certain ranking order in each tree, the order of the node positions represents the ranking order to a certain extent, and the Key value is the information used to represent the data filtering conditions. Then, the correspondence between the Key value in the record information and its corresponding node position represents to a certain extent the data filtering conditions that can be used to filter out the data stored in Tree1 and the ranking information of the data under the data filtering conditions. The memory reads the record information from the disk. In this application scenario, the memory does not need to cache or store the product data itself, but only needs to store the record information, which reduces the burden on the memory. The product data itself is cached or stored on the disk, realizing local caching at the disk level. The Key value in this application scenario is to determine the storage address (tree address) of the tree where the expected query data is located on the disk based on the Key to be queried in the subsequent data query solution, and then find the expected query data based on the tree address.

[0108] The following combination Figure 5 and Figure 6 The data query process is described as follows.

[0109] On the displayed operational query page, the user enters the information to be queried, such as data filtering criteria (positive reviews of 3C products) and product data ranked (ranked 1st to 5th) under these criteria. The business system receives the user's query information and implements the data query using AOP technology. The business system identifies the user's entered data filtering criteria and, based on the identified result "positive reviews of 3C products," generates a key value corresponding to this result. It then determines whether a record containing the key is stored in memory. If not, it further determines whether a record containing the key is stored on disk. If so, the record is loaded or read into memory using a memory-mapped file for subsequent use. If no record containing the key is stored on disk, the source data storage layer's API is called to read the data that meets the entered data filtering criteria and ranking information to disk. The data is then stored in a tree format, resulting in a record containing the data filtering criteria and ranking information. For a description of this process, please refer to the aforementioned instructions; any repetitions are omitted. Among them, the interactive read or write cache between the disk and the memory is realized in the form of Memory Mapped Files, which can speed up the reading or writing speed, save query time and improve query efficiency.

[0110] When it is determined that the memory stores record information including the KEY, determine whether the product data to be queried is cached on the disk, that is, whether there is ranking information corresponding to the KEY in the record information. If there is ranking information corresponding to the KEY, it means that the top 5 product data of 3C products have been cached on the disk. If there is no ranking information corresponding to the KEY value, it means that the top 5 product data have not been cached on the disk. Or if one ranking information is missing, such as the 5th ranking, it means that the product data ranked 5 has not been cached on the disk, that is, the product data ranked 5 is missing. In this case, it is necessary to call the API of the source data storage layer to read the data that meets the input data filtering conditions and the missing ranking to the disk, and store it in the form of a tree. The record information is updated, and the position of the missing node ranked 5 is updated to the original record information. For a description of this process, please refer to the above-mentioned relevant instructions, and the repeated parts will not be repeated.

[0111] When the desired product data is cached on disk, the business system determines the tree address corresponding to the KEY value in the record information based on the record information cached in memory, that is, finds the storage location of the tree of the desired data on disk based on the KEY. The business system finds the tree with the storage location, traverses the tree, and finds the node position in the tree corresponding to the ranking information entered by the user, such as the 1st to 5th node positions in the tree. The node data stored at this node position is the data the user desires to query. The business system transmits it to the operation end and displays it on the operation end query page for the user to view.

[0112] In this application scenario, the record information obtained for a tree can be called a metadata file; the record information for all trees recorded in the memory is called a metadata collection object, that is, the metadata collection object records the metadata files of all trees stored on the disk; the record information stored in the memory can be a metadata collection object.

[0113] In a practical application, there may be two trees on disk. One tree is used to store data for products ranked 1st to 5th in the 3C category. That is, nodes 1 to 5 in this tree are used to store products ranked 1st to 5th in terms of positive reviews. The other tree is used to store data for products ranked 6th to 10th in terms of positive reviews. For two or more trees stored on disk that meet the same data filtering criteria, these trees can be periodically merged. This not only saves disk space but also improves query efficiency. A specific merging solution might be to create five nodes for the first tree based on the original nodes (nodes 1 to 5 of the first tree, which store data for products ranked 1st to 5th). The data for products ranked 6th to 10th in terms of positive reviews stored in the second tree is then stored in these five newly created nodes. Alternatively, you can create a new tree and create 10 nodes for the tree, and store the product data ranked 1st to 10th in these 10 nodes. In this application scenario, the data query solution can be completed by one thread, and the data merging solution can be completed by another thread. The two do not affect each other, so this solution is called an asynchronous process. On the technical level, when the pre-set merging cycle for implementing the data merging solution, such as 48 hours, arrives, the business system reads the metadata collection object cached in the memory, traverses the metadata collection object, and merges the trees with the same KEY value stored on the disk in a zero-copy manner, which is equivalent to merging product data with certain identical characteristics, such as merging the top 10 product data in the 3C positive review category. Update the metadata file and metadata object collection. For subsequent query use. The reason why merging trees in the embodiment of the present application can improve query efficiency is that: compared to before merging, if a user wants to query the top 10 product data of 3C products with good reviews, it is necessary to query two trees, but after merging, only one tree needs to be queried, which can effectively shorten the query time and improve query efficiency. Among them, before merging the trees with the same KEY value stored on the disk in a zero-copy manner, it is necessary to read the trees from the disk and then merge them. This reading and writing method can be regarded as a solution for reading and writing data from disk to disk in a Zero-Copy manner. In addition, when the data that the user expects to query is queried on the disk, the business system also uses the Zero-Copy method to read the data expected to be queried and display it. This solution can be regarded as a solution for reading from disk to display page in a Zero-Copy manner. In this application scenario, the use of the Zero-Copy method to read and write data from disk to disk and from disk to display page can effectively shorten the data reading and writing speed and speed up query efficiency.

[0114] The above implementation logic in this application scenario can be implemented using AOP code. The AOP code in this application scenario implements disk-memory interactive read / write caching using memory-mapped files, implements disk-to-network data reading and disk-to-disk data reading and writing using zero-copy, and implements periodic data merging using the concept of LSM trees, improving the efficiency of subsequent data queries and achieving efficient local caching of large amounts of data on disk.

[0115] from Figure 5 As can be seen in the figure, the user enters data filtering criteria and the ranking information of the data they wish to query. Using a pre-notification, the LSM tree address corresponding to the key is retrieved from the metadata collection object stored in memory. Data is then sequentially read from the tree with that address using zero-copy. If no cache exists on disk, data is retrieved from the data source and a new LSM tree is created using a memory-mapped file. During the merge cycle, an asynchronous thread merges all LSM trees corresponding to the same key using zero-copy, improving query efficiency.

[0116] In this application scenario, the data to be queried is cached or stored on disk, achieving disk-level caching. Combined with zero-copy and memory-mapped file technologies, this can speed up caching or data queries and improve efficiency. The memory in this application scenario no longer caches the queried data; it only records or stores the recorded information. This reduces the burden on the memory and effectively avoids the problem of frequent memory replacement that may result from data caching in memory (equivalent to memory-level data caching). Furthermore, because the disk space is larger than the memory cache space, the amount of disk cached data is larger, which can support large amounts of parallel data queries. Furthermore, disks are cheaper than memory, effectively saving expenditure costs.

[0117] The present application embodiment provides a data query device, such as Figure 7 As shown, it includes: an obtaining unit 701, a first determining unit 702, a reading unit 703 and a second determining unit 704; wherein,

[0118] An obtaining unit 701 is configured to obtain information to be queried, wherein the information to be queried is used to indicate a data screening condition and ranking information under the data screening condition;

[0119] A first determining unit 702 is configured to determine, based on the record information in the memory, a storage address in the disk of data that meets the data screening condition and ranking information;

[0120] The reading unit 703 is configured to read the data from the storage address of the disk;

[0121] The second determining unit 704 is configured to use the read data as data expected to be queried under the information to be queried.

[0122] In an optional solution, the first determining unit 702 is configured to determine, based on the record information in the memory, a tree in which the data meeting the data screening condition and ranking information is located; and determine a tree address of the tree in the disk;

[0123] Correspondingly, the reading unit 703 is configured to traverse the tree in the disk based on the tree address to obtain a node position in the tree for storing the data of the ranking information; and read the data stored at the node position;

[0124] The second determining unit 704 is configured to use the read data as data expected to be queried under the information to be queried.

[0125] In an optional solution, the device further includes a judgment unit, configured to: judge whether record information of data meeting the data screening condition and ranking information is stored in the memory;

[0126] If the determination unit determines that the result is yes, the first determination unit 702 is triggered to determine the tree where the data that meets the data screening condition and ranking information is located according to the record information in the memory.

[0127] If the judgment unit judges as no, the creation unit of the query device is used to obtain data that meets the data screening conditions and the ranking information from the database; create a tree for the obtained data on the disk; assign a tree address to the tree, and assign a node position in the tree to the data.

[0128] In an optional solution, the device further includes a merging unit, configured to:

[0129] Obtain all data stored in the disk that meets the data screening conditions;

[0130] Obtaining attributes of each of the data;

[0131] Combine all data according to the properties.

[0132] In an optional solution, when all data that meets the data screening condition is located in at least two trees, the at least two trees are merged into one tree.

[0133] It can be understood that in actual applications, the obtaining unit 701, the first determining unit 702, the reading unit 703 and the second determining unit 704 in the device can all be implemented by the central processing unit (CPU), digital signal processor (DSP), microcontroller unit (MCU) or programmable gate array (FPGA) of the query device.

[0134] The embodiment of the present application provides a disk, such as Figure 8 As shown, it includes: a storage unit 801 and a recording unit 802; wherein,

[0135] The storage unit 801 is used to store data to be stored; the stored data to be stored has at least indication information, the indication information is used to indicate the data screening conditions for screening out the data to be stored and the ranking of the data to be stored under the data screening conditions;

[0136] The recording unit 802 is configured to record the indication information and the storage location of the data to be stored in the disk to obtain recording information; the recording information is used to query the data to be stored.

[0137] The embodiment of the present application provides a memory, such as Figure 9 As shown, it includes: an obtaining unit 901 and a storage unit 902; wherein,

[0138] An obtaining unit 901 is configured to obtain record information, wherein the record information is information recording indication information and a storage location of data stored on the disk, wherein the indication information is used to indicate a data screening condition for screening out the data to be stored and a ranking of the data to be stored under the data screening condition;

[0139] The storage unit 902 is used to store the record information, wherein the record information is used to query the data stored in the disk.

[0140] It should be noted that the relevant devices of the embodiments of the present application, such as data query devices, disks, and memory, solve problems based on principles similar to those of the aforementioned query methods and storage methods. Therefore, the implementation process and implementation principles of these devices can be found in the description of the implementation process and implementation principles of the aforementioned query methods and storage methods, and the repeated parts will not be repeated. Among them, the data query device has the same execution function as the aforementioned business system. For an understanding of the execution function of the data query device, please refer to the understanding of the execution function of the business system, and no specific limitations will be given for the repeated parts.

[0141] The present application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is characterized in that when the program is executed by a processor, it is used to perform at least Figures 1 to 6 The steps of any of the methods shown. The computer readable storage medium may be a memory. The memory may be Figure 10 The memory 62 is shown.

[0142] Figure 10 This is a hardware structure diagram of the data query device according to an embodiment of the present application. Figure 10 As shown, the data query device includes: a communication component 63 for data transmission, at least one processor 61 and a memory 62 for storing computer programs that can be run on the processor 61. The various components in the terminal are coupled together through a bus system 64. It can be understood that the bus system 64 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 64 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 10 Various buses are labeled as bus system 64.

[0143] Wherein, when the processor 61 executes the computer program, it at least performs Figures 1 to 6 Any of the steps of the method shown.

[0144] It is understood that the memory 62 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk or a magnetic tape. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory 62 described in the embodiments of the present application is intended to include, but is not limited to, these and any other suitable types of memory.

[0145] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 61. Processor 61 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in processor 61 or instructions in software form. The above processor 61 may be a general-purpose processor, a DSP, or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc. Processor 61 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium located in memory 62. Processor 61 reads the information in memory 62 and, in conjunction with its hardware, completes the steps of the above method.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0147] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0149] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0150] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0151] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0152] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0153] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0154] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A data query method, characterized in that: include: Obtaining information to be queried, the information to be queried being used to indicate a data screening condition and ranking information under the data screening condition; Determining, based on the recorded information in the memory, the storage address of the data that meets the data screening condition and ranking information on the disk; wherein different data are stored in the form of a tree at different storage addresses on the same disk; different storage addresses correspond to different trees; the recorded information is obtained by recording the data screening condition of the data to be stored, the ranking of the data to be stored under the data screening condition, and the tree address of the tree where the data to be stored is located; Reading the data from the storage address of the disk; The read data is used as the data expected to be queried under the information to be queried; The method further comprises: For at least two trees stored in the disk and belonging to the same data screening condition, the at least two trees are merged through an asynchronous thread.

2. The method according to claim 1, characterized in that The data is stored in the disk as data at a node position in a tree, and the tree stored in the disk has a tree address in the disk; Correspondingly, the storage address of the data in the disk that meets the data screening conditions and ranking information is determined based on the record information in the memory; Reading the data from the storage address of the disk and using the read data as expected data to be queried under the information to be queried includes: Determine the tree where the data that meets the data screening conditions and ranking information is located based on the record information in the memory; Determine the tree address of the tree on disk; Based on the tree address, the tree in the disk is traversed to determine the node position in the tree for storing the data of the ranking information; and the data stored at the node position is read; The read data is used as the data expected to be queried under the information to be queried.

3. The method according to claim 2, characterized in that The method further comprises: Determining whether the memory stores record information of data that meets the data screening condition and ranking information; If the judgment is yes, the tree where the data that meets the data screening condition and ranking information is located is determined based on the record information in the memory.

4. The method according to claim 3, characterized in that The method further comprises: If the judgment is no, Obtaining data that meets the data screening conditions and the ranking information from a database; Create a tree on disk for the obtained data; The tree is assigned a tree address, and the data is assigned a node position in the tree.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Obtain all data stored in the disk that meets the data screening conditions; Obtaining an attribute of each of the data, wherein the attribute represents a ranking of each of the data under the data screening condition; Combine all data according to the properties.

6. The method according to claim 5, characterized in that All data that meet the data screening conditions are located in at least two trees; Accordingly, all data are merged according to the attributes, including: According to attributes of all data stored in the at least two trees, the at least two trees are merged into one tree.

7. A data storage method, applied to a disk, characterized in that: include: Obtaining data to be stored; Storing the data to be stored; The stored data to be stored at least has indication information, where the indication information is used to indicate a data screening condition for screening out the data to be stored and a ranking of the data to be stored under the data screening condition; The indication information and the tree address of the tree where the data to be stored is located are recorded to obtain recording information; the recording information is used to determine the storage address of the data that meets the data screening condition and the ranking information on the disk; different data are stored in the form of trees at different storage addresses on the same disk; different storage addresses correspond to different trees; The data screening condition is used to determine at least two trees belonging to the same data screening condition from the different trees, so that the at least two trees are merged through an asynchronous thread.

8. A data storage method, applied to a memory, characterized in that: include: Obtaining record information, wherein the record information is information recording indication information and a tree address of a tree where data stored in the disk is located, wherein the indication information is used to indicate a data screening condition for screening out data to be stored and a ranking of the data to be stored under the data screening condition; Storing the record information, wherein the record information is used to determine the storage address of the data that meets the data screening condition and the ranking information on the disk; different data are stored in different storage addresses on the same disk in the form of a tree; different storage addresses correspond to different trees; The data screening condition is used to determine at least two trees belonging to the same data screening condition from the different trees, so that the at least two trees are merged through an asynchronous thread.

9. A data query device, characterized in that: include: an obtaining unit, configured to obtain information to be queried, wherein the information to be queried is used to indicate a data screening condition and ranking information under the data screening condition; A first determining unit is configured to determine, based on the recorded information in the memory, a storage address on the disk of data that meets the data screening condition and ranking information; wherein different data are stored in a tree format at different storage addresses on the same disk; different storage addresses correspond to different trees; and the recorded information is obtained by recording the data screening condition for the data to be stored, the ranking of the data to be stored under the data screening condition, and the tree address of the tree where the data to be stored is located; A reading unit, configured to read the data from the storage address of the disk; A second determining unit is configured to use the read data as data expected to be queried under the information to be queried; The second determining unit is further configured to merge at least two trees stored in the disk and belonging to the same data screening condition through an asynchronous thread.

10. A magnetic disk, characterized in that: include: A storage unit, used for storing data to be stored; The stored data to be stored at least has indication information, where the indication information is used to indicate a data screening condition for screening out the data to be stored and a ranking of the data to be stored under the data screening condition; a recording unit, configured to record the indication information and the tree address of the tree where the data to be stored is located, to obtain recording information; the recording information is used to determine the storage address of the data that meets the data screening condition and the ranking information on the disk; different data are stored in the form of trees at different storage addresses on the same disk; different storage addresses correspond to different trees; The data screening condition is used to determine at least two trees belonging to the same data screening condition from the different trees, so that the at least two trees are merged through an asynchronous thread.

11. A memory, characterized in that: include: an obtaining unit, configured to obtain record information, wherein the record information is information recording the indication information and the tree address of the tree where the data stored in the disk is located, wherein the indication information is used to indicate a data screening condition for screening out the data to be stored and a ranking of the data to be stored under the data screening condition; a storage unit for storing the record information, wherein the record information is used to determine the storage address of the data that meets the data screening condition and the ranking information on the disk; different data are stored in different storage addresses on the same disk in the form of a tree; different storage addresses correspond to different trees; The data screening condition is used to determine at least two trees belonging to the same data screening condition from the different trees, so that the at least two trees are merged through an asynchronous thread.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images