Data query method, device and equipment based on multi-level medium and storage medium

Abstract

The invention belongs to the field of artificial intelligence big data, and relates to a multi-level medium-based data query method, which comprises the following steps of: determining an LSM level position where a target business database is located based on a data query request; based on the LSM hierarchical position information, extracting corresponding hierarchical medium key information, and adding a medium statistical information histogram to calculate the data reading cost of the current data query request; and selecting an optimal data query path according to the data reading cost. The invention further provides a data query device and equipment based on the multi-level medium and a storage medium. In addition, the invention also relates to a block chain technology, and the data requested to be queried can be stored in a block chain. The method can be applied to a data query scene in the medical field, the LSM hierarchical medium key information is added when the statistical information isodepth histogram is used for calculating the data query cost, so that the cost estimation of database physical optimization is more accurate, the accuracy of a data query cost calculation result is improved, and then the data query efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and big data technology, and in particular to a data query method, device, equipment and storage medium based on multi-level media. Background Art

[0002] With the rise of artificial intelligence and big data technologies, data query has become a channel for users to quickly obtain the information they need in many scenarios. For example, in the medical field, the medical record information required by users can be queried from massive electronic medical records based on artificial intelligence models, which helps provide users with medical record references. In addition, the reading and management of massive case data can support functions such as disease auxiliary diagnosis, health analysis, and remote consultation.

[0003] In the era of big data, in order to improve data storage capabilities, the LSM hierarchical structure will be used to efficiently manage storage data. The first few levels of LSM hierarchical storage are high-speed media, and the last level L2 is low-speed media. Among them, high-speed media include NVME, and low-speed media include HDD, etc. The cost of the system can be reduced through mixed grading. In the LSM hierarchical storage structure, since the data is scattered in multiple media, the read operation may involve multiple disk accesses. When querying the database, there may be multiple query paths, so the optimal path with the minimum cost (that is, the shortest reading time) should be calculated based on statistical information to obtain the best performance.

[0004] In the traditional reading cost calculation method, the statistical information iso-depth histogram is usually used. The iso-depth histogram provides relevant information about the data distribution. The number of buckets can be calculated according to the scope of the data query, and then multiplied by the bucket depth to obtain the number of data items expected to be obtained, and then the reading cost can be calculated. However, the traditional information statistical histogram only contains information about the bucket and the number, that is, the traditional calculation method does not consider the medium where the data is distributed, and can only use the reading time of one medium. Due to the different media, the reading speed may differ by dozens of times, so there will be a large error in the cost estimation. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a data query method, device, equipment and storage medium based on multi-level media to solve the problem that traditional statistical histograms cannot flexibly calculate media-related information, resulting in inaccurate cost calculation results and inability to obtain the optimal query path.

[0006] In order to solve the above technical problems, the embodiment of the present application provides a data query method based on multi-level media, which adopts the following technical solution:

[0007] Determine the LSM hierarchical location of the target business database based on the data query request;

[0008] Based on the LSM hierarchical position, the corresponding hierarchical medium key information is extracted, and the medium statistical information histogram is added to comprehensively calculate the data reading cost of the current data query request;

[0009] An optimal data query path is selected according to the data reading cost, and a data query request is executed based on the optimal data query path.

[0010] Furthermore, before the step of determining the LSM hierarchical location of the target business database based on the data query request, the step further includes:

[0011] Parsing the data query request, where the parsed content includes at least the access mode and keywords of the query request;

[0012] An index structure is established based on the acquired access pattern and keyword structure, and a target business database is determined through the index structure.

[0013] Furthermore, the step of determining the LSM hierarchical location of the target business database based on the data query request specifically includes:

[0014] Retrieving the log file of the target business database according to the data query request;

[0015] Obtain basic characteristic information of data maintenance and management in the target business database from the log file;

[0016] The basic feature information is mapped to the hierarchical structure of the LSM tree according to the statistical information and / or index of the target business database to determine the LSM level at which the target business database associated with the current data query request is located.

[0017] Furthermore, the hierarchical media key information includes: data block attributes, hierarchical attributes, media statistical information histogram, and media read time. The step of adding the media statistical information histogram to comprehensively calculate the data read cost of the current data query request specifically includes:

[0018] Based on the LSM level where the target business database is located, extracting data block attributes, level attributes, and read time of the level medium of the LSM level, wherein the data block attributes include data block size, and the level attributes include the number of levels and IO performance corresponding to the number of levels;

[0019] Dynamically adjusting the read time of the hierarchical medium according to the data block size of the LSM level, the number of levels and the IO performance corresponding to the number of levels, and adding the adjusted read time of the hierarchical medium to the medium statistical information histogram to optimize the medium statistical information histogram;

[0020] Calculating the expected number of data items based on the optimized medium statistical information histogram, and obtaining information on the number of data items and corresponding storage interval information in the medium statistical information histogram;

[0021] The data reading cost is calculated based on the data number information, the storage interval information and the reading time of the hierarchical medium.

[0022] Furthermore, the step of calculating the expected number of data items based on the optimized medium statistical information histogram specifically includes:

[0023] Acquire data distribution characteristics of the target service database in the LSM level based on the medium statistical information histogram, and determine a data query range according to the data distribution characteristics;

[0024] Calculating the number of buckets of the medium statistical information histogram according to the data query range;

[0025] Based on the bucket number information and the bucket depth corresponding to the bucket, the expected number of data items to be obtained is calculated.

[0026] Furthermore, the calculation formula of the data reading cost is:

[0027] cost = n(H)*t(H)+n(L)*t(L);

[0028] Where n(H) represents the number of data items on the high-speed medium, n(L) represents the number of data items on the low-speed medium, t(H) represents the read time of the high-speed medium, and t(L) represents the read time of the low-speed medium.

[0029] In order to solve the above technical problems, the embodiment of the present application also provides a data query device based on multi-level media, which adopts the following technical solution:

[0030] An acquisition module is used to determine the LSM hierarchical location of the target business database based on a data query request;

[0031] A calculation module, for extracting corresponding hierarchical medium key information based on the LSM hierarchical position, and adding a medium statistical information histogram to comprehensively calculate the data reading cost of the current data query request;

[0032] An execution module is used to select an optimal data query path according to the data reading cost, and execute a data query request based on the optimal data query path.

[0033] Furthermore, the device also includes:

[0034] A parsing module, used to parse the request content of the data query request, where the parsed content at least includes the access mode and keywords of the query request;

[0035] The index module is used to establish an index structure based on the obtained access mode and keyword structure, and determine the target business database through the index structure.

[0036] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0037] Determine the LSM hierarchical location of the target business database based on the data query request;

[0038] Based on the LSM hierarchical position, the corresponding hierarchical medium key information is extracted, and the medium statistical information histogram is added to comprehensively calculate the data reading cost of the current data query request;

[0039] An optimal data query path is selected according to the data reading cost, and a data query request is executed based on the optimal data query path.

[0040] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0041] Determine the LSM hierarchical location of the target business database based on the data query request;

[0042] Based on the LSM hierarchical position, the corresponding hierarchical medium key information is extracted, and the medium statistical information histogram is added to comprehensively calculate the data reading cost of the current data query request;

[0043] An optimal data query path is selected according to the data reading cost, and a data query request is executed based on the optimal data query path.

[0044] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0045] The embodiment of the present application first determines the LSM hierarchical position of the target business database according to the user's data query request, then extracts the corresponding hierarchical media key information based on the LSM hierarchical position, adds the media statistical information histogram to comprehensively calculate the data reading cost of the current data query request, and finally selects the optimal data query path according to the data reading cost, and executes the data query request based on the optimal data query path. The embodiment of the present application helps to identify features such as query services by parsing the content in the query request and extracting key information from the selected LSM level to calculate the reading cost, so that according to these features, it can be mapped to the hierarchical structure of the LSM tree, and can quickly locate the main LSM level that may be involved in the query, determine the location where the data may be stored, avoid data search at unnecessary levels, reduce unnecessary hierarchical scanning and invalid queries, so that the reading cost can be calculated more efficiently and accurately, and the problem of large cost estimation errors caused by large differences in reading speeds due to different storage media is solved, and the reading cost calculation accuracy and reliability are significantly improved; at the same time, the optimal data query path is selected through the data reading cost to ensure the optimal query performance and significantly improve the query response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0048] Figure 2 A flowchart of an embodiment of a data query method based on multi-level media according to the present application;

[0049] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S201;

[0050] Figure 4 yes Figure 2 A flowchart of a specific implementation of step S202;

[0051] Figure 5 It is a schematic diagram of the histogram of statistical information of traditional computing query data;

[0052] Figure 6 It is a media statistical information histogram according to a specific implementation of the data query method based on multi-level media of the present application;

[0053] Figure 7 is a structural schematic diagram of an embodiment of a data query device based on multi-level media according to the present application;

[0054] Figure 8 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0056] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0057] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0058] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0059] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0060] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0061] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0062] It should be noted that the data query method based on multi-level media provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the data query device based on multi-level media is generally set in the server / terminal device.

[0063] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0064] Continue to refer Figure 2 , shows a flow chart of an embodiment of a data query method based on multi-level media according to the present application. According to different requirements, the order of the steps in the flow chart can be changed. The data query method based on the optimized statistical information histogram provided by the embodiment of the present application can be applied to any scenario where the optimal query path for data needs to be determined and used. The data query method based on the optimized statistical information histogram includes the following steps:

[0065] Step S201: Determine the LSM hierarchical location of the target business database based on the data query request.

[0066] In this embodiment, the electronic device on which the data query method based on the optimized statistical information histogram runs can receive the data query request sent by the user terminal through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wide band) connection, and other wireless connection methods currently known or to be developed in the future.

[0067] Specifically, before step S201, the method further includes:

[0068] Parsing the data query request, where the parsed content includes at least the access mode and keywords of the query request;

[0069] An index structure is established based on the acquired access pattern and keyword structure, and a target business database is determined through the index structure.

[0070] In one possible implementation, the method provided in the present application can be used in the medical field, and the data queried by the data query request can be medical data, such as personal health records, prescriptions, examination reports and other data.

[0071] In this embodiment, the query request includes the conditions for this query, such as the access mode and keywords of the query request. Among them, the access mode includes client access to the server, browser access to the server, peer-to-peer network access mode, etc. Keywords include key information parsed from the request sent by the user. For example, when medical staff retrieves the patient's medical information, the input keywords are the user's name, ID number, mobile phone number, etc. Different access modes and input keywords will differ in the indexing method. Therefore, by parsing the query request and obtaining detailed information on the access mode and keywords, it is helpful to establish a targeted index structure based on these detailed information, and further search and lock the target business database based on the index structure.

[0072] like Figure 3 As shown, specifically, step S201 includes:

[0073] Step S2011, retrieving the log file of the target business database according to the data query request;

[0074] Step S2012, obtaining basic characteristic information of data maintenance and management in the target business database from the log file;

[0075] Step S2013, mapping the basic feature information to the hierarchical structure of the LSM tree according to the statistical information and / or index of the target business database to determine the LSM level where the target business database associated with the current data query request is located.

[0076] In this embodiment, by retrieving the log file of the target business database, detailed information such as data writing and merging, data type, access frequency and data volume in the target business database can be obtained according to the log file of the target business database, and this information is helpful to understand the storage level of the data. Therefore, the statistical information or / index of the target business database can be used to map this information to the hierarchical structure of the LSM tree, so as to determine in which medium of the LSM level the target business database is distributed, and then obtain comprehensive media information to calculate a more accurate reading cost.

[0077] In this embodiment, determining the LSM hierarchical position of the target business database based on the data query request can specifically be to obtain the basic feature information of data maintenance and management in the target business database through the log file of the target business database determined according to the data query request, and use the statistical information or / and index of the target business database to map the basic feature information to the hierarchical structure of the LSM tree to determine the LSM level where the target business database associated with the current query task is located. Among them, the LSM tree is usually designed as a multi-layer structure, and the data compression degree and merging frequency of each layer are different, which affects the query efficiency. Therefore, using the statistical information about data distribution and access frequency in the database, such as the type of data, access frequency and data volume of the query, the level that the query will mainly involve can be estimated. According to these characteristics, the hierarchical structure of the LSM tree is mapped to determine the location where the data may be stored, such as different levels such as the memory table (MemTable) and SSTable, and quickly locate the main LSM level that the query may involve, reduce unnecessary level scanning, avoid data search at unnecessary levels, and significantly improve the query response speed. Using the statistical information about data distribution and access frequency in the database, the data volume, key value range, etc. of each level SSTable are understood. In addition, if the LSM tree or the database system on it supports indexes, such as Bloom filters, hash indexes, etc., the indexes can be used to quickly locate the possible levels and locations of data. Combining query conditions, statistical information, and indexes, it is possible to estimate the LSM levels that the query may involve.

[0078] It should be noted that, since the data in the LSM tree is merged in sequence, the newer data is usually located at the top level, while the older data is located at the bottom level. If the data range involved in the query spans multiple levels, it is necessary to consider the impact of the merge and compression operations between these levels on the query performance. To this end, this embodiment provides an implementation method: a strategy for optimizing queries can be formulated, such as selecting a suitable query path, using cache to reduce disk I / O operations, etc., while monitoring query performance and adjusting the optimization strategy according to actual conditions. In this way, even if the actual location of the data changes as the data is continuously written and merged, an optimization strategy can be adopted to adjust the determination of the hierarchical location to further improve the query efficiency. For example, real-time monitoring of which data is frequently updated and queried, dynamically adjusting the hierarchical structure of the LSM tree according to the frequency of data updates, such as independently storing frequently updated data in a high level with fewer levels to reduce merging overhead, or storing less accessed historical data in a low level with more levels to save high-level space. Through the comprehensive application of the above strategies, the LSM database can better adapt to application scenarios with frequent data updates, and significantly improve query performance while ensuring data consistency.

[0079] Step S202: based on the LSM hierarchical position, extract the corresponding hierarchical medium key information, add the medium statistical information histogram to comprehensively calculate the data reading cost of the current data query request.

[0080] The key information of the hierarchical media includes: data block attributes, hierarchical attributes, media statistics histogram and media read time.

[0081] like Figure 4 As shown, specifically, step S202 includes:

[0082] Step S2021, based on the LSM level where the target business database is located, extracting data block attributes, level attributes, and read time of the level medium of the LSM level, wherein the data block attributes include data block size, and the level attributes include the number of levels and IO performance corresponding to the number of levels;

[0083] Step S2022, dynamically adjusting the read time of the hierarchical medium according to the data block size of the LSM level, the number of levels and the IO performance corresponding to the number of levels, and adding the adjusted read time of the hierarchical medium to the medium statistical information histogram to optimize the medium statistical information histogram;

[0084] Step S2023, calculating the expected number of data items based on the optimized medium statistical information histogram, and obtaining the number of data items and the corresponding storage interval information in the medium statistical information histogram;

[0085] Step S2024: Calculate the data read cost based on the data number information, storage interval information, and the read time of the hierarchical medium.

[0086] In this embodiment, the LSM hierarchical structure may include multiple levels, and different levels may be different hierarchical media. Different storage media correspond to different IO performances. For example, the LSM hierarchical structure includes 3 levels, including 1 high-speed medium and 1 low-speed medium. Each layer may contain a certain number of data blocks. When the amount of data exceeds a certain threshold, a merge operation will be triggered to merge the data blocks into the next layer. Therefore, it is necessary to extract data block attributes to understand the storage status of the corresponding data blocks in the LSM layer of the data in the target business database. Among them, the data block attributes include the data block size, and may also include the data block location, compression rate, etc. Among them, the tools or APIs provided by the database management system can be used to extract the key information of the hierarchical media of the LSM tree layer. For example, for a NoSQL database such as HBase, the status and performance data of the LSM tree can be obtained through its management interface or command line tool.

[0087] Furthermore, by extracting the hierarchical attributes and data block attributes, a dynamic adjustment mechanism for hierarchical media read time can be established. When the hierarchical attributes are good, the I / O delay is short and the CPU decompression time is short; the data block is large and stored in high-speed media with a high compression rate, the read time of the corresponding hierarchical media can be optimized for these data, and then the impact of the media read time on the number of buckets is considered in the media statistical information histogram, so that the data reading efficiency is improved.

[0088] In this embodiment, the above-mentioned optimization is achieved by dynamically adjusting the medium read time by combining data block attributes and level attributes and then adding the medium statistical information histogram. The cost estimation method that combines level attributes and data block attributes is more conducive to ensuring the accuracy and reliability of read cost calculation.

[0089] Specifically, in step S2023, the step of calculating the expected number of data items based on the optimized medium statistical information histogram specifically includes:

[0090] Acquire data distribution characteristics of the target service database in the LSM level based on the medium statistical information histogram, and determine a data query range according to the data distribution characteristics;

[0091] Calculating the number of buckets of the medium statistical information histogram according to the data query range;

[0092] Based on the bucket number information and the bucket depth corresponding to the bucket, the expected number of data items to be obtained is calculated.

[0093] In this embodiment, the calculation of the expected number of data items requires the use of statistical information isobservation histograms. The isobservation histograms show the data distribution characteristics in the LSM level. Therefore, the data query range can be directly obtained based on the isobservation histograms. Based on the data query range, the corresponding bucket number information can be clearly obtained, where the bucket number information can be the number of buckets, for example: dividing the set in the range of 1.6 to 3.5 into 4 buckets or 6 buckets of different sizes. After calculating the number of buckets, the product of the number of buckets and the bucket depth is calculated to obtain the number of data items expected to be calculated.

[0094] More specifically, the above data reading cost calculation formula is shown in the following formula (1):

[0095] cost=n(H)*t(H)+n(L)*t(L) (1)

[0096] Where n(H) represents the number of data items on the high-speed medium, n(L) represents the number of data items on the low-speed medium, t(H) represents the read time of the high-speed medium, and t(L) represents the read time of the low-speed medium.

[0097] In this embodiment, there may be multiple query paths when querying the database, so the optimal query path with the minimum cost (shortest reading time) is calculated based on statistical information to obtain the best performance. The query cost can be simplified as: the expected number of data items to be obtained * the medium reading time.

[0098] In this embodiment, by analyzing the data distribution characteristics of the target business database in the LSM level based on the media statistical information histogram, the distribution of data on the physical storage medium can be accurately identified to avoid unnecessary full table scans, thereby significantly reducing the number and delay of I / O operations and improving data access speed. Based on the bucket number information combined with the corresponding bucket depth of the bucket, the expected number of data items is calculated. This strategy finely controls the query granularity and ensures that the query results are not only fast but also accurate.

[0099] To better illustrate that this application adds hierarchical medium key information to the histogram, and estimates the cost by combining the bucket depth and the number of elements in the bucket, it is possible to improve the estimation accuracy and select a better and more accurate query path. As a specific implementation method, this embodiment provides the calculation process and comparison results of the traditional calculation query data and the data query method based on the optimized statistical information histogram.

[0100] like Figure 5 As shown in the figure, the traditional computing query data process: for a given set {1.6, 1.9, 1.9, 2.0, 2.4, 2.6, 2.7, 2.7, 2.8, 2.9, 3.4, 3.5}, and 4 buckets are generated, then the final equal-depth histogram will contain four buckets [1.6, 1.9], [2.0, 2.6], [2.7, 2.8], [2.9, 3.5], and the bucket depth is 3. If the queried data is between 2.0 and 2.8, it just covers two buckets [2.0, 2.6], [2.7, 2.8], and the bucket depth is 3, that is, each bucket has three data, and there are two buckets in total, so this query has 6 data. For hierarchical storage, if table t1 has 1000 data, 200 of which are distributed on L0 / L1 and 800 are distributed on L2. The histogram can be used to calculate that there are 1000 data items, but since the media on which the data is distributed is not considered, the read cost is 1000*media read time. Since the read speed may differ by dozens of times for different media, there will be a large error in the cost estimate.

[0101] like Figure 6As shown, the query process of the data query method based on the optimized statistical information histogram provided by the present application: the histogram contains four buckets [1,100], [101,260], [261,500], [501,850], and the bucket depth is 100. Each bucket only has a quantity. According to the distribution of the data plus the medium-related information, the data from 1 to 500 is stored on the low-speed medium, and the data from 501 to 1000 is stored on the high-speed medium. Then the histogram can be expressed as [1,100]@L, [101,260]@L, [261,500]@L, [501,850]@H. @L means on the low-speed medium, and @H means on the high-speed medium. If you need to calculate the reading of data from 260 to 1000 (data query range), this interval covers the two buckets [261,500] and [501,850] (bucket number information). When estimating the number of data items based on statistical information, the traditional statistical information that does not include the medium returns a value of: 2 buckets * 100 bucket depth, which is 200. After adding the medium-related information (that is, after adding the medium reading time), the data conditions returned by the statistical information cannot be simply superimposed. In addition to the number of data, the medium reading time is also included. The returned results are [260,500]@L and [501,850]@H. Then: n(H) is the number of data in [501,850]@H, that is, its bucket depth (100) * the number of buckets (100), and n(L) is also 100. If t(H) is 1.5 and t(L) is 2.0, then substituting them into formula (1) yields a data reading cost of 350, where t(H) and t(L) are the medium reading times and are constants.

[0102] By comparison, the data query method based on the optimized statistical information histogram that adds key information of hierarchical media can more accurately calculate the reading cost, and can use the database to select a more appropriate query path, so that the database physical optimization can be more accurate in cost estimation. Compared with the traditional calculation query data process, it has higher accuracy, higher efficiency, and better calculation effect. At the same time, when estimating the number of data based on statistical information, the data conditions returned by the statistical information are not simply superimposed. In addition to the number of data, it also contains media information, which realizes a more accurate calculation of the reading cost, and can use the database to select a more appropriate query path. By providing an accurate evaluation of the reading performance of different levels, the problem of computational complexity caused by the performance differences of media at different levels is solved.

[0103] Step S203 , selecting an optimal data query path according to the data reading cost, and executing a data query request based on the optimal data query path.

[0104] In this embodiment, the optimal data query path can be selected according to the data reading cost, for example, the path with the lowest cost is selected for query, which optimizes query performance, reduces query delay, and solves the performance inconsistency problem caused by multi-path selection.

[0105] In an embodiment of the present invention, the LSM hierarchical position of the target business database is determined by analyzing the user's data query request, and then based on the LSM hierarchical position, the corresponding hierarchical media key information is extracted, and the media statistical information histogram is added to comprehensively calculate the data reading cost of the current data query request. Finally, the optimal data query path is selected according to the data reading cost, and the data query request is executed based on the optimal data query path. The embodiment of the present application parses the content in the query request, extracts key information from the selected LSM level to calculate the reading cost, which helps to identify features such as query services, so that according to these features, they are mapped to the hierarchical structure of the LSM tree, and can quickly locate the main LSM levels that may be involved in the query, determine the location where the data may be stored, avoid data search at unnecessary levels, reduce unnecessary hierarchical scanning and invalid queries, so that the reading cost can be calculated more efficiently and accurately. The problem of large cost estimation errors caused by large differences in reading speeds due to different media is solved, and the accuracy and reliability of reading cost calculation are significantly improved; at the same time, the optimal data query path is selected through the data reading cost to ensure optimal query performance and significantly improve the query response speed.

[0106] It should be emphasized that in order to further ensure the privacy and security of the data requested for the above query, it can also be stored in a node of a blockchain. The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain is essentially a decentralized database. It is a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. The blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0107] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0108] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0109] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0110] Further references Figure 7 , as a response to the above Figure 2 The present application provides an embodiment of a data query device based on a multi-level medium. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0111] like Figure 7 As shown, the data query device 700 based on multi-level media described in this embodiment includes: an acquisition module 701, a calculation module 702 and an execution module 703. Among them:

[0112] An acquisition module 701 is used to determine the LSM hierarchical location of the target business database based on the data query request;

[0113] A calculation module 702 is used to extract the corresponding hierarchical medium key information based on the LSM hierarchical position, and add the medium statistical information histogram to comprehensively calculate the data reading cost of the current data query request;

[0114] The execution module 703 is used to select an optimal data query path according to the data reading cost, and execute the data query request based on the optimal data query path.

[0115] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the data query method based on the optimized statistical information histogram in the above-mentioned implementation mode, and are not described in detail here.

[0116] In some optional implementations of this embodiment, the device further includes:

[0117] A parsing module, used to parse the request content of the data query request, where the parsed content at least includes the access mode and keywords of the query request;

[0118] The index module is used to establish an index structure based on the obtained access mode and keyword structure, and determine the target business database through the index structure.

[0119] In some optional implementations of this embodiment, the acquisition module includes:

[0120] A first acquisition submodule, used to retrieve the log file of the target business database according to a data query request;

[0121] A second acquisition submodule is used to acquire basic characteristic information of data maintenance and management in the target business database from the log file;

[0122] A mapping submodule is used to map the basic feature information to the hierarchical structure of the LSM tree according to the statistical information and / or index of the target business database to determine the LSM level where the target business database associated with the current data query request is located.

[0123] In some optional implementations of this embodiment, the hierarchical media key information extracted by the identification module includes: data block attributes, hierarchical attributes, media statistical information histogram and media reading time, and the calculation module specifically includes:

[0124] An extraction submodule, configured to extract data block attributes, layer attributes, and read time of layer media of the LSM layer based on the LSM layer where the target business database is located, wherein the data block attributes include data block size, and the layer attributes include the number of layers and IO performance corresponding to the number of layers;

[0125] An adjustment submodule, used for dynamically adjusting the read time of the hierarchical medium according to the data block size of the LSM level, the number of levels and the IO performance corresponding to the number of levels, and adding the adjusted read time of the hierarchical medium to the medium statistical information histogram to optimize the medium statistical information histogram;

[0126] A first calculation submodule, used for calculating the expected number of data items to be obtained based on the optimized medium statistical information histogram, and obtaining the number of data items and the corresponding storage interval information in the medium statistical information histogram;

[0127] The second calculation submodule is used to calculate the data reading cost based on the data number information, the storage interval information and the reading time of the hierarchical medium.

[0128] In some optional implementations of this embodiment, the second calculation submodule specifically includes:

[0129] An acquisition subunit, configured to acquire data distribution characteristics of the target service database in the LSM level based on the medium statistical information histogram, and determine a data query range according to the data distribution characteristics;

[0130] A first calculation subunit, configured to calculate the number of buckets of the medium statistical information histogram according to the data query range;

[0131] The second calculation subunit is used to calculate the expected number of data items to be obtained based on the bucket number information and the bucket depth corresponding to the bucket.

[0132] In some optional implementations of this embodiment, the calculation formula for calculating the data reading cost by the second calculation submodule is:

[0133] cost = n(H)*t(H)+n(L)*t(L);

[0134] Where n(H) represents the number of data items on the high-speed medium, n(L) represents the number of data items on the low-speed medium, t(H) represents the read time of the high-speed medium, and t(L) represents the read time of the low-speed medium.

[0135] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the data query method based on the optimized statistical information histogram in the above-mentioned implementation mode, and are not described in detail here.

[0136] To solve the above technical problems, the present application also provides a computer device. Figure 8 , Figure 8 This is a basic structural block diagram of the computer device in this embodiment.

[0137] The computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 8 with components 81-83, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0138] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0139] The memory 81 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 81 can be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 81 can also be an external storage device of the computer device 8, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 8. Of course, the memory 81 can also include both the internal storage unit of the computer device 8 and its external storage device. In this embodiment, the memory 81 is generally used to store the operating system and various application software installed on the computer device 8, such as computer-readable instructions of a data query method based on an optimized statistical information histogram, etc. In addition, the memory 81 can also be used to temporarily store various types of data that have been output or are to be output.

[0140] The processor 82 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 82 is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 82 is used to run the computer-readable instructions stored in the memory 81 or process data, such as running the computer-readable instructions of the data query method based on the optimization statistical information histogram.

[0141] The network interface 83 may include a wireless network interface or a wired network interface. The network interface 83 is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0142] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0143] In this embodiment, the embodiment of the present application first determines the LSM hierarchical position of the target business database according to the user's data query request, then extracts the corresponding hierarchical media key information based on the LSM hierarchical position, adds the media statistical information histogram to comprehensively calculate the data reading cost of the current data query request, and finally selects the optimal data query path according to the data reading cost, and executes the data query request based on the optimal data query path. The embodiment of the present application helps to identify features such as query services by parsing the content in the query request and extracting key information from the selected LSM level to calculate the reading cost, so that according to these features, it can be mapped to the hierarchical structure of the LSM tree, and can quickly locate the main LSM level that may be involved in the query, determine the location where the data may be stored, avoid data search at unnecessary levels, and reduce unnecessary hierarchical scanning and invalid queries; thereby, the reading cost can be calculated more efficiently and accurately, solving the problem of large cost estimation errors caused by large differences in reading speeds due to different media, and significantly improving the accuracy and reliability of reading cost calculation; at the same time, the optimal data query path is selected through the data reading cost to ensure optimal query performance and significantly improve the query response speed.

[0144] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data query method based on multi-layer media as described above.

[0145] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0146] In this embodiment, the embodiment of the present application first determines the LSM hierarchical position of the target business database according to the user's data query request, then extracts the corresponding hierarchical media key information based on the LSM hierarchical position, adds the media statistical information histogram to comprehensively calculate the data reading cost of the current data query request, and finally selects the optimal data query path according to the data reading cost, and executes the data query request based on the optimal data query path. The embodiment of the present application helps to identify features such as query services by parsing the content in the query request and extracting key information from the selected LSM level to calculate the reading cost, so that according to these features, it can be mapped to the hierarchical structure of the LSM tree, and can quickly locate the main LSM level that may be involved in the query, determine the location where the data may be stored, avoid data search at unnecessary levels, and reduce unnecessary hierarchical scanning and invalid queries; thereby, the reading cost can be calculated more efficiently and accurately, solving the problem that the reading speed may differ by dozens of times due to different media, so that the cost estimation will have errors, significantly improving the reading cost calculation accuracy and reliability, and at the same time, selecting the optimal data query path through the data reading cost to ensure the optimal query performance and significantly improve the query response speed.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the method steps described in each embodiment of the present application.

[0148] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data query method based on multi-level media, characterized in that: The steps include: Determine the LSM hierarchical location of the target business database based on the data query request; Based on the LSM hierarchical position, the corresponding hierarchical medium key information is extracted, and the medium statistical information histogram is added to comprehensively calculate the data reading cost of the current data query request; An optimal data query path is selected according to the data reading cost, and a data query request is executed based on the optimal data query path.

2. The data query method based on multi-level media according to claim 1, characterized in that: Before the step of determining the LSM hierarchical position of the target business database based on the data query request, the method further includes: Parsing the data query request, where the parsed content includes at least the access mode and keywords of the query request; An index structure is established based on the acquired access pattern and keyword structure, and a target business database is determined through the index structure.

3. The data query method based on multi-level media according to claim 2, characterized in that: The step of determining the LSM hierarchical location of the target business database based on the data query request specifically includes: Retrieving the log file of the target business database according to the data query request; Obtain basic characteristic information of data maintenance and management in the target business database from the log file; The basic feature information is mapped to the hierarchical structure of the LSM tree according to the statistical information and / or index of the target business database to determine the LSM level at which the target business database associated with the current data query request is located.

4. The data query method based on multi-level media according to claim 1, characterized in that: The hierarchical media key information includes: data block attributes, hierarchical attributes, media statistical information histogram, and media read time. The step of adding the media statistical information histogram to comprehensively calculate the data read cost of the current data query request specifically includes: Based on the LSM level where the target business database is located, extracting data block attributes, level attributes, and read time of the level medium of the LSM level, wherein the data block attributes include data block size, and the level attributes include the number of levels and IO performance corresponding to the number of levels; Dynamically adjusting the read time of the hierarchical medium according to the data block size of the LSM level, the number of levels and the IO performance corresponding to the number of levels, and adding the adjusted read time of the hierarchical medium to the medium statistical information histogram to optimize the medium statistical information histogram; Calculating the expected number of data items based on the optimized medium statistical information histogram, and obtaining information on the number of data items and corresponding storage interval information in the medium statistical information histogram; The data reading cost is calculated based on the data number information, the storage interval information and the reading time of the hierarchical medium.

5. The data query method based on multi-level media according to claim 4, characterized in that: The step of calculating the expected number of data items based on the optimized medium statistical information histogram specifically includes: Acquire data distribution characteristics of the target service database in the LSM level based on the medium statistical information histogram, and determine a data query range according to the data distribution characteristics; Calculating the number of buckets of the medium statistical information histogram according to the data query range; Based on the bucket number information and the bucket depth corresponding to the bucket, the expected number of data items to be obtained is calculated.

6. The data query method based on multi-level media according to claim 5, characterized in that: The calculation formula of the data reading cost is: cost = n(H)*t(H)+n(L)*t(L); Where n(H) represents the number of data items on the high-speed medium, n(L) represents the number of data items on the low-speed medium, t(H) represents the read time of the high-speed medium, and t(L) represents the read time of the low-speed medium.

7. A data query device based on multi-level media, characterized in that: include: An acquisition module is used to determine the LSM hierarchical location of the target business database based on a data query request; A calculation module, for extracting corresponding hierarchical medium key information based on the LSM hierarchical position, and adding a medium statistical information histogram to comprehensively calculate the data reading cost of the current data query request; An execution module is used to select an optimal data query path according to the data reading cost, and execute a data query request based on the optimal data query path.

8. The data query device based on multi-level media according to claim 7, characterized in that: The device also includes: A parsing module, used to parse the request content of the data query request, where the parsed content at least includes the access mode and keywords of the query request; The index module is used to establish an index structure based on the obtained access mode and keyword structure, and determine the target business database through the index structure.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the multi-layer medium-based data query method as claimed in any one of claims 1 to 6 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the multi-layer medium-based data query method according to any one of claims 1 to 6 are implemented.

Images