Data reading method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data reading method and device, electronic equipment and a storage medium, and belongs to the technical field of databases. The method comprises the following steps: determining a first page number occupied by a corresponding table-level composite index based on the area size of a candidate area in an initial data table, a preset index field number and a preset index type; dividing pages contained in the candidate area based on the first page number to determine index pages and data pages of the first page number; constructing a table-level composite index based on a region-level preset bloom filter and a preset double-layer sparse index, and storing the table-level composite index to an index page; storing field data of all candidate pages in the candidate area to a data page, and constructing a target area structure corresponding to the candidate area based on the stored index page and the data page; and constructing a target data table based on the multiple target area structures of the same initial data table, and performing data reading based on the target data table. According to the embodiment of the invention, the data reading efficiency between the database nodes can be improved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data reading method and device, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of emerging technologies and businesses such as big data and the Internet of Things, the amount of data generated is growing at a high speed. When users query data with complex requirements from massive data, it involves not only querying or manipulating one or several records in a relational table, but also data analysis and information integration of tens of millions of records in multiple tables. In addition, with the rapid increase in the amount of data, massive data also brings pressure on storage costs. In order to solve the problem that the storage capacity of traditional high availability (HA) deployment database clusters is doubled compared to a single machine, related technologies propose to build a resource pooling architecture in combination with segment page storage to solve the problem of too many page storage file handles, and it can be well compatible with other database storage layer modules currently based on page storage.

[0003] At present, under the resource pooling architecture of related technologies, the master node has read and write permissions, the backup node only has read permissions, all nodes share a copy of the data, and in the cluster of database nodes, the master node manages the status of all resources, and the master and backup nodes interact through the network. Taking a backup node read request as an example, the backup node will first apply for the S lock of a page before reading it. After judging the holding situation, the master node will let the holding node pass the page to the requesting node through the network and let the requesting node also hold the S lock. In the On-Line Analytical Processing (OLAP) scenario, this method will introduce a large amount of network interaction between nodes while bringing real-time consistency to the backup machine, which can easily reduce the read performance of the cluster. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a data reading method and device, an electronic device, and a storage medium, which can avoid unnecessary network interactions between primary and standby nodes in an OLAP scenario, improve the reading performance of database nodes, and thus improve the efficiency of data reading.

[0005] To achieve the above object, a first aspect of an embodiment of the present application provides a data reading method, the method comprising:

[0006] Obtaining a region size, a preset number of index fields, and a preset index type of a candidate region in an initial data table, wherein the initial data table includes a plurality of candidate regions, and the candidate region includes a plurality of candidate pages;

[0007] Determine the first number of pages occupied by the table-level composite index corresponding to the candidate zone based on the zone size, the preset number of index fields, and the preset index type;

[0008] Dividing the pages included in the candidate area into pages based on the first number of pages to determine index pages and data pages of the candidate area, where the number of index pages is the first number of pages;

[0009] The table-level composite index is constructed based on a preset Bloom filter and a preset double-layer sparse index, and the table-level composite index is stored in the index page; the preset Bloom filter is a pre-constructed Bloom filter based on the zone level, the preset double-layer sparse index includes a first sparse index based on the zone level and a second sparse index based on the page level, the first sparse index is used to indicate the data index field of the candidate zone, and the second sparse index is used to indicate the data index field of each candidate page in the candidate zone;

[0010] storing the field data of all the candidate pages in the candidate area into the data page, and constructing a target area structure corresponding to the candidate area based on the stored index page and the data page;

[0011] A target data table is constructed based on the multiple target area structures of the same initial data table, and data is read based on the target data table.

[0012] In some embodiments, the reading of data based on the target data table includes:

[0013] receiving a read request, wherein the read request includes the target data table and a query predicate condition;

[0014] Determine a target area list based on the target data table, the target area list including a plurality of target areas;

[0015] Selecting a to-be-filtered area from the plurality of target areas, and acquiring a target index page corresponding to the to-be-filtered area, wherein the target index page stores the table-level composite index corresponding to the to-be-filtered area;

[0016] Pre-filtering the query predicate condition based on the table-level composite index corresponding to the to-be-filtered area to obtain a composite filtering result;

[0017] If the composite filtering result indicates that the area to be filtered contains data that meets the query predicate condition, a page query is performed based on the composite filtering result, the query predicate condition and the data page corresponding to the area to be filtered to determine a page to be read;

[0018] Data reading is performed by scanning the read page.

[0019] In some embodiments, the pre-filtering of the query predicate condition based on the table-level composite index corresponding to the to-be-filtered area to obtain a composite filtering result includes:

[0020] Performing zone-level filtering on the query predicate condition based on the preset Bloom filter corresponding to the to-be-filtered zone and the first sparse index based on the zone level to obtain a zone-level filtering result;

[0021] If the zone-level filtering result indicates that the zone to be filtered contains data that meets the query predicate condition, a page-level sparse index detection is performed on the table-level composite index corresponding to the zone to be filtered based on the query predicate condition to obtain a page-level sparse index detection result;

[0022] If the page-level sparse index detection result indicates that the table-level composite index corresponding to the area to be filtered contains the second sparse index to be filtered, performing page-level filtering on the query predicate condition based on the second sparse index to obtain the page-level filtering result;

[0023] The composite filtering result is determined based on the page-level filtering results corresponding to all the second sparse indexes to be filtered.

[0024] In some embodiments, if the zone-level filtering result indicates that the to-be-filtered zone contains data that meets the query predicate condition, before performing page-level sparse index detection on the table-level composite index corresponding to the to-be-filtered zone based on the query predicate condition and obtaining the page-level sparse index detection result, the method further includes:

[0025] If the zone-level filtering result indicates that the zone to be filtered does not contain data that meets the query predicate condition, updating the zone to be filtered based on the plurality of target zones;

[0026] The query predicate condition is pre-filtered according to the updated table-level composite index corresponding to the to-be-filtered area to obtain a composite filtering result until all the target areas are pre-filtered.

[0027] In some embodiments, performing a page query based on the composite filtering result, the query predicate condition, and the data page corresponding to the to-be-filtered area to determine a page to read includes:

[0028] If the composite filtering result indicates that data that meets the query predicate condition is matched in the second sparse index of the to-be-filtered area, and the corresponding memory contains pages that meet the query predicate condition, the read page is determined from the data pages based on the matched second sparse index; or,

[0029] If the composite filtering result indicates that data that meets the query predicate condition is matched in the second sparse index of the area to be filtered, and the corresponding memory does not contain pages that meet the query predicate condition, a data page request is sent to the corresponding distributed memory service module, and the read page returned by the distributed memory service module is received.

[0030] In some embodiments, obtaining a target index page corresponding to the area to be filtered includes:

[0031] If it is detected that the corresponding memory contains multiple index pages of the target area, the target index page corresponding to the area to be filtered is obtained; or

[0032] If it is detected that the corresponding memory does not contain multiple index pages of the target area, an index page request is sent to the corresponding distributed memory service module, and the target index page corresponding to the to-be-filtered area returned by the distributed memory service module is received.

[0033] In some embodiments, the preset Bloom filter in the candidate region is constructed in the following manner:

[0034] Performing bit calculation based on the zone size, the number of fields of the table-level composite index, the unit page storage size, and preset optimization parameters to obtain a target bit number of the preset Bloom filter;

[0035] The preset Bloom filter corresponding to the candidate area is constructed based on the shared bit array of the target number of bits and a preset hash function.

[0036] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a data reading device, the device comprising:

[0037] An acquisition module, used to acquire a region size, a preset number of index fields, and a preset index type of a candidate region in an initial data table, wherein the initial data table includes a plurality of candidate regions, and the candidate region includes a plurality of candidate pages;

[0038] A page number determination module, configured to determine the first number of pages occupied by the table-level composite index corresponding to the candidate zone based on the zone size, the preset number of index fields and the preset index type;

[0039] a page division module, configured to divide the pages included in the candidate area into pages based on the first number of pages, and determine index pages and data pages of the candidate area, wherein the number of the index pages is the first number of pages;

[0040] A composite index construction module, configured to construct the table-level composite index based on a preset Bloom filter and a preset double-layer sparse index, and store the table-level composite index in the index page; the preset Bloom filter is a pre-constructed zone-level Bloom filter, the preset double-layer sparse index includes a first zone-level sparse index and a second page-level sparse index, the first sparse index is used to indicate a data index field of the candidate zone, and the second sparse index is used to indicate a data index field of each candidate page in the candidate zone;

[0041] A storage module, used for storing the field data of all the candidate pages in the candidate area into the data page, and constructing a target area structure corresponding to the candidate area based on the stored index page and the data page;

[0042] A data reading module is used to construct a target data table based on a plurality of target area structures of the same initial data table, and to read data based on the target data table.

[0043] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0044] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, the method described in the first aspect is implemented.

[0045] The embodiments of the present application provide a data reading method and device, an electronic device, and a storage medium. First, a region size, a preset number of index fields, and a preset index type of a candidate region in an initial data table are obtained, wherein the initial data table includes multiple candidate regions, and the candidate region includes multiple candidate pages; further, based on the region size, the preset number of index fields, and the preset index type, a first number of pages occupied by a table-level composite index corresponding to the candidate region is determined; further, based on the first number of pages, pages included in the candidate region are divided into pages to determine index pages and data pages of the candidate region, wherein the number of index pages is the first number of pages; further, a table-level composite index is constructed based on a preset Bloom filter and a preset double-layer sparse index, And store the table-level composite index to the index page, the preset Bloom filter is a pre-built Bloom filter based on the zone level, the preset double-layer sparse index includes a first sparse index based on the zone level and a second sparse index based on the page level, the first sparse index is used to indicate the data index field of the candidate zone, and the second sparse index is used to indicate the data index field of each candidate page in the candidate zone; further, store the field data of all candidate pages in the candidate zone to the data page, and build the target zone structure corresponding to the candidate zone based on the stored index page and data page; further, build a target data table based on multiple target zone structures of the same initial data table, and read data based on the target data table. The embodiment of the present application proposes an innovative way to build a target data table, that is, the index page and the data page are divided in the target zone structure of the target data table, and the index page stores the table-level composite index built based on the preset Bloom filter and the preset double-layer sparse index, which can play a hierarchical screening effect in subsequent data reading and avoid reading invalid pages. Therefore, compared with the related art, the present application can effectively avoid unnecessary network interactions between the primary and standby nodes in the OLAP scenario, improve the read performance of the database nodes, and thus improve the efficiency of data reading. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is an architectural diagram of a resource pooling architecture provided by related technologies;

[0047] Figure 2 It is a schematic diagram of an organization form based on segment-page storage provided in an embodiment of the present application;

[0048] Figure 3 is a first flow chart of a data reading method provided in an embodiment of the present application;

[0049] Figure 4 is a schematic diagram of a structure of a candidate area provided in an embodiment of the present application;

[0050] Figure 5 It is a structural schematic diagram of dividing pages in a candidate area structure provided by an embodiment of the present application;

[0051] Figure 6 It is a structural diagram of storing a corresponding table-level composite structure in an index page of a candidate area provided by an embodiment of the present application;

[0052] Fig. 7A It is a specific structural diagram of a storage table-level composite index provided in an embodiment of the present application;

[0053] Figure 7B is another specific structural diagram of a storage table-level composite index provided in an embodiment of the present application;

[0054] Figure 8 It is a schematic diagram of a tuple storage structure of a table-level composite index provided in an embodiment of the present application;

[0055] Fig. 9 yes Figure 3 A flowchart of step S340 in FIG.

[0056] Fig.10 is a filtering schematic diagram of a preset Bloom filter provided in an embodiment of the present application;

[0057] Fig.11 yes Figure 3 A flowchart of step S360 in FIG.

[0058] Fig.12 yes Fig.11 A flowchart of step S1140 in FIG.

[0059] Fig.13 is a specific flow chart of the data reading method provided in the embodiment of the present application;

[0060] Fig.14 is a structural schematic diagram of a data reading device provided in an embodiment of the present application;

[0061] Fig.15 It is a hardware structure diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0065] First, some nouns involved in this application are analyzed:

[0066] Paged storage: a storage method that logically organizes, allocates and manages storage space in units of pages, and stores data in a fixed format in the pages. Under the paged storage mechanism, the data of each table independently corresponds to a logically large file (up to 32TB), and the logical file is divided into multiple physical files of a fixed size (default 1GB) and stored in the corresponding directory. A table is divided into multiple partition tables and split into multiple sub-tables, and the number of files required at the bottom layer is large.

[0067] Segment-page storage: Logically, segment-page storage is organized, stored, allocated, and managed in the form of segments, extents, and pages. A data table in the database corresponds to a logical segment, and all data in the table is stored on the segment. Each segment will mount multiple extents, each extent is a continuous physical page, and an extent contains multiple pages. Extents are not necessarily continuous, but pages in the same extent are continuous. In the physical file, the same type of extent is stored in the same physical file. A single physical file is up to 1GB in size, and different physical files store data with different forknums.

[0068] Database node: refers to an independent entity in the database system that is responsible for storing and managing data. A database node can be a physical server, virtual server, or other computing resource.

[0069] Scan operator: It is an operator used for traversing results such as tables, result sets, and linked list subqueries, and obtains a tuple as the input of the upper node each time. Scan operators include Sequence Scan (SeqScan), Index Scan (IndexScan), Bitmap Scan (BitmapHeapScan), etc.

[0070] Bloom filter: It is a highly space-efficient probabilistic data structure used to determine whether an element is in a set. Bloom filters usually consist of a very long binary vector (bit array) and a series of random mapping functions (hash functions). When an element is added to the Bloom filter, the hash value is calculated by the hash function and the corresponding bit is set to 1. When querying, the given element can be hashed again. If all corresponding bits are 1, the element may exist; otherwise, the element must not be there.

[0071] Shared lock (S lock): refers to a lock that allows multiple transactions to read resources at the same time, but cannot modify resources. S locks can ensure that resources can be shared and read between multiple transactions, but prevent modification, thereby avoiding data inconsistency. Among them, multiple transactions can hold S locks at the same time, allowing concurrent reading.

[0072] Predicate Condition: It is a conditional expression used to filter data in programming languages ​​and database queries. Predicate conditions are usually used to determine whether the value of a certain attribute or field meets specific conditions, thereby deciding whether to select or operate the corresponding data record.

[0073] Distributed Memory Service (DMS): It is a dynamic library integrated into the database. It transmits page contents through the Transmission Control Protocol / Remote Direct Memory Access (TCP / RDMA) network, integrates the main and standby memories, and provides memory pooling capabilities to achieve real-time consistency read functions for the standby machine. Memory pooling achieves real-time exchange of main and standby pages through the distributed memory service DMS component, providing real-time consistency capabilities for the standby machine. That is, after the host transaction is committed, it can be read immediately on the standby machine, and there is no delayed reading phenomenon (the transaction isolation level is Read-Committed).

[0074] Transaction visibility: refers to which modifications of other transactions can be seen by the database read operation in a transaction. This is mainly affected by the transaction isolation level. Different isolation levels determine the difference in data visibility between transactions.

[0075] With the rapid development of emerging technologies and businesses such as big data and the Internet of Things, the amount of data generated is growing at a high speed. When users query data with complex requirements from massive data, it involves not only querying or manipulating one or several records in a relational table, but also data analysis and information synthesis of tens of millions of records in multiple tables. In addition, with the rapid increase in the amount of data, massive data also brings pressure on storage costs. Based on the above problems, researchers in the database field have proposed the On-Line Analytical Processing (OLAP) technology. OLAP is specially designed to support complex analytical operations, focusing on decision support for decision makers and senior managers, and can quickly and flexibly perform complex query processing on large amounts of data according to the requirements of analysts. In the OLAP scenario, its data often has the following characteristics: First, its data volume is very large, often at the terabyte (TB) or petabyte (PB) level. The read operation of this level of data volume poses a challenge to the scanning performance of the database; second, the possibility of data being changed after writing is low; third, the actual distribution of its data in the storage medium often shows aggregation in some fields.

[0076] As the amount of data increases dramatically, the massive amount of data also brings pressure on storage costs. In order to solve the problem that the storage capacity of the traditional high availability (HA) deployment database cluster is doubled compared to that of a single machine, related technologies have proposed a way to introduce resource pooling features in the database. Figure 1 As shown, Figure 1 It is an architectural diagram of the resource pooling architecture provided by the relevant technology. The resource pooling feature provides a new HA deployment form, which enables all nodes in the cluster to share a storage, while reducing storage costs, and bringing excellent features such as real-time consistent read of the backup machine and low recovery time objective (RTO) time.

[0077] Among them, segment page storage is a storage architecture that is different from traditional page storage. It can solve the problem of too many page storage file handles and is well compatible with other database storage layer modules currently based on page storage. Figure 2 As shown, Figure 2This is a schematic diagram of an organization form based on segment-page storage provided by an embodiment of the present application, in which the possible values ​​of an Extent size are 64K (1K = 1024 bytes), 1M (1M = 1024K), 8M and 64M, etc., and the size of a data page is fixed at 8K. Therefore, combining the segment-page storage method to build a resource pooling architecture can solve the problem of too many page storage file handles, and can be well compatible with other database storage layer modules currently based on page storage.

[0078] Currently, if Figure 1 As shown, under the resource pooling architecture of the related technology, the master node has read and write permissions, the standby node only has read permissions, and all nodes share a copy of the data. In a cluster of database nodes, there are multiple database nodes in a cluster, and resources refer to data pages and regular locks, etc., and locks and page resources are managed by the DMS module. Among them, the master node manages the status of all resources, and the master and standby nodes interact through the network. Taking a standby node read request as an example, the standby node will first apply for the S lock of the page before reading a page. To this end, the standby node will go to the master node to request the S lock of the page, and the master node will respond according to the current status of the page. If the page is currently not held by any nodes in the cluster, it will allow the requesting node to read from the storage and hold the S lock; if there is a node in the cluster currently writing to the page, the requesting node needs to wait; if there are other nodes in the cluster holding the S lock, the master node will allow the holding node to pass the page to the requesting node through the network and let the requesting node also hold the S lock. However, in the On-Line Analytical Processing (OLAP) scenario, while this method brings real-time consistency to the standby machine, it also introduces a large amount of network interaction between nodes, which can easily reduce the read performance of the cluster.

[0079] Based on this, the embodiments of the present application provide a data reading method and device, an electronic device, and a storage medium, which can avoid unnecessary network interactions between primary and standby nodes in an OLAP scenario, improve the reading performance of database nodes, and thus improve the efficiency of data reading.

[0080] The data reading method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms; the software can be an application that implements the data reading method, etc., but is not limited to the above forms.

[0081] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network personal computers (Personal Computer, PC), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0082] See also Figure 3 , Figure 3 is an optional flow chart of the data reading method provided in the embodiment of the present application. Figure 3 The method may specifically include but is not limited to steps S310 to S360. Figure 3 These six steps are introduced in detail.

[0083] Step S310, obtaining the region size, the number of preset index fields and the preset index type of the candidate region in the initial data table;

[0084] Step S320, determining the number of first pages occupied by the table-level composite index corresponding to the candidate zone based on the zone size, the number of preset index fields, and the preset index type;

[0085] Step S330, dividing the pages included in the candidate area based on the first number of pages to determine index pages and data pages of the candidate area, where the number of index pages is the first number of pages;

[0086] Step S340, constructing a table-level composite index based on the preset Bloom filter and the preset double-layer sparse index, and storing the table-level composite index in the index page;

[0087] Step S350, storing the field data of all candidate pages in the candidate area into the data page, and constructing the target area structure corresponding to the candidate area based on the stored index page and data page;

[0088] Step S360: construct a target data table based on multiple target area structures of the same initial data table, and read data based on the target data table.

[0089] In steps S310 to S360 of some embodiments, the data reading method provided by the embodiments of the present application can be used in both traditional clusters using segment page storage and resource pooling architecture clusters. That is, when facing a large amount of data read operation in a traditional cluster using segment page storage, the method can filter multiple data pages at a time through the Extent-level composite index to avoid invalid disk IO. In addition, the embodiments of the present application can be better applied to tables using segment page storage, thereby greatly improving data reading performance.

[0090] In step S310 of some embodiments, the initial data table refers to a data table initially set up and constructed based on a segment-page storage structure. The storage of the initial data table is organized, stored, allocated, and managed in the form of segments, extents, and pages. An initial data table corresponds to a logical segment, and all data in the table exists on the segment. The initial data table includes multiple candidate areas (i.e., each segment will mount multiple extents), and the candidate areas include multiple candidate pages (i.e., each extent will contain multiple pages).

[0091] Among them, the zone size of the candidate zone refers to the space occupied by the candidate zone in the initial data table, which can be the number of rows, the number of pages or other metrics. The preset number of index fields refers to the number of fields used to establish a table-level composite index. By setting the preset number of index fields and the corresponding preset index fields, you can know which fields will be used to build the index in order to speed up the retrieval operation, and these fields should be selected based on the query usage frequency and data distribution. The preset index type refers to a pre-set index type, and the index type can include a Bloom index and a sparse index. In addition, this application does not specifically limit the space used by the index page.

[0092] In step S320 of some embodiments, further, the number of first pages occupied by the table-level composite index corresponding to the candidate zone may be determined based on the zone size, the number of preset index fields, and the preset index type.

[0093] In step S330 of some embodiments, since the spatial size of the candidate area is set, the number of pages contained therein is also fixed. After determining the first number of pages occupied by the table-level composite index corresponding to the candidate area, the pages contained in the candidate area can be divided into pages based on the first number of pages to determine the index pages and data pages of the candidate area. The number of index pages is the first number of pages, and the remaining pages in the candidate area are data pages for storing table data. In other words, the candidate area can be constructed based on the index pages and data pages. Figure 4 As shown, Figure 4 A structural diagram of the candidate areas provided for the embodiment of the present application. Among them, for the candidate areas extent0 and extent1 contained in the initial database, the corresponding storage space size is 64K. Based on the area size, the preset number of index fields and the preset index type, it can be determined that the first number of pages occupied by the table-level compound index corresponding to extent0 is the number of pages contained in area D1, and the first number of pages occupied by the table-level compound index corresponding to extent1 is the number of pages contained in area D9. It can be understood that area D1 in extent0 is used to store the table-level compound index corresponding to extent0, and areas D2 to D8 are used to store the real data corresponding to extent0; area D9 in extent1 is used to store the table-level compound index corresponding to extent1, and areas D10 to D16 are used to store the real data corresponding to extent1.

[0094] In step S340 of some embodiments, the preset Bloom filter is a pre-built Bloom filter based on the zone level. The preset double-layer sparse index includes a first sparse index based on the zone level and a second sparse index based on the page level, the first sparse index is used to indicate the data index field of the candidate zone, and the second sparse index is used to indicate the data index field of each candidate page in the candidate zone. The index page divided by the candidate zone is used to store the table-level composite index built based on the preset Bloom filter and the preset double-layer sparse index.

[0095] It should be noted that sparse indexing is an indexing technology that only stores key information of part of the data, rather than all the data. This can save storage space and improve query efficiency in some cases. The preset double-layer sparse index created in this application is a sparse index based on Extent-Page double-layer filtering, which means that the index is first filtered at the Extent level and then further refined to the Page level. This hierarchical filtering can more effectively locate the page containing the required data.

[0096] It should be noted that the index page used to store the table-level composite index can include two parts: the preset Bloom filter page (i.e., the page used to store the Extent-level preset Bloom filter) and the sparse index storage page (i.e., the page used to store the preset double-layer sparse index). In the index page, the sparse index storage page is stored in the front, and the preset Bloom filter page is stored in the back, and the header of the index page can record the offset of the Extent-level preset Bloom filter in the index page. Please refer to Figure 5 , Figure 5 It is a structural diagram of dividing pages in the candidate area structure provided by an embodiment of the present application. Among them, a table file storing the initial data table may include multiple Segments, a Segment is equivalent to an initial data table, a Segment may contain multiple Extents (i.e., candidate areas), and an Extent may contain multiple pages. At this time, in combination with the above steps S310 to S330, the first several pages of the Extent header can be used as index pages for storing the corresponding table-level composite index, and the remaining pages are used as data pages for storing the real data corresponding to the candidate area.

[0097] Please note that Figure 6 , Figure 6 It is a structural diagram of storing the corresponding table-level composite structure in the index page of the candidate zone provided by an embodiment of the present application. Among them, in the index page, for the storage of the preset double-layer sparse index, a column-first storage method can be adopted (but this method is not limited, it is only an example), that is, the sparse index involved can be stored in a continuous storage space on the index page. First, the data index field based on the first sparse index at the zone level can be stored in the first index page, that is, the maximum data value (max) and the minimum data value (min) in the established index field are saved. Figure 6 As shown, the min / max of the extent includes 4 data index fields, colum1 to colum4, and the data index field corresponding to colum1 at this time represents the min and max of all pages counted from the perspective of the extent. In the subsequent index pages, the min / max of the candidate pages page0 to page n (n represents the total number of pages contained in the extent - 1) contained in the extent can be stored, and at this time, page 0 to page n in the data index field of the same colum all belong to the subfield of the data index field corresponding to the colum. In this embodiment, the sparse index of a column can be put together, and the structure can also be adjusted according to actual needs, which will not be elaborated.

[0098] For example, see Fig. 7A and Figure 7B , Fig. 7A for Figure 4 The specific structural diagram of area D1 in extent0 used to store the table-level composite index corresponding to extent0 is shown in FIG. Figure 7B for Figure 4 A schematic diagram of the specific structure of area D1 in extent1 for storing the table-level composite index corresponding to extent1. Fig. 7A , the min / max of the extent contains 4 data index fields, colum1 to colum4, and the extent contains candidate pages page 1 to page 7. For the data index field of extent in colum1, the min (i.e., the minimum value of the field) is -1, and the max (i.e., the maximum value of the field) is 57. At this time, the min (i.e., the minimum value of the field) of the data index field of page1 in colum1 is -1, and the max (i.e., the maximum value of the field) is 10, and the min (i.e., the minimum value of the field) of the data index field of page2 in colum1 is 11, and the max (i.e., the maximum value of the field) is 20, and so on. In addition, the min (i.e., the minimum value of the field) of the data index field of page7 in colum1 is 48, and the max (i.e., the maximum value of the field) is 57. In this way, the value range of the data index field of each candidate page in the same colum is a subset of the value range of the data index field of the extent in the same colum. Based on this, it can be preliminarily determined whether the extent contains the data to be read based on the value range of the data index field at the page where the extent is located.

[0099] It should be noted that the page used to store the preset double-layer word count index may include multiple tuples. The first of all sparse index tuples is the Extent-level sparse index tuple, followed by sparse index tuples at each page level. Figure 8 , Figure 8 It is a schematic diagram of a tuple storage structure of a table-level composite index provided in an embodiment of the present application. Among them, a tuple in the tuple structure of the preset double-layer sparse index can store the index information of a candidate area or candidate page, such as column 1min and column 1max store the index field value interval at the Extent level, column 2min and column 2max store the index field value interval at a page level in the Extent, and so on.

[0100] See also Fig. 9 , Fig. 9 is an optional flow chart of step S340 provided in an embodiment of the present application. In some embodiments, step S340 may specifically include steps S910 to S930. Fig. 9These two steps are introduced in detail.

[0101] Step S910, performing bit calculation based on the zone size, the number of fields of the table-level composite index, the unit page storage size, and the preset optimization parameters to obtain the target bit number of the preset Bloom filter;

[0102] Step S920 , constructing a preset Bloom filter corresponding to the candidate area based on the shared bit array of the target bit number and the preset hash function.

[0103] In step S910 and step S920 of the above embodiment, the preset Bloom filter constructed by the present application can be an adaptive multi-key Bloom filter based on the number of fields at the Extent level and the perception of the Extent size, which can be better used for efficient existence judgment on fields whose values ​​are enumeration types.

[0104] In the embodiment of the present application, all fields used to establish the preset Bloom filter can share a bit array (bitarray), that is, a shared bit array. The target number of bits of the shared bit array can be determined by the size of the Extent in which it is located and the number of fields used to establish the index. Therefore, the process of determining the target number of bits in the embodiment of the present application can be calculated by the following formula:

[0105] A=K*n*(m / 8KB)

[0106] Among them, A represents the number of bits used by the shared bit array, that is, the target number of bits; n is the number of fields used to establish a table-level composite index, that is, the number of fields of the table-level composite index; m is the area size of the candidate area corresponding to the preset Bloom filter; K is a preset optimization parameter, which can be a hyperparameter set according to actual needs and is not limited; 8KB represents the unit page storage size.

[0107] It should be noted that the larger the number of bits used in the shared bit array, the smaller the probability of making an error, but if the number of bits is too large, it will also cause many problems. Therefore, based on the zone size, the number of fields of the table-level composite index, the unit page storage size and the preset optimization parameters, this application can determine the target number of bits that better meets the requirements and needs of the candidate zone, better improve the read performance of the database node, and thus improve the efficiency of data reading.

[0108] In some embodiments, see Fig.10 , Fig.10It is a filtering schematic diagram of a preset Bloom filter provided in an embodiment of the present application. Among them, after determining the target number of bits, a shared bit array with a length of the target number of bits can be created first, and all bits are initialized to 0. When inserting an element, q different hash functions can be used to calculate the hash value corresponding to the attribute value of the element (such as attribute1: value1 and attribute2: value2 in the figure), and then these hash values ​​are mapped to the corresponding positions of the shared bit array, and the bits at these positions are set to 1. When querying whether an element is in a set, k hash functions are also used to calculate the hash value of the element, and then the positions corresponding to these hash values ​​in the shared bit array are checked. If the bits at all positions are 1, then the element may be in the set; if the bit at any position is 0, then the element is definitely not in the set.

[0109] In step S350 of some embodiments, further, the field data of all candidate pages in the candidate area may be stored in the data page, and a target area structure corresponding to the candidate area may be constructed based on the stored index page and data page.

[0110] The embodiment of the present application optimizes the segment page storage in the predicate pre-filtering framework based on the composite index under the segment page storage, and divides the pages in each Extent into index pages and data pages, wherein the index page is used to store the composite index, and the data page is used to store the real data. In addition, when creating a table, the user can specify the index field and index type according to the characteristics of the data, and select a certain proportion of pages in each Extent as index pages, and the rest as data pages.

[0111] In step S360 of some embodiments, after multiple target areas are constructed based on the target area structure construction method of the present application, a target data table can be constructed based on multiple target area structures of the same initial data table, and data can be read based on the target data table.

[0112] See also Fig.11 , Fig.11 is an optional flowchart of step S360 provided in an embodiment of the present application. In some embodiments, the data reading based on the target data table in step S360 may specifically include steps S1110 to S1160. Fig.11 These six steps are introduced in detail.

[0113] Step S1110, receiving a read request;

[0114] Step S1120, determining a target area list based on the target data table;

[0115] Step S1130, selecting a to-be-filtered area from a plurality of target areas, and obtaining a target index page corresponding to the to-be-filtered area;

[0116] Step S1140, pre-filtering the query predicate condition based on the table-level composite index corresponding to the area to be filtered, to obtain a composite filtering result;

[0117] Step S1150, if the composite filtering result indicates that the area to be filtered contains data that meets the query predicate condition, a page query is performed based on the composite filtering result, the query predicate condition and the data page corresponding to the area to be filtered to determine the page to be read;

[0118] Step S1160, reading data by scanning the reading page.

[0119] In step S1110 of some embodiments, a read request refers to a request by a target node (from a client or other system) to apply to other nodes for querying and reading data. A read request may be an identifier, condition, SQL statement, etc. of a specific data item. The target node may be a master node or a standby node in the same node cluster. The read request includes a target data table and a query predicate condition. The target data table refers to a data table where the data to be read is located, and the target data table may be constructed based on the target area structure of the present application. The query predicate condition refers to a condition for filtering out the data to be read in the target data table, and is used to filter the data stored in the target data table. The query predicate condition may also be specifically limited to the page in the target data table to be read, without specific limitation. For example, the SQL statement corresponding to the read request is "select * from tbl where colum1 <-80 and colum4>750". The goal of the read request is to return all data that simultaneously meet the conditions that the value of column1 is less than -80 and the value of column4 is greater than 750 from the tbl table. If no record meets both conditions at the same time, the result of the query will be an empty set. At this time, the target data table of the read request is the tb1 table, and the query predicate condition is "colum1<-80 and colum4>750".

[0120] In step S1120 of some embodiments, all target areas included in the target data table can be known based on the known target data table, so that a target area list can be obtained, that is, the target area list includes multiple target areas.

[0121] In step S1130 of some embodiments, further, a to-be-filtered area is selected from a plurality of target areas. This is because each target area represents a portion of data stored in the target data table, and these areas need to be checked to find data that meets the conditions. Therefore, each target area in the target data table needs to be filtered in turn as an to-be-filtered area to achieve complete reading of the data that meets the conditions in the target data table. When judging the to-be-filtered area, first obtain the target index page corresponding to the to-be-filtered area. At this time, the target index page refers to the page that stores the table-level composite index corresponding to the to-be-filtered area proposed in the above embodiments.

[0122] In some embodiments, obtaining the target index page corresponding to the to-be-filtered area may specifically include:

[0123] If it is detected that the corresponding memory contains multiple index pages of the target area, the target index page corresponding to the area to be filtered is obtained; or,

[0124] If it is detected that the corresponding memory does not contain index pages of multiple target areas, an index page request is sent to the corresponding distributed memory service module, and the target index page corresponding to the to-be-filtered area returned by the distributed memory service module is received.

[0125] It is understandable that when other nodes receive a read request, they will first check whether the memory shared by the nodes of the database already contains index pages of multiple target areas related to the target query, that is, check whether the memory hits the page corresponding to the area to be filtered. If it does, the target index page corresponding to the area to be filtered is directly obtained from the memory. In this way, the efficiency of data reading can be improved, because memory access is much faster than retrieving data from disk or remote service.

[0126] If the index page corresponding to the area to be filtered is not found in the memory and the resource pooling architecture is currently in use, a request can be made to the DMS module corresponding to the node, which also points to the distributed nature of the data storage architecture. In other cases, data can be read directly from the disk. During this process, the system sends an index page request to the distributed memory service module, which includes information about the target area to be queried. Once the distributed memory service module processes the request, it returns the target index page corresponding to the area to be filtered, and these target index pages can be used to perform subsequent read operations.

[0127] In step S1140 of some embodiments, after the target index page of the area to be filtered is determined, the query predicate condition may be pre-filtered based on the table-level composite index corresponding to the area to be filtered stored in the target index page to obtain a composite filtering result.

[0128] See also Fig.12 , Fig.12is an optional flow chart of step S1140 provided in an embodiment of the present application. In some embodiments, step S1140 may specifically include steps S1210 to S1240. Fig.12 These four steps are introduced in detail.

[0129] Step S1210, performing zone-level filtering on the query predicate condition based on the preset Bloom filter corresponding to the to-be-filtered zone and the first sparse index based on the zone level to obtain a zone-level filtering result;

[0130] Step S1220, if the zone-level filtering result indicates that the zone to be filtered contains data that meets the query predicate condition, a page-level sparse index detection is performed on the table-level composite index corresponding to the zone to be filtered based on the query predicate condition to obtain a page-level sparse index detection result;

[0131] Step S1230, if the page-level sparse index detection result indicates that the table-level composite index corresponding to the to-be-filtered area contains the second sparse index to be filtered, the query predicate condition is filtered at the page level based on the second sparse index to obtain a page-level filtering result;

[0132] Step S1240: determining a composite filtering result based on the page-level filtering results corresponding to all the second sparse indexes to be filtered.

[0133] In step S1210 of some embodiments, during specific filtering, the query predicate condition can be first filtered at the zone level based on the preset Bloom filter corresponding to the zone to be filtered and the first sparse index based on the zone level to obtain the zone level filtering result. That is to say, the extent level index filtering is performed first. When the table-level composite index is established in the present application, the preset Bloom filter and the sparse index have no dependency on each other and can be established independently. Therefore, for a field in the table that has a table-level composite index established, when the table-level composite index is used for zone level filtering, if the field has a preset Bloom filter based on the zone level, the preset Bloom filter is used for filtering. If the first sparse index based on the zone level is established, the first sparse index based on the zone level is used for filtering. There is no limitation on the specific filtering order and the specific filtering method. Furthermore, if one of the two results is negative, it is determined that the zone to be filtered does not contain data that meets the query predicate condition.

[0134] In step S1220 of some embodiments, if the zone-level filtering result indicates that the zone to be filtered contains data that meets the query predicate condition, a page-level sparse index check is further performed on the target page in the table-level composite index corresponding to the zone to be filtered based on the query predicate condition to determine whether there is an available second sparse index based on the page level, so as to obtain a page-level sparse index check result.

[0135] In some embodiments, if the zone-level filtering result in step S1220 indicates that the to-be-filtered zone contains data that meets the query predicate condition, a page-level sparse index detection is performed on the table-level composite index corresponding to the to-be-filtered zone based on the query predicate condition. Before obtaining the page-level sparse index detection result, the data reading method of the present application may further include:

[0136] If the zone-level filtering result indicates that the zone to be filtered does not contain data that meets the query predicate condition, the zone to be filtered is updated based on the multiple target zones;

[0137] The query predicate condition is pre-filtered according to the updated table-level composite index corresponding to the to-be-filtered area to obtain a composite filtering result until all target areas are pre-filtered.

[0138] It is understandable that if the area-level filtering result indicates that the area to be filtered does not contain data that meets the query predicate condition, it means that the current area to be filtered does not have data that meets the requirements, and the next target area of ​​the target data table can be judged, and the area to be filtered can be updated based on multiple target areas. Further, the query predicate condition is pre-filtered according to the table-level composite index corresponding to the updated area to be filtered, that is, S1140 and subsequent steps are repeatedly executed to obtain a composite filtering result until all target areas are pre-filtered.

[0139] It is understandable that if the extent-level zone-level filtering result is yes, and there is a sparse index on the field in the table-level composite index, the page-level sparse index is used to filter the data when reading each page. If the result is no, there is no required data on the page, otherwise the data page is scanned. Under a non-resource pooling architecture, the read and write process of the index page is consistent with the read and write process of the data page. Among them, the table-level composite index is updated while writing data. When responding to a read request, first determine the list of zones involved in the read request, and request the index page of each processed zone. If the filtering result is no, all subsequent pages under the zone are skipped. If the zone-level filtering result is yes, the corresponding sparse index of the data page is used for filtering when the scanning operator processes each data page.

[0140] In step S1230 of some embodiments, if the page-level sparse index detection result indicates that the table-level composite index corresponding to the area to be filtered contains a second sparse index to be filtered based on the page level, the query predicate condition is filtered at the page level based on the second sparse index to obtain a page-level filtering result. At this time, the page-level filtering result may be empty, that is, there is no data that meets the requirements in the corresponding page, or it may be a specific matching data field range, that is, the corresponding page contains data that meets the requirements, so as to facilitate subsequent specific data reading on the data page.

[0141] It should be noted that if the page-level sparse index detection result indicates that the table-level composite index corresponding to the area to be filtered does not contain the second sparse index to be filtered based on the page level, the target page of the area to be filtered is directly scanned.

[0142] In step S1240 of some embodiments, after performing page-level filtering on all target pages in the area to be filtered, a composite filtering result can be determined based on the page-level filtering results corresponding to all second sparse indexes to be filtered and the results of directly scanning the pages. The composite filtering result at this time includes all data field ranges in the area to be filtered that meet the query predicate condition requirements.

[0143] In step S1150 of some embodiments, if the composite filtering result indicates that the area to be filtered contains data that meets the query predicate condition, a page query is performed based on the composite filtering result, the query predicate condition, and the data page corresponding to the area to be filtered to determine the read page. Among them, data can be directly read from the scanned page. At this time, the read page is the page corresponding to the data field range that meets the query predicate condition requirements. In this way, it can be ensured that only necessary data pages are scanned, further improving data reading efficiency.

[0144] In some embodiments, performing page query based on the composite filtering result, the query predicate condition and the data page corresponding to the to-be-filtered area in step S1150 to determine the page to be read may specifically include:

[0145] If the composite filtering result indicates that data that meets the query predicate condition is matched in the second sparse index of the to-be-filtered area, and the corresponding memory contains pages that meet the query predicate condition, the read page is determined from the data page based on the matched second sparse index; or,

[0146] If the composite filtering result indicates that no data that meets the query predicate conditions is matched in the second sparse index of the area to be filtered, and the corresponding memory does not contain pages that meet the query predicate conditions, a data page request is sent to the corresponding distributed memory service module, and the read page returned by the distributed memory service module is received.

[0147] It is understandable that if the composite filtering result indicates that data that meets the query predicate condition is matched in the second sparse index of the to-be-filtered area, and the memory corresponding to the node contains a page that meets the query predicate condition, it means that the data page stored in the memory can hit the corresponding data. Otherwise, it is necessary to send a data page request to the distributed memory service module corresponding to the node to obtain the read page.

[0148] In step S1160 of some embodiments, further, the requested data is obtained by scanning and reading the determined read page. This step can extract the required information from the physical data page and return it to the user or application. Among them, data reading may involve decoding, conversion and formatting the data into a form that is easy for the user to understand, for example, returning JSON, XML or other formats, which are not specifically limited here.

[0149] It should be noted that, after the present application completes data reading of a page of a to-be-filtered area by scanning, the to-be-filtered area will be further updated until all target areas of the target data table are judged and complete data is read.

[0150] Thus, the present invention proposes an efficient composite index to filter predicate conditions (i.e., query condition data), wherein the adaptive multi-key Bloom filter based on the number of fields at the Extent level and the Extent size perception, and the preset sparse index of the Extent-Page double-layer filtering are both low-cost and efficient, and each filtering operation of these two indexes has a time complexity of O(1), which means that the time overhead of each filtering operation is extremely short, and each successful filtering may save several to thousands of data pages of disk reading and network overhead, which can greatly speed up the read operation. At the same time, the composite index at the Extent level in this application does not need to introduce the Extent level lock, but only needs to rely on the original data page level lock. In this way, because the S lock at the data page level is obtained, it does not affect the modification of other pages at the Extent level. And under the resource pooling architecture, the composite index will not be affected by the transaction visibility during use, and there is no need to specially consider the version visibility problem when using it.

[0151] It should be noted that the composite index in this application can filter fields of any data type and can be established on any field without any special restrictions when used. The function of this application only relies on segment page storage, has no other dependencies, and does not conflict with existing functions. This function is directly enabled through parameters in the table creation SQL statement when used. In summary, the present invention has good ease of use.

[0152] For example, assume that the statement corresponding to the read request is "select * from tbl where colum1<-80 and colum4>750". Figure 4 , Fig. 7A and Figure 7B, the target area in the target data table tbl includes extent0 and extent1. When pre-filtering, first read the index page of extent0. From the condition colum1<-80, filtering in the first sparse index based on the area level of extent0, it can be seen that there is no data that meets the requirements in extent0, so the data page scanning of this extent can be skipped directly. Further, read the index page of extent1. From the first sparse index based on the area level of extent1, it can be seen that extent1 contains data that meets the requirements. At this time, the second sparse index based on the page level corresponding to page10 to page16 in extent1 can be compared in sequence. It can be known that page10 and page16 contain data that meets the conditions, that is, the data field range of colum1 in page10 is [-90,-70], including the data of "colum1<-80", and the data field range of colum4 in page16 is [701,810], including the data of "colum4>750". Further, the data in page10 and page16 can be scanned and read, and other pages can be skipped.

[0153] It should be noted that the embodiment of the present application is used to perform fast existence judgment on fields of all value types. The present application establishes a preset double-layer sparse index on the field specified by the user. The sparse index maintains the min and max values ​​of the field at the Extent and Page levels. Min takes the minimum value among all versions, and max takes the maximum value among all versions, which can avoid the problem of version visibility. When filtering, first use the first sparse index at the Extent level to filter. If the filter result is no, all pages contained in the Extent are skipped. If the filter result is yes, use the second sparse index at the Page level to filter when each data page is actually read. This method has no restrictions on the field value type and is more friendly to range-type predicate conditions.

[0154] like Fig.13 As shown, Fig.13 It is a specific flow chart of the data reading method provided by the embodiment of the present application. Taking a standby node sending a read request to a master node as an example, the data reading method may specifically include steps S1301 to S1310.

[0155] Step S1301, start;

[0156] Step S1302, receiving a read request;

[0157] Step S1303, determining a target area list involved (equivalent to determining the target area list based on the target data table in the read request);

[0158] Step S1304, selecting a to-be-filtered area from a plurality of target areas, and requesting a target index page corresponding to the to-be-filtered area;

[0159] Step S1305, determining whether the index page memory is hit (equivalent to detecting that the corresponding memory contains index pages of multiple target areas), if not, executing step S1306, if yes, executing step S1307;

[0160] Step S1306, requesting an index page through a distributed memory service module;

[0161] Step S1307, judging whether the extent-level index filtering based on the to-be-filtered area contains data that meets the requirements (equivalent to judging the area-level filtering result of area-level filtering of the query predicate condition based on the preset Bloom filter corresponding to the to-be-filtered area and the first sparse index at the area level), if yes, that is, it contains, executing step S1308; if not, executing step S1313;

[0162] Step S1308, determine whether there is an available page-level sparse index (equivalent to determining whether the table-level composite index corresponding to the to-be-filtered area contains the second sparse index to be filtered). If so, that is, it is contained, and step S1309 is executed; if not, that is, it is not contained, and step S1310 is executed;

[0163] Step S1309, determining whether the page-level sparse index filters out data that meets the requirements, if so, that is, it is included, and executing step S1310; if not, that is, it is not included, and executing step S1309;

[0164] Step S1310, determining whether the data page memory is hit (equivalent to the corresponding memory containing the data page that meets the query predicate condition), if yes, that is, it is contained, executing step S1312; if no, that is, it is not contained, executing step S1311;

[0165] Step S1311, requesting a data page through a distributed memory service module;

[0166] Step S1312, scanning the hit pages;

[0167] Step S1313: the page of the current area to be filtered or the area to be filtered is processed, and the page of the next area to be filtered or the area to be filtered is processed.

[0168] The data reading method provided in the embodiment of the present application optimizes the segment page storage, that is, constructs a predicate pre-filtering framework based on the composite index under the segment page storage. By constructing an adaptive multi-key Bloom filter that is aware of the number of fields at the Extent level and the Extent size to perform page filtering, the sparse index of the field used by the predicate condition can be quickly located in the memory. In addition, the bitarray size used for the Bloom filter can be determined based on the number of fields specified by the user for establishing the Bloom filter and the Extent size in which it is located, so that the constructed preset Bloom filter is more in line with the user's needs and effectively improves the efficiency of data screening. Therefore, the present application can avoid unnecessary network interactions between the primary and standby nodes in the OLAP scenario, improve the read performance of the database node, and thus improve the efficiency of data reading.

[0169] See also Fig.14 , Fig.14 : is a schematic diagram of the structure of a data reading device provided in an embodiment of the present application. The device can implement the data reading method of the above embodiment. The device may specifically include:

[0170] An acquisition module 1410 is used to acquire a region size, a preset index field number, and a preset index type of a candidate region in an initial data table, wherein the initial data table includes multiple candidate regions, and the candidate regions include multiple candidate pages;

[0171] A page number determination module 1420, configured to determine the first number of pages occupied by the table-level composite index corresponding to the candidate zone based on the zone size, the preset number of index fields, and the preset index type;

[0172] A page division module 1430 is used to divide the pages included in the candidate area based on the first number of pages to determine index pages and data pages of the candidate area, where the number of index pages is the first number of pages;

[0173] The composite index construction module 1440 is used to construct a table-level composite index based on a preset Bloom filter and a preset double-layer sparse index, and store the table-level composite index in an index page; the preset Bloom filter is a pre-constructed Bloom filter based on the zone level, the preset double-layer sparse index includes a first sparse index based on the zone level and a second sparse index based on the page level, the first sparse index is used to indicate a data index field of a candidate zone, and the second sparse index is used to indicate a data index field of each candidate page in the candidate zone;

[0174] The storage module 1450 is used to store the field data of all candidate pages in the candidate area into the data page, and to construct the target area structure corresponding to the candidate area based on the stored index page and data page;

[0175] The data reading module 1460 is used to construct a target data table based on multiple target area structures of the same initial data table, and read data based on the target data table.

[0176] It should be noted that the data reading device of the embodiment of the present application is used to implement the data reading method of the above embodiment. The data reading device of the embodiment of the present application corresponds to the above data reading method. Please refer to the above data reading method for the specific processing process, which will not be repeated here.

[0177] The embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the data reading method of the embodiment of the present application when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0178] See also Fig.15 , Fig.15 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0179] The processor 1510 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0180] The memory 1520 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1520 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1520, and the processor 1510 calls and executes the data reading method of the embodiment of this application;

[0181] Input / output interface 1530, used to implement information input and output;

[0182] Communication interface 1540, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0183] bus 1550 , which transmits information between the various components of the device (e.g., processor 1510 , memory 1520 , input / output interface 1530 , and communication interface 1540 );

[0184] The processor 1510 , the memory 1520 , the input / output interface 1530 , and the communication interface 1540 are connected to each other in communication within the device via the bus 1550 .

[0185] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program is used to enable a computer to execute the data reading method in the above embodiment.

[0186] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0187] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0188] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0189] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0191] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0192] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0193] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0194] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0196] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0197] The preferred embodiments of the present invention are described with reference to the accompanying drawings, but the scope of the present invention is not limited by the above. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A data reading method, characterized in that: The method comprises: Obtaining a region size, a preset number of index fields, and a preset index type of a candidate region in an initial data table, wherein the initial data table includes a plurality of candidate regions, and the candidate region includes a plurality of candidate pages; Determine the first number of pages occupied by the table-level composite index corresponding to the candidate zone based on the zone size, the preset number of index fields, and the preset index type; Dividing the pages included in the candidate area into pages based on the first number of pages to determine index pages and data pages of the candidate area, where the number of index pages is the first number of pages; The table-level composite index is constructed based on a preset Bloom filter and a preset double-layer sparse index, and the table-level composite index is stored in the index page; the preset Bloom filter is a pre-constructed Bloom filter based on the zone level, the preset double-layer sparse index includes a first sparse index based on the zone level and a second sparse index based on the page level, the first sparse index is used to indicate the data index field of the candidate zone, and the second sparse index is used to indicate the data index field of each candidate page in the candidate zone; storing the field data of all the candidate pages in the candidate area into the data page, and constructing a target area structure corresponding to the candidate area based on the stored index page and the data page; A target data table is constructed based on the multiple target area structures of the same initial data table, and data is read based on the target data table.

2. The method according to claim 1, characterized in that The step of reading data based on the target data table includes: receiving a read request, wherein the read request includes the target data table and a query predicate condition; Determine a target area list based on the target data table, the target area list including a plurality of target areas; Selecting a to-be-filtered area from the plurality of target areas, and acquiring a target index page corresponding to the to-be-filtered area, wherein the target index page stores the table-level composite index corresponding to the to-be-filtered area; Pre-filtering the query predicate condition based on the table-level composite index corresponding to the to-be-filtered area to obtain a composite filtering result; If the composite filtering result indicates that the area to be filtered contains data that meets the query predicate condition, a page query is performed based on the composite filtering result, the query predicate condition and the data page corresponding to the area to be filtered to determine a page to be read; Data reading is performed by scanning the read page.

3. The method according to claim 2, characterized in that The pre-filtering of the query predicate condition based on the table-level composite index corresponding to the to-be-filtered area to obtain a composite filtering result includes: Performing zone-level filtering on the query predicate condition based on the preset Bloom filter corresponding to the to-be-filtered zone and the first sparse index based on the zone level to obtain a zone-level filtering result; If the zone-level filtering result indicates that the zone to be filtered contains data that meets the query predicate condition, a page-level sparse index detection is performed on the table-level composite index corresponding to the zone to be filtered based on the query predicate condition to obtain a page-level sparse index detection result; If the page-level sparse index detection result indicates that the table-level composite index corresponding to the area to be filtered contains the second sparse index to be filtered, performing page-level filtering on the query predicate condition based on the second sparse index to obtain the page-level filtering result; The composite filtering result is determined based on the page-level filtering results corresponding to all the second sparse indexes to be filtered.

4. The method according to claim 3, characterized in that If the zone-level filtering result indicates that the to-be-filtered zone contains data that meets the query predicate condition, before performing page-level sparse index detection on the table-level composite index corresponding to the to-be-filtered zone based on the query predicate condition and obtaining the page-level sparse index detection result, the method further includes: If the zone-level filtering result indicates that the zone to be filtered does not contain data that meets the query predicate condition, updating the zone to be filtered based on the plurality of target zones; The query predicate condition is pre-filtered according to the updated table-level composite index corresponding to the to-be-filtered area to obtain a composite filtering result until all the target areas are pre-filtered.

5. The method according to claim 2, characterized in that: Performing a page query based on the composite filtering result, the query predicate condition, and the data page corresponding to the to-be-filtered area to determine a page to be read includes: If the composite filtering result indicates that data that meets the query predicate condition is matched in the second sparse index of the to-be-filtered area, and the corresponding memory contains pages that meet the query predicate condition, the read page is determined from the data pages based on the matched second sparse index; or, If the composite filtering result indicates that data that meets the query predicate condition is matched in the second sparse index of the area to be filtered, and the corresponding memory does not contain pages that meet the query predicate condition, a data page request is sent to the corresponding distributed memory service module, and the read page returned by the distributed memory service module is received.

6. The method according to claim 2, characterized in that The step of obtaining a target index page corresponding to the area to be filtered includes: If it is detected that the corresponding memory contains multiple index pages of the target area, the target index page corresponding to the area to be filtered is obtained; or If it is detected that the corresponding memory does not contain multiple index pages of the target area, an index page request is sent to the corresponding distributed memory service module, and the target index page corresponding to the to-be-filtered area returned by the distributed memory service module is received.

7. The method according to any one of claims 1 to 6, characterized in that: The preset Bloom filter in the candidate area is constructed in the following way: Performing bit calculation based on the zone size, the number of fields of the table-level composite index, the unit page storage size, and preset optimization parameters to obtain a target bit number of the preset Bloom filter; The preset Bloom filter corresponding to the candidate area is constructed based on the shared bit array of the target number of bits and a preset hash function.

8. A data reading device, characterized in that: The device comprises: An acquisition module, used to acquire a region size, a preset number of index fields, and a preset index type of a candidate region in an initial data table, wherein the initial data table includes a plurality of candidate regions, and the candidate region includes a plurality of candidate pages; A page number determination module, configured to determine the first number of pages occupied by the table-level composite index corresponding to the candidate zone based on the zone size, the preset number of index fields and the preset index type; a page division module, configured to divide the pages included in the candidate area into pages based on the first number of pages, and determine index pages and data pages of the candidate area, wherein the number of the index pages is the first number of pages; A composite index construction module, configured to construct the table-level composite index based on a preset Bloom filter and a preset double-layer sparse index, and store the table-level composite index in the index page; the preset Bloom filter is a pre-constructed zone-level Bloom filter, the preset double-layer sparse index includes a first zone-level sparse index and a second page-level sparse index, the first sparse index is used to indicate a data index field of the candidate zone, and the second sparse index is used to indicate a data index field of each candidate page in the candidate zone; A storage module, used for storing the field data of all the candidate pages in the candidate area into the data page, and constructing a target area structure corresponding to the candidate area based on the stored index page and the data page; A data reading module is used to construct a target data table based on a plurality of target area structures of the same initial data table, and to read data based on the target data table.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images