Data retrieval method and device, equipment and storage medium
By distinguishing the intervals of high and low duplicate values in the database, using the B+ tree index structure and sequential scanning strategy, the problem of low query efficiency in the duplicate value area of B tree index and bitmap index is solved, and more efficient data retrieval is achieved.
Patent Information
- Application Number
- CN202510361575.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-25
- Publication Date
- 2025-07-22
AI Technical Summary
Existing B-tree indexes and bitmap indexes cannot filter more data when querying areas with more duplicate values in the database, resulting in a small difference in the amount of read and return data, which cannot improve query efficiency. Instead, it increases the overhead of scanning inodes and reduces retrieval efficiency.
By distinguishing between high-repetition value intervals and low-repetition value intervals, different data retrieval strategies are adopted: build a B+ tree index structure for low-repetition value intervals for querying, and sequentially scan and store data for high-repetition value intervals.
Efficiently locate and query data in the database, reduce query response time, avoid additional overhead caused by scanning inodes, significantly improve query efficiency, and improve database performance and stability.
Smart Images
Figure CN120353834A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of data processing, and in particular, to a data retrieval method, apparatus, device, and storage medium. Background Art
[0002] In modern information systems, as the core component for data storage and management, the performance of a database directly affects the efficiency of business systems and the user experience. Index technology is a key means to improve the query performance of databases. Among them, B-tree indexes and bitmap indexes have become widely used index structures due to their excellent data search capabilities.
[0003] However, when the numerical distribution of data stored in a database is unbalanced, for regions with a large number of duplicate values in the database, when querying using these two index structures, namely B-tree indexes and bitmap indexes, a large amount of data cannot be filtered, resulting in a small difference between the amount of data read and returned. In this case, using these two indexes not only fails to improve the query efficiency, but instead increases the overhead due to scanning index nodes, reducing the retrieval efficiency. Summary of the Invention
[0004] Embodiments of the present disclosure provide a data retrieval method, apparatus, device, and storage medium to solve the problem in the related art that for regions with a large number of duplicate values in a database, when querying using these two index structures, namely B-tree indexes and bitmap indexes, a large amount of data cannot be filtered, resulting in a small difference between the amount of data read and returned, not only failing to improve the query efficiency, but instead increasing the overhead due to scanning index nodes and reducing the retrieval efficiency.
[0005] In a first aspect, embodiments of the present disclosure provide a data retrieval method, the method comprising: Receiving a data retrieval request, and parsing the data retrieval request to obtain a target retrieval value corresponding to the data retrieval request; Determining whether the target retrieval value belongs to a pre-determined high-duplicate value interval or a low-duplicate value interval; wherein, the degree of duplication of values in the high-duplicate value interval is higher than a preset duplication threshold; the degree of duplication of values in the low-duplicate value interval is not higher than the duplication threshold; In the case where the target retrieval value belongs to the low-duplicate value interval, constructing a B+-tree index structure, and querying the target retrieval value through the constructed B+-tree index structure; in the case where the target retrieval value belongs to the high-duplicate value interval, querying the target retrieval value by sequentially scanning the stored data.
[0006] In a second aspect, embodiments of the present disclosure provide a data retrieval apparatus, the apparatus comprising: A receiving module, configured to receive a data retrieval request, and parse the data retrieval request to obtain a target retrieval value corresponding to the data retrieval request; A determination module, configured to determine whether the target retrieval value belongs to a pre-determined high repetition value interval or a low repetition value interval; wherein, the repetition degree of the values in the high repetition value interval is higher than a preset repetition degree threshold; the repetition degree of the values in the low repetition value interval is not higher than the repetition degree threshold. A retrieval module, configured to, when the target retrieval value belongs to the low repetition value interval, construct a B+ tree index structure and query the target retrieval value through the constructed B+ tree index structure; when the target retrieval value belongs to the high repetition value interval, query the target retrieval value by sequentially scanning the stored data.
[0007] In a third aspect, an embodiment of the present disclosure provides a data retrieval device, including: a processor; and a memory configured to store computer-executable instructions, and the computer-executable instructions, when executed, cause the processor to implement the steps of the method described in the first aspect above.
[0008] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method described in the first aspect above.
[0009] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, which includes a computer program, and the computer program, when executed by a processor, implements the steps of the method described in the first aspect above.
[0010] The above at least one technical solution provided by the embodiments of the present invention can achieve the following technical effects: In the embodiments of the present invention, after receiving a data retrieval request, the data retrieval request can be parsed to obtain the target retrieval value corresponding to the data retrieval request, and then it can be determined whether the target retrieval value belongs to a pre-determined high repetition value interval or a low repetition value interval; wherein, the repetition degree of the values in the high repetition value interval is higher than a preset repetition degree threshold; and the repetition degree of the values in the low repetition value interval is not higher than the repetition degree threshold. After the determination, if the target retrieval value belongs to the low repetition value interval, a B+ tree index structure can be constructed and the target retrieval value can be queried through the constructed B+ tree index structure; if the target retrieval value belongs to the high repetition value interval, the target retrieval value can be queried by sequentially scanning the stored data.
[0011] In the embodiments of the present invention, when faced with a situation where there are many duplicate values in the database, different data retrieval strategies can be adopted by precisely distinguishing high and low duplicate value intervals. Specifically, for the low duplicate value interval, a B+ tree index structure is constructed to quickly locate and query the target retrieval value, reducing the query response time and providing users with a more efficient data retrieval service. For the high duplicate value interval, the traditional index scanning method is abandoned, and a strategy of sequentially scanning the stored data is adopted, avoiding the additional overhead caused by scanning index nodes. Since the data in the high duplicate value interval itself has poor filterability, sequential scanning can directly obtain the data, avoiding frequent reciprocating access between the index nodes and the data storage pages, thus significantly improving the query efficiency and effectively solving the problem in the related art that for regions with many duplicate values in the database, the B-tree index and bitmap index cannot filter much data during query, resulting in a small difference between the amount of data read and returned, not only unable to improve the query efficiency, but also reducing the retrieval efficiency due to the increased overhead of scanning index nodes. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the technical solutions in one or more embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. Figure 1 It is a schematic flowchart of a data retrieval method provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of a data index structure provided by an embodiment of the present invention; Figure 3 It is a schematic diagram of the module composition of a data retrieval device 300 provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of the hardware structure of a data retrieval device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0013] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present disclosure, in order to make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the specific embodiments and corresponding drawings of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
[0014] The technical solutions provided by various embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.
[0015] See also Figure 1 , Figure 1 A flow chart of a data retrieval method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method comprises the following steps: Step 102: Receive a data retrieval request, and parse the data retrieval request to obtain a target retrieval value corresponding to the data retrieval request.
[0016] Step 104: Determine whether the target search value belongs to a predetermined high repetition value interval or a low repetition value interval; wherein the repetition of the value in the high repetition value interval is higher than a preset repetition threshold; and the repetition of the value in the low repetition value interval is not higher than the repetition threshold.
[0017] Step 106: When the target search value belongs to a low repetition value interval, a B+ tree index structure is constructed, and the target search value is queried through the constructed B+ tree index structure; when the target search value belongs to a high repetition value interval, the target search value is queried by sequentially scanning the stored data.
[0018] In the embodiment of the present invention, a data retrieval request may be received, and the received data retrieval request may be parsed to obtain a target retrieval value corresponding to the data retrieval request, that is, a value that the data retrieval request wants to retrieve.
[0019] After obtaining the target search value, it can be determined whether the target search value belongs to a predetermined high repetition value interval or a low repetition value interval.
[0020] In one embodiment of the present invention, a high repetition value interval and a low repetition value interval may be predetermined, wherein the repetition value of the values in the high repetition value interval may be higher than a preset repetition threshold, and the repetition of the values in the low repetition value interval may not be higher than the preset repetition threshold.
[0021] In one example, the repetition threshold may include a repetition number threshold and a repetition ratio threshold. When determining a high repetition value interval and a low repetition value interval, the currently stored data may be first divided into a plurality of data intervals, and then, for any of the divided data intervals, it may be determined whether its value satisfies a preset repetition condition. The preset repetition condition may be: the number of identical values is greater than a preset repetition number threshold; the ratio of the number of identical values to the total number of values in the data interval to which it belongs is greater than a preset repetition ratio threshold.
[0022] For any data interval, if its value meets at least one of the above preset duplication degree conditions, it can be determined that the any data interval is a high-duplication value interval; otherwise, it can be determined that the any data interval is a low-duplication value interval, that is, if its value does not meet the above preset duplication degree conditions, it is determined that the any data interval is a low-duplication value interval.
[0023] After determining whether the target retrieval data belongs to a high-duplication value interval or a low-duplication value interval, different retrieval strategies can be determined according to the difference in the data interval to which the target retrieval data belongs. Specifically, when the target retrieval data belongs to a low-duplication value interval, a B+ tree index structure can be constructed, and the target retrieval value can be queried through the constructed B+ tree index structure; while when the target retrieval value belongs to a high-duplication value interval, the index structure can be skipped, and the target retrieval value can be directly queried by sequentially scanning the currently stored data.
[0024] In the embodiment of the present invention, when the target retrieval value to be queried belongs to a low-duplication value interval, the database can construct a B+ tree index structure to quickly locate and query data through the B+ tree index structure; if the target retrieval value to be queried falls within a high-duplication value interval, when constructing a B+ tree index structure and querying data through the B+ tree index structure, there is almost no difference in the amount of data read and the number of returns. The database needs to abandon using the B+ tree structure and choose to directly use the method of sequentially scanning the data for data query. Since there is no need to repeatedly access the B+ tree nodes and the actual data storage pages, the actual data query performance will be improved compared to querying through the B+ tree structure.
[0025] In an example, the values stored in the database can be as shown in Table 1: Table 1
[0026] When dividing the data intervals of the data stored in Table 1, it can be divided into three data intervals, which are: the first data interval where the value is not greater than 0, the second data interval where the value is greater than 0 and not greater than 1000, and the third data interval where the value is greater than 1000. Then, the duplication degree thresholds can be set to include a duplication quantity threshold and a duplication ratio threshold, and the duplication quantity threshold is 100, and the duplication ratio threshold is 10%. Then, it can be determined that the first data interval is a high-duplication interval, and the second data interval and the third data interval are low-duplication intervals. When retrieving data, when the target retrieval data to be retrieved belongs to the first data interval, it can be directly queried by sequentially scanning the stored data; while when the target retrieval data to be retrieved belongs to the second data interval and the third data interval, a B+ tree index structure can be constructed, and the query can be performed through the constructed B+ tree index structure. Therefore, it can be obtained as Figure 2Schematic diagram of the data index structure shown
[0027] In one example, considering that the users of the database have in-depth knowledge of the actual distribution of data values and can make relatively accurate predictions, and even know specifically which actual distributions are uneven during the design and planning, that is, which data has extremely high duplicate values. Therefore, when creating an index in the database, it is possible to directly specify which value range does not require index maintenance through the set declarative syntax, that is, it is possible to directly specify the low-duplicate-value interval for which an index needs to be created and the high-duplicate-value interval for which an index does not need to be created.
[0028] For example, the declarative syntax can be: create index “index name” on “table”(“field 1”<0 ignore)) Alternatively, during use, when it is found that the data values appear as described in this application, the low-duplicate-value interval and the high-duplicate-value interval can be dynamically adjusted: alter index “index name”(“field 1”<5 ignore)) In an embodiment of the present invention, it is also possible to receive a data modification request for stored data, where the data modification request may include one of the following: a data insertion request, a data deletion request, and a data update request.
[0029] After receiving the data modification request, the data modification request can be parsed to obtain the target modification value corresponding to the data modification request, where the target modification value is specifically the value that needs to be modified this time. Then, it can be determined whether the target modification value belongs to the high-duplicate-value interval or the low-duplicate-value interval determined above.
[0030] If the target modification value belongs to the low-duplicate-value interval, the target index node in the B+ tree index structure can be determined according to the target modification value, and the target index node and the stored data are modified. That is, both the index and the stored data need to be modified; when the target modification value belongs to the high-duplicate-value interval, the stored data is modified according to the target modification value.
[0031] In an embodiment of the present invention, for high-duplicate values, during each modification operation, such as INSERT, UPDATE, and DELETE operations, the index node change can be directly skipped, and only the actual data storage content needs to be maintained. Because the data values are highly repetitive, the most critical sorting ability of the index is not reflected, so these index node maintenance operations are all ineffective. Omitting these actions can greatly improve efficiency and bring technical benefits in the following two scenarios: 1. Canceling all these inode change operations can most intuitively reduce unnecessary operations and save a large amount of valuable CPU, memory, or IO resources.
[0032] 2. To ensure data consistency, the database implements a large number of internal coordination mechanisms. Among them, the use of mutex locks is particularly crucial. In the embodiments of the present invention, for the part of data with high duplicate values, these mutex lock-related operations can be omitted. For example, when changing an index leaf node, the database internally locks the node and releases it after the change. In the case of high duplicate values, even if the content of an index page node is the same, but only a new record pointer pointing to a specific data page needs to be added, this locking operation still persists. If multiple records are operated on simultaneously, a mutex waiting problem will occur. Since the content of the high duplicate value part is cancelled in the new structure, the related operations and the conflicts caused by these operations can be eliminated, so the performance of the database and the stability of the database product will be greatly improved.
[0033] In an embodiment of the present invention, when the data modification request is a data insertion request, it can be determined whether the duplicate degree of the values in the low duplicate value interval is higher than the above-mentioned preset duplicate degree threshold, and when the duplicate degree of the values in the low duplicate value interval is higher than the duplicate degree threshold, the low duplicate value interval is adjusted to a high duplicate value interval.
[0034] Correspondingly, when the data modification request is a data deletion request, it can be determined whether the duplicate degree of the values in the high duplicate value interval is not higher than the above-mentioned preset duplicate degree threshold, and when the duplicate degree of the values in the high duplicate value interval is not higher than the duplicate degree threshold, the high duplicate value interval is adjusted to a low duplicate value interval.
[0035] In addition, the embodiments of the present invention can also dynamically adjust the high duplicate value interval and the low duplicate value interval according to a preset dynamic update period. Specifically, when the duplicate degree of the values in the high duplicate value interval is not higher than the preset duplicate degree threshold, the high duplicate value interval is adjusted to a low duplicate value interval; and when the duplicate degree of the values in the low duplicate value interval is higher than the preset duplicate degree threshold, the low duplicate value interval is adjusted to a high duplicate value interval.
[0036] Through the above steps, it is possible to dynamically adjust the high - repetition - degree interval and the low - repetition - degree interval, thereby significantly improving the performance and adaptability of the database. As the data continuously changes, the repetition degree of the data is not fixed. By dynamically adjusting the interval, it can be ensured that the database always adopts the most suitable retrieval strategy. When an interval originally belongs to the low - repetition - degree interval and its repetition degree rises to meet the high - repetition - degree interval standard with data updates, after timely adjustment, when querying the data in this interval, the database can abandon the mismatched B + tree index structure and adopt sequential scanning, avoiding the overhead caused by maintaining a complex index structure and improving the query efficiency. Conversely, if the repetition degree of an interval decreases and it is adjusted from the high - repetition - degree interval to the low - repetition - degree interval, the database can reuse the B + tree index structure for efficient retrieval again. This dynamic adjustment mechanism enables the database to flexibly respond to data changes, effectively balance the index maintenance cost and query efficiency, improve the overall operating efficiency and stability of the database, and meet diverse business requirements.
[0037] In the embodiment of the present invention, after receiving a data retrieval request, the data retrieval request can be parsed to obtain the target retrieval value corresponding to the data retrieval request. Then, it can be determined whether the target retrieval value belongs to a pre - determined high - repetition - value interval or a low - repetition - value interval. Among them, the repetition degree of the values in the high - repetition - value interval is higher than a preset repetition - degree threshold, while the repetition degree of the values in the low - repetition - value interval is not higher than this repetition - degree threshold. After determination, if the target retrieval value belongs to the low - repetition - value interval, a B + tree index structure can be constructed and the target retrieval value can be queried through the constructed B + tree index structure. If the target retrieval value belongs to the high - repetition - value interval, the target retrieval value can be queried by sequentially scanning the stored data.
[0038] In the embodiment of the present invention, when faced with a situation where there are many duplicate values in the database, it is possible to accurately distinguish between high - and low - repetition - value intervals and adopt different data retrieval strategies. Specifically, for the low - repetition - value interval, a B + tree index structure is constructed to quickly locate and query the target retrieval value, reducing the query response time and providing a more efficient data retrieval service for users. For the high - repetition - value interval, the traditional index scanning method is abandoned, and the strategy of sequentially scanning the stored data is adopted, avoiding the additional overhead caused by scanning index nodes. Since the data in the high - repetition - value interval itself has poor filterability, sequential scanning can directly obtain the data, avoiding frequent back - and - forth access between the index nodes and the data storage pages, thereby significantly improving the query efficiency and effectively solving the problem in the related art that for the area with many duplicate values in the database, the B - tree index and the bitmap index cannot filter a large amount of data during query, resulting in a small difference in the amount of data read and returned, not only unable to improve the query efficiency but also reducing the retrieval efficiency due to the increased overhead of scanning index nodes.
[0039] Corresponding to the above data retrieval method, an embodiment of the present invention further provides a data retrieval device, Figure 3 A schematic diagram of the module composition of the data retrieval device 300 provided in an embodiment of the present invention is shown in FIG. Figure 3 As shown, the data retrieval device 300 includes: The receiving module 301 is used to receive a data retrieval request, and parse the data retrieval request to obtain a target retrieval value corresponding to the data retrieval request; Determination module 302, used to determine whether the target search value belongs to a predetermined high repetition value interval or a low repetition value interval; wherein the repetition of the value in the high repetition value interval is higher than a preset repetition value threshold; and the repetition of the value in the low repetition value interval is not higher than the repetition value threshold; The retrieval module 303 is used to construct a B+ tree index structure when the target retrieval value belongs to the low repetition value interval, and query the target retrieval value through the constructed B+ tree index structure; when the target retrieval value belongs to the high repetition value interval, query the target retrieval value by sequentially scanning the stored data.
[0040] Optionally, the repetition threshold includes a repetition quantity threshold and a repetition ratio threshold; the method further includes ( Figure 3 not shown): A division module 304, used for dividing the stored data into a plurality of data intervals before determining whether the target search value belongs to a predetermined high repetition value interval or a low repetition value interval; The first judgment module 305 is used to determine, for any data interval, that the data interval is a high repetition value interval when the value satisfies at least one preset repetition condition; and to determine that the data interval is a low repetition value interval when the value does not satisfy the repetition condition; wherein the repetition condition is: the number of identical values is greater than the repetition number threshold; the ratio of the number of identical values to the total number of values in the data interval to which it belongs is greater than the repetition ratio threshold.
[0041] Optionally, the method further comprises ( Figure 3 not shown): The acquisition module 306 is used to construct a B+ tree index structure when the target search value belongs to the low repetition value interval, and query the target search value through the constructed B+ tree index structure; when the target search value belongs to the high repetition value interval, after querying the target search value by sequentially scanning the stored data, receive a data modification request; the data modification request includes one of the following: a data insertion request, a data deletion request, and a data update request; A parsing module 307, configured to parse the data modification request to obtain a target modification value corresponding to the data modification request; A second determination module 308, configured to determine whether the target modification value belongs to the high repetition value range or the low repetition value range; A modification module 309, configured to, when the target modification value belongs to the low repetition value range, determine a target index node in the B+ tree index structure according to the target modification value, and modify the target index node and the stored data; when the target modification value belongs to the high repetition value range, modify the stored data according to the target modification value.
[0042] Optionally, the method further includes ( Figure 3 not shown in A first adjustment module 310, configured to, when the target modification value belongs to the low repetition value range, determine a target index node in the B+ tree index structure according to the target modification value, and modify the target index node and the stored data; when the target modification value belongs to the high repetition value range, after modifying the stored data according to the target modification value, when the data modification request is a data insertion request, determine whether the repetition degree of the values in the low repetition value range is higher than the repetition degree threshold, and when the repetition degree of the values in the low repetition value range is higher than the repetition degree threshold, adjust the low repetition value range to a high repetition value range; A second adjustment module 311, configured to, when the data modification request is a data deletion request, determine whether the repetition degree of the values in the high repetition value range is not higher than the repetition degree threshold, and when the repetition degree of the values in the high repetition value range is not higher than the repetition degree threshold, adjust the high repetition value range to a low repetition value range.
[0043] Optionally, the method further includes ( Figure 3 not shown in A specifying module 312, configured to specify the high repetition value range and the low repetition value range through declarative syntax set manually before determining whether the target retrieval value belongs to a pre-determined high repetition value range or a low repetition value range.
[0044] Optionally, the method further includes ( Figure 3 not shown in The third adjustment module 313 is configured to, when the target retrieval value belongs to the low repetition value range, construct a B+ tree index structure and query the target retrieval value through the constructed B+ tree index structure; when the target retrieval value belongs to the high repetition value range, query the target retrieval value by sequentially scanning the stored data, and then dynamically adjust the high repetition value range and the low repetition value range according to a preset dynamic update period. The third adjustment module 313 is configured to: When the repetition degree of the values in the high repetition value range is not higher than a preset repetition degree threshold, adjust the high repetition value range to a low repetition value range; when the repetition degree of the values in the low repetition value range is higher than the preset repetition degree threshold, adjust the low repetition value range to a high repetition value range.
[0045] In an embodiment of the present invention, after receiving a data retrieval request, the data retrieval request can be parsed to obtain a target retrieval value corresponding to the data retrieval request. Then, it can be determined whether the target retrieval value belongs to a pre-determined high repetition value range or a low repetition value range. Among them, the repetition degree of the values in the high repetition value range is higher than a preset repetition degree threshold; while the repetition degree of the values in the low repetition value range is not higher than this repetition degree threshold. After determination, if the target retrieval value belongs to the low repetition value range, a B+ tree index structure can be constructed and the target retrieval value can be queried through the constructed B+ tree index structure; if the target retrieval value belongs to the high repetition value range, the target retrieval value can be queried by sequentially scanning the stored data.
[0046] When facing the situation of a large number of duplicate values in the database, the embodiment of the present invention can accurately distinguish between high and low repetition value ranges and adopt different data retrieval strategies. Specifically, for the low repetition value range, a B+ tree index structure is constructed to quickly locate and query the target retrieval value, reducing the query response time and providing a more efficient data retrieval service for users; while for the high repetition value range, the traditional index scanning method is abandoned and a strategy of sequentially scanning the stored data is adopted, avoiding the additional overhead caused by scanning index nodes. Since the data in the high repetition value range itself has poor filterability, sequential scanning can directly obtain the data, avoiding frequent reciprocating access between the index node and the data storage page, thereby significantly improving the query efficiency and effectively solving the problem in the related art that for the area with a large number of duplicate values in the database, the B-tree index and the bitmap index cannot filter a large amount of data during query, resulting in a small difference in the amount of read and returned data, not only unable to improve the query efficiency, but also reducing the retrieval efficiency due to the increased overhead of scanning index nodes.
[0047] Corresponding to the above data retrieval method, the embodiment of the present invention further provides a data retrieval device. Figure 4Schematic diagram of the hardware structure of a data retrieval device provided by an embodiment of the present invention.
[0048] The data retrieval device may be a terminal device or a server, etc. for retrieving data provided in the above embodiment.
[0049] The data retrieval device may vary greatly due to different configurations or performances, and may include one or more processors 401 and a memory 402. One or more storage applications or data may be stored in the memory 402. Among them, the memory 402 may be transient storage or persistent storage. The application programs stored in the memory 402 may include one or more modules (not shown in the figure), and each module may include a series of computer-executable instructions in the data retrieval device. Further, the processor 401 may be set to communicate with the memory 402 and execute a series of computer-executable instructions in the memory 402 on the data retrieval device. The data retrieval device may also include one or more power supplies 403, one or more wired or wireless network interfaces 404, one or more input / output interfaces 405, and one or more keyboards 406.
[0050] Specifically, in this embodiment, the data retrieval device includes a memory and one or more programs, where one or more programs are stored in the memory, and one or more programs may include one or more modules, and each module may include a series of computer-executable instructions in the data retrieval device and is configured to be executed by one or more processors in the above embodiment.
[0051] In an embodiment of the present invention, after receiving a data retrieval request, the data retrieval request may be parsed to obtain a target retrieval value corresponding to the data retrieval request. Then, it may be determined whether the target retrieval value belongs to a pre-determined high-repeat value interval or a low-repeat value interval; where the repeatability of the values in the high-repeat value interval is higher than a preset repeatability threshold; and the repeatability of the values in the low-repeat value interval is not higher than the repeatability threshold. After the determination, if the target retrieval value belongs to the low-repeat value interval, a B+ tree index structure may be constructed, and the target retrieval value may be queried through the constructed B+ tree index structure; if the target retrieval value belongs to the high-repeat value interval, the target retrieval value may be queried by sequentially scanning the stored data.
[0052] In the embodiments of the present invention, when faced with a situation where there are many duplicate values in the database, it is possible to accurately distinguish between high and low duplicate value intervals and adopt different data retrieval strategies. Specifically, for the low duplicate value interval, a B+ tree index structure is constructed to quickly locate and query the target retrieval value, reducing the query response time and providing users with a more efficient data retrieval service. For the high duplicate value interval, the traditional index scan method is abandoned, and a strategy of sequentially scanning the stored data is adopted, avoiding the additional overhead caused by scanning index nodes. Since the data in the high duplicate value interval itself has poor filterability, sequential scanning can directly obtain the data, avoiding frequent reciprocating access between the index node and the data storage page, thus significantly improving the query efficiency and effectively solving the problem in the related art that for regions with many duplicate values in the database, the two index structures of B-tree index and bitmap index cannot filter much data during query, resulting in a small difference in the amount of read and returned data, not only unable to improve the query efficiency, but instead reducing the retrieval efficiency due to the increased overhead of scanning index nodes.
[0053] Another embodiment of the present disclosure also provides a computer-readable storage medium for storing computer-executable instructions, which, when executed by a processor, implement the above process.
[0054] The storage medium in the embodiments of the present disclosure can implement each process of the above data retrieval method embodiment and achieve the same effects and functions, which will not be repeated here.
[0055] Another embodiment of the present disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above process.
[0056] The computer program product in the embodiments of the present disclosure can implement each process of the above data retrieval method embodiment and achieve the same effects and functions, which will not be repeated here.
[0057] In each embodiment of the present disclosure, the computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disc, etc.
[0058] In the 1990s, it was obvious to distinguish whether an improvement to a technology was a hardware improvement (e.g., improvement to the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement to the method flow). However, with the development of technology, many improvements to method flows today can be regarded as direct improvements to the hardware circuit structure. Almost all designers obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement to a method flow cannot be implemented with a hardware entity module. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logical function is determined by the customer programming the device. The designer can program by himself to "integrate" a digital system on a PLD, without having to ask a chip manufacturer to design and fabricate a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compiler used in program development and writing. The original code before compilation also has to be written in a specific programming language, which is called a Hardware Description Language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones currently are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that as long as the method flow is slightly logically programmed with the above-mentioned several hardware description languages and programmed into the integrated circuit, it is easy to obtain the hardware circuit that implements the logical method flow.
[0059] The controller can be implemented in any suitable manner. For example, the controller can take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of the controller include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that, in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to make the controller implement the same function in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as either software modules implementing the method or structures within the hardware component.
[0060] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0061] For the convenience of description, when describing the above devices, they are described separately as various units according to their functions. Of course, when implementing the embodiments of the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0062] Those skilled in the art should understand that one or more embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, one or more embodiments of the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.
[0063] This disclosure is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices produce means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.
[0064] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means that implement the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.
[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.
[0066] It should also be noted that the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, commodity or device including the said element.
[0067] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. One or more embodiments of the present disclosure may also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.
[0068] Each embodiment in the present disclosure is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and reference can be made to the corresponding parts of the method embodiments for the relevant content.
[0069] The above description is only for the embodiments of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, various changes and modifications can be made to the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included within the scope of the claims of the present disclosure.
Claims
1. A data retrieval method, characterized in that, The method comprises: Receiving a data retrieval request, and parsing the data retrieval request to obtain a target retrieval value corresponding to the data retrieval request; Determine whether the target search value belongs to a predetermined high repetition value interval or a low repetition value interval; wherein the repetition of the value in the high repetition value interval is higher than a preset repetition value threshold; and the repetition of the value in the low repetition value interval is not higher than the repetition value threshold; When the target retrieval value belongs to the low repetition value interval, a B+ tree index structure is constructed, and the target retrieval value is queried through the constructed B+ tree index structure; when the target retrieval value belongs to the high repetition value interval, the target retrieval value is queried by sequentially scanning the stored data.
2. The method according to claim 1, wherein The repetition threshold includes a repetition number threshold and a repetition ratio threshold; before determining whether the target search value belongs to a predetermined high repetition value interval or a low repetition value interval, the method includes: Divide the stored data into multiple data intervals; For any data interval, when the value satisfies at least one preset repetition condition, the data interval is determined to be a high repetition value interval; when the value does not satisfy the repetition condition, the data interval is determined to be a low repetition value interval; wherein the repetition condition is: the number of identical values is greater than the repetition number threshold; the ratio of the number of identical values to the total number of values in the data interval to which it belongs is greater than the repetition ratio threshold.
3. The method according to claim 1, wherein In the case where the target search value belongs to the low repetition value interval, constructing a B+ tree index structure, and querying the target search value through the constructed B+ tree index structure; When the target search value belongs to the high repetition value interval, after querying the target search value by sequentially scanning the stored data, the method further includes: Receiving a data modification request; the data modification request includes one of the following: a data insertion request, a data deletion request, and a data update request; Parsing the data modification request to obtain a target modification value corresponding to the data modification request; Determining whether the target modified value belongs to the high repetition value interval or the low repetition value interval; When the target modification value belongs to the low repetition value interval, the target index node in the B+ tree index structure is determined according to the target modification value, and the target index node and the stored data are modified; when the target modification value belongs to the high repetition value interval, the stored data is modified according to the target modification value.
4. The method according to claim 3, characterized in that In the case where the target modification value belongs to the low repetition value interval, determining the target index node in the B+ tree index structure according to the target modification value, and modifying the target index node and the stored data; In the case where the target modification value belongs to the high repetition value interval, after modifying the stored data according to the target modification value, the method further includes: When the data modification request is a data insertion request, determine whether the repetition degree of the values in the low repetition value interval is higher than the repetition degree threshold, and in the case where the repetition degree of the values in the low repetition value interval is higher than the repetition degree threshold, adjust the low repetition value interval to a high repetition value interval; When the data modification request is a data deletion request, determine whether the repetition degree of the values in the high repetition value interval is not higher than the repetition degree threshold, and in the case where the repetition degree of the values in the high repetition value interval is not higher than the repetition degree threshold, adjust the high repetition value interval to a low repetition value interval.
5. The method according to claim 1, wherein Before determining whether the target retrieval value belongs to a pre-determined high repetition value interval or a low repetition value interval, the method further includes: Specify the high repetition value interval and the low repetition value interval through declarative syntax set manually.
6. The method according to claim 1, wherein In the case where the target retrieval value belongs to the low repetition value interval, construct a B+ tree index structure and query the target retrieval value through the constructed B+ tree index structure; After querying the target retrieval value by sequentially scanning the stored data in the case where the target retrieval value belongs to the high repetition value interval, the method further includes: Dynamically adjust the high repetition value interval and the low repetition value interval according to a preset dynamic update period; The dynamically adjusting the high repetition value interval and the low repetition value interval includes: In the case where the repetition degree of the values in the high repetition value interval is not higher than the preset repetition degree threshold, adjust the high repetition value interval to a low repetition value interval; in the case where the repetition degree of the values in the low repetition value interval is higher than the preset repetition degree threshold, adjust the low repetition value interval to a high repetition value interval.
7. A data retrieval device, characterized in that, The device includes: A receiving module, configured to receive a data retrieval request, and parse the data retrieval request to obtain a target retrieval value corresponding to the data retrieval request; A determining module, configured to determine whether the target retrieval value belongs to a pre-determined high repetition value interval or a low repetition value interval; wherein, the repetition degree of the values in the high repetition value interval is higher than the preset repetition degree threshold; the repetition degree of the values in the low repetition value interval is not higher than the repetition degree threshold; A retrieval module, configured to construct a B+ tree index structure and query the target retrieval value through the constructed B+ tree index structure in the case where the target retrieval value belongs to the low repetition value interval; query the target retrieval value by sequentially scanning the stored data in the case where the target retrieval value belongs to the high repetition value interval.
8. A data retrieval device, characterized in that, Includes: A memory, a processor, and a computer program stored on the memory and executable on the processor, and when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, characterized in that, The computer program product includes a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6 above.