A method for constructing a data query bitmap index, a data query method and a device

By obtaining and preprocessing data from the database and creating query bitmap indexes, the problem of low data retrieval efficiency under complex search conditions is solved, and fast and efficient data query is achieved.

CN117573678BActive Publication Date: 2024-11-15BEIJING PACTERA JINXIN TECH LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311676114.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-07
Publication Date
2024-11-15
Estimated Expiration
2043-12-07

AI Technical Summary

Technical Problem

The prior art cannot realize rapid data retrieval under complex search conditions, has low retrieval efficiency, and the deployment of distributed systems is complex and cumbersome.

Method used

By obtaining basic business data from the database, preprocessing and filtering out data that cannot be thresholded, determining the mapping threshold for querying basic data, forming an application data table and creating a query bitmap index to achieve fast query.

Benefits of technology

It improves data retrieval efficiency and real-timeness, reduces complex data search processes, and simplifies feedback on data query results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117573678B_ABST
    Figure CN117573678B_ABST
Patent Text Reader

Abstract

The present application provides a method for constructing a data query bitmap index, a data query method and a device, which pre-processes basic business data to obtain query basic data that can be thresholded; for each query basic data, the mapping threshold of the query basic data is determined according to the data attribute of the query basic data; based on each query basic data and the mapping threshold of the query basic data, an application data table is formed; using the mapping threshold of each query basic data, by classifying and processing each query basic data in the application data table, a query bitmap index of the application data table is created, so as to realize the query of the query basic data in the application data table by using the query bitmap index. In this way, the mapping threshold of the query basic data included in the application data table can be used to create a query bitmap index of the application data table, and with the help of the query bitmap index, the query of data can be quickly realized, thereby improving the retrieval efficiency and timeliness of data under large data volumes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for constructing a data query bitmap index, a data query method and a device. Background Art

[0002] In the banking management business scenario, since the number of users involved on the bank side is relatively large, if you want to retrieve users and / or data, you have to retrieve the huge number of customer lists in the database to obtain the relevant data.

[0003] At present, data retrieval is usually achieved by relying on the storage of structured data in a database and using the database's query engine. However, under such a technical route, if complex retrieval conditions are encountered, it is impossible to quickly retrieve data under huge data volumes, and the real-time performance of the retrieval is relatively low. In addition, in distributed file systems, distributed storage of data is usually achieved with the help of the system structure of the distributed system, and then distributed search and analysis engines (such as Elasticsearch) are used for data retrieval. Although this method can improve the retrieval efficiency of data under large data volumes, the implementation steps are relatively cumbersome, the logical structure is relatively complex, and the operation and deployment are also relatively troublesome. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a method for constructing a data query bitmap index, a data query method and a device, so as to solve the problems of low data query efficiency and complicated query process in the prior art.

[0005] The present application provides a method for constructing a data query bitmap index, the method comprising:

[0006] Obtain basic business data from the basic data table pre-stored in the database;

[0007] Preprocessing the basic business data, filtering out the basic business data that cannot be processed by threshold value, and obtaining query basic data that can be processed by threshold value;

[0008] For each query basic data, determining a mapping threshold of the query basic data according to the data attribute of the query basic data;

[0009] Based on each query basic data and a mapping threshold of the query basic data, an application data table is formed;

[0010] By utilizing the mapping threshold of each query basic data and classifying each query basic data in the application data table, a query bitmap index of the application data table is created, so as to implement the query of the query basic data in the application data table using the query bitmap index.

[0011] In a possible implementation manner, determining the mapping threshold of the query basic data according to the data attribute of the query basic data includes:

[0012] Determine the data type of the query basic data according to the data attribute of the query basic data;

[0013] When the data type indicates that the query basic data is numerical data, determining a mapping threshold of the query basic data based on an actual data value of the query basic data and a preset numerical range under a data attribute to which the query basic data belongs;

[0014] When the data type indicates that the query basic data is non-numeric data, a mapping threshold of the query basic data is determined according to a preset mapping relationship under a data attribute to which the query basic data belongs.

[0015] In a possible implementation, forming an application data table based on each query basic data and a mapping threshold of the query basic data includes:

[0016] According to the data table writing structure of the database, each query basic data and the mapping threshold of the query basic data are written into the application data table as query data items in the application data table.

[0017] In a possible implementation manner, the use of the mapping threshold of each query basic data and classification processing of each query basic data in the application data table to create a query bitmap index of the application data table includes:

[0018] Using the mapping threshold of each query basic data, the query basic data with the same mapping threshold are divided into the same bitmap index group;

[0019] For each bitmap index group, a mapping threshold corresponding to any query basic data in the bitmap index group is determined as a sub-query bitmap index of the bitmap index group, so as to implement a query of the query basic data in the bitmap index group by using the sub-query bitmap index;

[0020] Based on the sub-query bitmap index of each bitmap index group, a query bitmap index of the application data table is created.

[0021] The present application also provides a data query method, which includes:

[0022] In response to receiving a data retrieval request, reading a data retrieval condition from the data retrieval request;

[0023] Using the data retrieval condition, by querying the query result data cached in the memory cache, determining whether there is a data query result in the memory cache that meets the data retrieval condition;

[0024] If not, calling the SQL encapsulation unit to generate a retrieval SQL statement for data query by converting the data retrieval condition into a data retrieval threshold;

[0025] By executing the retrieval SQL statement, with the help of the query bitmap index of the application data table in the database, a data query result that meets the data retrieval condition is obtained from the application data table; wherein the query bitmap index of the application data table is constructed by the above-mentioned data query bitmap index construction method;

[0026] The data query result is fed back to the client.

[0027] In a possible implementation, the calling SQL encapsulation unit converts the data retrieval condition into a data retrieval threshold to generate a retrieval SQL statement for performing data query, including:

[0028] Calling the SQL encapsulation unit to convert the data retrieval condition into a data retrieval threshold according to a mapping rule between the retrieval condition and the retrieval threshold;

[0029] Substitute the data retrieval threshold into the SQL statement template to obtain the retrieval SQL statement.

[0030] In a possible implementation, executing the search SQL statement to obtain a data query result satisfying the data search condition from the application data table by using a query bitmap index of the application data table in the database includes:

[0031] By executing the retrieval SQL statement, by comparing the data retrieval threshold carried by the retrieval SQL statement with the query bitmap index of the application data table in the database, determining the sub-query bitmap index in the query bitmap index that matches the data retrieval threshold;

[0032] The query basic data in the target bitmap index group corresponding to the sub-query bitmap index is determined as a data query result that satisfies the data retrieval condition.

[0033] In a possible implementation, the data query method further includes:

[0034] Counting the number of query basic data included in the data query result;

[0035] When the data quantity is less than a preset quantity threshold, the data query result is cached in the memory cache.

[0036] The present application also provides a device for constructing a data query bitmap index, the device comprising:

[0037] A data acquisition module is used to acquire basic business data from a basic data table pre-stored in the database;

[0038] A threshold processing module is used to pre-process the basic business data, filter out the basic business data that cannot be processed by thresholding, and obtain query basic data that can be processed by thresholding;

[0039] A threshold mapping module, used for determining, for each query basic data, a mapping threshold of the query basic data according to data attributes of the query basic data;

[0040] A data table creation module, used for forming an application data table based on each query basic data and a mapping threshold value of the query basic data;

[0041] A bitmap index creation module is used to utilize a mapping threshold of each query basic data and classify each query basic data in the application data table to create a query bitmap index of the application data table, so as to utilize the query bitmap index to implement a query of the query basic data in the application data table.

[0042] In a possible implementation, when the threshold mapping module is used to determine the mapping threshold of the query basic data according to the data attribute of the query basic data, the threshold mapping module is used to:

[0043] Determine the data type of the query basic data according to the data attribute of the query basic data;

[0044] When the data type indicates that the query basic data is numerical data, determining a mapping threshold of the query basic data based on an actual data value of the query basic data and a preset numerical range under a data attribute to which the query basic data belongs;

[0045] When the data type indicates that the query basic data is non-numeric data, a mapping threshold of the query basic data is determined according to a preset mapping relationship under a data attribute to which the query basic data belongs.

[0046] In a possible implementation, when the data table creation module is used to form the application data table based on each query basic data and the mapping threshold of the query basic data, the data table creation module is used to:

[0047] According to the data table writing structure of the database, each query basic data and the mapping threshold of the query basic data are written into the application data table as query data items in the application data table.

[0048] In a possible implementation, when the bitmap index creation module is used to create a query bitmap index for the application data table by classifying each query basic data in the application data table using a mapping threshold of each query basic data, the bitmap index creation module is used to:

[0049] Using the mapping threshold of each query basic data, the query basic data with the same mapping threshold are divided into the same bitmap index group;

[0050] For each bitmap index group, a mapping threshold corresponding to any query basic data in the bitmap index group is determined as a sub-query bitmap index of the bitmap index group, so as to implement a query of the query basic data in the bitmap index group by using the sub-query bitmap index;

[0051] Based on the sub-query bitmap index of each bitmap index group, a query bitmap index of the application data table is created.

[0052] The present application also provides a data query device, which includes:

[0053] A data reading module, configured to read a data retrieval condition from the data retrieval request in response to receiving the data retrieval request;

[0054] A first query module, used to use the data retrieval condition to determine whether there is a data query result satisfying the data retrieval condition in the memory cache by querying the query result data cached in the memory cache;

[0055] A statement generation module is used for, if not, calling the SQL encapsulation unit to generate a retrieval SQL statement for data query by converting the data retrieval condition into a data retrieval threshold;

[0056] A second query module is used to obtain a data query result satisfying the data retrieval condition from the application data table by executing the retrieval SQL statement with the aid of a query bitmap index of the application data table in the database; wherein the query bitmap index of the application data table is constructed by the above-mentioned data query bitmap index construction device;

[0057] The result feedback module is used to feed back the data query result to the client.

[0058] In a possible implementation, when the statement generation module is used to call the SQL encapsulation unit to convert the data retrieval condition into a data retrieval threshold to generate a retrieval SQL statement for performing a data query, the statement generation module is used to:

[0059] Calling the SQL encapsulation unit to convert the data retrieval condition into a data retrieval threshold according to a mapping rule between the retrieval condition and the retrieval threshold;

[0060] Substitute the data retrieval threshold into the SQL statement template to obtain the retrieval SQL statement.

[0061] In a possible implementation manner, when the second query module is used to obtain a data query result satisfying the data retrieval condition from the application data table by executing the retrieval SQL statement with the aid of a query bitmap index of the application data table in the database, the second query module is used to:

[0062] By executing the retrieval SQL statement, by comparing the data retrieval threshold carried by the retrieval SQL statement with the query bitmap index of the application data table in the database, determining the sub-query bitmap index in the query bitmap index that matches the data retrieval threshold;

[0063] The query basic data in the target bitmap index group corresponding to the sub-query bitmap index is determined as a data query result that satisfies the data retrieval condition.

[0064] In a possible implementation, the data query device further includes a cache module, and the cache module is used to:

[0065] Counting the number of query basic data included in the data query result;

[0066] When the data quantity is less than a preset quantity threshold, the data query result is cached in the memory cache.

[0067] An embodiment of the present application also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the method for constructing a data query bitmap index and the steps of the data query method are performed.

[0068] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for constructing a data query bitmap index and the steps of the data query method are executed.

[0069] The data query bitmap index construction method, data query method and device provided in the embodiment of the present application obtain basic business data from the basic data table pre-stored in the database; pre-process the basic business data, filter out the basic business data that cannot be thresholded in the basic business data, and obtain the query basic data that can be thresholded; for each query basic data, determine the mapping threshold of the query basic data according to the data attribute of the query basic data; form an application data table based on each query basic data and the mapping threshold of the query basic data; use the mapping threshold of each query basic data to classify each query basic data in the application data table to create a query bitmap index for the application data table, so as to use the query bitmap index to realize the query of the query basic data in the application data table. In this way, in the process of arranging the basic business data into the application data table, the mapping threshold of the query basic data included in the application data table can be used to create a query bitmap index for the application data table, so that when there is a query demand, the query bitmap index can be used to quickly realize the query of the data, thereby improving the retrieval efficiency and timeliness of data under a large amount of data. In addition, in order to facilitate users to perform data queries, the data query results can be cached in the memory cache based on the actual situation of the data retrieval results, so that in the subsequent retrieval process, the data retrieval results can be quickly fed back; and, by matching the query bitmap index, data query is realized, and there is no need to traverse the query memory data, so that the data query results can be quickly located and queried, thereby reducing the complex data search process and improving the feedback efficiency of the data query results.

[0070] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0072] Figure 1 A flowchart of a method for constructing a data query bitmap index provided in an embodiment of the present application;

[0073] Figure 2 A flowchart of a data query method provided in an embodiment of the present application;

[0074] Figure 3 A schematic diagram of a data query process provided by an embodiment of the present application;

[0075] Figure 4 A schematic diagram of the structure of a device for constructing a data query bitmap index provided in an embodiment of the present application;

[0076] Figure 5 A schematic diagram of the structure of a data query device provided in an embodiment of the present application;

[0077] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0079] Through research, it is found that, at present, data retrieval is usually achieved by relying on the storage structured data of the database and using the database query engine. However, under such a technical route, if complex retrieval conditions are encountered, it is impossible to achieve fast data retrieval under huge data volume, and the real-time performance of the retrieval is relatively low. In addition, in distributed file systems, distributed storage of data is usually achieved with the help of the system structure of the distributed system, and then distributed search and analysis engines (such as Elasticsearch) are used for data retrieval. Although this method can improve the retrieval efficiency of data under large data volume, the implementation steps are relatively cumbersome, the logical structure is relatively complex, and the operation and deployment are also relatively troublesome.

[0080] Based on this, an embodiment of the present application provides a method for constructing a data query bitmap index. In the process of organizing basic business data into an application data table, the mapping threshold of the query basic data included in the application data table can be used to create a query bitmap index for the application data table, so that when there is a query demand, the query bitmap index can be used to quickly realize data query, thereby improving the retrieval efficiency and timeliness of data under large data volumes.

[0081] See also Figure 1 , Figure 1 The flowchart of a method for constructing a data query bitmap index provided in an embodiment of the present application is as follows. Figure 1 As shown in , the method for constructing a data query bitmap index provided in an embodiment of the present application includes:

[0082] S101. Obtain basic business data from a basic data table pre-stored in a database.

[0083] S102 , pre-processing the basic business data, filtering out the basic business data that cannot be processed by threshold value, and obtaining query basic data that can be processed by threshold value.

[0084] S103 . For each query basic data, determine a mapping threshold of the query basic data according to data attributes of the query basic data.

[0085] S104: Form an application data table based on each query basic data and a mapping threshold of the query basic data.

[0086] S105. Using the mapping threshold of each query basic data, by classifying each query basic data in the application data table, a query bitmap index of the application data table is created, so as to use the query bitmap index to implement the query of the query basic data in the application data table.

[0087] A method for constructing a data query bitmap index provided in an embodiment of the present application targets basic business data pre-stored in a database. First, the basic business data is pre-processed to filter out the basic business data that cannot be thresholded, so as to reduce the amount of data that needs to be processed during the thresholding process; then, for the query basic data that can be thresholded, a mapping threshold corresponding to the query basic data is used to create a query bitmap index of an application data table to which the query basic data belongs, so that when there is a query demand, the query basic data can be quickly retrieved, thereby improving the retrieval efficiency of data retrieval in the database.

[0088] In step S101, a basic data table is pre-stored in the database (MPP), and the basic data table includes business data generated by customers in the process of handling business. Here, considering that some business data cannot be fed back as query results, in order to avoid these business data from interfering with the data query process, it is necessary to further obtain basic business data that can be used as query results from the basic data table pre-stored in the database.

[0089] Here, considering that the query bitmap index will be established by the "threshold processing" method in the subsequent process, in order to avoid the basic business data that cannot be thresholded from interfering with the subsequent thresholding process and increasing the unnecessary data processing volume of the terminal device, the basic business data that cannot be thresholded can be filtered out through preprocessing before thresholding.

[0090] In step S102, the extracted basic business data is preprocessed to screen and clean the basic business data, and the basic business data that cannot be thresholded is filtered out to obtain query basic data that can be thresholded, thereby reducing the amount of data processing in the subsequent thresholding process and avoiding interference with the thresholding process.

[0091] Here, the query basic data that can be thresholded have different data attributes, and the difference in data attributes will affect the threshold mapping process of the query basic data. Therefore, when determining the mapping threshold of the query basic data, it is necessary to further refer to the data attributes of the query basic data.

[0092] In step S103, for each query basic data, a "threshold processing" is performed on the query basic data in combination with the data attribute of the query basic data to determine a mapping threshold of the query basic data.

[0093] Among them, data attributes may include gender, age, amount and other attributes.

[0094] Here, the data attributes of the query basic data may affect the data type of the query basic data. For query basic data of different data types, the "mapping rules" for mapping the mapping threshold of the query data type are also different. Therefore, it is also necessary to combine the actual data type of the query basic data to realize the "mapping" of the mapping threshold of the query data type.

[0095] In one implementation, step S103 includes:

[0096] S1031. Determine the data type of the query basic data according to the data attribute of the query basic data.

[0097] The data type may include numerical data and non-numerical data; for query basic data of different data types, due to the limitation of the data type, there are differences in the way of determining the mapping threshold.

[0098] S1032. When the data type indicates that the query basic data is numerical data, a mapping threshold of the query basic data is determined based on an actual data value of the query basic data and a preset numerical range under a data attribute to which the query basic data belongs.

[0099] In this step, when the data type of the query basic data is numerical data, the actual data value of the query basic data needs to be used in the mapping process, combined with the preset numerical range under the data attribute to which the query basic data belongs. By determining the preset numerical range to which the query basic data belongs, the mapping threshold corresponding to the preset numerical range is determined as the mapping threshold of the query basic data.

[0100] Exemplarily, taking the data attribute of the query basic data as "deposit amount" as an example, multiple preset value ranges can be set in advance for data such as "deposit amount", range 1: (1000-9999) corresponding to the mapping threshold 1000, range 2: (10000-99999) corresponding to the mapping threshold 10000, range 3: (100000-999999) corresponding to the mapping threshold 100000; if the actual data value of the query basic data is 5500, the query basic data belongs to "Range 1: (1000-9999)", at this time, the mapping threshold corresponding to "Range 1" (that is, 1000) can be determined as the mapping threshold of the query basic data.

[0101] S1033: When the data type indicates that the query basic data is non-numeric data, determine a mapping threshold of the query basic data according to a preset mapping relationship under a data attribute to which the query basic data belongs.

[0102] In this step, when the data type of the query basic data is non-numeric data, at this time, in the mapping process, it is necessary to map the query basic data to the mapping threshold according to the preset mapping relationship under the data attribute to which the query basic data belongs.

[0103] Exemplarily, taking the data attribute of the query basic data as "gender" as an example, multiple preset mapping relationships can be set in advance for data such as "gender", relationship 1: when "gender" is "female", the mapping threshold is 1, range 2: when "gender" is "male", the mapping threshold is 0; if the query basic data is "female", the mapping threshold corresponding to the query basic data is "1".

[0104] In step S104, each query basic data and the mapping threshold of the query basic data are integrated to form an application data table covering the query basic data and the mapping threshold corresponding to the query basic data, so as to construct a query bitmap index of the application data table using the mapping threshold of the query basic data.

[0105] Here, the application data table stored in the database has certain requirements on the writing structure. Therefore, when forming the application data table, it is necessary to integrate the query basic data and the mapping threshold corresponding to the query basic data according to the data table writing structure.

[0106] In one implementation, step S104 includes: writing each query basic data and the mapping threshold of the query basic data into the application data table as a query data item in the application data table according to the data table writing structure of the database.

[0107] In this step, after writing the section according to the preset data table in the database stored in the application data table, each query basic data and the mapping threshold of the query basic data are integrated, and each query basic data and the mapping threshold of the query basic data are written into the application data table as query data items in the application data table to obtain an application data table covering the query basic data and the mapping thresholds corresponding to the query basic data.

[0108] Here, the query bitmap index of the application data table can be constructed by using the mapping threshold of each query basic data in the application data table, so as to assist the data retrieval process by using the query bitmap index.

[0109] In step S105, each query basic data in the application data table is classified and processed using the mapping threshold of each query basic data, and a query bitmap index of the application data table is created by assigning a query bitmap index to the query basic data with the same mapping threshold; further, in the process of data retrieval, the query bitmap index can be used to realize fast query of relevant query basic data in the application data table.

[0110] In one implementation, step S105 includes:

[0111] S1051 . Using the mapping threshold of each query basic data, the query basic data with the same mapping threshold are divided into the same bitmap index group.

[0112] In this step, the query basic data in the application data table are classified in combination with the mapping threshold of each query basic data, and the query basic data with the same mapping threshold are divided into the same bitmap index group.

[0113] S1052. For each bitmap index group, determine a mapping threshold corresponding to any query basic data in the bitmap index group as a sub-query bitmap index of the bitmap index group, so as to implement a query of the query basic data in the bitmap index group using the sub-query bitmap index.

[0114] In this step, for each bitmap index group, since the query basic data in the bitmap index group have the same mapping threshold, the mapping threshold corresponding to any query basic data in the bitmap index group can be determined as the sub-query bitmap index of the bitmap index group; so that in the process of data retrieval, the query basic data in the bitmap index group can be accurately located and queried with the help of the sub-query bitmap index.

[0115] S1053: Create a query bitmap index of the application data table based on the sub-query bitmap index of each bitmap index group.

[0116] The method for constructing a data query bitmap index provided in an embodiment of the present application obtains basic business data from a basic data table pre-stored in a database; pre-processes the basic business data to filter out the basic business data that cannot be thresholded, and obtains query basic data that can be thresholded; for each query basic data, determines a mapping threshold of the query basic data according to the data attribute of the query basic data; forms an application data table based on each query basic data and the mapping threshold of the query basic data; uses the mapping threshold of each query basic data to classify each query basic data in the application data table to create a query bitmap index for the application data table, so as to use the query bitmap index to implement query of the query basic data in the application data table. In this way, in the process of organizing the basic business data into the application data table, the mapping threshold of the query basic data included in the application data table can be used to create a query bitmap index for the application data table, so that when there is a query demand, the query bitmap index can be used to quickly realize data query, thereby improving the retrieval efficiency and timeliness of data under huge data volumes; and, by matching the query bitmap index to realize data query, there is no need to traverse the query memory data, so that the data query results can be quickly located and queried, thereby reducing the complex data search process and improving the feedback efficiency of the data query results.

[0117] See also Figure 2 , Figure 2 A flowchart of a data query method provided in an embodiment of the present application. Figure 2 As shown in , the data query method provided by the embodiment of the present application includes:

[0118] S201. In response to receiving a data retrieval request, read a data retrieval condition from the data retrieval request.

[0119] S202: Using the data retrieval condition, by querying the query result data cached in the memory cache, determine whether there is a data query result that meets the data retrieval condition in the memory cache.

[0120] S203: If not, call the SQL encapsulation unit to generate a retrieval SQL statement for data query by converting the data retrieval condition into a data retrieval threshold.

[0121] S204: By executing the search SQL statement, with the help of the query bitmap index of the application data table in the database, a data query result that meets the data search condition is obtained from the application data table.

[0122] S205: Feedback the data query result to the client.

[0123] A data query method provided in an embodiment of the present application, when receiving a data retrieval request, if there is no data query result that meets the data retrieval conditions in the memory cache, with the help of the query bitmap index of the application data table, the basic data for query in the application data table can be quickly located and queried, thereby improving the feedback efficiency of the data query result.

[0124] In step S201, in response to receiving a data retrieval request sent by a client, the business service interface analyzes the data retrieval request, reads data retrieval conditions from the data retrieval request, and uses the data retrieval conditions to implement fast retrieval of data in the memory cache.

[0125] Here, in order to quickly respond to users' data retrieval requests and improve the feedback efficiency of data query results, during the historical retrieval process, some historical retrieval results will be cached in the memory cache, so that in the subsequent data retrieval process, if a data retrieval request with the same data retrieval conditions is received, the data query results can be fed back without searching the database; therefore, when a data retrieval request is received, data retrieval in the memory cache must be prioritized.

[0126] In step S202, the extracted data retrieval condition is used to query the data query results cached in the memory cache and obtained in the historical retrieval process to determine whether there are data query results that meet the data retrieval condition in the memory cache.

[0127] If there are data query results that meet the data retrieval conditions in the memory cache, the data query results can be directly obtained from the memory cache and feedback of the data query results can be realized, thereby simplifying the data retrieval process and improving data query efficiency.

[0128] Here, if there is no data query result that meets the data retrieval condition in the memory cache, it is necessary to further query the application data table in the database to obtain the data query result that meets the data retrieval condition; at this time, it is necessary to further generate a retrieval SQL statement for searching the database;

[0129] At the same time, in order to be able to use the query bitmap index in the application data table to achieve fast query of data query results, considering that the query bitmap index of the application data table is in the form of a threshold, when generating a retrieval SQL statement, it is also necessary to convert the data retrieval condition into the form of a data retrieval threshold to facilitate comparison with the query bitmap index of the application data table.

[0130] In step S203, if there is no data query result satisfying the data retrieval condition in the memory cache, the SQL encapsulation unit may be called to generate a retrieval SQL statement for performing data query in the database by converting the data retrieval condition into a data retrieval threshold.

[0131] In one implementation, step S203 includes:

[0132] S2031. Calling the SQL encapsulation unit to convert the data retrieval condition into a data retrieval threshold according to a mapping rule between the retrieval condition and the retrieval threshold.

[0133] In this step, by calling the SQL encapsulation unit, each data retrieval condition is mapped into the form of a data retrieval threshold one by one according to the mapping rules between the retrieval condition and the retrieval threshold, so that during the retrieval process, the query basic data that meets the data retrieval condition can be determined by comparing the data retrieval threshold with the sub-query bitmap index in the query bitmap index one by one.

[0134] The mapping rules between different search conditions and search thresholds are different. Taking the data search condition "gender" as an example, the mapping rule between the search condition and the search threshold is: if the data search condition is "female", the corresponding data search threshold is "1"; if the data search condition is "male", the corresponding data search threshold is "0".

[0135] S2032: Substitute the data retrieval threshold into an SQL statement template to obtain the retrieval SQL statement.

[0136] In step S204, the data query control unit executes the generated retrieval SQL statement, calls the query method interface of the database (MPP), and completes the query in the application data table with the help of the query bitmap index of the application data table in the database to obtain data query results that meet the data retrieval conditions from the application data table.

[0137] The query bitmap index of the application data table is constructed by the data query bitmap index construction method described in the above embodiment.

[0138] In one implementation, step S204 includes:

[0139] S2041. By executing the retrieval SQL statement, and by comparing the data retrieval threshold carried by the retrieval SQL statement with the query bitmap index of the application data table in the database, a sub-query bitmap index in the query bitmap index that matches the data retrieval threshold is determined.

[0140] In this step, during the execution of the retrieval SQL statement, the data retrieval threshold carried in the retrieval SQL statement is compared with the sub-query bitmap index in the query bitmap index of the application data table to determine the sub-query bitmap index that matches the data retrieval threshold.

[0141] Corresponding to the above embodiment, when the data retrieval threshold corresponding to the data retrieval condition is "1", the sub-query bitmap index with a value of "1" can be considered to be a sub-query bitmap index that matches the data retrieval threshold.

[0142] S2042: Determine the query basic data in the target bitmap index group corresponding to the sub-query bitmap index as the data query result that satisfies the data retrieval condition.

[0143] In this step, each query basic data is pre-divided into multiple bitmap index groups in the application data table, and the query basic data in each bitmap index group has the same mapping threshold, and the sub-query bitmap index of the bitmap index group is the mapping threshold of any query basic data in the bitmap index group; it can be seen that if the sub-query bitmap index of the target bitmap index group matches the data retrieval threshold, it can be said that the query basic data in the target bitmap index group are all data query results that meet the data retrieval conditions corresponding to the data retrieval threshold.

[0144] In step S205, the data query results are fed back to the client so that the user can view the data query results.

[0145] Here, in order to facilitate the user to perform data query, the data query results can be cached in the memory cache based on the actual situation of the data retrieval results, so that in the subsequent retrieval process, the data retrieval results can be quickly fed back.

[0146] Considering that if the data query results with a large data stream are stored in the memory cache, it will affect the number of data query results cached in the memory cache. In order to cache as many data query results as possible, after obtaining the data query results, you can choose whether to cache the data query results obtained from this query in the memory cache based on the number of data in the data query results.

[0147] In one embodiment, the data query method further includes: counting the number of query basic data included in the data query result; and caching the data query result in the memory cache when the number of data is less than a preset number threshold.

[0148] In this step, after obtaining the data query results that meet the data retrieval conditions, the data query results included in the statistical data query results can be cached in the memory cache if the data quantity is less than the preset quantity threshold (for example, the data quantity is less than 10,000). If the data quantity is greater than or equal to the preset quantity threshold (for example, the data quantity is greater than or equal to 10,000), the data query results are cached.

[0149] The data query method provided by the embodiment of the present application, in response to receiving a data retrieval request, reads out the data retrieval condition from the data retrieval request; uses the data retrieval condition to query the query result data cached in the memory cache to determine whether there is a data query result that meets the data retrieval condition in the memory cache; if not, calls the SQL encapsulation unit to generate a retrieval SQL statement for data query by converting the data retrieval condition into a data retrieval threshold; by executing the retrieval SQL statement, obtains the data query result that meets the data retrieval condition from the application data table with the help of the query bitmap index of the application data table in the database; and feeds back the data query result to the client. In this way, the query bitmap index of the application data table can be used to realize the rapid positioning and query of the query basic data in the application data table, thereby improving the feedback efficiency of the data query result.

[0150] See also Figure 3 , Figure 3 A data query flow diagram provided in an embodiment of the present application. Figure 3As shown, the data query process involves a client used by the user, a memory cache, a data query module, a database (MPP) and a data preprocessing module; the user sends a data retrieval request through the client; the data query module receives the data retrieval request through the business service interface, and queries whether there is a data query result in the memory cache that meets the data retrieval condition indicated by the data retrieval request; if so, the data query module directly returns the data query result to the client through the business service interface; if not, a retrieval SQL statement for data query is generated through the SQL encapsulation unit; the data query control unit executes the generated retrieval SQL statement, calls the query method interface of the database (MPP), and obtains the data query result that meets the data retrieval condition from the application data table with the help of the query bitmap index of the application data table in the database.

[0151] The data preprocessing module includes a data input unit, a data verification unit, a threshold processing unit and a data output unit; the data input unit obtains basic business data from a basic data table pre-stored in a database; the data verification unit pre-processes the obtained basic business data, filters out the basic business data that cannot be thresholded, and obtains query basic data that can be thresholded; the threshold processing unit performs threshold processing on the query basic data to determine the mapping threshold of the query basic data; the data output unit integrates the query basic data and its corresponding mapping threshold to obtain an application data table; for the application data table, a query bitmap index of the application data table is created.

[0152] See also Figure 4 , Figure 4 A schematic diagram of a data query bitmap index construction device provided in an embodiment of the present application. Figure 4 As shown in , the construction device 400 includes:

[0153] The data acquisition module 410 is used to acquire basic business data from a basic data table pre-stored in the database;

[0154] A threshold processing module 420 is used to pre-process the basic business data, filter out the basic business data that cannot be processed by thresholding, and obtain query basic data that can be processed by thresholding;

[0155] A threshold mapping module 430 is used to determine, for each query basic data, a mapping threshold of the query basic data according to data attributes of the query basic data;

[0156] A data table creation module 440, configured to form an application data table based on each query basic data and a mapping threshold of the query basic data;

[0157] A bitmap index creation module 450 is used to utilize a mapping threshold of each query basic data and classify each query basic data in the application data table to create a query bitmap index of the application data table, so as to implement a query of the query basic data in the application data table using the query bitmap index.

[0158] Furthermore, when the threshold mapping module 430 is used to determine the mapping threshold of the query basic data according to the data attribute of the query basic data, the threshold mapping module 430 is used to:

[0159] Determine the data type of the query basic data according to the data attribute of the query basic data;

[0160] When the data type indicates that the query basic data is numerical data, determining a mapping threshold of the query basic data based on an actual data value of the query basic data and a preset numerical range under a data attribute to which the query basic data belongs;

[0161] When the data type indicates that the query basic data is non-numeric data, a mapping threshold of the query basic data is determined according to a preset mapping relationship under a data attribute to which the query basic data belongs.

[0162] Furthermore, when the data table creation module 440 is used to form an application data table based on each query basic data and a mapping threshold of the query basic data, the data table creation module 440 is used to:

[0163] According to the data table writing structure of the database, each query basic data and the mapping threshold of the query basic data are written into the application data table as query data items in the application data table.

[0164] Further, when the bitmap index creation module 450 is used to create a query bitmap index of the application data table by classifying each query basic data in the application data table using the mapping threshold of each query basic data, the bitmap index creation module 450 is used to:

[0165] Using the mapping threshold of each query basic data, the query basic data with the same mapping threshold are divided into the same bitmap index group;

[0166] For each bitmap index group, a mapping threshold corresponding to any query basic data in the bitmap index group is determined as a sub-query bitmap index of the bitmap index group, so as to implement a query of the query basic data in the bitmap index group by using the sub-query bitmap index;

[0167] Based on the sub-query bitmap index of each bitmap index group, a query bitmap index of the application data table is created.

[0168] The data query bitmap index construction device provided in the embodiment of the present application obtains basic business data from a basic data table pre-stored in a database; pre-processes the basic business data, filters out the basic business data that cannot be thresholded in the basic business data, and obtains query basic data that can be thresholded; for each query basic data, determines the mapping threshold of the query basic data according to the data attribute of the query basic data; forms an application data table based on each query basic data and the mapping threshold of the query basic data; uses the mapping threshold of each query basic data to classify each query basic data in the application data table to create a query bitmap index for the application data table, so as to use the query bitmap index to realize the query of the query basic data in the application data table. In this way, in the process of organizing the basic business data into the application data table, the mapping threshold of the query basic data included in the application data table can be used to create a query bitmap index for the application data table, so that when there is a query demand, the query bitmap index can be used to quickly realize data query, thereby improving the retrieval efficiency and timeliness of data under huge data volumes; and, by matching the query bitmap index to realize data query, there is no need to traverse the query memory data, so that the data query results can be quickly located and queried, thereby reducing the complex data search process and improving the feedback efficiency of the data query results.

[0169] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a data query device provided in an embodiment of the present application. Figure 5 As shown in , the data query device 500 includes:

[0170] The data reading module 510 is used to read the data retrieval condition from the data retrieval request in response to receiving the data retrieval request;

[0171] A first query module 520 is used to use the data retrieval condition to determine whether there is a data query result that meets the data retrieval condition in the memory cache by querying the query result data cached in the memory cache;

[0172] A statement generation module 530 is used to, if not, call the SQL encapsulation unit to generate a retrieval SQL statement for data query by converting the data retrieval condition into a data retrieval threshold;

[0173] The second query module 540 is used to obtain data query results that meet the data retrieval condition from the application data table by executing the retrieval SQL statement with the help of the query bitmap index of the application data table in the database; wherein the query bitmap index of the application data table is constructed by the above-mentioned data query bitmap index construction device;

[0174] The result feedback module 550 is used to feed back the data query result to the client.

[0175] Furthermore, when the statement generation module 530 is used to call the SQL encapsulation unit and convert the data retrieval condition into a data retrieval threshold to generate a retrieval SQL statement for performing data query, the statement generation module 530 is used to:

[0176] Calling the SQL encapsulation unit to convert the data retrieval condition into a data retrieval threshold according to a mapping rule between the retrieval condition and the retrieval threshold;

[0177] Substitute the data retrieval threshold into the SQL statement template to obtain the retrieval SQL statement.

[0178] Furthermore, when the second query module 540 is used to obtain a data query result satisfying the data retrieval condition from the application data table by executing the retrieval SQL statement with the aid of a query bitmap index of the application data table in the database, the second query module 540 is used to:

[0179] By executing the retrieval SQL statement, by comparing the data retrieval threshold carried by the retrieval SQL statement with the query bitmap index of the application data table in the database, determining the sub-query bitmap index in the query bitmap index that matches the data retrieval threshold;

[0180] The query basic data in the target bitmap index group corresponding to the sub-query bitmap index is determined as a data query result that satisfies the data retrieval condition.

[0181] Furthermore, the data query device 500 further includes a cache module (not shown in the figure), and the cache module is used to:

[0182] Counting the number of query basic data included in the data query result;

[0183] When the data quantity is less than a preset quantity threshold, the data query result is cached in the memory cache.

[0184] The data query device provided by the embodiment of the present application, in response to receiving a data retrieval request, reads out the data retrieval condition from the data retrieval request; uses the data retrieval condition to query the query result data cached in the memory cache to determine whether there is a data query result that meets the data retrieval condition in the memory cache; if not, calls the SQL encapsulation unit to generate a retrieval SQL statement for data query by converting the data retrieval condition into a data retrieval threshold; obtains the data query result that meets the data retrieval condition from the application data table by executing the retrieval SQL statement with the help of the query bitmap index of the application data table in the database; and feeds back the data query result to the client. In this way, the query bitmap index of the application data table can be used to realize the rapid positioning and query of the query basic data in the application data table, thereby improving the feedback efficiency of the data query result.

[0185] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown in , the electronic device 600 includes a processor 610 , a memory 620 and a bus 630 .

[0186] The memory 620 stores machine-readable instructions executable by the processor 610. When the electronic device 600 is running, the processor 610 communicates with the memory 620 via the bus 630. When the machine-readable instructions are executed by the processor 610, the above-mentioned Figure 1 The steps of the method for constructing a data query bitmap index in the method embodiment shown are as follows Figure 2 The steps of the data query method in the method embodiment shown and the specific implementation methods can be found in the method embodiment, which will not be repeated here.

[0187] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of the method for constructing a data query bitmap index in the method embodiment shown are as follows Figure 2 The steps of the data query method in the method embodiment shown and the specific implementation methods can be found in the method embodiment, which will not be repeated here.

[0188] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0189] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0190] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0192] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0193] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for constructing a data query bitmap index. It is characterized in that The construction method comprises: Obtain basic business data from the basic data table pre-stored in the database; Preprocessing the basic business data, filtering out the basic business data that cannot be processed by threshold value, and obtaining query basic data that can be processed by threshold value; For each query basic data, determining a mapping threshold of the query basic data according to the data attribute of the query basic data; Based on each query basic data and a mapping threshold of the query basic data, an application data table is formed; Using the mapping threshold of each query basic data, by classifying each query basic data in the application data table, a query bitmap index of the application data table is created, so as to use the query bitmap index to implement the query of the query basic data in the application data table; The method of utilizing the mapping threshold of each query basic data and classifying each query basic data in the application data table to create a query bitmap index of the application data table includes: Using the mapping threshold of each query basic data, the query basic data with the same mapping threshold are divided into the same bitmap index group; For each bitmap index group, a mapping threshold corresponding to any query basic data in the bitmap index group is determined as a sub-query bitmap index of the bitmap index group, so as to implement a query of the query basic data in the bitmap index group by using the sub-query bitmap index; Based on the sub-query bitmap index of each bitmap index group, a query bitmap index of the application data table is created.

2. The construction method according to claim 1, It is characterized in that The step of determining a mapping threshold of the query basic data according to a data attribute of the query basic data includes: Determine the data type of the query basic data according to the data attribute of the query basic data; When the data type indicates that the query basic data is numerical data, determining a mapping threshold of the query basic data based on an actual data value of the query basic data and a preset numerical range under a data attribute to which the query basic data belongs; When the data type indicates that the query basic data is non-numeric data, a mapping threshold of the query basic data is determined according to a preset mapping relationship under a data attribute to which the query basic data belongs.

3. The construction method according to claim 1, It is characterized in that The forming of the application data table based on each query basic data and the mapping threshold of the query basic data includes: According to the data table writing structure of the database, each query basic data and the mapping threshold of the query basic data are written into the application data table as query data items in the application data table.

4. A data query method, It is characterized in that The data query method comprises: In response to receiving a data retrieval request, reading a data retrieval condition from the data retrieval request; Using the data retrieval condition, by querying the query result data cached in the memory cache, determining whether there is a data query result in the memory cache that meets the data retrieval condition; If not, calling the SQL encapsulation unit to generate a retrieval SQL statement for data query by converting the data retrieval condition into a data retrieval threshold; By executing the retrieval SQL statement, with the help of the query bitmap index of the application data table in the database, a data query result that meets the data retrieval condition is obtained from the application data table; wherein the query bitmap index of the application data table is constructed by the data query bitmap index construction method described in any one of claims 1 to 3; The data query result is fed back to the client.

5. The data query method according to claim 4, It is characterized in that The calling SQL encapsulation unit generates a retrieval SQL statement for performing data query by converting the data retrieval condition into a data retrieval threshold, including: Calling the SQL encapsulation unit to convert the data retrieval condition into a data retrieval threshold according to a mapping rule between the retrieval condition and the retrieval threshold; Substitute the data retrieval threshold into the SQL statement template to obtain the retrieval SQL statement.

6. The data query method according to claim 4, It is characterized in that The step of executing the search SQL statement and obtaining a data search result satisfying the data search condition from the application data table by means of a query bitmap index of the application data table in the database includes: By executing the retrieval SQL statement, by comparing the data retrieval threshold carried by the retrieval SQL statement with the query bitmap index of the application data table in the database, determining the sub-query bitmap index in the query bitmap index that matches the data retrieval threshold; The query basic data in the target bitmap index group corresponding to the sub-query bitmap index is determined as a data query result that satisfies the data retrieval condition.

7. The data query method according to claim 4, It is characterized in that The data query method further includes: Counting the number of query basic data included in the data query result; When the data quantity is less than a preset quantity threshold, the data query result is cached in the memory cache.

8. A device for constructing a data query bitmap index, It is characterized in that The construction device comprises: A data acquisition module is used to acquire basic business data from a basic data table pre-stored in the database; A threshold processing module is used to pre-process the basic business data, filter out the basic business data that cannot be processed by thresholding, and obtain query basic data that can be processed by thresholding; A threshold mapping module, used for determining, for each query basic data, a mapping threshold of the query basic data according to data attributes of the query basic data; A data table creation module, used for forming an application data table based on each query basic data and a mapping threshold value of the query basic data; A bitmap index creation module is used to create a query bitmap index of the application data table by using a mapping threshold of each query basic data and classifying each query basic data in the application data table, so as to use the query bitmap index to implement a query of the query basic data in the application data table; When the bitmap index creation module is used to create a query bitmap index for the application data table by classifying each query basic data in the application data table using the mapping threshold of each query basic data, the bitmap index creation module is used to: Using the mapping threshold of each query basic data, the query basic data with the same mapping threshold are divided into the same bitmap index group; For each bitmap index group, a mapping threshold corresponding to any query basic data in the bitmap index group is determined as a sub-query bitmap index of the bitmap index group, so as to implement a query of the query basic data in the bitmap index group by using the sub-query bitmap index; Based on the sub-query bitmap index of each bitmap index group, a query bitmap index of the application data table is created.

9. A data query device, It is characterized in that The data query device comprises: A data reading module, configured to read a data retrieval condition from the data retrieval request in response to receiving the data retrieval request; A first query module, used to use the data retrieval condition to determine whether there is a data query result satisfying the data retrieval condition in the memory cache by querying the query result data cached in the memory cache; A statement generation module is used for, if not, calling the SQL encapsulation unit to generate a retrieval SQL statement for data query by converting the data retrieval condition into a data retrieval threshold; A second query module is used to obtain a data query result that meets the data retrieval condition from the application data table by executing the retrieval SQL statement with the help of a query bitmap index of the application data table in the database; wherein the query bitmap index of the application data table is constructed by the data query bitmap index construction device according to claim 8; The result feedback module is used to feed back the data query result to the client.