Data query method and device, computer equipment and storage medium

Abstract

The invention relates to a data query method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps of obtaining a data query request for a data warehouse and a plurality of pre-query results pre-configured for the data warehouse, extracting an expected query dimension and an expected dimension condition from the data query request, determining the expected query dimension and the expected dimension condition in the data query request, and sending the expected query dimension and the expected dimension condition to the data warehouse; and according to the pre-query condition corresponding to each pre-query result, determining the pre-query dimension and the pre-query dimension condition of each pre-query result, and by adopting the method, under the condition that each pre-query dimension comprises the expected query dimension and the pre-query dimension condition corresponding to the expected query dimension comprises the expected dimension condition, determining the pre-query dimension of each pre-query result. The pre-query result is configured in advance, and the query result of the data query request is determined based on the pre-query result corresponding to the expected query dimension, so that the time for generating the query result is saved, and the data query efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data warehouses, and in particular to a data query method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] In today's data-driven era, enterprises are increasingly dependent on data. The real-time, accuracy and comprehensiveness of data are directly related to the decision-making efficiency and market response speed of enterprises. As an important technology in the field of big data processing and analysis, data warehouse is gradually becoming a key tool for enterprises to achieve real-time data insights.

[0003] In the traditional technology, when querying the data in the data warehouse according to the query statement after the statement optimization, there is still the problem of low data query efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a data query method, device, computer equipment, computer-readable storage medium and computer program product that can improve data query efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a data query method. The method comprises:

[0006] Obtain a data query request for a data warehouse and a plurality of pre-query results pre-configured for the data warehouse; extract query conditions from the data query request; the query conditions are characterized by an expected query dimension and an expected dimension condition for the expected query dimension; determine the pre-query dimension and pre-query dimension condition of each pre-query result according to the pre-query condition corresponding to each pre-query result; in a case where each pre-query dimension contains the expected query dimension and the pre-query dimension condition corresponding to the expected query dimension contains the expected dimension condition, determine the query result of the data query request based on the pre-query result corresponding to the expected query dimension.

[0007] In one of the embodiments, the method further includes: obtaining multiple historical query records; each of the historical query records includes a historical query dimension and a historical dimension condition for the historical query dimension; when there are at least two target query records containing the same target query dimension in each of the historical query records, merging the historical dimension conditions corresponding to the target query dimension to obtain the target dimension condition; based on the pre-query condition represented by the target query dimension and the target dimension condition, constructing a pre-query result for the data warehouse.

[0008] In one of the embodiments, the method further includes: when the expected query dimension is not included in each of the pre-query dimensions, or the pre-query dimension condition corresponding to the expected query dimension does not include the expected dimension condition, determining multiple dimensional query values ​​matching the expected query dimension and the expected dimensional condition in the fact table of the data warehouse; performing statistics on each of the dimensional query values ​​according to a statistical method matching the expected query dimension to obtain an index value corresponding to the expected query dimension; and determining a query result of the data query request based on the index value.

[0009] In one of the embodiments, the data warehouse includes an updated real-time data warehouse, and the method further includes: obtaining a full data table in the real-time data warehouse and an incremental data table within a preset processing cycle; batch processing the full data table and the incremental data table according to the preset processing cycle to obtain batch processing results; updating the real-time data warehouse according to the batch processing results to obtain an updated real-time data warehouse.

[0010] In one of the embodiments, extracting the query condition from the data query request includes: writing the data query request into a server cache of a server, and determining a query frequency of the server cache; extracting the query condition from the data query request when the query frequency is less than or equal to a frequency threshold; and blocking the query process corresponding to the data query request when the query frequency is less than or greater than the frequency threshold until the query frequency is less than or equal to the frequency threshold.

[0011] In one of the embodiments, the method further includes: determining a computing capability of the server; and determining a frequency threshold that matches the computing capability, wherein the computing capability and the frequency threshold are positively correlated.

[0012] In a second aspect, the present application also provides a data query device. The device comprises:

[0013] An acquisition module is used to acquire a data query request for a data warehouse and a plurality of pre-query results pre-configured for the data warehouse; a condition extraction module is used to extract query conditions from the data query request; the query conditions are represented by expected query dimensions and expected dimension conditions for the expected query dimensions; a pre-query module is used to determine the pre-query dimensions and pre-query dimension conditions of each pre-query result according to the pre-query conditions corresponding to each pre-query result; a query module is used to determine the query result of the data query request based on the pre-query results corresponding to the expected query dimensions, when each pre-query dimension contains the expected query dimension and the pre-query dimension conditions corresponding to the expected query dimension contain the expected dimension conditions.

[0014] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0015] Obtain a data query request for a data warehouse and a plurality of pre-query results pre-configured for the data warehouse; extract query conditions from the data query request; the query conditions are characterized by an expected query dimension and an expected dimension condition for the expected query dimension; determine the pre-query dimension and pre-query dimension condition of each pre-query result according to the pre-query condition corresponding to each pre-query result; in a case where each pre-query dimension contains the expected query dimension and the pre-query dimension condition corresponding to the expected query dimension contains the expected dimension condition, determine the query result of the data query request based on the pre-query result corresponding to the expected query dimension.

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0017] Obtain a data query request for a data warehouse and a plurality of pre-query results pre-configured for the data warehouse; extract query conditions from the data query request; the query conditions are characterized by an expected query dimension and an expected dimension condition for the expected query dimension; determine the pre-query dimension and pre-query dimension condition of each pre-query result according to the pre-query condition corresponding to each pre-query result; in a case where each pre-query dimension contains the expected query dimension and the pre-query dimension condition corresponding to the expected query dimension contains the expected dimension condition, determine the query result of the data query request based on the pre-query result corresponding to the expected query dimension.

[0018] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0019] Obtain a data query request for a data warehouse and a plurality of pre-query results pre-configured for the data warehouse; extract query conditions from the data query request; the query conditions are characterized by an expected query dimension and an expected dimension condition for the expected query dimension; determine the pre-query dimension and pre-query dimension condition of each pre-query result according to the pre-query condition corresponding to each pre-query result; in a case where each pre-query dimension contains the expected query dimension and the pre-query dimension condition corresponding to the expected query dimension contains the expected dimension condition, determine the query result of the data query request based on the pre-query result corresponding to the expected query dimension.

[0020] The above data query method, device, computer equipment, storage medium and computer program product obtain a data query request for a data warehouse and a plurality of pre-query results pre-configured for the data warehouse, extract query conditions from the data query request, and the query conditions are characterized by the expected query dimension and the expected dimension condition for the expected query dimension. The expected query dimension and the expected dimension condition are determined in the data query request, which is conducive to subsequent condition matching and helps to improve the efficiency of data query. According to the pre-query conditions corresponding to each pre-query result, the pre-query dimension and the pre-query dimension condition of each pre-query result are determined, and the pre-query dimension and the pre-query dimension condition are included in the pre-query result, which is conducive to subsequent condition matching. With the above method, when the expected query dimension is included in each pre-query dimension and the pre-query dimension condition corresponding to the expected query dimension includes the expected dimension condition, since the pre-query result is configured in advance, the query result of the data query request is determined based on the pre-query result corresponding to the expected query dimension, which saves the time of generating the query result, and there is no need to query the data in the data warehouse separately, which greatly improves the efficiency of data query. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 An application environment diagram of a data query method in an embodiment;

[0022] Figure 2 A schematic diagram of a flow chart of a data query method in an embodiment;

[0023] Figure 3 A flowchart of a method for realizing real-time insurance business analysis based on flow-batch integration technology in one embodiment;

[0024] Figure 4 is a structural block diagram of a data query device in an embodiment;

[0025] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] The data query method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0028] The server 104 may obtain a data query request for the data warehouse and a plurality of pre-query results pre-configured for the data warehouse from the terminal 102. The server 104 may extract a query condition from the data query request, and the query condition may be characterized by an expected query dimension and an expected dimension condition for the expected query dimension. The server 104 may determine the pre-query dimension and the pre-query dimension condition of each pre-query result according to the pre-query condition corresponding to each pre-query result. The server 104 may determine the query result of the data query request based on the pre-query result corresponding to the expected query dimension, when each pre-query dimension includes the expected query dimension and the pre-query dimension condition corresponding to the expected query dimension includes the expected dimension condition. The server 104 may send the query result to the terminal 102.

[0029] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0030] In one embodiment, Figure 2 As shown, a data query method is provided, which is applied to Figure 1 The server 104 in FIG. 1 is used as an example to illustrate, including:

[0031] S202, obtaining a data query request for the data warehouse and a plurality of pre-query results pre-configured for the data warehouse.

[0032] Among them, a data warehouse can be a system for storing and managing data from multiple sources. A data warehouse can integrate, clean and standardize data scattered in different operational databases to form a unified data view.

[0033] A data warehouse can be a real-time data warehouse, a batch data warehouse, or a hybrid data warehouse. A real-time data warehouse can process data in real time or near real time, usually supporting data delays of milliseconds to seconds, and is suitable for scenarios that require fast response. A batch data warehouse uses a batch processing mode, and data processing has a certain delay. It usually needs to wait until data collection is completed before processing, which has a long delay. A hybrid data warehouse can combine the advantages of a real-time data warehouse and a batch data warehouse, and is suitable for scenarios that need to process real-time data and historical data at the same time. This architecture can flexibly respond to different business needs, meeting the needs of real-time data analysis while retaining the ability to process historical data in batches. The implementation of a hybrid data warehouse often relies on a batch-stream integrated architecture. By introducing real-time stream processing technology, real-time data collection, processing, and analysis are achieved, greatly improving the timeliness of data.

[0034] The data query request may be a request for performing data query in a data warehouse, and may be any one of a Json-Server data query, a database query request, an API query request, and an HTTP request data query.

[0035] The pre-query result can be a query result pre-selected in the data warehouse for possible query requirements. For example, if a user prefers to query the payment method and payment amount, the pre-query result can be the specific method of payment and the specific value of the payment amount. Since the pre-query result is pre-configured information, in the data query scenario, it can save the process of querying data, matching data, and packaging data, reduce the time of querying data, and help improve data query efficiency.

[0036] Specifically, the server may determine its own service interface for querying the data warehouse, and the server may read a data query request for the data warehouse from the service interface.

[0037] In one of the embodiments, in order to avoid repeated queries and to relieve the computing pressure of the computing storage layer used to process data query requests, the server can cooperate with caching technology to avoid repeated queries. Specifically, the server can write the data query request into the server cache of the server and determine the query frequency of the server cache. The server can extract the query condition from the query request when the query frequency is less than or equal to the frequency threshold. The server can block the query process corresponding to the query request when the query frequency is less than or greater than the frequency threshold until the query frequency is less than or equal to the frequency threshold.

[0038] Specifically, the server may also obtain multiple historical query records. The server may determine pre-query results pre-configured for the data warehouse from multiple historical query records. The pre-query results may be historical query results with the same historical query dimension. The pre-query results may also be historical query results with the same historical query dimension and the same historical dimension conditions.

[0039] S204: extracting query conditions from the data query request.

[0040] The query condition may represent a desired query scope. The query may be characterized by a desired query dimension and a desired dimension condition for the desired query dimension.

[0041] The query condition may include at least one desired query dimension and at least one desired dimension condition for the at least one desired query dimension. For example, the query condition may include two desired query dimensions and respective desired dimension conditions for the two desired query dimensions.

[0042] The expected query dimension may be a query dimension expected by the user. For example, the expected query dimension may be institution number, insurance type, payment type, payment amount, etc.

[0043] The expected dimension condition may be a condition for the expected query dimension, for example, common insurance type, online payment type, payment amount less than XX yuan, etc.

[0044] The type of the data query request may be any one of a Json-Server data query request, a database query request, an API query request, and an HTTP request. Specifically, the server may determine a type extraction method according to the type corresponding to the data query request. The server extracts query conditions from the data query request according to the type extraction method.

[0045] S206: Determine the pre-query dimension and pre-query dimension condition of each pre-query result according to the pre-query condition corresponding to each pre-query result.

[0046] The pre-query dimension may represent the query dimension included in the pre-query result, for example, dimensions such as institution number, insurance type, payment type, and payment amount.

[0047] The pre-query dimension condition may indicate the query condition corresponding to the query dimension in the pre-query result, for example, common insurance, special insurance, online payment, prepayment, offline payment, payment amount less than YY yuan, etc.

[0048] It can be understood that a pre-query result may include a pre-query dimension and a pre-query dimension condition for the pre-query dimension.

[0049] S208, when each pre-query dimension includes the expected query dimension and the pre-query dimension condition corresponding to the expected query dimension includes the expected dimension condition, determine the query result of the data query request based on the pre-query result corresponding to the expected query dimension.

[0050] The query result of the data query request may represent the query result returned to the data warehouse. The query result of the data query request may be an encapsulated data query response message.

[0051] Each pre-query dimension includes an expected query dimension, which may indicate that there is a pre-query dimension that is the same as or similar to the expected query dimension in each pre-query dimension.

[0052] The pre-query dimension condition corresponding to the expected query dimension includes the expected dimension condition, which can indicate that the range corresponding to the pre-query dimension condition includes the range of the expected dimension condition. For example, the expected query dimension can be the payment amount, the pre-query dimension can be the payment amount, the expected query condition can be that the payment amount is less than 1,000 yuan, and if the pre-query dimension condition is that the payment amount is less than 1,500 yuan. It can be considered that the pre-query result including the pre-query dimension and the pre-query dimension condition can be the query result of the data query request.

[0053] The expected query dimension may correspond to multiple pre-query results. Specifically, the server may determine the query result of the data query request based on each pre-query message.

[0054] In one embodiment, the server may query and determine the result importance of each pre-query result of each pre-query dimension condition according to the condition similarity between each pre-query dimension condition and the expected dimension condition. The server may determine the pre-query result with the greatest result importance as the query result of the data query request.

[0055] In the above data query method, a data query request for a data warehouse and a plurality of pre-query results pre-configured for the data warehouse are obtained, and query conditions are extracted from the data query request. The query conditions are represented by the expected query dimension and the expected dimension condition for the expected query dimension. The expected query dimension and the expected dimension condition are determined in the data query request, which is conducive to subsequent condition matching and helps to improve the efficiency of data query. According to the pre-query conditions corresponding to each pre-query result, the pre-query dimension and the pre-query dimension condition of each pre-query result are determined, and the pre-query dimension and the pre-query dimension condition are included in the pre-query result, which is conducive to subsequent condition matching. With the above method, when the expected query dimension is included in each pre-query dimension and the pre-query dimension condition corresponding to the expected query dimension includes the expected dimension condition, since the pre-query result is configured in advance, the query result of the data query request is determined based on the pre-query result corresponding to the expected query dimension, which saves the time of generating the query result, and there is no need to query the data in the data warehouse separately, which greatly improves the efficiency of data query.

[0056] In one embodiment, the data query method also includes: obtaining multiple historical query records, each historical query record including a historical query dimension and a historical dimension condition for the historical query dimension, and when there are at least two target query records containing the same target query dimension in each historical query record, merging the historical dimension conditions corresponding to the target query dimension to obtain the target dimension condition, and constructing a pre-query result for the data warehouse based on the pre-query condition represented by the target query dimension and the target dimension condition.

[0057] The historical query record may represent the query history information of the data warehouse, or may represent the query history information obtained by each user when querying the data warehouse.

[0058] The historical query record may include historical query dimensions and historical dimension conditions for the historical query dimensions. The historical query dimensions may include dimensions such as institution number, insurance type, payment type, and payment amount. The historical dimension conditions may include the insurance type being a common insurance type, the insurance type being a special insurance type, the payment method being online payment, the payment method being online prepayment, the payment method being online and offline payment, and the payment amount being less than YY yuan.

[0059] The target dimension condition can be the merged historical dimension condition. For example, the historical dimension condition can be that the payment amount is 0 to 1,000 yuan, and the payment amount is 80 to 3,000 yuan. The target dimension condition can be 0 to 3,000 yuan.

[0060] The target query dimension may be the same query dimension in each historical query dimension. For example, if there are two historical query dimensions that are both payment amount dimensions, the payment amount dimension may be determined as the target query dimension.

[0061] Specifically, the server may filter multiple target historical query records in each historical query record. The historical query dimensions of each target historical query record are the same. The server may merge the historical dimension conditions of each historical query record to obtain the target dimension condition. The server may construct a pre-query result for the data warehouse based on the pre-query condition represented by the historical query dimension and the target dimension condition.

[0062] Specifically, the server may further select at least two target query results from each historical query result, and select the target query results having the same historical query dimension and the same historical dimension conditions of each target query result. The server may determine the target query result as a pre-query result pre-configured for the data warehouse.

[0063] In this embodiment, multiple historical query records are obtained, each of which includes a historical query dimension and a historical dimension condition for the historical query dimension. When there are at least two target query records containing the same target query dimension in each historical query record, the historical dimension conditions corresponding to the target query dimension are merged to obtain the target dimension condition. Based on the pre-query condition represented by the target query dimension and the target dimension condition, a pre-query result for the data warehouse is constructed. The pre-query result is constructed based on the historical query record. The pre-query result reflects the user's query dimension preference, thereby improving the accuracy of subsequent query results.

[0064] In one embodiment, the data query method also includes: when the expected query dimension is not included in each pre-query dimension, or the pre-query dimension condition corresponding to the expected query dimension does not include the expected dimension condition, determining multiple dimensional query values ​​that match the expected query dimension and the expected dimension condition in the fact table of the data warehouse, and statistically analyzing the query values ​​of each dimension according to the statistical method of matching the expected query dimension to obtain the indicator value corresponding to the expected query dimension, and determining the query result of the data query request based on the indicator value.

[0065] Among them, the fact table of the data warehouse can be a key component for storing and analyzing business measurement data. The fact table includes information such as measurement values ​​and dimension keys. Measurement values ​​can represent the main data in the fact table and can perform numerical calculations, such as sales, order quantity, product unit price, etc. Measurement values ​​are divided into cumulative and non-cumulative types, among which cumulative measurements are the most useful and can provide meaningful summary information. Dimension keys can represent foreign keys pointing to dimension tables, through which fact tables are connected to dimension tables, thereby providing contextual information for measurements. Dimension tables contain attributes related to specific facts.

[0066] The fact table specifically includes multiple query values ​​contained in each dimension.

[0067] The dimension query value may be a query value in the fact table that matches the desired query dimension and the desired dimension condition.

[0068] The statistical method may indicate a specific calculation method of the dimension query value. The number of dimension query values ​​may be multiple. The statistical method may be summing, taking extreme values, etc.

[0069] The index value may represent a statistical value corresponding to each dimension query value. For example, the index value may be at least one of an average value corresponding to each dimension query value, a maximum value corresponding to each dimension query value, and a minimum value corresponding to each dimension query value.

[0070] Specifically, the server may determine the indicator value as the query result of the data query request.

[0071] In this embodiment, when the expected query dimension is not included in each pre-query dimension, or the pre-query dimension condition corresponding to the expected query dimension does not include the expected dimension condition, multiple dimensional query values ​​that match the expected query dimension and the expected dimension condition are determined in the fact table of the data warehouse, and the query values ​​of each dimension are counted according to the statistical method of the expected query dimension match to obtain the indicator value corresponding to the expected query dimension, and the query result of the data query request is determined based on the indicator value. The query result includes the indicator value, which is conducive to providing a more intuitive reference for the user.

[0072] In one embodiment, the data warehouse includes an updated real-time data warehouse, and the data query method also includes: obtaining the full data table in the real-time data warehouse and the incremental data table within a preset processing cycle, batch processing the full data table and the incremental data table according to the preset processing cycle to obtain batch processing results, and updating the real-time data warehouse according to the batch processing results to obtain an updated real-time data warehouse.

[0073] Among them, the real-time data warehouse can process data in real time or near real time, usually supporting data delays of milliseconds to seconds, and is suitable for scenarios that require fast response.

[0074] A full data table can store all data at a certain point in time, regardless of whether the data has changed. Every time it is updated, the old data will be overwritten, so historical data changes cannot be recorded.

[0075] Incremental data tables can represent data that has changed since the last data load, and do not contain data that has not changed.

[0076] The expected processing cycle can be a manually configured processing cycle of the data warehouse, for example, one hour or one day.

[0077] The batch processing result may indicate the processing result of a data table. The batch processing result may include the full processing result and the incremental processing result. The full processing result may indicate the result obtained by processing the full data table. The incremental processing result may indicate the result obtained by processing the incremental data table and the full data table. The full data table needs to recalculate all data, while the incremental data table only needs to process the newly added data. For batch processing of incremental data, the incremental data is usually merged with the full data to generate the full data at the current time point.

[0078] The batch processing result may include a full processing result and an incremental processing result. Specifically, the server may update the real-time data warehouse according to the full processing result and the incremental processing result to obtain an updated real-time data warehouse.

[0079] In one embodiment, the server may update the real-time data warehouse according to the full processing result to obtain a first data warehouse. The server may update the real-time data warehouse according to the incremental processing result to obtain a second data warehouse. The server may determine the data warehouse with higher integrity, whichever is the first data warehouse or the second data warehouse, as the updated real-time data warehouse.

[0080] In this embodiment, the full data table in the real-time data warehouse and the incremental data table within a preset processing cycle are obtained, and the full data table and the incremental data table are batch processed according to the preset processing cycle to obtain the batch processing results. The real-time data warehouse is updated according to the batch processing results to obtain the updated real-time data warehouse, and the data warehouse is updated to maintain the accuracy of the data, thereby improving the accuracy of subsequent data queries.

[0081] In one embodiment, extracting query conditions from a data query request includes: writing the data query request into a server cache of a server, determining a query frequency of the server cache, extracting the query conditions from the data query request when the query frequency is less than or equal to a frequency threshold, and blocking a query process corresponding to the data query request when the query frequency is less than or greater than the frequency threshold until the query frequency is less than or equal to the frequency threshold.

[0082] The server cache may refer to the memory or disk of the server where data is temporarily stored.

[0083] The query frequency can represent the call frequency of the server's memory or disk. The higher the query frequency, the more query requests and the more query times of the query requests. The query frequency can be dynamically changed. The query frequency can be represented by the ratio of the number of query requests and the number of query times of the query requests to the unit time.

[0084] The frequency threshold may represent a threshold configured for the query frequency.

[0085] The query process may represent a server CPU process for processing a query request.

[0086] Blocking a query process means that the process no longer occupies CPU resources while waiting for an event to occur, but gives up CPU resources to other processes. This mechanism helps improve the overall efficiency of the system because when a process is blocked, the CPU can be used to perform other tasks.

[0087] Specifically, the server may determine the query frequency of the server cache in real time. The server may extract the query condition from the data query request when the query frequency is less than or equal to the frequency threshold. The server may block the query process corresponding to the data query request when the query frequency is less than or equal to the frequency threshold until the query frequency is less than or equal to the frequency threshold.

[0088] In this embodiment, the data query request is written into the server cache of the server, and the query frequency of the server cache is determined. When the query frequency is less than or equal to the frequency threshold, the query condition is extracted from the data query request. When the query frequency is less than or greater than the frequency threshold, the query process corresponding to the data query request is blocked until the query frequency is less than or equal to the frequency threshold, thereby avoiding repeated queries and sending multiple query requests to the computing storage layer, thereby improving data query efficiency.

[0089] In one embodiment, the data query method further includes: determining the computing capacity of the server, determining a frequency threshold that matches the computing capacity, and the computing capacity and the frequency threshold are positively correlated.

[0090] Among them, computing power can represent the processing ability of the server for various tasks.

[0091] Stronger computing power means that the server can process various tasks (such as query tasks) faster, and accordingly, the server can process more tasks.

[0092] There are many ways to determine the computing capacity of the server. Specifically, the server can obtain its own CPU information, memory information, and storage information. The server can determine the computing capacity of the server based on the CPU information, memory information, and storage information.

[0093] Among them, CPU information can represent relevant information of the server CPU. The server CPU is the core component of the server, responsible for executing program instructions and processing data. For example, CPU information may include parameters such as the number of cores, main frequency, and cache size. A high-performance multi-core CPU can handle multiple tasks at the same time, improving the processing speed and efficiency of the server.

[0094] Memory information can represent information related to the server's memory. Server memory is where the server runs programs and stores temporary data. Memory capacity and speed directly affect the server's read and write speed and data processing capabilities. Larger memory capacity can ensure that the server can handle more tasks at the same time, while high-speed memory can increase the read and write speed of data, thereby improving overall computing efficiency.

[0095] Storage information can represent the relevant information of server storage. Server storage is used to store data for a long time. Common types include mechanical hard disk drives (HDDs) and solid-state drives (SSDs). SSDs usually provide faster read and write speeds and are suitable for applications that require higher data read and write speeds. The choice of storage capacity should be determined based on business needs to ensure that there is enough space to store data.

[0096] In one embodiment, the server may determine the first computing capacity, the second computing capacity, and the third computing capacity respectively based on the CPU information, the memory information, and the storage information. The server may determine the smallest computing capacity among the first computing capacity, the second computing capacity, and the third computing capacity as the computing capacity of the server.

[0097] In this embodiment, the computing power of the server is determined, and a frequency threshold that matches the computing power is determined. The computing power and the frequency threshold are positively correlated. The higher the frequency threshold, the more data query requests are processed per unit time, thereby improving the data query efficiency as much as possible.

[0098] In one embodiment, Figure 3 As shown, a method for realizing real-time insurance business analysis based on flow-batch integration technology is provided, including:

[0099] S301, obtaining multiple historical query records.

[0100] Each historical query record includes a historical query dimension and a historical dimension condition for the historical query dimension.

[0101] Exemplarily, a CDC tool supporting a business database is used to collect incremental data generated by the business system into an incremental data storage location in a fixed format.

[0102] Kafka is used as the message middleware to receive data transmitted by the upper-level collection module in real time, and seven days of incremental data are saved by default. Since the received data are all CURD logs of the business database, the order of receiving data needs to be guaranteed, so separate TOPICs are set for different business systems, and each TOPIC has only one partition.

[0103] Flink is used as a streaming computing engine to consume data in Kafka. After simple calculations, the data is pushed to the downstream computing and storage modules. All programs in this module are based on Exactly Once semantics and cooperate with Flink's checkpoint function to ensure the integrity and consistency of incremental data processing.

[0104] The distributed database Doris is used as the backend computing storage engine of the entire real-time data warehouse and the query engine of the report. Its asynchronous materialized view function is mainly used to complete the complex association and calculation of incremental data at a high frequency.

[0105] Oceanbase is used to save the dimensional data or related full data required for incremental data association to support Flink's real-time, high-frequency, fast short queries.

[0106] The batch data processing is completed by using a scheduled scheduling method, and the batch calculation results are pushed to the real-time data warehouse so that the data in the real-time data warehouse can be verified and repaired.

[0107] The front-end reporting system can directly connect to this module to obtain real-time data warehouse data for display. The module uses API calls internally and cooperates with caching technology to avoid repeated queries and send multiple query requests to the computing storage layer.

[0108] Provides an efficient development platform and guaranteed running stability for Flink programs. The program has built-in script mode and wizard mode. The wizard mode allows developers to visually complete the development of a Flink program. It is friendly to developers of different levels and provides excellent support for one-click deployment, resource management, monitoring and alarm, and application restart of Flink programs.

[0109] S302: When there are at least two target query records containing the same target query dimension in each historical query record, merge each historical dimension condition corresponding to the target query dimension to obtain the target dimension condition.

[0110] S303: construct a pre-query result for the data warehouse based on the pre-query condition represented by the target query dimension and the target dimension condition.

[0111] S304, obtaining a data query request for the data warehouse and a plurality of pre-query results pre-configured for the data warehouse.

[0112] The data warehouse includes the updated real-time data warehouse, and the full data table in the real-time data warehouse and the incremental data table within the preset processing cycle are obtained. According to the preset processing cycle, the full data table and the incremental data table are batch processed to obtain the batch processing results. According to the batch processing results, the real-time data warehouse is updated to obtain the updated real-time data warehouse.

[0113] S305: Determine the computing capacity of the server.

[0114] S306: Determine a frequency threshold that matches the computing capability.

[0115] Among them, computing power and frequency threshold are positively correlated.

[0116] S307, writing the data query request into the server cache of the server, and determining the query frequency of the server cache.

[0117] S308: When the query frequency is less than or equal to the frequency threshold, extract the query condition from the data query request.

[0118] S309: When the query frequency is less than or equal to the frequency threshold, the query process corresponding to the data query request is blocked until the query frequency is less than or equal to the frequency threshold.

[0119] The query condition is represented by an expected query dimension and an expected dimension condition for the expected query dimension.

[0120] S310: Determine the pre-query dimension and pre-query dimension condition of each pre-query result according to the pre-query condition corresponding to each pre-query result.

[0121] S311, determine whether each pre-query dimension includes the expected query dimension, and whether the pre-query dimension condition corresponding to the expected query dimension includes the expected dimension condition. If so, execute S312, if not, execute S313.

[0122] S312: Determine a query result of the data query request based on the pre-query result corresponding to the expected query dimension.

[0123] S313, determining multiple dimensional query values ​​matching the expected query dimension and the expected dimensional condition in the fact table of the data warehouse.

[0124] S314, statistics are performed on the query values ​​of each dimension according to the statistical method matched with the expected query dimension to obtain the index value corresponding to the expected query dimension.

[0125] S315: Determine a query result of the data query request based on the indicator value.

[0126] In this embodiment, a data query request for a data warehouse and a plurality of pre-query results pre-configured for the data warehouse are obtained, and query conditions are extracted from the data query request. The query conditions are represented by the expected query dimension and the expected dimension condition for the expected query dimension. The expected query dimension and the expected dimension condition are determined in the data query request, which is conducive to subsequent condition matching and helps to improve the efficiency of data query. According to the pre-query conditions corresponding to each pre-query result, the pre-query dimension and the pre-query dimension condition of each pre-query result are determined, and the pre-query dimension and the pre-query dimension condition are included in the pre-query result, which is conducive to subsequent condition matching. With the above method, when the expected query dimension is included in each pre-query dimension and the pre-query dimension condition corresponding to the expected query dimension includes the expected dimension condition, since the pre-query result is configured in advance, the query result of the data query request is determined based on the pre-query result corresponding to the expected query dimension, which saves the time of generating the query result, and there is no need to query the data in the data warehouse separately, which greatly improves the efficiency of data query.

[0127] The above method can also bring the following beneficial effects: (1) It lowers the threshold for real-time data warehouse construction, reduces the number of technical components involved, and does not require learning a large number of big data technologies. (2) It reduces the difficulty of implementation. Except for the data collection step, other steps can be developed in the form of SQL. (3) It improves the stability of the real-time data warehouse architecture and reduces the risk of instability caused by Flink programs. (4) The Doris database and OceanBase computing engine used are both domestically produced trust products and have high universality in the industry.

[0128] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0129] Based on the same inventive concept, the embodiment of the present application also provides a data query device for implementing the data query method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data query device embodiments provided below can refer to the limitations on the data query method above, and will not be repeated here.

[0130] In one embodiment, Figure 4 As shown, a data query device is provided, including: an acquisition module 401, a condition extraction module 402, a pre-query module 403 and a query module 404, wherein:

[0131] The acquisition module 401 is used to acquire a data query request for the data warehouse and a plurality of pre-query results pre-configured for the data warehouse;

[0132] The condition extraction module 402 is used to extract the query condition from the data query request; the query condition is represented by the expected query dimension and the expected dimension condition for the expected query dimension;

[0133] The pre-query module 403 is used to determine the pre-query dimension and pre-query dimension condition of each pre-query result according to the pre-query condition corresponding to each pre-query result;

[0134] The query module 404 is used to determine the query result of the data query request based on the pre-query result corresponding to the expected query dimension when the expected query dimension is included in each pre-query dimension and the pre-query dimension condition corresponding to the expected query dimension includes the expected dimension condition.

[0135] In one of the embodiments, the data query device also includes: a historical query module, used to obtain multiple historical query records; each historical query record includes a historical query dimension, and a historical dimension condition for the historical query dimension; when there are at least two target query records containing the same target query dimension in each historical query record, the historical dimension conditions corresponding to the target query dimension are merged to obtain the target dimension condition; based on the pre-query condition represented by the target query dimension and the target dimension condition, a pre-query result for the data warehouse is constructed.

[0136] In one of the embodiments, the data query device also includes: an indicator value determination module, which is used to determine multiple dimensional query values ​​that match the expected query dimension and the expected dimension condition in the fact table of the data warehouse when the expected query dimension is not included in each pre-query dimension or the pre-query dimension condition corresponding to the expected query dimension does not include the expected dimension condition; statistics are performed on the query values ​​of each dimension according to the statistical method of matching the expected query dimension to obtain the indicator value corresponding to the expected query dimension; and the query result of the data query request is determined based on the indicator value.

[0137] In one of the embodiments, the data query device also includes: a data warehouse update module, which is used to obtain the full data table in the real-time data warehouse and the incremental data table within a preset processing cycle; batch processing the full data table and the incremental data table according to the preset processing cycle to obtain batch processing results; and update the real-time data warehouse according to the batch processing results to obtain an updated real-time data warehouse.

[0138] In one of the embodiments, the condition extraction module 402 is also used to write the data query request into the server cache of the server and determine the query frequency of the server cache; when the query frequency is less than or equal to the frequency threshold, extract the query condition from the data query request; when the query frequency is less than or greater than the frequency threshold, block the query process corresponding to the data query request until the query frequency is less than or equal to the frequency threshold.

[0139] In one of the embodiments, the data query device further includes: a threshold determination module, which is used to determine the computing capacity of the server; determine a frequency threshold that matches the computing capacity, and the computing capacity and the frequency threshold are positively correlated.

[0140] Each module in the above data query device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0141] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data query request data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data query method is implemented.

[0142] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the above method steps are implemented when the processor executes the computer program.

[0144] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method steps are implemented.

[0145] In one embodiment, a computer program product is provided, comprising a computer program, which implements the above method steps when executed by a processor.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0147] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data query method, characterized in that: The method comprises: Obtaining a data query request for a data warehouse and a plurality of pre-query results pre-configured for the data warehouse; Extracting a query condition from the data query request; the query condition is represented by an expected query dimension and an expected dimension condition for the expected query dimension; Determining the pre-query dimensions and pre-query dimension conditions of each pre-query result according to the pre-query conditions corresponding to each pre-query result; In a case where each of the pre-query dimensions includes the expected query dimension and the pre-query dimension condition corresponding to the expected query dimension includes the expected dimension condition, the query result of the data query request is determined based on the pre-query result corresponding to the expected query dimension.

2. The method according to claim 1, characterized in that The method further comprises: Acquire multiple historical query records; each of the historical query records includes a historical query dimension and a historical dimension condition for the historical query dimension; In the case where there are at least two target query records containing the same target query dimension in each of the historical query records, merging the historical dimension conditions corresponding to the target query dimension to obtain the target dimension condition; Based on the pre-query condition represented by the target query dimension and the target dimension condition, a pre-query result for the data warehouse is constructed.

3. The method according to claim 1, characterized in that The method further comprises: In a case where the expected query dimension is not included in each of the pre-query dimensions, or the pre-query dimension condition corresponding to the expected query dimension does not include the expected dimension condition, determining a plurality of dimensional query values ​​matching the expected query dimension and the expected dimension condition in a fact table of the data warehouse; According to the statistical method of matching the expected query dimension, statistics are performed on the query values ​​of each dimension to obtain the index value corresponding to the expected query dimension; Based on the indicator value, a query result of the data query request is determined.

4. The method according to claim 1, characterized in that: The data warehouse includes an updated real-time data warehouse, and the method further includes: Obtain the full data table in the real-time data warehouse and the incremental data table within the preset processing cycle; According to the preset processing cycle, batch processing is performed on the full data table and the incremental data table to obtain batch processing results; The real-time data warehouse is updated according to the batch processing result to obtain an updated real-time data warehouse.

5. The method according to claims 1 to 4, characterized in that The extracting the query condition from the data query request includes: Writing the data query request into a server cache of the server, and determining a query frequency of the server cache; When the query frequency is less than or equal to a frequency threshold, extracting a query condition from the data query request; When the query frequency is less than or equal to the frequency threshold, the query process corresponding to the data query request is blocked until the query frequency is less than or equal to the frequency threshold.

6. The method according to claim 5, characterized in that The method further comprises: determining the computing capacity of the server; A frequency threshold matching the computing capability is determined, wherein the computing capability and the frequency threshold are positively correlated.

7. A data query device, characterized in that: The device comprises: An acquisition module, used to acquire a data query request for a data warehouse and a plurality of pre-query results pre-configured for the data warehouse; A condition extraction module, used to extract a query condition from the data query request; the query condition is represented by an expected query dimension and an expected dimension condition for the expected query dimension; A pre-query module, used to determine the pre-query dimension and pre-query dimension condition of each of the pre-query results according to the pre-query condition corresponding to each of the pre-query results; A query module is used to determine the query result of the data query request based on the pre-query result corresponding to the expected query dimension, when the expected query dimension is included in each of the pre-query dimensions and the pre-query dimension condition corresponding to the expected query dimension includes the expected dimension condition.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images