Index data query method, device and equipment

Abstract

The invention discloses an index data query method, device and equipment, and relates to the technical field of computers, the method comprises the following steps: receiving a query index request initiated by a user, and obtaining a query dimension associated with a query index; judging whether query dimensions associated with the query indexes are materialized and stored in a materialized view of a fast area or not; if the query dimensions associated with the query indexes are materialized and stored in the materialized view of the fast area, reading data of the query indexes from the materialized view of the fast area; otherwise, reading the data of the query index from the common area in a full-table scanning mode; and integrating the data of the plurality of query indexes read from the fast area and the common area, and generating a unified query result based on preset service logic. According to the method, the existence of all dimensions associated with each query index in the fast zone materialization view is dynamically judged, the optimal data reading path is intelligently selected for different indexes, and the overall query response time is shortened.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus and device for querying indicator data. Background Technology

[0002] As enterprises deepen their digital transformation, data analytics has become a core support for operational decision-making. In a massive data environment, efficiently performing multi-dimensional, interactive, real-time data queries and analyses is a key technological requirement for current data platforms. Especially in typical scenarios such as business statistics, users need to flexibly and quickly explore and visualize the same set of core indicators based on different dimensional combinations, which places extremely high demands on the response speed and flexibility of underlying data services.

[0003] Currently, common industry solutions primarily rely on a single data storage model. One approach is to directly respond to queries using a full table scan based on a data table storing all detailed data. While this approach ensures query flexibility and data integrity, it significantly increases query time when handling multi-dimensional cross-references or complex joins due to the massive amount of data that needs to be scanned and computed, making it difficult to meet the performance requirements of interactive analysis. Another approach is to pre-create materialized views for specific dimension combinations, making query performance optimization dependent on the hit rate of these materialized views. However, when the dimension combinations involved in a user's query request are not fully covered by any materialized view, the system cannot effectively utilize the existing optimized structure and must still revert to the inefficient full table scan path.

[0004] Therefore, the core flaw of existing technologies lies in their inability to intelligently adapt to dynamically changing query needs when querying multiple datasets in combination, resulting in low overall query performance.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of this application is to provide a method, apparatus, and device for querying indicator data, which aims to solve the technical problem of low overall query performance when querying multiple datasets in a mixed manner, as the data cannot be intelligently adapted to dynamically changing query requirements.

[0007] To achieve the above objectives, this application proposes a method for querying indicator data, the method comprising: Receive query metrics requests initiated by users and obtain the query dimensions associated with the query metrics; Determine whether all query dimensions associated with the query metrics have been materialized and stored in the materialized view of the fast zone; If all query dimensions associated with the query metric have been materialized and stored in the materialized view of the fast zone, then the data of the query metric is read from the materialized view of the fast zone; otherwise, the data of the query metric is read from the normal zone through a full table scan. Data from multiple query metrics read from the fast zone and the normal zone are integrated, and a unified query result is generated based on preset business logic.

[0008] In one embodiment, the fast area is used to store data containing frequently used dimensions, and the normal area is used to store source data tables containing all dimensions.

[0009] In one embodiment, constructing the materialized view of the fast zone includes: Based on historical query logs, analyze and identify frequently used metrics and dimension combinations; Based on the combination of frequently used indicators and dimensions, the source data table of the ordinary area is pre-aggregated and calculated to generate a materialized view. The materialized view is stored in the fast zone.

[0010] In one embodiment, the step of analyzing and identifying frequently used combinations of metrics and dimensions based on historical query logs includes: Extract a set of triples from historical query logs, where the triples include indicators, dimension combinations, and query frequency; A multi-objective optimization function is constructed with the goal of maximizing total query revenue and minimizing total storage cost. A fast non-dominated sorting genetic algorithm is used to optimize the set of triples under a given total storage space budget constraint to obtain a Pareto optimal solution set. Based on a preset business priority strategy, a set of materialized views to be constructed is selected from the Pareto optimal solution set.

[0011] In one embodiment, the step of determining whether all query dimensions associated with the query metrics have been materialized and stored in the materialized view of the fast zone includes: Construct a dimension-storage location mapping table, in which each query dimension in the mapping table corresponds to a unique dimension feature vector; Convert the query dimensions associated with the query metrics into a corresponding set of dimensional feature vectors; Using the set of dimensional feature vectors as keys, a fast prediction component built on a Bloom filter is queried in batches. If the fast prediction component returns that no feature vector exists in any dimension, the query result is determined to be negative, and the data reading process from the ordinary area is triggered. If the feature vectors of the dimensions returned by the fast prediction component all exist, then continue to query the dimension-storage location mapping table to confirm for the second time that the query dimensions associated with the query indicators have been materialized and stored in the materialized view of the fast area.

[0012] In one embodiment, the update step of the fast region includes: Monitor new query logs and calculate the popularity trends of various metric-dimension combinations; The fast non-dominated sorting genetic algorithm is executed based on a preset time period to generate a new Pareto optimal solution set. The new Pareto optimal solution set is compared with the currently materialized view to generate a view update operation sequence, which includes a new materialized view to be constructed and an old materialized view to be discarded. The view update operation sequence is executed during periods of low system load.

[0013] In one embodiment, the step of integrating data from multiple query metrics read from the fast zone and the normal zone, and generating a unified query result based on preset business logic, includes: Assign a unified query batch identifier to data result sets that are read from different data areas but belong to the same query indicator request; Based on the query batch identifier, the data result set is loaded into the distributed memory workspace; Within the distributed memory workspace, based on the display dimensions specified in the query metric request, data from different data areas are subjected to dimension alignment, association, and aggregation processing to generate intermediate results with unified dimensions. The intermediate results are calculated based on preset business logic to generate unified query results.

[0014] In one embodiment, the step of dimensional alignment of data from different data regions includes: For each data result set, construct a memory bitmap index on the stated display dimension; Based on the memory bitmap index, AND, OR, and NOT logical operations are performed to identify the intersection and difference relationships of different data result sets in the display dimension; Based on the intersection and difference relationships, zero-padding, filling, or association operations are performed on the data in different data areas to complete dimension alignment.

[0015] Furthermore, to achieve the above objectives, this application also proposes an indicator data query device, which includes: The request receiving module is used to receive query metric requests initiated by users and obtain the query dimensions associated with the query metrics. The dimension judgment module is used to determine whether all query dimensions associated with the query metrics have been materialized and stored in the materialized view of the fast zone; The data reading module is used to read the data of the query indicator from the materialized view of the fast zone if all the query dimensions associated with the query indicator have been materialized and stored in the materialized view of the fast zone; otherwise, it reads the data of the query indicator from the normal zone through a full table scan. The data integration module is used to integrate data from multiple query indicators read from the fast zone and the normal zone, and generate unified query results based on preset business logic.

[0016] In addition, to achieve the above objectives, this application also proposes an indicator data query device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the indicator data query method described above.

[0017] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the indicator data query method described above.

[0018] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the indicator data query method described above.

[0019] One or more technical solutions proposed in this application have at least the following technical effects: This application receives query metric requests initiated by users, obtains the query dimensions associated with the query metric, determines whether all query dimensions associated with the query metric have been materialized and stored in the materialized view of the fast zone, and if so, reads the data of the query metric from the materialized view of the fast zone; otherwise, reads the data of the query metric from the normal zone using a full table scan; integrates the data of multiple query metrics read from the fast zone and the normal zone, and generates a unified query result based on preset business logic. This application dynamically determines the existence of all dimensions associated with each query metric in the materialized view of the fast zone, intelligently selects the optimal data reading path for different metrics, and reduces the overall query response time. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the first embodiment of the indicator data query method in this application. Figure 2 This is a flowchart illustrating the second embodiment of the indicator data query method in this application. Figure 3 This is a flowchart illustrating the third embodiment of the indicator data query method in this application. Figure 4 This is a schematic diagram of the module structure of the indicator data query device according to an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the indicator data query method in this application embodiment.

[0023] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0024] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0025] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0026] Because existing technologies cannot intelligently adapt to dynamically changing query needs when querying multiple datasets in combination, the overall query performance is low.

[0027] This application provides a solution that receives a user-initiated query indicator request, obtains the query dimensions associated with the query indicator, determines whether all the query dimensions associated with the query indicator have been materialized and stored in the materialized view of the fast zone, and if all the query dimensions associated with the query indicator have been materialized and stored in the materialized view of the fast zone, then reads the data of the query indicator from the materialized view of the fast zone; otherwise, reads the data of the query indicator from the normal zone through a full table scan; integrates the data of multiple query indicators read from the fast zone and the normal zone, and generates a unified query result based on preset business logic.

[0028] Based on this, the embodiments of this application provide a method for querying indicator data, referring to... Figure 1 , Figure 1This is a flowchart illustrating the first embodiment of the indicator data query method of this application.

[0029] In this embodiment, the indicator data query method includes steps S10 to S40: Step S10: Receive the query indicator request initiated by the user and obtain the query dimension associated with the query indicator; It should be noted that, in this embodiment, a query indicator request refers to an instruction generated by a user through interactive operations such as selection and dragging on the front-end visual analysis interface, requesting specific business metrics, including at least one or more query indicators. Query indicators are numerical metrics used to measure business performance, such as sales revenue, order volume, and user activity. Query dimensions refer to the angle or grouping criteria for observing and analyzing these business indicators, such as time dimensions (year, month, day), regional dimensions (country, province, city), and product dimensions. By receiving and parsing the user's original request, each query indicator contained in the request is accurately identified, and all query dimensions logically related to these query indicators are further obtained, ensuring that the system can accurately respond to the user's complex multidimensional analysis needs.

[0030] Step S20: Determine whether all query dimensions associated with the query metrics have been materialized and stored in the materialized view of the fast zone; It should be noted that, in this embodiment, the fast zone refers to a dedicated data storage area in the system for storing high-performance query results, distinct from the ordinary zone that stores the original full data; a materialized view refers to a database object that physically stores predefined query results (usually pre-aggregated data), aiming to avoid lengthy real-time connections and aggregation calculations during queries, thereby significantly improving query speed. By determining the state of the indicator-dimensional combination in the fast zone, the system accurately determines the optimal data acquisition source for each query indicator, such as directly reading efficient materialized results or reverting to the basic data source for calculation.

[0031] Step S30: If all query dimensions associated with the query metric have been materialized and stored in the materialized view of the fast zone, then read the data of the query metric from the materialized view of the fast zone; otherwise, read the data of the query metric from the normal zone using a full table scan. It should be noted that, in this embodiment, the ordinary area refers to the storage area containing the source data table with full dimension and detailed data; the full table scan method refers to an operation mode in which the database system directly reads and processes the entire or most of the data table, which is usually accompanied by high I / O overhead and computation latency. This embodiment executes a differentiated data acquisition strategy based on the judgment result of step S20. For query indicators with a judgment result of "yes", the system directly reads its calculated data from the materialized view of the fast area, avoiding complex real-time calculations and improving response speed.

[0032] Step S40: Integrate the data of multiple query indicators read from the fast zone and the normal zone, and generate a unified query result based on preset business logic.

[0033] It should be noted that, in this embodiment, integration refers to the process of merging, associating, calculating, and ultimately organizing multiple datasets from different data sources, which may have different data structures and granularities, into a regular data result set that can be directly used by the front end; the preset business logic refers to predefined processing rules used to standardize indicator technical rules, data association conditions, and the final result format. In this embodiment, since different query indicators may be read from the "fast zone" and the "normal zone" respectively, the returned data needs to be aligned and associated under a common display dimension to ensure that the final output of unified query results is consistent and accurate in terms of data structure and business meaning.

[0034] In one possible practical approach, the system assigns a query batch identifier to all data subsets of the same query request, loads the query batch identifier into a distributed memory workspace, and then quickly aligns the data based on technologies such as bitmap indexes. Finally, it performs association and calculations according to business logic to generate a unified result.

[0035] This embodiment establishes a two-layer storage structure of fast zone and normal zone, and performs dynamic routing judgment and data reading based on query index granularity. The system can automatically guide high-frequency queries to efficient materialized views, while ensuring full-dimensional data support for low-frequency or special queries. With the help of an efficient data integration engine, the results of heterogeneous data sources are seamlessly unified. Thus, while maintaining query flexibility, the overall response time under the mixed query mode is significantly reduced, and the efficiency and resource utilization of massive data interaction analysis are improved.

[0036] In one possible implementation, the fast zone is used to store data containing frequently used dimensions, and the normal zone is used to store source data tables containing all dimensions.

[0037] It should be noted that, in this embodiment, "full dimension" refers to all possible analytical perspectives defined and collected by the enterprise under a specific business theme, ensuring data completeness; the source data table refers to the finest-grained, unaggregated raw data set. The fast zone does not physically store the raw data directly, but rather stores materialized views generated after pre-aggregation calculations based on the source data tables of the ordinary zone; these materialized views are projections and summaries of the ordinary zone data on specific high-frequency dimension combinations.

[0038] In one possible implementation, the construction of the materialized view of the fast zone includes: Based on historical query logs, analyze and identify frequently used metrics and dimension combinations; Based on the combination of frequently used indicators and dimensions, the source data table of the ordinary area is pre-aggregated and calculated to generate a materialized view. The materialized view is stored in the fast zone.

[0039] It should be noted that, in this embodiment of the application, the typical recorded content of the historical query log includes the queried indicator, the dimension used, the filtering conditions, the query duration, and the execution duration; the combination of frequently used indicators and dimensions refers to the pairing relationship between specific indicators and dimensions identified from the historical query log through statistical analysis, which are repeatedly requested more than a certain threshold within the statistical time window; pre-aggregation calculation refers to the process of performing grouping, summarizing, and other calculation operations on the source data table of the ordinary area in advance according to specific indicator and dimension combinations before the user query occurs, and persistently storing the calculation results.

[0040] This embodiment uses intelligent analysis and preprocessing to pre-calculate and store the most likely accessed data in the fast zone, thereby shifting the real-time computing burden during queries to the system's idle time and achieving a qualitative improvement in query performance. Through in-depth analysis of historical query patterns, the system can accurately allocate limited storage and computing resources to the data combinations that best enhance the overall query experience, achieving intelligent query acceleration.

[0041] In one specific implementation, the "normal zone" of an enterprise-level data platform might store a sales fact table named `fact_sales`, which includes metrics such as sales products and payment periods, as well as key-value pairs related to dimensions such as ranking, customers, salespersons, and warranty periods. Through analysis of historical queries, the system discovered that over 80% of queries only involve the combination of the two dimensions: sales products and ranking. Therefore, the system pre-calculates and stores a materialized view named `mv_sales_summary` in the fast zone. This view only contains the two high-frequency dimensions of sales products and ranking, along with aggregated metrics such as the total number of current sales products. When a query hits these dimensions, the system can directly read data from this materialized view, avoiding a full table scan of the massive `fact_sales` table.

[0042] Furthermore, referring to Figure 2 The second embodiment of the indicator data query method of this application provides a flowchart, based on the above. Figure 2 The illustrated embodiment further refines the step of "analyzing and identifying frequently used metrics and dimension combinations based on historical query logs," including steps A201 to A204: Step A201: Extract a set of triples based on historical query logs. The triples include indicators, dimension combinations, and query frequency. It should be noted that in this embodiment, each triple contains three basic elements: the metric refers to the specific business metric targeted by the query request, such as sales revenue or number of users; the dimension combination refers to the ordered set of all dimensions used in this query, such as [time dimension, region dimension]; and the query frequency refers to the total number of times this specific metric and dimension combination was queried within the statistical time range. This step is performed by the log analysis module, which transforms the unstructured historical query logs into a normalized data set that can be processed by subsequent algorithms.

[0043] Step A202: Construct a multi-objective optimization function with the optimization objectives of maximizing total query revenue and minimizing total storage cost; It should be noted that, in this embodiment, the total query benefit refers to the total query time that the system may save over its lifetime if a materialized view is constructed. Its calculation is typically based on the product of the view's query frequency and the performance improvement per query. The total storage cost refers to the sum of physical storage space required to store all selected materialized views. By simultaneously considering both performance gains and resource consumption, the system can avoid extreme situations that may result from single-objective optimization. For example, the query benefit defined by the system for each triple is the product of its query frequency and the estimated time required to perform a full table scan if the query misses the materialized view.

[0044] Step A203: Using the fast non-dominated sorting genetic algorithm, the set of triples is optimized under the given total storage space budget constraint to obtain the Pareto optimal solution set; It should be noted that, in this embodiment, the fast non-dominated sorting genetic algorithm searches for solutions that simultaneously optimize multiple objectives in the solution space by simulating the natural selection process; the total storage space budget constraint refers to the upper limit of physical storage space allocated by the system to the fast region; the Pareto optimal solution set refers to the set of solutions among all feasible solutions that cannot improve any objective without harming other objectives, representing the best trade-off between different objectives. The fast non-dominated sorting genetic algorithm iteratively evolves the candidate solution population through operations such as selection, crossover, and mutation, and uses fast non-dominated sorting and crowding calculation to maintain the diversity and convergence of solutions, ultimately outputting a set of optimal trade-off solutions distributed on the Pareto front.

[0045] Optionally, given a 5TB storage budget, the optimization engine runs a fast non-dominated sorting genetic algorithm to optimize a set of 500 candidate triples. After 100 iterations, the algorithm outputs a Pareto optimal solution set containing 15 non-dominated solutions, each representing a materialized view construction scheme that achieves a different balance between query benefits and storage costs.

[0046] Step A204: Based on the preset business priority strategy, select the set of materialized views to be constructed from the Pareto optimal solution set.

[0047] It should be noted that, in this embodiment, the preset business priority strategy refers to business rules predefined based on the actual operational needs of the enterprise, used to make the final business decision among multiple technically feasible solutions obtained from multi-objective optimization. This combines the technical optimization results with actual business needs to make the final decision that best serves the enterprise's interests.

[0048] This embodiment quantitatively analyzes historical query patterns and, under strict storage space constraints, precisely balances query performance improvement with resource consumption, automatically outputting a set of materialized views that is optimal in both technical and business aspects. This ensures that limited storage resources have more efficient overall query performance. In other words, the decision to construct materialized views is transformed from subjective judgment based on human experience to an objective and scientific calculation process based on historical data and multi-objective optimization algorithms.

[0049] Furthermore, referring to Figure 3 The third embodiment of the indicator data query method of this application provides a flowchart, based on the above. Figure 3 The illustrated embodiment further refines the step of "determining whether all query dimensions associated with the query metrics have been materialized and stored in the materialized view of the fast zone," including steps A301 to A305: Step A301: Construct a dimension-storage location mapping table, wherein each query dimension in the mapping table corresponds to a unique dimension feature vector; It should be noted that, in this embodiment, the dimension-storage location mapping table records the location status of each query dimension in the data storage system; the query dimension refers to the basic angle that users can use to group and filter data, such as "time," "region," and "product category"; the dimension feature vector refers to the fixed-length, unique numerical representation converted from the text identifier of each dimension through a specific algorithm. By establishing an efficient and accurate query index, the dimension concept at the business level is mapped to a storage status identifier that the system can quickly identify.

[0050] Step A302: Convert the query dimensions associated with the query metrics into a corresponding set of dimension feature vectors; It should be noted that, in the embodiments of this application, the set of dimensional feature vectors represents all dimensional features involved in a specific query request.

[0051] Step A303: Using the set of dimensional feature vectors as keys, batch query a fast prediction component built based on a Bloom filter; It should be noted that in this embodiment, the Bloom filter is used to quickly determine whether an element is not in a set; the fast prediction component is specifically responsible for performing a preliminary, extremely fast existence determination on the dimension set. Utilizing the Bloom filter's extremely high query speed and extremely low memory usage, query requests that need to fall back to the normal area are quickly filtered out.

[0052] Step A304: If the fast prediction component returns that no feature vector exists in any dimension, the query result is determined to be negative, and the data reading process from the ordinary area is triggered. It should be noted that, in this embodiment of the application, "yes or no" specifically refers to the conclusion that the system determines that the set of dimensions associated with the current query metric is not completely covered by the materialized view of the fast zone, thereby optimizing the processing efficiency of negative query paths and improving the overall response speed of the system.

[0053] Step A305: If the feature vectors of the dimensions returned by the fast prediction component all exist, then continue to query the dimension-storage location mapping table to confirm for the second time that the query dimensions associated with the query indicators have been materialized and stored in the materialized view of the fast area.

[0054] This embodiment optimizes the processing efficiency of negative query paths by introducing a two-level judgment architecture of "fast prediction by Bloom filter + precise confirmation by dimension-storage location mapping table".

[0055] In one possible implementation, the update step of the fast region includes: Monitor new query logs and calculate the popularity trends of various metric-dimension combinations; The fast non-dominated sorting genetic algorithm is executed based on a preset time period to generate a new Pareto optimal solution set. The new Pareto optimal solution set is compared with the currently materialized view to generate a view update operation sequence, which includes a new materialized view to be constructed and an old materialized view to be discarded. The view update operation sequence is executed during periods of low system load.

[0056] It should be noted that, in this embodiment, the trend of popularity change refers to the direction and magnitude of the change in the access frequency of each indicator-dimensional combination calculated by statistical methods (such as the query frequency growth rate within a sliding time window); the preset time period refers to the execution period set in advance based on the frequency of business data changes and the system maintenance window, such as weekly or monthly; the view update operation sequence refers to a series of ordered operation instructions required to synchronize the current materialized view set to the new Pareto optimal solution set; through view update management, the system enables the data content of the fast zone to adapt to changes in business query patterns, and continuously maintains query performance optimization through periodic optimization calculations and careful scheduling execution.

[0057] In one possible implementation, the step of integrating data from multiple query metrics read from the fast zone and the normal zone, and generating a unified query result based on preset business logic, includes: Assign a unified query batch identifier to data result sets that are read from different data areas but belong to the same query indicator request; Based on the query batch identifier, the data result set is loaded into the distributed memory workspace; Within the distributed memory workspace, based on the display dimensions specified in the query metric request, data from different data areas are subjected to dimension alignment, association, and aggregation processing to generate intermediate results with unified dimensions. The intermediate results are calculated based on preset business logic to generate unified query results.

[0058] It should be noted that, in this embodiment, the query batch identifier refers to a globally unique identifier assigned by the system to each independent query indicator request, used to associate and track all data subsets serving that request; dimension alignment refers to the process of unifying datasets from different data sources, which may be at different dimensional granularities, to the same set of display dimensions specified by the user query through data completion, padding, or mapping; this embodiment solves the problem of heterogeneous data source fusion caused by mixed reading of data from the "fast zone" and the "normal zone", and can ultimately generate a unified data view that is consistent in dimensions, accurate in business, and can be directly used by the front end.

[0059] Optionally, the data integration engine receives a query request that simultaneously display "Sales Revenue" (read from the materialized view `mv_sales` in the fast zone) and "Profit Margin" (calculated from a full table scan of the `fact_orders` table in the normal zone). The engine first assigns a unique query batch identifier to this request. Then, it tags the pre-aggregated data from `mv_sales` and the real-time calculation results from `fact_orders` with this query batch identifier and loads them onto different nodes in the distributed in-memory workspace. In the in-memory workspace, the integration engine recognizes that the user-specified display dimensions are year and product category. The data in `mv_sales` is complete in both dimensions, while the original data in `fact_orders` is missing records for a certain product category in a particular year. Therefore, the system performs dimension alignment, padding the missing data rows with zeros. Next, the engine joins the two datasets based on the two keys, year and product category, and calculates the profit margin for each dimension combination according to preset business logic (e.g., profit margin = profit / sales revenue). Ultimately, the engine generates a unified query result set, in which each row contains four fields: year, product category, sales amount, and profit margin, and returns it to the front end for chart rendering.

[0060] In one possible implementation, the step of dimensional alignment of data from different data regions includes: For each data result set, construct a memory bitmap index on the stated display dimension; Based on the memory bitmap index, AND, OR, and NOT logical operations are performed to identify the intersection and difference relationships of different data result sets in the display dimension; Based on the intersection and difference relationships, zero-padding, filling, or association operations are performed on the data in different data areas to complete dimension alignment.

[0061] It should be noted that, in this embodiment, the display dimension refers to the set of dimensions explicitly specified by the user in this query request for grouping and displaying results; the memory bitmap index refers to a bit vector-based index structure quickly created for each data result set in a distributed memory workspace, where each bit represents the existence state of a unique combination of dimension key values ​​(1 indicates existence, 0 indicates non-existence), which has extremely high query and computation efficiency; the intersection and difference relationships are set relationships obtained through logical operations (such as AND, OR, NOT) of the bitmap index, used to accurately characterize the overlap and missing information of different datasets in common display dimensions; zero-padding, filling, or association operations are specific data processing actions taken for data missingness identified in the difference relationship: among them, zero-padding is usually used to fill missing dimensions for numerical indicators, and filling may involve default values ​​or interpolation. This embodiment quickly diagnoses the dimensional differences between datasets through efficient bitmap operations and performs precise data repair or connection, thereby generating a complete and consistent data foundation in the specified display dimensions, clearing structural obstacles for generating unified query results.

[0062] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the indicator data query method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0063] This application also provides an indicator data query device, please refer to... Figure 4 The indicator data query device includes: The request receiving module 10 is used to receive query indicator requests initiated by users and obtain the query dimensions associated with the query indicators. The dimension judgment module 20 is used to determine whether all query dimensions associated with the query indicators have been materialized and stored in the materialized view of the fast zone; The data reading module 30 is used to read the data of the query indicator from the materialized view of the fast zone if all the query dimensions associated with the query indicator have been materialized and stored in the materialized view of the fast zone; otherwise, it reads the data of the query indicator from the normal zone by a full table scan. The data integration module 40 is used to integrate data from multiple query indicators read from the fast zone and the normal zone, and generate a unified query result based on preset business logic.

[0064] The indicator data query device provided in this application, employing the indicator data query method in the above embodiments, can solve the technical problem of low overall query performance due to the inability to intelligently adapt to dynamically changing query needs when querying multiple datasets in combination. Compared with the prior art, the beneficial effects of the indicator data query device provided in this application are the same as those of the indicator data query method provided in the above embodiments, and other technical features in the indicator data query device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0065] This application provides an indicator data query device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the indicator data query method in the first embodiment described above.

[0066] The following is for reference. Figure 5 The diagram illustrates a structural schematic of a data query device suitable for implementing embodiments of this application. The data query device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The indicator data query device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0067] like Figure 5As shown, the indicator data query device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the indicator data query device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the index data query device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows index data query devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0068] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0069] The indicator data query device provided in this application, employing the indicator data query method in the above embodiments, can solve the technical problem of low overall query performance due to the inability to intelligently adapt to dynamically changing query needs when querying multiple datasets in combination. Compared with the prior art, the beneficial effects of the indicator data query device provided in this application are the same as those of the indicator data query method provided in the above embodiments, and other technical features of this indicator data query device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0070] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0071] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0072] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the index data query method in the above embodiments.

[0073] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0074] The aforementioned computer-readable storage medium may be included in the indicator data query device; or it may exist independently and not be assembled into the indicator data query device.

[0075] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the indicator data query device, the indicator data query device performs the following actions: receives a query indicator request initiated by a user and obtains the query dimensions associated with the query indicator; determines whether all query dimensions associated with the query indicator have been materialized and stored in the materialized view of the fast zone; if all query dimensions associated with the query indicator have been materialized and stored in the materialized view of the fast zone, it reads the data of the query indicator from the materialized view of the fast zone; otherwise, it reads the data of the query indicator from the normal zone using a full table scan; and integrates the data of multiple query indicators read from the fast zone and the normal zone to generate a unified query result based on preset business logic.

[0076] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0077] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0078] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0079] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described indicator data query method. This solves the technical problem of low overall query performance due to the inability to intelligently adapt to dynamically changing query requirements when querying multiple datasets in combination. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the indicator data query method provided in the above embodiments, and will not be elaborated upon here.

[0080] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the indicator data query method described above.

[0081] The computer program product provided in this application can solve the technical problem of low overall query performance due to the inability to intelligently adapt to dynamically changing query requirements when querying multiple datasets in combination. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the indicator data query method provided in the above embodiments, and will not be repeated here.

[0082] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for querying indicator data, characterized in that, The method for querying indicator data includes: Receive query metrics requests initiated by users and obtain the query dimensions associated with the query metrics; Determine whether all query dimensions associated with the query metrics have been materialized and stored in the materialized view of the fast zone; If all query dimensions associated with the query metric have been materialized and stored in the materialized view of the fast zone, then the data of the query metric is read from the materialized view of the fast zone; otherwise, the data of the query metric is read from the normal zone through a full table scan. Data from multiple query metrics read from the fast zone and the normal zone are integrated, and a unified query result is generated based on preset business logic.

2. The indicator data query method as described in claim 1, characterized in that, The fast zone is used to store data containing frequently used dimensions, and the normal zone is used to store source data tables containing all dimensions.

3. The indicator data query method as described in claim 2, characterized in that, The construction of the materialized view of the fast zone includes: Based on historical query logs, analyze and identify frequently used metrics and dimension combinations; Based on the combination of frequently used indicators and dimensions, the source data table of the ordinary area is pre-aggregated and calculated to generate a materialized view. The materialized view is stored in the fast zone.

4. The indicator data query method as described in claim 3, characterized in that, The steps for analyzing and identifying frequently used metrics and dimension combinations based on historical query logs include: Extract a set of triples from historical query logs, where the triples include indicators, dimension combinations, and query frequency; A multi-objective optimization function is constructed with the goal of maximizing total query revenue and minimizing total storage cost. A fast non-dominated sorting genetic algorithm is used to optimize the set of triples under a given total storage space budget constraint to obtain a Pareto optimal solution set. Based on a preset business priority strategy, a set of materialized views to be constructed is selected from the Pareto optimal solution set.

5. The indicator data query method as described in claim 4, characterized in that, The step of determining whether all query dimensions associated with the query metrics have been materialized and stored in the materialized view of the fast zone includes: Construct a dimension-storage location mapping table, in which each query dimension in the mapping table corresponds to a unique dimension feature vector; Convert the query dimensions associated with the query metrics into a corresponding set of dimensional feature vectors; Using the set of dimensional feature vectors as keys, a fast prediction component built on a Bloom filter is queried in batches. If the fast prediction component returns that no feature vector exists in any dimension, the query result is determined to be negative, and the data reading process from the ordinary area is triggered. If the feature vectors of the dimensions returned by the fast prediction component all exist, then continue to query the dimension-storage location mapping table to confirm for the second time that the query dimensions associated with the query indicators have been materialized and stored in the materialized view of the fast area.

6. The indicator data query method as described in claim 5, characterized in that, The update steps for the fast zone include: Monitor new query logs and calculate the popularity trends of various metric-dimension combinations; The fast non-dominated sorting genetic algorithm is executed based on a preset time period to generate a new Pareto optimal solution set. The new Pareto optimal solution set is compared with the currently materialized view to generate a view update operation sequence, which includes a new materialized view to be constructed and an old materialized view to be discarded. The view update operation sequence is executed during periods of low system load.

7. The indicator data query method as described in claim 6, characterized in that, The step of integrating data from multiple query metrics read from the fast zone and the normal zone, and generating a unified query result based on preset business logic, includes: Assign a unified query batch identifier to data result sets that are read from different data areas but belong to the same query indicator request; Based on the query batch identifier, the data result set is loaded into the distributed memory workspace; Within the distributed memory workspace, based on the display dimensions specified in the query metric request, data from different data areas are subjected to dimension alignment, association, and aggregation processing to generate intermediate results with unified dimensions. The intermediate results are calculated based on preset business logic to generate unified query results.

8. The indicator data query method as described in claim 7, characterized in that, The step of dimensional alignment of data from different data regions includes: For each data result set, construct a memory bitmap index on the stated display dimension; Based on the memory bitmap index, AND, OR, and NOT logical operations are performed to identify the intersection and difference relationships of different data result sets in the display dimension; Based on the intersection and difference relationships, zero-padding, filling, or association operations are performed on the data in different data areas to complete dimension alignment.

9. A device for querying indicator data, characterized in that, The indicator data query device includes: The request receiving module is used to receive query metric requests initiated by users and obtain the query dimensions associated with the query metrics. The dimension judgment module is used to determine whether all query dimensions associated with the query metrics have been materialized and stored in the materialized view of the fast zone; The data reading module is used to read the data of the query indicator from the materialized view of the fast zone if all the query dimensions associated with the query indicator have been materialized and stored in the materialized view of the fast zone; otherwise, it reads the data of the query indicator from the normal zone through a full table scan. The data integration module is used to integrate data from multiple query indicators read from the fast zone and the normal zone, and generate unified query results based on preset business logic.

10. A device for querying indicator data, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the indicator data query method as described in any one of claims 1 to 7.

Images