Data query method, visual data processing method and device thereof, electronic device and storage medium

Abstract

This application discloses a data query method, a visualization data processing method and apparatus, as well as an electronic device and a storage medium. The data query method includes: pre-configuring the update frequency of dimension table data; pre-configuring the association relationship between the fact table dataset and the dimension table dataset; pre-querying the dimension table data at the update frequency, and storing the pre-queried dimension table data using a predetermined data structure; querying the fact table data in response to the data query; and mapping the field values ​​in the queried fact table data to the associated field values ​​in the dimension table data based on the association relationship, to obtain the data query result.

Description

Technical Field

[0001] This application relates to the field of big data, and in particular to business intelligence (BI) technology, specifically to data query methods, visualization data processing methods and apparatus, electronic devices and storage media. Background Technology

[0002] Currently, Business Intelligence (BI) is widely used in the field of big data, involving tools and technologies for data collection, integration, analysis, and visualization. BI technologies are typically used to transform raw data collected from different data sources into understandable data or meaningful information, in order to support business decisions by analyzing the facts revealed by the data.

[0003] To facilitate multidimensional analysis using Business Intelligence (BI) tools, a data warehouse platform is typically built to extract business data. Within the data warehouse platform, star schemas or snowflake schemas are usually employed for data modeling and ETL (Extract, Transform, Load) based on the relationships between dimension tables and fact tables. This allows BI tools to generate Structured Query Language (SQL) for data queries and visualization during interactive multidimensional data analysis.

[0004] When using BI tools for such multidimensional data analysis, there are currently two main approaches. The first requires the BI tool to generate SQL queries to join and query the dimension and fact tables in real time. The second approach involves pre-joining the dimension and fact tables to generate a wide table for the BI tool to query.

[0005] However, real-time join queries between dimension and fact tables generated by BI tools cannot guarantee query efficiency, impacting user experience. If, to improve query efficiency, the dimension and fact tables are pre-joined in the underlying data to generate a wide table, and this data is redundantly stored in the database, this consumes additional storage space. Furthermore, when the dimension table information changes, the historical data in the wide table needs to be recalculated, making the wide table data generation process difficult to maintain. As data generation speed and data volume rapidly increase, these existing problems will become increasingly severe.

[0006] The background description is provided for the purpose of understanding the relevant technologies in this field and is not intended as an admission of prior art. Summary of the Invention

[0007] In this regard, the embodiments of the present invention aim to provide a solution that can effectively solve or alleviate any of the above-mentioned technical problems, thereby ensuring query efficiency when using business intelligence (BI) tools for interactive multidimensional analysis, avoiding large amounts of redundant data storage, and reducing data maintenance workload.

[0008] In this regard, a data query method is provided in the first aspect, which may include:

[0009] Pre-configured dimension table data update frequency;

[0010] The pre-configured relationship between fact table data and dimension table data;

[0011] Pre-query dimension table data at the aforementioned update frequency, and store the pre-queried dimension table data;

[0012] In response to the data query, query the fact table data in real time;

[0013] Based on the aforementioned relationship, the field values ​​in the queried fact table data are mapped to the associated field values ​​in the dimension table data to obtain the data query results.

[0014] Optionally, the dimension table data obtained from the pre-query is stored, including:

[0015] The dimension table data is cached using a predetermined hash table data structure.

[0016] Optionally, the predetermined hash table data structure includes a first hash table, a second hash table nested in the first hash table, and a third hash table nested in the second hash table, wherein the third hash table corresponds to the row index of the dimension table data, the second hash table corresponds to the column index of the dimension table data, and the first hash table corresponds to the table index of the dimension table data.

[0017] Optionally, based on the association, the field values ​​in the queried fact table data are mapped to the associated field values ​​in the dimension table data to obtain data query results, including:

[0018] Based on the aforementioned relationships, determine the fields that need to be mapped;

[0019] The query results of the fact table data are traversed, and the row index of the dimension table data is obtained by using the field name and field value of the field, and the column index is obtained by using the field name of the field;

[0020] Retrieve the list of dimension table field values ​​based on the row index;

[0021] Retrieve the dimension table field values ​​based on the column index;

[0022] Replace the field values ​​in the query results of the fact table data with the field values ​​of the obtained dimension table.

[0023] Optionally, the data query method may further include:

[0024] Dimension table data and fact table data are generated from online transaction processing (OLTP) source data through extract-transform-load (ETL) processing.

[0025] In a second aspect, a visualization data processing method is provided, which may include:

[0026] In response to user actions in business intelligence tools, determine the data dimensions to be analyzed and / or the computational functions used for data analysis;

[0027] Based on a defined data dimension and / or calculation function, the data query method according to any one of claims 1 to 6 is triggered to obtain data query results;

[0028] The data query results are displayed in a visual report format within the business intelligence tool for the user's multidimensional data analysis.

[0029] In a third aspect, a data query apparatus is provided, which may include:

[0030] The first pre-configuration unit is used to pre-configure the update frequency of dimension table data;

[0031] The second pre-configuration unit is used to pre-configure the association relationship between the fact table dataset and the dimension table dataset;

[0032] The preloading unit is used to pre-query dimension table data at the update frequency and store the pre-queried dimension table data.

[0033] The query unit is used to query fact table data in response to the data query.

[0034] The mapping unit is used to map the field values ​​in the queried fact table data to the associated field values ​​in the dimension table data based on the association relationship, so as to obtain the data query results.

[0035] In a fourth aspect, a visualization data processing apparatus is provided, which may include:

[0036] The determination unit is used to determine the data dimensions to be analyzed and / or the calculation functions used for data analysis in response to user actions in business intelligence tools.

[0037] A triggering unit is used to trigger a data query performed using the data query device of claim 8 based on a determined data dimension and / or calculation function, so as to obtain a data query result.

[0038] The visualization unit is used to display the data query results in a visual report format within the business intelligence tool, for the user's multidimensional data analysis.

[0039] In a fifth aspect, an electronic device is provided, which may include: a processor and a memory storing a computer program, the processor being configured to implement the method described in any embodiment of the present invention when running the computer program.

[0040] In a sixth aspect, a storage medium is provided that stores a computer program configured to be executed to implement the method described in any embodiment of the present invention.

[0041] The data query method in this embodiment of the invention proposes pre-configuring the update frequency of dimension table data; pre-configuring the association relationship between fact table data and the dimension table data; pre-querying dimension table data at the update frequency and storing the pre-queried dimension table data; responding to the data query and querying fact table data in real time; and mapping the field values ​​in the queried fact table data to the associated field values ​​in the dimension table data based on the association relationship to obtain the data query result. This ensures query efficiency without requiring pre-associating dimension tables and fact tables to generate a wide table; only the dimension tables and fact tables need to be generated. Furthermore, BI tools can perform multi-dimensional data analysis and association queries in real time based on the set association relationships, reducing redundant data storage space, decreasing the development workload for data producers, and ensuring data real-time performance.

[0042] Other optional features and technical effects of the embodiments of the present invention are partly described below and partly apparent from reading this document. Attached Figure Description

[0043] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The elements shown are not limited to the scale shown in the drawings, and the same or similar reference numerals in the drawings denote the same or similar elements, wherein:

[0044] Figure 1 A system architecture diagram for data querying in BI tools is shown;

[0045] Figure 2 Example diagrams of fact tables and dimension tables used in data warehousing are shown;

[0046] Figure 3 A connection block diagram is shown for implementing the data query method according to embodiments of this application;

[0047] Figure 4 An exemplary flowchart of a data query method according to an embodiment of this application is shown;

[0048] Figure 5 An exemplary flowchart of a data query method according to an embodiment of this application is shown;

[0049] Figure 6An exemplary flowchart of a visualization data processing method according to an embodiment of this application is shown;

[0050] Figure 7 A block diagram of a data query device according to an embodiment of the present invention is shown;

[0051] Figure 8 A block diagram of a visualization data processing apparatus according to an embodiment of the present invention is shown;

[0052] Figures 9A to 9E A schematic diagram illustrating the implementation of data querying using BI tools in a data warehouse system according to an embodiment of the present invention is shown;

[0053] Figure 10 A schematic architecture diagram of an electronic device capable of implementing the method according to embodiments of the present invention is shown. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0055] In this paper, Business Intelligence (BI) refers to the use of modern data warehousing technology, online analytical processing technology, data mining, and data visualization technology to perform data analysis in order to realize business value. In this embodiment of the invention, the BI tool preferably includes a front-end visualization tool that applies business intelligence to perform data analysis and mining. This front-end visualization tool can interface with and is built upon the back-end data warehouse.

[0056] In this article, a data warehouse (DW) refers to a data management system designed to support various business intelligence (BI) operations, particularly analysis. Typically, data warehouses are primarily used for performing queries and analyses, such as those used for online analytical processing (OLAP) and data mining. Data warehouses usually contain large amounts of historical data and have a wide range of data sources, such as transactional data obtained through online transaction processing (OLTP).

[0057] In this paper, Online Analytical Processing (OLAP) is a core component for high-speed, multidimensional analysis of large amounts of data from data warehouses, data marts, or other unified, centralized data stores. Current OLAP systems are generally based on data warehouses, extracting subsets of large datasets from the data warehouse and storing them in OLAP storage after necessary aggregation for front-end analytical tools, such as front-end BI tools, to read and analyze. It will be understood that the form and implementation of OLAP differ from Online Transaction Processing (OLTP), which is typically used to implement unified, implemented business processing activities. It should be noted that the data query described in this embodiment is based on OLAP and is unrelated to OLTP.

[0058] In this paper, Extract-Transform-Load (ETL) is a data pipeline used to collect data from different data sources. In embodiments of this invention, ETL components / tools can collect, extract, transform, and store transactional data, such as OLTP data, in a data warehouse.

[0059] In this article, a fact table, also known as a fact data table, is a type of table used in a data warehouse, along with dimension tables, to represent and store data. Fact tables typically contain relevant attributes of business events, also known as measures, and attribute values ​​are usually numerical, continuous fields. A key characteristic of fact tables is that they can hold large amounts of summative data. For example, Figure 2 An exemplary fact table is shown, such as Loan Amount Fact Table 210.

[0060] In this article, dimension tables are another type of table used in data warehouses, alongside fact tables, to represent the stored data. Dimension tables typically contain dimension codes and multiple attributes under that dimension, thus representing the structure of the data organization from a dimensional perspective. For example, Figure 2 The document shows several exemplary dimension tables, such as user dimension table 220, time dimension table 230, loan product dimension table 240, and geographic dimension table 250.

[0061] The wide table mentioned in the background section refers to a table that pre-associates and generates dimension tables and fact tables, and the association between the dimension tables and fact tables, as well as the generation of the wide table, are implemented and stored at the physical level. The data query method in this embodiment avoids the pre-generation of such a wide table.

[0062] In big data applications, data typically flows into a data warehouse from various data sources, where it is ultimately processed and stored for use by various front-end applications, such as business intelligence (BI) tools. Therefore, data in the data warehouse can be used by BI tools for interactive, multi-dimensional analysis to discover optimal business outcomes or support business decisions.

[0063] Figure 1A system architecture diagram for data querying in BI tools is shown. A data warehouse is a central repository for integrated data from one or more different data sources, used to store historical data in one place. For example... Figure 1 As shown, the system can handle multiple OLTP databases 102a, 102b, and 102c. As previously mentioned, Online Transaction Processing (OLTP) can be used, for example, to process online transactions of various business systems. Therefore, the OLTP process may have a large number of short-lived online transactions and may include a large number of real-time business query processing. It will be understood that this OLTP-based business query processing differs from data querying according to embodiments of the present invention. Accordingly, Figure 1 The business data (not shown) in the OLTP databases 102a, 102b, and 102c shown can be accessed, updated, or rewritten by business systems. These OLTP databases 102a, 102b, and 102c are, for example, relational databases.

[0064] Furthermore, business data in OLTP databases 102a, 102b, and 102c can also undergo Extract-Transform-Load (ETL) processing. For example... Figure 1As shown, OLTP databases 102a, 102b, and 102c can interface with ETL component 104, thereby allowing business data (source data) in OLTP databases 102a, 102b, and 102c to be processed by ETL component 104. Such business data (source data) can be, for example, structured and / or unstructured data, which can be processed using extract-transform-load (ETL) component 104. ETL component 104 can first extract data from structured or unstructured data sources; then clean, transform, and store the data in data warehouse 106 or data marts 108a, 108b, and 108c within the data warehouse. In various embodiments, ETL component 104 can extract data from source systems, such as various business databases, and convert data from different source systems into a unified data warehouse format. In various embodiments, ETL component 104 transforms received data, for example, by applying business rules (e.g., calculating new metrics and dimensions), cleaning (e.g., mapping), filtering (e.g., selecting only certain columns to load), splitting (e.g., splitting columns into multiple columns), merging data from multiple sources (e.g., lookup, merge), transposing rows and columns, and validating. In various embodiments, the ETL system component loads the data into a data warehouse. For example, ETL component 104 may load data into data warehouse 106 or its data marts 108a, 108b, 108c for use by upper-layer services, such as BI tool 110 based on Online Analytical Processing (OLAP). Optionally, the ETL component may be a distributed system that distributes the load across many (sub)components performing ETL. In optional embodiments, various detection mechanisms, such as a change detection component (not shown), may also be included so that ETL processing is performed by ETL component 104 when data in the OLTP database is updated and / or changed.

[0065] This allows the ETL component 104 to generate fact table data and dimension table data that can be stored in data warehouse 106, such as data marts 108a, 108b, and 108c within data warehouse 106. In some embodiments, fact table data and dimension table data for a single topic can be stored in a single data mart. In some embodiments, a single data mart can store a fact table and its associated dimension table. Figure 2 An example diagram of fact tables and dimension tables used in a data warehouse is shown. Figure 2 The diagram illustrates a fact table 210, which contains IDs and business attribute values ​​for various dimension tables, in this case, "Credit Amount". Figure 2 The illustrated embodiment also shows several illustrative dimension tables, such as user dimension table 220, time dimension table 230, (loan) product dimension table 240, and geographic dimension table 250. These dimension tables can be generated using various models, such as... Figure 2 The star schema shown is linked to the fact table and stored in a data warehouse, such as a data mart. In some embodiments, other models, such as snowflake schemas or constellation schemas, can also be used to link the fact table and dimension tables.

[0066] In addition, such as Figure 1 Furthermore, an OLAP component 112 may also be provided. In the illustrated embodiment, the OLAP component is integrated into the BI tool 110, allowing users to perform interactive multidimensional query operations using the BI tool. In an alternative embodiment, the OLAP component may also be integrated into the data warehouse.

[0067] In some embodiments of the present invention, the data warehouse can be built on Hadoop, such as a data warehouse implemented with Apache Hive, in which HDFS can be used for data storage. ETL components can be integrated internally within the Hive data warehouse, or external ETL tools can be used. Hive can map structured data files to database tables, provide SQL-based query methods, and translate SQL statements into tasks for different computing engines for analysis and querying, so as to be used for Online Analytical Processing (OLAP), such as for Business Insights (BI) tools, thereby facilitating users to perform relevant analysis and decision-making.

[0068] When using BI tools for such multidimensional data analysis, users typically use the BI tools to generate SQL queries to join dimension and fact tables in real time and perform data queries. Alternatively, BI tool maintainers can pre-join dimension and fact tables to generate wide tables, allowing users to quickly query data using the BI tools later.

[0069] However, the above solutions suffer from a difficulty in balancing real-time performance and cost-effectiveness. Real-time join queries between dimension and fact tables generated by BI tools cannot guarantee query efficiency, impacting user experience. If, to improve query efficiency, the dimension and fact tables are pre-joined in the underlying data to generate a wide table, and this data is redundantly stored in the database, it consumes additional storage space. Furthermore, when the dimension table information changes, the historical data in the wide table needs to be recalculated, making the wide table data generation process difficult to maintain. As data generation speed and data volume rapidly increase, these existing problems will become increasingly severe.

[0070] Therefore, embodiments of the present invention provide a data query method, a visualization data processing method and apparatus, as well as related electronic devices and storage media, which can be applied to interactive multidimensional analysis using BI tools. For example... Figure 3As shown, the data query method of this embodiment of the invention can pre-configure report datasets, pre-set dimension table datasets and fact table datasets 310; pre-configure data update frequency 320 so that BI tools can pre-load and cache dimension table data 330 according to the update frequency; and also pre-configure the association relationship 340 between the fact table dataset and the dimension table dataset; thus, when the user uses the BI tool to perform multidimensional analysis queries 350, by querying the fact table data 360, according to the pre-configured association relationship, the fields in the fact table data are mapped to the associated fields of the dimension table 370, thereby realizing multidimensional interactive queries 380.

[0071] By utilizing the embodiments of this invention, while ensuring query efficiency, there is no need to develop wide tables in advance, and historical data does not need to be re-run when dimension table data is updated, thus improving data output efficiency and reducing data development workload. Conversely, by pre-loading dimension tables in a cached manner, this invention avoids real-time database join queries and redundant data storage. Furthermore, the cached dimension table data structure facilitates rapid table data joins. In addition, this invention fully considers changes in business dimension table data, preventing these changes from affecting the accuracy of report data. Furthermore, report configuration is more convenient and flexible; only the join relationships need to be set in the dataset, simplifying report configuration.

[0072] Furthermore, this document also discloses several embodiments, which, while possessing the inventive spirit of the present invention, can have different implementation forms and include different features.

[0073] For example Figure 4 A data query method according to an embodiment of the present invention is shown, the data query method may include:

[0074] S410: Pre-configured dimension table data update frequency.

[0075] Optionally, prior to step S410, the report set and, consequently the dataset, may be pre-configured.

[0076] In some embodiments, a report set may also be referred to as a report dataset. It may include a chart set.

[0077] In some embodiments, the pre-configured dataset includes selecting a given data mart in order to configure the report set. In some embodiments, the data query method according to the invention can also be a method implemented on different data marts.

[0078] The inventors realized that the update and change rate of dimension table data is often slower than that of fact table data, and far slower than that of highly real-time business data. However, the attributes of dimensions are not static and still change slowly over time.

[0079] In this embodiment of the invention, the update frequency can be pre-configured based on the update and change frequency of the dimension tables in the data warehouse and data mart.

[0080] In some embodiments, a first update frequency can be set for the data in the first dimension table, and a second update frequency can be set for the data in the second dimension table.

[0081] S420: Pre-configure the relationship between fact table data and dimension table data.

[0082] In some embodiments, the association may also be cached in cache database 114 as described below or in another database.

[0083] S430: Pre-query dimension table data at update frequency and store the pre-queried dimension table data.

[0084] In some embodiments, storing the dimension table data obtained from the pre-query is equivalent to caching the dimension table data in the BI tool. For example... Figure 1 As shown, the BI tool can integrate a cache database 114, such as a cache database that uses a hash map data structure for caching. In another embodiment of the invention, the cache database can be constructed in the form of EhCache or Redis. Those skilled in the art will understand that the cache database 114 is independent of the data warehouse 106 or its data marts 108a, 108b, 108c, thereby allowing the BI tool to access the dimension table data of the cache database in near real-time when the user performs interactive queries.

[0085] In some embodiments, storing the dimension table data obtained from the pre-query may include caching the dimension table data using a predetermined hash map data structure.

[0086] In some embodiments, the predetermined hash table data structure includes a first hash table, a second hash table nested within the first hash table, and a third hash table nested within the second hash table. The third hash table corresponds to the row index of the dimension table data, the second hash table corresponds to the column index of the dimension table data, and the first hash table corresponds to the table index of the dimension table data.

[0087] In this embodiment, the data structure is a Map. <String,Map<String,Map<String,Object> >>

[0088] In a further embodiment, the specific content of the HashMap is represented using JSON as {"dimension table dataset name": {"index":{"field name_field value":"row index"},"meta":{"field name":"column index"},"data":{"row index":["field value"]}}}. In this embodiment, the HashMap is a "linked list hash" data structure, that is, a combination of array and linked list. The underlying structure of HashMap is an array structure, and each item in the array is a linked list. When a new HashMap is created, an array is initialized. The specific data structure of HashMap will not be described in detail here.

[0089] S440: Responds to data queries and queries fact table data in real time.

[0090] S450: Based on the relationship, the field values ​​in the queried fact table data are mapped to the related field values ​​in the dimension table data to obtain the data query results.

[0091] In some embodiments, the data query may be an OLAP data query, more preferably a data cube-based data query. Figures 9A to 9E A schematic diagram illustrating the implementation of data querying using BI tools in a data warehouse system according to an embodiment of the present invention is shown. Figures 9A to 9E As shown, when using BI tools for interactive multidimensional data analysis queries based on OLAP, operations such as slicing data cube 900 to 910, 920, 930, and 940; drilling down to 950; dicing; rotating; and rolling up were implemented. In response to this query operation, fields in the fact table are mapped to corresponding fields in the dimension table based on the association relationships.

[0092] In a further embodiment, such as Figure 5 As shown, based on this relationship, the field values ​​in the queried fact table data are mapped to the related field values ​​in the dimension table data to obtain the data query results, which may include:

[0093] S451: Based on the association relationship, determine the fields that need to be mapped;

[0094] S452: Traverse the query results of the fact table data, use the field name and field value to get the row index of the dimension table data, and use the field name to get the column index;

[0095] S453: Retrieve the list of dimension table field values ​​based on row index;

[0096] S454: Retrieve dimension table field values ​​based on column index;

[0097] S455: Replace the field values ​​in the query results of the fact table data with the obtained dimension table field values.

[0098] In one specific embodiment, the mapping process of querying the fact table and mapping field values ​​in the fact table data to field values ​​in the dimension table through the configured association relationships is as follows: Obtain the fields to be mapped based on the configured dimension table association relationships; traverse the query results, using field name + field value to obtain the row index of the dimension table cached data, and using the field name to obtain the column index; obtain the list of field values ​​based on the row index, and obtain the dimension table field values ​​based on the column index; replace the query result field values ​​with the dimension table field values. This index mapping achieves the following effects: on the one hand, the metrics of the fact table, which may change frequently, can be accurately obtained after the query; on the other hand, the field values ​​in the dimension table can be obtained through fast mapping.

[0099] In embodiments of the present invention, such as Figure 6 As shown, a data visualization processing method is also provided, which may include:

[0100] S610: In response to user actions in business intelligence tools, determine the data dimensions to be analyzed and / or the calculation functions used for data analysis;

[0101] In some embodiments, determining the data dimensions to be analyzed and / or the computational functions used for data analysis includes the operations on the data cube described above.

[0102] S620: Based on a defined data dimension and / or calculation function, trigger a data query method to obtain data query results;

[0103] In some embodiments, the data query method is the data query method according to the embodiments of the present invention.

[0104] S630: Displays data query results in a visual report format within business intelligence tools for users' multidimensional data analysis.

[0105] In some embodiments, the visualization report display may include a chart display, which is implemented based on the query results.

[0106] In this embodiment of the invention, the data query method and the data visualization processing method can be applied to various BI tools, including but not limited to modifications of existing BI tools and novel BI tools that implement the methods or functions described above in this embodiment of the invention. The type of existing BI tools is not limited; for example, they can be commercially available BI tools, such as the BI tools or modules integrated into the fintech platform provided by the applicant.

[0107] In embodiments of the present invention, such as Figure 7As shown, a data query device 700 is also provided. This data query device 700 may include a first pre-configuration unit 710 for pre-configuring the update frequency of dimension table data; a second pre-configuration unit 720 for pre-configuring the association relationship between the fact table dataset and the dimension table dataset; a pre-loading unit 730 for pre-querying dimension table data at the update frequency and storing the pre-queried dimension table data; a query unit 740 for querying fact table data in response to a data query; and a mapping unit 750 for mapping field values ​​in the queried fact table data to associated field values ​​in the dimension table data based on the association relationship, to obtain the data query result.

[0108] Optionally, not shown in the figure, the mapping unit 750 may include a determining subunit configured to determine the fields to be mapped based on the association relationship; a traversal subunit configured to traverse the query results of the fact table data, obtain the row index of the dimension table data using the field name and field value, and obtain the column index using the field name; a row index obtaining subunit configured to obtain the list of dimension table field values ​​based on the row index; a column index obtaining subunit configured to obtain the dimension table field values ​​based on the column index; and a replacement subunit configured to replace the field values ​​in the query results of the fact table data with the obtained dimension table field values.

[0109] In embodiments of the present invention, such as Figure 8 As shown, a visualization data processing device 800 is also provided, which may include a determining unit 810, used to determine the data dimension to be analyzed and / or the calculation function used for data analysis in response to a user's operation in a business intelligence tool; a triggering unit 820, used to trigger a data query performed by the data query device described in this embodiment of the invention based on the determined data dimension and / or calculation function, so as to obtain data query results; and a visualization unit 830, used to display the data query results in a visualization report format in the business intelligence tool for the user's multidimensional data analysis.

[0110] One or more features of the system or apparatus in the embodiments of the present invention can be applied to the method to implement new embodiments, and vice versa.

[0111] In an embodiment of the present invention, an electronic device is provided, comprising: a processor and a memory storing a computer program, the processor being configured to execute the method of any embodiment of the present invention when running the computer program.

[0112] Figure 10 A schematic diagram is shown of a method or electronic device 1000 that can implement embodiments of the present invention. In some embodiments, the system according to embodiments of the present invention may include a plurality of electronic devices as shown in the figure, for example, implemented by a plurality of distributed electronic devices.

[0113] like Figure 10 As shown, the electronic device 1000 includes a processor 1001, which can perform various appropriate operations and processes based on programs and / or data stored in read-only memory (ROM) 1002 or programs and / or data loaded from storage portion 1008 into random access memory (RAM) 1003. The processor 1001 may be a multi-core processor or may contain multiple processors. In some embodiments, the processor 1001 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. Various programs and data required for the operation of the electronic device 1000 are also stored in RAM 1003. The processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. An input / output (I / O) interface 1005 is also connected to bus 1004.

[0114] The processor and memory described above are used together to execute the program stored in the memory. When the program is executed by the computer, it can implement the steps or functions of the model generation method and recognition method described in the above embodiments.

[0115] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1010 as needed so that computer programs read from it can be installed into storage section 1008 as needed. Figure 10 The diagram only shows a portion of the components and does not imply that the computer system 1000 only includes... Figure 10 The components shown.

[0116] Furthermore, the systems, devices, modules, or units described in the embodiments of this invention can be implemented by a computer or its associated components, such as... Figure 10 The electronic device shown. For example, the data query device and / or visualization data processing device or its components described in the embodiments of the present invention may include multiple such... Figure 10 The computer shown. The data query device and / or visualization data processing device or its components according to embodiments of the present invention may have, for example... Figure 10 The computer architecture or other computer architecture and / or type shown.

[0117] In an embodiment of the present invention, a storage medium is provided, the storage medium storing a computer program configured to execute the method of any embodiment of the present invention when run.

[0118] Storage media in embodiments of the present invention include articles that are permanent and non-permanent, removable and non-removable, capable of storing information by any method or technology. Examples of storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0119] The methods, programs, systems, apparatuses, etc., in the embodiments of the present invention can be executed or implemented in one or more networked computers, or practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks can be performed by remote processing devices connected via a communication network.

[0120] Those skilled in the art will understand that the embodiments described in this specification can be provided as methods, systems, or computer program products. Therefore, those skilled in the art will realize that the functional modules / units or controllers and related method steps described in the above embodiments can be implemented in software, hardware, or a combination of both.

[0121] Unless explicitly stated otherwise, the actions or steps of the methods and procedures described in the embodiments of the present invention do not necessarily have to be performed in a specific order and can still achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0122] This document describes several embodiments of the present invention; however, for the sake of brevity, the descriptions of the embodiments are not exhaustive, and identical or similar features or parts between the embodiments may be omitted. In this document, "one embodiment," "some embodiments," "example," "specific example," or "some examples" refers to embodiments applicable to at least one, but not all, of the present invention. The above terms do not necessarily refer to the same embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of the different embodiments or examples.

[0123] The exemplary systems and methods of the present invention have been specifically shown and described with reference to the above embodiments, which are merely examples of the best mode for implementing the systems and methods. Those skilled in the art will understand that various changes can be made to the embodiments of the systems and methods described herein without departing from the spirit and scope of the invention as defined in the appended claims when implementing the systems and / or methods.

Claims

1. A data query method, characterized by, The method comprises: pre-configuring a dimension table data update frequency; pre-configuring an association relationship between fact table data and the dimension table data; pre-querying dimension table data at the update frequency, and storing the pre-querying dimension table data; in response to the data query, querying fact table data in real time; based on the association relationship, mapping a field value in the queried fact table data to an associated field value in the dimension table data to obtain a data query result.

2. The data query method of claim 1, wherein, Storing the pre-querying dimension table data comprises: caching the dimension table data in a predetermined hash table data structure.

3. The data query method of claim 2, wherein, The predetermined hash table data structure comprises a first hash table, a second hash table nested in the first hash table, and a third hash table nested in the second hash table, wherein the third hash table corresponds to a row index of the dimension table data, the second hash table corresponds to a column index of the dimension table data, and the first hash table corresponds to a table index of the dimension table data.

4. The data query method of any one of claims 1 to 3, wherein, Based on the association relationship, mapping a field value in the queried fact table data to an associated field value in the dimension table data to obtain a data query result comprises: based on the association relationship, determining a field that needs to be mapped; traversing the query result of the fact table data, using the field name and field value of the field to obtain the row index of the dimension table data, and using the field name of the field to obtain the column index; obtaining a dimension table field value list according to the row index; obtaining a dimension table field value according to the column index; replacing the field value in the query result of the fact table data with the obtained dimension table field value.

5. The data query method of any one of claims 1 to 3, wherein, Further comprising: generating dimension table data and fact table data from online transaction processing (OLTP) source data through extract-transform-load (ETL) processing.

6. The data query method of claim 4, wherein, Further comprising: generating dimension table data and fact table data from online transaction processing (OLTP) source data through extract-transform-load (ETL) processing.

7. A method of visualizing data processing, characterized by, The method comprises: in response to a user's operation in a business intelligence tool, determining a data dimension to be analyzed and / or a calculation function for data analysis; based on the determined data dimension and / or calculation function, triggering the data query method according to any one of claims 1 to 6 to obtain a data query result; displaying the data query result in a report mode in the business intelligence tool for multi-dimensional data analysis by the user.

8. A data query apparatus, characterized by comprising: The method comprises: a first pre-configuration unit for pre-configuring a dimension table data update frequency; a second pre-configuration unit for pre-configuring an association relationship between a fact table data set and the dimension table data set; a pre-loading unit for pre-querying dimension table data at the update frequency, and storing the pre-querying dimension table data; a query unit for querying fact table data in real time in response to the data query; a mapping unit for mapping a field value in the queried fact table data to an associated field value in the dimension table data based on the association relationship to obtain a data query result.

9. A visualizing data processing apparatus, characterized by The method comprises: a determination unit for determining a data dimension to be analyzed and / or a calculation function for data analysis in response to a user's operation in a business intelligence tool; triggering a data query by the data query apparatus of claim 8 based on the determined data dimension and / or the calculation function to obtain a data query result; visualizing the data query result in the business intelligence tool in a report manner for multidimensional data analysis of the user.

10. An electronic device, comprising: comprising: a processor and a memory having stored thereon a computer program, the processor being configured to implement the method of any one of claims 1 to 7 when the computer program is run.

11. A storage medium, characterized by the storage medium having stored thereon a computer program configured such that, when the computer program is run by a processor, the method of any one of claims 1 to 7 is implemented.

Images