A method and device for unified storage of indicator result data
By designing a flexible data table, including general dimension columns, indicator columns and indicator unique dimension columns, and supporting capacity expansion, it solves the management difficulties and complex query problems of indicator data storage solutions in the existing technology, and achieves efficient indicator result data Storage and query reduce operation and maintenance costs.
Patent Information
- Application Number
- CN202310413526.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-17
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2043-04-17
AI Technical Summary
In the prior art, the index data storage solution has problems such as many data tables, difficulty in management, redundant data, inconsistent data, high resource consumption, and inability to directly query and complex number fetching.
Design an elastic data table, including general dimension columns, indicator columns and metric columns unique to indicators, supports horizontal and vertical expansion, and realizes unified storage and efficient query of indicator result data by dynamically adding data columns and segmenting data files.
It solves problems such as many data tables, management difficulties, data redundancy, and inability to directly query, and realizes efficient storage and query of indicator result data, reducing operation and maintenance costs.
Smart Images

Figure CN116401254B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of indicator result data storage, and in particular to a method and device for unified storage of indicator result data. Background Art
[0002] In the current fierce market competition environment, enterprises continue to strengthen internal refined management, gradually build a digital management system with indicators as the core, keep a close eye on the core indicator data of enterprise operations, and build and continuously improve the comprehensive and unified indicator system of enterprises. At the same time, the problems of enterprise indicator management continue to expand, especially in the big data environment. The storage and query of massive indicator data poses a huge challenge to the data team and the technical team.
[0003] In order not to affect the normal operation of existing business systems, enterprises usually build subject-level or enterprise-level data warehouses based on relational databases or big data platforms, extract data from various business systems into the data warehouse, and perform hierarchical calculations and storage. When data analysis of business application scenarios is proposed, the business analysis team will sort out the indicators and analysis dimensions related to data analysis, and the data development team will then perform data modeling, write data ETL scripts, and configure scheduled tasks for data calculations, and finally store the indicator data that meets business needs in the corresponding data tables.
[0004] While meeting the current data needs of the business, the indicator data storage solution must not only meet the performance requirements of data queries, but also quickly respond to changing business needs without affecting existing businesses. Faced with complex business data needs, the data development team usually uses the following methods to store indicator data.
[0005] The first technical solution is to store indicator data in different data tables according to themes and analysis scenarios. The dealer analysis-summary data table (daily granularity) designed as shown in Table 1 below contains three indicators: the number of creations, the number of targets, and the number of qualified people. These three indicators include five analysis dimensions in this analysis scenario: month, province, region, dealer, and activity. This data table will collect statistics on a daily basis.
[0006] Table 1: Dealer analysis - summary data table (daily granularity) example
[0007]
[0008] The advantage of this technical solution is that it is easy to understand, can be adapted to different databases and big data platforms, and can directly meet the data retrieval requirements of upper-level businesses. This is also a common solution for traditional data warehouses, and there are mature modeling technologies and modeling specifications to match it. However, with the rapid changes in business in the digital age, the problems of this technical solution have also become apparent. First, in the digital age, the demand for data analysis has increased rapidly, resulting in an increasing number of data tables in the data warehouse, which makes management difficult. Secondly, the rapid changes in business have high maintenance costs for data tables and are likely to affect other businesses. Redesigning data tables in order not to affect existing businesses is likely to lead to data redundancy and data inconsistency. Finally, more and more data tables have increased the demand for storage and computing resources, resulting in a rapid increase in the operation and maintenance costs of enterprises.
[0009] The second technical solution is to design a general indicator result data storage solution based on Gaoban. This technical solution includes an indicator result data table and some configuration information tables. In addition to fixed dimensions such as date, province, region, etc., the indicator result data table also reserves N dimensions such as DIM1, DIM2...DIMN. Different indicators may correspond to different dimensions and numbers of dimensions. In this indicator result table, the same reserved dimension field (such as DIM1) may correspond to different dimensions of different indicators. For example, in the data record of the indicator of the number of creations, the DIM1 dimension corresponds to the activity dimension, while in the data record of the sales indicator, the DIM1 dimension corresponds to the store. Since the indicator values are all numeric types, the values of all indicators can be stored in the same column, such as the indicator value column defined in the table.
[0010] Table 2: Example of indicator result data table
[0011]
[0012] In addition to the indicator result data table, you also need to design the corresponding dimension information table, indicator dimension relationship table, etc. The dimension information table contains information such as dimension type, dimension name, dimension English name, dimension description, dimension classification, dimension status, etc. The indicator dimension relationship table is used to associate indicators and dimensions and to define the dimension columns in the indicator result data table in detail.
[0013] Table 3: Example of dimension information table
[0014]
[0015]
[0016] Table 4: Example of indicator dimension relationship table
[0017]
[0018] The advantage of this high-level table-based technical solution is that it can uniformly store all indicator result data. As the business changes, new indicators can be continuously added without the need to design indicator result data storage tables multiple times. Each indicator can be calculated independently to ensure the consistency of indicator caliber and indicator results. This technical solution can also be implemented using traditional relational databases or big data platforms. However, the problem is also obvious. The indicator result data cannot be directly queried, and data acquisition is complicated. The upper-level business applications and BI reporting tools cannot directly read the data. When the data needs to be used, it is necessary to extract the required data from this set of indicator result data storage solutions based on the business's demand for indicator data and create data that meets the needs.
[0019] The first technical solution has problems such as multiple data tables, difficult management, data redundancy, data inconsistency, and high resource consumption; the second technical solution cannot directly query data, data acquisition is complicated, and multiple data extractions are required. Therefore, the inventor believes that it is necessary to propose a method for unified storage of indicator result data. Summary of the invention
[0020] The embodiments of the present application provide a method and device for unified storage of indicator result data, which are used to solve technical problems in the prior art such as multiple data tables, difficult management, data redundancy, inability to directly query data, and complex data acquisition.
[0021] The present application embodiment provides a method for uniformly storing indicator result data, generating an elastic data table, which includes:
[0022] Generate common dimension columns, indicator columns, and dimension columns unique to indicators;
[0023] The common dimension refers to the dimension that all indicators have. The column name and data type of the common dimension column are fixed and will not change with the indicator.
[0024] The indicator column refers to the column that stores the indicator result data. Each indicator corresponds to a data column, which is used to store the indicator result data;
[0025] The dimension columns unique to the indicators refer to the unique dimensions that each indicator has in addition to the common dimensions. The dimension columns unique to the indicators have their own corresponding data columns in the table. The dimension columns unique to the indicators store the unique data in the indicator result data.
[0026] The elastic data table is stored in a storage medium.
[0027] Furthermore, the elastic data table supports horizontal expansion and vertical expansion. The horizontal expansion refers to dynamically adding data columns of the elastic data table when writing data in the indicator column. The vertical expansion refers to the elastic data table dynamically dividing all indicator data into different data files, automatically performing file division when writing indicator data, and writing different indicator data into different data files.
[0028] Furthermore, all indicator result data are uniformly stored in the elastic data table, indicator result data with the same dimensions in the indicator result data are stored in the same data column of the elastic data table, and indicator result data with different dimensions in the indicator result data are stored in independent data columns.
[0029] Furthermore, the method for uniformly storing indicator result data is used to store enterprise operation and management data.
[0030] Furthermore, the storage medium is a volatile storage medium or a non-volatile storage medium.
[0031] The embodiment of the present application also provides a device for uniformly storing indicator result data, including:
[0032] The data table generation module generates a flexible data table, including:
[0033] Generate common dimension columns, indicator columns, and dimension columns unique to indicators;
[0034] The common dimension refers to the dimension that all indicators have. The column name and data type of the common dimension column are fixed and will not change with the indicator.
[0035] The indicator column refers to the column that stores the indicator result data. Each indicator corresponds to a data column, which is used to store the indicator result data;
[0036] The dimension columns unique to the indicators refer to the unique dimensions that each indicator has in addition to the common dimensions. The dimension columns unique to the indicators have their own corresponding data columns in the table. The dimension columns unique to the indicators store the unique data in the indicator result data.
[0037] The storage stores the elastic data table in a storage medium.
[0038] Furthermore, the elastic data table supports horizontal expansion and vertical expansion. The horizontal expansion refers to dynamically adding data columns of the elastic data table when writing data in the indicator column. The vertical expansion refers to the elastic data table dynamically dividing all indicator data into different data files, automatically performing file division when writing indicator data, and writing different indicator data into different data files.
[0039] Furthermore, the storage medium is a volatile storage medium or a non-volatile storage medium.
[0040] An embodiment of the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps of the aforementioned method when executing the computer program.
[0041] The embodiment of the present application also provides a computer-readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to execute the aforementioned method.
[0042] The embodiments provided in this application have at least the following beneficial effects:
[0043] The present invention provides a method for unified storage of indicator result data, generating an elastic data table, which includes: generating a general dimension column, an indicator column and an indicator-specific dimension column; wherein the general dimension refers to a dimension that all indicators have, and the column name and data type of the general dimension column are fixed and will not change with the indicator; the indicator column refers to a column that stores indicator result data, and each indicator corresponds to a data column for storing indicator result data; the indicator-specific dimension column refers to a unique dimension that each indicator has in addition to the general dimension, and the indicator-specific dimension column has a corresponding data column; the indicator-specific dimension column stores unique data in the indicator result data; and the elastic data table is stored in a storage medium. The technical solution of the present application can solve the technical problems in the prior art such as multiple data tables, difficult management, data redundancy, inability to directly query data, and complex data acquisition. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0045] Figure 1 A schematic diagram of a method flow for unified storage of indicator result data provided in an embodiment of the present application;
[0046] Figure 2 A schematic diagram of a query module for uniformly storing indicator result data provided in an embodiment of the present application;
[0047] Figure 3 A query flow chart for unified storage of indicator result data provided in an embodiment of the present application. DETAILED DESCRIPTION
[0048] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0049] Please refer to Figure 1 , the present application provides a method for uniformly storing indicator result data, generating an elastic data table, which includes:
[0050] Generate common dimension columns, indicator columns, and dimension columns unique to indicators;
[0051] The common dimension refers to the dimension that all indicators have. The column name and data type of the common dimension column are fixed and will not change with the indicator.
[0052] The indicator column refers to the column that stores the indicator result data. Each indicator corresponds to a data column, which is used to store the indicator result data;
[0053] The dimension columns unique to the indicators refer to the unique dimensions that each indicator has in addition to the common dimensions. The dimension columns unique to the indicators have their own corresponding data columns in the table. The dimension columns unique to the indicators store the unique data in the indicator result data.
[0054] The elastic data table is stored in a storage medium.
[0055] In order to overcome the deficiencies of the prior art, the present invention proposes a method for uniformly storing indicator result data, and provides a distributed data storage and query method, device and system.
[0056] The method of uniformly storing indicator result data is to design a flexible data table, which can uniformly store the result data of all indicators in the flexible data table. The same dimensions of these indicators are stored in the same data column of the flexible data table, while the different dimensions between indicators are stored in their own independent data columns. Different columns correspond to different dimensions of different indicators. The results of the flexible data table are shown in Table 5 below:
[0057] Table 5: Example of elastic data table structure
[0058]
[0059] Table 5 consists of three types of columns: general dimension columns, indicator columns, and indicator-specific dimension columns. General dimension columns refer to dimensions that all indicators have, such as time, indicator name, indicator code, indicator unit, indicator version, etc. The column names and data types of these columns are fixed and will not change with different indicators. Indicator columns refer to columns that store indicator values. Each indicator has a specific data column for storing the indicator value, and the column data type is associated with the indicator. Indicator-specific dimension columns refer to indicators that, in addition to general dimensions, each indicator has other specific dimensions, and these dimensions have corresponding data columns. Taking the data in rows 3 and 4 in Table 5 as an example, row 3 represents the data record with indicator code C01 and indicator name A. The record indicates that the data time is January 1, 2010, the version is V1.0, and the unit is person. The value of the indicator is stored in the i1 data column and also contains two dimensions d1 and d2. The values of these two dimensions are D1 and D2 respectively. Row 4 represents the data of indicator B. The indicator value is stored in the i2 data column, which contains the dN dimension, and the value of the dimension is DN. It can be seen that although indicator A and indicator B are different indicators, the values of the two indicators are stored in different data columns, including some of the same dimensions and their own unique dimensions, but they can also be stored in a unified elastic data table.
[0060] The data table of traditional database technology requires an initial table creation operation, and all columns and column types are defined when the table is created. The elastic data table in the present invention does not require a table creation operation, the table structure is defined when the data is written, and dynamic expansion is supported. The expansion here includes horizontal expansion and vertical expansion. Horizontal expansion means that the data columns of the elastic data table can be dynamically increased when the indicator result data is written. Vertical expansion means that the elastic data table dynamically divides all indicator result data into different data files, and the file division is automatically performed when the indicator result data is written, and different indicator result data is written into different data files.
[0061] Data warehouses built on traditional databases or big data platforms cannot support the above-mentioned elastic data tables. In order to implement elastic data tables, the present invention proposes a system and storage engine device that supports distributed data storage and query. The distributed architecture is adopted in technology. Each node module can support high availability of services, including management node modules, data node modules, query node modules, such as Figure 2 shown.
[0062] The management node module performs unified management and data scheduling for the entire system, including managing metadata information and data sharding information of elastic data tables, managing data distribution of data node modules, etc. The management node module uses Zookeeper to achieve automatic switching between active and standby modes, ensuring high availability of the management node module's services.
[0063] The query node module provides data query request services to the outside world, and supports HTTP and JDBC connection methods for query. The query node module caches the information of the elastic data table in the entire system, including table structure information, data sharding information, and data sharding maintained by each data node module. The query node module intelligently splits the query request into multiple sub-queries, distributes the sub-queries to the best data node module, merges and summarizes the results returned by each sub-query, and returns the final result to the request end.
[0064] The data node module is used to store data files of the elastic data table and provide data query services. After receiving the subquery forwarded by the query node module, the data node module will find the data shards that meet the query conditions according to the query conditions, and use multiple threads to read the data records that meet the conditions from the data files respectively, and then merge and summarize multiple data from different data shards, and finally return the query results of the subquery to the query node module.
[0065] The index result data is usually obtained by final statistics based on data that has been processed for multiple times. In order to write the index result data more conveniently, the storage engine device can start a data writing task in the data node module, which includes an SQL statement for calculating the index result. The data node module connects to the source database through JDBC and sends the SQL statement to the source database. The source database completes the calculation of the index result and waits for the source database to return the query result. After the source database returns the query result, the data node module will receive the returned index result data and write it to the data shard file. When writing data to the data shard file, it is divided according to the strategy. The strategy refers to classifying the indicators according to the subject or business classification of the indicators. Indicators of the same subject or business classification are stored in the same data shard. In a data shard, it can be approximated and evenly distributed according to the number of records of the index result data, and a batch of index result data can be written to different data files of the same data shard. Each data file contains the value of the index result data and metadata information such as the data column stored in the data file. The above-mentioned data node module will preload the metadata information in each data shard file.
[0066] As mentioned above, after the requester initiates a query request, it is first received by the query node, which splits the query into multiple sub-queries and distributes them to multiple data node modules. The data nodes read data records from different data shards and return the merged and summarized data to the query node module. The query node module then merges and summarizes the data returned by multiple data node modules, and finally returns the final calculation result to the requester. The specific process is as follows Figure 3 As shown:
[0067] After the requester connects to the query node module via HTTP or JDBC and initiates a query request, the system processes the query request as follows:
[0068] 1. Query node module: receives the query request initiated by the requester;
[0069] 2. Query node module: Analyze the requested data table, requested indicators and other information in the query request, and find the data shards where the data that meets the conditions is located, and the data node modules where each data shard is located based on the information of the elastic data table in the cached system, and select the most appropriate data node module based on the load and other information;
[0070] 3. Query node module: Based on the analysis results of the previous step, the query request is split into multiple sub-queries, and appropriate query conditions are added to reduce the amount of data processed by the data node module;
[0071] 4. Query node module: distribute subqueries to multiple most appropriate data node modules;
[0072] 5. Data node module: receives sub-query requests distributed by the query node module;
[0073] 6. Data node module: analyzes the data table requested in the subquery request, and determines the data shard where the data may be based on the request conditions and pre-loaded metadata information;
[0074] 7. Data node module: Use multi-threading technology for parallel processing to read data that meets the conditions from each data shard file;
[0075] 8. Data node module: merge / aggregate data obtained from different data shards in the data node module;
[0076] 9. Data node module: returns the local merge / aggregate results to the query node module;
[0077] 10. Query node module: After the query node module distributes the subquery to the data node module, it waits for each data node module to complete the processing of the subquery and receives the local merge / aggregate results returned by each data node module;
[0078] 11. Query node module: after receiving partial results, merge / aggregate the data results returned by multiple data node modules;
[0079] 12. Query node module: returns the final query result to the requester.
[0080] The embodiment of the present application also provides a device for uniformly storing indicator result data, including:
[0081] The data table generation module generates a flexible data table, including:
[0082] Generate common dimension columns, indicator columns, and dimension columns unique to indicators;
[0083] The common dimension refers to the dimension that all indicators have. The column name and data type of the common dimension column are fixed and will not change with the indicator.
[0084] The indicator column refers to the column that stores the indicator result data. Each indicator corresponds to a data column, which is used to store the indicator result data;
[0085] The dimension column unique to the indicator refers to the unique dimension that each indicator has in addition to the common dimension, and the dimension column unique to the indicator has a corresponding data column; the dimension column unique to the indicator stores the unique data in the indicator result data;
[0086] The storage stores the elastic data table in a storage medium.
[0087] Furthermore, the elastic data table supports horizontal expansion and vertical expansion. The horizontal expansion refers to dynamically adding data columns of the elastic data table when writing data in the indicator column. The vertical expansion refers to the elastic data table dynamically dividing all indicator data into different data files, automatically performing file division when writing indicator data, and writing different indicator data into different data files.
[0088] Furthermore, all indicator result data are uniformly stored in the elastic data table, indicator result data with the same dimensions in the indicator result data are stored in the same data column of the elastic data table, and indicator result data with different dimensions in the indicator result data are stored in independent data columns.
[0089] Furthermore, the method for uniformly storing indicator result data is used to store enterprise operation and management data.
[0090] Furthermore, the storage medium is a volatile storage medium or a non-volatile storage medium.
[0091] An embodiment of the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps of the aforementioned method when executing the computer program.
[0092] The embodiment of the present application also provides a computer-readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to execute the aforementioned method.
[0093] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0094] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0095] The above is only the embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A method for unified storage of indicator result data, It is characterized in that Generates a flexible data table, including: Generate common dimension columns, indicator columns, and dimension columns unique to indicators; Among them, the common dimension refers to the dimension that all indicators have. The column name and data type of the common dimension column are fixed and will not change with the change of the indicator; The indicator column refers to the column that stores the indicator result data. Each indicator corresponds to a data column, which is used to store the indicator result data; The unique dimension of an indicator refers to the unique dimension that each indicator has in addition to the common dimension. The unique dimension column of the indicator has its own corresponding data column in the table; the unique dimension column of the indicator stores the unique data in the indicator result data; Storing the elastic data table in a storage medium; The elastic data table supports horizontal expansion and vertical expansion. The horizontal expansion refers to dynamically adding data columns of the elastic data table when writing data in the indicator column. The vertical expansion refers to dynamically dividing all indicator data into different data files by the elastic data table. When the indicator data is written, the file division is automatically performed, and different indicator data is written into different data files. All indicator result data are uniformly stored in the elastic data table, the indicator result data of the same dimension in the indicator result data are stored in the same data column of the elastic data table, and the indicator result data of different dimensions in the indicator result data are stored in independent data columns; The method for uniformly storing the indicator result data is used to store the enterprise operation and management data; The storage medium is a volatile storage medium or a non-volatile storage medium; The method for uniformly storing indicator result data employs a device for uniformly storing indicator result data, comprising: A data table generation module generates a flexible data table, including generating common dimension columns, indicator columns, and dimension columns unique to indicators; Among them, the common dimension refers to the dimension that all indicators have. The column name and data type of the common dimension column are fixed and will not change with the change of the indicator; The indicator column refers to the column that stores the indicator result data. Each indicator corresponds to a data column, which is used to store the indicator result data; The unique dimension of an indicator refers to the unique dimension that each indicator has in addition to the common dimension. The unique dimension of the indicator is listed in the table with its own corresponding data column; the unique dimension column of the indicator stores the unique data in the indicator result data; Storing the elastic data table in a storage medium; The elastic data table supports horizontal expansion and vertical expansion. The horizontal expansion refers to dynamically adding data columns of the elastic data table when writing data in the indicator column. The vertical expansion refers to dynamically dividing all indicator data into different data files by the elastic data table. When the indicator data is written, the file division is automatically performed, and different indicator data is written into different data files. The storage medium is a volatile storage medium or a non-volatile storage medium; The method for unified storage of indicator result data is applied to a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the method for unified storage of indicator result data when executing the computer program; The method for uniformly storing indicator result data employs a computer-readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to execute the method for uniformly storing indicator result data; The method for unified storage of indicator result data uses a system and storage engine device that supports distributed data storage and query, including a management node module, a data node module, and a query node module; The management node module is used to uniformly manage and schedule data for the entire system, including managing metadata information and data sharding information of the elastic data table, managing data distribution of the data node module, and the management node module uses Zookeeper to achieve automatic switching of the master and standby modes; The query node module is used to provide external data query request services, and supports HTTP and JDBC connection methods for query. The query node module will cache the information of the elastic data table in the entire system, including table structure information, data sharding information, and data sharding maintained by each data node module. The query node module intelligently splits the query request into multiple sub-queries, distributes the sub-queries to the best data node module, merges and summarizes the results returned by each sub-query, and returns the final result to the request end; The data node module is used to store the data file of the elastic data table and provide data query services. After receiving the subquery forwarded by the query node module, the data node module will find the data shards that meet the query conditions according to the query conditions, and use multiple threads to read the data records that meet the conditions from the data file respectively, and then merge and summarize multiple data from different data shards, and finally return the query result of the subquery to the query node module; In order to write the indicator result data, the storage engine device starts a data writing task in the data node module. The task includes an SQL statement for calculating the indicator result. The data node module connects to the source database through JDBC and sends the SQL statement to the source database. The source database completes the calculation of the indicator result data and waits for the source database to return the query result. After the source database returns the query result, the data node module receives the returned indicator result data and writes it to the data shard file. When the indicator result data is written to the data shard file, it is divided according to the strategy. The strategy refers to classifying the indicators according to the subject or business classification of the indicators. Indicators of the same subject or business classification are stored in the same data shard. In the same data shard, the indicator result data is approximately evenly distributed according to the number of records of the indicator result data, and a batch of indicator result data is written to different data files of the same data shard. Each data file contains the value of the indicator result data and also contains the data column metadata information stored in the data file. The data node module preloads the metadata information in each data shard file. After the request end connects to the query node module via HTTP or JDBC and initiates a query request, the query request processing flow is as follows: S1. The query node module: receives the query request initiated by the request end; S2. The query node module: analyzes the data table and the requested index information in the query request, and finds the data shards where the data that meets the conditions is located, and the data node modules where each data shard is located, based on the information of the elastic data table in the cached system, and selects the most appropriate data node module based on the load information; S3. The query node module: Based on the analysis results of the previous step, the query request is split into multiple sub-queries, and appropriate query conditions are added to reduce the amount of data processed by the data node module; S4. The query node module: distributes the subqueries to a plurality of the most appropriate data node modules; S5. The data node module: receiving the sub-query request distributed by the query node module; S6. The data node module: analyzes the data table requested in the subquery request, and determines the data shard where the data may be based on the request conditions and preloaded metadata information; S7. The data node module: uses multi-threading technology for parallel processing to read data that meets the conditions from each data shard file; S8. The data node module: merging or aggregating the data obtained from different data shards in the data node module; S9. The data node module: returns the local merge or summary results to the query node module; S10. The query node module: after distributing the subquery to the data node module, the query node module waits for each data node module to complete processing of the subquery, and receives the local merged or summarized results returned by each data node module; S11. The query node module: after receiving the local results, merge or aggregate the data results returned by the multiple data node modules; S12. The query node module returns the final query result to the request end.
