Big data index management method and system based on weak model

Abstract

The embodiment of the invention provides a big data index management method and system based on a weak model, and the method comprises the steps: selecting a data source based on a business scene, writing an SQL statement, recognizing dimension information and index information based on the SQL statement, and writing the dimension information and the index information into an index table, correspondingly writing the SQL statement and the index table into a distributed column database; receiving a query request of the user-defined index, querying SQL statements based on parameter key information of the query request in combination with basic information of the user-defined index, assembling the SQL statements, generating user-defined SQL statements, and performing statement optimization on the user-defined SQL statements; index codes are created according to the dimension information and the index information, detail indexes are generated based on the index codes, synchronization parameters of the detail indexes are set, and the detail indexes comprise basic indexes, derivative indexes and composite indexes.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a big data index management method and system based on a weak model. Background Art

[0002] With the development of information systems in the big data era, when enterprises handle business, the business scale is constantly increasing, the data processing scale is gradually becoming huge, and the amount of data is constantly increasing. Enterprises are facing challenges in the management and analysis of massive business data. It is necessary to introduce some management systems to help process and query data to achieve effective management of business data.

[0003] However, in response to the above problems, traditional systems at the current stage still have low query efficiency and cannot meet real-time requirements when facing a large amount of data. Therefore, there is an urgent need for a big data index management method based on a weak model that can solve the above problems. Summary of the Invention

[0004] In view of the problems existing in the prior art, embodiments of the present invention provide a big data index management method and system based on a weak model.

[0005] Embodiments of the present invention provide a big data index management method based on a weak model, and the method includes: Select a data source based on a business scenario, write an SQL statement, identify dimension information and index information based on the SQL statement, write the dimension information and index information into an index table, and write the SQL statement and the index table into a distributed columnar database correspondingly; Receive a query request for a custom index, query an SQL statement based on the parameter key information of the query request, combine the basic information of the custom index, assemble the SQL statement to generate a custom SQL statement, and optimize the statement of the custom SQL statement; Create an index code according to the dimension information and index information, generate detailed indexes based on the index code, and set synchronization parameters for the detailed indexes. The detailed indexes include basic indexes, derivative indexes, and composite indexes.

[0006] In one of the embodiments, the method further includes: Read the data source data, write the data source data into a temporary table in a buffer queue, and generate corresponding partition data files in combination with partition rules; Replace the original partition file with the partition data file through Stream Load of the distributed columnar database, and delete the temporary table.

[0007] In one of the embodiments, the method further includes: Receive custom metrics of the OpenAPI interface, extract key information of the custom metrics based on the business scenario and user requests, where the key information includes specified metrics and specified dimensions.

[0008] In one embodiment, the method further includes: Parse the query request through the JsqlParser library to obtain SQL statements for the specified metrics and specified dimensions, and search for data source tables based on the SQL statements for the specified metrics and specified dimensions. Based on the table relationships between the derived metrics and composite metrics, add join conditions to the corresponding SQL statements, and select corresponding aggregation functions to aggregate the SQL statements according to the aggregation methods of the specified metrics and specified dimensions to generate custom SQL statements.

[0009] In one embodiment, the method further includes: Detect whether there is an index for the SQL fields in the custom SQL statement. When there is no index for the SQL fields, detect the execution efficiency of the SQL statement, and create corresponding indexes for the SQL fields whose execution efficiency is less than the preset threshold. Detect whether there are multiple nested subqueries in the custom SQL statement. When there are multiple nested subqueries in the custom SQL statement, replace the multiple nested subqueries through join operations.

[0010] In one embodiment, the method further includes: Encapsulate the SQL template of the custom SQL statement and convert it to the target format according to the query request. Store the query results of the SQL statement in the LRU cache, and periodically update the cache invalidation results of the SQL statements in the LRU cache.

[0011] An embodiment of the present invention provides a big data metric management system based on a weak model, and the system includes: A writing module, configured to select a data source based on a business scenario, write an SQL statement, identify dimension information and metric information based on the SQL statement, write the dimension information and metric information into a metric table, and correspondingly write the SQL statement and the metric table into a distributed columnar database. A query module, configured to receive a query request for custom metrics, query an SQL statement based on the parameter key information of the query request, combine with the basic information of the custom metrics, assemble the SQL statement to generate a custom SQL statement, and optimize the statement of the custom SQL statement. An index configuration module, configured to create an index code according to the dimension information and index information, generate detailed indexes based on the index code, and set synchronization parameters of the detailed indexes, where the detailed indexes include basic indexes, derivative indexes, and composite indexes.

[0012] In one embodiment, the system further includes: A temporary module, configured to read the data source data, write the data source data into a temporary table in a buffer queue, and generate corresponding partition data files in combination with partition rules; A replacement module, configured to replace the original partition file with the partition data file through Stream Load of a distributed columnar database, and delete the temporary table.

[0013] In view of the above, in one or more embodiments of this specification, a data source is selected based on a business scenario, and an SQL statement is written. The dimension information and index information are identified based on the SQL statement, and the dimension information and index information are written into an index table. The SQL statement and the index table are correspondingly written into a distributed columnar database. A query request for a custom index is received. Based on the parameter key information of the query request and in combination with the basic information of the custom index, the SQL statement is queried, the SQL statement is assembled to generate a custom SQL statement, and the custom SQL statement is optimized. An index code is created according to the dimension information and index information, detailed indexes are generated based on the index code, and synchronization parameters of the detailed indexes are set. The detailed indexes include basic indexes, derivative indexes, and composite indexes. In this way, through steps of data writing, index management, data querying, and data updating, real-time updating and high-performance querying of data corresponding to business requirements can be completed, providing fast response for complex data queries as well, enabling users to obtain analysis results in a short time, providing timely and accurate data support for enterprise decision-making, and improving the efficiency and flexibility of data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0015] Figure 1 is a flowchart of a big data index management method based on a weak model provided by an embodiment of this specification.

[0016] Figure 2 is a schematic structural diagram of a big data index management system based on a weak model provided by an embodiment of this specification. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] Reference will now be made to exemplary embodiments to discuss the subject matter described herein. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and thus implement the subject matter described herein, and is not a limitation on the scope of protection, applicability, or examples set forth in the claims. Changes can be made to the functions and arrangements of the elements discussed without departing from the scope of protection of the content of this specification. Each example can omit, substitute, or add various processes or components as needed. For example, the methods described can be performed in a different order than the described order, and each step can be added, omitted, or combined. Additionally, the features described relative to some examples can also be combined in other examples.

[0018] As used herein, the term "comprising" and its variants denote open terms, meaning "including but not limited to". The term "based on" means "at least partially based on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc. can refer to different or the same objects. Other definitions may be included below, whether explicit or implicit. Unless clearly specified in the context, the definition of a term is consistent throughout the specification.

[0019] As Figure 1 shown, an embodiment of the present invention provides a big data metric management method based on a weak model, including: Step S101, select a data source based on a business scenario, write an SQL statement, identify dimension information and metric information based on the SQL statement, write the dimension information and metric information into a metric table, and correspondingly write the SQL statement and the metric table into a distributed columnar database.

[0020] Specifically, when building an efficient intelligent metrics engine, it is first necessary to write the data from the data source into the database, that is, the data writing process. When writing data, clarify the business requirements according to the business scenario and determine which data sources contain data valuable for the current business scenario. Different types of data sources are suitable for different application scenarios; for example, transaction processing is usually more suitable for using relational databases such as MySQL or PostgreSQL, while large-scale data analysis can use distributed storage systems like Hive. After selecting the data source, since the data from the data source will flow to the distributed columnar database StarRocks, the connectivity and compatibility between the data source and StarRocks should be determined. StarRocks is an enterprise-level MPP (Massively Parallel Processing) database system designed for real-time analysis. It can provide fast data query capabilities and efficient analysis performance, support real-time query of large datasets, and is suitable for online analytical processing (OLAP) scenarios. Moreover, features such as materialized views, index mechanisms, and partitioning in StarRocks, as well as the secondary cache feature of the metrics engine, can improve query performance. At the same time, high-performance data processing is achieved by using technologies such as columnar storage and vectorized execution.

[0021] Furthermore, according to the business logic and goals of the business scenario, identify key metrics. Key metrics include metric information and dimension information. Among them, metric information is the relevant data information of performance metrics, and dimension information is the dimension information of the corresponding users and vendors. Taking the business scenario of e-commerce data analysis as an example, in an e-commerce data analysis scenario, the sales volume metrics for different regions and different product categories are metric information, as well as the related dimension information of users, such as age distribution and gender ratio. Write SQL statements based on the metric information and dimension information to ensure that the SQL statements can accurately reflect the business rules. Then use JsqlParser (a Java library for parsing and modifying SQL queries) to parse the previously written metric definition SQL, obtain all relevant column information, including column names and their data types, and then convert the parsed columns and their types into the data types supported by StarRocks, and create the corresponding metric table, including but not limited to primary keys, indexes, partitioning strategies, etc.

[0022] Furthermore, when building the metric table, a one-metric-per-table pattern can be followed, but is not limited to this. That is, each table focuses on storing a certain or certain types of key metrics under specific business logic. This helps to simplify the data model, making each table structure clear and easy to maintain. In addition, a more compact data layout means lower storage overhead and less space waste.

[0023] Furthermore, to determine the timeliness and consistency of data, based on the information in the metric definition, a rule engine or configuration file is developed to automatically identify and select the most suitable DataX plugin for the current task. DataX is an offline synchronization tool for heterogeneous data sources, mainly used to efficiently migrate data between different data storage systems, supporting multiple data sources such as relational databases (MySQL, Oracle, etc.), NoSQL databases (HBase, MongoDB, etc.), and cloud storage services. It can enhance the scalability of the system and meet the diverse needs of different business scenarios.

[0024] In addition, when writing data to the data source through the DataX plugin, the data source can also be intelligently split, that is, the most suitable splitting strategy is automatically selected according to the data volume. If the data volume is large, the plugin will break down a single large task into multiple subtasks to achieve batch reading and processing. When connecting to the source data source through JDBC, the data read can be first written into a buffer queue instead of directly writing to the target data source. This can smooth the load peak and prevent the target system from being overloaded due to a large amount of data pouring in instantaneously. Then, a table for storing temporary data is pre-created in the target data source (StarRocks). After all the data is successfully written into the temporary table, the data is read according to the established partitioning rules and corresponding partition data files are generated. Using the Stream Load feature of StarRocks, the newly generated partition data files are efficiently replaced with the original partition files in the form of a data stream. After the data synchronization and update are completed and verified without errors, the temporary table is deleted to free up storage space.

[0025] Step S102: Receive a query request for custom metrics. Based on the key information of the parameters of the query request and combined with the basic information of the custom metrics, query the SQL statement, assemble the SQL statement to generate a custom SQL statement, and optimize the statement of the custom SQL statement.

[0026] Specifically, after the metric engine starts working based on the metric definition, it will receive diverse query requests for custom metrics from different business systems or users. The interface for receiving metric query requests can be an open interface, such as a RESTful API interface based on the OpenAPI standard, allowing external systems to initiate data query requests in a standardized manner and dynamically adjust query parameters according to their own business logic. In the user's custom query request, there are customizations for metrics and dimensions. Specific metrics of interest (such as sales volume, click-through rate, etc.) and dimensions (such as time, geographical location, product category, etc.) can be specified in the request to obtain targeted data for market strategy analysis.

[0027] Furthermore, when parsing the query statement, the JSqlParser library can be used to deeply parse the incoming SQL query statement and extract key information such as metrics, dimensions, and aggregation methods. Then, based on the extracted key information and combined with the configuration database of the metric engine, the data source tables involved in the query are determined. When the query involves multiple tables, JSqlParser can automatically add appropriate JOIN statements according to the predefined table relationships and select the corresponding aggregation functions according to the aggregation method and grouping of the key information. Based on the above steps, the parsing steps of the query statement can be illustrated by an example: For example, for the query statement "SELECT SUM(sales) AS total_sales, region FROM sales_data WHERE date BETWEEN '2023-01-01' AND '2023-12-31' GROUP BY region", JSQLPARSER can recognize that "SUM(sales)" is the aggregation function, indicating that the total sales (metric) needs to be calculated, "region" is the grouping dimension, and "date BETWEEN '2023-01-01' AND '2023-12-31'" is the query condition. When determining the data source table, if the sales_data table contains sales data and has sales and region fields, it will become the base table for the query. When adding the join condition, if the region dimension is stored in another region_info table and the sales_data table and the region_info table are associated through the region_id field, the JOIN statement "JOIN region_info ON sales_data.region_id = region_info.region_id" will be automatically generated. When processing aggregation and grouping, the SQL statement "SELECT SUM (sales) AS total_sales, region FROM sales_data JOIN region_info ON sales_data.region_id = region_info.region_id WHERE date BETWEEN '2023-01-01' AND '2023-12-31' GROUP BY region" will be generated. Additionally, during the query, the metric table can also be directly determined without involving the query of the data source table, which can improve the query speed and efficiency of the data.

[0028] In addition, after parsing the query statement to obtain the custom SQL statement, in order to enhance the code reusability and maintainability, the generated SQL statement can be encapsulated in a template form. Thus, the same template can be reused in different scenarios, and only the parameter values need to be replaced.

[0029] Furthermore, further optimization is performed on the SQL statement. The optimization content includes but is not limited to: index optimization, checking whether appropriate indexes already exist for the fields involved in the SQL statement, especially for fields frequently appearing in the WHERE clause, JOIN conditions, and ORDER BY clause, etc. If it is detected that some fields lack necessary indexes, appropriate index creation operations can be recommended and executed based on the data access pattern and query frequency. For example, if the date field is often used for time range filtering, an index can be created for the date field on the sales_data table. For dimensional fields (such as region or category), indexes can also be created to accelerate grouping and sorting operations.

[0030] In addition, the optimization content includes but is not limited to: query rewriting. For queries containing multiple nested subqueries or complex logic, the metric engine can rewrite them, including converting nested subqueries into equivalent JOIN operations to reduce the execution time. Or identifying and eliminating duplicate or unnecessary calculation steps, such as avoiding performing the same operation on the same column multiple times in an aggregation function. Or merging multiple independent filtering conditions into a more compact form.

[0031] In addition, the optimization content includes but is not limited to: utilization of statistical information. Interact with the database system to regularly collect statistical information about the table structure, number of rows, field value distribution, etc. Based on the latest statistical information, the metric engine can dynamically adjust the query execution plan. As the data characteristics change, the optimization strategy can be updated in a timely manner to ensure that the best performance state is always maintained. For example, if the data volume of a certain table grows rapidly, the selection of indexes may need to be re-evaluated; for data with skewed distribution, a partition pruning strategy can be selected to reduce the scan range; using covering indexes to avoid table access operations to further improve query efficiency.

[0032] Furthermore, for the query results, they can be converted into the format required by the user according to the user requirements in the query statement. For example, if it is necessary to return data to the front end or an external system through an API interface, the results can be encapsulated into the JSON format for subsequent transmission and processing. Additionally, after the query is completed, check whether there are the same query conditions. If so, directly read the results from the cache to avoid repeated calculations. The storage strategy of the cache can adopt the LRU eviction strategy and use the LRU (Least Recently Used) algorithm to manage the cache space. When the cache reaches the capacity limit, automatically remove the least recently used data item to ensure that the latest and most frequently used metric data is always retained in the cache. Moreover, considering data timeliness, a reasonable cache expiration time (TTL, Time To Live) can also be defined. Expired data should be updated or cleared in a timely manner to ensure the accuracy of the query results.

[0033] Step S103: Create metric codes according to the dimension information and metric information, generate detailed metrics based on the metric codes, and set the synchronization parameters of the detailed metrics. The detailed metrics include basic metrics, derived metrics, and composite metrics.

[0034] Specifically, to ensure data atomicity and consistency and support incremental updates and full-coverage updates, for the update timing, it can be, but is not limited to, the update of custom metrics after a custom metric query. It can also update the StarRocks data regularly to ensure data timeliness and accuracy. During the data update process, incremental updates and full-coverage updates of data can be achieved through the partitioning function of StarRocks and the XA two-phase commit transaction of the metric engine. The new data source can be selected according to the requirements of the custom SQL statement, and then the new data source information can be connected, such as through a graphical interface or an API interface, and a new data source entry can be added on the system management interface. Then enter and configure the detailed information of the new data source entry, including but not limited to the database type, server address, port number, username, and password, etc. Then determine the specific data source and data table for which the dimension needs to be configured, dynamically parse the structure of the selected data table, obtain the information of all columns, select the required columns from the parsed column information as dimensions, specify the dimension data format, and perform the configuration.

[0035] Furthermore, the update of data can also include the update of metric codes. Define a specific SQL query model for subsequent metric calculations, that is, write an SQL query statement to define the calculation logic of metrics, including aggregation functions (such as SUM, COUNT), grouping conditions (such as GROUP BY), filtering conditions (such as WHERE), etc. Then use the JSqlParser library to parse the SQL statement, extract the key fields and logic, and match the parsed fields with the dimension information configured in step S103 to complete the creation of metric codes. Then create detailed metrics according to the metric codes, and ensure that each detailed metric can be updated and synchronized regularly. Specifically, select a suitable SQL query model for the current business scenario from the saved metric code library, create or reference a data table according to the SQL query model in the metric code, ensure that the data storage structure is reasonable, and then define basic metrics, derivative metrics, and composite metrics. Among them, the basic metric defines the measurement of a certain behavior under the business process, such as the number of ad clicks, the number of orders, etc., and ensure that each atomic metric has a determined data type, algorithm description, and naming. The derivative metric describes the business process, such as the effective click-through rate, conversion rate, etc. The effective click volume = click volume (basic metric) / display volume (basic metric). The composite metric is further calculated through the basic metrics and derivative metrics under the same dimension group, such as the average order amount, customer lifetime value, etc. The average order amount = total sales / number of orders, or the composite metric is further calculated by freely combining and arranging the basic metrics, composite metrics, and derivative metrics through four arithmetic operations, such as gross profit margin, net profit margin, etc. According to the task scheduling information, complete the regular update of data and parameter synchronization.

[0036] Furthermore, in this embodiment, the data query in step S102 and the data update steps in steps S103 and S104 are not limited to the process sequence in this embodiment. In this embodiment, after detecting the data query of the custom metric (or after the period of regular update expires), the data of the custom metric after the data query is updated. For the actual data query and data update steps, it is also possible to discover possible problems (such as data missing, duplicate records, etc.) in the data update process through data query after the data update, so as to perform targeted optimization and adjustment.

[0037] In addition, after creating detailed metrics including basic metrics, derivative metrics, and composite metrics, it can support rapid response to complex business logics and provide the best query experience for end-users. For example, when a user queries through basic metrics, they can parse the basic metric information by extracting information such as technical caliber, metric definition, aggregation method, and dimensions, and then dynamically generate an SQL query statement. Connect to the StarRocks metric query engine through the JDBC protocol and execute the generated SQL statement to obtain data. When a user queries through composite metrics and derivative metric data, they can decompose the query statement to the basic metric granularity, determine the required data tables, then integrate the dimension information and calculation rules of the data tables, dynamically generate an SQL statement, and then submit the query through the JDBC protocol and process the returned results, thus ensuring rapid response to complex business logics, providing comprehensive and accurate data support for end-users, and enabling enterprises to deploy and apply more flexibly in the face of different business requirements and technical environments.

[0038] A big data metric management method based on a weak model provided by an embodiment of the present invention selects a data source based on a business scenario and writes an SQL statement. Identifies dimension information and metric information based on the SQL statement, writes the dimension information and metric information into a metric table, and correspondingly writes the SQL statement and the metric table into a distributed columnar database; receives a query request for a custom metric, queries the SQL statement based on the parameter key information of the query request and in combination with the basic information of the custom metric, assembles the SQL statement to generate a custom SQL statement, and optimizes the custom SQL statement; creates a metric code based on the dimension information and metric information, generates detailed metrics based on the metric code, and sets synchronization parameters for the detailed metrics. The detailed metrics include basic metrics, derivative metrics, and composite metrics. In this way, it can complete real-time update and high-performance query of data corresponding to business requirements through steps of data writing, metric management, data query, and data update, also provides rapid response for complex data queries, enables users to obtain analysis results in a short time, provides timely and accurate data support for enterprise decision-making, and improves the efficiency and flexibility of data analysis.

[0039] Please refer to Figure 2 , Figure 2 is a schematic structural diagram of a big data metric management system provided by an embodiment of the present application. As Figure 2 shown, the system includes: A writing module S201, configured to select a data source based on a business scenario and write an SQL statement, identify dimension information and metric information based on the SQL statement, write the dimension information and metric information into a metric table, and correspondingly write the SQL statement and the metric table into a distributed columnar database; A query module S202, configured to receive a query request for custom metrics, query an SQL statement based on the parameter key information of the query request and in combination with the basic information of the custom metrics, assemble the SQL statement to generate a custom SQL statement, and optimize the statement of the custom SQL statement; An index configuration module S203, configured to create an index code according to the dimension information and index information, generate detailed metrics based on the index code, and set synchronization parameters for the detailed metrics, where the detailed metrics include basic metrics, derivative metrics, and composite metrics.

[0040] In another embodiment, a big data metric management system based on a weak model further includes: A temporary module, configured to read the data source data, write the data source data into a temporary table in a buffer queue, and generate corresponding partition data files in combination with partition rules; A replacement module, configured to replace the original partition file with the partition data file through Stream Load of a distributed columnar database, and delete the temporary table.

[0041] The specific embodiments of the present specification have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired results. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A big data metric management method based on a weak model, the method comprising: Select a data source based on a business scenario, write an SQL statement, identify dimension information and metric information based on the SQL statement, write the dimension information and metric information into a metric table, and correspondingly write the SQL statement and the metric table into a distributed columnar database; Receive a query request for a custom metric, query an SQL statement based on the parameter key information of the query request and in combination with the basic information of the custom metric, assemble the SQL statement to generate a custom SQL statement, and optimize the statement of the custom SQL statement; Create a metric code according to the dimension information and metric information, generate detailed metrics based on the metric code, and set synchronization parameters for the detailed metrics, where the detailed metrics include basic metrics, derived metrics, and composite metrics.

2. The method according to claim 1, wherein The identifying the metric information and dimension information of the data source includes: Read the data source data, write the data source data into a temporary table in a buffer queue, and generate corresponding partition data files in combination with partition rules; Replace the original partition file with the partition data file through Stream Load of the distributed columnar database, and delete the temporary table.

3. The method according to claim 1, characterized in that, The receiving the query request for the custom metric and extracting the key information of the custom metric includes: Receive a custom metric of an OpenAPI interface, and extract the key information of the custom metric based on the business scenario and the user request, where the key information includes a specified metric and a specified dimension.

4. The method according to claim 3, characterized in that, The finding a target metric table based on the key information and assembling an SQL statement through the SQL statement corresponding to the target metric table to generate a custom SQL statement includes: Parse the query request through the JsqlParser library to obtain the SQL statement of the specified metric and the specified dimension, and find a data source table based on the SQL statement of the specified metric and the specified dimension; Add a join condition to the corresponding SQL statement based on the table relationship between the derived metrics and the composite metrics, and select a corresponding aggregation function to aggregate the SQL statement according to the aggregation method of the specified metric and the specified dimension to generate a custom SQL statement.

5. The method according to claim 1, wherein The optimizing the statement of the custom SQL statement includes: Detect whether an index exists for an SQL field in the custom SQL statement. When the SQL field does not have an index, detect the execution efficiency of the SQL statement, and create a corresponding index for the SQL field whose execution efficiency is less than a preset threshold; Detect whether there are multiple nested subqueries in the custom SQL statement. When there are multiple nested subqueries in the custom SQL statement, replace the multiple nested subqueries through a join operation.

6. The method according to claim 1, characterized in that The method further includes: Encapsulate the SQL template of the custom SQL statement and convert it into a target format according to the query request; Store the SQL statement query result in an LRU cache, and periodically update the cache invalidation result of the SQL statement in the LRU cache.

7. A big data metric management system based on a weak model, characterized in that, The system includes; A writing module, which is used to select a data source based on a business scenario, write an SQL statement, identify dimension information and metric information based on the SQL statement, write the dimension information and metric information into a metric table, and write the SQL statement and the metric table into a distributed columnar database correspondingly; A query module, which is used to receive a query request for custom metrics, query an SQL statement based on the parameter key information of the query request and in combination with the basic information of the custom metrics, assemble the SQL statement to generate a custom SQL statement, and optimize the statement for the custom SQL statement; A metric configuration module, which is used to create a metric code according to the dimension information and metric information, generate detailed metrics based on the metric code, and set synchronization parameters for the detailed metrics, where the detailed metrics include basic metrics, derivative metrics, and composite metrics.

8. The system according to claim 7, wherein The system further includes: A temporary module, which is used to read the data source data, write the data source data into a temporary table in a buffer queue, and generate corresponding partition data files in combination with partition rules; A replacement module, which is used to replace the original partition file with the partition data file through Stream Load of the distributed columnar database and delete the temporary table.

Images