Index data preprocessing method and device and computer readable storage medium
By building and optimizing the data preprocessing index map, the problems of poor flexibility, poor performance and high change cost in the metric data processing methods in the business intelligence field are solved, and high performance, flexibility and automated metric data processing are achieved.
Patent Information
- Application Number
- CN202411738248.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-02-11
AI Technical Summary
The existing indicator data processing methods in the field of business intelligence have problems such as poor flexibility, poor performance and high change costs, which are difficult to meet the rapid changes and efficient computing needs of indicator queries.
An index data preprocessing method is adopted to receive pre-computing scenario configuration information, obtain index definition information and model information, analyze data statistics information, and build a data preprocessing index graph IndexGraph. This method uses heuristic strategies and ROI-based cost rules to optimize the data preprocessing index map, generate high RIO data preprocessing index map, and automatically convert it into data preprocessing tasks to realize automated data processing and query.
It realizes high performance and flexibility of indicator query, reduces the cost of indicator changes, improves the degree of automation of data processing, and ensures a balance between user needs, system capabilities and data characteristics.
Smart Images
Figure CN119415524A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of business intelligence (BI) and big data computing technology, and in particular to an indicator data preprocessing method, device and computer-readable storage medium. Background Art
[0002] With the rapid development of information technology, data has become an important asset for enterprises. How to effectively manage and analyze this data and extract valuable information is crucial to the development of enterprises. Business Intelligence (BI) is a set of technologies and practices used to help decision makers make more informed decisions by analyzing data in the enterprise; BI systems usually include data warehouses, data mining, data analysis, data visualization, and dashboards, which can help enterprises extract valuable information from large amounts of data and transform it into actionable insights.
[0003] Data analysis in the field of business intelligence is based on the calculation and analysis of data indicators based on the data in the data warehouse. The existing methods are as follows: Index query directly uses the source data in the data warehouse, and the data processing logic is maintained by data developers, which will have a great impact on quality and stability. At the same time, it cannot meet the changes in the use of indicators, has poor flexibility, and performance cannot be guaranteed.
[0004] The development logic is defined through the indicator platform, and then published to the big data platform for calculation, and the calculation results are used for query. This will make the indicator changes very heavy and less flexible.
[0005] By brute-force enumerating possible combinations of indicators, calculating and storing these results, all indicator queries can be satisfied quickly. However, there will be a great waste of storage costs and computing resources, and the cost of change will be very high. Changes will lead to recalculation, and the flexibility is extremely poor.
[0006] In view of this, the present invention patent is proposed. Summary of the invention
[0007] In order to solve the above problems, the present invention proposes an indicator data preprocessing method, device and computer-readable storage medium. Specifically, the following technical solutions are adopted: An indicator data preprocessing method, comprising: Receive pre-calculated scene configuration information; Obtain indicator definition information and model information required for the pre-calculation scenario according to the pre-calculation scenario configuration information, analyze the data tables referenced by each model in the model information, and obtain data statistical information; Get system information; The pre-calculation scenario configuration information, data statistics information and system information are collected, and a data pre-processing index graph IndexGraph is constructed according to a heuristic strategy.
[0008] As an optional implementation manner of the present invention, in an indicator data preprocessing method of the present invention, the data table referenced by each model in the model information is analyzed to obtain data statistical information including: Analyze the data tables referenced by the model, and obtain data statistics including the number of rows, number of partitions, column cardinality, column null value ratio, and column combination cardinality.
[0009] As an optional embodiment of the present invention, in an indicator data preprocessing method of the present invention, the collecting of the pre-calculated scenario configuration information, data statistical information and system information, and constructing and generating a data preprocessing index graph IndexGraph according to a heuristic strategy includes: Determine the model information required for each indicator based on the indicator definition information in the pre-calculated scenario configuration information; According to the model information, a data preprocessing index is formed by the aggregation results of at least one dimension and index of a model within a predetermined time range; According to the model information, all the dimension combinations in a model are combined with the indicators to form a preprocessing index tree Index Tree with a tree structure. Each node of the preprocessing index tree Index Tree is a preprocessing index Index. The preprocessing index tree Index Tree has a hierarchical relationship. The upper preprocessing index Index is generated by the lower preprocessing index Index. A preprocessing index tree Index Tree is created for each model according to the model information, and the preprocessing index tree Index Tree is combined and constructed according to the relationship between the indicators to obtain the data preprocessing index graph IndexGraph.
[0010] As an optional implementation manner of the present invention, an indicator data preprocessing method of the present invention includes: Selecting an applicable optimization rule Rule from a rule pool according to the data preprocessing index graph IndexGraph; Load applicable optimization rules Rule one by one, optimize the data preprocessing index graph IndexGraph, and generate a high RIO data preprocessing index graph IndexGraph; Therein, a plurality of optimization rules are preset in the rule pool, and the optimization rules include heuristic rules and ROI-based cost rules.
[0011] As an optional implementation manner of the present invention, an indicator data preprocessing method of the present invention includes: Receive the optimized and generated high RIO data preprocessing index graph IndexGraph, and convert the data preprocessing index Index in the high RIO data preprocessing index graph IndexGraph into a data preprocessing task; The dependency relationship between data preprocessing tasks is constructed according to the dependency relationship between the data preprocessing indexes Index in the high RIO data preprocessing index graph IndexGraph.
[0012] As an optional implementation manner of the present invention, an indicator data preprocessing method of the present invention includes: According to the data preprocessing task, start the preprocessing of historical data; The preprocessing of historical data is automatically performed according to the dependencies between data preprocessing tasks; After completing the preprocessing of historical data, incremental scheduling is performed on the result data of the preprocessing work.
[0013] As an optional implementation manner of the present invention, an indicator data preprocessing method of the present invention includes: Receive indicator query requests; According to the index query request, obtaining the best data preprocessing index Index or the best data preprocessing index Index combination based on the high RIO data preprocessing index graph IndexGraph; Obtain the result data of the corresponding data preprocessing task execution according to the best data preprocessing index Index, and return the query result; Alternatively, according to the best data preprocessing index combination, the result data of the corresponding data preprocessing tasks are obtained respectively, and after performing combined calculations on the result data, the query result is returned.
[0014] The present invention also provides an indicator data preprocessing device, comprising: A pre-calculation scene configuration module receives pre-calculation scene configuration information; The statistical analysis module obtains the indicator definition information and the model information required for the pre-calculated scenario according to the pre-calculated scenario configuration information, analyzes the data table referenced by each model in the model information, and obtains data statistical information; System information acquisition module, to obtain system information; The index construction module collects the pre-calculated scene configuration information, data statistics information and system information, and constructs and generates a data pre-processing index graph IndexGraph according to a heuristic strategy.
[0015] As an optional implementation of the present invention, an indicator data preprocessing device of the present invention includes a rule optimization module: Selecting an applicable optimization rule Rule from a rule pool according to the data preprocessing index graph IndexGraph; Load applicable optimization rules Rule one by one, optimize the data preprocessing index graph IndexGraph, and generate a high RIO data preprocessing index graph IndexGraph; Therein, a plurality of optimization rules are preset in the rule pool, and the optimization rules include heuristic rules and ROI-based cost rules.
[0016] The indicator data preprocessing device also includes a task management module: Receive the optimized and generated high RIO data preprocessing index graph IndexGraph, and convert the data preprocessing index Index in the high RIO data preprocessing index graph IndexGraph into a data preprocessing task; The dependency relationship between data preprocessing tasks is constructed based on the dependency relationship between data preprocessing indexes in the high RIO data preprocessing index graph IndexGraph; According to the data preprocessing task, start the preprocessing of historical data; The preprocessing of historical data is automatically performed according to the dependencies between data preprocessing tasks; After completing the preprocessing of historical data, incremental scheduling is performed on the result data of the preprocessing work.
[0017] The present invention also provides a computer-readable storage medium storing a computer-executable program. When the computer executable program is executed, the indicator data preprocessing method is implemented.
[0018] Compared with the prior art, the present invention has the following beneficial effects: An index data preprocessing method of the present invention weighs the three aspects of cost, flexibility and performance, and selects valuable data for processing through heuristic rules and ROI-based cost rules. The index requirements and performance requirements specified by the user are abstracted into parameters, and the system information and data statistical information are also abstracted into parameters, which are optimized through the optimization rule Rule, and the final output needs to be pre-calculated. Then, the high RIO data preprocessing index graph IndexGraph is automatically converted into a processing task, and the system automatically completes the data processing. When the user finally queries the index, the system will find the best Index or Index combination to meet the user's query. In this way, ROI is achieved for high-value and computational processing, automated data processing task processing, and high-performance and flexible queries.
[0019] The index data preprocessing method of the present invention has the following technical effects: 1. Indicators and calculation storage costs: By converting user scenario requirements, indicator development semantics, and system information into optimization strategy parameters. Through heuristic rules and ROI-based cost rule optimization, output the dimensions / indicators / time ranges that are most worth pre-calculating. Fully consider user needs, system capabilities, and data characteristics to avoid waste caused by brute force enumeration.
[0020] 2. Performance and flexibility of index query: By generating a tree-shaped data preprocessing index graph IndexGraph, we can find the best index or index combination to meet the query and ensure the performance of the query. Through special processing, we ensure that all indicators can be calculated again, avoiding enumerating all possible queries, thus having good flexibility.
[0021] 3. Automatic data processing: Compared with traditional indicator processing, which requires writing a large number of data processing tasks, data processing tasks can be automatically constructed by parsing the definition of indicators and automatically generating indexes, and users no longer need to develop a large number of data processing scripts. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 A flowchart of an indicator data preprocessing method according to an embodiment of the present invention; Figure 2 A flowchart of constructing a data preprocessing index graph IndexGraph in an indicator data preprocessing method according to an embodiment of the present invention; Figure 3 An example graph of an index data preprocessing method constructed to obtain a data preprocessing index graph IndexGraph in an embodiment of the present invention; Figure 4 A flowchart of rule optimization for a data preprocessing index graph IndexGraph in an indicator data preprocessing method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0023] To make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them.
[0024] Therefore, the following detailed description of the embodiments of the present invention is not intended to limit the scope of the invention claimed for protection, but merely represents some embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0025] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.
[0026] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0027] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the invention product is usually placed when in use, or the orientation or positional relationship commonly understood by those skilled in the art. Such terms are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.
[0028] See also Figure 1 As shown, a method for preprocessing indicator data in this embodiment includes: Receive pre-calculated scene configuration information; Obtain indicator definition information and model information required for the pre-calculation scenario according to the pre-calculation scenario configuration information, analyze the data tables referenced by each model in the model information, and obtain data statistical information; Get system information; The pre-calculation scenario configuration information, data statistics information and system information are collected, and a data pre-processing index graph IndexGraph is constructed according to a heuristic strategy.
[0029] In the field of business intelligence, pre-computation is an important means to accelerate index query analysis. However, the larger the pre-computation tasks and data volume, the more complicated the operation and maintenance, and the heavier the index / model changes. A method for preprocessing index data in this embodiment aims to produce higher-value data processing and solve the index query performance and automated data processing issues in the index middle station. The whole idea is to establish a data preprocessing index graph IndexGraph for index query, rather than pre-calculating all dimension combinations. An index query will match the best satisfying index or index combination to achieve a balance between performance, pre-computation cost, and flexibility. Users can also control the trade-offs among the three through computing scenario configuration information.
[0030] The model in the model information of this embodiment refers to a structured data organization method formed by associating multiple data tables, including descriptive attributes and measurable values.
[0031] The pre-calculation scenario configuration information of this embodiment includes common dimensions, necessary dimensions, date dimensions, indicators, and related business limitations, etc., wherein common dimensions are dimensions commonly used in the analysis process, such as user ID, device type, geographic location, etc. Common dimensions are generally used for data analysis across multiple business scenarios. In the pre-calculation scenario, common dimensions may be used to group and aggregate indicators. Necessary dimensions are dimensions specific to a certain business scenario or query requirement, and they are essential to meeting specific business needs. In pre-calculation, necessary dimensions are often used together with common dimensions to define the scope and objectives of data preprocessing. The date dimension is an indispensable part of data analysis and is used to group and aggregate data by time. In pre-calculation, the date dimension is often used to construct time series data so that users can query and analyze historical data trends. The date dimension may include different time granularities such as year, month, day, hour, minute, etc. Indicators are the core of data analysis and are used to quantify business performance or user behavior.
[0032] In pre-computation scenarios, indicators are usually pre-computed and stored for fast retrieval and query. Common indicators include atomic indicators, derived indicators, and derived indicators. Related business constraints: Related business constraints are constraints specific to business needs and are used to filter or limit the scope of data. These constraints may include specific user groups, time ranges, geographical restrictions, etc. In pre-computation, business constraints are used to define which data should be included in the pre-computation results.
[0033] This embodiment generates various pre-calculated scenes (scenes) by pre-calculating scene configuration information, including multiple dimensions (Dimension), indicator (Metrics) combinations and storage query requirement attributes. For example: all dimensions and indicators used in a report page can be queried in seconds through scene configuration acceleration. Among them, dimension refers to the category or attribute of data used for analysis and description based on the model (Model). Metrics refers to the definition of aggregatable measurement values based on the model (Model) to quantify business problems.
[0034] The system information of this embodiment includes information of infrastructure such as storage space, CPU, memory, number of host nodes, etc. The main purpose of obtaining the system information is to calculate ROI.
[0035] In an indicator data preprocessing method of this embodiment, the data table referenced by each model in the model information is analyzed to obtain data statistical information including: Analyze the data tables referenced by the model. The obtained data statistics mainly include table statistics, column statistics, index statistics, partition and bucket statistics, such as the number of rows, number of partitions, column cardinality, column null value ratio, and column combination cardinality, which are used as reference for subsequent optimization.
[0036] In an indicator data preprocessing method of this embodiment, the collecting of the pre-calculation scenario configuration information, data statistical information and system information, and constructing and generating a data pre-processing index graph IndexGraph according to a heuristic strategy includes: Determine the model information required for each indicator based on the indicator definition information in the pre-calculated scenario configuration information; According to the model information, a data preprocessing index is formed by the aggregation results of at least one dimension and index of a model within a predetermined time range; According to the model information, all the dimension combinations in a model are combined with the indicators to form a preprocessing index tree Index Tree with a tree structure. Each node of the preprocessing index tree Index Tree is a preprocessing index Index. The preprocessing index tree Index Tree has a hierarchical relationship. The upper preprocessing index Index is generated by the lower preprocessing index Index. A preprocessing index tree Index Tree is created for each model according to the model information, and the preprocessing index tree Index Tree is combined and constructed according to the relationship between the indicators to obtain the data preprocessing index graph IndexGraph.
[0037] See also Figure 3As shown, this is an example diagram of the data preprocessing index graph IndexGraph constructed in this embodiment.
[0038] Further, see Figure 4 As shown, a method for preprocessing indicator data in this embodiment includes: According to the data preprocessing index graph IndexGraph, an applicable optimization rule Rule is selected from the rule pool. Specifically, the optimization rule Rule is as follows: the system initializes a cost threshold for whether to generate an index, and then the system generates an actual cost threshold according to the above statistical information. The two are compared. If the latter is greater than the former, the index needs to be generated, and if it is less than, the index does not need to be generated; Load applicable optimization rules Rule one by one, optimize the data preprocessing index graph IndexGraph, and generate a high RIO data preprocessing index graph IndexGraph; Among them, the rule pool pre-sets multiple optimization rules, including heuristic rules and ROI-based cost rules; in the pre-computation field, the ROI-based cost rule matching is used to determine whether to perform pre-computation operations. As an indicator to measure investment benefits, ROI can help determine whether the pre-computation investment (including computing resources, time, etc.) can bring sufficient benefits in the future (such as improved query performance, accelerated data analysis, etc.) in the pre-computation scenario.
[0039] It can be seen that an indicator data preprocessing method of this embodiment converts user scenario requirements, indicator development semantics, and system information into parameters of the optimization strategy. Through heuristic rules and ROI-based cost rule optimization, a high RIO data preprocessing index graph IndexGraph is derived, and the dimensions / indicators / time ranges most worthy of precalculation are output. Fully consider user needs, system capabilities, and data characteristics to avoid waste caused by brute force enumeration.
[0040] The optimization rule Rule of this embodiment can process the Index / IndexTree / IndexGraph in the data preprocessing index graph IndexGraph, including processing means such as rewriting / clipping / merging / splitting.
[0041] A method for preprocessing indicator data in this embodiment includes: Receive the optimized and generated high RIO data preprocessing index graph IndexGraph, and convert the data preprocessing index Index in the high RIO data preprocessing index graph IndexGraph into a data preprocessing task; The dependency relationship between data preprocessing tasks is constructed according to the dependency relationship between the data preprocessing indexes Index in the high RIO data preprocessing index graph IndexGraph.
[0042] In this embodiment, an indicator data preprocessing method automatically converts the high RIO data preprocessing index graph IndexGraph into a data preprocessing task, and the system automatically completes the data processing. When the user finally queries the indicator, the system will find the best data preprocessing index Index or data preprocessing index Index combination in the high RIO data preprocessing index graph IndexGraph to meet the user's query. In this way, ROI is achieved for high value and computing processing, automated data processing task processing, and high-performance and flexible query.
[0043] A method for preprocessing indicator data in this embodiment includes: According to the data preprocessing task, start the preprocessing of historical data; The preprocessing of historical data is automatically performed according to the dependencies between data preprocessing tasks; After completing the preprocessing of historical data, incremental scheduling is performed on the result data of the preprocessing work. "Incremental scheduling" is a data processing and scheduling strategy. It is mainly used to selectively schedule and process newly generated or changed data after the preprocessing work is completed, rather than reprocessing the entire data set, to ensure that the data is accurate and timely.
[0044] A method for preprocessing indicator data in this embodiment includes: Receive indicator query requests; According to the index query request, obtaining the best data preprocessing index Index or the best data preprocessing index Index combination based on the high RIO data preprocessing index graph IndexGraph; Obtain the result data of the corresponding data preprocessing task execution according to the best data preprocessing index Index, and return the query result; Alternatively, according to the best data preprocessing index combination, the result data of the corresponding data preprocessing tasks are obtained respectively, and after performing combined calculations on the result data, the query result is returned.
[0045] In summary, an index data preprocessing method of the present embodiment weighs the three aspects of cost, flexibility, and performance, and selects valuable data for processing through heuristic rules and ROI-based cost rules. The index requirements and performance requirements specified by the user are abstracted into parameters, and the system information and data statistical information are also abstracted into parameters, which are optimized through the optimization rule Rule, and the final output needs to be pre-calculated. Then, the high RIO data preprocessing index graph IndexGraph is automatically converted into a processing task, and the system automatically completes the data processing. When the user finally queries the index, the system will find the best Index or Index combination to meet the user's query. In this way, ROI is achieved for high-value and computational processing, automated data processing task processing, and high-performance and flexible queries.
[0046] The index data preprocessing method of this embodiment has the following technical effects: 1. Indicators and calculation storage costs: By converting user scenario requirements, indicator development semantics, and system information into optimization strategy parameters. Through heuristic rules and ROI-based cost rule optimization, output the dimensions / indicators / time ranges that are most worth pre-calculating. Fully consider user needs, system capabilities, and data characteristics to avoid waste caused by brute force enumeration.
[0047] 2. Performance and flexibility of index query: By generating a tree-shaped data preprocessing index graph IndexGraph, we can find the best index or index combination to meet the query and ensure the performance of the query. Through special processing, we ensure that all indicators can be calculated again, avoiding enumerating all possible queries, thus having good flexibility.
[0048] 3. Automatic data processing: Compared with traditional indicator processing, which requires writing a large number of data processing tasks, data processing tasks can be automatically constructed by parsing the definition of indicators and automatically generating indexes, and users no longer need to develop a large number of data processing scripts.
[0049] See also Figure 2 As shown, this embodiment also provides an indicator data preprocessing device, including: A pre-calculation scene configuration module receives pre-calculation scene configuration information; The statistical analysis module obtains the indicator definition information and the model information required for the pre-calculated scenario according to the pre-calculated scenario configuration information, analyzes the data table referenced by each model in the model information, and obtains data statistical information; System information acquisition module, to obtain system information; The index construction module collects the pre-calculated scene configuration information, data statistics information and system information, and constructs and generates a data pre-processing index graph IndexGraph according to a heuristic strategy.
[0050] In the field of business intelligence, pre-computation is an important means to accelerate index query analysis. However, the larger the pre-computation tasks and data volume, the more complicated the operation and maintenance, and the heavier the index / model changes. An index data pre-processing device in this embodiment aims to produce higher-value data processing and solve the index query performance and automated data processing issues in the index middle station. The whole idea is to establish a data pre-processing index graph IndexGraph for index query, rather than pre-calculating all dimension combinations. An index query will match the best satisfying index or index combination to achieve a balance between performance, pre-computation cost, and flexibility. Users can also control the trade-offs among the three through computing scenario configuration information.
[0051] Further, see Figure 4 As shown, an indicator data preprocessing device of this embodiment includes a rule optimization module: Selecting an applicable optimization rule Rule from a rule pool according to the data preprocessing index graph IndexGraph; Load applicable optimization rules Rule one by one, optimize the data preprocessing index graph IndexGraph, and generate a high RIO data preprocessing index graph IndexGraph; Wherein, a plurality of optimization rules are pre-set in the rule pool, and the optimization rules include heuristic rules and cost rules based on ROI; The indicator data preprocessing device also includes a task management module: Receive the optimized and generated high RIO data preprocessing index graph IndexGraph, and convert the data preprocessing index Index in the high RIO data preprocessing index graph IndexGraph into a data preprocessing task; The dependency relationship between data preprocessing tasks is constructed according to the dependency relationship between the data preprocessing indexes Index in the high RIO data preprocessing index graph IndexGraph.
[0052] According to the data preprocessing task, start the preprocessing of historical data; The preprocessing of historical data is automatically performed according to the dependencies between data preprocessing tasks; After completing the preprocessing of historical data, incremental scheduling is performed on the result data of the preprocessing work.
[0053] In this embodiment, an index data preprocessing device is used. The index construction module converts user scenario requirements, indicator development semantics, and system information into parameters of the optimization strategy. The rule optimization module derives the high RIO data preprocessing index graph IndexGraph through heuristic rules and ROI-based cost rule optimization, and outputs the dimensions / indicators / time ranges that are most worth precalculating. Fully consider user needs, system capabilities, and data characteristics to avoid waste caused by brute force enumeration.
[0054] In this embodiment, a method for preprocessing index data is used. The task management module automatically converts the high RIO data preprocessing index graph IndexGraph into a data preprocessing task, and the system automatically completes the data processing. When the user finally queries the index, the system will search for the best data preprocessing index Index or data preprocessing index Index combination in the high RIO data preprocessing index graph IndexGraph to meet the user's query. This achieves ROI for high value and computational processing, automated data processing task processing, and high-performance and flexible querying.
[0055] This embodiment also provides a computer-readable storage medium storing a computer-executable program. When the computer-executable program is executed, the indicator data preprocessing method as described above is implemented.
[0056] The computer-readable storage medium described in this embodiment may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus, or a device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.
[0057] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the indicator data preprocessing method.
[0058] The electronic device is presented in the form of a general-purpose computing device. The processor may be one or more than one and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.
[0059] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.
[0060] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).
[0061] It should be understood that the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons and keyboards. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.
[0062] Through the above description of the implementation mode, it is easy for a person skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), or it can be distributed and stored on the network, as long as it enables the electronic device to execute the method according to the present invention.
[0063] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described in the present invention. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the above specific implementation methods. Therefore, any modification or equivalent replacement of the present invention; and all technical solutions and improvements thereof that do not depart from the spirit and scope of the invention are included in the scope of the claims of the present invention.
Claims
1. A method for preprocessing index data, characterized in that: include: Receive pre-calculated scene configuration information; Obtain indicator definition information and model information required for the pre-calculation scenario according to the pre-calculation scenario configuration information, analyze the data tables referenced by each model in the model information, and obtain data statistical information; Get system information; The pre-calculation scenario configuration information, data statistics information and system information are collected, and a data pre-processing index graph IndexGraph is constructed according to a heuristic strategy.
2. The index data preprocessing method according to claim 1, characterized in that: The data tables referenced by each model in the model information are analyzed to obtain data statistics information including: Analyze the data tables referenced by the model, and obtain data statistics including the number of rows, number of partitions, column cardinality, column null value ratio, and column combination cardinality.
3. The index data preprocessing method according to claim 1, characterized in that: The collecting of the pre-calculation scene configuration information, data statistics information and system information, and constructing and generating a data pre-processing index graph IndexGraph according to a heuristic strategy includes: Determine the model information required for each indicator based on the indicator definition information in the pre-calculated scenario configuration information; According to the model information, a data preprocessing index is formed by the aggregation results of at least one dimension and index of a model within a predetermined time range; According to the model information, all the dimension combinations in a model are combined with the indicators to form a preprocessing index tree Index Tree with a tree structure. Each node of the preprocessing index tree Index Tree is a preprocessing index Index. The preprocessing index tree Index Tree has a hierarchical relationship. The upper preprocessing index Index is generated by the lower preprocessing index Index. A preprocessing index tree Index Tree is created for each model according to the model information, and the preprocessing index tree Index Tree is combined and constructed according to the relationship between the indicators to obtain the data preprocessing index graph IndexGraph.
4. The index data preprocessing method according to claim 1, characterized in that: include: According to the data preprocessing index graph IndexGraph, an applicable optimization rule Rule is selected from a rule pool; the applicable optimization rules Rule are loaded one by one, and the data preprocessing index graph IndexGraph is optimized to generate a high RIO data preprocessing index graph IndexGraph; Therein, a plurality of optimization rules are preset in the rule pool, and the optimization rules include heuristic rules and ROI-based cost rules.
5. The index data preprocessing method according to claim 4, characterized in that: include: Receive the optimized and generated high RIO data preprocessing index graph IndexGraph, and convert the data preprocessing index Index in the high RIO data preprocessing index graph IndexGraph into a data preprocessing task; The dependency relationship between data preprocessing tasks is constructed according to the dependency relationship between the data preprocessing indexes Index in the high RIO data preprocessing index graph IndexGraph.
6. The index data preprocessing method according to claim 5, characterized in that: include: According to the data preprocessing task, start the preprocessing of historical data; The preprocessing of historical data is automatically performed according to the dependencies between data preprocessing tasks; After completing the preprocessing of historical data, incremental scheduling is performed on the result data of the preprocessing work.
7. The index data preprocessing method according to claim 6, characterized in that: include: Receive indicator query requests; According to the index query request, obtaining the best data preprocessing index Index or the best data preprocessing index Index combination based on the high RIO data preprocessing index graph IndexGraph; Obtain the result data of the corresponding data preprocessing task execution according to the best data preprocessing index Index, and return the query result; Alternatively, according to the best data preprocessing index combination, the result data of the corresponding data preprocessing tasks are obtained respectively, and after performing combined calculations on the result data, the query result is returned.
8. An indicator data preprocessing device, characterized in that: include: A pre-calculation scene configuration module receives pre-calculation scene configuration information; The statistical analysis module obtains the indicator definition information and the model information required for the pre-calculated scenario according to the pre-calculated scenario configuration information, analyzes the data table referenced by each model in the model information, and obtains data statistical information; System information acquisition module, to obtain system information; The index construction module collects the pre-calculated scene configuration information, data statistics information and system information, and constructs and generates a data pre-processing index graph IndexGraph according to a heuristic strategy.
9. The index data preprocessing device according to claim 8, characterized in that: Including rule optimization module: Selecting an applicable optimization rule Rule from a rule pool according to the data preprocessing index graph IndexGraph; Load applicable optimization rules Rule one by one, optimize the data preprocessing index graph IndexGraph, and generate a high RIO data preprocessing index graph IndexGraph; Wherein, a plurality of optimization rules are pre-set in the rule pool, and the optimization rules include heuristic rules and cost rules based on ROI; The indicator data preprocessing device also includes a task management module: Receive the optimized and generated high RIO data preprocessing index graph IndexGraph, and convert the data preprocessing index Index in the high RIO data preprocessing index graph IndexGraph into a data preprocessing task; The dependency relationship between data preprocessing tasks is constructed based on the dependency relationship between data preprocessing indexes in the high RIO data preprocessing index graph IndexGraph; According to the data preprocessing task, start the preprocessing of historical data; The preprocessing of historical data is automatically performed according to the dependencies between data preprocessing tasks; After completing the preprocessing of historical data, incremental scheduling is performed on the result data of the preprocessing work.
10. A computer-readable storage medium storing a computer-executable program, characterized in that: When the computer executable program is executed, an indicator data preprocessing method as described in any one of claims 1 to 7 is implemented.