A data standard-based algorithm automatic generation device and method

Abstract

The application discloses a kind of algorithm automatic generation device based on data standard, including mutually electrically connected data standard module, data modeling module and algorithm generation module;Wherein, data standard module, for constructing the standard system of general data in wireless big data field;Data modeling module is used to divide level according to the difference of data model focus point, and sequentially arrange the logical model and physical model of data according to level, and mapping relationship is established between data in standard system and logical model and physical model;Algorithm generation module is used to automatically generate data processing algorithm according to mapping relationship.The application solves the gap between standard formulation and standard landing, improves the accuracy and timeliness of data standard landing.

Description

Technical Field

[0001] This invention belongs to the field of wireless big data, and specifically relates to an automatic algorithm generation device and method based on data standards. Background Technology

[0002] Wireless big data is characterized by its large volume, diverse data sources, rich data types, and complex reporting metrics. To better utilize wireless big data and provide a foundation for data analysis in planning, construction, maintenance, optimization, and operation, data governance is necessary. Current methods primarily involve separately developing data standards and data. However, since data development relies on manual implementation of standards, inconsistencies arise between the developed algorithms and the actual data standards, hindering their effective implementation. Furthermore, wireless systems currently have numerous and rapidly evolving metrics, while manual development has long development cycles, making it difficult to quickly adapt to these changes. Summary of the Invention

[0003] The purpose of this invention is to provide an automatic algorithm generation device and method based on data standards, thereby bridging the gap between standard setting and standard implementation, and improving the accuracy and timeliness of data standard implementation.

[0004] To solve the above-mentioned technical problems, the technical solution of the present invention is: an automatic algorithm generation device based on data standards, comprising a data standard module, a data modeling module, and an algorithm generation module electrically connected to each other; wherein,

[0005] The data standards module is used to construct a standard system for general data in the field of wireless big data;

[0006] The data modeling module is used to divide the data into levels according to the differences in the focus of the data model, and to build the logical model and physical model of the data in order of the level, and to establish the mapping relationship between the data in the standard system and the logical model and physical model;

[0007] The algorithm generation module is used to automatically generate data processing algorithms based on mapping relationships.

[0008] It also includes an algorithm execution module, which is used to schedule data processing algorithm tasks and execute data processing algorithms on the data.

[0009] It also includes a system service module for user interaction control and system initialization operations.

[0010] The data standard module includes a terminology standard submodule, an indicator standard submodule, and an indicator constraint submodule; wherein...

[0011] The Terminology Standards submodule is used to define atomic metrics for data and provide standard guidance for data modeling.

[0012] The indicator standards submodule is used to define the indicator standards for data, providing basic information for the automatic generation of data processing algorithms; the indicator standards should at least include the type of indicator and the calculation formula of the indicator.

[0013] The indicator constraint submodule is used to define and describe the constraints related to the indicators, providing auxiliary information for the automatic generation of data processing algorithms.

[0014] The data modeling module includes a logical modeling submodule and a physical modeling submodule; wherein...

[0015] The logical model modeling submodule is used to construct the logical model of the data model and establish the mapping relationship between the logical model and the data.

[0016] The physical modeling submodule is used to build physical models based on logical models, so as to define different physical properties for different databases.

[0017] The algorithm generation module includes an algorithm materialization submodule and an algorithm optimization submodule; wherein...

[0018] The algorithm materialization submodule is used to automatically generate a data query plan based on data standard information and data model information;

[0019] The algorithm optimization submodule is used to dynamically optimize the generated data query plan and automatically generate data processing algorithms.

[0020] A method for automatically generating algorithms based on data standards is also provided, including the following steps:

[0021] In the data standards module, define terminology standards, indicator standards, and constraint standards to form a standard system for general data in the wireless field;

[0022] In the data modeling module, establish logical and physical models of the data, and establish a mapping relationship between the data in the standard system and the logical and physical models;

[0023] By calling the algorithm generation module, the corresponding model and its corresponding indicator data processing algorithm are generated.

[0024] The algorithm generation method of the algorithm generation module is as follows:

[0025] Query plan generation: Based on the mapping relationship between the logical model and the data, a query plan is constructed for each field's data according to the indicator calculation formula information in the standard system.

[0026] Query plan optimization involves adjusting and optimizing the query plan to obtain a query plan with better performance.

[0027] The algorithm generates and combines the optimized query plan into a complete SQL statement, while also generating the corresponding task configuration information and packaging them together to form a data processing algorithm.

[0028] The specific steps for generating a query plan are as follows:

[0029] The calculation formulas corresponding to each field are parsed, and the related terms are searched in the original table. If the corresponding terms are found, the algorithm fragment for this field is generated through the following steps:

[0030] Choose an appropriate aggregation function based on the type of index aggregation;

[0031] Based on the constraints associated with the indicators, the terms are filtered, and a CASE WHEN statement is generated to filter terms that do not meet the constraints.

[0032] Based on the calculation formula of the indicator, the final algorithm fragment of the indicator for that field is generated.

[0033] The specific steps for query plan optimization are as follows:

[0034] For common constraints, extract them into the WHERE clause to avoid the performance loss caused by performing constraint filtering on each field.

[0035] For scenarios that require joining dimension table information, determine whether it is supplementary dimension information and whether it should be used as the clustered primary key. If not, postpone the join execution until after data aggregation to avoid performance loss caused by joining large amounts of data.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] It bridges the gap between standard setting and standard implementation, improving the accuracy and timeliness of data standard implementation. Attached Figure Description

[0038] Figure 1 This is an overall framework diagram of an embodiment of the present invention;

[0039] Figure 2 The flowchart below shows the automatic generation algorithm of an embodiment of the present invention. Detailed Implementation

[0040] The invention will be further described below with reference to specific examples and accompanying drawings.

[0041] This invention provides an automatic algorithm generation device based on data standards, such as... Figure 1 As shown, it includes:

[0042] The Data Standards Module is used to build a standardized, universal data standards system for the wireless domain. Its sub-modules include: Terminology Standards, which define atomic metrics for data and provide standard guidance for detailed data modeling; Indicator Standards, which define indicator standards, including indicator types and calculation formulas, providing foundational information for the automatic generation of data processing algorithms; and Indicator Constraints, which define information describing the constraints related to indicators, providing auxiliary information for the automatic generation of data processing algorithms.

[0043] The data modeling module is used to construct logical and physical data models based on a hierarchical data modeling system and establish mapping relationships with data standards. The logical modeling submodule is used to build the logical model of the data model and provide mapping management between the logical model and relevant data standards. The physical modeling submodule is used to build the physical model based on the logical model and define different physical attributes for different databases.

[0044] The algorithm generation module is used to automatically generate data processing algorithms, ensuring the implementation of data standards. Its sub-module, algorithm materialization, automatically generates query plans based on data standard and data model information. The algorithm optimization sub-module dynamically optimizes the generated query plans and automatically generates data processing algorithms.

[0045] The system service module provides the user interface and handles system initialization. The user interface module offers a user-friendly interface model, facilitating user-defined standards, model design, algorithm generation, and task submission. The system initialization module is responsible for the installation and deployment of the entire system, as well as the import of initialization data.

[0046] The algorithm execution module is used to schedule data processing algorithm tasks and complete the specific execution of the data. The task scheduling module performs the scheduling function for algorithm tasks, with scheduling types including time-driven and data-driven. The algorithm execution module is the computing engine, including but not limited to database engines, big data computing engines such as Spark / Hive / Flink, etc.

[0047] This invention provides an automatic algorithm generation method based on data standards, such as... Figure 2 As shown, it includes the following steps:

[0048] Step 1: Define terminology standards, indicator standards, and constraint standards in the data standards module. Specifically, the terminology definition method includes: name, business meaning, naming conventions, version number, creator, approver, creation time, and modification time. The indicator definition method includes: name, business meaning, naming conventions, version number, creator, approver, creation time, modification time, indicator type, indicator aggregation function, indicator calculation formula, and the constraints the indicator depends on. The constraint definition method includes: name, meaning, version number, creator, approver, creation time, modification time, constraint algorithm, and scope of application.

[0049] Step 2: Define the data model and the data standards associated with each field in the data modeling module. Based on the different focuses of the data model, it is divided into different layers, including: detailed data layer, aggregated data layer, common dimension data layer, and application data layer. The logical model designs the names of the logical tables and their contained fields. Fields can be specifically divided into dimension foreign key fields and terminology / metric measurement fields. The terminology and metric fields establish relationships with the terminology and metric standards defined in the data standards, allowing the program to obtain information such as the calculation formulas of the metrics based on these relationships. The physical model is built based on the logical model and, depending on the specific database, establishes the physical model corresponding to the logical model.

[0050] Step 3: By calling the algorithm generation module, generate the data processing algorithms for the corresponding model and its corresponding indicators. Specifically, the data query plan constructs the data generation algorithm for each field by specifying the logical model and the indicator calculation formula information in the data standard associated with each field. This involves parsing the calculation formula for each field, searching for the relevant basic measures (terms) in the original table, and generating an algorithm fragment for that field through three steps: First, selecting an appropriate aggregation function based on the indicator aggregation type, including SUM, COUNT, DISTINCT, etc.; Second, filtering the basic measures (terms) based on the constraints associated with the indicator, generating a CASE WHEN statement to filter terms that do not meet the constraints; Third, generating the final algorithm fragment for that indicator based on its calculation formula. Since directly generated algorithm fragments have performance issues, the query plan needs to be adjusted to achieve the optimal query algorithm performance. The specific method is as follows: First, for common constraints, extract them into the WHERE clause to avoid performance loss caused by performing constraint filtering on each field. Second, for scenarios requiring the association of dimension table information, determine whether it is supplementary dimension information and whether it should be used as the clustered primary key. If not, the association can be postponed until after data aggregation to avoid performance loss caused by large-scale data association. The data query plan generated based on the above steps is then concatenated into a complete SQL statement. Simultaneously, the task configuration information for this algorithm is generated. The algorithm task is then packaged for use by the subsequent task scheduling and execution modules.

[0051] This invention, based on data standards and the relationship between standards and models, automatically generates data processing algorithms, solving the problem of inconsistencies between data standards and their implementation. It improves the efficiency and accuracy of standard implementation through automation.

[0052] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data standard-based algorithm automatic generation device, characterized by, Comprise data standard module, data modeling module and algorithm generation module which are electrically connected with each other;Wherein, The data standard module is used for constructing the standard system of general data in the field of wireless big data; The data modeling module is used for dividing levels according to the difference of data model concerns, and arranging the logical model and the physical model of data in order according to the levels, and establishing the mapping relationship between the data in the standard system and the logical model and the physical model; The algorithm generation module is used for automatically generating data processing algorithm according to the mapping relationship; The data standard module comprises a term standard submodule, an index standard submodule and an index constraint condition submodule;Wherein, The term standard submodule is used for defining the atomic measurement standard of data, and providing standard guidance for data modeling; The index standard submodule is used for defining the index standard of data, and providing basic information for the automatic generation of data processing algorithm;The index standard at least comprises the type of index and the calculation formula of index; The index constraint condition submodule is used for defining the constraint condition related to index, and providing auxiliary information for the automatic generation of data processing algorithm; The data modeling module comprises a logical model modeling submodule and a physical model modeling submodule;Wherein, The logical model modeling submodule is used for constructing the logical model of data model, and establishing the mapping relationship between the logical model and the data; The physical model modeling submodule is used for constructing the physical model based on the logical model, so as to define different physical properties of different databases; The algorithm generation module comprises an algorithm materialization submodule and an algorithm optimization submodule;Wherein, The algorithm materialization submodule is used for automatically generating the query plan of data based on the data standard information and the data model information; The algorithm optimization submodule is used for dynamically optimizing the generated data query plan and automatically generating data processing algorithm.

2. The apparatus for automatic generation of algorithms based on data criteria according to claim 1, characterized in that, Further comprising an algorithm execution module, which is used for scheduling data processing algorithm task and executing data processing algorithm on data.

3. The apparatus according to claim 1, wherein, Further comprising a system service module, which is used for user interaction control and system initialization operation.

4. A method for automatically generating a device using an algorithm based on data criteria as claimed in claim 1, characterized in that, Comprise the following steps: Define the term standard, the index standard and the constraint standard in the data standard module, and form the standard system of general data in the field of wireless; Establish the logical model and the physical model of data in the data modeling module, and establish the mapping relationship between the data in the standard system and the logical model and the physical model; Generate the data processing algorithm corresponding to the model and the corresponding index by calling the algorithm generation module.

5. The method of claim 4, wherein, The algorithm generation method of the algorithm generation module is: Query plan generation, construct the query plan of each field corresponding to the data according to the mapping relationship between the logical model and the data and the index calculation formula information in the standard system; Query plan optimization, adjust the query plan to obtain the query plan with better performance; Algorithm generation, splice the query plan obtained by optimization to form a complete SQL statement, and generate the supporting task configuration information at the same time, and pack them together to form the data processing algorithm.

6. The method of claim 5, wherein, The specific steps of query plan generation are: Analyze the calculation formula corresponding to each field, find the corresponding term in the original table if the term is found, and generate the algorithm segment of this field by the following steps: According to the index aggregation type, a suitable aggregation function is selected; According to the constraint condition associated with the index, the terms are filtered to generate a case when statement to filter the terms that do not meet the constraint condition; According to the calculation formula of the index, the final algorithm segment of the field index is generated.

7. The method of claim 6, wherein, The specific steps of the query plan optimization are as follows: For the common constraint condition, the constraint is extracted to the where statement for constraint, so as to avoid the performance loss caused by the constraint filtering of each field; For the scene needing to associate the dimension table information, it is judged whether the supplementary dimension information is used as the aggregation primary key, if not, the association is postponed to the data aggregation, so as to avoid the performance loss caused by the large data association.

Images