Index data extraction method and device based on field blood relationship

Abstract

The invention provides an index data extraction method and device based on field blood relationship, and the method comprises the steps: S10, extracting a core SQL code, and converting the core SQL code into a standard SQL code; s20, analyzing the standard SQL code into an atomic operation; s30, complementing the atomic operation, and storing the complemented code in a code metadatabase; s40, obtaining indexes and dimension information; s50, performing logic extraction on the code metadatabase according to the indexes and the dimension information to generate corresponding extraction codes; and S60, performing data extraction according to the extraction code. The index data extraction method based on the field consanguinity has the beneficial effects that the working efficiency and accuracy of data extraction are improved, the access of indexes is more flexible and variable, more individual requirements can be supported, and the operation efficiency of codes is higher through optimization and integration.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for extracting indicator data based on field lineage. Background Art

[0002] In the prior art, the data granularity of the report application format of the data center is fixed. To meet the personalized needs of business analysts, business analysts are required to perform data extraction by restoring the application code logic from the underlying detail layer model through self-service analysis.

[0003] Since the applications of the data center are calculated through complex data processing logic in multiple links, there are several problems in restoring the code logic of the application:

[0004] 1) The workload of restoring the existing logic by code is large;

[0005] 2) Manual code refactoring is prone to errors;

[0006] 3) The data center and backend are mainly based on SQL code, and it is difficult for business analysts to understand the code;

[0007] 4) Due to processing and integration by multiple nodes, code redundancy leads to poor operating performance.

[0008] Therefore, a completely new data extraction method is needed. Summary of the invention

[0009] The technical problem to be solved by the present invention is to provide a method and device for extracting indicator data based on field lineage, which can automatically parse existing codes and automatically reconstruct and generate new codes for data extraction according to the personalized needs of business analysts.

[0010] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for extracting indicator data based on field lineage, comprising:

[0011] S10, extracting the core SQL code, and converting the core SQL code into a standard SQL code;

[0012] S20, parsing standard SQL code into atomic operations;

[0013] S30, completing the atomic operation, and storing the completed code in a code metadata database;

[0014] S40, obtaining indicator and dimension information;

[0015] S50, performing logical extraction on the code metadata database according to the indicator and dimension information, and generating corresponding extraction codes;

[0016] S60: extract data according to the extraction code.

[0017] Furthermore, in step S30, it further includes: performing standard name calculation on the field, including:

[0018] S31. Calculate the relationship between two fields through the blood relationship of the fields;

[0019] S32, by using a clustering algorithm, the same fields are integrated into a category of indicators to obtain an aggregate field;

[0020] S33. The name that appears most times in the aggregate field is used as the standard name of the aggregate field.

[0021] Furthermore, in step S50, the logic extraction specifically includes:

[0022] S51. Obtain the lineage of each indicator and remove the code of irrelevant atomic operations;

[0023] S52, locate the model that meets the indicator dimension, and obtain the data acquisition logic set for generating the corresponding indicator;

[0024] S53. Integrate the data acquisition logic set to obtain the data acquisition logic of each indicator.

[0025] Furthermore, after step S53, the method further includes: integrating the indicators into a report.

[0026] Furthermore, the step of integrating the indicators into a report includes:

[0027] S541. Expand the dimension of the master data corresponding to the indicator;

[0028] S542. Aggregate the indicators into the same dimension and perform multi-table association.

[0029] Furthermore, in step 52, a step of optimizing the model is also included, including:

[0030] S521. Filter out the models that meet the indicator dimension requirements;

[0031] S522. Score each model using a formula, where the formula is: score = number of indicators met / product of weight values ​​of each dimension of the model.

[0032] Furthermore, S50 also includes a step of optimizing the extracted code.

[0033] The present invention also relates to an indicator data extraction device based on field lineage, comprising an extraction module, a conversion module, a parsing module, a completion module, an acquisition module, a generation module and a storage module;

[0034] The extraction module is used to extract the core SQL code and extract data according to the extraction code;

[0035] The conversion module is used to convert the core SQL code into a standard SQL code;

[0036] The parsing module is used to parse the standard SQL code into atomic operations;

[0037] The completion module is used to complete the atomic operation;

[0038] The storage module is used to store the completed code in the code metadata database;

[0039] The acquisition module is used to acquire indicator and dimension information;

[0040] The generation module is used to perform logical extraction on the code metadata database according to the indicators and dimension information, and generate corresponding extraction codes.

[0041] The present invention also relates to an electronic device, characterized in that it comprises:

[0042] one or more processors;

[0043] A storage device for storing one or more programs;

[0044] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-described methods.

[0045] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, wherein the program implements the above-described method when executed by a processor.

[0046] The beneficial effects of the present invention are: providing an indicator data extraction method based on field lineage, which improves the work efficiency and accuracy of data extraction, makes the indicator data acquisition more flexible and changeable, and can support more personalized needs. Through optimization and integration, the code runs more efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and other objects, features and advantages of the present disclosure will become more apparent by describing in detail its exemplary embodiments with reference to the accompanying drawings. The accompanying drawings described below are only some embodiments of the present disclosure, and it is clear to a person skilled in the art that other accompanying drawings can be obtained from these accompanying drawings without creative work.

[0048] Figure 1 It is a flow chart of the indicator data extraction method based on field lineage of the present invention;

[0049] Figure 2 A schematic diagram of a sub-flow of the indicator data extraction method based on field lineage of the present invention;

[0050] Figure 3 It is another sub-flow diagram of the indicator data extraction method based on field lineage of the present invention;

[0051] Figure 4 It is another sub-flow diagram of the indicator data extraction method based on field lineage of the present invention;

[0052] Figure 5 It is another sub-flow diagram of the indicator data extraction method based on field lineage of the present invention;

[0053] Figure 6 A block diagram of an indicator data extraction device based on field lineage of the present invention;

[0054] Figure 7 The present invention is a block diagram of an electronic device. DETAILED DESCRIPTION

[0055] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present invention will be comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0056] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solution of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.

[0057] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0058] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0059] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the present inventive concept. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0060] Those skilled in the art will appreciate that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present invention, and therefore cannot be used to limit the protection scope of the present invention.

[0061] See also Figures 1 to 5 , a method for extracting indicator data based on field lineage, comprising:

[0062] S10, extracting the core SQL code, and converting the core SQL code into a standard SQL code;

[0063] S20, parsing standard SQL code into atomic operations;

[0064] S30, completing the atomic operation, and storing the completed code in a code metadata database;

[0065] S40, obtaining indicator and dimension information;

[0066] S50, performing logical extraction on the code metadata database according to the indicator and dimension information, and generating corresponding extraction codes;

[0067] S60: extract data according to the extraction code.

[0068] In this embodiment, since the application of the data middle platform is obtained through multiple links and complex data processing logic, the code is distributed in different programs, which is not convenient for query. Therefore, it is necessary to extract the core code in each program and parse the code through the SQL parsing engine. In order to improve the reusability and readability of the code, the code can be atomized and uniformly structured, and each processing logic can be decomposed into atomic nodes, and then integrated into a complete processing process through blood relationship. At the same time, the information is converted into a unified metadata structure for storage for subsequent use. In order to facilitate user queries, it can also be displayed in the form of a field-level blood relationship diagram.

[0069] Specifically, in step S10, an adapter is provided for different programs to extract the core SQL code, and perform standardized SQL conversion to obtain standard SQL code. For example, by parsing the cpt file of the Fanruan report, the ds data set module or the corresponding processing logic code therein is instantiated into standard SQL code according to the panel parameters.

[0070] In step S20, the standard SQL code is parsed by an SQL parsing engine such as the JSQLParser package, and the standard SQL code is parsed into atomic operations one by one. Similarly, other parsing engines can also be used instead.

[0071] In step S30, the information of each atomic operation is supplemented and improved through the database metadata information (or the data dictionary maintained by the data center). The code is converted into a unified metadata structure for storage. For example, "select cola, colb from table1, table2" can be converted into "select table1.cola, table2.colb from table1, table2" through the database metadata information, so as to clarify the source table of each field. Table 1 is the atomic operation data structure table, and Table 2 is the data structure table of the Expression ion expression.

[0072]

[0073] Table 1

[0074]

[0075] Table 2

[0076] In step S40, the user can select indicators and dimensions from the existing data center application according to personalized needs. Code generation and data extraction can be performed based on the indicators and dimension information selected by the user.

[0077] In step S50, the program will automatically generate new codes that can extract target indicators and dimensions based on the codes in the code metadata database and use field metadata information for logical extraction, as well as multiple processing steps such as code generation, thereby improving the efficiency of data extraction.

[0078] The code is generated by reusing existing SQL, eliminating the need for manual repetitive development, greatly improving the development efficiency of personalized needs. At the same time, it ensures the consistency of data extraction logic and improves data accuracy. The entire process is mainly handled automatically by the program, and the user is only responsible for adjusting the program parameters, which reduces the technical threshold for data extraction.

[0079] In step S60, data extraction is performed using the automatically generated new code, thereby ensuring the accuracy of data extraction.

[0080] As a preferred embodiment of this embodiment, in step S30, it further includes: performing standard name calculation on the field, including:

[0081] S31. Calculate the relationship between two fields through the blood relationship of the fields;

[0082] S32, by using a clustering algorithm, the same fields are integrated into a category of indicators to obtain an aggregate field;

[0083] S33. The name that appears most times in the aggregate field is used as the standard name of the aggregate field.

[0084] In the specific implementation, since the same indicator can be counted in different models, but because the model fields in the data center may have the same name but different meanings or the same meaning but different names, it is difficult for users to quickly locate which models can extract indicator data. Therefore, it is necessary to use the blood relationship between the fields to count the relationship between the model fields and bind a standard name to each field for user query convenience.

[0085] Through standard name calculation, the problem of model fields with the same name but different meanings is solved, thus improving data quality.

[0086] Determine the blood relationship between two fields and mark the two fields according to the blood relationship. For example, if A is directly derived from B and has not been processed, it is marked as "1", indicating that they are completely equal;

[0087] If A is derived from B and is only processed by aggregate functions (SUM, AVG, MAX, MIN) or IFNULL, CAST and other functions, the identifier is "2", indicating that they are basically equal;

[0088] If A is derived from B and processed by other functions, it is marked as "3", indicating a derivative relationship;

[0089] If A comes from B and C and is processed by other functions, it is marked as "4" to indicate a composite relationship;

[0090] If A and B appear in the JOIN condition, they are marked as "1", indicating that they are completely equal.

[0091] Perform cluster calculation on the fields represented as 1 and 2 to obtain the aggregated field. For example, if AB is equal and BC is equal, then the fields are aggregated into ABC.

[0092] The name that appears most frequently in the aggregated field is used as the standard name of the aggregated field. That is, the field name that appears most frequently in field ABC is used as the standard name of the indicator.

[0093] After calculating the standard names of the fields, completely equal or substantially equal fields are aggregated and assigned standard names, which greatly facilitates user queries.

[0094] Through standard name calculation, the association between different model fields is established, providing more options for the source of data for indicators.

[0095] As a preference in this embodiment, in step S50, the logic extraction specifically includes:

[0096] S51. Obtain the lineage of each indicator and remove the code of irrelevant atomic operations;

[0097] S52, locate the model that meets the indicator dimension, and obtain the data acquisition logic set for generating the corresponding indicator;

[0098] S53. Integrate the data acquisition logic set to obtain the data acquisition logic of each indicator.

[0099] In the specific implementation, the lineage of the corresponding indicator is obtained from the source report of each indicator, and the code of irrelevant atomic operations is eliminated, so as to achieve the effect of streamlining the code and improve the efficiency and readability of the code. For example, A in the original application is composed of multiple indicators, and the data can be obtained from B, C, and D respectively. Now only one of the indicators is taken, and the indicator does not need to be obtained from B. In this case, B and subsequent related lineage nodes can be eliminated, and only C, D and subsequent related lineage nodes are retained.

[0100] Using a recursive method, trace back from the application to the data source to locate the model that can support the indicator dimension. If the data is output from the upper source table A, SQLA is put into set A. If not, continue to recursively search for table B, table C, etc. by lineage until the requirements are met. Finally, the calculation SQL set (SqlA, SqlB, etc.) that generates the indicator and the corresponding source table X are obtained and output.

[0101] For example, if the indicator requires the brand dimension and Table A does not meet the requirement, the query continues forward by lineage and Table B can support it. In this case, the logical set for obtaining data for this indicator is the set of SqlA and SqlB.

[0102] Finally, multiple SQLs in the SQL set are integrated into one SQL to obtain the data acquisition logic of the indicator.

[0103] Specifically, first rewrite the SQLA in the collection to the with subquery mode, and use session_ as the with table name;

[0104] Assemble the with subquery before the final application's data retrieval SQLA. Since SQL calls have advantages and disadvantages, the with table needs to be assembled in reverse order.

[0105] Replace the original table A, table B, etc. in the SQL with session_, session_, etc. to get a new SQL.

[0106] By atomically decomposing the code and reorganizing its lineage (single indicator calculation logic extraction), the data of the indicator can be obtained more deeply into the underlying detailed model, making the data granularity finer and supporting data analysis scenarios in more dimensions.

[0107] As a preference in this embodiment, after step S53, the method further includes: integrating the indicators into a report.

[0108] During the specific implementation, after obtaining the data collection SQL logic for each indicator, it is necessary to integrate the indicators into a report to facilitate subsequent operations.

[0109] Specifically, the step of integrating the indicators into a report includes:

[0110] S541. Expand the dimension of the master data corresponding to the indicator;

[0111] S542. Aggregate the indicators into the same dimension and perform multi-table association.

[0112] In specific implementation, since indicator integration requires the same dimensions for each indicator, the master data model that can be directly associated is queried based on the standard name of the field in the field metadata information, and the related dimensions are brought out. It is determined whether they can be directly associated. The judgment is made through the primary key information of the two tables (based on the comparison of the standard names). If Table A contains all the primary key fields of Table B, then Table A and Table B can be directly associated. The association condition is all the primary key fields of Table B. The program adds Table A to associate Table B, and brings out all the dimension fields of Table B. If Table B has a product field, and Table G is the product master data (the primary key is the product), then Table B can expand the corresponding attributes such as brand and category.

[0113] Since the association of data in different dimensions will produce a Cartesian product, it is necessary to first aggregate them by dimension and perform SUM() processing between the indicators.

[0114] As a preference in this embodiment, step 52 further includes a step of optimizing the model, including:

[0115] S521. Filter out the models that meet the indicator dimension requirements;

[0116] S522. Score each model using a formula, where the formula is:

[0117] Score = the product of the number of indicators met / the weight values ​​of each dimension of the model.

[0118] In the specific implementation, since the code only integrates each single-indicator SQL separately, there is repeated code, and the execution performance and readability of the code are not optimal. Therefore, in order to improve the execution efficiency of the code, the model needs to be optimized.

[0119] The optimal model recommendation algorithm can be used to find a more suitable model to replace the existing table to improve code efficiency.

[0120] Specifically, it can be judged from the standard name that the model contains all the required dimensions and at least one indicator, and the models that meet the indicator dimension requirements are screened out, that is, the model set.

[0121] Each model in the model set is scored based on the number of indicators it meets and the dimensions of the model. The more indicators it meets, the higher the score. The more dimensions a model has (the larger the amount of data), the lower the score. The dimensions are weighted according to the amount of data in the dimension value. The formula is:

[0122] Score = the product of the number of indicators met / the weight values ​​of each dimension of the model.

[0123] Please refer to Table 1. The data required are: date, brand, sales volume, delivery quantity and inventory number. According to the formula, it can be calculated that the optimal models are Model C and Model D.

[0124]

[0125] Table 1

[0126] As a preferred embodiment of this embodiment, S50 also includes a step of optimizing the extracted code.

[0127] In specific implementation, the newly generated data retrieval SQL logic integrates the logic of multiple atomic operations. The same atomic operation may be called multiple times. It is necessary to merge atomic operations with the same or similar logic to achieve code simplification and optimization to ensure code execution performance and readability.

[0128] Specifically, the atomic operations are classified, and the atomic operations that use the same data source table and have the same output dimension are classified into one category (the atomic operations with different dimensions cannot be combined); in each category of nodes, the common filtering conditions of the atomic operations are first extracted and put into the where or having attributes of the atomic operation data structure. For the individual indicators, they are put into the expression of each indicator in outputCol to generate new atomic operations; the original atomic operations (with subqueries) are replaced with new atomic operations.

[0129] The code is decomposed into atoms, and the code related to the requirements is extracted from the existing source code according to the field-level blood relationship information, and redundant code is eliminated to make the code more concise; through the optimal model recommendation algorithm, the model with the best performance that meets the requirements is selected for data extraction, which reduces the amount of statistical calculation data for the system; through code merging and optimization, the operation of duplicate code is reduced.

[0130] Please refer to Table 2 to Table 4. There are two atomic operations A and B, which are used to obtain data from table A, sales_money and out_qty. Through the code merging algorithm, they can be merged into an atomic node C.

[0131]

[0132] Table 2

[0133]

[0134] Table 3

[0135]

[0136] Table 4

[0137] From the above description, it can be seen that the beneficial effects of the present invention are: providing an indicator data extraction method based on field lineage, providing an indicator data extraction method based on field lineage, which improves the work efficiency and accuracy of data extraction, and the data acquisition of indicators is more flexible and changeable, which can support more personalized needs. Through optimization and integration, the code runs more efficiently.

[0138] The following are device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments of the present invention, please refer to the method embodiments of the present invention.

[0139] See also Figure 6 , this embodiment discloses an indicator data extraction device based on field lineage, including an extraction module, a conversion module, a parsing module, a completion module, an acquisition module, a generation module and a storage module.

[0140] In the indicator data extraction device based on field lineage, the extraction module is used to extract the core SQL code and perform data extraction according to the extraction code;

[0141] The conversion module is used to convert the core SQL code into a standard SQL code;

[0142] The parsing module is used to parse the standard SQL code into atomic operations;

[0143] The completion module is used to complete the atomic operation;

[0144] The storage module is used to store the completed code in the code metadata database;

[0145] The acquisition module is used to acquire indicator and dimension information;

[0146] The generation module is used to perform logical extraction on the code metadata database according to the indicators and dimension information, and generate corresponding extraction codes.

[0147] In this embodiment, the core code in each program is extracted through the extraction module, and the core SQL code of each program is converted into standard SQL code through the conversion module, and the SQL parsing engine is called by the parsing module to perform code parsing. In order to improve the reusability and readability of the code, it is necessary to atomize and uniformly structure the code, decompose each processing logic into atomic nodes, and then complete the atomic operation through the completion module, and store the completed code in the code meta database through the storage module. When the user needs to extract data according to personalized needs, he can arbitrarily select indicators and dimensions in the data middle platform application. The device can obtain the indicators and dimension information selected by the user through the acquisition module, and finally the generation module performs logical extraction on the code meta database according to the indicators and dimension information to generate the corresponding extraction code.

[0148] As a preferred embodiment of this embodiment, it also includes a calculation module, an integration module and a naming module. Among them, the calculation module is used to calculate the relationship between two fields through the blood relationship of the fields; the integration module is used to integrate the same fields into a type of indicator through a clustering algorithm to obtain an aggregate field; the naming module is used to use the name with the most appearances in the aggregate field as the standard name of the aggregate field.

[0149] In order to help users quickly locate which models can be used to extract indicator data, it is necessary to count the relationships between the fields of each model through the blood relationship between the fields, and bind a standard name to each field to facilitate user query.

[0150] As a preferred embodiment of this embodiment, a removal module is also included. The removal module is used to obtain the lineage of each indicator and remove the code of irrelevant atomic operations; the generation module is also used to locate the model that meets the indicator dimension and obtain the data acquisition logic set for generating the corresponding indicator; the integration module is also used to integrate the data acquisition logic set to obtain the data acquisition logic of each indicator.

[0151] Obtain the lineage of the corresponding indicator from the source report of each indicator, and remove the code of irrelevant atomic operations, so as to achieve the effect of streamlining the code and improve the efficiency and readability of the code. Use a recursive method to trace the source from the application to the data source end and locate the model that can support the indicator dimension closest to it.

[0152] As a preferred embodiment of the present invention, the integration module is also used to integrate various indicators into one report, and also includes an expansion module and an aggregation module. The expansion module is used to expand the dimension of the master data corresponding to the indicator; the aggregation module is used to aggregate various indicators into the same dimension and perform multi-table association.

[0153] Since indicator integration requires the same dimensions for each indicator, the master data model that can be directly associated is queried based on the standard name of the field in the field metadata information, and the related dimensions are brought out. It is determined whether they can be directly associated. The judgment is made based on the primary key information of the two tables (based on the comparison of the standard names). If Table A contains all the primary key fields of Table B, then Table A and Table B can be directly associated. The association condition is all the primary key fields of Table B. The program adds Table A to associate Table B and brings out all the dimension fields of Table B. If Table B has a product field, and Table G is the product master data (the primary key is the product), then Table B can expand the corresponding attributes such as brand and category.

[0154] Since the association of data in different dimensions will produce a Cartesian product, it is necessary to first aggregate them by dimension and perform SUM() processing between the indicators.

[0155] As a preferred embodiment of this embodiment, a screening module and a scoring module are also included. The screening module is used to screen out models that meet the indicator dimension requirements; the scoring module is used to score each model through a formula, and the formula is: score = the number of indicators met / the product of the weight values ​​of each dimension of the model.

[0156] Since the code only integrates the SQL statements of each single indicator separately, there is duplicate code, and the execution performance and readability of the code are not optimal. Therefore, in order to improve the execution efficiency of the code, the model needs to be optimized.

[0157] The optimal model recommendation algorithm can be used to find a more suitable model to replace the existing table to improve code efficiency.

[0158] As a preference in this embodiment, an optimization module is also included, and the optimization module is used to optimize the extraction code.

[0159] In specific implementation, the newly generated data retrieval SQL logic integrates the logic of multiple atomic operations. The same atomic operation may be called multiple times. It is necessary to merge atomic operations with the same or similar logic to achieve code simplification and optimization to ensure code execution performance and readability.

[0160] Specifically, the atomic operations are classified into one category, and the atomic operations that use the same data source table and have the same output dimension are classified into one category (the atomic operations with different dimensions cannot be combined);

[0161] In each type of node, first extract the common filtering conditions of atomic operations and put them into the where or having attribute of the atomic operation data structure. For the individual indicators, put them into the expression of each indicator in outputCol to generate new atomic operations;

[0162] The original atomic operation (with subquery) is replaced with a new atomic operation, thereby streamlining and optimizing the code.

[0163] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment.

[0164] Refer to the following Figure 7 The electronic device according to this embodiment of the present invention will be described. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0165] like Figure 7 As shown, the electronic device is in the form of a general computing device. The components of the electronic device may include, but are not limited to: at least one processing unit, at least one storage unit, a bus connecting different system components (including the storage unit and the processing unit), a display unit, etc.

[0166] The storage unit stores a program code, which can be executed by the processing unit, so that the processing unit performs the steps of various exemplary embodiments of the present invention described in the above-mentioned method for extracting indicator data based on field lineage. For example, the processing unit can perform the following steps: Figures 1 to 5 Follow the steps shown in .

[0167] The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) and / or a cache memory unit, and may further include a read-only memory unit (ROM).

[0168] The storage unit may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0169] The bus may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0170] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication may be performed through an input / output (I / O) interface. In addition, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter may communicate with other modules of the electronic device through a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0171] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to perform the above method according to the implementation of the present invention.

[0172] A computer readable medium, a program product for implementing the above method according to an embodiment of the present invention, can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited to this. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0173] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0174] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0175] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0176] The computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the computer-readable medium implements the following functions:

[0177] Extract the core SQL code and convert it into standard SQL code;

[0178] Parse standard SQL code into atomic operations;

[0179] Complete the atomic operation and store the completed code in the code metadata database;

[0180] Get indicator and dimension information;

[0181] Perform logical extraction of the code metadata database based on the indicators and dimension information, and generate corresponding extraction codes;

[0182] Data extraction was performed according to the extraction code.

[0183] Those skilled in the art will appreciate that the above modules may be distributed in the device according to the description of the embodiment, or may be changed accordingly and used in one or more devices that are different from the present embodiment. The modules and / or units and / or sub-units of the above embodiments may be combined into one module and / or unit and / or sub-unit, or may be further divided into multiple modules and / or units and / or sub-units and / or sub-modules.

[0184] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.

[0185] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structure, configuration or implementation method described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

[0186] In addition, the structures, proportions, sizes, etc. shown in this specification and the drawings are only used to match the contents disclosed in the specification for those skilled in the art to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention, so they have no substantial technical significance. Any modification of the structure, change of the proportion relationship or adjustment of the size, without affecting the technical effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical contents disclosed in the present invention. At the same time, the terms such as "on", "first", "second" and "one" quoted in this specification are only for the convenience of description, and are not used to limit the scope of the implementation of the present invention. The change or adjustment of their relative relationship should also be regarded as the scope of the implementation of the present invention without substantially changing the technical contents.

Claims

1. A method for extracting indicator data based on field lineage, comprising: S10, extracting the core SQL code, and converting the core SQL code into a standard SQL code; S20, parsing standard SQL code into atomic operations; S30, completing the atomic operation, and storing the completed code in a code metadata database; S40, obtaining indicator and dimension information; S50, performing logical extraction on the code metadata database according to the indicator and dimension information, and generating corresponding extraction codes; S60: extract data according to the extraction code.

2. The method according to claim 1, characterized in that: In step S30, it also includes: calculating the standard name of the field, including: S31. Calculate the relationship between two fields through the blood relationship of the fields; S32, by using a clustering algorithm, the same fields are integrated into a category of indicators to obtain an aggregate field; S33. The name that appears most times in the aggregate field is used as the standard name of the aggregate field.

3. The method according to claim 1, characterized in that: In step S50, the logic extraction specifically includes: S51. Obtain the lineage of each indicator and remove the code of irrelevant atomic operations; S52, locate the model that meets the indicator dimension, and obtain the data acquisition logic set for generating the corresponding indicator; S53. Integrate the data acquisition logic set to obtain the data acquisition logic of each indicator.

4. The method according to claim 3, characterized in that: After step S53, the method further includes: integrating the indicators into a report.

5. The method according to claim 4, characterized in that: The step of integrating the indicators into one report includes: S541. Expand the dimension of the master data corresponding to the indicator; S542. Aggregate the indicators into the same dimension and perform multi-table association.

6. The method according to claim 3, characterized in that: In step 52, the step of optimizing the model is also included, including: S521. Filter out models that meet the indicator dimension requirements; S522. Score each model using a formula, where the formula is: score = number of indicators met / product of weight values ​​of each dimension of the model.

7. The method according to claim 1, characterized in that: S50 also includes a step of optimizing the extracted code.

8. A device for extracting index data based on field lineage, characterized in that: It includes an extraction module, a conversion module, a parsing module, a completion module, an acquisition module, a generation module and a storage module; The extraction module is used to extract the core SQL code and extract data according to the extraction code; The conversion module is used to convert the core SQL code into a standard SQL code; The parsing module is used to parse the standard SQL code into atomic operations; The completion module is used to complete the atomic operation; The storage module is used to store the completed code in the code metadata database; The acquisition module is used to acquire indicator and dimension information; The generation module is used to perform logical extraction on the code metadata database according to the indicators and dimension information, and generate corresponding extraction codes.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images