Data asset-based data development method and system

By obtaining and grouping the fields and standard names of data assets in data development, calculating the shortest associated path length, and selecting the appropriate table, the problem of inefficiency in traditional data development is solved and a more efficient data development process is achieved.

CN118885482AActive Publication Date: 2024-11-01JIANGSU KUNSHAN RURAL COMMERCIAL BANK CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410985976.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-01
Estimated Expiration
2044-07-23

AI Technical Summary

Technical Problem

In the process of data development, as the data table increases, traditional methods need to select tables first and then look up fields, which leads to inefficiency, time-consuming and complexity.

Method used

By obtaining the data table fields and standard names in the data asset system, calculating the data standard names list, selecting standard names according to needs, displaying the selected tables and fields, and carrying out data development, and using the preferred implementation method to encode the topic or system as Directory, grouping to form a data standard list, calculate the shortest associated path length through the graph database, select the most appropriate table, reduce table combinations, and improve efficiency.

Benefits of technology

The traditional process of selecting tables first and then searching fields has been changed, which reduces the time to find the required fields, reduces the complexity and number of tables in data development, and improves the efficiency and quality of data development.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118885482A_ABST
    Figure CN118885482A_ABST
Patent Text Reader

Abstract

The invention discloses a data development method based on data assets, and the method comprises the steps: obtaining all data tables in a data asset system, and storing the fields of the data tables and corresponding standard names in a data asset list; calculating a data standard name list from the data asset list table; selecting a data standard name according to requirements, and determining a standard selection result list; and according to a determination standard, selecting a result list, displaying all selected tables and fields, and carrying out data development. According to the method, the required assets are retrieved through the data standard of the data assets, and the selected table is obtained by calculating the least table combinations, so that the traditional process of firstly selecting the table and then searching the field is changed, the time for searching the required field is shortened, and the data development efficiency is improved. And meanwhile, as the minimum table combinations required by the selected data assets are calculated, the number of tables in the data development process is reduced, the complexity of data development is reduced, and the efficiency and quality of data development are improved to a certain extent.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a data development method and system based on data assets. Background Art

[0002] With the continuous development of information technology, data has become increasingly important. In the financial industry, data warehouses or data marts store a large amount of data. Data developers process the data in data warehouses or data marts to generate indicators or report data required by the business.

[0003] In the process of data development, as the number of data tables continues to increase, the efficiency of data development has become an urgent problem to be solved. In the traditional data development process, you need to select the table first and then find the field, which often takes a lot of time and energy to find the required field. Therefore, how to improve the efficiency of data development has become an important research topic. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a data development method and system based on data assets.

[0005] A data development method based on data assets, comprising the following steps:

[0006] Obtain each data table in the data asset system, and store the fields of the data table and the corresponding standard names in the data asset list;

[0007] Calculate the list of data standard names from the data asset list table;

[0008] Select the data standard name according to the requirements and determine the list of standard selection results;

[0009] Select the result list according to the determined criteria, display all the selected tables and fields, and perform data development.

[0010] As a preferred implementation, the subject or system code of the fields in the data asset list is used as the first-level directory, and the standard code of the fields in the data asset list is used as the second-level directory, and the data asset list tables are grouped to form the data standard list.

[0011] As a preferred implementation, a candidate standard list is determined based on the required field information;

[0012] Determine the list of standard selection results based on the requirements and the candidate standard list table.

[0013] As a preferred implementation, the field information includes subject or system coding information of the field and standard coding information of the field;

[0014] Select a corresponding standard from the data standard list based on the subject or system coding information of the required field and the standard coding information of the field, and save the selected standard into the candidate standard list.

[0015] As a preferred implementation, it is determined whether the selected standard specifies the use of a standard from a table in the data asset list. If so, the standard and its corresponding table information are stored in the standard selection result list, and the standard is removed from the candidate standard list;

[0016] If not, the table corresponding to the selected standard in the data asset list is calculated, and the table information obtained by the calculation result is stored in the standard selection result list, and the standard is removed from the candidate standard list;

[0017] Repeat this step until all the criteria in the candidate criteria list are clear.

[0018] As a preferred implementation, in the process of calculating the table corresponding to the selected standard in the data asset list:

[0019] First, determine whether there is a table in the standard selection result list that includes the standard information. If so, the standard also uses the table in the standard selection result list, and stores the standard and its corresponding table information in the judgment standard selection result list. Otherwise:

[0020] All table information in the data asset list is traversed, and the table including the most standards in the candidate standard list is selected.

[0021] As a preferred implementation, when traversing all table information in the data asset list, if multiple tables simultaneously include the same number of standards in the candidate standard list, then:

[0022] The number of times a table is used is used as the weight, and the table with the largest weight is selected.

[0023] As a preferred implementation, when traversing all table information in the data asset list, if multiple tables simultaneously include the same number of standards in the candidate standard list, then:

[0024] According to the graph database, the shortest association path length between each table and the existing tables in the standard selection result list is calculated, and the table with the smallest path length value is selected.

[0025] As a preferred implementation, a result list is selected according to a determined standard, all selected tables and fields are displayed, and data development is performed, including:

[0026] Inter-table association rules, data filtering rules, data grouping rules and data grouping filtering rules.

[0027] As a preferred implementation, after data development is completed, the generated processing rules are converted into job information, table relationship information and field mapping information, and job configuration is generated based on the above information to build a corresponding data warehouse model.

[0028] As a preferred implementation, the calculation results of each step are stored in a table format.

[0029] The present invention also provides a data development system based on data assets, comprising:

[0030] a processor; a database; and a memory, wherein a program is stored,

[0031] When the processor executes the program, any of the aforementioned data asset-based data development methods is implemented.

[0032] Compared with the prior art, the present invention retrieves the required assets through the data standard of the data assets, and obtains the selected table by calculating the minimum table combination, thereby changing the traditional process of first selecting the table and then searching for the field, reducing the time for finding the required field, and improving the data development efficiency. At the same time, because the minimum table combination required for calculating the selected data assets is reduced, the number of tables in the data development process is reduced, the complexity of data development is reduced, and the efficiency and quality of data development are improved to a certain extent.

[0033] Other features and advantages of the present invention will be described in the following description, and part of them will become obvious from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings are only for the purpose of illustrating specific embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0035] Figure 1 The figure is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0036] It is easy to understand that according to the technical solution of the present invention, without changing the essential spirit of the present invention, a person skilled in the art can imagine various embodiments of the present invention. Therefore, the following specific embodiments and drawings are only exemplary descriptions of the technical solution of the present invention, and should not be regarded as the whole of the present invention or as a limitation or limitation of the technical solution of the present invention. On the contrary, the purpose of providing these embodiments is to enable those skilled in the art to understand the present invention more thoroughly. The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the innovative concept of the present invention.

[0037] Example

[0038] A data development method based on data assets, comprising the following steps:

[0039] Obtain each data table in the data asset system, and store the fields of the data table and the corresponding standard names in the data asset list;

[0040] Calculate the list of data standard names from the data asset list table;

[0041] Select the data standard name according to the requirements and determine the list of standard selection results;

[0042] Select the result list according to the determined criteria, display all the selected tables and fields, and perform data development.

[0043] As a preferred method, the data asset list tables are grouped with the subject or system code of the fields in the data asset list as the first-level directory and the standard code of the fields in the data asset list as the second-level directory to form the data standard list.

[0044] As a preferred method, a list of candidate standards is determined based on the required field information;

[0045] Determine the list of standard selection results based on the requirements and the candidate standard list table.

[0046] As a preferred manner, the field information includes subject or system coding information of the field and standard coding information of the field;

[0047] Select a corresponding standard from the data standard list based on the subject or system coding information of the required field and the standard coding information of the field, and save the selected standard into the candidate standard list.

[0048] As a preferred method, it is determined whether the selected standard specifies the use of a standard from a table in the data asset list. If so, the standard and its corresponding table information are stored in the standard selection result list, and the standard is removed from the candidate standard list;

[0049] If not, the table corresponding to the selected standard in the data asset list is calculated, and the table information obtained by the calculation result is stored in the standard selection result list, and the standard is removed from the candidate standard list;

[0050] Repeat this step until all the criteria in the candidate criteria list are clear.

[0051] As a preferred method, in the process of calculating the table corresponding to the selected standard in the data asset list:

[0052] First, determine whether there is a table in the standard selection result list that includes the standard information. If so, the standard also uses the table in the standard selection result list, and stores the standard and its corresponding table information in the judgment standard selection result list. Otherwise:

[0053] All table information in the data asset list is traversed, and the table including the most standards in the candidate standard list is selected.

[0054] As a preferred implementation, when traversing all table information in the data asset list, if multiple tables simultaneously include the same number of standards in the candidate standard list, then:

[0055] The number of times a table is used is used as the weight, and the table with the largest weight is selected.

[0056] As a preferred method, when traversing all table information in the data asset list, if multiple tables simultaneously include the same number of standards in the candidate standard list, then:

[0057] According to the graph database, the shortest association path length between each table and the existing tables in the standard selection result list is calculated, and the table with the smallest path length value is selected.

[0058] As a preferred method, select the result list according to the determined criteria, display all the selected tables and fields, and perform data development, including:

[0059] Inter-table association rules, data filtering rules, data grouping rules and data grouping filtering rules.

[0060] As a preferred method, after data development is completed, the generated processing rules are converted into job information, table relationship information and field mapping information, and job configuration is generated based on the above information to build a corresponding data warehouse model.

[0061] As a preferred manner, the calculation results of each step are stored in a table format.

[0062] The present invention also provides a data development system based on data assets, comprising:

[0063] a processor; a database; and a memory, wherein a program is stored,

[0064] When the processor executes the program, any of the aforementioned data asset-based data development methods is implemented.

[0065] In actual operation, in order to save the calculation results of each step, a series of tables will be established in the calculation system to store the result data of each step, so as to facilitate the subsequent step call or external display. Of course, in actual operation, various tools such as text documents can also be used to save the calculation results of each step.

[0066] In this embodiment, the calculation results of each step are stored in the form of a table. Of course, as a data processing method, the data development method based on data assets described in the present invention may also include other calculation information in the table storing the result data of each step in order to effectively control and implement the calculation process. The addition or non-addition of such information does not affect the implementation of the data development method based on data assets described in the present invention. As an embodiment, the following series of data tables can be used to store the calculation results of each step in the data development process:

[0067] Table 1 Data asset list table DATA_ASSET_DETAIL;

[0068] Table 2 Candidate standard list table CANDIDATE_STD_LISTS;

[0069] Table 3 Standard selection result list table SELECTION_STD_RESULTS;

[0070] Table 4 Job information table MAP_JOB_INFO;

[0071] Table 5 table relationship information table MAP_TABLE_RELATED;

[0072] Table 6 Field mapping information table MAP_MAPPING_COL_INFO

[0073] The table template constructed above can be pre-placed in the storage device of the system.

[0074] The tables described below are two-dimensional tables of relational databases, each of which contains multiple columns, each of which is called a field. Each field corresponds to a standard in the data asset. The standard described here is a specification for a type of data. For example, the private customer number is a standard that exists in multiple tables, such as personal credit card information, personal loan customer credit ledger, and other tables.

[0075] like Figure 1As shown, the data development method based on data assets shown in this embodiment includes the following steps:

[0076] S100, obtaining each data table in the data asset system, and storing the fields of the data table and the corresponding standard names in the data asset list table DATA_ASSET_DETAIL.

[0077] Table 1 Data asset list table DATA_ASSET_DETAIL (partial example)

[0078]

[0079]

[0080] The meanings of the items in Table 1 are as follows:

[0081] BELGTO_SCHEMA is the Schema to which the table belongs, indicating the database mode to which the table belongs.

[0082] TABLE_CODE is the table name,

[0083] TABLE_NAME is the Chinese name of the table.

[0084] COL_CODE is the field name,

[0085] COL_NAME is the Chinese name of the field.

[0086] SYS_CODE is the subject or system code, indicating the system or subject to which the table belongs. The system code is stored before data development, that is, modeling, is completed. After data development is completed, that is, after modeling, the corresponding subject code is stored.

[0087] SYS_NAME is the name of the subject or system, that is, the Chinese name corresponding to SYS_CODE.

[0088] STD_CODE is the standard English name,

[0089] STD_NAME is the standard Chinese name.

[0090] The purpose of this step is to use the data asset reading device to read the list of all tables from the data asset system, store the list information into the storage device, and form a data asset list table for the entire data warehouse to facilitate subsequent annotation selection and data development and use.

[0091] S200, calculating a data standard name list from the data asset list table;

[0092] By grouping the subject or system codes and standard codes in Table 1, a list is formed with the subject or system codes as the first level and the standard codes as the second level, and the list is displayed through a display device;

[0093] For example, according to the data in Table 1, the first-level directory should be INTS (international settlement system), and the second-level directory list should include the data standard list of CORE_CUST_NO (core customer number), MONEXCH_OBJ (foreign exchange settlement and sale object), COLLCT_DATE (collection date), COLLCT_CURRCD (collection currency), and COLLCT_AMT (collection amount), and each of these standards may correspond to fields in multiple tables.

[0094] S300, selecting a data standard name according to the requirements and determining a list of standard selection results;

[0095] Usually, when a user selects a field, he may know some field information he needs, but he does not know which tables contain the field, or the user knows that some tables contain the required field information, but he does not know which tables to select first. Therefore, in this embodiment, the standard selection result list can be determined according to the standard corresponding to the field and the data standard name list obtained in S200;

[0096] More specifically, when the user determines the required field information, that is, the corresponding standard, the required standard can be stored in the candidate standard list table CANDIDATE_STD_LISTS. For example, in this embodiment, the user selects CORE_CUST_NO (core customer number), MONEXCH_OBJ (foreign exchange settlement and sale object), COLLCT_DATE (collection date), COLLCT_CURRCD (collection currency), and COLLCT_AMT (collection amount) of INTS (international settlement system) to construct an event flow table of export bill purchase of export collection financing type, that is, to obtain Table 2 candidate standard list table CANDIDATE_STD_LISTS:

[0097]

[0098] The meanings of the items in Table 2 are as follows: TASK_ID is the task number,

[0099] SELECTED_STDS is a list of standards selected by the user, including all the standards selected by the user.

[0100] CANDIDATE_STDS is a list of candidate standards, which indicates the list of candidate standards whose source table has not yet been determined. When the standard in this item has its source table determined, the standard will be removed from this item. When this column of candidate standards is cleared to empty, the selection is considered complete.

[0101] At the same time, a blank table 3 is obtained, namely the standard selection result list table SELECTION_STD_RESULTS:

[0102] TASK_ID SELECTION_STD BELGTO_SCHEMA TABLE_CODE COL_CODE 1 CORE_CUST_NO [NULL] [NULL] [NULL] 1 MONEXCH_OBJ [NULL] [NULL] [NULL] 1 COLLCT_DATE [NULL] [NULL] [NULL] 1 COLLCT_CURRCD [NULL] [NULL] [NULL] 1 COLLCT_AMT [NULL] [NULL] [NULL]

[0103] The meanings of the items in Table 3 are as follows:

[0104] TASK_ID is the task number, corresponding to the task number in Table 2.

[0105] SELECTION_STD is the standard selected by the user.

[0106] TABLE_CODE is the table name, which corresponds to the table name in Table 1, that is, the table corresponding to the determined selection criteria;

[0107] The following are configurable items:

[0108] BELGTO_SCHEMA is the schema to which the table belongs, corresponding to the schema to which the table in Table 1 belongs.

[0109] COL_CODE is the field name, corresponding to the field name in Table 1.

[0110] Next, determine one by one whether the selected standard specifies the use of a standard from a table in the data asset list. If so, store the standard and its corresponding table information in the standard selection result list, and remove the standard from the candidate standard list;

[0111] If not, the table corresponding to the selected standard in the data asset list is calculated, and the table information obtained by the calculation result is stored in the standard selection result list, and the standard is removed from the candidate standard list;

[0112] At the same time, in the process of calculating the table corresponding to the selected standard in the data asset list:

[0113] First, determine whether there is a table in the standard selection result list that includes the standard information. If so, the standard also uses the table in the standard selection result list, and stores the standard and its corresponding table information in the judgment standard selection result list. Otherwise:

[0114] All table information in the data asset list is traversed, and the table including the most standards in the candidate standard list is selected.

[0115] Repeat this step until all the criteria in the candidate criteria list are cleared.

[0116] Specifically in this embodiment, the user specifies CORE_CUST_NO (core customer number) using the field of the national customer information (SDM_INTS_CUST) table, thereby removing the CORE_CUST_NO (core customer number) standard in CANDIDATE_STDS of the candidate standard list table CANDIDATE_STD_LISTS in Table 2, and storing the standard and its corresponding table information: the national customer information (SDM_INTS_CUST) table in Table 3 standard selection result list table SELECTION_STD_RESULTS, thereby obtaining the updated Table 2 candidate standard list table CANDIDATE_STD_LISTS and Table 3:

[0117] Table 2 Candidate Standard List CANDIDATE_STD_LISTS

[0118]

[0119] Table 3 Standard selection result list SELECTION_STD_RESULTS

[0120] TASK_ID SELECTION_STD BELGTO_SCHEMA TABLE_CODE COL_CODE 1 CORE_CUST_NO SDM SDM_INTS_CUST HX_CUST_ID 1 MONEXCH_OBJ [NULL] [NULL] [NULL] 1 COLLCT_DATE [NULL] [NULL] [NULL] 1 COLLCT_CURRCD [NULL] [NULL] [NULL] 1 COLLCT_AMT [NULL] [NULL] [NULL]

[0121] Afterwards, it is found that the MONEXCH_OBJ (foreign exchange settlement and sale object) standard also exists in the SDM_INTS_CUST table. Therefore, the MONEXCH_OBJ (foreign exchange settlement and sale object) standard is removed from CANDIDATE_STDS of the candidate standard list table CANDIDATE_STD_LISTS in Table 2, and the standard and its corresponding table information: the SDM_INTS_CUST table is stored in the SELECTION_STD_RESULTS table of the standard selection result list in Table 3, thereby obtaining the updated CANDIDATE_STD_LISTS table in Table 2 and Table 3:

[0122] Table 2 Candidate Standard List CANDIDATE_STD_LISTS

[0123]

[0124]

[0125] Table 3 Standard selection result list SELECTION_STD_RESULTS

[0126] TASK_ID SELECTION_STD BELGTO_SCHEMA TABLE_CODE COL_CODE 1 CORE_CUST_NO SDM SDM_INTS_CUST HX_CUST_ID 1 MONEXCH_OBJ SDM SDM_INTS_CUST HX_CUST_ID 1 COLLCT_DATE [NULL] [NULL] [NULL] 1 COLLCT_CURRCD [NULL] [NULL] [NULL] 1 COLLCT_AMT [NULL] [NULL] [NULL]

[0127] Since the existing tables in the standard selection result list table SELECTION_STD_RESULTS in Table 3 do not contain the remaining COLLCT_DATE, COLLCT_CURRCD, and COLLCT_AMT standards, all table information in the data asset list is traversed to select the table that includes the most standards in the candidate standard list.

[0128] In this embodiment, the COLLCT_DATE, COLLCT_CURRCD, and COLLCT_AMT standards can all be found in the SDM_INTS_EXCLMASTER (Export Bill Payment for Export Collection Financing) table, so the table information is stored in Table 3, and the three standards are deleted from the candidate standard list in Table 2;

[0129] After the selection is completed, the data in Table 2 and Table 3 are as follows:

[0130] Table 2 Candidate Standard List CANDIDATE_STD_LISTS

[0131]

[0132] Table 3 Standard selection result list SELECTION_STD_RESULTS

[0133]

[0134] It should be noted that when selecting a table here, if multiple tables include the same number of standards in the candidate standard list, the table with the largest weight can be selected based on the number of times the table is used;

[0135] Alternatively, the shortest association path length between each table and the existing tables in the standard selection result list can be calculated based on the graph database, and the table with the smallest path length value can be selected (refer to the method in patent CN115328883A);

[0136] You can also select a table randomly.

[0137] At this point, in Table 2, the candidate standard list table CANDIDATE_STD_LISTS, the standards in the candidate standard list list have been cleared, and the determination of the standard selection result list has been completed;

[0138] S400, selecting a result list according to the determined standard, displaying all selected tables and fields, and performing data development.

[0139] All selected tables and fields are displayed on a display device for users to conduct data development; users choose to add or remove fields from the optional fields and add corresponding field processing rules, and fill in association rules, filtering conditions, grouping rules, group filtering rules, etc.

[0140] For example: by filling in the mapping rules, the fields are processed;

[0141] By filling in the association rules between tables, that is, the join conditions in the query statement, the association conditions between multiple tables are supplemented;

[0142] Filter the data results by filling in the filtering conditions, that is, the where conditions in the query;

[0143] Group the data by filling in the grouping rules, that is, the group conditions in the query statement;

[0144] By filling in the grouping filter rules, that is, the having conditions in the query statement, the results after the data is grouped are filtered.

[0145] Here, during the development process, the user is also allowed to replace the selected table according to the requirements, or to re-specify the table for more standards and re-run the above process;

[0146] You can also use a data warehouse modeling method and a method proposed by the system (patent CN115328883A) for data development to simplify data development work.

[0147] After completing the above steps, the generated processing rules will be recorded in the operation information table shown in Table 4, the table relationship information table shown in Table 5, and the field mapping information table shown in Table 6 preset in the storage device.

[0148] Table 4 Job information table MAP_JOB_INFO (partial example)

[0149]

[0150] Table 4 Job Information Table MAP_JOB_INFO stores basic information of the job, including basic configuration of the job, scheduling frequency, filtering conditions, grouping conditions, group filtering conditions, etc. Table 4 is a partial example of actual cases, showing filtering conditions, grouping conditions, group filtering conditions, etc. The meaning of each item in Table 4 is as follows:

[0151] JOB_NAME is the job name,

[0152] DATABASE_ENV is the database schema where the target table is located.

[0153] WHERE_INFO is a filtering rule used to filter data results.

[0154] GROUP_INFO is the grouping rule used to group the results.

[0155] HAVING_INFO is a group filtering rule, which is used together with the grouping rule to filter the grouping results.

[0156] COMMENT is the table comment.

[0157] Table 5 Table relationship information table MAP_TABLE_RELATED (partial example)

[0158]

[0159] Table 5 Table relationship information table MAP_TABLE_RELATED stores path association information, ie, association rules in S400.

[0160] The meanings of the items in Table 5 are as follows:

[0161] JOB_CODE is the job number. Multiple rows with the same job number represent the original table set contained in a model table and the association relationship between the original tables.

[0162] JOB_NAME is the job name,

[0163] SERIAL_NUM is the source table serial number.

[0164] DATABASE_ENV is the source table database schema.

[0165] JOIN_TABLE is the name of the source table.

[0166] TABLE_ALIAS is the alias of the source table.

[0167] JOIN_TYPE is the association mode, of which there are 4 types of association modes: 1 represents inner join, 2 represents left outer join, 3 represents right outer join, and 4 represents full outer join;

[0168] JOIN_CONDITION is the association rule.

[0169] Table 6 Field mapping information table MAP_MAPPING_COL_INFO (partial example)

[0170]

[0171]

[0172] Table 6 Field mapping information table MAP_MAPPING_COL_INFO is used to store the fields to be retained and processing rules in S400.

[0173] The meanings of the items in Table 6 are as follows:

[0174] JOB_CODE is the job number,

[0175] JOB_NAME is the job name,

[0176] DATABASE_ENV is the target database schema.

[0177] TAB_NAME is the target table name.

[0178] COL_NO is the target table field number.

[0179] COL_NAME is the target field name,

[0180] COL_COMMENT is the target field comment.

[0181] TYPE_NAME is the target field type.

[0182] COL_LENGTH is the target field length,

[0183] COL_SCALE is the target field precision,

[0184] IS_PK is whether it is a primary key.

[0185] SOURE_TAB_NAME is the source table name,

[0186] SOURE_COL_VALUE is the source table field name,

[0187] SOURCE_ALIAS is the alias of the source table field.

[0188] SOURE_TYPE is the source table field type,

[0189] SOURE_LENGTH is the source table field length,

[0190] SOURE_SCALE is the source table field precision,

[0191] COL_MAPPING is the field mapping relationship.

[0192] Finally, based on the above table information, the job configuration is generated and the corresponding data warehouse model is constructed.

[0193] This solution retrieves the required assets through the data standards of data assets, and obtains the selected table by calculating the minimum table combination, thereby changing the traditional process of selecting the table first and then searching for the field, reducing the time to find the required field, reducing the number of tables in the data development process, reducing the complexity of data development, and improving data development efficiency to a certain extent.

[0194] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

[0195] It should be understood that in order to simplify the present invention and help those skilled in the art understand the various aspects of the present invention, in the above description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes described in a single embodiment or described with reference to a single figure. However, the present invention should not be interpreted as the features included in the exemplary embodiments are all necessary technical features of the patent claims.

[0196] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0197] It should be understood that the modules, units, components, etc. included in the device of an embodiment of the present invention can be adaptively changed to be set in a device different from the embodiment. The different modules, units, or components included in the device of the embodiment can be combined into one module, unit, or component, or they can be divided into multiple sub-modules, sub-units, or sub-components.

[0198] The modules, units or components in the embodiments of the present invention may be implemented in hardware, or in software running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) may be used in practice to implement the embodiments of the present invention. The present invention may also be implemented as a computer program product or computer-readable medium for executing part or all of the methods described herein.

Claims

1. A data development method based on data assets, characterized in that: Obtain each data table in the data asset system, and store the fields of the data table and the corresponding standard names in the data asset list; Calculate the list of data standard names from the data asset list table; Select the data standard name according to the requirements and determine the list of standard selection results; Select the result list according to the determined criteria, display all the selected tables and fields, and perform data development.

2. The data development method based on data assets according to claim 1, characterized in that: The data standard list is formed by grouping the data asset list tables with the subject or system code of the fields in the data asset list as the first-level directory and the standard code of the fields in the data asset list as the second-level directory.

3. The data development method based on data assets according to claim 1, characterized in that: Determine the candidate standard list based on the required field information; Determine the list of standard selection results based on the requirements and the candidate standard list table.

4. The data development method based on data assets according to claim 3, characterized in that: The field information includes subject or system coding information of the field and standard coding information of the field; Select a corresponding standard from the data standard list based on the subject or system coding information of the required field and the standard coding information of the field, and save the selected standard into the candidate standard list.

5. The data development method based on data assets as claimed in claim 3, characterized in that: Determine whether the selected standard specifies the use of a standard from a table in the data asset list. If so, store the standard and its corresponding table information in the standard selection result list, and remove the standard from the candidate standard list; If not, the table corresponding to the selected standard in the data asset list is calculated, and the table information obtained by the calculation result is stored in the standard selection result list, and the standard is removed from the candidate standard list; Repeat this step until all the criteria in the candidate criteria list are clear.

6. The data development method based on data assets according to claim 5, characterized in that: In the process of calculating the table corresponding to the selected standard in the data asset list: First, determine whether there is a table in the standard selection result list that includes the standard information. If so, the standard also uses the table in the standard selection result list, and stores the standard and its corresponding table information in the judgment standard selection result list. Otherwise: All table information in the data asset list is traversed, and the table including the most standards in the candidate standard list is selected.

7. The data development method based on data assets according to claim 6, characterized in that: When traversing all table information in the data asset list, if multiple tables include the same number of standards in the candidate standard list, then: The number of times a table is used is used as the weight, and the table with the largest weight is selected.

8. The data development method based on data assets according to claim 6, characterized in that: When traversing all table information in the data asset list, if multiple tables include the same number of standards in the candidate standard list, then: According to the graph database, the shortest association path length between each table and the existing tables in the standard selection result list is calculated, and the table with the smallest path length value is selected.

9. The data development method based on data assets according to claim 1, characterized in that: Select the result list according to the determined criteria, display all the selected tables and fields, and perform data development, including: Inter-table association rules, data filtering rules, data grouping rules and data grouping filtering rules.

10. The data development method based on data assets according to claim 9, characterized in that: When data development is completed, the generated processing rules are converted into job information, table relationship information and field mapping information, and job configuration is generated based on the above information to build the corresponding data warehouse model.

11. The data development method based on data assets according to any one of claims 1 to 10, characterized in that: The calculation results of each step are stored in table form.

12. A data development system based on data assets, characterized in that: include: processor; database; and a memory storing a program, When the processor executes the program, the data development method based on data assets described in any one of claims 1 to 10 is implemented.

Citation Information

Patent Citations

  • Data query method and device, data middle ground and storage medium

    CN111930823A

  • Data table standardization method, device and equipment and computer storage medium

    CN114648010A

  • Data management method and system based on dimension modeling

    CN117573646A

  • Field verification method

    CN117688040A

  • System, Method, and Computer Program Products for Modeling Complex Hierarchical Metadata with Multi-Generational Terms

    US20230385248A1