A data development method and system based on data assets

Abstract

This invention discloses a data development method based on data assets. The method involves acquiring data tables from a data asset system and storing the fields and corresponding standard names of these tables in a data asset list. A list of data standard names is then calculated from the data asset list. Data standard names are selected according to requirements, and a list of selected standard names is determined. Based on this list, all selected tables and fields are displayed, and data development is performed. This invention retrieves the required assets through data asset standards and obtains the selected tables by calculating the minimum number of table combinations. This changes the traditional process of selecting tables first and then searching for fields, reducing the time spent searching for required fields and improving data development efficiency. Furthermore, by calculating the minimum number of table combinations required for the selected data assets, the number of tables in the data development process is reduced, lowering the complexity of data development and improving both efficiency and quality to a certain extent.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a data development method and system based on data assets. Background Technology

[0002] With the continuous development of information technology, data has become increasingly important. In the financial industry, data warehouses or data marts store massive amounts of data. Data developers process the data in data warehouses or data marts to generate the metrics or reports needed for business operations.

[0003] In the data development process, as the number of data tables continues to increase, the efficiency of data development becomes a pressing issue. Traditional data development requires selecting the table first and then searching for the fields, which often consumes a significant amount of time and effort to find the necessary fields. Therefore, improving the efficiency of data development has become an important research topic. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a data development method and system based on data assets.

[0005] A data development method based on data assets includes the following steps:

[0006] Obtain each data table from the data asset system and store the fields and corresponding standard names of the data tables in the data asset list;

[0007] Calculate the list of data standard names from the data asset inventory table;

[0008] Select data standard names according to requirements and determine the list of standard selection results;

[0009] Select a list of results based on the established criteria, display all selected tables and fields, and then proceed with data development.

[0010] In a preferred implementation, the data asset list is grouped into a data standard list by using the subject or system code of the fields in the data asset list as the first-level directory and the standard code of the fields in the data asset list as the second-level directory.

[0011] As a preferred implementation method, a list of candidate criteria is determined based on the required field information;

[0012] Based on the requirements and the list of candidate standards, determine the list of selected standards.

[0013] In a preferred embodiment, the field information includes the field's subject or system encoding information and the field's standard encoding information;

[0014] Select the appropriate standard from the data standard list based on the subject or system encoding information of the required field and the standard encoding information of the field, and save the selected standard into the candidate standard list.

[0015] As a preferred implementation, it is determined whether the selected standard specifies the use of a standard from a table in the data asset list. If so, the standard and its corresponding table information are stored in the standard selection result list and the standard is removed from the candidate standard list.

[0016] If not, calculate the table corresponding to the selected standard in the data asset list, store the table information obtained from the calculation results in the standard selection result list, and remove the standard from the candidate standard list;

[0017] Repeat this step until the list of candidate standards is cleared.

[0018] As a preferred implementation, in the process of calculating the table corresponding to the selected standard in the data asset list:

[0019] First, determine if any existing tables in the standard selection result list contain information about the standard. If so, the standard also uses an existing table in the standard selection result list, and the standard and its corresponding table information are stored in the standard selection result list. Otherwise:

[0020] Iterate through all table information in the data asset list and select the table that includes the most criteria from the candidate criteria list.

[0021] As a preferred implementation, when traversing all table information in the data asset inventory, if multiple tables simultaneously include the same number of standards from the candidate standard list, then:

[0022] The table with the highest weight is selected based on the number of times it is used.

[0023] As a preferred implementation, when traversing all table information in the data asset inventory, if multiple tables simultaneously include the same number of standards from the candidate standard list, then:

[0024] Based on the graph database, calculate the shortest association path length between each table and the existing tables in the standard selection result list, and select the table with the smallest path length value.

[0025] As a preferred implementation, a result list is selected based on defined criteria, all selected tables and fields are displayed, and data development is performed, including:

[0026] Inter-table join rules, data filtering rules, data grouping rules, and data grouping filtering rules.

[0027] As a preferred implementation, once the data development is complete, the generated processing rules are transformed into job information, table relationship information, and field mapping information, and job configurations are generated based on the above information to construct the corresponding data warehouse model.

[0028] In a preferred implementation, the calculation results of each step are stored in tabular form.

[0029] This invention also provides a data development system based on data assets, comprising:

[0030] Processor; database; and memory, in which programs are stored.

[0031] When the processor executes the program, it implements any of the aforementioned data asset-based data development methods.

[0032] Compared with existing technologies, this invention retrieves the required assets through data asset standards and obtains the selected tables by calculating the minimum number of table combinations. This changes the traditional process of selecting tables first and then searching for fields, reducing the time spent searching for required fields and improving data development efficiency. Simultaneously, by calculating the minimum number of table combinations required for the selected data assets, the number of tables in the data development process is reduced, lowering the complexity of data development and improving the efficiency and quality of data development to a certain extent.

[0033] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0034] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0035] Figure 1 This is a flowchart illustrating an embodiment of the present invention. Detailed Implementation

[0036] It is readily understood that, based on the technical solution of this invention, various embodiments of the invention can be conceived by those skilled in the art without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention. Rather, these embodiments are provided to enable those skilled in the art to gain a more thorough understanding of the invention. Preferred embodiments of the invention are described below in conjunction with the accompanying drawings, which form part of this application and, together with the embodiments of the invention, serve to illustrate the innovative concept of the invention.

[0037] Example

[0038] A data development method based on data assets includes the following steps:

[0039] Obtain each data table from the data asset system and store the fields and corresponding standard names of the data tables in the data asset list;

[0040] Calculate the list of data standard names from the data asset inventory table;

[0041] Select data standard names according to requirements and determine the list of standard selection results;

[0042] Select a list of results based on the established criteria, display all selected tables and fields, and then proceed with data development.

[0043] As a preferred approach, the data asset list is grouped into the data standard list by using the subject or system code of the fields in the data asset list as the first-level directory and the standard code of the fields in the data asset list as the second-level directory.

[0044] As a preferred approach, a list of candidate criteria is determined based on the required field information;

[0045] Based on the requirements and the list of candidate standards, determine the list of selected standards.

[0046] As a preferred embodiment, the field information includes the field's subject or system encoding information and the field's standard encoding information;

[0047] Select the appropriate standard from the data standard list based on the subject or system encoding information of the required field and the standard encoding information of the field, and save the selected standard into the candidate standard list.

[0048] As a preferred approach, it is determined whether the selected standard specifies the use of a standard derived from a table in the data asset inventory. If so, the standard and its corresponding table information are stored in the standard selection result list, and the standard is removed from the candidate standard list.

[0049] If not, calculate the table corresponding to the selected standard in the data asset list, store the table information obtained from the calculation results in the standard selection result list, and remove the standard from the candidate standard list;

[0050] Repeat this step until the list of candidate standards is cleared.

[0051] As a preferred approach, in the process of calculating the table corresponding to the selected standard in the data asset inventory:

[0052] First, determine if any existing tables in the standard selection result list contain information about the standard. If so, the standard also uses an existing table in the standard selection result list, and the standard and its corresponding table information are stored in the standard selection result list. Otherwise:

[0053] Iterate through all table information in the data asset list and select the table that includes the most criteria from the candidate criteria list.

[0054] As a preferred implementation, when traversing all table information in the data asset inventory, if multiple tables simultaneously include the same number of standards from the candidate standard list, then:

[0055] The table with the highest weight is selected based on the number of times it is used.

[0056] As a preferred approach, when iterating through all table information in the data asset inventory, if multiple tables simultaneously include the same number of standards from the candidate standard list, then:

[0057] Based on the graph database, calculate the shortest association path length between each table and the existing tables in the standard selection result list, and select the table with the smallest path length value.

[0058] As a preferred approach, a list of results is selected based on established criteria, displaying all selected tables and fields, and data development is performed, including:

[0059] Inter-table join rules, data filtering rules, data grouping rules, and data grouping filtering rules.

[0060] As a preferred approach, once the data development is complete, the resulting processing rules are transformed into job information, table relationship information, and field mapping information. Based on this information, job configurations are generated, and the corresponding data warehouse model is constructed.

[0061] As a preferred approach, the calculation results of each step are stored in tabular form.

[0062] This invention also provides a data development system based on data assets, comprising:

[0063] Processor; database; and memory, in which programs are stored.

[0064] When the processor executes the program, it implements any of the aforementioned data asset-based data development methods.

[0065] In practice, to facilitate saving the calculation results obtained from each step, a series of tables are created in the calculation system to store the results data for subsequent steps or for external display. Of course, in practice, various tools such as text documents can also be used to save the calculation results of each step.

[0066] In this embodiment, the calculation results of each step will be stored in the form of a table. Of course, as a data processing method, the data development method based on data assets described in this invention may include other calculation information in the table storing the results of each step in order to effectively control and implement the calculation process. The addition or omission of this information does not affect the implementation of the data development method based on data assets described in this invention. As an example, the following series of data tables can be used to store the calculation results of each step during the data development process:

[0067] Table 1 Data Asset List Table DATA_ASSET_DETAIL;

[0068] Table 2 Candidate Criterion List (CANDIDATE_STD_LISTS);

[0069] Table 3. List of Standard Selection Results (SELECTION_STD_RESULTS)

[0070] Table 4 Job Information Table MAP_JOB_INFO;

[0071] Table 5 shows the relationship information in the MAP_TABLE_RELATED table.

[0072] Table 6 Field Mapping Information Table MAP_MAPPING_COL_INFO

[0073] The table templates constructed above can be pre-placed in the system's storage device.

[0074] The tables described below are two-dimensional tables in a relational database, where each table contains multiple columns, each of which is called a field. Each field corresponds to a standard in the data assets. The standard referred to here is a specification for a type of data. For example, a personal customer number is a standard, which exists in multiple tables, such as personal credit card information, personal loan customer credit ledgers, and other types of tables.

[0075] like Figure 1As shown in this embodiment, the data development method based on data assets includes the following steps:

[0076] S100: Obtain each data table in the data asset system and store the fields of the data tables and their corresponding standard names in the data asset list table DATA_ASSET_DETAIL.

[0077] Table 1. Data Asset List Table DATA_ASSET_DETAIL (Partial Example)

[0078]

[0079]

[0080] The meanings of each item in Table 1 are as follows:

[0081] BELGTO_SCHEMA is the schema to which the table belongs, indicating the database schema to which the table belongs.

[0082] TABLE_CODE is the table name.

[0083] TABLE_NAME is the Chinese name of the table.

[0084] COL_CODE is the field name.

[0085] COL_NAME is the Chinese name of the field.

[0086] SYS_CODE is the subject or system code, indicating the system or subject to which the table belongs. The system code is stored before data development (i.e., modeling) is completed, and the corresponding subject code is stored after modeling is finished.

[0087] SYS_NAME is the name of the topic or system to which it belongs, which is the Chinese name corresponding to SYS_CODE.

[0088] STD_CODE is the standard English name.

[0089] STD_NAME is the standard Chinese name.

[0090] The purpose of this step is to use the data asset reading device to read the list of all tables in the data asset system, store the list information in the storage device, and form a data asset list table for the entire data warehouse, which will facilitate subsequent labeling, selection and data development.

[0091] S200, calculate the list of data standard names from the data asset list table;

[0092] By grouping the subject or system codes and standard codes in Table 1, a list is formed with the subject or system codes as the first level and the standard codes as the second level, and then displayed on a display device;

[0093] For example, based on the data in Table 1, the first-level directory should be INTS (International Settlement System), and the second-level directory list should include a list of data standards such as CORE_CUST_NO (core customer number), MONEXCH_OBJ (foreign exchange settlement object), COLLCT_DATE (collection date), COLLCT_CURRCD (collection currency), and COLLCT_AMT (collection amount). Each of these standards may correspond to fields in multiple tables.

[0094] S300: Select the data standard name according to the requirements and determine the list of standard selection results;

[0095] Typically, when a user selects fields, they may know some field information they need, but they don't know which tables contain that field, or they may know that some tables contain the required field information, but they don't know which tables to select first. Therefore, in this embodiment, the standard selection result list can be determined according to the standard corresponding to the field and the list of data standard names obtained in S200.

[0096] More specifically, once the user has determined the required field information, i.e., the corresponding standard, the required standard can be stored in the candidate standard list table CANDIDATE_STD_LISTS. For example, in this embodiment, the user selected CORE_CUST_NO (core customer number), MONEXCH_OBJ (foreign exchange settlement object), COLLCT_DATE (collection date), COLLCT_CURRCD (collection currency), and COLLCT_AMT (collection amount) of INTS (international settlement system) to construct an event log table for export collection financing of export bill discounting, thus obtaining the candidate standard list table CANDIDATE_STD_LISTS in Table 2:

[0097]

[0098] The meanings of each item in Table 2 are as follows: TASK_ID is the task number.

[0099] SELECTED_STDS is a list of criteria selected by the user, including all criteria selected by the user.

[0100] CANDIDATE_STDS is a list of candidate standards, representing a list of candidate standards whose source tables have not yet been determined. When the source table of a standard in this item is determined, the standard is removed from this item. When the candidate standards in this column are cleared to empty, the selection is considered complete.

[0101] At the same time, a blank Table 3 is obtained, namely the standard selection result list table SELECTION_STD_RESULTS:

[0102] TASK_ID SELECTION_STD BELGTO_SCHEMA TABLE_CODE COL_CODE 1 CORE_CUST_NO [NULL] [NULL] [NULL] 1 MONEXCH_OBJ [NULL] [NULL] [NULL] 1 COLLCT_DATE [NULL] [NULL] [NULL] 1 COLLCT_CURRCD [NULL] [NULL] [NULL] 1 COLLCT_AMT [NULL] [NULL] [NULL]

[0103] The meanings of each item in Table 3 are as follows:

[0104] TASK_ID is the task number, which corresponds to the task number in Table 2.

[0105] SELECTION_STD is the standard selected by the user.

[0106] TABLE_CODE is the table name, which corresponds to the table name in Table 1, that is, the table corresponding to the determined selection criteria;

[0107] The following are configurable options:

[0108] BELGTO_SCHEMA is the schema to which the table belongs, corresponding to the schema to which the table belongs in Table 1.

[0109] COL_CODE is the field name, corresponding to the field name in Table 1.

[0110] Next, determine whether each selected standard specifies the use of a standard from a table in the data asset list. If so, store the standard and its corresponding table information in the standard selection result list and remove the standard from the candidate standard list.

[0111] If not, calculate the table corresponding to the selected standard in the data asset list, store the table information obtained from the calculation results in the standard selection result list, and remove the standard from the candidate standard list;

[0112] Meanwhile, during the process of calculating the corresponding table in the data asset inventory for the selected standard:

[0113] First, determine if any existing tables in the standard selection result list contain information about the standard. If so, the standard also uses an existing table in the standard selection result list, and the standard and its corresponding table information are stored in the standard selection result list. Otherwise:

[0114] Iterate through all table information in the data asset list and select the table that includes the most criteria from the candidate criteria list.

[0115] Repeat this step until the list of candidate standards is cleared.

[0116] Specifically, in this embodiment, the user specifies that CORE_CUST_NO (core customer number) uses a field from the SDM_INTS_CUST table for domestic customer information. This removes the CORE_CUST_NO (core customer number) standard from the CANDIDATE_STDS table of the CANDIDATE_STD_LISTS table in Table 2, and stores this standard and its corresponding table information (SDM_INTS_CUST table) into the SELECTION_STD_RESULTS table of Table 3, thus obtaining the updated CANDIDATE_STD_LISTS table in Table 2 and Table 3.

[0117] Table 2 Candidate Criteria List (CANDIDATE_STD_LISTS)

[0118]

[0119] Table 3. List of Standard Selection Results (SELECTION_STD_RESULTS)

[0120] TASK_ID SELECTION_STD BELGTO_SCHEMA TABLE_CODE COL_CODE 1 CORE_CUST_NO SDM SDM_INTS_CUST HX_CUST_ID 1 MONEXCH_OBJ [NULL] [NULL] [NULL] 1 COLLCT_DATE [NULL] [NULL] [NULL] 1 COLLCT_CURRCD [NULL] [NULL] [NULL] 1 COLLCT_AMT [NULL] [NULL] [NULL]

[0121] Subsequently, it was discovered that the MONEXCH_OBJ (Foreign Exchange Settlement Object) standard also exists in the International Settlement Customer Information (SDM_INTS_CUST) table. Therefore, the MONEXCH_OBJ (Foreign Exchange Settlement Object) standard was removed from the CANDIDATE_STDS table of the Candidate Standard List table CANDIDATE_STD_LISTS in Table 2, and the standard and its corresponding table information: the International Settlement Customer Information (SDM_INTS_CUST) table, were stored in the SELECTION_STD_RESULTS table of the Standard Selection Result List in Table 3, thus obtaining the updated Candidate Standard List table CANDIDATE_STD_LISTS in Table 2 and Table 3:

[0122] Table 2 Candidate Criteria List (CANDIDATE_STD_LISTS)

[0123]

[0124]

[0125] Table 3. List of Standard Selection Results (SELECTION_STD_RESULTS)

[0126] TASK_ID SELECTION_STD BELGTO_SCHEMA TABLE_CODE COL_CODE 1 CORE_CUST_NO SDM SDM_INTS_CUST HX_CUST_ID 1 MONEXCH_OBJ SDM SDM_INTS_CUST HX_CUST_ID 1 COLLCT_DATE [NULL] [NULL] [NULL] 1 COLLCT_CURRCD [NULL] [NULL] [NULL] 1 COLLCT_AMT [NULL] [NULL] [NULL]

[0127] Since the existing tables in the SELECTION_STD_RESULTS table of Table 3 do not contain the remaining COLLCT_DATE, COLLCT_CURRCD, and COLLCT_AMT standards, all table information in the data asset list is traversed to select the table that contains the most standards from the candidate standard list.

[0128] In this embodiment, the COLLCT_DATE, COLLCT_CURRCD, and COLLCT_AMT standards can all be found in the SDM_INTS_EXCLMASTER (Export Collection Financing Export Bill Discount) table. Therefore, the information from this table is stored in Table 3, and these three standards are deleted from the candidate standard list in Table 2.

[0129] After selection, the data in Tables 2 and 3 are as follows:

[0130] Table 2 Candidate Criteria List (CANDIDATE_STD_LISTS)

[0131]

[0132] Table 3. List of Standard Selection Results (SELECTION_STD_RESULTS)

[0133]

[0134] It should be noted that when selecting a table here, if multiple tables contain the same number of standards from the candidate standard list, the table with the highest weight can be selected based on the number of times it is used.

[0135] Alternatively, based on the graph database, the shortest association path length between each table and the existing tables in the standard selection result list can be calculated, and the table with the smallest path length value can be selected (refer to the method in patent CN115328883A).

[0136] Alternatively, you can choose a table randomly.

[0137] At this point, the candidate standard list in Table 2, CANDIDATE_STD_LISTS, has been cleared, and the determination of the standard selection result list has been completed.

[0138] S400 selects a list of results based on defined criteria, displays all selected tables and fields, and performs data development.

[0139] Display all selected tables and fields on the display device for users to develop data; users can select to add or remove fields from the available fields and the corresponding field processing rules, and fill in association rules, filtering conditions, grouping rules, group filtering rules, etc.

[0140] For example, fields can be processed by filling in mapping rules;

[0141] By filling in the join rules between tables, that is, the join conditions in the query statement, the join conditions between multiple tables are supplemented;

[0142] The data results can be filtered by filling in the filter conditions, i.e., the WHERE clause in the query;

[0143] Data is grouped by specifying grouping rules, i.e., the group condition in the query statement;

[0144] The results of grouped data can be filtered by filling in the grouping filtering rules, i.e., the having condition in the query statement.

[0145] Here, during the development process, users are also allowed to replace the selected table as needed, or to re-specify tables for more criteria and rerun the above process.

[0146] Alternatively, a data warehouse modeling method and system proposed in this paper (patent CN115328883A) can be used for data development, which simplifies the data development work.

[0147] After the above steps are completed, the generated processing rules will be recorded in the job information table shown in Table 4, the table relationship information table shown in Table 5, and the field mapping information table shown in Table 6, which are preset in the storage device.

[0148] Table 4. Job Information Table MAP_JOB_INFO (Partial Example)

[0149]

[0150] Table 4, the MAP_JOB_INFO table, stores basic job information, including basic job configuration, scheduling frequency, filtering conditions, grouping conditions, and group filtering conditions. Table 4 provides a partial example of a real-world case, illustrating the filtering conditions, grouping conditions, and group filtering conditions. The meanings of each item in Table 4 are as follows:

[0151] JOB_NAME is the job name.

[0152] DATABASE_ENV is the database schema where the target table resides.

[0153] WHERE_INFO is a filtering rule used to filter the data results.

[0154] GROUP_INFO specifies the grouping rule used to group the results.

[0155] HAVING_INFO is a grouping filtering rule, used in conjunction with grouping rules to filter grouped results.

[0156] COMMENT is a table comment.

[0157] Table 5 shows the relationship information in the MAP_TABLE_RELATED table (partial example).

[0158]

[0159] Table 5, MAP_TABLE_RELATED, stores path association information, i.e., association rules in S400.

[0160] The meanings of each item in Table 5 are as follows:

[0161] JOB_CODE is the job number. Multiple rows with the same job number represent the set of original tables contained in a model table and the relationships between the original tables.

[0162] JOB_NAME is the job name.

[0163] SERIAL_NUM is the source table sequence number.

[0164] DATABASE_ENV is the database schema for the source table.

[0165] JOIN_TABLE is the name of the source table.

[0166] TABLE_ALIAS is the alias for the source table.

[0167] JOIN_TYPE specifies the join type, with four possible join types: 1 represents inner join, 2 represents left outer join, 3 represents right outer join, and 4 represents full outer join.

[0168] JOIN_CONDITION is the association rule.

[0169] Table 6 Field Mapping Information Table MAP_MAPPING_COL_INFO (Partial Example)

[0170]

[0171]

[0172] Table 6, the field mapping information table MAP_MAPPING_COL_INFO, is used to store the fields to be retained and processing rules in S400.

[0173] The meanings of each item in Table 6 are as follows:

[0174] JOB_CODE is the job number.

[0175] JOB_NAME is the job name.

[0176] DATABASE_ENV is the target database schema.

[0177] TAB_NAME is the target table name.

[0178] COL_NO is the field number in the target table.

[0179] COL_NAME is the name of the target field.

[0180] COL_COMMENT is a comment for the target field.

[0181] TYPE_NAME is the type of the target field.

[0182] COL_LENGTH is the length of the target field.

[0183] COL_SCALE represents the precision of the target field.

[0184] IS_PK indicates whether it is a primary key.

[0185] SOURE_TAB_NAME is the source table name.

[0186] SOURE_COL_VALUE is the field name in the source table.

[0187] SOURCE_ALIAS is an alias for a field in the source table.

[0188] SOURE_TYPE specifies the field type of the source table.

[0189] SOURE_LENGTH is the length of the field in the source table.

[0190] SOURE_SCALE specifies the precision of the source table field.

[0191] COL_MAPPING represents the field mapping relationship.

[0192] Finally, based on the information in the above tables, job configurations are generated, and the corresponding data warehouse model is constructed.

[0193] This solution retrieves the required assets through data asset standards and obtains the selected tables by calculating the minimum number of table combinations. This changes the traditional process of selecting tables first and then searching for fields, reducing the time spent searching for required fields, reducing the number of tables in the data development process, reducing the complexity of data development, and improving data development efficiency to a certain extent.

[0194] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

[0195] It should be understood that, in order to simplify the present invention and help those skilled in the art understand its various aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes described in a single embodiment or with reference to a single figure. However, the present invention should not be construed as including all features in the exemplary embodiments as essential technical features of the claims of this patent.

[0196] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0197] It should be understood that the modules, units, components, etc., included in the device of one embodiment of the present invention can be adaptively changed to be placed in a device different from that embodiment. Different modules, units, or components included in the device of the embodiment can be combined into a single module, unit, or component, or they can be divided into multiple sub-modules, sub-units, or sub-components.

[0198] The modules, units, or components in the embodiments of the present invention can be implemented in hardware, in software running on one or more processors, or in a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement the embodiments of the present invention. The present invention can also be implemented on a computer program product or computer-readable medium for performing some or all of the methods described herein.

Claims

1. A data development method based on data assets, characterized in that, Obtain each data table from the data asset system and store the fields and corresponding standard names of the data tables in the data asset list; Calculate the list of data standard names from the data asset inventory table; Select data standard names according to requirements and determine the list of standard selection results; Select a list of results based on the established criteria, display all selected tables and fields, and then proceed with data development. The data asset list is grouped into a data standard list by using the subject or system code of the fields in the data asset list as the first-level directory and the standard code of the fields in the data asset list as the second-level directory. Based on the required field information, determine the list of candidate criteria; Based on the requirements and the list of candidate standards, determine the list of selected standards. Determine whether the selected standard specifies the use of a standard from a table in the data asset inventory. If so, store the standard and its corresponding table information in the standard selection result list and remove the standard from the candidate standard list. If not, calculate the table corresponding to the selected standard in the data asset list, store the table information obtained from the calculation results in the standard selection result list, and remove the standard from the candidate standard list; Repeat this step until the list of candidate standards is cleared; In the process of calculating the corresponding table in the data asset inventory for the selected criteria: First, determine if any existing tables in the standard selection result list contain information about the standard. If so, the standard also uses an existing table in the standard selection result list, and the standard and its corresponding table information are stored in the standard selection result list. Otherwise: Iterate through all table information in the data asset list and select the table that includes the most criteria from the candidate criteria list.

2. The data development method based on data assets as described in claim 1, characterized in that, The field information includes the subject or system encoding information of the field and the standard encoding information of the field; Select the appropriate standard from the data standard list based on the subject or system encoding information of the required field and the standard encoding information of the field, and save the selected standard into the candidate standard list.

3. The data development method based on data assets as described in claim 1, characterized in that, When iterating through all tables in the data asset inventory, if multiple tables simultaneously include the same number of criteria from the candidate criterion list, then: The table with the highest weight is selected based on the number of times it is used.

4. The data development method based on data assets as described in claim 1, characterized in that, When iterating through all tables in the data asset inventory, if multiple tables simultaneously include the same number of criteria from the candidate criterion list, then: Based on the graph database, calculate the shortest association path length between each table and the existing tables in the standard selection result list, and select the table with the smallest path length value.

5. The data development method based on data assets as described in claim 1, characterized in that, Based on the established criteria, a list of results is selected, displaying all selected tables and fields, and data development is performed, including: Inter-table join rules, data filtering rules, data grouping rules, and data grouping filtering rules.

6. The data development method based on data assets as described in claim 5, characterized in that, Once the data development is complete, the resulting processing rules are transformed into job information, table relationship information, and field mapping information. Based on this information, job configurations are generated, and the corresponding data warehouse model is constructed.

7. The data development method based on data assets as described in any one of claims 1-6, characterized in that, The calculation results for each step are stored in tabular form.

8. A data development system based on data assets, characterized in that, include: processor; database; And memory, in which programs are stored. When the processor executes the program, it implements the data development method based on data assets as described in any one of claims 1-6.

Images