Data asset management method and system based on development process

Abstract

The invention provides a data asset management method and system based on a development process. The data asset management method based on the development process comprises the following steps: configuring table structure information and metadata; the server receives the table structure information and performs data analysis to obtain DDL script information; according to the DDL script information, generating a finally executed DDL script; executing the finally executed DDL script; judging an execution result of the finally executed DDL script, and recording an execution result log; the metadata information is stored, and management and storage of the data assets are completed. According to the technical scheme, through metadata identification, storage and complex SQL analysis conversion, full-process automation and configurability of data assets are achieved, the research and development cost is reduced, and the reuse rate is increased.

Description

Background Art

[0002] With the deepening of Digital Sunshine, data analysis, data development, and data ETL work based on big data AI big model applications are frequent, the sample data management of data assets is chaotic, and sample data cannot be obtained quickly and efficiently. At the same time, data assets are facing pain points such as missing business information, high search and development costs, low reuse rate, redundant asset information, and lack of version management.

[0003] This is the pain point: difficulty in finding development, redundant asset information, and lack of version management.

[0004] In order to better serve the business with data, it is urgent to find a fast and efficient method. Summary of the invention

[0005] The present application provides a data asset management method and system based on development process, which is used to achieve fast, efficient and complex database DDL processing and data asset management.

[0006] In a first aspect, a data asset management method based on a development process is provided, comprising the following steps:

[0007] Configure table structure information and metadata;

[0008] The server receives the table structure information, performs data analysis, and obtains DDL script information;

[0009] Generate a DDL script to be finally executed according to the DDL script information;

[0010] Execute the DDL script that is finally executed;

[0011] Determine the execution result of the DDL script that is finally executed, and record the execution result log;

[0012] Store metadata information to complete the management and storage of data assets.

[0013] In the above technical solution, the table structure information and metadata are configured; the server receives the table structure information and performs data analysis to obtain DDL script information; based on the DDL script information, a DDL script to be finally executed is generated; the DDL script to be finally executed is executed; the execution result of the DDL script to be finally executed is determined, and the execution result log is recorded; the metadata information is stored to complete the management and storage of data assets; through metadata identification, storage, and complex SQL analysis and transformation, the full process automation and configurability of data assets are realized, R&D costs are reduced, and the reuse rate is improved.

[0014] In a specific implementation scheme, the table structure information includes the basic structure and business field information of the table, wherein:

[0015] The basic structure of the table includes table name, column name, data type, field length, primary key default value, and remarks.

[0016] In a specific implementation scheme, the server receives the table structure information, performs data parsing, and obtains the DDL script information in the following steps:

[0017] The server receives the table structure information and performs data analysis;

[0018] Determine the database type, concatenate different DDL scripts, and obtain DDL script information.

[0019] In a specific implementation scheme, the steps of generating the DDL script to be finally executed according to the DDL script information are specifically as follows:

[0020] According to the DDL script information, through the page parameter entry, interactive judgment is made with the backend database query, and the final executed DDL script is spliced ​​and generated according to the comparison of the database table structure information and field differences.

[0021] In a specific implementation scheme, the step of executing the DDL script that is finally executed is specifically as follows:

[0022] Configure a database connection identifier, obtain a database connection, and execute the DDL script that is finally executed.

[0023] In a specific implementation scheme, the steps of storing metadata information and completing the management and storage of data assets are as follows:

[0024] The table structure information and the business field information are stored in the data asset management system database to complete the management and storage of data assets.

[0025] In a specific implementation scheme, before executing the task of the finally executed DDL script, it is confirmed that the metadata management system has maintained all metadata information of the data source.

[0026] In a specific implementation scheme, a database connection is obtained through JDBC, and the DDL script that is finally executed is automatically executed.

[0027] In a specific implementation scheme, based on the data analysis generated by the large model, the table structure information is subjected to data parsing.

[0028] Second, a data asset management system based on a development process is provided, including:

[0029] Configuration module, used to configure table structure information and metadata;

[0030] The parsing module is used for receiving the table structure information at the server side, and performing data parsing to obtain DDL script information;

[0031] A DDL script generation module, used to generate a DDL script to be finally executed according to the DDL script information;

[0032] An execution module, used for executing the DDL script finally executed;

[0033] A result module is used to determine the execution result of the DDL script finally executed and record the execution result log;

[0034] The storage module is used to store metadata information and complete the management and storage of data assets.

[0035] In the above technical solution, the table structure information and metadata are configured; the server receives the table structure information and performs data analysis to obtain DDL script information; based on the DDL script information, a DDL script to be finally executed is generated; the DDL script to be finally executed is executed; the execution result of the DDL script to be finally executed is determined, and the execution result log is recorded; the metadata information is stored to complete the management and storage of data assets; through metadata identification, storage, and complex SQL analysis and transformation, the full process automation and configurability of data assets are realized, R&D costs are reduced, and the reuse rate is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a data asset management method based on a development process provided in an embodiment of the present application;

[0037] Figure 2 This is a structural block diagram of the data asset management system based on the development process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The present application is further described in detail below through the accompanying drawings and embodiments. Through these descriptions, the characteristics and advantages of the present application will become clearer and more specific.

[0039] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0040] In addition, the technical features involved in different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0041] To facilitate understanding of the data asset management method and system based on the development process provided by the embodiment of the present application, its application scenario is first explained. The data asset management method and system based on the development process provided by the embodiment of the present application are used to achieve fast, efficient and complex database DDL processing and data asset management. With the deepening of Digital Sunshine, data analysis, data development, and data ETL work based on big data AI big model applications are frequent, and the sample data management of data assets is chaotic, and sample data cannot be obtained quickly and efficiently. At the same time, data assets face pain points such as missing business information, high search and development costs, low reuse rate, redundant asset information, and lack of version management. For this reason, the embodiment of the present application provides a data asset management method and system based on the development process to achieve fast, efficient and complex database DDL processing and data asset management. The following is a detailed description of the embodiment in conjunction with specific drawings.

[0042] refer to Figure 1 and Figure 2 , Figure 1 A flowchart of a data asset management method based on a development process provided in an embodiment of the present application; Figure 2 This is a structural block diagram of the data asset management system based on the development process provided in an embodiment of the present application.

[0043] exist Figure 1 In the embodiment of the present application, a data asset management method based on a development process is provided, comprising the following steps:

[0044] Configure table structure information and metadata;

[0045] The server receives the table structure information, performs data analysis, and obtains DDL script information;

[0046] Generate a DDL script to be finally executed according to the DDL script information;

[0047] Execute the DDL script that is finally executed;

[0048] Determine the execution result of the DDL script that is finally executed, and record the execution result log;

[0049] Store metadata information to complete the management and storage of data assets.

[0050] In the above technical solution, the table structure information and metadata are configured; the server receives the table structure information and performs data analysis to obtain DDL script information; based on the DDL script information, a DDL script to be finally executed is generated; the DDL script to be finally executed is executed; the execution result of the DDL script to be finally executed is determined, and the execution result log is recorded; the metadata information is stored to complete the management and storage of data assets; through metadata identification, storage, and complex SQL analysis and transformation, the full process automation and configurability of data assets are realized, R&D costs are reduced, and the reuse rate is improved.

[0051] In this embodiment, in order to solve the confusion of metadata information management, provide metadata information support for downstream AI large models, and realize the automatic generation of business prompt words, a data asset management method based on the development process is provided:

[0052] Including: users can customize the configuration of related table structure supplementary information, not limited to field name, data type, field notes and other information; through standardized development process configuration, automatically realize the table creation, modification, and deletion process, while retaining operation tracks and data supplementary information, including but not limited to data dimensions, data granularity, update frequency, business meaning; realize fast, efficient and complex database DDL processing and data asset management, that is, realize the data asset management method based on the development process through simple configuration. This method can realize the automatic generation and management of data assets through scientific configuration management on demand. This system is a visualization tool developed to serve data analysis based on large models, and is implemented based on the Sunshine Data Data Management System.

[0053] Specifically, this solution uses a Java Web application developed with Spring Boot. Users can configure the page to parse table structure information for different databases through the data management method of the big data environment, splice different DDL SQLs according to different database types, connect to multiple source databases according to the configuration information, execute DDL scripts, and record log information and data asset information.

[0054] In a specific implementation scheme, the table structure information includes the basic structure and business field information of the table, wherein:

[0055] The basic structure of the table includes table name, column name, data type, field length, primary key default value, and remarks.

[0056] Specifically, configure the table structure information and metadata: Here you can configure the database table structure information. In addition to configuring the basic structure of the table, table name, column name, data type, field length, primary key default value, remarks, etc., you also need to add other business meaning information of the table according to the specific business content to provide data support for downstream large model applications.

[0057] In a specific implementation scheme, the server receives the table structure information, performs data parsing, and obtains the DDL script information in the following steps:

[0058] The server receives the table structure information and performs data analysis;

[0059] Determine the database type, concatenate different DDL scripts, and obtain DDL script information.

[0060] Specifically, the server receives table structure information, performs data analysis, determines six database types, including Hive, Oracle, MySQL, OceanBase, and Clickhouse, based on user configuration, and splices different DDL scripts using the SQL dialect syntax of different databases.

[0061] In a specific implementation scheme, the steps of generating the DDL script to be finally executed according to the DDL script information are specifically as follows:

[0062] According to the DDL script information, through the page parameter entry, interactive judgment is made with the backend database query, and the final executed DDL script is spliced ​​and generated according to the comparison of the database table structure information and field differences.

[0063] Specifically, the DDL script is automatically generated: through the page parameter entry, interactive judgment with the background database query, comparison based on the database table structure information, and splicing the DDL script that needs to be executed in the end according to the field differences; for example: adding a new table, modifying the table name, field name, field type length, etc.

[0064] In a specific implementation scheme, the step of executing the DDL script that is finally executed is specifically as follows:

[0065] Configure a database connection identifier, obtain a database connection, and execute the DDL script that is finally executed.

[0066] Specifically, the target database is connected to execute the DDL script, the database connection identifier is configured by the user, the database connection is obtained through the JDBC method, and the DDL script is automatically executed.

[0067] It should be noted that before executing the DDL script task, you need to confirm that the metadata management system has maintained all metadata information of the data source.

[0068] In a specific implementation scheme, the steps of storing metadata information and completing the management and storage of data assets are as follows:

[0069] The table structure information and the business field information are stored in the data asset management system database to complete the management and storage of data assets.

[0070] Specifically, metadata information is stored, and user configuration table structure information and business field information are stored in the data asset management system database to facilitate downstream data collection and data display.

[0071] In the above technical solution, each step is used in conjunction to achieve the effect of flexible management and storage of data assets. In view of this, through metadata identification, storage, and complex SQL analysis and conversion, the full process automation and configurability of data assets are realized, R&D costs are reduced, and the reuse rate is increased. Support is provided for the development and analysis of data for regulatory submission, opening results, and reports.

[0072] In a specific implementation scheme, before executing the task of the finally executed DDL script, it is confirmed that the metadata management system has maintained all metadata information of the data source.

[0073] In a specific implementation scheme, a database connection is obtained through JDBC, and the DDL script that is finally executed is automatically executed.

[0074] In a specific implementation scheme, based on the data analysis generated by the large model, the table structure information is subjected to data parsing.

[0075] exist Figure 2 In the embodiment of the present application, a data asset management system based on a development process is also provided, including:

[0076] Configuration module, used to configure table structure information and metadata;

[0077] The parsing module is used for receiving the table structure information at the server side, and performing data parsing to obtain DDL script information;

[0078] A DDL script generation module, used to generate a DDL script to be finally executed according to the DDL script information;

[0079] An execution module, used for executing the DDL script finally executed;

[0080] A result module is used to determine the execution result of the DDL script finally executed and record the execution result log;

[0081] The storage module is used to store metadata information and complete the management and storage of data assets.

[0082] In the above technical solution, the table structure information and metadata are configured; the server receives the table structure information and performs data analysis to obtain DDL script information; based on the DDL script information, a DDL script to be finally executed is generated; the DDL script to be finally executed is executed; the execution result of the DDL script to be finally executed is determined, and the execution result log is recorded; the metadata information is stored to complete the management and storage of data assets; through metadata identification, storage, and complex SQL analysis and transformation, the full process automation and configurability of data assets are realized, R&D costs are reduced, and the reuse rate is improved.

[0083] Furthermore, the technical solution of the present application does not require the configuration of complex desensitization rule definitions, does not require the standardization of a certain entity, and the fields for desensitization can be dynamically adjusted to apply different desensitization rules to different data in different result sets, thereby achieving full process configuration and automation.

[0084] Those skilled in the art will appreciate that the present application may be implemented as a system, method or computer program product.

[0085] Therefore, the present disclosure may be specifically implemented in the following forms, namely: it may be completely hardware, it may be completely software (including firmware, resident software, microcode, etc.), or it may be a combination of hardware and software, generally referred to herein as a "circuit", "module" or "system". In addition, in some embodiments, the present application may also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium may contain computer-readable program code.

[0086] Any combination of one or more computer-readable media can be used. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0087] Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art can change, modify, replace and modify the above embodiments within the scope of the present application. On this basis, a variety of replacements and improvements can be made to the present application, all of which fall within the scope of protection of the present application.

Claims

1. A data asset management method based on development process, characterized in that: The following steps are involved: Configure table structure information and metadata; The server receives the table structure information, performs data analysis, and obtains DDL script information; Generate a DDL script to be finally executed according to the DDL script information; Execute the DDL script that is finally executed; Determine the execution result of the DDL script that is finally executed, and record the execution result log; Store metadata information to complete the management and storage of data assets.

2. The data asset management method based on development process according to claim 1 is characterized in that: The table structure information includes the basic structure and business field information of the table, wherein: The basic structure of the table includes table name, column name, data type, field length, primary key default value, and remarks.

3. The data asset management method based on development process according to claim 2 is characterized in that: The server receives the table structure information, performs data parsing, and obtains the DDL script information, specifically: The server receives the table structure information and performs data analysis; Determine the database type, concatenate different DDL scripts, and obtain DDL script information.

4. The data asset management method based on development process according to claim 3 is characterized in that: The steps of generating the DDL script to be finally executed according to the DDL script information are specifically as follows: According to the DDL script information, through the page parameter entry, interactive judgment is made with the backend database query, and the final executed DDL script is spliced ​​and generated according to the comparison of the database table structure information and field differences.

5. The data asset management method based on development process according to claim 4 is characterized in that: The steps of executing the DDL script finally executed are specifically: Configure a database connection identifier, obtain a database connection, and execute the DDL script that is finally executed.

6. The data asset management method based on development process according to claim 5 is characterized in that: The steps to store metadata information and complete the management and storage of data assets are as follows: The table structure information and the business field information are stored in the data asset management system database to complete the management and storage of data assets.

7. The data asset management method based on development process according to claim 6 is characterized in that: Before executing the task of the DDL script to be finally executed, it is confirmed that the metadata management system has maintained all metadata information of the data source.

8. The data asset management method based on development process according to claim 7 is characterized in that: The database connection is obtained through JDBC, and the DDL script to be finally executed is automatically executed.

9. The data asset management method based on development process according to claim 8 is characterized in that: Based on the data analysis generated by the large model, the table structure information is parsed.

10. A data asset management system based on development process, characterized in that: include: Configuration module, used to configure table structure information and metadata; The parsing module is used for receiving the table structure information at the server side, and performing data parsing to obtain DDL script information; A DDL script generation module, used to generate a DDL script to be finally executed according to the DDL script information; An execution module, used for executing the DDL script finally executed; A result module is used to determine the execution result of the DDL script finally executed and record the execution result log; The storage module is used to store metadata information and complete the management and storage of data assets.

Images