Metadata acquisition method based on AIGC

Abstract

The invention discloses a metadata acquisition method based on AIGC, and relates to the technical field of data acquisition. Comprising the steps of 1, pre-registering data sources: pre-registering and managing multiple data sources, 2, identifying and formulating an acquisition task, 3, generating task configuration and executing the task, 4, verifying a source library and generating a report: after the acquisition task is completed, performing source library verification on acquired metadata, and after the verification is passed, generating a detailed acquisition report, and step 5, carrying out manual reexamination and exception handling: carrying out manual reexamination on correct data in the acquisition report for online use, carrying out acquisition task correction on abnormal data occurring in the acquisition report, and carrying out acquisition, reporting and reexamination processes again until the data is accurate and successfully online.

Description

Technical Field

[0001] The invention discloses a metadata collection method based on AIGC, and relates to the technical field of data collection. Background Art

[0002] In the current field of software engineering and data governance, metadata collection is a key link to ensure data quality and promote effective data management and utilization. With the rapid development of artificial intelligence technology, especially the rise of large language models, such as AIGC, it is possible to automatically and intelligently handle complex tasks. Its metadata collection is even more complicated. However, the current metadata collection methods often rely on manual operations, which are not only inefficient but also prone to errors, especially when facing large-scale and diverse data sources. This problem is particularly prominent. Summary of the invention

[0003] In view of the problems of the prior art, the present invention provides a metadata collection method based on AIGC, which integrates advanced artificial intelligence big model technology to realize the comprehensive automation of data source reading, collection task formulation, and metadata collection process, thereby significantly improving the efficiency and quality of metadata collection in the data governance process. This method aims to reduce manual intervention, improve collection accuracy, accelerate the data governance cycle, and provide more efficient and reliable data support for enterprises and organizations.

[0004] The specific scheme proposed by the present invention is:

[0005] The present invention provides a metadata collection method based on AIGC, comprising:

[0006] Step 1: Pre-register data sources: Pre-register and manage multiple data sources.

[0007] Step 2: Identify and formulate collection tasks: Based on the description text of the collection task provided, the AIGC platform automatically identifies and associates the corresponding data source.

[0008] Step 3: Generate task configuration and execute the task: According to the selected collection tool, automatically generate the adapted task configuration, execute the collection task, collect metadata from the associated data source, and perform necessary processing and storage.

[0009] Step 4: Source database verification and report generation: After the collection task is completed, the source database of the collected metadata is verified. After the verification is passed, a detailed collection report is generated for subsequent manual review and use.

[0010] Step 5: Manual review and exception handling: The correct data in the collection report will be manually reviewed for online use. For abnormal data in the collection report, correct the collection task and re-execute the collection, reporting and review process until the data is accurate and successfully put online.

[0011] Furthermore, when pre-registering data sources in step 1 of the AIGC-based metadata collection method, relational databases, non-relational databases, and file systems are covered to facilitate identification and location of required data sources in subsequent tasks.

[0012] Furthermore, in step 2 of the metadata collection method based on AIGC, the identified data source is analyzed according to the collection task. If the data source is a database, the database type is automatically determined, and the most appropriate collection tool is selected according to the database type to ensure the accuracy and efficiency of the collection.

[0013] Furthermore, in step 3 of the metadata collection method based on AIGC, an adapted task configuration is automatically generated, including configuring collection rules, frequency, and format parameters.

[0014] The present invention also provides a metadata collection device based on AIGC, comprising a registration module, a task collection formulation module, a configuration execution module, a verification module and a review module.

[0015] Registration module pre-registers data sources: pre-registers and manages multiple data sources,

[0016] The task collection formulation module identifies and formulates collection tasks: based on the description text of the provided collection task, it automatically identifies and associates it with the corresponding data source based on the AIGC platform.

[0017] The configuration execution module generates task configuration and executes the task: according to the selected collection tool, it automatically generates the adapted task configuration, executes the collection task, collects metadata from the associated data source, and performs necessary processing and storage.

[0018] Verification module Source database verification and report generation: After the collection task is completed, the collected metadata is verified by the source database. After the verification is passed, a detailed collection report is generated for subsequent manual review and use.

[0019] The review module performs manual review and exception handling: the correct data in the collection report is manually reviewed before being put online. For abnormal data in the collection report, the collection task is corrected and the collection, reporting and review processes are re-executed until the data is accurate and successfully put online.

[0020] Furthermore, when the registration module of the metadata collection device based on AIGC pre-registers the data source, it covers relational databases, non-relational databases, and file systems, so as to facilitate the identification and positioning of the required data source in subsequent tasks.

[0021] Furthermore, the task collection formulation module of the metadata collection device based on AIGC analyzes the identified data source according to the collection task. If the data source is a database, it automatically determines the database type and selects the most appropriate collection tool according to the database type to ensure the accuracy and efficiency of the collection.

[0022] Furthermore, the configuration execution module of the metadata collection device based on AIGC automatically generates an adaptive task configuration, including configuration collection rules, frequency, and format parameters.

[0023] The benefits of the present invention are:

[0024] Significantly improve efficiency: Through automated processes, manual operations are greatly reduced and the efficiency of metadata collection is improved.

[0025] Improve collection quality: Use AIGC technology to accurately identify data sources and database types, select the best collection tools, and improve collection quality.

[0026] Reduce complexity and cost: Provide a one-stop solution to simplify the metadata collection process and reduce technical barriers and maintenance costs.

[0027] Enhanced flexibility and scalability: Supports multiple data sources and database types, and is easy to expand to new data sources to meet changing needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0029] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0030] Example 1

[0031] The present invention provides a metadata collection method based on AIGC, comprising:

[0032] Step 1: Pre-register data sources: Pre-register and manage multiple data sources. When pre-registering data sources, include relational databases, non-relational databases, file systems, etc., to facilitate identification and location of required data sources in subsequent tasks.

[0033] Step 2: Identify and formulate collection tasks: Based on the description text of the collection task provided, the AIGC platform automatically identifies and associates the corresponding data source, greatly reducing the need for manual intervention. The identified data source is analyzed according to the collection task. If the data source is a database, the database type is automatically determined, and the most appropriate collection tool is selected based on the database type to ensure the accuracy and efficiency of the collection.

[0034] Step 3: Generate task configuration and execute tasks: Automatically generate adaptive task configurations based on the selected collection tool, including configuring collection rules, frequency, format parameters, executing collection tasks, collecting metadata from associated data sources, and performing necessary processing and storage.

[0035] Step 4: Source database verification and report generation: After the collection task is completed, the source database of the collected metadata is verified. After the verification is passed, a detailed collection report is generated for subsequent manual review and use.

[0036] Step 5: Manual review and exception handling: The correct data in the collection report will be manually reviewed for online use. For abnormal data in the collection report, correct the collection task and re-execute the collection, reporting and review process until the data is accurate and successfully put online.

[0037] The method of the present invention realizes the comprehensive automation and intelligence of the data source reading, collection task formulation, and metadata collection process of the artificial intelligence large model AIGC through the steps of pre-registered data source management, intelligent task identification and formulation, adaptive task generation and execution, source library verification and report generation, as well as manual review and exception handling, and significantly improves the efficiency and quality of metadata collection in the data governance process.

[0038] Example 2

[0039] The present invention also provides a metadata collection device based on AIGC, comprising a registration module, a task collection formulation module, a configuration execution module, a verification module and a review module.

[0040] Registration module pre-registers data sources: pre-registers and manages multiple data sources,

[0041] The task collection formulation module identifies and formulates collection tasks: based on the description text of the provided collection task, it automatically identifies and associates it with the corresponding data source based on the AIGC platform.

[0042] The configuration execution module generates task configuration and executes the task: according to the selected collection tool, it automatically generates the adapted task configuration, executes the collection task, collects metadata from the associated data source, and performs necessary processing and storage.

[0043] Verification module Source database verification and report generation: After the collection task is completed, the collected metadata is verified by the source database. After the verification is passed, a detailed collection report is generated for subsequent manual review and use.

[0044] The review module performs manual review and exception handling: the correct data in the collection report is manually reviewed before being put online. For abnormal data in the collection report, the collection task is corrected and the collection, reporting and review processes are re-executed until the data is accurate and successfully put online.

[0045] As the information interaction and execution process between the modules of the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0046] Likewise, the benefits of the device of the present invention are:

[0047] Significantly improve efficiency: Through automated processes, manual operations are greatly reduced and the efficiency of metadata collection is improved.

[0048] Improve collection quality: Use AIGC technology to accurately identify data sources and database types, select the best collection tools, and improve collection quality.

[0049] Reduce complexity and cost: Provide a one-stop solution to simplify the metadata collection process and reduce technical barriers and maintenance costs.

[0050] Enhanced flexibility and scalability: Supports multiple data sources and database types, and is easy to expand to new data sources to meet changing needs.

[0051] It should be noted that not all steps and modules in the above-mentioned processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0052] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A metadata collection method based on AIGC, characterized by: include: Step 1: Pre-register data sources: Pre-register and manage multiple data sources. Step 2: Identify and formulate collection tasks: Based on the description text of the collection task provided, the AIGC platform automatically identifies and associates the corresponding data source. Step 3: Generate task configuration and execute the task: According to the selected collection tool, automatically generate the adapted task configuration, execute the collection task, collect metadata from the associated data source, and perform necessary processing and storage. Step 4: Source database verification and report generation: After the collection task is completed, the collected metadata is verified in the source database. After the verification is passed, a detailed collection report is generated for subsequent manual review and use. Step 5: Perform manual review and exception handling: The correct data in the collection report will be manually reviewed for online use. For abnormal data in the collection report, correct the collection task and re-execute the collection, reporting and review process until the data is accurate and successfully put online.

2. The metadata collection method based on AIGC according to claim 1, characterized in that When pre-registering data sources in step 1, relational databases, non-relational databases, and file systems are covered to facilitate identification and location of required data sources in subsequent tasks.

3. The metadata collection method based on AIGC according to claim 1, characterized in that In step 2, the identified data source is analyzed according to the collection task. If the data source is a database, the database type is automatically determined, and the most appropriate collection tool is selected according to the database type to ensure the accuracy and efficiency of the collection.

4. The metadata collection method based on AIGC according to claim 1, characterized in that In step 3, the adapted task configuration is automatically generated, including the configuration of collection rules, frequency, and format parameters.

5. A metadata collection device based on AIGC, characterized in that It includes registration module, task collection and formulation module, configuration execution module, verification module and review module. Registration module pre-registers data sources: pre-registers and manages multiple data sources, The task collection formulation module identifies and formulates collection tasks: based on the description text of the provided collection task, it automatically identifies and associates it with the corresponding data source based on the AIGC platform. The configuration execution module generates task configuration and executes the task: according to the selected collection tool, it automatically generates the adapted task configuration, executes the collection task, collects metadata from the associated data source, and performs necessary processing and storage. Verification module Source database verification and report generation: After the collection task is completed, the collected metadata is verified by the source database. After the verification is passed, a detailed collection report is generated for subsequent manual review and use. The review module performs manual review and exception handling: the correct data in the collection report is manually reviewed before being put online. For abnormal data in the collection report, the collection task is corrected and the collection, reporting and review processes are re-executed until the data is accurate and successfully put online.

6. The metadata collection device based on AIGC according to claim 5, characterized in that When the registration module pre-registers data sources, it covers relational databases, non-relational databases, and file systems to facilitate identification and location of required data sources in subsequent tasks.

7. The metadata collection device based on AIGC according to claim 5, characterized in that The task collection formulation module analyzes the identified data sources according to the collection tasks. If the data source is a database, it automatically determines the database type and selects the most appropriate collection tool based on the database type to ensure the accuracy and efficiency of the collection.

8. The metadata collection device based on AIGC according to claim 5, characterized in that The configuration execution module automatically generates the adapted task configuration, including the configuration of collection rules, frequency, and format parameters.

Images