Metadata generation system and metadata generation method

Abstract

The present application provides a metadata generation system and a metadata generation method. The metadata generation system comprises a storage device and a processor. The storage device stores a data collection module and a data analysis module. The processor is coupled to the storage device. The processor executes the data collection module to collect original data and obtain first data. The processor executes the data analysis module to analyze the first data and generate second data. The data analysis module compares the second data with samples to generate metadata.

Description

TECHNICAL FIELD

[0001] The present application relates to a data generation technique, in particular, a metadata generation system and a metadata generation method. BACKGROUND

[0002] In data arrangement, the existing metadata establishment method needs to be observed by personnel after data, and the given person's knowledge and experience are determined. In other words, the traditional metadata generation process must be checked and viewed by personnel to complete the setting. However, when data checking is performed in an enterprise, there are thousands of tables and fields, and the data may reach hundreds of millions. For large data, the data checking personnel cannot check all the data to develop specifications. The traditional way is to use the way of sampling part of the data, and to develop metadata according to the sampling results, so errors are prone to occur. SUMMARY

[0003] The present application is a metadata generation system and a metadata generation method that can automatically generate metadata.

[0004] According to an embodiment of the present application, the metadata generation system of the present application includes a storage device and a processor. The storage device stores a data acquisition module and a data analysis module. The processor is coupled to the storage device. The processor executes the data acquisition module to acquire raw data and obtain first data. The processor executes the data analysis module to analyze the first data and generate second data. The data analysis module compares the second data with samples to generate metadata.

[0005] According to an embodiment of the present application, the metadata generation method of the present application includes the following steps: acquiring raw data by a data acquisition module and obtaining first data; analyzing the first data by a data analysis module and generating second data; and comparing the second data with samples by the data analysis module to generate metadata.

[0006] Based on the above, the metadata generation system and the metadata generation method of the present application can automatically analyze raw data and compare samples (pattern comparison) to automatically classify and effectively generate corresponding metadata.

[0007] In order to make the above features and advantages of the present application more obvious and easy to understand, the following embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a schematic diagram of a metadata generation system of an embodiment of the present application;

[0009] Figure 2 is a flowchart of a metadata generation method of an embodiment of the present application;

[0010] Figure 3 is a schematic diagram of generating metadata of an embodiment of the present application.

[0011] Reference Signs List

[0012] 100: metadata generation system;

[0013] 101: original data;

[0014] 102: first data;

[0015] 103: second data;

[0016] 104: metadata;

[0017] 110: processor;

[0018] 120: storage device;

[0019] 130: data collection module;

[0020] 131: data collector;

[0021] 132: field collector;

[0022] 140: data analysis module;

[0023] 141: style analysis module;

[0024] 142: style comparison module;

[0025] 143: data generation module;

[0026] 150: data adjustment module;

[0027] 200: style database;

[0028] S210-S230: steps. DETAILED DESCRIPTION

[0029] Reference will now be made in detail to exemplary embodiments of the present application, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.

[0030] Figure 1 is a schematic diagram of a metadata generation system of an embodiment of the present application. Reference is made to Figure 1The metadata generation system 100 comprises a processor 110 and a storage device 120. The processor 110 is coupled to the storage device 120. The storage device 120 can store a data collection module 130, a data analysis module 140 and a data adjustment module 150. The processor 110 can also be coupled to an external or internal style database 200. The processor 110 can access the storage device 120 to execute the data collection module 130, the data analysis module 140 and the data adjustment module 150. In this embodiment, a person can input or the processor 110 can automatically read a pre-stored raw data, and the data collection module 130 can collect the raw data. The data analysis module 140 can analyze the data collection result of the raw data and compare the data with the style database 200 to automatically generate metadata. Furthermore, the processor 110 can output the metadata to a human-machine interaction interface, and the human-machine interaction interface can provide corresponding adjustment instructions to the processor 110 according to the adjustment result of the adjusted metadata, so that the data adjustment module 150 can adjust the metadata according to the adjustment instructions to generate adjusted metadata. In an embodiment, the adjustment instructions can also be automatically generated or pre-set by the processor 110.

[0031] In the present embodiment, the processor 110 can comprise, for example, a central processing unit (CPU), or other programmable general purpose or special purpose microprocessors (Microprocessor), digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuits, ASIC), programmable logic devices (Programmable Logic Device, PLD), other similar processing circuitry, or a combination of these devices. The storage device 120 can comprise a memory and / or a database, wherein the memory can be, for example, a non-volatile memory (Non-Volatile Memory, NVM). The storage device 120 can store relevant programs, modules, systems or algorithms for implementing embodiments of the present application for the processor 110 to access and execute to implement the relevant functions and operations described in embodiments of the present application. In the present embodiment, the data acquisition module 130, the data analysis module 140 and the data adjustment module 150 can be implemented, for example, in a program language such as JSON (JavaScript Object Notation), Extensible Markup Language (Extensible Markup Language, XML) or YAML, but the present application is not limited thereto. In the present embodiment, the metadata generation system 100 can be specifically implemented in a personal computer (Personal Computer, PC), an on-premises server (Server) or a cloud server, and the present application is not limited thereto. In an embodiment, the metadata generation system 100 can also be integrated in an enterprise resource planning (Enterprise Resource Planning, ERP) system to provide a service of automatically generating metadata.

[0032] Figure 2 is a flowchart of a metadata generation method of an embodiment of the present application. Referring to Figure 1 and Figure 2The metadata generation system 100 of the present embodiment can automatically generate metadata by performing the following steps S210-S230. In the present embodiment, the processor 110 can acquire raw data and execute the data acquisition module 130, the data analysis module 140, and the data adjustment module 150. In step S210, the data acquisition module 130 can perform data acquisition on the raw data and acquire first data. In the present embodiment, the raw data can be, for example, in the form of a table. The processor 110 can acquire a portion of the raw data according to an acquisition range to acquire the first data, wherein the acquisition range can be pre-set by the system or selected by a person. In step S220, the data analysis module 140 can analyze the first data and generate second data. In the present embodiment, the data analysis module 140 can analyze multiple rows or multiple columns of the first data respectively to generate multiple field information of the second data and multiple style information, multiple type information, and multiple length information corresponding to the multiple field information. In step S230, the data analysis module 140 can perform sample comparison (style comparison) on the second data to generate metadata. In the present embodiment, the style database 200 can be used to store multiple style reference information corresponding to different data specifications and multiple form reference information, multiple length reference information, multiple type definition reference information, and multiple suggested name reference information corresponding to the multiple style reference information. The data analysis module 140 can read the style database 200 to compare the second data according to the aforementioned reference data to automatically determine the information content in the second data that complies with the data specifications and automatically compile to generate metadata. In this way, the metadata generation system 100 of the present embodiment can efficiently automatically generate metadata.

[0033] Figure 3 is a schematic diagram of generating metadata according to an embodiment of the present disclosure. Referring to Figure 1 and Figure 3 The present embodiment further specifically illustrates the flow of generating metadata. In the present embodiment, the data acquisition module 130 can include a data collector 131 and a field collector 132. The data collector 131 can acquire first data 102 from raw data 101, and the field collector 132 can acquire field information for generating metadata from the raw data 101. For example, the data collector 131 can acquire a portion of the raw data 101 according to an acquisition range to acquire the first data 102. The raw data 101 can be, for example, in the form of a table. The raw data 101 can be as shown in Table 1 below, and the first data 102 can be as shown in Table 2 below. The first data 102 is a portion of the raw data 101.

[0034]

[0035] Table 1

[0036] Ammy Sales AmmyLin@house.com 02-21124654 25 Female Roger Engineer RogerLee@fisic.com 03-45635124 29 Male Jeff Manager JeffWu@kingd.com 02-45648111 37 Male Tony Sales TonyChen@gmail.com 07-06466811 33 Male Ann Presale AnnHu@yahoo.com 04-45681234 40 Female

[0037] Table 2

[0038] In this embodiment, the data analysis module 140 can include a data analysis module 141, a pattern matching module 142, and a data generation module 143. The data analysis module 141 can then acquire the first data 102 and analyze the first data 102 to generate the second data 103. The data analysis module 141 can calculate the data pattern and can determine, for example, that the pattern that is more than 50% or the highest among the same type of data is the main pattern. The second data 103 can include a plurality of field information and a plurality of pattern information, a plurality of type information, and a plurality of length information corresponding to the plurality of field information. For example, the second data 103 can be as shown in Table 3 below, wherein the information content of the pattern information is expressed in the form of the syntax of the regular expression (RE) of the industry standard, but the present application is not limited thereto.

[0039]

[0040] Table 3

[0041] The data matching module 142 can then match the second data 103 by the pattern database 200. The pattern matching module 142 can update the second data 103 according to the matching result. The pattern database 200 can store a plurality of pattern reference information corresponding to different data specifications and a plurality of shape reference information, a plurality of length reference information, a plurality of type definition reference information, and a plurality of suggested name reference information corresponding to the plurality of pattern reference information. For example, the pattern database 200 can store the reference information content as shown in Table 4 below. The pattern matching module 142 can match all possible data types of each pattern in the pattern database 200, and set the data type that matches as a candidate. Also, if there is no match to the pattern, the pattern matching module 142 performs format setting processing according to the data content.

[0042]

[0043] Table 4

[0044] Next, the field collector 132 can obtain the field information for generating the metadata 104 from the raw data 101 and provide to the data generation module 143. The field information can be, for example, the fields "Name", "Title", "Email", "Phone", "Gender", "Age" in Table 1, and used to sequentially update (replace) "Field 1", "Field 2", "Field 3", "Field 4", "Field 5", "Field 6" in the second data 103 of Table 3. The data generation module 143 can set the field information in the plurality of fields in the updated second data according to the field information to generate the metadata 104. For example, the metadata 104 can be as shown in Table 5 below. It is noted that if there is information of the suggested name in the style reference information, the data generation module 143 will use the field information of the suggested name to preferentially replace the object. Also, since there are two information styles and information lengths corresponding to the "Local Phone" (one is the style and length format generated by the style analysis result in the second data 103, and the other is the style and length format pre-stored in the style database 200), the data generation module 143 can first store the data contents of the two different information styles and information lengths in the metadata 104 and mark them. The data generation module 143 can mark this non-compliant data so that the metadata 104 can include the data marked based on the non-compliance.

[0045]

[0046] Table 5

[0047] Next, the data adjustment module 150 can output the metadata 104 to a human computer interaction (HCI) interface (or an application programming interface (API)), and the HCI interface can provide an adjustment instruction to the data adjustment module 150 according to the adjustment result of adjusting the metadata. Therefore, the data adjustment module 150 can adjust the marked data in the metadata according to the adjustment instruction to generate adjusted metadata to generate the adjusted metadata as shown in Table 6 below. For this, a person can, for example, select the style information and length information in the local phone to determine the content of the final metadata. That is, the metadata generation system 100 can automatically generate metadata with corresponding data standards, and can also perform screening according to the person's discretion to generate the final metadata and publish it for use by other applications.

[0048]

[0049] Table 6

[0050] In summary, the metadata generation system and the metadata generation method of the present application can effectively improve the data processing efficiency and generate a consistent metadata generation means. The metadata generation system and the metadata generation method of the present application can greatly improve the efficiency of personnel in data asset inventory. Personnel do not need to perform data inventory in a comprehensive visual manner, and can find reasonable metadata by the data analysis style function of the metadata generation system, and only need to spend less time processing non-standard data. The metadata generation system and the metadata generation method of the present application can reuse the style database to achieve consistent standard processing of common data, such as identity card number, mobile phone number or signal card number, etc.

[0051] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A metadata generation system, characterized in that, include: Storage device, storage data acquisition module, and data analysis module; as well as The processor is coupled to the storage device. The processor executes the data acquisition module to acquire raw data and obtain first data. The processor executes the data analysis module to analyze the first data and generate second data. The data analysis module performs sample comparison on the second data to generate metadata. The data acquisition module includes a data collector and a field collector. The data collector is used to obtain the first data from the raw data, and the field collector is used to obtain field information for generating the metadata from the raw data. The first data is a part of the raw data, and the field information is another part of the raw data. The data analysis module includes a style analysis module, which performs style analysis on the first data and generates the second data. The second data includes multiple field information and multiple style information, multiple type information, and multiple length information corresponding to the multiple field information. The original data is in tabular form, and the processor collects a portion of the original data according to the collection range to obtain the first data. The data analysis module analyzes multiple rows or columns of the first data to generate the multiple field information of the second data, as well as the multiple style information, multiple type information, and multiple length information corresponding to the multiple field information.

2. The metadata generation system according to claim 1, characterized in that, The data analysis module includes a style comparison module and a data generation module. The style comparison module compares the second data with the style database, and the data generation module generates the metadata based on the comparison results and the field information.

3. The metadata generation system according to claim 2, characterized in that, The style comparison module updates the second data based on the comparison result, and the data generation module sets the multiple field information in the updated second data according to the field information to generate the metadata.

4. The metadata generation system according to claim 2, characterized in that, The style database is used to store multiple style reference information corresponding to different data specifications, as well as multiple form reference information, multiple length reference information, multiple type definition reference information, and multiple suggested name reference information corresponding to the multiple style reference information.

5. The metadata generation system according to claim 1, characterized in that, The storage device also stores a data adjustment module, wherein the processor executes the data adjustment module to adjust the metadata according to adjustment instructions.

6. The metadata generation system according to claim 5, characterized in that, The metadata includes data that is marked based on non-compliance with specifications, and the data adjustment module adjusts the marked data in the metadata according to the adjustment instructions to generate adjusted metadata.

7. The metadata generation system according to claim 5, characterized in that, The processor outputs the metadata to the human-computer interaction interface, and the human-computer interaction interface provides the adjustment instruction to the processor based on the adjustment result of the adjusted metadata.

8. A method for generating metadata, characterized in that, include: The raw data is collected through the data acquisition module, and the first data and field information are obtained. The first data is analyzed by the data analysis module, and the second data is generated. as well as The data analysis module performs sample comparison on the second data to generate metadata. The first data is a part of the original data, and the field information is another part of the original data. The second data includes multiple field information, as well as multiple style information, multiple type information, and multiple length information corresponding to the multiple field information. The original data mentioned above is in tabular form. The steps for obtaining the first data include: A portion of the original data is collected according to the collection range to obtain the first data; and The steps for generating the second data include: Analyze multiple rows or columns of the first data to generate the multiple field information of the second data, as well as the multiple style information, multiple type information, and multiple length information corresponding to the multiple field information.

9. The metadata generation method according to claim 8, characterized in that, The steps for generating the second data include: By comparing the second data with the style database; and The metadata is generated based on the comparison results and the field information.

10. The metadata generation method according to claim 9, characterized in that, The steps for generating the second data further include: Update the second data according to the comparison results; and The steps for generating the metadata include: Based on the field information, the updated field information in the second data is set to generate the metadata.

11. The metadata generation method according to claim 9, characterized in that, The style database is used to store multiple style reference information corresponding to different data specifications, as well as multiple form reference information, multiple length reference information, multiple type definition reference information, and multiple suggested name reference information corresponding to the multiple style reference information.

12. The metadata generation method according to claim 8, characterized in that, Also includes: Adjust the metadata according to the adjustment instructions.

13. The metadata generation method according to claim 12, characterized in that, The metadata includes data that has been flagged for not conforming to the specifications. The steps for adjusting the metadata include: The tagged data in the metadata is adjusted according to the adjustment instructions to generate adjusted metadata.

14. The metadata generation method according to claim 12, characterized in that, Also includes: Output the metadata to the human-computer interaction interface; as well as The adjustment instruction is generated based on the adjustment result of the metadata through the human-computer interaction interface.

Images