Functional metadata system and method for data quality detection

Abstract

The invention provides a function metadata method for data quality detection, which comprises the following steps of: establishing function metadata based on historical data, the function metadata comprising an acceptable mode, data distribution and data definition; and performing at least one of acceptable mode detection, data distribution detection and data definition detection on the data content according to the functional metadata to obtain a detection result. According to the technical scheme, the error occurrence possibility can be reduced, and time and manual checking cost are saved.

Description

Technical Field

[0001] The present invention relates to a computer system and an operating method thereof, and more particularly to a functional metadata system and method for data quality detection. Background Art

[0002] Data quality is crucial for a variety of applications, including infographics, data mining, business analytics, or training machine learning models. Flawed data can lead to statistically inaccurate or inappropriate feature sizes.

[0003] To address the challenges of data quality control, relevant fields have been diligently seeking solutions, but for a long time, no suitable method has been developed. Therefore, how to more efficiently detect data quality is a key research and development topic and a goal that urgently needs improvement in related fields. Summary of the Invention

[0004] The present invention proposes a functional metadata system and method for data quality detection to improve the problems of the prior art.

[0005] In some embodiments of the present invention, a functional metadata system for data quality testing includes a storage device and a processor electrically connected to the storage device. The storage device stores data content. The processor performs a data quality test on the data content to obtain a test result. The data quality test is selected from the group consisting of an acceptable pattern test, a data distribution test, and a data-defined test.

[0006] In some embodiments of the present invention, the data content includes a string content, and the processor detects whether the format of the string content matches an acceptable pattern.

[0007] In some embodiments of the present invention, the data content includes digital content, and the processor performs data distribution based on historical data to define a default reasonable interval for the digital content.

[0008] In some embodiments of the present invention, the data content includes a character string, and the processor determines whether the character string matches a predefined attribute in the data definition.

[0009] In some embodiments of the present invention, the data content includes digital content, and the processor determines whether the digital content is abnormal based on preset common sense standards.

[0010] In some embodiments of the present invention, the functional metadata method for data quality detection proposed by the present invention includes the following steps: establishing functional metadata based on historical data, the functional metadata including acceptable patterns, data distribution and data definitions; based on the functional metadata, performing at least one of acceptable pattern detection, data distribution detection and data definition detection on the data content to obtain a detection result.

[0011] In some embodiments of the present invention, the data content includes digital content, and the step of detecting data distribution of the data content includes: pre-fitting historical data to obtain a probability distribution of the historical data, and analyzing whether the digital content is an outlier based on the data.

[0012] In some embodiments of the present invention, the data content includes string content, and the step of detecting an acceptable pattern of the data content includes: analyzing the string content format through an artificial intelligence model, and then detecting whether the format of the string content matches the acceptable pattern.

[0013] In some embodiments of the present invention, the data content includes string content, and the step of performing data definition detection on the data content includes: when the format of the string content does not match the acceptable pattern, using an artificial intelligence model to determine whether the format of the string content conforms to the standard answer of the data definition, as a basis for confirming the correctness of the string content.

[0014] In some embodiments of the present invention, the data content includes digital content, and the step of performing data definition detection on the data content includes: defining predetermined numerical boundaries of a preset category based on preset common sense standards; if the digital content belongs to the default category, determining whether the digital content exceeds the predetermined numerical boundaries.

[0015] In summary, the technical solution of the present invention has significant advantages and beneficial effects compared to the existing technology. The functional metadata system and method for data quality detection of the present invention overcomes the shortcomings of the previous technology, thereby reducing the possibility of errors and saving time and costs of manual verification.

[0016] The above description will be described in detail below with reference to implementation examples, and a further explanation of the technical solution of the present invention will be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To make the above and other objects, features, advantages and embodiments of the present invention more apparent, the accompanying drawings are described as follows:

[0018] Figure 1 is a block diagram of a functional metadata system for data quality detection according to some embodiments of the present invention; and

[0019] Figure 2is a flowchart of a functional metadata method for data quality detection according to some embodiments of the present invention.

[0020] Component number description

[0021] 100 Functional metadata system for data quality detection

[0022] 110 Storage Device

[0023] 120 processors

[0024] 150 Transmission Device

[0025] 190 Computer Devices

[0026] 200 Functional metadata methods for data quality detection

[0027] Steps S201 and S202 DETAILED DESCRIPTION

[0028] In order to make the description of the present invention more detailed and complete, reference is made to the accompanying drawings and various embodiments described below, in which the same numbers represent the same or similar components. On the other hand, well-known components and steps are not described in the embodiments to avoid unnecessary limitations on the present invention.

[0029] Please refer to Figure 1 The technical aspect of the present invention is a functional metadata system 100 for data quality detection, which can be applied to data governance or widely used in related technical links. In some embodiments of the present invention, data governance refers to a set of practical methods, strategies, and roles related to the collection, management, and utilization of data, the purpose of which is to ensure that data provides as much value as possible within the organization. The functional metadata system 100 for data quality detection of this technical aspect can achieve considerable technological progress and has wide industrial utilization value. The following will be combined with Figure 1 The specific implementation of the functional metadata system 100 for data quality detection is described below.

[0030] It should be understood that various implementations of the functional metadata system 100 for data quality detection can be combined with Figure 1 In the following description, for ease of explanation, many specific details are further set to provide a comprehensive description of one or more embodiments. However, the present technology can be implemented without these specific details. In other examples, in order to effectively describe these embodiments, known structures and devices are shown in block diagram form. The term "for example" is used herein to mean "as an example, instance, or illustration." Any embodiment described herein as "for example" is not necessarily to be interpreted as better or superior to other embodiments.

[0031] Figure 1 FIG is a block diagram of a functional metadata system 100 for data quality detection according to an embodiment of the present invention. Figure 1 As shown, the functional metadata system 100 for data quality detection includes a storage device 110, a processor 120, and a transmission device 150. For example, the storage device 110 can be a hard disk, a flash memory, or other storage media; the processor 120 can be a central processing unit, a controller, or other circuit; and the transmission device 150 can be a transmission interface, a transmission line, a network device, a communication device, or other transmission media.

[0032] Architecturally, the storage device 110 is electrically connected to the processor 120, and the processor 120 is electrically connected to the transmission device 150, so that data can be transmitted between the transmission device 150 and the computer device 190. In practice, for example, the computer device 190 may be a personal computer, a mobile phone, an input / output device, or other electronic device. It should be understood that in the embodiments and the scope of the patent application, the description involving "electrical connection" may generally refer to a component being indirectly electrically coupled to another component through other components, or a component being directly electrically connected to another component without going through other components. For example, the storage device 110 may be a built-in storage device directly electrically connected to the processor 120, or the storage device 110 may be an external storage device indirectly connected to the processor 120 through a line.

[0033] In practice, for example, computer device 190 inputs data content. During use, transmission device 150 receives the data content, and storage device 110 stores the data content. Processor 120 performs a data quality test on the data content to obtain a test result. In some embodiments of the present invention, the data quality test is selected from the group consisting of an acceptable pattern test, a data distribution test, and a data-defined test.

[0034] Regarding the detection of acceptable patterns, in some embodiments of the present invention, the data content includes a string of characters, and the processor 120 detects whether the format of the string of characters matches an acceptable pattern (e.g., a default pattern acceptable to a computer). For example, the string of characters may include 2012Q1, where the format represents the year and quarter in the Gregorian calendar. If the acceptable pattern includes ^(20\d{2}(Q[1-4]))$, the processor 120 determines that the format of the string of characters matches the acceptable pattern.

[0035] Alternatively, in practice, for example, the string content may include 20120101, which represents the year, month, and day in the Christian era. If the acceptable pattern includes YYYYMMDD, the processor 120 determines that the format of the string content matches the acceptable pattern.

[0036] Regarding the aforementioned data distribution detection, in some embodiments of the present invention, the data content includes digital content, and processor 120 performs data distribution based on historical data, thereby defining a default reasonable range for the digital content. In practice, for example, the maximum value among multiple maximum values ​​of multiple historical digital data of the same type serves as the upper limit, and the minimum value among multiple minimum values ​​of multiple historical digital data of the same type serves as the lower limit. The range between the upper and lower limits is defined as the preset reasonable range. If the digital content of the same type falls within the default reasonable range, processor 120 determines that the digital content is normal; conversely, if the digital content of the same type falls outside the default reasonable range, processor 120 determines that the digital content is abnormal.

[0037] Regarding the aforementioned data definition detection, in some embodiments of the present invention, the data content includes a string, and processor 120 determines whether the string matches predefined attributes in the data definition. In practice, for example, steel, cement, and partitions can be categorized as a material group. Although no other data currently exists in the material group, if new data (e.g., the aforementioned string) is present and the string matches the attributes of the predefined material group in the data definition, processor 120 can determine that the string is valid.

[0038] Regarding the detection of the aforementioned data definition, in some embodiments of the present invention, the data content includes digital content, and processor 120 determines whether the digital content is abnormal based on preset common-sense standards. In practice, for example, processor 120 may automatically preset an upper limit for room temperature to 50 degrees Celsius. If the digital content reflecting the room temperature is greater than 50 degrees Celsius, processor 120 determines that the digital content is abnormal.

[0039] To further explain the operation method of the functional metadata system 100 for data quality detection, please also refer to Figures 1 and 2 , Figure 2 FIG. 2 is a flow chart of a functional metadata method 200 for data quality detection according to an embodiment of the present invention. Figure 2 As shown, the functional metadata method 200 for data quality detection includes steps S201 and S202 (it should be understood that the steps mentioned in this embodiment, except for those specifically stated in their order, can be adjusted in order according to actual needs, and can even be executed simultaneously or partially simultaneously).

[0040] The functional metadata method 200 for data quality detection can be in the form of a computer program product on a non-transitory computer-readable recording medium having a plurality of computer-readable instructions embodied therein. Suitable recording media may include any of the following: non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electronically erasable programmable read-only memory (EEPROM); volatile memory, such as static access memory (SRAM), dynamic access memory (SRAM), and double data rate random access memory (DDR-RAM); optical storage devices, such as compact disc read-only memory (CD-ROM) and digital versatile disc read-only memory (DVD-ROM); and magnetic storage devices, such as hard disk drives and floppy disk drives.

[0041] In step S201, processor 120 creates functional metadata based on historical data. The functional metadata includes acceptable patterns, data distribution, and data definitions. In step S202, processor 120 performs at least one of an acceptable pattern check, a data distribution check, and a data definition check on the data content based on the functional metadata to obtain a test result.

[0042] Regarding the detection of the above-mentioned acceptable pattern, in some embodiments of the present invention, the data content includes string content. In step S202, the format of the string content is analyzed by an artificial intelligence model to detect whether the format of the string content matches the acceptable pattern.

[0043] In practice, for example, the artificial intelligence model can be a built-in artificial intelligence model (such as a trained neural network model, a natural language processing model, etc.), and the storage device 110 stores the artificial intelligence model. In step S202, the processor 120 executes the built-in artificial intelligence model to analyze the format of the string content, and then detects whether the format of the string content matches an acceptable pattern.

[0044] Alternatively, the AI ​​model may be an external AI model (e.g., ChatGPT). In step S202, processor 120 analyzes the format of the string content via transmission device 150 using the external AI model to detect whether the format of the string content matches an acceptable pattern. For example, the string content may include F-FLOZEN, which has the format "English word - English word." If the acceptable pattern includes ^[AZ]+-[AZ]+$, processor 120 determines that the format of the string content matches the acceptable pattern.

[0045] Regarding the aforementioned data distribution detection, in some embodiments of the present invention, the data content includes digital content. In step S202, historical data is pre-fitted to obtain a probability distribution of the historical data, which is then used to analyze whether the digital content is an outlier. In practice, for example, processor 120 may execute a Python package named fitter to perform data fitting. Processor 120 sets a threshold value corresponding to a probability below a predetermined standard (e.g., an extremely low probability that can be ruled out in practice) to determine whether the digital content is an outlier.

[0046] Regarding the detection of the data definition, in some embodiments of the present invention, the data content includes a string content. In step S202, it is determined whether the string content is a possible content. In practice, for example, the processor 120 determines whether the string content matches a predefined attribute in the data definition.

[0047] It should be understood that "possible content" is similar to "acceptable mode", but slightly different. The string content of some fields cannot be formatted using the above-mentioned acceptable mode method. For example, there is a field called "destination" in the database stored in the storage device 110. The actual goods may have a new shipping location in a certain country. The processor 120 determines whether the string content (such as the new location) matches the pre-defined attribute (a certain country) in the data definition to automatically determine that the new location is also in a certain country, so there is no need to manually add a standard answer to update.

[0048] Furthermore, regarding the aforementioned data definition detection, in some embodiments of the present invention, the data content includes a string. In step S202, when the format of the string content does not match the aforementioned acceptable pattern, the artificial intelligence model determines whether the string content format conforms to the standard answer of the data definition, thereby confirming the correctness of the string content. For example, if the string content is CZO2, the processor 120 queries the artificial intelligence model: "Is CZO2 the same format as UM08, CZO1, TP05, etc., and is the data definition a factory code?" If the artificial intelligence model responds with a yes, the processor 120 determines that the string content (CZO2) is correct.

[0049] Regarding the detection of the aforementioned data definition, in some embodiments of the present invention, the data content includes digital content. In step S202, a predetermined numerical boundary (e.g., a vehicle speed of 300 km / h) for a predetermined category (e.g., an ordinary car) is defined based on a preset common-sense standard. If the digital content belongs to the default category, a determination is made as to whether the digital content exceeds the predetermined numerical boundary to eliminate unreasonable numerical values, such as Celsius temperature or speed, because these values ​​have common-sense reasonable values.

[0050] Regarding the above-mentioned detection results, in some embodiments of the present invention, the detection results may include four commonly used data quality indicators: missing, redundancy, inconsistency and outliers. In practice, for example, the standard value range of each indicator is between 0 and 1, 0 means no problem, and 1 means that relevant problems are detected in all fields. Compared with traditional data management, through the above-mentioned functional metadata system 100 for data quality detection and the functional metadata method 200 for data quality detection, each indicator has been improved, especially the "inconsistency" and "outlier" categories, the effect is particularly significant.

[0051] In summary, the technical solutions of the present invention offer significant advantages and benefits over existing technologies. The functional metadata system 100 and functional metadata method 200 for data quality testing of the present invention address the shortcomings of prior technologies, thereby reducing the likelihood of errors and saving time and the cost of manual verification.

[0052] Although the present invention has been disclosed above in terms of embodiments, this is not intended to limit the present invention. Anyone skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope of the appended patent applications.

Claims

1. A functional metadata system for data quality detection, characterized in that: Include: a storage device storing data content; as well as A processor is electrically connected to the storage device, and performs a data quality test on the data content to obtain a test result. The data quality test is selected from a group consisting of an acceptable pattern test, a data distribution test, and a data definition test.

2. The functional metadata system for data quality detection according to claim 1, characterized in that: The data content includes a string of content, and the processor detects whether a format of the string of content matches the acceptable pattern.

3. The functional metadata system for data quality detection according to claim 1, characterized in that: The data content includes digital content, and the processor performs the data distribution based on historical data to define a default reasonable range for the digital content.

4. The functional metadata system for data quality detection according to claim 1, characterized in that: The data content includes a character string, and the processor determines whether the character string matches a predefined attribute in the data definition.

5. The functional metadata system for data quality detection according to claim 1, characterized in that: The data content includes digital content, and the processor determines whether the digital content is abnormal based on a preset common sense standard.

6. A functional metadata method for data quality detection, characterized in that: The following steps are involved: Creating functional metadata based on historical data, the functional metadata including an acceptable model, a data distribution, and a data definition; as well as At least one of the detection of the acceptable mode, the detection of the data distribution and the detection of the data definition is performed on a data content according to the functional metadata to obtain a detection result.

7. The functional metadata method for data quality detection according to claim 6, characterized in that: The data content includes digital content, and the step of detecting the data distribution of the data content includes: A data fitting is performed on the historical data in advance to obtain a probability distribution of the historical data, and is used to analyze whether the digital content is an abnormal value.

8. The functional metadata method for data quality detection according to claim 6, characterized in that: The data content includes a character string, and the step of detecting the acceptable mode of the data content includes: A format of the string content is analyzed by an artificial intelligence model to detect whether the format of the string content matches the acceptable pattern.

9. The functional metadata method for data quality detection according to claim 8, characterized in that: The data content includes a character string, and the step of detecting the data definition on the data content includes: When the format of the string content does not match the acceptable pattern, the artificial intelligence model is used to determine whether the format of the string content conforms to the standard answer defined by the data, as a basis for confirming the correctness of the string content.

10. The functional metadata method for data quality detection according to claim 6, characterized in that: The data content includes digital content, and the step of detecting the data definition on the data content includes: A predetermined numerical boundary for defining a predetermined category based on a predetermined common sense criterion; as well as If the digital content belongs to the default category, it is determined whether the digital content exceeds the predetermined value boundary.

Images