Data completeness analysis system and data completeness analysis method

Abstract

A data completeness analysis system and a data completeness analysis method, the data completeness analysis system comprising a storage device, a field data description file generation module and a generalized data completeness analysis module. The storage device is used to store a plurality of native data. The field data description file generation module generates a plurality of field data description files corresponding to the plurality of native data. The generalized data completeness analysis module determines the scores of the consistency indexes of the native data according to the generated field data description files. The generalized data completeness analysis module obtains the data of the label to be analyzed from the native data according to the classification labels of each field data description file, and obtains the scores of the integrity index, the correctness index, the validity index and the refinement index corresponding to the data of the label to be analyzed. A data completeness analysis method is also disclosed herein.

Description

Technical Field

[0001] The embodiments of the present invention mainly relate to a data completeness analysis technology, and more particularly to a data completeness analysis technology for analyzing data completeness of different data orientations through different completeness analysis modules and indicators. Background Art

[0002] As data collection becomes more convenient, the amount of available data increases rapidly, and data analysis technology also develops rapidly. Effective big data analysis results rely on good data quality, so data quality is an important issue in data analysis. The current data quality diagnosis methods can be divided into data analysis experts using programming language analysis or using analysis software packages on the market. However, the current analysis methods for data quality (completeness) are not comprehensive enough, the diagnosis process has not been standardized, and there is a lack of subsequent application inspection and optimization. Summary of the invention

[0003] In view of the above problems, embodiments of the present invention provide a data completeness analysis system and a data completeness analysis method.

[0004] According to one embodiment of the present invention, a data completeness analysis system is provided, comprising a storage device, a field data description file generation module and a completeness analysis module. The storage device is used to store a plurality of native data. The field data description file generation module is coupled to the storage device and is used to generate a plurality of field data description files corresponding to the plurality of native data. The generalized data completeness analysis module is coupled to the storage device and the field data description file generation module to obtain the plurality of native data and the plurality of field data description files. The generalized data completeness analysis module can determine the scores of a consistency index of the native data according to the plurality of field data description files. The generalized data completeness analysis module can obtain the data of the label to be analyzed from the plurality of native data according to the classification labels of each of the field data description files, and obtain the scores of a completeness index, a correctness index, a validity index and a refinement index corresponding to the data of the label to be analyzed. The generalized data completeness analysis module can determine the data completeness corresponding to the data of the label to be analyzed based on the score of the above consistency index, as well as the score of the above integrity index, the score of the above correctness index, the score of the above validity index and the score of the above refinement index corresponding to the data of the label to be analyzed.

[0005] In some embodiments, the data completeness analysis system also includes a native data completeness analysis module. The native data completeness analysis module is coupled to the storage device and the field data description file generation module to obtain the multiple native data and the multiple field data description files. The native data completeness analysis module can obtain the scores of the integrity index, the correctness index, the validity index and the refinement index corresponding to each of the native data based on the multiple native data and the multiple field data description files. The native data completeness analysis module can determine the data completeness of each of the native data based on the scores of the integrity index, the correctness index, the validity index and the refinement index corresponding to each of the native data.

[0006] In some embodiments, the data completeness analysis system further includes a specific data completeness analysis module. The specific data completeness analysis module is coupled to the storage device and the field data description file generation module to obtain the multiple raw data and the multiple field data description files. The specific data completeness analysis module can obtain specific data from the multiple raw data according to a specific template, and obtain the score of the integrity index, the score of the correctness index, the score of the validity index, the score of the refinement index, the score of a fit index and the score of a data volume index corresponding to the specific data. The specific data completeness analysis module can determine the data completeness of the specific data according to the score of the integrity index, the score of the correctness index, the score of the validity index, the score of the refinement index, the score of the fit index and the score of the data volume index corresponding to the specific data.

[0007] In the embodiment of the present invention, the suitability index is used to determine whether the content included in the above-mentioned specific data meets the content required by the above-mentioned specific template.

[0008] In an embodiment of the present invention, the data volume indicator is used to determine whether the data volume of the content included in the specific data is sufficient according to a learning curve.

[0009] In the embodiment of the present invention, the integrity indicator is used to determine whether data in a data field is missing.

[0010] In the embodiment of the present invention, the correctness index is used to determine whether the data has format errors, value errors, and association errors.

[0011] In the embodiment of the present invention, the validity index is used to determine whether the data has abnormal cluster distribution and whether the value of the data exceeds a critical value.

[0012] In an embodiment of the present invention, the refinement index is used to determine whether repeated content or distribution occurs in the data.

[0013] The consistency index is used to determine whether the fields between multiple native data are related and whether their names and recording methods are consistent.

[0014] According to an embodiment of the present invention, a data completeness analysis method is provided. The steps of the data completeness analysis method include: generating multiple field data description files corresponding to multiple native data through a field data description file generation module; obtaining the multiple native data and the multiple field data description files through a generalized data completeness analysis module; judging the scores of a consistency index of the native data according to the multiple field data description files through the generalized data completeness analysis module; obtaining the data of the label to be analyzed from the multiple native data according to the classification labels of each of the field data description files through the generalized data completeness analysis module, and obtaining the scores of a completeness index, a correctness index, a validity index and a refinement index corresponding to the data of the label to be analyzed; and judging the data completeness corresponding to the data of the label to be analyzed according to the scores of the consistency index, and the scores of the completeness index, the correctness index, the validity index and the refinement index corresponding to the data of the label to be analyzed through the generalized data completeness analysis module.

[0015] As for other additional features and advantages of the present invention, those skilled in the art may make slight changes and modifications to the data completeness analysis system and method disclosed in the implementation method of the present invention without departing from the spirit and scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a block diagram showing a data readiness analysis system 100 according to an embodiment of the present invention.

[0017] Figure 2 is a flow chart of a data completeness analysis method according to an embodiment of the present invention.

[0018] Figure 3 is a flow chart of a data completeness analysis method according to another embodiment of the present invention.

[0019] Figure 4 is a flow chart of a data completeness analysis method according to another embodiment of the present invention.

[0020] Description of reference numerals:

[0021] 100: Data completeness analysis system

[0022] 110: Storage device

[0023] 120: Field data description file generation module

[0024] 130: Native data completeness analysis module

[0025] 140: Generalized data completeness analysis module

[0026] 150: Specific data completeness analysis module

[0027] S210~S250, S310~S330, S410~S430: Steps DETAILED DESCRIPTION

[0028] This section describes the preferred method of implementing the present invention, which is intended to illustrate the spirit of the present invention rather than to limit the scope of protection of the present invention. The scope of protection of the present invention shall be determined by the claims.

[0029] Figure 1 FIG. 1 is a block diagram showing a data completeness analysis system 100 according to an embodiment of the present invention. Figure 1 As shown, the data completeness analysis system 100 may include a storage device 110, a field data description file generation module 120, a native data completeness analysis module 130, a generalized data completeness analysis module 140, and a specific data completeness analysis module 150. Figure 1 The block diagram shown in the figure is only for the convenience of explaining the embodiments of the present invention, but the present invention is not limited to Figure 1 The data completeness analysis system 100 may also include other components.

[0030] According to an embodiment of the present invention, the collected multiple native data may be stored in the storage device 110. In addition, according to an embodiment of the present invention, the collected multiple native data may be converted into a tabular form first. It is particularly noted that the native data described in the present invention refers to native data that has been converted into a tabular form. That is to say, the native data stored in the storage device 110 in the embodiment of the present invention represents multiple data tables. The native data stored in the storage device 110 can be provided to the native data completeness analysis module 130, the generalized data completeness analysis module 140 and the specific data completeness analysis module 150 for data completeness analysis.

[0031] According to an embodiment of the present invention, when the collected native data is to be analyzed for data completeness, the field data description file generation module 120 can obtain the native data from the storage device 110, and generate a field data description file corresponding to each native data (data table) according to the obtained native data (e.g., multiple data tables). In each field data description file, the field name of each native data (data table), the field data form (type) corresponding to the field of each native data (data table), the category label (category) corresponding to the field of each native data (data table), and the field association between the native data (data tables) can be recorded, but the present invention is not limited thereto.

[0032] The column name of each native data (data table) can be a product name, a product serial number, a name of a substance, a name of a person, or a measurement value, but the present invention is not limited thereto. The column data form corresponding to the column of each native data (data table) can include an integer (which can be marked as int), a string (which can be marked as varchar), a Boolean value (which can be marked as bool), or a date (which can be marked as data), but the present invention is not limited thereto. The classification label corresponding to the column of each native data (data table) can include people, machines, raw materials, methods, environments, places, or measurements, but the present invention is not limited thereto. The content corresponding to the column of each native data (data table) can be a description corresponding to the column name. Tables 1 and 2 are used as examples below for illustration.

[0033] product Recording Project Serial number Numeric A temperature 1 31.5 A speed 1 3648 A Voltage 1 0.89 A temperature 2 31 A speed 2 3680 A Voltage 2 0.89 … … … …

[0034] Table 1

[0035] Field Name The content of the column Field data format Category Tags product Product Name String other Recording Project Product Attributes String other Serial number Product No. Boolean value other Numeric The value of the product attribute Integer Measurement

[0036] Table 2

[0037] Table 1 is a schematic diagram of a native data S according to an embodiment of the present invention. As shown in Table 1, the native data S shown in Table 1 relates to a data table of product A. It is particularly noted that the native data S only displays the data of product A, but the present invention is not limited thereto. Other native data may also include data tables of other products. In addition, the information of product A is taken as an example in the native data S of Table 1, but the present invention is not limited thereto. The native data may also be other data tables corresponding to different analysis topics, such as: environmental measurement, character analysis, event analysis, etc. Table 2 is a schematic diagram of a column data description file according to an embodiment of the present invention. As shown in Table 2, the column data description file shown in Table 2 displays the column name, data form, and classification label corresponding to each column of the native data S of Table 1. It is particularly noted that Table 2 is only used to illustrate an embodiment of the present invention, but the present invention is not limited thereto. The column data description file may also include other information, for example, between multiple different native data tables, column association information may be established based on the data similarity between different columns.

[0038] According to an embodiment of the present invention, when the raw data completeness analysis module 130 is to analyze the data completeness of raw data (e.g., multiple data tables), the raw data to be analyzed can be obtained from the storage device 110, and the field data description file corresponding to the raw data to be analyzed can be obtained from the field data description file generation module 120. Then, the raw data completeness analysis module 130 can calculate the scores of the completeness index, the accuracy index, the validity index, and the compaction index corresponding to each raw data according to the obtained raw data and the field data description file. The raw data completeness analysis module 130 can then generate an overall score of the data completeness of the corresponding raw data according to the scores of the completeness index, the accuracy index, the validity index, and the compaction index corresponding to each raw data.

[0039] According to an embodiment of the present invention, the completeness index is used to determine whether the data of a data field is missing or missing. According to an embodiment of the present invention, the native data completeness analysis module 130 can calculate the ratio of missing data for each field of the native data, and calculate the score of the completeness index corresponding to the native data based on the ratio of missing data for each field. According to an embodiment of the present invention, if the ratio of missing data for a field is greater than a critical value, the native data completeness analysis module 130 can generate a prompt message to inform the user that the data completeness of this field may be insufficient.

[0040] According to an embodiment of the present invention, the correctness index is used to determine whether the data has format errors, numerical errors, and association errors. According to an embodiment of the present invention, the native data completeness analysis module 130 can calculate the score of the correctness index corresponding to the native data based on the percentage of data errors (format errors, numerical errors, and association errors) contained in the native data.

[0041] According to an embodiment of the present invention, the native data completeness analysis module 130 can determine whether the data contained in each field of the native data has a format error according to the field data form corresponding to the field recorded in the field data description file. For example, if the field data form corresponding to a field is an integer (int), but the data contained therein is a string (varchar), the native data completeness analysis module 130 will determine that the field has a format error. It should be noted that the above example is only used to illustrate an embodiment of the present invention, but the present invention is not limited thereto.

[0042] According to one embodiment of the present invention, the native data completeness analysis module 130 can determine whether the data contained in each column of the native data is greater than or less than an upper limit value or a lower limit value to determine whether a numerical error occurs. For example, if the data recorded in a column is height, but a value less than 0 appears, the native data completeness analysis module 130 will determine that a numerical error occurs in the column. According to another embodiment of the present invention, the native data completeness analysis module 130 can determine whether the data contained in each column of the native data has an unreasonable value to determine whether a numerical error occurs. For example, if the data recorded in a column is gender, but data unrelated to gender appears, the native data completeness analysis module 130 will determine that a numerical error occurs in the column. It should be noted that the above examples are only used to illustrate embodiments of the present invention, but the present invention is not limited thereto.

[0043] According to one embodiment of the present invention, the native data completeness analysis module 130 can determine whether an error occurs in the association of data in the fields of the native data, so as to determine whether an association error occurs. For example, if the data recorded in one field of the native data is Fahrenheit temperature, and the data recorded in another field is Celsius temperature, when an error occurs in the association between the Fahrenheit temperature and its corresponding Celsius temperature, the native data completeness analysis module 130 will determine that an association error occurs in the native data. For another example, if the data recorded in one field of the native data is the start time, and the data recorded in another field is the end time, when a start time is later than its corresponding end time, the native data completeness analysis module 130 will determine that an association error occurs in the native data. It should be noted that the above examples are only used to illustrate the embodiments of the present invention, but the present invention is not limited thereto.

[0044] According to an embodiment of the present invention, the validity index is used to determine whether the data has abnormal cluster distribution, and whether the value of the data (for example, the number of outliers) exceeds a critical value. According to an embodiment of the present invention, the native data completeness analysis module 130 can determine whether the data corresponding to each field of the native data is too dispersed or too concentrated to determine whether the native data has abnormal cluster distribution. When the data corresponding to the field is too dispersed or too concentrated, the analysis value of the native data will be reduced due to the excessive dispersion or concentration of the samples. Therefore, when the data corresponding to each field of the native data is too dispersed or too concentrated, the native data completeness analysis module 130 will determine that the native data has abnormal cluster distribution. For example, if the data recorded in a field of the native data is a device model, when the device model recorded in the field is too dispersed (that is, the collected samples are dispersed in different models), the native data completeness analysis module 130 will determine that the native data has abnormal cluster distribution. If the data recorded in a column of the native data is gender, when the data recorded in the column is too concentrated (i.e., the collected samples are too concentrated, for example: the recorded genders are all boys), the native data completeness analysis module 130 will determine that the native data has an abnormal cluster distribution. In addition, according to another embodiment of the present invention, the native data completeness analysis module 130 can determine whether the value corresponding to each column of the native data exceeds a critical value to determine whether the sample collected in the native data is abnormal. When the value corresponding to a column of the native data exceeds a critical value, the native data completeness analysis module 130 will determine that the sample collected in the native data is abnormal. The native data completeness analysis module 130 can calculate the score of the validity index corresponding to the native data based on the results of determining whether the native data has an abnormal cluster distribution and whether the value of the native data exceeds a critical value. It should be noted that the above examples are only used to illustrate the embodiments of the present invention, but the present invention is not limited thereto.

[0045] According to an embodiment of the present invention, the refinement index is used to determine whether there is repeated content or distribution in the data. According to an embodiment of the present invention, the native data completeness analysis module 130 can determine whether there are repeated fields or repeated data in the native data to calculate the refinement index score corresponding to the native data.

[0046] According to an embodiment of the present invention, when the generalized data completeness analysis module 140 is to analyze the data completeness of each classification label of the original data (e.g., multiple data tables), the generalized data completeness analysis module 140 may first obtain the original data to be analyzed from the storage device 110, and obtain the field data description file corresponding to the original data to be analyzed from the field data description file generation module 120. The generalized data completeness analysis module 140 will determine the correlation between each original data to obtain the consistency index score. In addition, the generalized data completeness analysis module 140 will extract the data of the field corresponding to the label to be analyzed from each original data according to the classification label recorded in each field data description file (i.e., the classification label corresponding to the field of each original data), and calculate the completeness index score, accuracy index score, validity index score, and compaction index score corresponding to the data of the label to be analyzed. Finally, the generalized data completeness analysis module 140 can generate an overall score of data completeness corresponding to the data of the label to be analyzed based on the score of the consistency index and the scores of the integrity index, the correctness index, the validity index and the refinement index corresponding to the data of the label to be analyzed. Taking Table 1 and Table 2 as an example, if the classification label to be analyzed by the generalized data completeness analysis module 140 is "measurement", the generalized data completeness analysis module 140 will extract the data of the column with the classification label "measurement" from the original data to perform data completeness analysis.

[0047] According to an embodiment of the present invention, the consistency index is used to determine whether the fields between multiple native data are related and whether their names and recording methods are consistent. For example, if field B1 of native data A1 and field B2 of native data A2 correspond to the same data, but the field names of field B1 and field B2 are inconsistent or the field data forms of field B1 and field B2 are inconsistent, the generalized data completeness analysis module 140 will determine that field B1 of native data A1 and field B2 of native data A2 are inconsistent. When the names and recording methods of the same fields corresponding to each other in native data are inconsistent, it will easily cause confusion in analysis. The generalized data completeness analysis module 140 will calculate the score of the consistency index based on the coverage ratio of the native data corresponding to each other. It should be noted that the above examples are only used to illustrate the embodiments of the present invention, but the present invention is not limited thereto.

[0048] According to an embodiment of the present invention, the generalized data completeness analysis module 140 calculates the scores of the completeness index, the correctness index, the validity index and the refinement index corresponding to the data of the label that needs to be analyzed, and the native data completeness analysis module 130 calculates the scores of the completeness index, the correctness index, the validity index and the refinement index corresponding to the native data, which is similar to the calculation method, so it will not be repeated here.

[0049] According to an embodiment of the present invention, when the specific data completeness analysis module 150 is to analyze the data completeness of specific data corresponding to a specific template in the native data (e.g., multiple data tables), the specific data completeness analysis module 150 can first obtain the native data to be analyzed from the storage device 110, and obtain the field data description file corresponding to the native data to be analyzed from the field data description file generation module 120. The specific template described in the embodiment of the present invention can be a setting that represents specific data required for a specific application, a specific scenario, or a specific requirement. Table 3 will be used for explanation below.

[0050]

[0051]

[0052] Table 3

[0053] Table 3 is a schematic diagram of specific data according to an embodiment of the present invention. Referring to Table 1, if the specific template is an analysis of the data completeness of product A, the specific data completeness analysis module 150 will obtain specific data about product A from the native data according to the information of the specific template (as shown in Table 3). It should be noted that Table 3 is only used to illustrate an embodiment of the present invention, but the present invention is not limited thereto. The specific template can also be an analysis of different applications.

[0054] The specific data completeness analysis module 150 can obtain the specific data from the raw data according to the information of the specific template, and calculate the scores of the completeness index, the accuracy index, the validity index, the compaction index, the fitness index, and the quantity index corresponding to the specific data. Finally, the specific data completeness analysis module 150 can generate an overall score of data completeness corresponding to the specific data according to the scores of the completeness index, the accuracy index, the validity index, the compaction index, the fitness index, and the quantity index corresponding to the specific data.

[0055] According to an embodiment of the present invention, the compatibility index is used to determine whether the content contained in the specific data meets the content required by the specific template. The specific data completeness analysis module 150 can determine whether the fields contained in the specific data meet the fields required by the specific template, so as to calculate the score of the compatibility index corresponding to the specific data. According to an embodiment of the present invention, the specific data completeness analysis module 150 can determine whether the content contained in the current specific data can meet the content required by the specific template based on the historical data corresponding to the specific template, so as to calculate the score of the compatibility index corresponding to the specific data.

[0056] According to an embodiment of the present invention, the data volume indicator is used to determine whether the data volume of the content contained in the specific data is sufficient based on a learning curve. The learning curve can be generated using a data learning method. The specific data completeness analysis module 150 can generate a learning curve based on the data volume of the content contained in the specific data, and calculate based on the slope of the learning curve to generate a score for the data volume indicator. When the slope of the learning curve is smaller, the data volume of the content contained in the specific data is more sufficient. When the slope of the learning curve is larger, the data volume of the content contained in the specific data is more insufficient.

[0057] According to an embodiment of the present invention, the way in which the specific data completeness analysis module 150 calculates the scores of the integrity index, the correctness index, the validity index and the refinement index corresponding to the specific data is similar to the way in which the native data completeness analysis module 130 calculates the scores of the integrity index, the correctness index, the validity index and the refinement index corresponding to the native data, and therefore, it will not be repeated here.

[0058] According to the native data completeness analysis module 130, the generalized data completeness analysis module 140, and the specific data completeness analysis module 150 proposed in the embodiment of the present invention, the data completeness analysis system 100 can respectively generate analysis results of data completeness of different aspects of the data. In some embodiments, only the analysis results generated by the native data completeness analysis module 130, the generalized data completeness analysis module 140, and the specific data completeness analysis module 150 can be used to determine the data completeness of the data. In some embodiments, the analysis results generated by multiple of the native data completeness analysis module 130, the generalized data completeness analysis module 140, and the specific data completeness analysis module 150 can be used to conduct a comprehensive evaluation to determine the data completeness of the data.

[0059] Figure 2 FIG. 1 is a flow chart of a data completeness analysis method according to an embodiment of the present invention. The data completeness analysis method can be applied to the data completeness analysis system 100. Figure 2As shown, in step S210, the field data description file generation module of the data completeness analysis system 100 generates multiple field data description files corresponding to multiple original data. In step S220, a generalized data completeness analysis module of the data completeness analysis system 100 obtains multiple original data and multiple field data description files. In step S230, the generalized data completeness analysis module of the data completeness analysis system 100 determines the score of a consistency index of the original data according to the multiple field data description files. In step S240, the generalized data completeness analysis module of the data completeness analysis system 100 obtains the data of the label to be analyzed from the multiple original data according to the classification label of each field data description file, and obtains the score of a completeness index, a correctness index, a validity index and a refinement index corresponding to the data of the label to be analyzed. In step S250, the generalized data completeness analysis module of the data completeness analysis system 100 determines the data completeness corresponding to the data of the label to be analyzed based on the score of the consistency index and the scores of the integrity index, correctness index, validity index and refinement index corresponding to the data of the label to be analyzed.

[0060] Figure 3 FIG. 1 is a flow chart of a data completeness analysis method according to an embodiment of the present invention. The data completeness analysis method may be applied to the data completeness analysis system 100. Figure 3 As shown, in step S310, the data completeness analysis system 100 generates multiple field data description files corresponding to multiple original data. In step S320, a native data completeness analysis module of the data completeness analysis system 100 obtains multiple original data and multiple field data description files. In step S330, the native data completeness analysis module of the data completeness analysis system 100 obtains the scores of the integrity index, the correctness index, the validity index and the refinement index corresponding to each original data according to the multiple original data and the multiple field data description files. In step S340, the native data completeness analysis module of the data completeness analysis system 100 determines the data completeness of each original data according to the scores of the integrity index, the correctness index, the validity index and the refinement index corresponding to each original data.

[0061] Figure 4 FIG. 1 is a flow chart of a data completeness analysis method according to an embodiment of the present invention. The data completeness analysis method may be applied to the data completeness analysis system 100. Figure 4As shown, in step S410, the field data description file generation module of the data completeness analysis system 100 generates multiple field data description files corresponding to multiple original data. In step S420, a specific data completeness analysis module of the data completeness analysis system 100 obtains multiple original data and multiple field data description files. In step S430, the specific data completeness analysis module of the data completeness analysis system 100 obtains specific data from multiple original data according to a specific template, and obtains the scores of the integrity index, the correctness index, the validity index, the refinement index, the fit index and the data volume index corresponding to the specific data. In step S440, the specific data completeness analysis module of the data completeness analysis system 100 determines the data completeness of the specific data according to the scores of the integrity index, the correctness index, the validity index, the refinement index, the fit index and the data volume index.

[0062] According to the data completeness analysis method proposed in the present invention, different aspects of data completeness can be analyzed through different analysis modules and indicators.

[0063] Serial numbers in this specification and claims, such as "first", "second", etc., are only for convenience of explanation and have no sequential relationship with each other.

[0064] The steps of the methods and algorithms disclosed in the specification of the present invention can be directly applied to hardware and software modules or a combination of the two by executing a processor. A software module (including execution instructions and related data) and other data can be stored in a data storage device, such as a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electronically erasable programmable read-only memory (EEPROM), a temporary register, a hard disk, a portable hard disk, a compact disk read-only memory (CD-ROM), a DVD or any other computer-readable storage medium format in the technology commonly used in this field. A storage medium can be coupled to a machine device, for example, such as a computer / processor (for convenience of explanation, the processor is represented in this specification), and the above-mentioned processor can read information (such as program code) and write information to the storage medium. A storage medium can integrate a processor. An application-specific integrated circuit (ASIC) includes a processor and a storage medium. A user device includes an application-specific integrated circuit. In other words, the processor and the storage medium are included in the user device in a way that is not directly connected to the user device. In addition, in some embodiments, any suitable computer program product includes a readable storage medium, wherein the readable storage medium includes program code related to one or more disclosed embodiments. In some embodiments, the computer program product may include packaging materials.

[0065] The above paragraphs are described using multiple levels. Obviously, the teachings of this article can be implemented in multiple ways, and any specific architecture or function disclosed in the examples is only a representative situation. Based on the teachings of this article, any person skilled in the art should understand that each level disclosed in this article can be implemented independently or two or more levels can be combined for implementation.

[0066] Although the present disclosure has been disclosed as above by way of embodiments, it is not intended to limit the present disclosure. Any person skilled in the art may make some changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the invention shall be determined by the definition of the claims.

Claims

1. A data completeness analysis system, comprising: A storage device for storing a plurality of raw data; a field data description file generating module, coupled to the storage device, and used for generating a plurality of field data description files corresponding to the plurality of native data, wherein the field data description files record at least one of the field names of the native data, the field data forms corresponding to the fields of the native data, the classification labels corresponding to the fields of the native data, and the field associations between the native data; and a generalized data completeness analysis module, coupled to the storage device and the field data description file generation module, to obtain the plurality of raw data and the plurality of field data description files; The generalized data completeness analysis module determines the scores of a consistency index of the plurality of native data according to the plurality of field data description files, wherein the consistency index is used to determine whether the fields between the plurality of native data are related and whether their names and recording methods are consistent; The generalized data completeness analysis module obtains the data of the label to be analyzed from the plurality of raw data according to the classification label of each of the field data description files, and obtains a completeness index score, a correctness index score, a validity index score, and a refinement index score corresponding to the data of the label to be analyzed; The generalized data completeness analysis module determines the data completeness corresponding to the data of the label to be analyzed according to the score of the consistency index, and the score of the integrity index, the score of the correctness index, the score of the validity index and the score of the refinement index corresponding to the data of the label to be analyzed; The data completeness analysis system further includes: a raw data completeness analysis module, coupled to the storage device and the field data description file generation module, to obtain the plurality of raw data and the plurality of field data description files; The native data completeness analysis module obtains the score of the completeness index, the score of the correctness index, the score of the validity index, and the score of the refinement index corresponding to each of the native data according to the plurality of native data and the plurality of field data description files; The native data completeness analysis module determines the data completeness of each of the native data according to the score of the integrity index, the score of the correctness index, the score of the validity index and the score of the refinement index corresponding to each of the native data; The generalized data completeness analysis module and the native data completeness analysis module respectively generate analysis results of data completeness of different aspects of the data. The analysis results generated by the generalized data completeness analysis module and the native data completeness analysis module are used to conduct a comprehensive evaluation to determine the data completeness.

2. The data readiness analysis system according to claim 1, further comprising: a specific data completeness analysis module, coupled to the storage device and the field data description file generation module, to obtain the plurality of raw data and the plurality of field data description files; The specific data completeness analysis module obtains specific data from the plurality of raw data according to a specific template, and obtains the score of the integrity index, the score of the correctness index, the score of the effectiveness index, the score of the refinement index, the score of a suitability index and the score of a data volume index corresponding to the specific data; The specific data completeness analysis module determines the data completeness of the specific data based on the score of the integrity index, the score of the correctness index, the score of the effectiveness index, the score of the refinement index, the score of the suitability index and the score of the data volume index.

3. The data completeness analysis system as described in claim 2, wherein the above-mentioned suitability index is used to determine whether the content included in the above-mentioned specific data meets the content required by the above-mentioned specific template.

4. The data completeness analysis system as claimed in claim 2, wherein the data volume indicator is used to determine whether the data volume of the content included in the specific data is sufficient according to a learning curve.

5. The data completeness analysis system as claimed in claim 2, wherein the completeness indicator is used to determine whether data in a data field is missing.

6. The data completeness analysis system as claimed in claim 2, wherein the correctness index is used to determine whether the data has format errors, value errors and correlation errors.

7. The data completeness analysis system as claimed in claim 2, wherein the effectiveness index is used to determine whether the data has abnormal cluster distribution and whether the value of the data exceeds a critical value.

8. The data completeness analysis system as claimed in claim 2, wherein the refinement index is used to determine whether repeated content or distribution occurs in the data.

9. A data completeness analysis method, comprising: Generating a plurality of field data description files corresponding to the plurality of native data, wherein the field data description files record at least one of field names of the native data, field data forms corresponding to the fields of the native data, classification labels corresponding to the fields of the native data, and field associations between the native data; Obtaining the plurality of raw data and the plurality of field data description files through a generalized data completeness analysis module; By means of the generalized completeness data analysis module, a consistency index score of the plurality of native data is determined according to the plurality of field data description files, wherein the consistency index is used to determine whether the fields of the plurality of native data are related and whether their names and recording methods are consistent; By means of the generalized data completeness analysis module, data of the label to be analyzed is obtained from the plurality of raw data according to the classification label of each of the field data description files, and a score of a completeness index, a score of a correctness index, a score of a validity index, and a score of a refinement index corresponding to the data of the label to be analyzed is obtained; and The generalized data completeness analysis module determines the data completeness corresponding to the data of the label to be analyzed according to the score of the consistency index, and the score of the integrity index, the score of the correctness index, the score of the validity index, and the score of the refinement index corresponding to the data of the label to be analyzed; The above data completeness analysis method also includes: Obtaining the plurality of native data and the plurality of field data description files through a native data completeness analysis module; Obtaining, by the native data completeness analysis module, the score of the integrity index, the score of the correctness index, the score of the validity index, and the score of the refinement index corresponding to each of the native data according to the plurality of native data and the plurality of field data description files; and Determining the data completeness of each of the above-mentioned original data by the above-mentioned original data completeness analysis module according to the score of the above-mentioned integrity index, the score of the above-mentioned correctness index, the score of the above-mentioned validity index and the score of the above-mentioned refinement index corresponding to each of the above-mentioned original data; The generalized data completeness analysis module and the native data completeness analysis module respectively generate analysis results of data completeness of different aspects of the data. The analysis results generated by the generalized data completeness analysis module and the native data completeness analysis module are used to conduct a comprehensive evaluation to determine the data completeness.

10. The data completeness analysis method according to claim 9, further comprising: Obtaining the plurality of raw data and the plurality of field data description files through a specific data completeness analysis module; Obtaining specific data from the plurality of raw data according to a specific template through the specific data completeness analysis module, and obtaining the score of the completeness index, the score of the correctness index, the score of the validity index, the score of the refinement index, the score of a compatibility index, and the score of a data volume index corresponding to the specific data; and The data completeness of the above-mentioned specific data is determined by the above-mentioned specific data completeness analysis module according to the score of the above-mentioned integrity index, the score of the above-mentioned correctness index, the score of the above-mentioned effectiveness index, the score of the above-mentioned refinement index, the score of the above-mentioned suitability index and the score of the above-mentioned data volume index.

11. The data completeness analysis method as claimed in claim 10, wherein the above-mentioned suitability index is used to determine whether the content included in the above-mentioned specific data meets the content required by the above-mentioned specific template.

12. The data completeness analysis method as claimed in claim 10, wherein the data volume indicator is used to determine whether the data volume of the content included in the specific data is sufficient according to a learning curve.

13. The data completeness analysis method of claim 10, wherein the completeness index is used to determine whether data in a data field is missing.

14. The data completeness analysis method as claimed in claim 10, wherein the correctness index is used to determine whether the data has format errors, value errors and correlation errors.

15. The data completeness analysis method as claimed in claim 10, wherein the validity index is used to determine whether the data has abnormal cluster distribution and whether the value of the data exceeds a critical value.

16. The data completeness analysis method as claimed in claim 10, wherein the refinement index is used to determine whether repeated content or distribution occurs in the data.

Images