Analysis system and analysis method for automatically detecting semi-structured data quality problems

Abstract

The application discloses an analysis system and an analysis method for automatically detecting semi-structured data quality problems, the analysis method takes JSON data in semi-structured data as a specific research object, and comprises the following steps: a parsing mode module is used for parsing semi-structured data into an aggregated schema tree; a data quality problem monitoring module is used for automatically detecting potential data quality problems of data according to the aggregated schema tree and a data quality space; a visualization generation module is used for visualizing the aggregated schema tree and the data quality problems, helping users to interactively find and view the data quality problems; and a data cleaning module is used for user configuration and solution of the data quality problems. The system can help users to quickly and effectively locate and clean the data quality problems of semi-structured data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of semi-structured data processing, and in particular to an analysis system and method for automatically detecting quality problems of semi-structured data. BACKGROUND

[0002] In the real world, enterprises and individuals usually analyze data to make value judgments and economic decisions, and the data quality of the original data directly affects the reliability, interpretability and authenticity of this process. Real-life data often contains various data quality problems, such as missing values, inconsistent values, or duplicate values. These data quality problems have multiple sources, such as human error, data integration, and non-standard coding style. The conclusions drawn from analyzing low-quality data often contain errors, which can lead to significant economic losses. Therefore, data quality problems must be discovered and solved before using the data.

[0003] Currently, most of the work of the industry and academia is focused on solving the quality problems of tabular data. Tabular data belongs to structured data, which has a clearly defined structure and conforms to a pre-defined data model. However, with the development of the network, semi-structured data is used more frequently, so it is crucial to ensure its data quality. For example, in the data analysis scenario, data workers need to clean the data as a pre-step for decision analysis. In addition, in the scenario of NoSQL database quality maintenance, data managers need to ensure that the data quality meets the preset standards. JSON data, as a typical and commonly used semi-structured data, has a clear hierarchical structure and is the mainstream format for network transmission and content storage. However, there are only a few studies on semi-structured data quality.

[0004] Compared with structured data, semi-structured data has the following two main differences:

[0005] First, semi-structured data does not follow a specific pattern. Semi-structured data often does not have a pre-defined pattern when it is generated, and its pattern can change at any time, for example, a property in JSON can be added, deleted, or changed.

[0006] Second, semi-structured data is hierarchical and each layer contains specific information. Structured data usually displays data as a flat table, but semi-structured data can be nested layer by layer, containing more diverse information.

[0007] On one hand, the above two features make semi-structured data convenient for network information transmission; on the other hand, it makes the cleaning process of semi-structured data more difficult and time-consuming, and it is more difficult for users to find and solve data quality problems hidden under the hierarchy. Therefore, it is a challenging research problem to help users automatically detect data quality problems in semi-structured data.

[0008] Currently, many research works have proposed methods or frameworks for guiding semi-structured data quality management. These works mainly focus on JSON data and can be roughly divided into three categories: (1) The first category is data profiling, such as OpenRefine, Profiler, etc. Tools support generating statistical information, such as data type, structure, schema, statistical indicators, etc., to help users find problem data, but this kind of tool does not support processing semi-structured data; for example, JSONDiscoverer, STEED, JSON crack, SchemaDrill, etc. Tools can extract and visualize the schema from semi-structured data, but this kind of tool does not support detecting data quality problems; (2) The second category is data quality monitoring. Currently, the academic community has not yet summarized a recognized data quality description dimension, because the word "data quality" often depends on the context and implies the principle of "suitable for use". Laranjeiro et al. (Laranjeiro N, Soydemir S N, Bernardino J. A Survey on Data Quality: Classifying Poor Data [C] / / Proceedings of the IEEE Pacific Rim International Symposium on Dependable Computing.) Investigated and summarized the commonly used data quality description dimensions, such as accuracy, completeness, consistency. In addition, data quality monitoring is a circular process, mainly including three implementation ways, which are: using organized control mechanism, continuously detecting whether rules are violated, periodically monitoring and visualizing monitoring results; (3) The third category is data cleansing, such as Data Chamaleon is a framework that uses domain-specific language to convert JSON data, but it does not support automatic detection of data quality problems, and using domain-specific language coding has a certain learning cost for beginners; for example, JSON Formatter&Validator, JSON Cleaner, JSONFormatter.io, etc. Commercial products can only solve JSON data format errors (such as duplicate attributes, trailing commas, single and double quotes, etc.), format JSON data, and cover a narrow range of data quality. SUMMARY

[0009] In order to detect data quality problems in semi-structured data and continuously monitor the changed data, the present application proposes an analysis system and an analysis method for automatically detecting semi-structured data quality problems, receiving user uploaded original JSON data, performing type inference, similarity calculation, type aggregation, generating an aggregated schema tree, and automatically detecting semi-structured data quality problems using the aggregated schema tree and the original JSON data and supporting user configuration of data conversion operation parameters. The present application can greatly improve the efficiency of data workers in processing semi-structured data quality problems, support automatic detection of data quality problems, and support user interactive configuration of parameters to solve data quality problems. Moreover, the present application has good generality and universality, such as being migrated to other types of semi-structured data, such as XML.

[0010] The object of the present application is achieved by the following technical solutions:

[0011] An analysis system for automatically detecting semi-structured data quality problems, comprising:

[0012] A parsing schema module that receives user input of original JSON data or JSON data after several times of cleaning, and sequentially performs type inference, similarity calculation, and type aggregation on the JSON data, and outputs an intermediate data of a tree structure describing an aggregated schema, i.e., an aggregated schema tree;

[0013] A data quality problem detection module that receives the JSON data and the aggregated schema tree output by the parsing schema module, and classifies data quality from two dimensions based on the space of semi-structured data quality; and for each data quality problem in the space, correspondingly gives a rule-based method for detecting the current data quality problem, and finally obtains information including the name of the current data quality problem, specific classification, node code in the aggregated schema tree, and node related data;

[0014] A visualization generation module that takes the output of the parsing schema module and the data quality problem detection module as input, and generates a visualization view of data quality problems and schema after encoding mapping and layout generation;

[0015] A data cleaning module for performing cleaning operations on user input of JSON data according to user freely set configurations or system recommended configurations, and providing preview operations before and after data cleaning to help users clearly locate the source of the current data quality problem.

[0016] Further, the parsing schema module comprises a type inference submodule, a similarity calculation submodule, and a type aggregation submodule;

[0017] The type inference submodule infers six data types of JSON data, namely, string String, Boolean, null Null, number Number, dictionary Dict, and array Array; among them, String, Boolean, Null, and Number are simple data types, and Dict and Array are complex data types;

[0018] The similarity calculation submodule is configured to calculate the similarity of the complex data types;

[0019] The type aggregation submodule is configured to merge two complex data types with a similarity not less than a preset threshold according to the similarity calculation result. JSON data is essentially a tree, and the current node is an array, a dictionary, or a simple data type during traversal. These three cases are classified and processed. Since there are multiple nesting cases for node types, type aggregation is a recursive process. Specifically, at the root of the JSON data tree, only the root node is considered to be an array or a dictionary.

[0020] If the current node is a dictionary, dictionary internal merging is performed. First, the child nodes under the current dictionary are grouped according to the type of the attribute value. The grouped results of the child nodes of the current node are iteratively processed according to the dictionary, array, and simple data type classification:

[0021] (a) If the child node is a dictionary, first perform dictionary internal merging on each dictionary in the current group, and then check whether the dictionaries after internal merging can be further merged between dictionaries. If the similarity is not less than the preset threshold, further merging between dictionaries is performed. Otherwise, merging between dictionaries is not performed.

[0022] (b) If the child node is an array, first perform array internal merging on each data type in the current array, and then determine whether the array after internal merging can be further merged between arrays according to the similarity threshold.

[0023] (c) If the child node is a simple data type, directly initialize the node information of the aggregated mode tree and put it into the final result.

[0024] If the current node is an array, array internal merging is performed. First, the child nodes are grouped according to the type of the array elements. The grouped results of the child nodes of the current node are iteratively processed according to the dictionary, array, and simple data type classification. The processing procedures of these three data types are the same as those when the current node is a dictionary.

[0025] If the current node is a simple data type, it is directly initialized as the node information of the aggregation mode tree and put into the final result; at this time, the current node has no child node and does not need to be further aggregated.

[0026] Further, the similarity calculation uses the Jaccard index, and the threshold is the default data in the range of 0-1 or the data manually input by the user.

[0027] Further, the data quality problem detection module classifies data quality problems from two dimensions, the first dimension focuses on describing data quality, including consistency, repeatability, redundancy, integrity, accuracy and mode; the second dimension focuses on describing data objects in JSON data, including attribute key, attribute value, attribute key-value pair and structure.

[0028] Further, the data quality problem detection module marks each data quality problem detected in the space as ①, ② and ③ according to whether it can be accurately detected, roughly detected or difficult to be detected; for the data quality problems that can be accurately detected and roughly detected, the data quality problem detection module correspondingly gives the rule-based detection method; for the data quality problems that are difficult to be detected, the user needs to select the node in the tree and declare the problem.

[0029] Further, the rule-based detection method for the current data quality problem is divided into two categories:

[0030] The first category is based on the decisive rule, which directly judges by using the aggregation mode tree;

[0031] The second category is based on the heuristic rule, which directly traverses the input JSON and combines the aggregation mode tree to judge at the same time.

[0032] Further, each detected data quality problem outputs the name of the current data quality problem and the node code in the aggregation mode tree; the node code in the tree is used to locate the node and the data related to the node.

[0033] An analysis method for automatically detecting semi-structured data quality problems, which is realized based on an analysis system;

[0034] The method comprises the following steps:

[0035] Step one: the parsing mode module receives the user input of the original JSON data or the JSON data after several times of cleaning, and sequentially performs type inference, similarity calculation and type aggregation on the original JSON data, and outputs an intermediate data of a tree structure describing the aggregation mode, i.e. an aggregation mode tree;

[0036] Step 2: The data quality problem detection module receives the original JSON data and the aggregation pattern tree. Based on the semi-structured data quality space, it classifies the data quality from two dimensions. For each data quality problem in the space, it provides a rule-based method for detecting the current data quality problem, and finally obtains information including the name of the current data quality problem, its specific classification, the node code in the aggregation pattern tree involved, and the node-related data.

[0037] Step 3: The visualization generation module generates a visualization view of data quality issues and patterns by encoding mapping and layout generation of the results obtained in Step 1 and Step 2.

[0038] Step 4: The data cleaning module performs cleaning operations on the user-input raw JSON data or JSON data that has undergone several cleaning cycles based on visualized data quality issues;

[0039] Step 5: Repeat steps 1 through 4 with the cleaned JSON data until no data quality issues are detected or the iteration termination condition is met, and output high-quality JSON data.

[0040] The beneficial effects of this invention are as follows:

[0041] (1) This invention can help users quickly and effectively locate data quality problems in semi-structured data, and supports recommended transformation operations and generation mode parameters, as well as user interactive configuration forms to perform transformation operations on data quality problems, repeating until a clean JSON data is obtained.

[0042] (2) The present invention adopts a similarity-based pattern aggregation method to obtain an aggregated pattern tree, allowing users to quickly view JSON patterns.

[0043] (3) This invention combines the visualization of JSON schema with JSON data quality issues, visualizing the JSON schema as a tree and the JSON data quality issues as bubbles surrounding the nodes in the tree, allowing users to better discover and solve data quality issues in the context of JSON data structure. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the analysis method for automatically detecting quality problems in semi-structured data according to an embodiment of the present invention.

[0045] Figure 2 This is an interface diagram of the analysis system for automatically detecting quality problems in semi-structured data according to an embodiment of the present invention. Detailed Implementation

[0046] The present application will be described in detail below with reference to the accompanying drawings and preferred embodiments, the objects and effects of the present application will become more apparent, and it should be understood that the specific embodiments described herein are merely intended to explain the present application and are not intended to limit the present application.

[0047] The analysis system for automatically detecting semi-structured data quality problems in the embodiments of the present application aims to assist users in cleaning data quality problems existing in semi-structured data and obtaining high-quality data. The system includes a parsing mode module, a data quality problem detection module, a visualization generation module, and a data cleaning module.

[0048] I. Parsing mode module

[0049] The module includes a type inference submodule, a similarity calculation submodule, and a type aggregation submodule. The input of the module is the original JSON data uploaded or copied into the editor by the user or the JSON data after several cleanings. The module finally outputs an intermediate data describing the aggregated mode tree structure, i.e., the aggregated mode tree, by executing the three submodules of type inference (aiming to convert specific data into data types, i.e., the six data types of JSON), similarity calculation (aiming to calculate the similarity between arrays or dictionaries), and type aggregation (aiming to aggregate arrays or dictionaries with a similarity threshold). Each node in the tree includes all original data information of the current node, regular information describing the aggregation, information describing all types after aggregation of the current node, and subnode information.

[0050] As one of the implementation manners, in the present embodiment, when the user uploads the original JSON data or inputs the JSON data after several cleanings, the first step of type inference is mainly aimed at inferring the six data types of JSON, which are String, Boolean, Null, Number, Dict, and Array. Among them, String, Boolean, Null, and Number are simple data types, and Dict and Array are complex data types. Taking [1, "s"] as an example, the inference result is [Number, String]. The second step is similarity calculation, which is applied to complex data types and is used to judge whether the current complex data type can be aggregated at a given threshold. For Dict, it is judged according to whether the key names in the dictionary are the same, and for Array, it is judged according to whether the element types after aggregation in the array are the same. The similarity calculation uses Jaccard Index, and the threshold is the data in the range of 0-1 initially set by default or manually input by the user. Taking Figure 1For example, in the input JSON, the two dictionary key names under the array are title / genres / score and title / genre / score. Their union is title / genres / score / genre, with a length of 4; their intersection is title / score, with a length of 2, so their similarity is 2 / 4 = 0.5. If the initial threshold is set to 0.5, the two dictionaries can be merged.

[0051] The third step of type aggregation depends on the calculation result of the similarity. The type aggregation is performed on the array and the dictionary. Since the node type may have various nested conditions such as parent array nesting child array, parent dictionary nesting child array, parent array nesting child dictionary, and the like, the type aggregation process is a top-down recursive process. In particular, at the tree root of the JSON data, only two cases of the root node being an array or a dictionary are considered. The current node is discussed according to whether it is a dictionary, an array, or a simple data type.

[0052] If the current node is a dictionary, dictionary internal merging is performed. First, the keys (i.e., child nodes) under the current dictionary are grouped according to the type of the attribute value. The grouped results of the child nodes of the current node are discussed according to the dictionary, array, and simple data type in turn.

[0053] (a) If the child node is a dictionary, dictionary internal merging is first performed on each dictionary in the current group. Then, it is checked whether the dictionaries after internal merging can be further merged between dictionaries. Whether the dictionaries can be merged mainly depends on the similarity. If the similarity is not less than the preset threshold, the dictionaries can be merged, otherwise they cannot be merged.

[0054] (b) If the child node is an array, similarly, array internal merging is first performed on each data type in the current array. Whether the array after internal merging can be further merged between arrays is determined according to the similarity threshold;

[0055] (c) If the child node is a simple data type, it is directly initialized as the node information of the schema tree and placed in the final result.

[0056] If the current node is an array, array internal merging is performed. First, the child nodes after type inference of the array elements are grouped. The grouped results of the child nodes of the current node are discussed according to the dictionary, array, and simple data type in turn. The processing procedures of these three data types are the same as those when the current node is a dictionary.

[0057] If the current node is a simple data type, it is directly initialized as the node information of the aggregation schema tree and put into the final result; at this time, the current node has no child node and does not need to be further aggregated.

[0058] For example, the input JSON in Figure 1 The merged two dictionaries are a dictionary with four attribute keys, title, genres, genre and score. The genre and genres only appear in one dictionary, so they are marked with a '?' mark. The rating attribute under the score dictionary has multiple different types (str and num, respectively), so it is marked with a '|' mark. The genres correspond to an array that appears multiple times in the string type, so it is marked with a '*'. The output after generating the aggregation schema is a tree describing the data structure after aggregation. This tree also includes other information of the current node, such as the data path of all original data corresponding to the current node, the description symbol of the data type of the current node, etc.

[0059] II. Data quality problem detection module

[0060] The input of this module is the aggregation schema tree and the input of the parsing schema module. Based on the space of semi-structured data quality, the data quality is classified in two dimensions. For each data quality problem in the space, a rule-based method is given to detect the current data quality problem, and finally the information including the name of the current data quality problem, the specific classification, the node code in the aggregation schema tree and the related data of the node are obtained.

[0061] As one of the implementation ways, first, based on the real world JSON data and discussion with data experts, a space of semi-structured data quality is summarized. For each data quality problem in the space, a rule-based method is given to detect the current data quality problem. The detection of data quality problems can also be divided into two categories. The first category is based on the decisive rule, which directly judges by using the aggregation schema tree. The second category is based on the heuristic rule, which directly traverses the input JSON and judges together with the aggregation schema tree.

[0062] In this embodiment, based on 119 JSON data collected from real world and expert interviews, a data quality table as shown in Table 1 is summarized, which involves 27 data quality problems. This table classifies data quality problems from two perspectives. The first dimension focuses on describing data quality, which includes consistency, redundancy, completeness, accuracy and schema. The second dimension focuses on describing data objects in JSON data, which includes attribute key, attribute value, attribute key-value pair and structure. Consistency category contains 5 data quality problems. For example, key inconsistency means that there are different keys representing the same entity in the dictionary under the array. Redundancy category also contains 5 problems. For example, key repetition means that there are multiple repeated keys with the same name under a dictionary. Redundancy category also contains 5 problems, but it is different from redundancy. For example, redundant external structure means that different complex data structures are used to represent the same information across levels. Completeness category contains 3 problems. For example, key missing means that there is no key in a dictionary under the array that exists in other dictionaries. Accuracy category focuses more on the accuracy of data. For example, distribution outlier means that there are outliers in the values of the same key in the dictionary under the array. Schema is used to measure whether the current data design structure is easy to understand. For example, hierarchical key means that the hierarchical structure is represented in the key name.

[0063] Table 1 Semi-structured data quality problem table summarized according to two dimensions in the embodiment

[0064]

[0065] For each data quality problem in the space, the data quality problem detection module can be marked as 1, 2, and 3 respectively according to accurate detection, rough detection, and difficult detection. For data quality problems that can be accurately or roughly detected, the corresponding rule-based detection method is given. For data quality problems that are difficult to detect, the user needs to select the node in the tree and declare the problem. The detection of data quality problems is also divided into two categories. The first category is based on the decisive rule, which directly judges using the aggregation schema tree. The second category is based on the heuristic rule, which directly traverses the input JSON and combines the aggregation schema tree to judge at the same time. Each detected data quality problem contains the name of the current data quality problem and the node code in the aggregation schema tree. The node code in the tree is used to locate the node and the node-related data. For example, the hierarchical key under the schema category is detected using the decisive rule. Its input is the node in the aggregation schema tree. For each node, check whether all child attribute key names have a common prefix, and record the node number and related data with a common prefix. For example, when detecting the problem of all repeated key-value pairs in the repetition category, the heuristic rule is used. The rule first traverses the original JSON data. Since the repeated key-value pairs appear in the dictionary under the array, the traversal process records all the paths of the array of dictionaries encountered. After traversal, the arrays composed of dictionaries represented by all recorded paths are checked in turn. First, get the set of all attribute keys of these dictionaries, then traverse and store all values belonging to the attribute key in the current array in the form of a set. Finally, after traversal, if the number of values belonging to the key is equal to the length of the current array and the number of values belonging to the key after deduplication is equal to 1, it is considered that each dictionary under the array belongs to the same key. At this time, the node corresponding to the current path in the schema tree needs to be matched, and the data of the corresponding node is recorded.

[0066] III. Visualization generation module

[0067] The input of this module is the output of the parsing schema module and the data quality problem detection module. After encoding mapping (aiming to map the input data to graph drawing related information) and layout generation (aiming to determine the position of the nodes in the schema tree and their internal constituent elements and the position of the bubbles wrapping the nodes), a visual view of the data quality problems and the schema is generated. The aggregated schema tree is input as input, mainly used to visualize the aggregated schema, the nodes in the tree are encoded using rectangles, and the nodes contain information such as attribute keys, data types, data distribution, etc. The output of the data quality problem detection module is input as input, mainly used to visualize the data quality problems in the current JSON data. On the one hand, the data quality dimension problems are visualized using pie charts, and the data object dimension problems are visualized using stacked bar charts. On the other hand, each data quality associated node is wrapped using bubbles of different color encoding in the tree.

[0068] As one of the embodiments, firstly, the aggregated schema tree output by the schema parsing module is preprocessed by the encoding mapping submodule into a format containing visualized encoding forms, such as the types contained by the nodes, the size or color of the nodes, the position of the icons, etc. Then, each node is laid out on the SVG canvas using the Tree Layout in D3.js, while the convex hull is implemented according to the positioned nodes by the Bubble Set algorithm proposed by Collins et al. (Collins C, Penn G, Carpendale S. Bubble Sets: Revealing Set Relations with Isocontours over Existing Visualizations [J]. IEEE Transactions on Visualization and Computer Graphics, 2009, 15(6): 1009-1016.) to wrap the nodes in the tree. Each node in the tree is encoded with a fixed-width and fixed-height rectangle, which contains the attribute key name of the current node, the type icon of the current node, and if there are multiple data types in the current node, these icons are arranged vertically, and the icon size will be evenly distributed following the fixed height of the rectangle. On the left side of the node, that is, above the tail of the edge, there are four regular symbols to represent the information of the current aggregated node, such as the plus sign '+' representing whether the current node has multiple attribute key names, the question mark '?' representing whether the current attribute key appears once or zero times, etc. The base chart above the node describes the distribution of the data associated with the current node, and each data type in the current node has a corresponding chart, for example, if there are three data types in a node, there are three charts above the node. In addition, different data types correspond to different visualization methods. For example, the string type uses the category frequency column chart and the string length frequency histogram, and the array type uses the horizontal axis to represent the element size and the vertical axis to represent the maximum depth of the element, and the color-coded element frequency heat map. In order to facilitate display, a small icon will appear in the upper right corner of the chart when the user hovers the mouse over the chart, prompting the user that there is another way to display the current data type, and the user can freely switch. Then, the output of the data quality problem detection module is used to visualize the data quality problems in the current JSON data. On the one hand, the data quality dimension problems are visualized using pie charts, and the data object dimension problems are visualized using stacked column charts; on the other hand, the nodes associated with each data quality problem are wrapped by bubbles encoded with different colors according to different data quality dimensions in the tree. The visualization of the current JSON data quality problems needs to be completed after the node layout in the tree, because the position of the bubbles wrapping them can be determined after the position of the nodes associated with each data quality problem is determined. The specific display results are shown in Figure 2 ​

[0069] IV. Data cleaning module

[0070] The input of this module is the configuration parameter set by the user or recommended by the system. Based on the configuration parameter, the module cleans the JSON data input by the user and provides a preview operation before and after data cleaning, helping the user to clearly locate the source of the current data quality problem.

[0071] As one of the embodiments, when the user clicks a bubble in the aggregation mode view or directly selects the name of a data quality problem through interactive operation, the system automatically displays the parameter configuration form related to the current data quality problem. By default, for each data quality problem, there is a recommended conversion operation and its corresponding parameter configuration, which can be directly used by the user. The user can preview the data changes before and after cleaning and locate the source of the problem.

[0072] The analysis system for automatically detecting semi-structured data quality problems in the embodiment of the application is a Web-based client-server application system named JsonCurer. The server of the system implements the functions of "parsing mode", "data quality problem detection" and "data cleaning" in the above analysis method, and the client implements the "visual generation" function. As shown in Figure 2 There are three views, namely data view, mode view and data quality management view. The data quality management view is further divided into four sub-views, namely problem overview, possible problem list, problem details and data cleaning.

[0073] In the data view, the user can upload a JSON file or directly use the sample data provided by the system. After the user confirms the upload of the JSON file, the content of the JSON file is directly displayed in the data view, and then the aggregation schema tree and the visualization of data quality problems are presented in the schema view. At the same time, the data quality management view presents an overview of the data quality problems detected based on rules in the current JSON file, a problem list, and details of the currently selected data quality problem. In the schema view, the user can turn on or off the switch of the overview view to control whether to display the data distribution diagram above the node, adjust the array or dictionary, and control the merging degree of the current tree. Clicking on the data distribution diagram above the node highlights the current data in the data view (linked with the left view). Clicking on the bubble wrapped by the node locates the current data quality problem in the list of the data quality management view and matches its form (linked with the right view). In the possible problem list of the data quality management view, the user selects the data quality problem to be viewed, and the schema view automatically moves the center to the current bubble (linked with the center view). In addition, the schema view also supports zooming in and out, moving the canvas, and expanding and collapsing the nodes in the tree. In the data cleaning subgraph of the data management view, when the user configures the data conversion parameters, the user can click the preview button, and the data view will display the changes before and after the conversion of the current JSON data.

[0074] The analysis method for automatically detecting semi-structured data quality problems of the present application, as shown in Figure 1 , is implemented based on the above analysis system.

[0075] The method comprises the following steps:

[0076] Step one: the parsing schema module receives the user input of the original JSON data or the JSON data after several cleanings, and sequentially performs type inference, similarity calculation, and type aggregation on the original JSON data, and outputs an intermediate data of a tree structure describing the aggregation schema, i.e., an aggregation schema tree.

[0077] Step two: the data quality problem detection module receives the original JSON data and the aggregation schema tree, classifies the data quality from two dimensions based on the space of semi-structured data quality; and for each data quality problem in the space, correspondingly gives a method for detecting the current data quality problem based on rules, and finally obtains information including the name of the current data quality problem, the specific classification, the node code in the aggregation schema tree, and the node related data.

[0078] Step three: the visualization generation module generates a visualization view of data quality problems and schema by encoding mapping and layout generation after obtaining the results of steps one and two.

[0079] Step four: the data cleaning module cleans the original JSON data input by the user or the JSON data after several times of cleaning based on the visualized data quality problems;

[0080] Step five: the cleaned JSON data is repeatedly subjected to steps one to four until no data quality problem is detected or the iteration termination condition is reached, and high-quality JSON data is output.

[0081] Those skilled in the art can understand that the above description is only a preferred example of the application and is not used to limit the application, although the application has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions recorded in the foregoing examples or make equivalent replacement for part of the technical features. Any modification, equivalent replacement, etc. within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. An analysis system for automatically detecting quality problems in semi-structured data, characterized in that, The system includes: The parsing pattern module receives raw JSON data or JSON data that has been cleaned several times from the user, and performs type inference, similarity calculation and type aggregation on the JSON data in sequence, and outputs an intermediate data tree describing the aggregation pattern, namely the aggregation pattern tree. The data quality problem detection module receives the JSON data and the aggregated pattern tree output by the parsing mode module. Based on the semi-structured data quality space, it classifies the data quality from two dimensions. For each data quality problem in the space, it provides a rule-based method for detecting the current data quality problem, and finally obtains information including the name of the current data quality problem, its specific classification, the node code in the aggregated pattern tree, and related node data. The visualization generation module takes the outputs of the parsing pattern module and the data quality problem detection module as input, and generates a visualization view of data quality problems and patterns after encoding mapping and layout generation. The data cleaning module is used to clean the JSON data input by the user according to the configuration set by the user or the configuration recommended by the system, and provides a preview of the data before and after cleaning. The parsing mode module includes a type inference submodule, a similarity calculation submodule, and a type aggregation submodule; The type inference submodule infers the six data types of JSON data, namely: String Boolean, Null, Number, Dict, Array These six types; among them, String Boolean, Null, Number For simple data types, Dict, Array For complex data types; The similarity calculation submodule is used to calculate the similarity of the complex data types; The type aggregation submodule is used to merge two complex data types with a similarity of not less than a preset threshold based on the similarity calculation results. During the traversal process, if the current node is an array, dictionary, or simple data type, these three cases are classified and processed. In the root of JSON data, only the two cases of the root node being an array or dictionary are considered.

2. The analysis system for automatically detecting quality problems in semi-structured data according to claim 1, characterized in that, If the current node is a dictionary, then perform dictionary merging; first, group the child nodes under the current dictionary according to the type of the attribute value, and then iterate through the grouped child nodes of the current node in turn, discussing them according to the categories of dictionary, array, and simple data types: (a) If the child node is a dictionary, first merge each dictionary within the current group, and then check whether the dictionary after the internal merging can be further merged between dictionaries; if the similarity is not less than the preset threshold, then further merge between dictionaries; otherwise, do not merge between dictionaries. (b) If the child node is an array, first merge the data types within the current array, and then determine whether to further merge the merged arrays based on the similarity threshold. (c) If the child node is a simple data type, it is directly initialized with the node information of the aggregation pattern tree and put into the final result; If the current node is an array, then merge the array internally; first, group the child nodes according to the inferred array element type, and then traverse the grouped results of the current node's child nodes in turn, and discuss them according to dictionary, array, and simple data type categories: the processing flow of these three data types is the same as the processing flow of the three data types when the current node is a dictionary; If the current node is a simple data type, it is directly initialized with the node information of the aggregation pattern tree and put into the final result; at this time, the current node has no child nodes and no further aggregation is needed.

3. The analysis system for automatically detecting quality problems in semi-structured data according to claim 1, characterized in that, The similarity calculation uses the Jacquard index, with the threshold being data within the default range of 0-1 or data manually entered by the user.

4. The analysis system for automatically detecting quality problems in semi-structured data according to claim 1, characterized in that, The data quality problem detection module classifies data quality problems from two dimensions. The first dimension focuses on describing data quality, namely consistency, repeatability, redundancy, integrity, accuracy, and pattern. The second dimension focuses on describing data objects in JSON data, namely attribute keys, attribute values, attribute key-value pairs, and structure.

5. The analysis system for automatically detecting quality problems in semi-structured data according to claim 1, characterized in that, The data quality problem detection module categorizes each detected data quality problem in the space into three types: those that can be accurately detected, those that can be roughly detected, and those that are difficult to detect, as ①, ②, and ③, respectively. For data quality problems that can be accurately detected and those that can be roughly detected, the data quality problem detection module provides a rule-based detection method. For data quality problems that are difficult to detect, the user needs to select a node in the tree and declare the problem.

6. The analysis system for automatically detecting quality problems in semi-structured data according to claim 1, characterized in that, The rule-based methods for detecting current data quality problems fall into two categories: The first category is based on deterministic rules, which directly determine the outcome using an aggregated pattern tree; The second type is based on heuristic rules, which directly traverses the input JSON and combines it with the aggregation pattern tree for simultaneous judgment.

7. The analysis system for automatically detecting quality problems in semi-structured data according to claim 6, characterized in that, For each detected data quality issue, output the name of the current data quality issue and the node code in the aggregated pattern tree involved; The encoding of nodes in a tree is used to locate the nodes and the data associated with them.

8. An analytical method for automatically detecting quality problems in semi-structured data, characterized in that, This method is implemented based on the analysis system described in claim 1; The method includes the following steps: Step 1: The parsing mode module receives the raw JSON data input by the user or the JSON data after several cleanings, and performs type inference, similarity calculation and type aggregation on the raw JSON data in sequence, and outputs an intermediate data tree describing the aggregation mode, namely the aggregation mode tree. Step 2: The data quality problem detection module receives the original JSON data and the aggregation pattern tree. Based on the semi-structured data quality space, it classifies the data quality from two dimensions. For each data quality problem in the space, it provides a rule-based method for detecting the current data quality problem, and finally obtains information including the name of the current data quality problem, its specific classification, the node code in the aggregation pattern tree involved, and the node-related data. Step 3: The visualization generation module generates a visualization view of data quality problems and patterns by encoding mapping and layout generation of the results obtained in Step 1 and Step 2. Step 4: The data cleaning module performs cleaning operations on the user-input raw JSON data or JSON data that has undergone several cleaning cycles based on visualized data quality issues; Step 5: Repeat steps 1 through 4 with the cleaned JSON data until no data quality issues are detected or the iteration termination condition is met, and output high-quality JSON data.

Images