Method for detecting data modeling quality
By transforming technical and business specifications into structured rules and generating abstract syntax trees, we automatically check the quality of data modeling, and solve the problems of inefficiency and relying on manual audits in the existing technology, and achieve efficient and reliable data modeling quality control .
Patent Information
- Application Number
- CN202411105740.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-29
Smart Images

Figure CN119046151A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data modeling in IT software industry, and in particular to a method for detecting the quality of data modeling. Background Art
[0002] In the field of modern software development and database management, data modeling, as one of the core processes, plays a vital role. Data modeling is not only related to the structural design of information systems, but also directly affects the performance and stability of the system. A good data model can ensure the consistency, accuracy and accessibility of data, while a poor quality data model may lead to inefficient data processing and difficult maintenance.
[0003] With the rapid development of information technology, the industry has higher and higher requirements for data modeling, which not only includes the rationality of model design, but also involves data security, privacy protection, multi-platform compatibility, etc. Therefore, ensuring the quality of data modeling has become an important part of the database design and software development process.
[0004] At present, in the process of software development, data modeling is needed to complete the conversion of abstract data involved in the business into concrete database object data. In the process of modeling, certain business and technical specifications must be followed to ensure that the final data structure does not have known defects. Usually, after the data modeling is completed, it needs to be reviewed and tested by a dedicated person to discover and correct the problems in the design process, but these all require additional costs and rely on the technical level of the auditors. There is no guarantee that the data modeling will ultimately produce high-quality results. Therefore, in response to the above problems, a method for detecting the quality of data modeling is proposed. Summary of the invention
[0005] In view of the deficiencies in the prior art, the present invention provides a method for detecting data modeling quality, which solves the problems of low efficiency, high cost, reliance on manual review and difficulty in ensuring the final modeling quality in traditional data modeling quality control.
[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: A method for detecting data modeling quality, comprising:
[0007] S1. Organize the technical and business specifications within the industry or enterprise into preset rules, and classify each rule into multiple units;
[0008] S2, read the modeling script, perform lexical analysis to generate an abstract syntax tree containing key attribute information;
[0009] S3, processing the syntax tree, removing irrelevant content and classifying it, forming a set of items to be checked consisting of multiple units;
[0010] S4. Sequentially check the set of items to be checked and confirm defects for units that do not meet the preset rules.
[0011] Preferably, the syntax tree in step S2 includes all modeled table information, field information, and key information.
[0012] Preferably, step S1 specifically includes the following steps:
[0013] S1.1. Analyze and sort out industry standards and enterprise-specific technical and business requirements;
[0014] S1.2. Convert these requirements into structured rules, each of which targets a specific type of modeling problem or requirement.
[0015] S1.3. Break down rules into actionable units, with each unit focusing on a specific data modeling attribute or error type.
[0016] Preferably, step S2 specifically includes the following steps:
[0017] S2.1, read and parse the script files generated during the data modeling process;
[0018] S2.2. Use a lexical analyzer to analyze the script content and identify basic lexical elements;
[0019] S2.3. Build a syntax tree to capture and record the structure of each data table.
[0020] Preferably, step S3 specifically includes the following steps:
[0021] S3.1, perform secondary processing on the parsed syntax tree to remove irrelevant data content;
[0022] S3.2, segment and classify the remaining grammatical data according to the requirements of the preset rules;
[0023] S3.3. Form a set of items to be checked, each of which is a minimum data modeling element to be quality checked.
[0024] Preferably, step S4 specifically includes the following steps:
[0025] S4.1, sequentially checking the item set one by one, including obtaining the type of the minimum unit to be checked and applying the corresponding preset rules to each unit;
[0026] S4.2. Select corresponding multiple preset rule units from the preset rules according to the type, and perform a matching check on the minimum unit to be checked;
[0027] S4.3. If the inspection result does not conform to the preset rules, it is determined that there is a defect, otherwise it is considered normal.
[0028] A device for detecting data modeling quality, comprising:
[0029] A syntax extraction module is used to parse the data in the modeling output script file and extract syntax tree data from it;
[0030] The grammar formatting module is connected to the grammar extraction module and is used to filter the grammar tree grammar data, simplify, segment and classify the grammar data, and output a data set that conforms to the rule structure;
[0031] The rule detection module is connected to the grammar formatting module and is used to check the set of items to be checked item by item according to the preset rules, and output the existing content that does not conform to the preset rules.
[0032] The present invention provides a method for detecting the quality of data modeling. It has the following beneficial effects:
[0033] 1. The present invention detects the quality of data modeling in an automated manner, significantly reducing labor costs and time costs. The automated process reduces human errors, improves the consistency and reliability of detection, lowers the technical threshold for auditors and testers, and improves work efficiency.
[0034] 2. The present invention converts technical and business specifications into structured preset rules and refines them into operational units. This structured processing makes the rules easy to apply and update, while ensuring the systematic and comprehensive nature of the detection process.
[0035] 3. The method and apparatus of the present invention allow easy maintenance and updating of rules, support flexible rule expansion, and are critical for coping with rapidly changing business environments and technology updates. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 is a flow chart of the method of the present invention;
[0037] Figure 2 It is a framework diagram of the device of the present invention.
[0038] Among them, 201, grammar extraction module; 2, grammar formatting module; 3, rule detection module. DETAILED DESCRIPTION
[0039] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0040] Example:
[0041] Please refer to the attached Figure 1 , an embodiment of the present invention provides a method for detecting data modeling quality, comprising the following steps:
[0042] S1. Organize the technical and business specifications within the industry or enterprise into preset rules, and classify each rule into multiple units;
[0043] S2, read the modeling script, perform lexical analysis to generate an abstract syntax tree containing key attribute information;
[0044] S3, processing the syntax tree, removing irrelevant content and classifying it, forming a set of items to be checked consisting of multiple units;
[0045] S4. Sequentially check the set of items to be checked and confirm defects for units that do not meet the preset rules.
[0046] Specifically, in step S1, complex technical and business specifications are converted into structured preset rules and classified into multiple units. Each rule unit corresponds to a specific business scenario or technical requirement to ensure the comprehensiveness and applicability of the rules.
[0047] Through structuring and classification, the readability and operability of the rules are enhanced, making it easier for subsequent automated tools to apply these rules for data modeling quality detection.
[0048] In step S2, the modeling script file is parsed, and the lexical analyzer is used to extract key attributes, such as attribute information of tables, fields, and keys, and a detailed syntax tree is generated.
[0049] The syntax tree provides a structured representation of the modeling script, which facilitates the rapid and accurate extraction and analysis of the components of the data model, and improves the efficiency and accuracy of subsequent processing.
[0050] Lexical analysis converts text input into tokens, which are then further organized into a syntax tree, a structure that makes static analysis and manipulation of the program more direct and simple.
[0051] In step S3, the generated syntax tree is cleaned and classified to remove irrelevant content and classify it according to function or structure to form a clearly divided set of items to be checked.
[0052] The inspection process is simplified, ensuring that the inspection focus is on key and relevant data structures, and improving the accuracy and efficiency of the inspection.
[0053] Through screening and segmentation, each unit to be inspected is precisely defined, reducing the possibility of false positives and missed negatives, so that each step of the inspection can correspond to the preset rules in a targeted manner.
[0054] In step S4, according to the classified set of items to be checked, each unit is systematically checked one by one to compare whether it complies with the preset rules.
[0055] Automated checks reduce the need for manual review, improve the speed and accuracy of checks, and quickly identify and flag parts of data models that do not meet specifications.
[0056] Using a rule-driven approach, the system verifies each data unit according to predefined logic and parameters, thereby effectively identifying and marking those elements that do not meet the specifications.
[0057] Step S1 specifically includes the following steps:
[0058] S1.1. Analyze and sort out industry standards and enterprise-specific technical and business requirements;
[0059] S1.2. Convert these requirements into structured rules, each of which targets a specific type of modeling problem or requirement.
[0060] S1.3. Break down rules into actionable units, with each unit focusing on a specific data modeling attribute or error type.
[0061] Specifically, in step S1, we will conduct an in-depth analysis of the industry-recognized standards and the technical and business specifications formed by long-term internal practices of the enterprise. We will ensure a comprehensive understanding and accurate grasp of these requirements by collecting and evaluating industry literature, specifications issued by standard-setting organizations, and historical data model cases of the enterprise.
[0062] Ensure that the rules developed are consistent with current industry best practices and enterprise-specific needs, and enhance the applicability and practicality of the data model.
[0063] Through a comprehensive analysis of industry and corporate specifications, we can ensure that the rule set is technologically advanced and effective in practical applications, thus avoiding possible deviations and errors in the modeling process.
[0064] In step S1.2, abstract business and technical requirements are concretized into clear checkpoints and parameters, and each rule targets a specific modeling scenario, such as data type standards, field length restrictions, or keyword usage specifications.
[0065] After being converted into structured rules, it is easier for automated tools to execute, reducing the uncertainty of human judgment and improving the standardization and consistency of inspections.
[0066] Clear rules are easy for algorithms to parse and apply, and systematic checking of these rules through automated tools can accurately identify and correct deviations, thereby ensuring the quality and consistency of data models.
[0067] In step S1.3, the rules are refined into actionable units, such as field naming conventions, data type compatibility, index configuration best practices, etc. Each unit has clear inspection criteria and expected results.
[0068] This refinement allows each inspection unit to be operated and verified independently, improving the accuracy of problem location and the efficiency of correction.
[0069] The subdivided operation units are classified according to specific data modeling attributes, allowing specialized treatment of various types of problems. This approach improves the pertinence of detection and correction, reduces the risk of overall modeling errors, and facilitates subsequent maintenance and rule updates.
[0070] And in the above steps S1.1-S1.3, the modeling script will be extracted and parsed into an abstract syntax tree structure as defined in Table 1.
[0071] Table 1
[0072]
[0073]
[0074] The S2 step specifically includes the following steps:
[0075] S2.1, read and parse the script files generated during the data modeling process;
[0076] S2.2. Use a lexical analyzer to analyze the script content and identify basic lexical elements;
[0077] S2.3. Build a syntax tree to capture and record the structure of each data table.
[0078] The syntax tree in step S2 includes all the table information, field information, and key information of the model.
[0079] Specifically, step S2.1 involves opening and reading the data modeling script file, which includes identifying the file format, encoding, and accurately reading the file content for subsequent analysis.
[0080] Ensure that all modeling data is loaded into the analysis tool accurately and without errors, providing the basis for subsequent lexical and grammatical analysis.
[0081] By programmatically reading script files, large amounts of data can be processed automatically, reducing manual input errors and improving processing efficiency and data accuracy.
[0082] In step S2.2, the lexical analyzer breaks down the script content into basic lexical units, such as keywords, identifiers, operators, etc. This process involves scanning the code, identifying and classifying the words and symbols in the code.
[0083] This step systematically converts the script text into a structure that is easy to process further, laying the foundation for building a syntax tree.
[0084] Lexical analysis is the first step in the compilation process, and its purpose is to simplify the design of the compiler and improve its efficiency. By converting the input text into tokens, the complexity of syntax analysis is reduced and the efficiency and accuracy of parsing are improved.
[0085] In step S2.3, the lexical analysis results are used to construct an abstract syntax tree representing the data model structure. In the abstract syntax tree, each node represents a structure, such as a data table, field, and key. This tree structure reflects the hierarchical relationship and logical structure of the database model.
[0086] The abstract syntax tree provides a clear, hierarchical view of the data, making subsequent analysis, inspection, and transformation more intuitive and efficient.
[0087] A syntax tree is a tree-like data structure that represents the output of syntax analysis and describes the grammatical structure of the input text. By constructing a syntax tree, semantic analysis and code generation can be performed efficiently because it clearly shows the relationships and dependencies between data structures.
[0088] The S3 step specifically includes the following steps:
[0089] S3.1, perform secondary processing on the parsed syntax tree to remove irrelevant data content;
[0090] S3.2, segment and classify the remaining grammatical data according to the requirements of the preset rules;
[0091] S3.3. Form a set of items to be checked, each of which is a minimum data modeling element to be quality checked.
[0092] Specifically, step S3.1 involves checking the syntax tree generated by parsing and removing nodes that are not relevant to modeling quality inspection, such as comments, auxiliary codes, or metadata. Such processing helps to simplify the syntax tree and focus on the core data structure.
[0093] By eliminating unnecessary content, the complexity and computational burden of subsequent processing steps can be reduced, and the speed and efficiency of data processing can be improved.
[0094] Streamlined data structures are easier to manage and analyze, optimizing performance by reducing the amount of data processed, while also reducing the chance of errors and ensuring that the detection focus always remains on key elements.
[0095] In step S3.2, the elements in the abstract syntax tree, such as data tables, fields, and constraints, are systematically segmented and classified according to preset rules. This process ensures that each element is classified into the appropriate inspection category, such as data integrity, field validity, etc.
[0096] Classification processing enables each type of data model element to correspond to specific rule tests, improving the accuracy and relevance of the tests.
[0097] By classifying the elements of the data model, different validation logic can be applied according to different categories. This approach improves the pertinence and efficiency of rule verification because the problems and risks faced by data elements in each category may be different.
[0098] In step S3.3, key elements, such as specific field or table definitions, are extracted from the grammatical data after classification and organized into a set of items to be checked. Each item is a basic unit in the inspection process and is specifically evaluated for its quality.
[0099] This step provides clear inspection objectives and boundaries, allowing the automated inspection process to identify problems in a targeted manner, thereby improving the efficiency and quality of processing.
[0100] By breaking down the data model into its smallest elements, the most appropriate check rule can be applied to each element, ensuring that each checkpoint is fully evaluated. This detailed inspection helps improve the overall quality and consistency of the data model.
[0101] The S4 step specifically includes the following steps:
[0102] S4.1, sequentially checking the item set one by one, including obtaining the type of the minimum unit to be checked and applying the corresponding preset rules to each unit;
[0103] S4.2. Select corresponding multiple preset rule units from the preset rules according to the type, and perform a matching check on the minimum unit to be checked;
[0104] S4.3. If the inspection result does not conform to the preset rules, it is determined that there is a defect, otherwise it is considered normal.
[0105] Specifically, step 4.1 involves traversing the sorted set of items to be checked, identifying the specific type of each unit, and applying the preset rules for that type for preliminary inspection. This process ensures that each element is appropriately evaluated according to its characteristics.
[0106] Through typed checking, the most relevant rules can be applied more accurately, unnecessary checks can be reduced, and the efficiency and accuracy of checking can be improved.
[0107] Categorizing inspection items and applying specific rules can optimize the inspection process, because each type of data element may face different issues and constraints. This approach improves the targeting and effectiveness of inspections.
[0108] In step S4.2, select the applicable rule unit from the preset rule base according to the type of each unit. For example, for a data field, you may need to check its data type, length, whether null values are allowed, and other attributes. This step ensures that each check covers all relevant specification requirements.
[0109] Careful rule matching ensures that each unit is fully evaluated, increasing the likelihood of discovering potential problems and thus improving the overall quality of the data model.
[0110] By matching specific rule units for inspection, we can ensure that the evaluation process is both comprehensive and specific, avoiding details that may be overlooked by overly general inspections.
[0111] In step S4.3, after applying all relevant preset rules, the inspection results are evaluated. If the data unit is found to be inconsistent with the applied rules, it is marked as defective; if it is completely compliant, the unit is considered normal.
[0112] This step provides clear quality control results, which helps to quickly identify and correct defects in the data model and ensure that the quality of the data model meets the predetermined standards.
[0113] By comparing the check results with the preset rules, the data model's conformance can be objectively evaluated. This comparison is based on clear criteria, which helps to accurately identify problems and ensure the overall consistency and accuracy of the data model.
[0114] Please see attached Figure 2 , a device for detecting data modeling quality, comprising:
[0115] A syntax extraction module 201 is used to parse the data in the modeling output script file and extract syntax tree data therefrom;
[0116] The grammar formatting module 202 is connected to the grammar extraction module 201 and is used to filter the grammar tree grammar data, simplify, segment and classify the grammar data, and output a data set that conforms to the rule structure;
[0117] The rule detection module 203 is connected to the grammar formatting module 202, and is used to check the set of items to be checked item by item according to the preset rules, and output the existing content that does not conform to the preset rules.
[0118] Specifically, the grammar extraction module 201, the grammar formatting module 202, and the pattern detection module 203 are only divisions of the logical processing units, and can be physically combined or independent.
[0119] The syntax extraction module 201 is mainly used to read and analyze the modeling script, extract all table attribute information, all field attribute information and all key attribute information therein, decompose them into logical tags, and output them in the form of an abstract syntax tree;
[0120] The grammar formatting module 202 mainly performs secondary processing on the results of the grammar extraction module, removes useless data, simplifies, segments and classifies the remaining grammar, and outputs it as a data set that is easy to process.
[0121] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for detecting data modeling quality, characterized in that: The following steps are involved: S1. Organize the technical and business specifications within the industry or enterprise into preset rules, and classify each rule into multiple units; S2, read the modeling script, perform lexical analysis to generate an abstract syntax tree containing key attribute information; S3, processing the syntax tree, removing irrelevant content and classifying it, forming a set of items to be checked consisting of multiple units; S4. Sequentially check the set of items to be checked and confirm defects for units that do not meet the preset rules.
2. A method for detecting data modeling quality according to claim 1, characterized in that: The syntax tree in step S2 includes all table information, field information, and key information of the model.
3. The method for detecting data modeling quality according to claim 1, characterized in that: The S1 step specifically includes the following steps: S1.
1. Analyze and sort out industry standards and enterprise-specific technical and business requirements; S1.
2. Convert these requirements into structured rules, each of which targets a specific type of modeling problem or requirement. S1.
3. Break down rules into actionable units, with each unit focusing on a specific data modeling attribute or error type.
4. The method for detecting data modeling quality according to claim 1, characterized in that: The S2 step specifically includes the following steps: S2.1, read and parse the script files generated during the data modeling process; S2.
2. Use a lexical analyzer to analyze the script content and identify basic lexical elements; S2.
3. Build a syntax tree to capture and record the structure of each data table.
5. The method for detecting data modeling quality according to claim 1, characterized in that: The S3 step specifically includes the following steps: S3.1, perform secondary processing on the parsed syntax tree to remove irrelevant data content; S3.2, segment and classify the remaining grammatical data according to the requirements of the preset rules; S3.
3. Form a set of items to be checked, each of which is a minimum data modeling element to be quality checked.
6. The method for detecting data modeling quality according to claim 1, characterized in that: The S4 step specifically includes the following steps: S4.1, sequentially checking the item set one by one, including obtaining the type of the smallest unit to be checked, and applying the corresponding preset rules to each unit; S4.
2. Select corresponding multiple preset rule units from the preset rules according to the type, and perform a matching check on the minimum unit to be checked; S4.
3. If the inspection result does not conform to the preset rules, it is determined that there is a defect, otherwise it is considered normal.
7. A device for detecting data modeling quality, characterized in that: A method for detecting data modeling quality in cooperation with any one of claims 1 to 6, comprising: A syntax extraction module (201), which is used to parse the data in the modeling output script file and extract syntax tree data therefrom; A grammar formatting module (202), which is connected to the grammar extraction module (201), is used to filter the grammar tree grammar data, simplify, segment and classify the grammar data, and output a data set that conforms to the rule structure; The rule detection module (203) is connected to the grammar formatting module (202) and is used to check the set of items to be checked item by item according to the preset rules and output the existing content that does not conform to the preset rules.
