Method and system for automatic generation and execution of data checking rules based on large models

Abstract

The application provides a large model-based data check rule automatic generation and execution method and system, relates to the technical field of data quality management, and comprises the following steps: obtaining a to-be-checked data set and description information, generating a field relation graph and an initial check rule set, correcting the rules based on data value distribution characteristics, obtaining a target rule set through sample set evaluation and screening, and finally executing a check operation based on a hierarchical check topology structure.The application can automatically adapt to data characteristics to dynamically generate check rules, improve the accuracy and efficiency of data checking, and reduce the cost of manual intervention.

Description

Technical Field

[0001] This invention relates to the field of data quality management technology, and in particular to a method and system for automatically generating and executing data verification rules based on a large model. Background Technology

[0002] Data quality is fundamental to big data applications, and data validation, as a crucial step in ensuring data quality, plays a vital role in data processing, analysis, and application. With the continuous expansion of data scale and increasing data complexity, traditional manually written data validation rules are no longer sufficient to meet the demands of modern data processing. Traditional data validation typically relies on domain experts manually designing rules or using simple statistical methods for anomaly detection. While these methods have achieved some success in processing structured data, they face numerous challenges as business complexity and data diversification increase. Especially when dealing with large-scale, multi-source, and heterogeneous data, the relationships and constraints between data become extremely complex, making it difficult to cover all possible data anomalies simply by relying on manually defined rules.

[0003] However, existing technologies still have shortcomings. Existing automated validation rule generation methods lack a deep understanding of the complex dependencies between data fields, making it difficult to discover and express logical constraints and business rules between multiple fields. This results in insufficient coverage of the generated validation rules and an inability to identify complex anomalies involving multiple field relationships. Traditional validation rules often use fixed boundary values ​​or pattern matching, lacking the ability to adaptively learn from data distribution characteristics. When the data distribution changes, they cannot dynamically adjust rule parameters, causing a large number of false positives or false negatives, reducing the accuracy and reliability of validation. Existing technologies lack evaluation mechanisms and optimization strategies for validation rule quality, making it impossible to effectively filter and prioritize rules. This leads to wasted computing resources and low efficiency during the validation process, especially when dealing with massive amounts of data, where validation performance becomes a bottleneck restricting data processing efficiency.

[0004] In conclusion, there is an urgent need for a technical solution that can automatically understand data structures, generate high-quality verification rules, and execute them efficiently, which has significant practical and theoretical value. Summary of the Invention

[0005] This invention provides a method and system for automatically generating and executing data verification rules based on a large model, which can solve the problems in the prior art.

[0006] A first aspect of this invention provides a method for automatically generating and executing data validation rules based on a large model, comprising:

[0007] Obtain the dataset to be verified and its corresponding data pattern description information;

[0008] The data pattern description information is parsed to generate a field relationship graph. An initial verification rule set is generated based on the transmission path of the directed edges and the constraint attributes of the nodes in the field relationship graph. The value distribution characteristics of each field in the dataset to be verified are extracted, the deviation between the value distribution characteristics and the constraint conditions in the initial verification rule set is calculated, and the constraint boundary is adaptively adjusted to generate a modified verification rule set.

[0009] For each rule in the modified verification rule set, construct a positive sample set and a negative sample set from the dataset to be verified, calculate the data coverage and anomaly detection rate, determine the rule quality score, and filter and generate the target verification rule set based on the rule quality score;

[0010] The input and output fields of the rules in the target validation rule set are analyzed to construct a directed dependency graph. Based on the topological sorting of the directed dependency graph, the directed paths are linked into an execution chain. Isolated nodes and rule subsets without directed path connections are combined into parallel execution units to construct a hierarchical validation topology structure.

[0011] The verification operation is performed on the dataset to be verified based on the hierarchical verification topology, and the verification result is obtained.

[0012] In one optional embodiment, parsing the data pattern description information to generate a field relationship graph, and generating an initial validation rule set based on the propagation paths of directed edges and the constraint attributes of nodes in the field relationship graph includes:

[0013] The data schema description information is parsed to extract field identifiers and type declarations. Field identifiers are mapped to nodes in the field relationship graph, and type declarations are converted into constraint attributes of the corresponding nodes. Foreign key constraints and reference relationships in the data schema description information are identified, and directed edges are established between the corresponding nodes.

[0014] Loop detection is performed on the directed edges in the field relationship graph to identify the set of edges that form a closed loop. Constraint attributes are extracted for the nodes in the set of edges, and loop consistency verification rules are generated based on the mutual exclusion and compatibility of the constraint attributes.

[0015] Perform a depth-first traversal on the field relationship graph, record the node sequence and edge sequence traversed during the traversal, calculate the cumulative strength of the constraint attributes in the node sequence, and when the cumulative strength exceeds a preset strength threshold, establish constraint propagation rules between the start and end points of the edge sequence.

[0016] Identify source nodes with zero in-degree and sink nodes with zero out-degree in the field relationship graph, extract all connected paths between the source nodes and sink nodes, combine the constraint attributes on each connected path, and generate end-to-end full-link verification rules.

[0017] The loop consistency verification rule, the constraint propagation rule, and the end-to-end verification rule are merged to form an initial verification rule set.

[0018] In one optional embodiment, the process of extracting the value distribution characteristics of each field in the dataset to be verified, calculating the deviation between the value distribution characteristics and the constraints in the initial verification rule set, and adaptively adjusting the constraint boundaries to generate a modified verification rule set includes:

[0019] Each field in the dataset to be verified is scanned, all values ​​of the field are extracted to construct a value set, and sorted by numerical size to generate an ordered value sequence. The interval distance between adjacent values ​​in the ordered value sequence is calculated, and the position where the interval distance exceeds the preset interval threshold is identified as the segment point.

[0020] Extract the constraints corresponding to each field from the initial set of validation rules, and obtain the minimum and maximum allowed values;

[0021] Based on the ordered value sequence, count the number of underflow values ​​less than the minimum allowed value and the number of overflow values ​​greater than the maximum allowed value. Calculate the proportion of the number of underflow values ​​and the number of overflow values ​​to the total number of elements in the value set, and determine the lower deviation and upper deviation. When the lower deviation is greater than the upper deviation, expand the minimum allowed value downwards; when the upper deviation is greater than the lower deviation, expand the maximum allowed value upwards.

[0022] The ordered value sequence is divided into multiple subsequences according to the segmentation points. The number of values ​​corresponding to each subsequence is calculated. The subsequence with the most values ​​is selected to determine the main distribution interval. Independent constraint boundaries are set for the main distribution interval.

[0023] The expanded minimum allowable value, maximum allowable value, and independent constraint boundary are updated to the initial verification rule set to generate the modified verification rule set.

[0024] In one optional embodiment, for each rule in the modified verification rule set, a positive sample set and a negative sample set are constructed from the dataset to be verified, the data coverage and anomaly detection rate are calculated, a rule quality score is determined, and a target verification rule set is generated based on the rule quality score, including:

[0025] Iterate through each rule in the set of corrected verification rules, parse the constraint boundaries in the rules, and select data records whose values ​​are within the constraint boundaries from the dataset to be verified to form a positive sample set, and data records whose values ​​are outside the constraint boundaries to form a negative sample set.

[0026] Extract all field values ​​of each data record in the positive sample set and combine them into a positive sample feature vector. Perform a convex hull algorithm on the positive sample feature vector in the vector space to obtain a geometric envelope surface. Construct a regular boundary surface based on the constraint boundary. Calculate the spatial deviation between the geometric envelope surface and the regular boundary surface to obtain the regular fitting error. Determine the corrected data coverage based on the regular fitting error.

[0027] Extract all field values ​​of each data record in the negative sample set and combine them into a negative sample feature vector. Use each negative sample feature vector as a spatial coordinate point to calculate the shortest Euclidean distance to the geometric envelope. Based on the shortest Euclidean distance, determine the near-boundary anomaly set and the far-boundary anomaly set. Divide the geometric envelope into directional sectors, identify high-voltage directional sectors, determine boundary leakage, and determine the corrected anomaly identification rate.

[0028] The rule quality score is obtained by weighted summing of the corrected data coverage rate and the corrected anomaly detection rate, and the target verification rule set is generated by filtering.

[0029] In one optional embodiment, the near-boundary anomaly set and the far-boundary anomaly set are determined based on the shortest Euclidean distance, the geometric envelope is divided into directional sectors, high-pressure directional sectors are identified, boundary leaks are determined, and the corrected anomaly identification rate is determined, including:

[0030] Calculate the shortest Euclidean distance from the feature vector of each negative sample in the negative sample set to the geometric envelope. Statistically count all the shortest Euclidean distances and calculate the median to determine the distance threshold. Divide the negative samples whose shortest Euclidean distance is less than the distance threshold into the near-boundary outlier set and the remaining negative samples into the far-boundary outlier set.

[0031] The geometric envelope is meshed into a spherical grid. Based on the outward normal direction of the envelope, the space is divided into multiple directional sectors. The number of negative samples in the near-boundary outlier set in each directional sector is counted. The boundary pressure value of the directional sector is determined by calculating the ratio of the number of negative samples to the total number of near-boundary outliers.

[0032] High-pressure directional sectors are identified by identifying directional sectors where the boundary pressure value exceeds a preset pressure threshold. Multiple surface points are sampled within the envelope area corresponding to each high-pressure directional sector. The distance from each surface point to the nearest negative sample is calculated. The number of surface points whose distance is less than a preset permeability threshold is counted. The ratio of the number of surface points to the total number of sampling points is calculated to determine the boundary permeability.

[0033] The weighted average of the boundary permeability of all high-pressure sector sectors is calculated as the global boundary permeability. An anomaly identification rate attenuation coefficient is calculated based on the global boundary permeability. The original anomaly identification rate is multiplied by the attenuation coefficient to obtain the corrected anomaly identification rate.

[0034] In one optional embodiment, a directed dependency graph is constructed by parsing the input and output fields of the rules in the target validation rule set. Based on the topological sorting of the directed dependency graph, directed paths are chained together into execution chains. Isolated nodes and subsets of rules without directed path connections are combined into parallel execution units. The hierarchical validation topology structure is constructed by:

[0035] Traverse each rule in the target validation rule set, parse the input and output fields in the rule, and for any two rules, if the output field of the first rule and the input field of the second rule have an intersection, then build a dependency relationship between the first rule and the second rule, mark the first rule as the predecessor rule, and mark the second rule as the successor rule.

[0036] A directed dependency graph is constructed based on the dependency relationship between predecessor and successor rules. Each rule is treated as a node in the directed dependency graph, and the dependency relationship is treated as a directed edge. A topological sort is performed on the directed dependency graph to identify all directed paths in the directed dependency graph. The rules in each directed path are then chained together in the topological sort order to form an execution chain.

[0037] Identify isolated nodes with zero in-degree and zero out-degree in the directed dependency graph, as well as rule subsets without directed path connections, and merge the rules corresponding to the isolated nodes with the rules in the rule subset into parallel execution units.

[0038] The rules in the execution chain are assigned to the corresponding level depth according to the topological sorting order. The longest path length from the starting node to the current node of each rule is calculated to determine the level number of the rule. The rules in the parallel execution unit are assigned to the first level. The rules in the execution chain are assigned to the subsequent levels in ascending order of level number to construct a hierarchical verification topology structure.

[0039] A second aspect of this invention provides a system for automatically generating and executing data validation rules based on a large model, comprising:

[0040] The first unit is used to obtain the dataset to be verified and the corresponding data pattern description information;

[0041] The second unit is used to parse the data pattern description information to generate a field relationship graph, generate an initial verification rule set based on the transmission path of the directed edges and the constraint attributes of the nodes in the field relationship graph; extract the value distribution characteristics of each field in the dataset to be verified, calculate the deviation between the value distribution characteristics and the constraint conditions in the initial verification rule set, and adaptively adjust the constraint boundaries to generate a modified verification rule set.

[0042] The third unit is used to construct positive and negative sample sets from the dataset to be verified for each rule in the modified verification rule set, calculate the data coverage and anomaly detection rate, determine the rule quality score, and filter and generate the target verification rule set based on the rule quality score.

[0043] The fourth unit is used to parse the input and output fields of the rules in the target validation rule set to construct a directed dependency graph. Based on the topological sorting of the directed dependency graph, the directed paths are linked into an execution chain. Isolated nodes and rule subsets without directed path connections are combined into parallel execution units to construct a hierarchical validation topology structure.

[0044] The fifth unit is used to perform verification operations on the dataset to be verified based on the hierarchical verification topology and obtain the verification results.

[0045] A third aspect of the present invention provides an electronic device, comprising:

[0046] processor;

[0047] Memory used to store processor-executable instructions;

[0048] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0049] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0050] In this embodiment of the invention, a field relationship graph is generated by parsing data pattern description information, and the rules are adaptively adjusted in combination with the actual data distribution characteristics. This ensures that the generated verification rules not only conform to business semantic constraints but also adapt to the actual data distribution characteristics, greatly improving the accuracy and applicability of the verification rules and reducing the workload of manually defining rules. By constructing positive and negative sample sets to evaluate the quality of the rules and selecting target rule sets based on the scoring results, the final executed verification rules are ensured to have high data coverage and anomaly detection rate, effectively avoiding rule redundancy and misjudgment, and improving the accuracy and reliability of data verification. A hierarchical verification topology structure is constructed based on the dependencies between rules, optimizing the execution order and parallelism of verification rules. This ensures the correctness of the verification logic and improves the execution efficiency of the verification process, adapting to the high-efficiency verification needs of large-scale datasets and reducing the cost of data quality management. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the automatic generation and execution method of data validation rules based on a large model according to an embodiment of the present invention;

[0052] Figure 2 The flowchart shows the parallel processing of the two paths for rule quality assessment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0055] Figure 1 This is a flowchart illustrating the automatic generation and execution method of data validation rules based on a large model according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0056] Obtain the dataset to be verified and its corresponding data pattern description information;

[0057] The data pattern description information is parsed to generate a field relationship graph. An initial verification rule set is generated based on the transmission path of the directed edges and the constraint attributes of the nodes in the field relationship graph. The value distribution characteristics of each field in the dataset to be verified are extracted, the deviation between the value distribution characteristics and the constraint conditions in the initial verification rule set is calculated, and the constraint boundary is adaptively adjusted to generate a modified verification rule set.

[0058] For each rule in the modified verification rule set, construct a positive sample set and a negative sample set from the dataset to be verified, calculate the data coverage and anomaly detection rate, determine the rule quality score, and filter and generate the target verification rule set based on the rule quality score;

[0059] The input and output fields of the rules in the target validation rule set are analyzed to construct a directed dependency graph. Based on the topological sorting of the directed dependency graph, the directed paths are linked into an execution chain. Isolated nodes and rule subsets without directed path connections are combined into parallel execution units to construct a hierarchical validation topology structure.

[0060] The verification operation is performed on the dataset to be verified based on the hierarchical verification topology, and the verification result is obtained.

[0061] In one optional implementation, the data pattern description information is parsed to generate a field relationship graph, and an initial validation rule set is generated based on the transmission paths of directed edges and the constraint attributes of nodes in the field relationship graph, including:

[0062] The data schema description information is parsed to extract field identifiers and type declarations. Field identifiers are mapped to nodes in the field relationship graph, and type declarations are converted into constraint attributes of the corresponding nodes. Foreign key constraints and reference relationships in the data schema description information are identified, and directed edges are established between the corresponding nodes.

[0063] Loop detection is performed on the directed edges in the field relationship graph to identify the set of edges that form a closed loop. Constraint attributes are extracted for the nodes in the set of edges, and loop consistency verification rules are generated based on the mutual exclusion and compatibility of the constraint attributes.

[0064] Perform a depth-first traversal on the field relationship graph, record the node sequence and edge sequence traversed during the traversal, calculate the cumulative strength of the constraint attributes in the node sequence, and when the cumulative strength exceeds a preset strength threshold, establish constraint propagation rules between the start and end points of the edge sequence.

[0065] Identify source nodes with zero in-degree and sink nodes with zero out-degree in the field relationship graph, extract all connected paths between the source nodes and sink nodes, combine the constraint attributes on each connected path, and generate end-to-end full-link verification rules.

[0066] The loop consistency verification rule, the constraint propagation rule, and the end-to-end verification rule are merged to form an initial verification rule set.

[0067] In one specific implementation, when parsing the data pattern description information, a combination of regular expressions and a recursive descent parser is used to process the data pattern description text line by line. For example, when parsing "customer_idINT NOT NULL PRIMARY KEY", the field identifier "customer_id" and the type declaration "INT NOTNULL PRIMARY KEY" are extracted. The field identifier is mapped to nodes in the field relationship graph, and the type declaration is converted into the constraint attributes of the nodes, such as data type constraint (INT), NOT NULL constraint (NOT NULL), primary key constraint (PRIMARY KEY), etc. For foreign key constraint declarations such as "order_id INT REFERENCES orders(id)", it is identified that "order_id" references the "id" field of the "orders" table, and directed edges are established between the corresponding nodes. The attributes of the edges include reference type, cascading operation, and other information.

[0068] After the field relationship graph is constructed, cycle detection is performed on the directed edges in the graph. A depth-first search algorithm is used to mark visited nodes. When a node is found to be visited repeatedly, a cycle is identified. Assume the following relationship exists in the data schema: the `manager_id` table in the department table references the `employee_id` table in the employee table, and the `department_id` table in the employee table references the `id` table in the department table, thus forming a cycle. For the nodes in the cycle (department.id, department.manager_id, employee.employee_id, employee.department_id), their constraint attributes are extracted, such as `department.id` (INT, PRIMARY KEY, NOT NULL), `department.manager_id` (INT, FOREIGN KEY, NULL), `employee.employee_id` (INT, PRIMARY KEY, NOT NULL), and `employee.department_id` (INT, FOREIGN KEY, NOT NULL). Based on the mutual exclusion (e.g., NULL and NOT NULL are mutually exclusive) and compatibility (e.g., INT and INT are compatible) analysis of these constraint attributes, a loop consistency check rule is generated: "When the manager_id in the department table has a value, the department_id in the corresponding employee table record must point to the current department, forming a consistent management relationship closed loop."

[0069] A depth-first traversal is performed on the field relationship graph, recording the node and edge sequences during the traversal. The traversal starts from each node, visits its adjacent nodes, and proceeds recursively until it cannot continue. Assuming it starts from the `category_id` field in the `product` table, passes through the `id` and `parent_id` fields in the `category` table, and finally reaches the `id` field of another category record, forming the node sequence: `product.category_id` → `category.id` → `category.parent_id` → `category.id`. Simultaneously, the cumulative constraint strength along the path is calculated. The constraint strength is assigned based on the constraint attributes of each node, such as a primary key constraint strength of 0.8, a NOT NULL constraint strength of 0.6, a foreign key constraint strength of 0.7, and a type match constraint strength of 0.5. When the cumulative strength exceeds a preset threshold of 2.0, a constraint propagation rule is established between the start and end points of the path. In the example above, the cumulative strength is 2.6, therefore the rule is generated: "The `category_id` of a product must reference a valid category ID, and this category can be any level of category (including top-level categories or subcategories)."

[0070] Identify source nodes with zero in-degree (e.g., the `id` field in the user table) and sink nodes with zero out-degree (e.g., the `product_quantity` field in the order details table) in the field relationship graph, and extract all connected paths between them. For example, a connected path from `user_id` in the user table to `product_quantity` in the order details table is: `user_id` → `order_id` → `order_id` → `order_quantity`. Perform combined analysis on the constraint attributes along this path to generate an end-to-end full-link validation rule: "For any `product_quantity` field in an order details record, it must be ensured that its associated order exists and belongs to a valid user." Similarly, multiple full-link validation rules can be generated to cover various business scenarios from source nodes to sink nodes.

[0071] The loop consistency check rules, constraint transit rules, and end-to-end check rules are merged to form an initial check rule set. This rule set is then deduplicated and optimized, eliminating redundant rules, merging similar rules, and improving rule execution efficiency. For example, the rules "the order's user_id must reference a valid user ID" and "the user status associated with the order's user_id must be active" are merged into "the order's user_id must reference a valid user ID with an active status." The final check rule set can be exported as JSON or XML for use by the data validation engine.

[0072] In practical applications, when processing financial transaction data, the method in this embodiment can identify from the data schema description that the `account_id` field in the transaction table references the `id` field in the account table, and the `customer_id` field in the account table references the `id` field in the customer table. Through a constraint transitivity rule generator, a rule is automatically created: "The `account_id` of a transaction record must be associated with a valid account, and that account must belong to a verified customer." This rule can effectively intercept fraudulent transactions in real-time transaction processing, ensuring data integrity and business security.

[0073] In this embodiment, data verification rules are automatically extracted and generated from complex data pattern descriptions without the need for manual writing, which greatly improves data governance efficiency, reduces business errors caused by data inconsistency, and provides a reliable quality assurance mechanism for large-scale data processing.

[0074] In one optional implementation, the value distribution characteristics of each field in the dataset to be verified are extracted, the deviation between the value distribution characteristics and the constraints in the initial verification rule set is calculated, and the constraint boundaries are adaptively adjusted to generate a modified verification rule set, including:

[0075] Each field in the dataset to be verified is scanned, all values ​​of the field are extracted to construct a value set, and sorted by numerical size to generate an ordered value sequence. The interval distance between adjacent values ​​in the ordered value sequence is calculated, and the position where the interval distance exceeds the preset interval threshold is identified as the segment point.

[0076] Extract the constraints corresponding to each field from the initial set of validation rules, and obtain the minimum and maximum allowed values;

[0077] Based on the ordered value sequence, count the number of underflow values ​​less than the minimum allowed value and the number of overflow values ​​greater than the maximum allowed value. Calculate the proportion of the number of underflow values ​​and the number of overflow values ​​to the total number of elements in the value set, and determine the lower deviation and upper deviation. When the lower deviation is greater than the upper deviation, expand the minimum allowed value downwards; when the upper deviation is greater than the lower deviation, expand the maximum allowed value upwards.

[0078] The ordered value sequence is divided into multiple subsequences according to the segmentation points. The number of values ​​corresponding to each subsequence is calculated. The subsequence with the most values ​​is selected to determine the main distribution interval. Independent constraint boundaries are set for the main distribution interval.

[0079] The expanded minimum allowable value, maximum allowable value, and independent constraint boundary are updated to the initial verification rule set to generate the modified verification rule set.

[0080] In one specific implementation, for the dataset to be verified, field scanning and value sequence construction are performed. Data for each field in the dataset is read, and all values ​​are extracted to form a value set. For example, for a temperature sensor data field, its value set may include temperature values ​​such as {-5.2, 0.3, 1.5, 1.8, 2.1, 20.5, 21.0, 21.2, 21.5, 21.8, 22.0, 22.3, 22.5, 50.1}. These values ​​are sorted according to their numerical value to form an ordered value sequence.

[0081] After the ordered value sequence is constructed, the interval distance between adjacent values ​​in the sequence is calculated. In the temperature data example above, the interval between adjacent values ​​is {5.5, 1.2, 0.3, 0.3, 18.4, 0.5, 0.2, 0.3, 0.3, 0.2, 0.3, 0.2, 27.6}. If a preset interval threshold of 5.0 is set, positions where the interval distance exceeds the threshold will be marked as segment points. In this embodiment, two segment points are identified: the first is located between -5.2 and 0.3 (interval 5.5), and the second is located between 22.5 and 50.1 (interval 27.6).

[0082] Extract the constraints corresponding to the temperature field from the initial validation rule set. Assume the initial rules define the minimum allowed value for this field as 0.0 and the maximum allowed value as 40.0. Based on these constraints, identify underflow values ​​(less than the minimum allowed value) and overflow values ​​(greater than the maximum allowed value) in the value set. For example, the underflow value is {-5.2}, with a quantity of 1; the overflow value is {50.1}, with a quantity of 1.

[0083] Calculate the lower and upper deviations, which are the proportions of the number of overflowing and overflowing values ​​to the total number of values. In the example, the total number of values ​​is 14, so the lower deviation is 1 / 14 = 7.14%, and the upper deviation is 1 / 14 = 7.14%. When the lower and upper deviations are equal, both the minimum allowable value can be expanded downwards and the maximum allowable value can be expanded upwards simultaneously.

[0084] For the expansion of constraint boundaries, an adaptive adjustment strategy based on deviation is adopted. When the current deviation is large, the minimum allowable value is expanded downward; when the current deviation is large, the maximum allowable value is expanded upward. The expansion magnitude can be set to exceed the original boundary value range plus a certain amount of redundancy. In the example, the new minimum allowable value can be adjusted to -6.0 (slightly less than the minimum underflow value -5.2), and the new maximum allowable value can be adjusted to 51.0 (slightly greater than the maximum overflow value 50.1).

[0085] Based on the previously identified segmentation points, the ordered value sequence is divided into multiple subsequences. In the example, three subsequences are formed: {-5.2}, {0.3, 1.5, 1.8, 2.1, 20.5, 21.0, 21.2, 21.5, 21.8, 22.0, 22.3, 22.5}, and {50.1}. For each subsequence, the number of values ​​it contains is calculated, which are 1, 12, and 1 respectively. The subsequence with the most values ​​(the second subsequence, containing 12 values) is determined as the main distribution interval, with a range of [0.3, 22.5].

[0086] Independent constraint boundaries are set for the main distribution interval. These boundaries are used to identify the mainstream distribution range of the data and can be used for subsequent anomaly detection or data filtering. Considering that the data within the main distribution interval is relatively concentrated and representative, an independent constraint boundary [0.0, 23.0] can be set for this interval, slightly extending the original boundary to increase fault tolerance.

[0087] The expanded global constraints (minimum allowable value -6.0 and maximum allowable value 51.0) and the independent constraint boundaries [0.0, 23.0] of the main distribution interval are updated to the initial validation rule set to generate the revised validation rule set. The revised rules are more consistent with the actual data distribution characteristics, which can reduce the false alarm rate and improve the accuracy and practicality of data validation.

[0088] This can also be applied to other types of fields. For example, for a device's operating time field, the initial validation rule limits the range to [0, 24] hours, while the actual data values ​​are {0.5, 1.2, 1.8, 2.5, 3.0, 8.5, 9.0, 22.5, 23.0, 25.5, 26.0, 48.5}. The overflow value {25.5, 26.0, 48.5} is identified, with an upward deviation of 3 / 12 = 25%, significantly higher than the downward deviation of 0%. Therefore, the maximum allowable value is expanded upward to 50.0. Simultaneously, the main distribution interval is identified as [0.5, 9.0], and the independent constraint boundary is set to [0.0, 10.0], ultimately generating a corrected validation rule containing the global constraint [0, 50] and the main distribution interval constraint [0, 10].

[0089] In this embodiment, by intelligently analyzing the actual data distribution and adaptively adjusting the boundaries of the verification rules, while extracting the main distribution characteristics of the data, the accuracy of data verification can be effectively improved, and false alarms and false negatives caused by unreasonable rule settings can be reduced, providing more reliable technical support for data quality control.

[0090] In one optional implementation, for each rule in the modified verification rule set, a positive sample set and a negative sample set are constructed from the dataset to be verified, the data coverage and anomaly detection rate are calculated, a rule quality score is determined, and a target verification rule set is generated based on the rule quality score, including:

[0091] Iterate through each rule in the set of corrected verification rules, parse the constraint boundaries in the rules, and select data records whose values ​​are within the constraint boundaries from the dataset to be verified to form a positive sample set, and data records whose values ​​are outside the constraint boundaries to form a negative sample set.

[0092] Extract all field values ​​of each data record in the positive sample set and combine them into a positive sample feature vector. Perform a convex hull algorithm on the positive sample feature vector in the vector space to obtain a geometric envelope surface. Construct a regular boundary surface based on the constraint boundary. Calculate the spatial deviation between the geometric envelope surface and the regular boundary surface to obtain the regular fitting error. Determine the corrected data coverage based on the regular fitting error.

[0093] Extract all field values ​​of each data record in the negative sample set and combine them into a negative sample feature vector. Use each negative sample feature vector as a spatial coordinate point to calculate the shortest Euclidean distance to the geometric envelope. Based on the shortest Euclidean distance, determine the near-boundary anomaly set and the far-boundary anomaly set. Divide the geometric envelope into directional sectors, identify high-voltage directional sectors, determine boundary leakage, and determine the corrected anomaly identification rate.

[0094] The rule quality score is obtained by weighted summing of the corrected data coverage rate and the corrected anomaly detection rate, and the target verification rule set is generated by filtering.

[0095] In one specific implementation, during the rule quality assessment phase, each rule in the modified verification rule set is processed. For example, for a certain temperature sensor data, there is a rule in the modified verification rule set: "When the production batch is Class A, the temperature value should be between 20°C and 40°C." The constraint boundaries of this rule are parsed, namely the temperature value range [20, 40] and the condition that the production batch is Class A.

[0096] The dataset to be validated selects data records that meet certain criteria to form a positive sample set. For example, from 10,000 sensor records, 3,000 records with production batch A and temperature values ​​between 20℃ and 40℃ are selected. Simultaneously, 500 records with production batch A but temperature values ​​outside this range are used to form a negative sample set.

[0097] When processing the positive sample set, all field values ​​of each record are extracted. For example, a temperature record contains fields such as temperature, humidity, and timestamp. These values ​​are combined into a positive sample feature vector. For instance, the vector for the first record might be [32.5, 45.8, 1623456789], and the vector for the second record might be [28.7, 42.1, 1623456790]. The convex hull algorithm is then applied to these vectors to calculate the geometric envelope.

[0098] Specifically, the convex hull algorithm uses an incremental approach with a time complexity of O(n log n). Taking two-dimensional space as an example, the first two dimensions of the positive sample feature vectors are plotted on the coordinate axes, resulting in multiple scattered points. By connecting the outer vertices of these scattered points, a convex polygon is formed. In three-dimensional or higher-dimensional space, the same principle is used to construct the convex hull.

[0099] Based on the rule-based constraint boundary, a rule boundary surface is constructed. In the temperature rule example above, the rule boundary surface is a hyperplane truncated in the temperature dimension. The spatial deviation between the geometric envelope and the rule boundary surface is calculated to obtain the rule fitting error. Specifically, 100 points are uniformly sampled on the rule boundary surface, the distance from each point to the geometric envelope is calculated, and the average value is taken as the fitting error. Assuming the calculated average distance is 0.45, the rule fitting error is 0.45.

[0100] The corrected data coverage is determined based on the rule fitting error. When the fitting error is less than a preset threshold (e.g., 0.5), it indicates that the rule boundaries match the data distribution well, and a high coverage score is given. For example, the corrected data coverage is calculated as 0.85. When the fitting error is large, it indicates that the rule constraints are too loose or too strict, and the coverage score is reduced.

[0101] In processing the negative sample set, all field values ​​of each record are extracted and combined to form a negative sample feature vector. For example, the vector of the first negative sample record is [19.5, 41.2, 1623456899]. Each negative sample feature vector is treated as a coordinate point in space, and its shortest Euclidean distance to the geometric envelope of the positive samples is calculated. For example, the distance from the first negative sample to the geometric envelope is 0.8.

[0102] Based on the calculated distance values, negative samples are divided into near-boundary outliers and far-boundary outliers. For example, negative samples with a distance less than 2 are classified into the near-boundary outlier set, which contains 150 records; negative samples with a distance greater than or equal to 2 are classified into the far-boundary outlier set, which contains 350 records.

[0103] The geometric envelope is divided into directional sectors, typically 8 to 16 sectors. Taking three-dimensional space as an example, rays are extended in all directions from the center point of the envelope, dividing the space into multiple cone-shaped regions. The system counts the number of negative samples in each sector and identifies the sector with the highest density of outliers as the high-voltage directional sector.

[0104] When the density of abnormal points in a high-voltage sector exceeds a preset threshold (e.g., 3 points per unit volume), a boundary leakage risk is identified, and stronger rule constraints are needed in this direction. For example, if an abnormal point density of 5 points per unit volume is found in a sector with a temperature below 20°C, a boundary leakage is identified in that direction.

[0105] Based on the distribution of near-boundary and far-boundary anomaly sets and the boundary leakage determination results, a revised anomaly identification rate is determined. The calculation formula takes into account the uniformity of anomaly distribution and the degree of boundary leakage. For example, when anomalies are uniformly distributed and there is no obvious boundary leakage, a high identification rate score of 0.92 is given; when boundary leakage exists, the identification rate score is reduced to 0.75.

[0106] The rule quality score is obtained by weighting and summing the corrected data coverage rate and the corrected anomaly detection rate. Typically, the weights are set to 0.4 for coverage and 0.6 for detection rate. For example, the rule quality score = 0.4 × 0.85 + 0.6 × 0.75 = 0.79.

[0107] After calculating quality scores for all rules in the revised verification rule set, high-quality rules are selected based on a preset quality threshold (e.g., 0.7) to form the target verification rule set. For example, from 100 revised rules, 75 rules with quality scores higher than 0.7 are selected to form the target verification rule set. These rules can both effectively cover the normal data distribution and effectively identify anomalous data.

[0108] like Figure 2 The diagram shows the parallel processing flowchart for rule quality assessment via two paths.

[0109] In one optional implementation, the near-boundary anomaly set and the far-boundary anomaly set are determined based on the shortest Euclidean distance, the geometric envelope is divided into directional sectors, high-pressure directional sectors are identified, boundary leaks are determined, and the corrected anomaly identification rate is determined, including:

[0110] Calculate the shortest Euclidean distance from the feature vector of each negative sample in the negative sample set to the geometric envelope. Statistically count all the shortest Euclidean distances and calculate the median to determine the distance threshold. Divide the negative samples whose shortest Euclidean distance is less than the distance threshold into the near-boundary outlier set and the remaining negative samples into the far-boundary outlier set.

[0111] The geometric envelope is meshed into a spherical grid. Based on the outward normal direction of the envelope, the space is divided into multiple directional sectors. The number of negative samples in the near-boundary outlier set in each directional sector is counted. The boundary pressure value of the directional sector is determined by calculating the ratio of the number of negative samples to the total number of near-boundary outliers.

[0112] High-pressure directional sectors are identified by identifying directional sectors where the boundary pressure value exceeds a preset pressure threshold. Multiple surface points are sampled within the envelope area corresponding to each high-pressure directional sector. The distance from each surface point to the nearest negative sample is calculated. The number of surface points whose distance is less than a preset permeability threshold is counted. The ratio of the number of surface points to the total number of sampling points is calculated to determine the boundary permeability.

[0113] The weighted average of the boundary permeability of all high-pressure sector sectors is calculated as the global boundary permeability. An anomaly identification rate attenuation coefficient is calculated based on the global boundary permeability. The original anomaly identification rate is multiplied by the attenuation coefficient to obtain the corrected anomaly identification rate.

[0114] In one specific implementation, for the geometric envelope formed by the data validation rules, the shortest Euclidean distance from each negative sample feature vector to the envelope is calculated. Specifically, the validation rules are converted into constraint boundaries in a high-dimensional feature space to form the geometric envelope. For example, the validation rule "order amount must be greater than zero and less than the user's credit limit" can be converted into a constraint plane in the feature space. The negative sample feature vector represents the data record that violates the validation rule, such as a record with a negative order amount or exceeding the credit limit. When calculating the shortest Euclidean distance, the gradient descent method is used to find the point closest to the negative sample on the geometric envelope, and the Euclidean distance between the two points is calculated. For example, for a batch of data that violates the order amount rule, the distances between them and the constraint plane are calculated respectively, resulting in a set of distance values ​​[0.12, 0.05, 0.25, 0.08, 0.17, 0.04, 0.31, 0.09]. Sort all distance values ​​and take the median of 0.10 as the distance threshold. Negative samples with a distance less than 0.10 (0.05, 0.08, 0.04, 0.09) are classified as near-boundary outliers, and the remaining negative samples are classified as far-boundary outliers.

[0115] In the boundary adjustment phase of the data validation rules, the geometric envelope is meshed using a spherical grid to achieve spatial orientation analysis. By calculating the outward normal directions of points uniformly distributed on the geometric envelope, the space outside the envelope is divided into multiple directional sectors. In the order validation rule example, the constraint plane can be meshed into a 5×5 grid, forming 25 regions, with the normal direction of each region defining a directional sector. Subsequently, the number of negative samples in the near-boundary outlier set within each directional sector is counted. If a sector has 6 negative samples while the total number of near-boundary outliers is 24, then the boundary pressure value for that sector is 0.25. In this way, the boundary pressure values ​​for all directional sectors are calculated separately, forming a pressure distribution map.

[0116] Identifying sectors where the boundary pressure value exceeds a preset pressure threshold is a crucial step in optimizing validation rules. In the sensitivity analysis phase of data validation rules, if the preset pressure threshold is 0.15, sectors with a boundary pressure value of 0.25 are identified as high-pressure sectors. Within the envelope region corresponding to each high-pressure sector, multiple surface points are sampled, such as 100 points uniformly sampled in this region. The distance from each surface point to the nearest negative sample is calculated, and the number of surface points whose distance is less than the preset penetration threshold is counted. Assuming the penetration threshold is set to 0.05, if 35 out of the 100 sampled points have a distance less than 0.05 to the nearest negative sample, then the boundary penetration rate of this region is 0.35. This indicates that in this region, 35% of the validation rule boundary is easily breached by violating data.

[0117] During the adaptive adjustment phase of data verification rules, the weighted average of the boundary permeability of all high-pressure sector areas is calculated as the global boundary permeability. The weighting coefficient can be determined based on the boundary pressure value of each sector; the higher the pressure value, the greater the weight. Assuming there are three high-pressure sector areas with boundary permeability of 0.35, 0.28, and 0.42, corresponding to weights of 0.4, 0.25, and 0.35, the global boundary permeability is 0.35×0.4 + 0.28×0.25 + 0.42×0.35 = 0.357. The anomaly detection rate attenuation coefficient is calculated based on the global boundary permeability using a non-linear mapping function, such as attenuation coefficient = 1 - the square of the global boundary permeability, resulting in an attenuation coefficient of 1 - 0.357. 2 = 0.873. Multiplying the original anomaly detection rate (e.g., 0.95) by the attenuation factor, we get a corrected anomaly detection rate of 0.95 × 0.873 = 0.829. The corrected anomaly detection rate is used to adjust the enforcement strength of the verification rules, making the rules more lenient in high-pressure areas and reducing false positives.

[0118] This method is particularly effective in complex database table join validation scenarios. For example, traditional methods may lead to numerous false positives due to improperly set boundary conditions in multi-table join validation rules for orders, products, and inventory. The method in this embodiment analyzes the distribution of violating data in the feature space, identifying that high-pressure sectors are mainly concentrated in the boundary region where "order submission time is close to inventory update time." The corrected validation rules can appropriately relax the judgment criteria in this region, allowing for slight time overlap, thereby reducing false positives caused by differences in timestamp precision and significantly improving the user experience.

[0119] In existing technologies, data validation rules typically use fixed thresholds or simple statistical methods to set boundaries, lacking fine-grained analysis of abnormal distributions in rule boundary regions. This leads to rules being overly strict or too lenient in certain directions. Traditional methods often result in numerous false positives or false negatives in boundary regions, affecting the accuracy and efficiency of data processing. The method in this embodiment introduces geometric envelope analysis and directional sector partitioning to achieve direction-aware optimization of data validation rules. It quantifies and analyzes the validation pressure in different directions and adjusts the rule strength accordingly. Especially in multi-table join scenarios with complex boundary conditions, the validation accuracy is significantly improved, and processing efficiency is significantly enhanced, providing a more reliable and efficient quality assurance mechanism for large-scale data processing.

[0120] In one optional implementation, a directed dependency graph is constructed by parsing the input and output fields of the rules in the target validation rule set. Based on the topological sorting of the directed dependency graph, directed paths are chained together into execution chains. Isolated nodes and subsets of rules without directed path connections are combined into parallel execution units. The hierarchical validation topology structure is constructed by:

[0121] Traverse each rule in the target validation rule set, parse the input and output fields in the rule, and for any two rules, if the output field of the first rule and the input field of the second rule have an intersection, then build a dependency relationship between the first rule and the second rule, mark the first rule as the predecessor rule, and mark the second rule as the successor rule.

[0122] A directed dependency graph is constructed based on the dependency relationship between predecessor and successor rules. Each rule is treated as a node in the directed dependency graph, and the dependency relationship is treated as a directed edge. A topological sort is performed on the directed dependency graph to identify all directed paths in the directed dependency graph. The rules in each directed path are then chained together in the topological sort order to form an execution chain.

[0123] Identify isolated nodes with zero in-degree and zero out-degree in the directed dependency graph, as well as rule subsets without directed path connections, and merge the rules corresponding to the isolated nodes with the rules in the rule subset into parallel execution units.

[0124] The rules in the execution chain are assigned to the corresponding level depth according to the topological sorting order. The longest path length from the starting node to the current node of each rule is calculated to determine the level number of the rule. The rules in the parallel execution unit are assigned to the first level. The rules in the execution chain are assigned to the subsequent levels in ascending order of level number to construct a hierarchical verification topology structure.

[0125] In one specific implementation, a target verification rule set is obtained, which contains multiple rules, each defining input and output fields. The input and output field information of each rule is parsed from the rule set and stored in a rule metadata structure. For example, for a rule R1 that verifies user information, its input fields might include "user age" and "user occupation," and its output field might be "user credit rating." Another rule R2's input fields might include "user credit rating" and "historical transaction records," and its output field might be "loan amount." An intersection is identified between the output field "user credit rating" of R1 and the input field "user credit rating" of R2, thus establishing a dependency relationship between R1 and R2.

[0126] The dependency relationship is established by iterating through all rules using a double-loop loop. For each rule Ri in the rule set, the system iterates through all other rules Rj, checking if there is an intersection between the output field set of Ri and the input field set of Rj. If an intersection exists, a directed edge is established between Ri and Rj, indicating that Ri is the predecessor rule and Rj is the successor rule. This dependency relationship essentially describes the order constraint of rule execution.

[0127] After completing the dependency analysis, a directed dependency graph G=(V,E) is constructed, where V represents the set of rules and E represents the set of dependencies between rules. For example, if the rule set contains rules {R1, R2, R3, R4, R5} and the dependencies are {(R1, R2), (R2, R3), (R1, R4), (R5, R3)}, then the constructed directed dependency graph contains 5 nodes and 4 directed edges.

[0128] After constructing the directed dependency graph, perform a topological sorting algorithm on it. Maintain a queue Q, initially adding all nodes with an in-degree of 0 (i.e., rules without predecessors) to the queue. Iteratively remove a node from the queue, add it to the resulting sequence, and decrease the in-degree of all its successor nodes. If the in-degree of a successor node becomes 0, add it to the queue. Repeat this process until the queue is empty. For example, performing a topological sort on the above directed dependency graph yields the sequence [R1, R5, R2, R4, R3].

[0129] Based on the topological sorting results, all directed paths in the directed dependency graph are identified. A directed path is a path formed by a series of connected directed edges in the graph. In the example above, the directed paths include [R1→R2→R3], [R1→R4], and [R5→R3]. The rules in each directed path are then chained together according to the topological sorting order to form an execution chain.

[0130] Identify isolated nodes with an in-degree of 0 and an out-degree of 0, as well as rule subsets without directed path connections in the directed dependency graph. Isolated nodes are rules that have no dependency on other nodes in the graph; they neither depend on nor are depended upon by other rules. Rule subsets are subgraphs that are interconnected but not connected to other parts of the graph. These isolated nodes and disconnected rule subsets can be executed in parallel, so the system merges them into parallel execution units.

[0131] Based on the structure of the directed dependency graph, each rule is assigned a level number. Specifically, the longest path length from any node with an in-degree of 0 (the starting node) to the current node is calculated, and this length is the node's level number. Nodes with a level number of 0 (i.e., nodes with an in-degree of 0) and isolated nodes are assigned to the first level. For example, in the example above, R1 and R5 have a level number of 0 and are assigned to the first level; R2 and R4 have a level number of 1 and are assigned to the second level; and R3 has a level number of 2 and is assigned to the third level.

[0132] A hierarchical validation topology is constructed based on the rule hierarchy numbering. Rules at the same level can be executed in parallel, while rules at different levels are executed sequentially according to their hierarchical order. For example, rules R1 and R5 at the first level are executed in parallel; after they are completed, rules R2 and R4 at the second level are executed in parallel; and finally, rule R3 at the third level is executed.

[0133] The hierarchical verification topology constructed in this embodiment makes full use of the dependencies between rules, which not only ensures the correctness of the verification process (there will be no situation where subsequent rules are executed before dependent rules have been completed), but also improves the verification efficiency (by executing independent rules in parallel), significantly improving the performance of verification systems containing a large number of rules, and is particularly suitable for scenarios that require real-time verification.

[0134] The present invention provides an automatic data validation rule generation and execution system based on a large model, comprising:

[0135] The first unit is used to obtain the dataset to be verified and the corresponding data pattern description information;

[0136] The second unit is used to parse the data pattern description information to generate a field relationship graph, generate an initial verification rule set based on the transmission path of the directed edges and the constraint attributes of the nodes in the field relationship graph; extract the value distribution characteristics of each field in the dataset to be verified, calculate the deviation between the value distribution characteristics and the constraint conditions in the initial verification rule set, and adaptively adjust the constraint boundaries to generate a modified verification rule set.

[0137] The third unit is used to construct positive and negative sample sets from the dataset to be verified for each rule in the modified verification rule set, calculate the data coverage and anomaly detection rate, determine the rule quality score, and filter and generate the target verification rule set based on the rule quality score.

[0138] The fourth unit is used to parse the input and output fields of the rules in the target validation rule set to construct a directed dependency graph. Based on the topological sorting of the directed dependency graph, the directed paths are linked into an execution chain. Isolated nodes and rule subsets without directed path connections are combined into parallel execution units to construct a hierarchical validation topology structure.

[0139] The fifth unit is used to perform verification operations on the dataset to be verified based on the hierarchical verification topology and obtain the verification results.

[0140] A third aspect of the present invention provides an electronic device, comprising:

[0141] processor;

[0142] Memory used to store processor-executable instructions;

[0143] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0144] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0145] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically generating and executing data validation rules based on a large model, characterized in that, include: Obtain the dataset to be verified and its corresponding data pattern description information; The data pattern description information is parsed, the description text is processed line by line, and a field relationship graph is generated. An initial verification rule set is generated based on the transmission path of the directed edges and the constraint attributes of the nodes in the field relationship graph. The value distribution characteristics of each field in the dataset to be verified are extracted, the deviation between the value distribution characteristics and the constraint conditions in the initial verification rule set is calculated, and the constraint boundary is adaptively adjusted to generate a modified verification rule set. For each rule in the modified verification rule set, construct a positive sample set and a negative sample set from the dataset to be verified, calculate the data coverage and anomaly detection rate, determine the rule quality score, and filter and generate the target verification rule set based on the rule quality score; The input and output fields of the rules in the target validation rule set are analyzed to construct a directed dependency graph. Based on the topological sorting of the directed dependency graph, the directed paths are linked into an execution chain. Isolated nodes and rule subsets without directed path connections are combined into parallel execution units to construct a hierarchical validation topology structure. The verification operation is performed on the dataset to be verified based on the hierarchical verification topology, and the verification result is obtained.

2. The method according to claim 1, characterized in that, The data pattern description information is parsed, the description text is processed line by line, a field relationship graph is generated, and an initial validation rule set is generated based on the directed edge propagation path and node constraint attributes in the field relationship graph, including: The data schema description information is parsed, processed line by line, and field identifiers and type declarations are extracted. Field identifiers are mapped to nodes in the field relationship graph, and type declarations are converted into constraint attributes of the corresponding nodes. Foreign key constraints and reference relationships in the data schema description information are identified, and directed edges are established between the corresponding nodes. Loop detection is performed on the directed edges in the field relationship graph to identify the set of edges that form a closed loop. Constraint attributes are extracted for the nodes in the set of edges, and loop consistency verification rules are generated based on the mutual exclusion and compatibility of the constraint attributes. Perform a depth-first traversal on the field relationship graph, record the node sequence and edge sequence traversed during the traversal, calculate the cumulative strength of the constraint attributes in the node sequence, and when the cumulative strength exceeds a preset strength threshold, establish constraint propagation rules between the start and end points of the edge sequence. Identify source nodes with zero in-degree and sink nodes with zero out-degree in the field relationship graph, extract all connected paths between the source nodes and sink nodes, combine the constraint attributes on each connected path, and generate end-to-end full-link verification rules. The loop consistency verification rule, the constraint propagation rule, and the end-to-end verification rule are merged to form an initial verification rule set.

3. The method according to claim 1, characterized in that, Extract the value distribution characteristics of each field in the dataset to be verified, calculate the deviation between the value distribution characteristics and the constraints in the initial verification rule set, and adaptively adjust the constraint boundaries to generate a modified verification rule set, including: Each field in the dataset to be verified is scanned, all values ​​of the field are extracted to construct a value set, and sorted by numerical size to generate an ordered value sequence. The interval distance between adjacent values ​​in the ordered value sequence is calculated, and the position where the interval distance exceeds the preset interval threshold is identified as the segment point. Extract the constraints corresponding to each field from the initial set of validation rules, and obtain the minimum and maximum allowed values; Based on the ordered value sequence, count the number of underflow values ​​less than the minimum allowed value and the number of overflow values ​​greater than the maximum allowed value. Calculate the proportion of the number of underflow values ​​and the number of overflow values ​​to the total number of elements in the value set, and determine the lower deviation and upper deviation. When the lower deviation is greater than the upper deviation, expand the minimum allowed value downwards; when the upper deviation is greater than the lower deviation, expand the maximum allowed value upwards. The ordered value sequence is divided into multiple subsequences according to the segmentation points. The number of values ​​corresponding to each subsequence is calculated. The subsequence with the most values ​​is selected to determine the main distribution interval. Independent constraint boundaries are set for the main distribution interval. The expanded minimum allowable value, maximum allowable value, and independent constraint boundary are updated to the initial verification rule set to generate the modified verification rule set.

4. The method according to claim 1, characterized in that, For each rule in the modified verification rule set, a positive sample set and a negative sample set are constructed from the dataset to be verified. The data coverage and anomaly detection rate are calculated to determine the rule quality score. Based on the rule quality score, the target verification rule set is generated, including: Iterate through each rule in the set of corrected verification rules, parse the constraint boundaries in the rules, and select data records whose values ​​are within the constraint boundaries from the dataset to be verified to form a positive sample set, and data records whose values ​​are outside the constraint boundaries to form a negative sample set. Extract all field values ​​of each data record in the positive sample set and combine them into a positive sample feature vector. Perform a convex hull algorithm on the positive sample feature vector in the vector space to obtain a geometric envelope surface. Construct a regular boundary surface based on the constraint boundary. Calculate the spatial deviation between the geometric envelope surface and the regular boundary surface to obtain the regular fitting error. Determine the corrected data coverage based on the regular fitting error. Extract all field values ​​of each data record in the negative sample set and combine them into a negative sample feature vector. Use each negative sample feature vector as a spatial coordinate point to calculate the shortest Euclidean distance to the geometric envelope. Based on the shortest Euclidean distance, determine the near-boundary anomaly set and the far-boundary anomaly set. Divide the geometric envelope into directional sectors, identify high-voltage directional sectors, determine boundary leakage, and determine the corrected anomaly identification rate. The rule quality score is obtained by weighted summing of the corrected data coverage rate and the corrected anomaly detection rate, and the target verification rule set is generated by filtering.

5. The method according to claim 4, characterized in that, Based on the shortest Euclidean distance, near-boundary and far-boundary anomaly sets are determined. The geometric envelope is then divided into directional sectors to identify high-pressure directional sectors, determine boundary leaks, and establish a corrected anomaly identification rate, including: Calculate the shortest Euclidean distance from the feature vector of each negative sample in the negative sample set to the geometric envelope. Statistically count all the shortest Euclidean distances and calculate the median to determine the distance threshold. Divide the negative samples whose shortest Euclidean distance is less than the distance threshold into the near-boundary outlier set and the remaining negative samples into the far-boundary outlier set. The geometric envelope is meshed into a spherical grid. Based on the outward normal direction of the envelope, the space is divided into multiple directional sectors. The number of negative samples in the near-boundary outlier set in each directional sector is counted. The boundary pressure value of the directional sector is determined by calculating the ratio of the number of negative samples to the total number of near-boundary outliers. High-pressure directional sectors are identified by identifying directional sectors where the boundary pressure value exceeds a preset pressure threshold. Multiple surface points are sampled within the envelope area corresponding to each high-pressure directional sector. The distance from each surface point to the nearest negative sample is calculated. The number of surface points whose distance is less than a preset permeability threshold is counted. The ratio of the number of surface points to the total number of sampling points is calculated to determine the boundary permeability. The weighted average of the boundary permeability of all high-pressure sector sectors is calculated as the global boundary permeability. An anomaly identification rate attenuation coefficient is calculated based on the global boundary permeability. The original anomaly identification rate is multiplied by the attenuation coefficient to obtain the corrected anomaly identification rate.

6. The method according to claim 1, characterized in that, The input and output fields of the rules in the target validation rule set are parsed to construct a directed dependency graph. Based on the topological sorting of the directed dependency graph, directed paths are chained together into execution chains. Isolated nodes and subsets of rules without directed path connections are combined into parallel execution units. The hierarchical validation topology structure includes: Traverse each rule in the target validation rule set, parse the input and output fields in the rule, and for any two rules, if the output field of the first rule and the input field of the second rule have an intersection, then build a dependency relationship between the first rule and the second rule, mark the first rule as the predecessor rule, and mark the second rule as the successor rule. A directed dependency graph is constructed based on the dependency relationship between predecessor and successor rules. Each rule is treated as a node in the directed dependency graph, and the dependency relationship is treated as a directed edge. A topological sort is performed on the directed dependency graph to identify all directed paths in the directed dependency graph. The rules in each directed path are then chained together in the topological sort order to form an execution chain. Identify isolated nodes with zero in-degree and zero out-degree in the directed dependency graph, as well as rule subsets without directed path connections, and merge the rules corresponding to the isolated nodes with the rules in the rule subset into parallel execution units. The rules in the execution chain are assigned to the corresponding level depth according to the topological sorting order. The longest path length from the starting node to the current node of each rule is calculated to determine the level number of the rule. The rules in the parallel execution unit are assigned to the first level. The rules in the execution chain are assigned to the subsequent levels in ascending order of level number to construct a hierarchical verification topology structure.

7. A system for automatically generating and executing data validation rules based on a large model, used to implement the method described in any one of claims 1-6, characterized in that, include: The first unit is used to obtain the dataset to be verified and the corresponding data pattern description information; The second unit is used to parse the data pattern description information, process the description text line by line, generate a field relationship graph, generate an initial verification rule set based on the transmission path of the directed edges and the constraint attributes of the nodes in the field relationship graph; extract the value distribution characteristics of each field in the dataset to be verified, calculate the deviation between the value distribution characteristics and the constraint conditions in the initial verification rule set, and adaptively adjust the constraint boundaries to generate a modified verification rule set. The third unit is used to construct positive and negative sample sets from the dataset to be verified for each rule in the modified verification rule set, calculate the data coverage and anomaly detection rate, determine the rule quality score, and filter and generate the target verification rule set based on the rule quality score. The fourth unit is used to parse the input and output fields of the rules in the target validation rule set to construct a directed dependency graph. Based on the topological sorting of the directed dependency graph, the directed paths are linked into an execution chain. Isolated nodes and rule subsets without directed path connections are combined into parallel execution units to construct a hierarchical validation topology structure. The fifth unit is used to perform verification operations on the dataset to be verified based on the hierarchical verification topology and obtain the verification results.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images