Data quality detection and judgment method for industry field

Abstract

The invention provides a data quality detection and judgment method for the industry field, belongs to the field of big data in internet application, and adopts the technical scheme that the business field to which data belongs is coded into placeholder identification, and a detection rule and a target industry range are dynamically matched based on bit operation. Accurate real-time adaptation of the single data and the quality inspection rule is achieved, and therefore on-demand quality inspection efficiency and expandability of cross-industry data are remarkably improved on the premise that full scanning or manual maintenance of screening logic is not needed.

Description

Technical Field

[0001] The present invention relates to the fields of big data, data processing, and computer technology in Internet applications, and particularly to a method for detecting and judging data quality in industrial fields. Background Art

[0002] Data quality: refers to the degree to which data meets the usage purposes of data consumers and can satisfy the specific requirements of business scenarios in a business environment. In different business scenarios, data consumers have different requirements for data quality. Some people mainly focus on the accuracy and consistency of data, while others focus on the timeliness and relevance of data. Therefore, as long as the data can meet the usage purposes, it can be said that the data quality meets the requirements.

[0003] Industrial field: The industrial field is the basic unit of economic activities. Through classification, the economic structure can be systematically understood, strategies can be formulated, or research can be conducted. With the development of technology, the boundaries between industries are gradually blurred (such as "Internet + traditional industries"), but classification is still an important tool for analyzing problems.

[0004] Existing data quality detection cannot quickly judge whether a single piece of data needs to be detected in a scenario where different industry attributes are dynamically distinguished because it needs to uniformly execute batch rule verification on all data in the table or relies on static pre-judgment to screen the data to be detected, resulting in insufficient processing efficiency and business flexibility. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a method for detecting and judging data quality in industrial fields.

[0006] The technical solution of the present invention is as follows: A method for detecting and judging data quality in industrial fields, including (a) Predefining an industry coding table: Coding the industrial fields involved in the business into binary bit serial numbers in a fixed order to form an industry index sequence; (b) Generating a rule mask: Generating an industry detection mask for each data quality detection rule, and setting the bits corresponding to the industries to be detected in the mask to 1 and the remaining bits to 0; (c) Generating a data industry identifier: For the data to be detected, generating a binary identifier according to the industry to which it belongs, and setting the bits corresponding to the industry to which the data belongs in the identifier to 1; (d) Dynamic rule matching: Traversing the rule set and performing the following operations on each rule: (d1) Reading the industry detection mask (Mask_R) of the rule and the industry identifier (Tag_D) of the data; (d2) Calculating the matching value: Result = Mask_R & Tag_D; (d3) If Result ≥ 1, perform quality check on the current data, otherwise skip the check; (e) Detection result storage: record the rule identification and violation results that triggered the detection.

[0007] Furthermore, The step (d3) further includes the following matching pattern: Full match mode: detection is triggered when and only when Result == Mask_R; Any matching mode: When Result>0, the detection is triggered.

[0008] Furthermore, The industry detection mask supports dynamic expansion. When a new industry is added, historical rule compatibility is achieved by increasing the number of mask bits, and the original mask is automatically padded with zeros at high positions.

[0009] Furthermore, In the step (c), the data to be inspected is parsed to extract the industry to which it belongs; and a data industry identifier is generated according to the industry coding table.

[0010] Furthermore, The test result storage in step (e) includes the following fields: Violation rule mask (records the rule mask value that triggers the detection); Data industry identification (records the industry identification value of the inspected data); Match result value (Result).

[0011] Furthermore, The generation logic of the industry coding table includes: Generate composite identifiers for cross-industry data, allowing a single piece of data to occupy multiple industry positions; The industry coding order is strongly bound to the number of mask bits to ensure unique mapping.

[0012] Furthermore, When traversing quality detection rules, it supports selecting industry matching logic through predefined matching judgment mode parameters.

[0013] Specifically include: (1) Full match mode, namely AND logic: Judgment condition: The data industry identifier Tag_D must completely contain all industry bits set to 1 in the rule industry detection mask Mask_R; Calculation logic: If (Mask_R&Tag_D) == Mask_R, then trigger detection; otherwise skip; Application scenarios: Applicable to strict scenarios where the rules are only effective for a few industries at the same time; (2) Any matching pattern is OR logic: Judgment condition: Data industry identifier Tag_D and rule industry detection mask Mask_R have at least one common industry bit; Calculation logic: If (Mask_R&Tag_D) ≠ 0, then trigger the detection; otherwise skip; Application scenarios: Applicable to general scenarios where the rules are effective in any related industries.

[0014] The beneficial effects of the present invention are Millisecond-level real-time judgment: By replacing traditional string matching or database query with bit operations (AND / OR), the industry matching judgment time for a single piece of data is reduced from 10ms to μs (measured average 0.2μs / rule), supporting real-time processing of tens of millions of data streams.

[0015] Flexible matching strategy: supports both full match (AND) and any match (OR) modes. A single rule can be configured as: Need to hit multiple industries at the same time (such as cross-border compliance rules for finance and healthcare); It is triggered when any industry is hit (such as general data format verification rules).

[0016] Rule-level performance optimization: Industry prediction and pre-filtering of invalid rules can reduce redundant quality inspection calculations by 60%-90% (depending on the density of rule industry associations), significantly reducing CPU and memory usage.

[0017] Dynamic expansion without perception: When adding a new industry, only the number of mask bits is expanded (for example, from 8 bits to 16 bits), and the historical rule mask is automatically padded with zeros, without business downtime or data migration.

[0018] Accurate resource allocation: Through mask hit rate statistics (for example, the mask hit rate of rule A is 5%), inefficient rules can be optimized in a targeted manner to improve the overall system throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the workflow of the present invention; Figure 2 It is a full match mode (AND logic) flow chart; Figure 3 It is an either-match mode (OR logic) flow chart. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0021] The present invention provides a method for detecting and judging data quality in an industry field. By encoding the business field to which the data belongs as a placeholder identifier and based on the technical solution of dynamically matching the detection rules with the target industry scope through bit operations, the accurate and real-time adaptation of a single piece of data to the quality inspection rules is realized. Thus, without the need for full-scale scanning or manual maintenance of the screening logic, the on-demand quality inspection efficiency and scalability of cross-industry data are significantly improved.

[0022] The specific steps are as follows: Step 1: Dynamically bind the industry code and rules.

[0023] Establish an industry code table, and map each industry to a binary bit serial number in a preset order (for example, finance → bit 1, manufacturing → bit 2, medical → bit 3); Generate a dynamic industry detection mask for each quality inspection rule, and set the bits corresponding to the industries to be detected in the mask to 1, and the remaining bits to 0; Example: If the rule needs to detect finance (bit 1) and medical (bit 3), then the mask = 101 (binary); Bind the industry detection mask as a rule attribute to the detection logic.

[0024] Step 2: Generate the data industry identifier.

[0025] Analyze the data to be inspected and extract the industry to which it belongs; Generate a data industry identifier according to the industry code table, and set the bits corresponding to the industry to which the data belongs in the identifier to 1; Example: If the data belongs to manufacturing (bit 2), then the identifier = 010 (binary) Step 3: Determine the real-time industry matching.

[0026] Before performing the detection on each piece of data, traverse the quality rule set and perform the following operations: a. Read the industry detection mask (Mask_R) of the current rule and the data industry identifier (Tag_D); b. Calculate the industry matching value: Match = Mask_R & Tag_D; c. Judgment logic: If Match ≠ 0 (that is, at least one industry bit matches), then perform the rule detection on the current data; If Match = 0, skip this rule.

[0027] Step 4. Support for multi-industry combination rules.

[0028] When traversing the quality inspection rules in this step, it supports selecting the industry matching logic through predefined matching judgment mode parameters, specifically including: 1. Full match mode (AND logic): Judgment condition: The data industry identifier (Tag_D) must completely contain all industry bits set to 1 in the rule industry detection mask (Mask_R); Calculation logic: If (Mask_R & Tag_D) == Mask_R, trigger the detection; otherwise, skip.

[0029] Application scenario: Applicable to strict scenarios where the rule only takes effect for multiple industries simultaneously (for example, cross-border financial data needs to comply with the regulatory requirements of multiple countries at the same time).

[0030] 2. Any match mode (OR logic): Judgment condition: There is at least one common industry bit between the data industry identifier (Tag_D) and the rule industry detection mask (Mask_R); Calculation logic: If (Mask_R & Tag_D) ≠ 0, trigger the detection; otherwise, skip.

[0031] Application scenario: Applicable to general scenarios where the rule takes effect in any related industry (for example, basic data format verification rules).

[0032] 3. Example (full match mode): Industry code table definition: Finance = 001, Manufacturing = 010, Medical = 100; Rule R2 settings: Industry detection mask = 011 (need to detect finance + manufacturing industries); Match mode = full match; Data D3 industry identifier = 011 (finance + manufacturing industries): Operation: 011 & 011 = 011 → equal to Mask_R, trigger the detection; Data D4 industry identifier = 010 (only manufacturing industry): Operation: 011 & 010 = 010 ≠ Mask_R, skip the detection.

[0033] Example (any match mode) Rule R3 settings: Industry detection mask = 101 (need to detect finance or medical industries); Match mode = any match; Data D5 industry identifier = 100 (only the medical industry): Operation: 101 & 100 = 100 ≠ 0, triggering detection; Data D6 industry identifier = 010 (only the manufacturing industry): Operation: 101 & 010 = 000, skipping detection.

[0034] The above are only the preferred embodiments of the present invention, which are only used to illustrate the technical solutions of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.

Claims

1. A method for detecting and judging data quality in an industry field, characterized in that: include: (a) Predefined industry coding table: Encode the industry fields involved in the business into binary bit numbers in a fixed order to form an industry index sequence; (b) Rule mask generation: Generate an industry detection mask for each data quality detection rule. The corresponding bit of the industry to be detected in the mask is set to 1, and the remaining bits are set to 0; (c) Data industry identification generation: For the data to be tested, a binary identification is generated according to the industry to which it belongs, and the corresponding bit of the industry to which the data belongs in the identification is set to 1; (d) Dynamic rule matching: Traverse the rule set and perform the following operations on each rule: (d1) Read the industry detection mask Mask_R of the rule and the industry identifier Tag_D of the data; (d2) Calculate the matching value: Result = Mask_R & Tag_D; (d3) If Result ≥ 1, perform quality detection on the current data, otherwise skip the detection; (e) Detection result storage: record the rule identification and violation results that triggered the detection.

2. The method according to claim 1, characterized in that The generation logic of the industry coding table includes: Generate composite identifiers for cross-industry data, allowing a single piece of data to occupy more than one industry position; The industry coding order is strongly bound to the number of mask bits to ensure unique mapping.

3. The method according to claim 1, characterized in that The industry detection mask supports dynamic expansion. When a new industry is added, historical rule compatibility is achieved by increasing the number of mask bits, and the original mask is automatically padded with zeros at high positions.

4. The method according to claim 1, characterized in that In the step (c), the data to be inspected is parsed to extract the industry to which it belongs; and a data industry identifier is generated according to the industry coding table.

5. The method according to claim 1, characterized in that The step (d3) further includes the following matching pattern: Full match mode: detection is triggered when and only when Result == Mask_R; Any matching mode: When Result > 0, the detection is triggered.

6. The method according to claim 1, characterized in that The test result storage in step (e) includes the following fields: Violation rule mask, which records the rule mask value that triggers the detection; Data industry identification, recording the industry identification value of the inspected data; Match the result value Result.

7. The method according to claim 1, characterized in that When traversing quality detection rules, it supports selecting industry matching logic through predefined matching judgment mode parameters.

8. The method according to claim 7, characterized in that Specifically include: (1) Full match mode, namely AND logic: Judgment condition: The data industry identifier Tag_D must completely contain all industry bits set to 1 in the rule industry detection mask Mask_R; Calculation logic: If (Mask_R & Tag_D) == Mask_R, then trigger detection; otherwise skip; Application scenarios: Applicable to strict scenarios where the rules are only effective for a few industries at the same time; (2) Any matching pattern is OR logic: Judgment condition: There is at least one common industry bit between the data industry identifier Tag_D and the rule industry detection mask Mask_R; Calculation logic: If (Mask_R & Tag_D) ≠ 0, then trigger the detection; otherwise skip; Application scenario: Applicable to the general scenario where the rule takes effect in any associated industry.

Images