Information acquisition and treatment method based on HBase

Abstract

The invention relates to the technical field of information collection, and discloses an HBase-based information collection treatment method, which comprises the following steps that: step 1, a user submits a data problem to a problem list through a management tool so as to identify a to-be-processed data problem; step 2, the management team audits the submitted problem content, and determines specific requirements of data inspection and treatment according to an auditing result, including a data processing mode and an expected target data format and content; step 3, converting the audited demand into an executable check rule and forming a corresponding rule configuration file; and 4, extracting the rule by the scheduling program, receiving the to-be-processed data by the target data layer, executing an inspection program to judge whether the data is legal or not, and outputting a corresponding inspection result. According to the method, the technical scheme of cooperative governance based on HBase distributed storage and the rule engine is adopted, and the technical effects of efficiently processing semi-structured data, improving the automation degree of the governance process and flexibly adapting to service requirements are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information collection, in particular to an information collection management method based on HBase. BACKGROUND

[0002] With the rapid development of Internet of Things, industrial Internet and multi-source heterogeneous data collection technology, the data scale of information collection systems is growing exponentially, and the data types are gradually extending from structured to semi-structured and unstructured.

[0003] For example, the existing Chinese patent with publication number CN114661822A discloses an information collection management system and method based on a big data model algorithm. The system includes an HBase database cluster. A data management function block classifies and manages the user electricity information collected by the preposition communication function block and stores it into the HBase database cluster. The HBase database cluster is responsible for data storage and maintenance. The present application sets up an HBase database cluster. The database cluster function block is responsible for data storage and data analysis functions of the system. According to different business and data needs, the database cluster is divided into a relational database cluster and an HBase database cluster. The HBase distributed database can perform powerful and flexible data storage functions according to different collection terminal addresses, improve the scalability of the system, and facilitate system maintenance.

[0004] For example, the existing Chinese patent with publication number CN112364614A discloses a natural person information collection and reporting method based on big data grassroots socialized intelligent management. The method includes: forming a knowledge base based on the state and attributes of each life event; designing a collection form according to department business, avoiding repeated collection through label management, and analyzing, sorting, and merging fields in all department business forms that need to be collected; managing label items; standardizing electronic data as unique codes and entering them into the system data element; customizing and managing information collection forms according to department business needs; configuring form permissions according to the information collection process involved in department needs; removing duplicate fields with the same meaning according to the electronic form configured by the department; comparing with the municipal big data center; comparing and approving the data; and pushing the business data collected by the grassroots according to the requirements and auditing the data.

[0005] The above-mentioned patents have the following shortcomings: first, the processing speed is slow, and it takes half a day to process a large amount of data. Frequent code modifications on a single machine cannot quickly respond to business changes. Second, it is troublesome to change rules. When business needs change, the code needs to be manually changed simultaneously, which cannot keep up with changes. Third, the steps of checking and cleaning are not related to each other.

[0006] To this end, the present application proposes an information collection management method based on HBase to solve the above problems. SUMMARY

[0007] In view of the deficiencies of the prior art, the present application provides an information collection management method based on HBase to solve the problems in the above background.

[0008] To achieve the above object, the present application is implemented by the following technical scheme: an information collection management method based on HBase, comprising:

[0009] Step 1, the user submits the data problem to the problem list through the management tool to identify the data problem to be processed;

[0010] Step 2, the management team audits the content of the submitted problem, and according to the audit result, the specific requirements of data inspection and management are determined, including data processing method and expected target data format and content;

[0011] Step 3, the requirements after the audit are converted into executable inspection rules and the corresponding rule configuration file is formed;

[0012] Step 4, the scheduling program extracts the rules and receives the data to be processed from the target data layer to execute the inspection program to judge whether the data is legal, and outputs the corresponding inspection result;

[0013] Step 5, the inspection result is output to the tool storage layer and stored according to the time stamp index;

[0014] Step 6, the problem data in the inspection result is processed, including removing duplicate data, unifying format and filling missing values;

[0015] Step 7, the processed problem record is archived in the problem list;

[0016] Step 8, a timing scheduling mechanism is designed and implemented to periodically check the newly accessed data.

[0017] Preferably, substep 1.1, the user identifies the data problem to be processed in the management tool, and enters the problem description through the interface, the problem description including data source, problem nature and influence range;

[0018] Substep 1.2, the problem is preliminarily classified according to the problem description input by the user;

[0019] Substep 1.3, after the user confirms the problem description and classification, the problem is submitted to the problem list through the submit button, and the problem record is recorded to the database to generate a unique identifier ID;

[0020] Substep 1.4, after submission, the system automatically updates the problem status to "to be audited", and notifies the management team;

[0021] Sub-step 1.5, the management team can give preliminary feedback on the problem status during the review, set the processing priority according to the problem complexity and urgency;

[0022] There are the following judgments:

[0023] If the problem affects the business process, the priority is set to high;

[0024] If the problem is a potential risk, the priority is set to medium;

[0025] If the problem is a general data error, the priority is set to low.

[0026] Preferably, in step 2, the management team reviews the submitted problem content, and according to the review results, the specific needs of data inspection governance are determined, further comprising:

[0027] Sub-step 2.1: The management team preliminarily reviews the submitted problem content, judges whether the problem belongs to the scope of data governance, and if the problem is beyond the scope of governance, the problem is rejected and the submitter is fed back;

[0028] Sub-step 2.2: According to the problem type and the influence range, set the data inspection governance demand parameters, including the data cleaning range, the format conversion standard, and the missing value filling strategy:

[0029] The data cleaning range: the data field to be processed and the processing method;

[0030] The format conversion standard: the field type and constraint condition of the target data format;

[0031] The missing value filling strategy: the rule based on historical data mean or fixed value filling.

[0032] Sub-step 2.3: According to the urgency of the problem and the data influence range, the processing priority level is determined to be high, medium, and low:

[0033] High priority: the problem causes the business process to be interrupted or the data to be seriously inconsistent;

[0034] Medium priority: the problem affects part of the business but can be temporarily avoided;

[0035] Low priority: the problem involves non-critical data or potential risks;

[0036] Sub-step 2.4: Generate a data inspection governance demand document to clearly define the data processing method, target data format, and inspection rules, and confirm the demand content with the submitter.

[0037] Preferably, in step 3, the reviewed requirements are converted into executable inspection rules and corresponding rule configuration files are formed, further comprising:

[0038] Sub-step 3.1, rule element extraction:

[0039] According to the requirement document generated in step 2, the system extracts the elements required for the inspection rules, including target fields, inspection types and constraint parameters;

[0040] The target field is derived from the data cleaning range set in the requirement document;

[0041] The inspection type is set according to the problem type as value legality, format consistency or field integrity;

[0042] The constraint parameters include the field allowed value range, regular expression template and missing tolerance threshold;

[0043] The setting conditions are as follows:

[0044] If the inspection type is legality, the upper and lower value ranges corresponding to the field are extracted;

[0045] If the inspection type is format consistency, the regular expression is extracted for format matching;

[0046] If the inspection type is field integrity, the maximum allowed missing proportion is set;

[0047] Sub-step 3.2, inspection rule generation:

[0048] Based on the elements extracted in sub-step 3.1, formal inspection rules are automatically generated, each rule defines a corresponding judgment function, which determines whether the data is legal according to the corresponding field and parameter setting, and the inspection judgment logic is as follows:

[0049] If the field is numeric, the rule form is:

[0050] then result=FAIL,

[0051] If the field is a string type, the rule form is:

[0052] then result=FAIL,

[0053] If the field has missing values, the rule form is:

[0054] then result=FAIL,

[0055] Where, match(F k ,Pk ) is to judge whether the field value meets the regular expression template,

[0056] count null (F k ) is the number of null records of field F k , F k is the target field, R k is the value range, total records is the total number of records of the data to be checked;

[0057] Sub-step 3.3, rule configuration file generation and verification:

[0058] Format each rule defined in sub-step 3.2 into a configuration file in JSON or YAML format, which contains field name, rule type, threshold parameter and error prompt description;

[0059] After the configuration file is generated, the structural integrity and semantic consistency are checked;

[0060] Determination condition:

[0061] If the field in the configuration file does not appear in the data model, return the error of invalid field;

[0062] If there are multiple groups of conflicting rules for the same field, mark the conflict for manual processing;

[0063] If all rules are verified, mark the configuration state as executable.

[0064] Preferably, in step 4, the scheduler extracts rules and receives data to be processed by the target data layer to execute the checking program to determine whether the data is legal, and outputs the corresponding checking result, further comprising:

[0065] Sub-step 4.1, rule loading and data sharding:

[0066] The scheduler extracts the rule configuration file generated in step 3 from the tool storage layer, and parses the rule type, target field and parameter threshold;

[0067] The target data layer receives the data to be checked, dynamically shards the data into multiple data blocks according to the data size, and each data block corresponds to an HBase Region;

[0068] The sharding rule is as follows:

[0069] If the data volume exceeds the threshold S max , the number of shards is

[0070] If the data volume is less than the threshold S max , do not shard and directly execute the checking program.

[0071] where S max is the maximum data volume of single processing, N is the number of shards;

[0072] Sub-step 4.2: Start independent inspection process for each data block, and perform inspection according to the fields and threshold parameters in the rule configuration file;

[0073] The inspection program adopts the MapReduce framework, the Map stage applies rules to each data, and the Reduce stage summarizes the inspection results. The judgment logic is:

[0074] If the value of field F k exceeds the range R k , it is marked as illegal data;

[0075] If the format of field F k does not match the regular expression P k , it is marked as illegal data;

[0076] If the missing rate of field F k exceeds the threshold θ k , it is marked as illegal data;

[0077] The formula is as follows:

[0078]

[0079] where D i is the i-th data record;

[0080] Sub-step 4.3, merge the inspection results of all data blocks, and generate a global statistical report, the report content includes:

[0081] Total data volume T, qualified data volume T pass , illegal data volume T fail ;

[0082] Sub-step 4.4: Record the unique identifier of illegal data, the violation field and the error description to the temporary storage table, the record format includes:

[0083] Data row key RowKey i ; violation field F k ;

[0084] Error type ErrorType∈{value out of range, format error, missing limit}.

[0085] Preferably, in step 5, the inspection results are output to the tool storage layer and stored by timestamp index, further comprising:

[0086] Sub-step 5.1: Extract core fields from the inspection result outputted by step 4, including data quality score Q score , total data amount T, qualified data amount T pass , and illegal data amount T fail ;

[0087] Parse the result data and generate structured records, with timestamp t as the index primary key;

[0088] Sub-step 5.2: Calculate data quality score Q pass according to T and T score extracted in sub-step 5.1, with the formula as follows:

[0089]

[0090] Set compliance level according to the score result:

[0091] If Q score ≥ 95%, mark the inspection result as compliant;

[0092] If 80% ≤ Q score < 95%, mark as partially compliant;

[0093] If Q score < 80%, mark as severely abnormal;

[0094] Sub-step 5.3: Create HBase table in the tool storage layer, with timestamp t as row key, and design column family structure as follows:

[0095] Inspection result column family: store Q score , T, T pass , T fail ;

[0096] Metadata column family: store inspection task ID, compliance level, and associated rule configuration file version;

[0097] Sub-step 5.4: Write structured records into HBase table, and update storage status according to the write result:

[0098] If the write is successful, mark the storage status as completed;

[0099] If the write fails, record error log and trigger retry mechanism, with the upper limit of retry times being 3.

[0100] Preferably, in step 6, processing the problematic data in the inspection result includes removing duplicate data, unifying formats, and filling missing values, further including:

[0101] Sub-step 6.1: Extract the illegal data recorded in step 5 from the tool storage layer, and classify them into the following three categories according to the error type:

[0102] Duplicate data: records with duplicate primary key fields;

[0103] Format abnormal data: data with field values that do not conform to regular expressions or numerical ranges;

[0104] Missing data: data with a field missing rate exceeding the threshold.

[0105] Determination conditions:

[0106] If the data row key RowKey i appears multiple times in the temporary storage table, it is marked as duplicate data;

[0107] If the field value violates the format or range constraints in the rule configuration file of step 3, it is marked as format abnormal data;

[0108] If the number of missing field values exceeds the θ k set in the rule configuration file, it is marked as missing data;

[0109] Sub-step 6.2: Perform deduplication on the classified duplicate data, and filter unique records according to the primary key field and timestamp:

[0110] Primary key determination: If the RowKey i of multiple records is the same, it is determined to be a duplicate;

[0111] Timestamp filtering: Keep the record with the latest timestamp t

[0112] Set conditions:

[0113] The primary key field is defined by the HBase table design of the target data layer;

[0114] If the duplicate data has no timestamp field, the first occurrence is kept by default;

[0115] Sub-step 6.3: Process format abnormal and missing data:

[0116] Format unification: Convert field values according to the target format in the rule configuration file;

[0117] Missing value filling: Fill missing values according to the strategy in the rule configuration file:

[0118] If the field is numerical, fill it with the historical data mean μ;

[0119] If the field is enumerated, fill it with the default value;

[0120] If the field is of time type, fill in the current timestamp t.

[0121] Preferably, the timing scheduling mechanism in step 8 automatically performs the data checking task based on a specific time interval.

[0122] A terminal device adopts the HBase-based information collection management method to manage data.

[0123] A storage medium stores computer executable instructions for implementing the HBase-based information collection management method.

[0124] The present application provides an HBase-based information collection management method.

[0125] The present application adopts the technical scheme of HBase-based distributed storage and rule engine collaborative management, achieves efficient processing of semi-structured data, improves the automation degree of management process and flexible adaptation to business needs, and compared with the technical scheme of the prior art which relies on single script checking, manual intervention management process and static rule configuration, solves the problems of low efficiency, rule update lag and fragmented management process in processing large-scale semi-structured data in traditional methods. BRIEF DESCRIPTION OF DRAWINGS

[0126] Figure 1 The flowchart of the present application. DETAILED DESCRIPTION

[0127] In order to enable those skilled in the art to understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0128] The present application will be described in detail below with reference to the drawings:

[0129] Embodiment:

[0130] Please refer to the accompanying drawings Figure 1 The present application provides an HBase-based information collection management method, which comprises:

[0131] Step 1: the user submits data problems to the problem list through the management tool to identify the data problems to be processed;

[0132] Substep 1.2: according to the problem description input by the user, the problem is preliminarily classified;

[0133] Sub-step 1.3, after the user confirms the problem description and classification, the problem is submitted to the problem list through the submit button, and the problem is recorded to the database, generating a unique identifier ID;

[0134] Sub-step 1.4, after submission, the system automatically updates the problem status to "to be audited", and notifies the management team;

[0135] Sub-step 1.5, the management team can give preliminary feedback on the problem status during the audit, and set the processing priority according to the problem complexity and urgency;

[0136] There are the following judgments:

[0137] If the problem affects the business process, the priority is set to high;

[0138] If the problem is a potential risk, the priority is set to medium;

[0139] If the problem is a general data error, the priority is set to low;

[0140] Step 2, the management team audits the submitted problem content, and according to the audit result, determines the specific needs of data inspection governance, including data processing method and expected target data format and content;

[0141] Sub-step 2.1: The management team conducts preliminary audit on the submitted problem content, judges whether the problem belongs to the scope of data governance, and if the problem is beyond the scope of governance, the problem is rejected and the submitter is fed back;

[0142] Sub-step 2.2: According to the problem type and influence range, set the data inspection governance demand parameters, including data cleaning range, format conversion standard, missing value filling strategy:

[0143] Data cleaning range: data fields to be processed and processing method;

[0144] Format conversion standard: field type and constraint condition of target data format;

[0145] Missing value filling strategy: rules based on historical data mean or fixed value filling.

[0146] Sub-step 2.3: According to the urgency of the problem and the influence range of the data, the processing priority level is determined to be high, medium and low:

[0147] High priority: the problem causes business process interruption or serious data inconsistency;

[0148] Medium priority: the problem affects part of the business but can be temporarily avoided;

[0149] Low priority: the problem involves non-critical data or potential risk;

[0150] Sub-step 2.4: Generate a data inspection governance requirement document to specify the data processing method, target data format, and inspection rules, and confirm the requirement content with the submitter;

[0151] Step 3: Convert the audited requirements into executable inspection rules and form the corresponding rule configuration file;

[0152] Sub-step 3.1: Rule element extraction:

[0153] According to the requirement document generated in step 2, the system extracts the elements required for the inspection rules, including the target field, the inspection type, and the constraint parameters;

[0154] The target field is derived from the data cleaning range set in the requirement document;

[0155] The inspection type is set as value legality, format consistency, or field integrity according to the problem type;

[0156] The constraint parameters include the field allowed value range, regular expression template, and missing tolerance threshold;

[0157] The conditions are set as follows:

[0158] If the inspection type is legality, extract the upper and lower value range of the field;

[0159] If the inspection type is format consistency, extract the regular expression for format matching;

[0160] If the inspection type is field integrity, set the maximum allowed missing proportion;

[0161] Sub-step 3.2: Inspection rule generation:

[0162] Based on the elements extracted in sub-step 3.1, automatically generate formal inspection rules, each rule defines a corresponding judgment function, which determines whether the data is legal according to the corresponding field and parameter settings. The inspection judgment logic is as follows:

[0163] If the field is numeric, the rule form is:

[0164] then result = FAIL,

[0165] If the field is a string, the rule form is:

[0166] then result = FAIL,

[0167] If the field has missing values, the rule form is:

[0168] then result = FAIL,

[0169] Where match(F k , P k ) is to judge whether the field value meets the regular expression template,

[0170] count null (F k ) is the number of null records of field F k , F k is the target field, R k is the value range, and total records is the total number of records of the data to be checked;

[0171] Sub-step 3.3, rule configuration file generation and verification:

[0172] Format each rule defined in sub-step 3.2 into a configuration file in JSON or YAML format, which contains field name, rule type, threshold parameter and error prompt description;

[0173] After the configuration file is generated, the structural integrity and semantic consistency are verified;

[0174] Determination condition:

[0175] If the field in the configuration file does not exist in the data model, return the error of invalid field;

[0176] If there are multiple groups of conflicting rules for the same field, mark the conflict for manual processing;

[0177] If all rules are verified, mark the configuration state as executable;

[0178] Step 4, the scheduler extracts rules and receives data to be processed by the target data layer to execute the checking program to determine whether the data is legal, and outputs the corresponding checking result;

[0179] Sub-step 4.1, rule loading and data slicing:

[0180] The scheduler extracts the rule configuration file generated in step 3 from the tool storage layer, and parses the rule type, target field and parameter threshold;

[0181] The target data layer receives the data to be checked, dynamically slices it into multiple data blocks according to the data size, and each data block corresponds to an HBase Region;

[0182] The slicing rule is as follows:

[0183] If the data volume exceeds the threshold S max , the number of slices N

[0184] If the data volume is less than the threshold S max , then the checking procedure is directly executed without sharding.

[0185] Where S max is the maximum data volume for single processing, and N is the number of shards.

[0186] Sub-step 4.2: Start an independent checking process for each data block, and perform the checking according to the field and threshold parameters in the rule configuration file.

[0187] The checking procedure uses the MapReduce framework. The Map stage applies the rules to each data record, and the Reduce stage summarizes the checking results. The judgment logic is as follows:

[0188] If the value of field F k exceeds the range R k , then it is marked as illegal data.

[0189] If the format of field F k does not match the regular expression P k , then it is marked as illegal data.

[0190] If the missing rate of field F k exceeds the threshold θ k , then it is marked as illegal data.

[0191] The formula is as follows:

[0192]

[0193] Where D i is the i-th data record.

[0194] Sub-step 4.3: Merge the checking results of all data blocks to generate a global statistical report, which includes:

[0195] The total data volume T, the qualified data volume T pass , and the illegal data volume T fail .

[0196] Sub-step 4.4: Record the unique identifier of illegal data, the violating field, and the error description to a temporary storage table, and the record format includes:

[0197] Data row key RowKey i ; violating field F k .

[0198] Error type ErrorType∈{value out of range, format error, missing rate exceeding limit}.

[0199] Step 5: Output the checking results to the tool storage layer and store them by timestamp index.

[0200] Sub-step 5.1: Extract core fields from the inspection result output by step 4, including data quality score Q score , total data amount T, qualified data amount T pass , and illegal data amount T fail ;

[0201] Parse the result data and generate a structured record containing timestamp t as the index primary key;

[0202] Sub-step 5.2: Calculate the data quality score Q pass based on T and T score extracted in sub-step 5.1, with the formula:

[0203]

[0204] Set compliance level according to the score result:

[0205] If Q score ≥ 95%, mark the inspection result as compliant;

[0206] If 80% ≤ Q score < 95%, mark as partially compliant;

[0207] If Q score < 80%, mark as serious abnormality;

[0208] Sub-step 5.3: Create an HBase table in the tool storage layer, with timestamp t as the row key, and design the column family structure as follows:

[0209] Inspection result column family: store Q score , T, T pass , T fail ;

[0210] Metadata column family: store inspection task ID, compliance level, and associated rule configuration file version;

[0211] Sub-step 5.4: Write structured records to the HBase table, and update the storage status according to the write result:

[0212] If the write is successful, mark the storage status as completed;

[0213] If the write fails, record error logs and trigger a retry mechanism, with a maximum of 3 retries;

[0214] Step 6: Process the problematic data in the inspection result, including removing duplicate data, unifying formats, and filling missing values;

[0215] Sub-step 6.1: Extract the illegal data recorded in step 5 from the tool storage layer, and classify them into the following three categories according to the error type:

[0216] Duplicate data: records with duplicate primary key fields;

[0217] Format abnormal data: data with field values that do not conform to regular expressions or numerical ranges;

[0218] Missing data: data with a field missing rate exceeding the threshold.

[0219] Judgment conditions:

[0220] If the data row key RowKey i appears multiple times in the temporary storage table, it is marked as duplicate data;

[0221] If the field value violates the format or range constraints in the rule configuration file of step 3, it is marked as format abnormal data;

[0222] If the number of missing values exceeds the θ k set in the rule configuration file, it is marked as missing data;

[0223] Sub-step 6.2: Perform deduplication on the classified duplicate data, and filter unique records according to the primary key field and timestamp:

[0224] Primary key judgment: If the RowKey i of multiple records is the same, it is determined to be a duplicate;

[0225] Timestamp filtering: keep the record with the latest timestamp t

[0226] Set conditions:

[0227] The primary key field is defined by the HBase table design of the target data layer;

[0228] If the duplicate data has no timestamp field, the first occurrence is kept by default;

[0229] Sub-step 6.3: Process format abnormal and missing data:

[0230] Format uniformity: convert field values according to the target format in the rule configuration file;

[0231] Missing value filling: fill missing values according to the strategy in the rule configuration file:

[0232] If the field is numerical, fill the historical data mean μ;

[0233] If the field is enumerated, fill the default value;

[0234] If the field is time type, fill in the current timestamp t;

[0235] Step 7, archive the processed problem records to the problem list;

[0236] Step 8, design and implement a timing scheduling mechanism to periodically check newly accessed data.

[0237] Step 1 ensures that data governance problems are traceable and manageable through a standardized user problem submission process. After the user submits a problem through the management tool, the system automatically classifies the problem, generates a unique identifier, and sets the initial state, avoiding omissions or confusion caused by manual recording. The hierarchical mechanism of problem priority enables the management team to quickly identify critical problems and allocate resources reasonably. For example, after a business process interruption problem is marked as high priority, an emergency handling process is triggered immediately to reduce business losses. At the same time, the automatic notification mechanism of problem submission and status update reduces communication costs and improves problem response efficiency.

[0238] Step 2 converts vague user problems into executable technical indicators through multi-level review and demand parameter setting. The management team determines the governance scope based on business impact, avoiding wasting resources on ineffective problems. By setting data cleaning range, format conversion standard, and missing value filling strategy, it ensures that subsequent rule generation and data processing have clear basis. For example, for the problem of chaotic date format, set the target format to "YYYY-MM-DD" to directly guide the rule configuration in step 3. Priority classification and demand document generation provide standardized input for subsequent automated governance, reducing manual decision-making.

[0239] Step 3 automatically converts business requirements into machine-executable inspection rules, significantly improving the efficiency and accuracy of rule generation. By extracting target fields, inspection types, and constraint parameters, the rule engine can dynamically adapt to different data types. For example, for temperature sensor data, automatically generate a "numeric range -20℃ ~ 80℃" legality rule.

[0240] Step 4 utilizes the distributed characteristics of HBase and the MapReduce framework to achieve efficient parallel inspection of massive data. Data sharding strategy and dynamic threshold setting balance computing resources and processing speed. For example, when processing 10TB of data at a time, it is automatically split into 100 shards for parallel execution, reducing the inspection time to 1 / 20 of the traditional single-machine solution. Global statistical reporting and separate storage of illegal data records provide accurate input for subsequent governance while avoiding pollution of original data. Structured records in temporary storage tables can be directly used for targeted repair in step 6.

[0241] Step 5 realizes traceability and quantitative evaluation of historical inspection results through timestamp index storage and quality score mechanism. Q score, compliance level, etc. are persisted to HBase, supporting quick retrieval by time range. For example, when Q score When the compliance level is below 80%, the rule version traceable to a specific time point is used to assist in troubleshooting configuration errors. The retry mechanism and column family design of the storage state take into account the reliability and query efficiency of data writing, providing a foundation for quality monitoring.

[0242] Step 6 implements targeted repair for illegal data classification, solving problems such as data redundancy, format disorder, and missing values. De-duplication ensures data uniqueness, avoiding misleading business analysis due to repeated calculations. Format unification and missing value filling directly improve data availability. For example, missing temperature data is filled with the historical average of 25°C, meeting the input requirements of subsequent analysis models. The strong association between repair logic and rule configuration files ensures the consistency of repair actions and business goals.

[0243] Step 7 forms a complete evidence chain of the governance process through a closed-loop archiving mechanism. The repaired problem records are persisted to the problem list, supporting post-auditing and rule optimization. For example, frequent "format error" problems can be statistically analyzed to drive the management team to optimize data collection interfaces. The association between archived records and original problems enhances the explainability of the governance process and provides evidence for compliance review.

[0244] Step 8 realizes the continuity and preventiveness of data governance through automated timing tasks. Regularly checking newly accessed data prevents the accumulation of problem data. For example, newly added device data triggers a check immediately after access, preventing format errors from spreading to business systems. The combination of scheduling mechanisms and the elastic expansion capability of HBase dynamically adapts to changes in data size, ensuring the stability and real-time performance of the governance process.

[0245] A terminal device adopts an information collection governance method based on HBase for data governance.

[0246] A storage medium stores computer executable instructions for implementing an information collection governance method based on HBase.

[0247] The terminal device realizes the whole-process closed-loop management from data collection to governance by integrating the HBase-based information collection governance method. The device utilizes the distributed storage capability of HBase to support high-concurrency data writing and real-time query, and ensures efficient processing of massive time-series data in the industrial Internet of Things and intelligent terminal scenarios. For example, in power equipment monitoring, the terminal device can collect sensor data in real time, quickly identify voltage abnormalities or temperature out-of-range problems through the parallel check mechanism of step 4, and automatically clean up data based on the repair strategy of step 6 to prevent dirty data from flowing into the business system. At the same time, the timing scheduling mechanism of step 8 ensures that the device continuously monitors new data to prevent problem accumulation. This scheme enables the terminal device to have "self-governance" capability, reduces dependence on cloud computing power, improves data reliability in edge scenarios, and significantly reduces device operation and maintenance costs by dynamically adapting to business changes through the rule engine.

[0248] The storage medium solidifies the HBase-based governance process to provide "out-of-the-box" governance capabilities for various systems. The computer instructions stored in the medium encapsulate the rule configuration template of step 3, the storage structure design of step 5, and the archiving strategy of step 7, so users can quickly deploy a complete governance process by loading the configuration file without repeated development.

[0249] Although embodiments of the present application have been shown and described, it will be understood by those having ordinary skill in the art that various changes, modifications, substitutions and alterations can be made hereto without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A HBase-based information collection management method, characterized in that, Comprising: Step 1, the user submits data problems to the problem list through the management tool to identify data problems to be processed; Step 2, the management team audits the content of the submitted problem, and according to the audit result, the specific requirements of data inspection governance are determined, including data processing method and expected target data format and content; Step 3, the audited requirements are converted into executable inspection rules and the corresponding rule configuration file is formed; Step 4, the scheduling program extracts the rules and receives the data to be processed from the target data layer to execute the inspection program to determine whether the data is legal, and outputs the corresponding inspection result; Step 5, output the inspection result to the tool storage layer and store it by timestamp index; Step 6, process the problem data in the inspection result, including removing duplicate data, unifying format and filling missing values; Step 7, archive the processed problem record to the problem list; Step 8, design and implement a timing scheduling mechanism to periodically check newly accessed data. 2.The HBase-based information collection management method of claim 1, wherein, Sub-step 1.1, the user identifies the data problem to be processed in the management tool, and enters the problem description through the interface, which includes data source, problem nature and impact range; Sub-step 1.2, the problem is preliminarily classified according to the problem description input by the user; Sub-step 1.3, after the user confirms the problem description and classification, the problem is submitted to the problem list through the submit button, and the problem record is recorded to the database, generating a unique identifier ID; Sub-step 1.4, after submission, the system automatically updates the problem status to "to be audited", and notifies the management team; Sub-step 1.5, the management team can give preliminary feedback on the problem status during the audit period, and set the processing priority according to the problem complexity and urgency; There are the following judgments: If the problem affects the business process, the priority is set to high; If the problem is a potential risk, the priority is set to medium; If the problem is a general data error, the priority is set to low. 3.The HBase-based information collection management method of claim 1, wherein, In step 2, the management team audits the content of the submitted problem, and according to the audit result, the specific requirements of data inspection governance are determined, further including: Sub-step 2.1: The management team preliminarily audits the content of the submitted problem to determine whether the problem belongs to the scope of data governance. If the problem is beyond the scope of governance, the problem is rejected and the submitter is fed back; Sub-step 2.2: According to the problem type and impact range, set the data inspection governance requirement parameters, including data cleaning range, format conversion standard, and missing value filling strategy: The data cleaning range: the data field to be processed and the processing method; The format conversion standard: the field type and constraint condition of the target data format; The missing value filling strategy: the rule based on historical data mean or fixed value filling. Sub-step 2.3: According to the urgency of the problem and the data impact range, the processing priority level is determined to be high, medium and low: High priority: the problem causes the business process to be interrupted or the data to be seriously inconsistent; Medium priority: the problem affects part of the business but can be temporarily avoided; Low priority: the problem involves non-critical data or potential risk; Sub-step 2.4: Generate a data inspection governance requirement document to clearly define the data processing method, target data format and inspection rule, and confirm the requirement content with the submitter.

4. The HBase-based information collection management method of claim 1, wherein, The step 3 further comprises: Sub-step 3.1, rule element extraction: According to the requirement document generated in step 2, the system extracts the elements required for the check rule, including the target field, the check type and the constraint parameter; The target field is derived from the data cleaning range set in the requirement document; The check type is set according to the problem type as value legality, format consistency or field integrity; The constraint parameter includes the field allowed value range, the regular expression template and the missing tolerance threshold; The setting conditions are as follows: If the check type is legality, the upper and lower value ranges of the field are extracted; If the check type is format consistency, the regular expression is extracted for format matching; If the check type is field integrity, the maximum allowed missing proportion is set; Sub-step 3.2, check rule generation: Based on the elements extracted in sub-step 3.1, the formalized check rule is automatically generated, and each rule defines a corresponding judgment function to determine whether the data is legal according to the corresponding field and parameter setting. The check judgment logic is as follows: If the field is numeric, the rule form is: If the field is string, the rule form is: If the field has missing values, the rule form is: Wherein, match(F k ,P k ) is to judge whether the field value conforms to the regular expression template, count null (F k ) is the number of null records of field F k , F k is the target field, R k is the value range, total records is the total number of records of the data to be checked; Sub-step 3.3, rule configuration file generation and verification: Each rule defined in sub-step 3.2 is formatted into a configuration file in JSON or YAML form, which contains the field name, rule type, threshold parameter and error prompt description; After the configuration file is generated, the structural integrity and semantic consistency are verified; The judgment conditions are as follows: If the field in the configuration file does not exist in the data model, return the error of invalid field; If there are multiple conflicting rules for the same field, mark it for manual processing; If all rules are verified, mark the configuration state as executable.

5. The HBase-based information collection management method of claim 1, wherein, In step 4, the scheduler extracts the rules and receives the data to be processed from the target data layer to execute the check program to determine whether the data is legal, and outputs the corresponding check result, which further comprises: Sub-step 4.1, rule loading and data slicing: The scheduler extracts the rule configuration file generated in step 3 from the tool storage layer, parses the rule type, target field and parameter threshold; The target data layer receives the data to be checked, dynamically slices it into multiple data blocks according to the data size, and each data block corresponds to a HBaseRegion; The slicing rule is as follows: If the data volume exceeds the threshold S max , the number of shards If the data volume is less than the threshold S max Then, the checking program is executed directly without sharding. Where S max is the maximum data volume for a single processing, and N is the number of shards. Sub-step 4.2: Start an independent check process for each data block, and execute the check according to the field and threshold parameter in the rule configuration file; The check program uses the MapReduce framework, the Map stage applies the rule to each data, and the Reduce stage summarizes the check result, and the judgment logic is as follows: If the value of field F k exceeds the range R k , it is marked as illegal data; If the format of field F k does not match the regular expression P k , it is marked as illegal data. If the missing rate of field F k exceeds threshold θ k , it is marked as illegal data; The formula is as follows: wherein D i is the i-th data record; Sub-step 4.3, merge the check results of all data blocks to generate a global statistical report, which includes: Total data amount T, qualified data amount T pass , illegal data amount T fail ; Sub-step 4.4: Record the unique identifier of the illegal data, the violation field and the error description to the temporary storage table, and the record format includes: Data row key RowKey i ; Violation field F k ; ErrorType∈{value out of range, format error, missing over limit}.

6. The HBase-based information collection management method of claim 1, wherein, The step 5, the check result is output to the tool storage layer and stored with timestamp index, further comprising: Sub-step 5.1: Extract core fields from the inspection result output by step 4, including data quality score Q score , total data amount T, qualified data amount T pass , and illegal data amount T fail ; Parsing the result data and generating structured records, including timestamp t as index primary key; Sub-step 5.2: Calculate the data quality score Q pass , according to the T and T score extracted in sub-step 5.1 , the formula is: Setting compliance level according to the score result: If Q score ≥ 95%, the marking inspection result is compliant; If 80% < Q score < 95%, then it is marked as partially compliant; If Q score <80%, it is marked as a severe anomaly; Sub-step 5.3: Create HBase table in the tool storage layer, with timestamp t as row key, and design column family structure as follows: Check result column family: store Q score , T, T pass , T fail ; Metadata column family: store check task ID, compliance level and associated rule configuration file version; Sub-step 5.4: Write structured records into HBase table, and update storage status according to the write result: If the write is successful, mark the storage status as completed; If the write fails, record error log and trigger retry mechanism, with a maximum of 3 retries.

7. The HBase-based information collection management method of claim 1, wherein, In step 6, the existing problem data in the check result is processed, including removing duplicate data, unifying format and filling missing values, further comprising: Sub-step 6.1: Extract illegal data recorded in step 5 from the tool storage layer, and classify them into the following three categories according to error type: Duplicate data: records with duplicate primary key fields; Format exception data: data with field values that do not conform to regular expressions or numerical value ranges; Missing data: data with field missing rate exceeding the threshold. Judgment conditions: If the data row key RowKey i If the data row key RowKey i If the data row key RowKey i If the data row key RowKey i If the data row key RowKey <000 If the field value violates the format or range constraints in the rule configuration file of step 3, it is marked as format exception data; If the number of missing values in a field exceeds the θ set in the rule configuration file k , it is marked as missing data; Sub-step 6.2: Perform deduplication operation on the classified duplicate data, and select unique records according to the primary key field and timestamp: Primary key determination: If the RowKey of multiple records are the same, it is determined as a duplicate. i same, it is determined as a duplicate. Timestamp filtering: keep the record with the latest timestamp t and delete the rest of the duplicate records; Set conditions: The primary key field is defined by the HBase table design of the target data layer; If the duplicate data has no timestamp field, the first occurrence data is kept by default; Sub-step 6.3: Process format exception and missing data: Format unification: convert field values according to the target format in the rule configuration file; Missing value filling: fill missing values according to the strategy in the rule configuration file: If the field is numerical, fill with the historical data mean μ; If the field is enumeration, fill with the default value; If the field is time type, fill with the current timestamp t. 8.The HBase-based information collection management method of claim 1, wherein, The timing scheduling mechanism in step 8 automatically executes data check tasks based on specific time intervals.

9. A terminal device, comprising: A HBase-based information collection management method according to claims 1-8 is used for data management.

10. A storage medium, characterized by Computer executable instructions for implementing the HBase-based information collection management method according to claims 1-8 are stored.

Images