Coal transaction data quality control method and system

Abstract

The invention discloses a coal transaction data quality control method and system, relates to the technical field of data quality control, adopts a layered architecture design, and comprises a data source layer, a data acquisition layer, a data cleaning layer, a data storage layer, a data quality monitoring layer, a data application layer and a user interaction layer. The data source layer is used for gathering multi-channel coal transaction related data of an enterprise direct reporting system, a market acquisition interface and a sensor network, and covering transaction core data, logistics information, quality detection data and market quotation data; according to the coal transaction data quality management and control method and system, through the multi-source adaptive acquisition module and the unified data format, in combination with the timestamp and the data source identifier, the problems of manual filling errors and difficult multi-source data adaptation are solved, and the data acquisition accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of data quality control technology, and more specifically, to a method and system for coal transaction data quality control. Background Technology

[0002] With the accelerated digital transformation of the coal industry, massive amounts of data are generated in the coal trading process, covering core transaction data (transaction price, transaction volume, and information of both parties), logistics information (transportation routes, arrival times), quality testing data (ash content, moisture content, calorific value), and market data (price index, supply and demand trends). This data has become a core strategic resource for coal enterprises in decision-making, operational efficiency optimization, and market risk management. The quality of the data directly determines the accuracy of enterprise decisions and market competitiveness.

[0003] However, existing methods for controlling the quality of coal trading data have significant shortcomings and are insufficient to meet the needs of digital transformation:

[0004] The data collection process is not standardized: it often uses a combination of manual entry and simple automated tools. Manual entry is prone to errors in transaction price entry and duplicate transaction volume recording due to human negligence and unfamiliarity with the business. Simple automated collection tools have limited functions and cannot adapt to multi-source data such as enterprise direct reporting, market crawlers, and sensors. They also lack data verification mechanisms, allowing erroneous and duplicate data to enter the system directly.

[0005] Inefficient data storage and integration: Traditional relational databases can only effectively store structured transaction data. Unstructured data (such as quality inspection reports and market research reports) are stored in a scattered manner using file systems. The data lacks connections and forms "data silos". Data sharing across departments and business systems requires manual integration, which is time-consuming, labor-intensive and prone to errors.

[0006] Weak data processing and analysis capabilities: Relying on simple statistical tools such as Excel to summarize data, it is impossible to achieve in-depth mining and advanced analysis, making it difficult to understand market trends and potential risks; at the same time, there is a lack of real-time monitoring of data quality, and data errors are only discovered in subsequent analysis, missing the best time to correct them and affecting the scientific nature of decision-making.

[0007] Lack of data traceability and optimization: The data flow path from collection to application is not recorded, making it impossible to quickly locate the root cause after quality problems occur; the lack of a closed-loop mechanism for user feedback and system optimization prevents the continuous improvement of data quality control capabilities.

[0008] Therefore, a method and system for quality control of coal transaction data is proposed to address the aforementioned technical issues. Summary of the Invention

[0009] In order to overcome the above-mentioned deficiencies of the prior art, this application provides a method and system for quality control of coal transaction data to solve the problems mentioned in the background art.

[0010] To achieve the above objectives, this application provides the following technical solution: a coal transaction data quality control system, adopting a layered architecture design, including a data source layer, a data acquisition layer, a data cleaning layer, a data storage layer, a data quality monitoring layer, a data application layer, and a user interaction layer. Each layer achieves data interaction and collaboration through standard interfaces and protocols.

[0011] The data source layer is used to aggregate coal transaction-related data from multiple channels, including the enterprise direct reporting system, market acquisition interface, and sensor network, covering core transaction data, logistics information, quality inspection data, and market data.

[0012] The data acquisition layer deploys a multi-source data acquisition module that is compatible with Web service interfaces, web crawler protocols, and Internet of Things protocols. It collects data periodically or in real time according to preset rules and adds timestamps and data source identifiers to the collected data.

[0013] The data cleaning layer integrates a data deduplication submodule, a data format verification submodule, and a data integrity check submodule. It uses hash algorithms, regular expressions, and interpolation algorithms to achieve data deduplication, format correction, and missing value completion, respectively.

[0014] The data storage layer uses a relational database to store structured data and a non-relational database to store unstructured data. It integrates multi-source data through a data warehouse and supports data sharing across business systems.

[0015] The data quality monitoring layer deploys a real-time monitoring submodule, a quality assessment submodule, and a source analysis submodule. It uses a stream computing framework to achieve real-time data quality assessment, locates the root cause of problems through data lineage technology, and supports threshold alarms.

[0016] The data application layer provides application services such as precise decision support, customer credit assessment, and market trend forecasting, and outputs business analysis results based on data.

[0017] The user interaction layer provides web and mobile terminal operation interfaces, supporting the visualization of data quality reports and the submission of user feedback.

[0018] Furthermore, the multi-source data acquisition module of the data acquisition layer includes an enterprise direct reporting data acquisition submodule, a market acquisition data submodule, and a sensor data acquisition submodule. The enterprise direct reporting data acquisition submodule connects to the enterprise's internal business system through a Web service interface to collect transaction prices, transaction volumes, and coal variety information in a unified data format. The market acquisition data submodule collects market price indices, supply and demand information, and policy updates through web crawling technology and API interface calls. The sensor data acquisition submodule receives and parses sensor data through an IoT gateway, covering coal quality testing and logistics positioning data.

[0019] Furthermore, the data deduplication submodule of the data cleaning layer employs deduplication logic based on hash algorithms and unique identifier fields. It performs hash operations on key fields such as transaction number and sensor device ID to generate unique hash values. Duplicate data is identified by comparing hash values, and the original field values ​​are further compared when hash collisions occur. The data format verification submodule verifies the format of transaction price values ​​and the "YYYY-MM-DD" format of dates using regular expressions and data type checks. Data with incorrect formats is automatically repaired or marked for manual processing. The data integrity check submodule determines integrity constraints based on business rules. Data with missing key fields is discarded directly, and data with missing non-key fields is filled using interpolation algorithms or default values.

[0020] Furthermore, the real-time monitoring submodule of the data quality monitoring layer deploys monitoring probes at key nodes in the data storage and application layers. It captures data operation events through database triggers and log analysis, and processes data samples in real time using the Apache Flink / Storm stream computing framework. It calculates data quality scores and compares them with thresholds to trigger audible and visual alarms and email notifications. The quality assessment submodule establishes an assessment index system including accuracy, completeness, consistency, timeliness, and redundancy. It periodically extracts sample data for quantitative assessment and generates a visual quality report. The source tracing analysis submodule records the data flow path through data lineage tracking technology, tracing back from the application layer to the acquisition layer to locate the root cause of the problem.

[0021] Furthermore, a method for quality control of coal transaction data includes the following steps:

[0022] S1. Standardized Collection of Multi-Source Data: Through the enterprise direct reporting, market collection, and sensor collection sub-modules of the data collection layer, data from multiple channels is collected according to preset rules, and after adding timestamps and data source identifiers, it is transmitted to the data cleaning layer.

[0023] S2. Intelligent Data Cleaning and Processing: The data cleaning layer sequentially passes through the deduplication, format verification, and integrity check sub-modules, using hash algorithms, regular expressions, and interpolation algorithms to process the data, generating cleaned data that is then transmitted to the data storage layer, and recording problem data logs.

[0024] S3. Layered Data Storage and Integration: The data storage layer stores structured and unstructured data through relational and non-relational databases respectively, integrates data using a data warehouse, and supports cross-departmental data sharing;

[0025] S4. Real-time quality monitoring and source tracing: The data quality monitoring layer captures data operations through monitoring probes, the stream computing framework evaluates the quality in real time, triggers threshold alarms, and locates the root cause of the problem through data lineage technology;

[0026] S5. Closed-loop feedback for continuous optimization: The user interaction layer collects user data quality feedback, combines it with monitoring data, and uses machine learning to classify and sort the feedback information to generate optimization strategies, adjust cleaning rules and monitoring indicators.

[0027] Furthermore, the specific process of S1 is as follows:

[0028] S11: The system loads the predefined acquisition task configuration, including acquisition frequency, data fields, and interface protocol;

[0029] S12: The enterprise direct reporting data collection submodule sends a data request message to the enterprise business system, and receives and parses transaction data transmitted in a unified format;

[0030] S13: The market data collection submodule accesses the target website periodically through web crawlers to parse web page content, or obtains structured market data through API calls;

[0031] S14: The sensor data acquisition submodule listens to sensor messages in real time and receives and parses coal quality and positioning data through the IoT gateway;

[0032] S15: Add a timestamp (format "YYYY-MM-DDHH:MM:SS") and a data source identifier (such as "Enterprise Direct Report-XX Company" or "Sensor-Coal Quality-001") to all collected data and transmit it to the data cleaning layer.

[0033] Furthermore, the specific process of S2 is as follows:

[0034] S21: The data cleaning layer receives the collected data and temporarily stores it in the buffer;

[0035] S22: The data deduplication submodule calculates hash values ​​for key fields, compares hash values ​​to remove duplicate data, and verifies the original field values ​​when hash collisions occur.

[0036] S23: The data format validation submodule validates the data format using regular expressions, automatically repairs correctable errors (such as date format conversion), and marks data with uncorrectable format errors.

[0037] S24: The data integrity check submodule checks whether the data meets the integrity constraints, discards missing data in key fields, and uses Lagrange interpolation to complete missing data in non-key fields.

[0038] S25: Transmit the cleaned data to the data storage layer and record detailed information on format errors and missing data to the log file.

[0039] Furthermore, the specific process of S4 is as follows:

[0040] S41: During system initialization, monitoring probes are deployed at key nodes in the data storage layer and application layer to start monitoring, evaluation, and tracing services;

[0041] S42: The real-time monitoring submodule captures data insertion, update, and deletion events through probes and transmits data samples to the monitoring server.

[0042] S43: The monitoring server uses a stream computing framework to process samples in real time and calculates data quality scores according to the indicator system.

[0043] S44: If the score is lower than the preset threshold, trigger an audible and visual alarm and email notification, and record the alarm information;

[0044] S45: The quality assessment submodule periodically extracts sample data for quantitative assessment and generates a visual report containing indicator scores and trend changes;

[0045] S46: When a data quality problem is discovered, the source tracing analysis submodule traces the data lineage information from the application layer to the collection layer, locates the problem link and cause, and generates a source tracing report.

[0046] Furthermore, the specific process of S5 is as follows:

[0047] S51: Users submit data quality issue feedback through the user interaction layer, including a description of the issue, the scope of the affected data, and the severity.

[0048] S52: The feedback optimization module receives feedback information, associates it with monitoring data, and classifies the feedback information using the K-Means clustering algorithm;

[0049] S53: Sort by problem severity and impact scope, and generate optimization strategies (such as adjusting cleaning rules and optimizing data collection interfaces).

[0050] S54: Submit the optimization strategy for review. Once approved, apply it to the system and update the relevant rules and parameters.

[0051] S55: Regularly review data quality trends, evaluate optimization effects, and iterate and improve management processes and technical solutions.

[0052] The technical effects and advantages of this application are as follows:

[0053] Compared with existing technologies, this coal transaction data quality control method and system achieves more standardized data collection: by using a multi-source adaptation collection module and a unified data format, combined with timestamps and data source identifiers, it solves the problems of manual entry errors and difficulties in multi-source data adaptation, thereby improving the accuracy of data collection.

[0054] More efficient data storage: Adopting a hierarchical storage architecture, it realizes the association and integration of structured and unstructured data, breaks down "data silos", effectively improves the efficiency of cross-departmental data sharing, and reduces the cost of manual integration;

[0055] More real-time quality monitoring: Through the stream computing framework and multi-dimensional indicator system, data quality monitoring can be achieved at the minute level, alarm response time is shortened, and erroneous data is prevented from affecting decision-making;

[0056] More accurate problem tracing: Data lineage technology records the entire data flow path, significantly reducing problem location time and making responsibility clearer; Continuous optimization of control capabilities: Closed-loop feedback mechanism combined with machine learning algorithms enables dynamic adjustment of system rules, and data quality scores continue to improve with optimization iterations, meeting the long-term digital operation needs of coal enterprises. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the system flow of this application;

[0058] Figure 2 This is a flowchart illustrating the method of this application;

[0059] Figure 3 This is a schematic diagram illustrating the specific process of standardized multi-source data collection in the method of this application;

[0060] Figure 4 This is a schematic diagram of the specific process of intelligent data cleaning in the method of this application;

[0061] Figure 5 This is a schematic diagram of the specific process of real-time quality monitoring and traceability in the method of this application;

[0062] Figure 6 This is a schematic diagram illustrating the specific process of closed-loop feedback continuous optimization in the method of this application.

[0063] The attached diagram is labeled as follows: 1. Data source layer; 2. Data acquisition layer; 3. Data cleaning layer; 4. Data storage layer; 5. Data quality monitoring layer; 6. Data application layer; 7. User interaction layer. Detailed Implementation

[0064] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0065] Example 1

[0066] As attached Figure 1 The coal transaction data quality control system shown adopts a layered architecture design, including data source layer 1, data acquisition layer 2, data cleaning layer 3, data storage layer 4, data quality monitoring layer 5, data application layer 6, and user interaction layer 7. Each layer realizes data interaction and collaboration through standard interfaces and protocols.

[0067] The data source layer 1 is used to aggregate coal transaction-related data from multiple channels, including the enterprise direct reporting system, market acquisition interface, and sensor network, covering core transaction data, logistics information, quality inspection data, and market data.

[0068] The data acquisition layer 2 is equipped with a multi-source data acquisition module that is compatible with Web service interfaces, web crawler protocols, and Internet of Things protocols. It collects data periodically or in real time according to preset rules and adds timestamps and data source identifiers to the collected data.

[0069] The multi-source data acquisition module of data acquisition layer 2 includes an enterprise direct reporting data acquisition submodule, a market data acquisition submodule, and a sensor data acquisition submodule. The enterprise direct reporting data acquisition submodule connects to the enterprise's internal business system through a Web service interface and collects transaction price, transaction volume, and coal variety information in a unified data format. The enterprise direct reporting data acquisition submodule sends data request messages conforming to the SOAP protocol to the enterprise's internal business system. The business system returns transaction data in a unified JSON format (e.g., {"Transaction Number":"JY20240601001","Price":580,"Transaction Volume":1000,"Buyer":"Company A","Seller":"Company B"}). The enterprise direct reporting data acquisition submodule parses the data and temporarily stores it.

[0070] The market data collection submodule uses web crawling technology and API calls to collect market price indices, supply and demand information, and policy updates. It periodically accesses platforms such as the "China Coal Market Network" using the Scrapy crawling framework, uses XPath to parse the HTML structure of web pages, and extracts market price indices (such as the "Bohai Rim Thermal Coal Price Index 850 points") and supply and demand information. Simultaneously, it obtains structured data (such as coal production data from the National Bureau of Statistics) through API calls to avoid data collection failures caused by changes in the web page structure.

[0071] The sensor data acquisition submodule receives and parses sensor data through an IoT gateway, covering coal quality testing and logistics positioning data. The market data acquisition submodule listens to MQTT topics (such as "coal / sensor / quality") through the IoT gateway, receives real-time data sent by coal quality sensors (detecting ash and moisture) and GPS sensors (transport vehicle location), and parses the data format (such as: {"Device ID":"S001","Ash":15.2,"Moisture":12.5,"Time":"2024-06-0110:00:00"}).

[0072] Add a uniform format timestamp (“YYYY-MM-DDHH:MM:SS”) and data source identifier (such as “Enterprise Direct Report-Company A”, “Sensor-Coal Quality-S001”, “Market-China Coal Market Network”) to all collected data to ensure data traceability, and then transmit the data to data cleaning layer 3;

[0073] The data cleaning layer 3 integrates a data deduplication submodule, a data format verification submodule, and a data integrity check submodule. It uses hash algorithms, regular expressions, and interpolation algorithms to respectively achieve data deduplication, format correction, and missing value completion.

[0074] The data deduplication submodule of the data cleaning layer 3 adopts deduplication logic based on hash algorithm and unique identifier field. It performs hash operation on key fields such as transaction number and sensor device ID to generate unique hash value. Duplicate data is identified by hash value comparison. When hash collision occurs, the original field value is further compared. The submodule extracts key data fields ("transaction number" for transaction data and "device ID + collection time" for sensor data) and uses MD5 hash algorithm to operate on key fields to generate unique hash value.

[0075] The data deduplication submodule of data cleaning layer 3 compares the hash values ​​of all data. If the hash values ​​are the same, it is determined to be duplicate data, and the earliest collected one is retained. If the hash values ​​are different but the core fields (such as transaction price and transaction volume) are completely identical, the original field values ​​are further verified. After confirming that it is duplicate, the redundant data is removed. For example, if two transaction data have the same hash value "e10adc3949ba59abbe56e057f20f883e5", they are determined to be duplicates, and the first one is retained.

[0076] The data format verification submodule uses regular expressions and data type checks to verify the format of transaction price values ​​and the date format "YYYY-MM-DD". It automatically repairs or marks data with format errors for manual processing. It attempts to automatically repair data with format errors (e.g., converting "2024 / 06 / 01" to "2024-06-01"). Data that cannot be repaired is marked as "format error" and the reason for the error is recorded (e.g., "the price is negative").

[0077] The data integrity check submodule determines integrity constraints based on business rules. Data with missing key fields is discarded directly, while data with missing non-key fields is filled using interpolation algorithms or default values. Based on coal trading business rules, the integrity constraints are determined as follows: transaction data must include "transaction number, price, transaction volume, and information of both buyer and seller", and sensor data must include "device ID, detection value, and acquisition time".

[0078] Data with missing critical fields (such as transaction data lacking "transaction number") is discarded directly; data with missing non-critical fields (such as transaction data lacking "remarks") is filled using Lagrange interpolation (applicable to numeric fields) or by filling with business default values ​​(applicable to text fields).

[0079] The cleaned data is transmitted to data storage layer 4, and detailed information (data content, error type, time) of format errors and missing data is recorded in the log file (stored in the MySQL log table) for subsequent manual review and processing.

[0080] The data storage layer 4 uses a relational database to store structured data and a non-relational database to store unstructured data. It integrates multi-source data through a data warehouse and supports data sharing across business systems.

[0081] Structured data storage: Transaction records, user information, sensor device information and other structured data are stored in a MySQL database. A database sharding and table partitioning strategy is adopted (tables are partitioned by "year-month", such as "Transaction Table_202406"). B+ tree indexes are built for key fields (transaction number, device ID) to improve query efficiency.

[0082] Unstructured data storage: Unstructured data such as coal quality test reports (PDF format) and market research reports (Word format) are stored in a MongoDB database. The GridFS file storage mechanism is used to divide large files into 256KB blocks for storage, while recording file metadata (file name, upload time, and associated transaction number) to facilitate subsequent related queries.

[0083] Multi-source data integration: Through the Hive data warehouse, structured data in MySQL and unstructured data metadata in MongoDB are linked and integrated according to key fields such as "transaction number" and "time". A data warehouse partition table is established (partitioned by "date", such as "dt=20240601"). This supports data query and analysis across departments and business systems. For example, by using "transaction number JY20240601001", transaction data, corresponding coal quality test report metadata, and market information data can be queried simultaneously.

[0084] The data quality monitoring layer 5 deploys a real-time monitoring submodule, a quality assessment submodule, and a source analysis submodule. It uses a stream computing framework to achieve real-time data quality assessment, locates the root cause of problems through data lineage technology, and supports threshold alarms.

[0085] The real-time monitoring submodule of the data quality monitoring layer 5 deploys monitoring probes at key nodes of the data storage layer 4 and the application layer. It captures data operation events through database triggers and log analysis, and processes data samples in real time using the Apache Flink / Storm stream computing framework. It calculates data quality scores and compares them with thresholds to trigger audible and visual alarms and email notifications.

[0086] System initialization configuration:

[0087] Deploy monitoring probes: Deploy monitoring probes at key nodes such as MySQL data insertion / update / deletion interfaces, MongoDB file upload interfaces, and Hive data query interfaces to capture data operation events;

[0088] Set evaluation indicators: Establish a data quality evaluation indicator system, including accuracy (consistency rate of data with authoritative sources), completeness (percentage of data without missing fields), consistency (cross-system data consistency rate), timeliness (percentage of data collection and generation time difference ≤ 10 minutes), and redundancy (percentage of duplicate data). The initial weight of each indicator is set to... ,in, , , , , These correspond to the weights of accuracy, completeness, consistency, timeliness, and redundancy, respectively.

[0089] Set alarm threshold: trigger an alarm when the overall data quality score is below 80; overall data quality score The calculation formula is: ,in The corresponding weights for accuracy, completeness, consistency, timeliness, and redundancy are assigned. To score the indicator, The total number of indicators;

[0090] The quality assessment submodule establishes an assessment index system including accuracy, completeness, consistency, timeliness, and redundancy. It periodically extracts sample data for quantitative assessment and generates a visual quality report. The probe captures data operation events (such as inserting a transaction record into MySQL or uploading a test report to MongoDB), collects data samples (including data content, operation time, and operation node), and transmits them to the monitoring server. The monitoring server uses the Apache Flink stream computing framework, establishes a 5-minute sliding window (1-minute sliding step), processes data samples in real time, and calculates scores for each quality index.

[0091] Accuracy is The formula for calculating the consistency rate between the compared data and the enterprise's ERP system (authoritative source) is as follows: ;

[0092] Completeness is The percentage of data without missing fields is calculated using the following formula: ;

[0093] Timeliness The percentage of data collection time and generation time with a difference of ≤10 minutes is calculated using the following formula: ;

[0094] Calculate the overall score. If the score is below 80, immediately trigger an audible and visual alarm (the audible and visual alarm connected to the monitoring server) and an email notification (sent to the data administrator's email address, including the alarm time, the problem data ID, and the indicator score). At the same time, record the alarm information to the MySQL alarm table.

[0095] The quality assessment submodule regularly extracts sample data from data storage layer 4 on a daily / weekly basis (the sample size is 10% of the total data for that day / week) and quantitatively calculates the scores of each quality indicator.

[0096] Generate a visual quality assessment report, using line charts to show the trend of indicator scores (e.g., the accuracy score has increased from 92% to 95% in the past 7 days) and pie charts to show the distribution of problematic data (e.g., 60% of the data has incorrect formatting, 30% has missing data, and 10% has duplicate data), and present it to the user through the user interaction layer 7;

[0097] The source tracing and analysis submodule records the data flow path through data lineage tracking technology, tracing the data from the application layer to the collection layer to locate the root cause of the problem.

[0098] When a data quality issue is detected (such as the inaccuracy of a certain transaction data), the source analysis submodule initiates data lineage tracing:

[0099] By tracing the data lineage stored in the Neo4j graph database (nodes are “data source - collection submodule - cleaning rules - storage table - application service”, and edges are “collected from - cleaned through - stored in - applied to”), we can trace upwards step by step from the data application layer 6 (such as the decision support submodule).

[0100] Example: The accuracy of certain transaction data is substandard. Tracing back, it was found that the data came from "Enterprise Direct Reporting - Company A". After being cleaned by "Hash Deduplication Rule V1.0" and "Format Validation Rule V2.0", it was stored in "Transaction Table_20240601". The root cause of the problem was an error in the original data entry of Company A's business system.

[0101] Generate a source tracing analysis report to identify the problematic stage (data collection layer), the cause (error in the original data), and the responsible party (Company A), providing a basis for problem resolution.

[0102] The data application layer 6 provides application services such as precise decision support, customer credit assessment, and market trend prediction, and outputs business analysis results based on data.

[0103] If users encounter data quality issues (such as missing coal quality indicators) when using data application layer 6 services (e.g., viewing decision reports and credit assessment results), they can submit feedback through the online form or email in user interaction layer 7. Feedback information includes: problem description ("missing coal quality indicators"), affected data ID ("JY20240601002"), problem severity (high / medium / low), and suggested solution ("add mandatory coal quality indicator validation").

[0104] The feedback collection submodule receives feedback information, stores it in the MySQL feedback table, and automatically associates it with the monitoring records of the data (such as whether the data has triggered an alarm).

[0105] The K-Means clustering algorithm is used to classify feedback information, grouping feedback with the same question type (e.g., all being "missing data") and involving the same data type (e.g., all being "transaction data") into one category. The clustering objective function is... ,in The number of clusters is dynamically adjusted based on the total feedback, and is generally set to 5-10. For the first A cluster, As a feedback sample, As the cluster center;

[0106] The feedback is sorted by "severity × number of data points affected". For example, feedback with "high severity + affecting 100 data points" has a higher priority than feedback with "low severity + affecting 10 data points".

[0107] For high-priority feedback, the system generates targeted optimization strategies:

[0108] If the feedback is "missing data", the optimization strategy is to "update the constraint rules of the data integrity check submodule and set the missing field as a required field".

[0109] If the feedback is "format error", the optimization strategy is to "upgrade the regular expression of the data format validation submodule to cover more abnormal formats";

[0110] The optimization strategy is submitted to the technical staff for review. Once approved, it is automatically applied to the system: such as updating the integrity constraint rules in the Drools rule engine or modifying the regular expression configuration file.

[0111] Example: In response to feedback about "missing coal quality indicators", the optimization strategy is to "add mandatory coal quality indicator verification to data collection layer 2, and prevent enterprises from submitting data without coal quality indicators". After approval, the form verification rules of the enterprise direct reporting data collection sub-module will be updated.

[0112] Regularly (monthly) review the effectiveness of the optimization strategy and compare the data quality index scores before and after optimization (e.g., the percentage of missing data was 15% before optimization and decreased to 5% after optimization) and the number of user feedback (e.g., the number of similar feedback items decreased from 30 per month to 5).

[0113] Based on the evaluation results, further iterate and optimize the strategy: if the optimization effect does not meet expectations (e.g., the proportion of missing data only drops to 12%), reanalyze the feedback information and adjust the optimization strategy (e.g., add secondary verification after data collection); if the effect meets the standard, solidify the optimization strategy into the system default rule, and start the next round of feedback collection and optimization to form a continuous improvement closed loop.

[0114] User interaction layer 7 provides web and mobile terminal operation interfaces, supporting the visualization of data quality reports and the submission of user feedback.

[0115] Example 2

[0116] Please see Figures 2-6 A method for quality control of coal transaction data, the method comprising the following steps:

[0117] S1. Standardized collection of multi-source data: Through the enterprise direct reporting, market collection, and sensor collection sub-modules of data collection layer 2, multi-channel data is collected according to preset rules, and after adding timestamps and data source identifiers, it is transmitted to data cleaning layer 3.

[0118] The specific process for S1 is as follows:

[0119] S11: The system loads the predefined acquisition task configuration, including acquisition frequency, data fields, and interface protocol;

[0120] S12: The enterprise direct reporting data collection submodule sends a data request message to the enterprise business system, and receives and parses transaction data transmitted in a unified format;

[0121] S13: The market data collection submodule accesses the target website periodically through web crawlers to parse web page content, or obtains structured market data through API calls;

[0122] S14: The sensor data acquisition submodule listens to sensor messages in real time and receives and parses coal quality and positioning data through the IoT gateway;

[0123] S15: Add timestamps (format "YYYY-MM-DDHH:MM:SS") and data source identifiers (such as "Enterprise Direct Report-XX Company" or "Sensor-Coal Quality-001") to all collected data and transmit them to data cleaning layer 3.

[0124] S2. Intelligent Data Cleaning and Processing: Data cleaning layer 3 sequentially passes through deduplication, format verification, and integrity check sub-modules, using hash algorithms, regular expressions, and interpolation algorithms to process the data, generating cleaned data that is then transmitted to data storage layer 4, where a log of problematic data is recorded.

[0125] The specific process for S2 is as follows:

[0126] S21: Data cleaning layer 3 receives the collected data and temporarily stores it in the buffer;

[0127] S22: The data deduplication submodule calculates hash values ​​for key fields, compares hash values ​​to remove duplicate data, and verifies the original field values ​​when hash collisions occur.

[0128] S23: The data format validation submodule validates the data format using regular expressions, automatically repairs correctable errors (such as date format conversion), and marks data with uncorrectable format errors.

[0129] S24: The data integrity check submodule checks whether the data meets the integrity constraints, discards missing data in key fields, and uses Lagrange interpolation to complete missing data in non-key fields.

[0130] S25: Transmit the cleaned data to data storage layer 4, and record detailed information on format errors and missing data to the log file.

[0131] S3. Layered Data Storage and Integration: Data storage layer 4 stores structured and unstructured data through relational databases and non-relational databases respectively, integrates data using a data warehouse, and supports cross-departmental data sharing;

[0132] S4. Real-time quality monitoring and source tracing: Data quality monitoring layer 5 captures data operations through monitoring probes, the stream computing framework evaluates quality in real time, triggers threshold alarms, and locates the root cause of the problem through data lineage technology;

[0133] The specific process for S4 is as follows:

[0134] S41: During system initialization, monitoring probes are deployed at key nodes in the data storage layer 4 and application layer to start monitoring, evaluation, and tracing services;

[0135] S42: The real-time monitoring submodule captures data insertion, update, and deletion events through probes and transmits data samples to the monitoring server.

[0136] S43: The monitoring server uses a stream computing framework to process samples in real time and calculates data quality scores according to the indicator system.

[0137] S44: If the score is lower than the preset threshold, trigger an audible and visual alarm and email notification, and record the alarm information;

[0138] S45: The quality assessment submodule periodically extracts sample data for quantitative assessment and generates a visual report containing indicator scores and trend changes;

[0139] S46: When a data quality problem is discovered, the source tracing analysis submodule traces the data lineage information from the application layer to the collection layer, locates the problem link and cause, and generates a source tracing report.

[0140] S5. Closed-loop feedback for continuous optimization: User interaction layer 7 collects user data quality feedback, combines it with monitoring data, and uses machine learning to classify and sort the feedback information to generate optimization strategies, adjust cleaning rules and monitoring indicators.

[0141] The specific process for S5 is as follows:

[0142] S51: Users submit data quality issue feedback through user interaction layer 7, including issue description, affected data range, and severity;

[0143] S52: The feedback optimization module receives feedback information, associates it with monitoring data, and classifies the feedback information using the K-Means clustering algorithm;

[0144] S53: Sort by problem severity and impact scope, and generate optimization strategies (such as adjusting cleaning rules and optimizing data collection interfaces).

[0145] S54: Submit the optimization strategy for review. Once approved, apply it to the system and update the relevant rules and parameters.

[0146] S55: Regularly review data quality trends, evaluate optimization effects, and iterate and improve management processes and technical solutions.

[0147] Finally: The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A coal transaction data quality control system, characterized in that, The architecture adopts a layered design, including a data source layer, a data acquisition layer, a data cleaning layer, a data storage layer, a data quality monitoring layer, a data application layer, and a user interaction layer. Each layer achieves data interaction and collaboration through standard interfaces and protocols. The data source layer is used to aggregate coal transaction-related data from multiple channels, including the enterprise direct reporting system, market acquisition interface, and sensor network, covering core transaction data, logistics information, quality inspection data, and market data. The data acquisition layer deploys a multi-source data acquisition module that is compatible with Web service interfaces, web crawler protocols, and Internet of Things protocols. It collects data periodically or in real time according to preset rules and adds timestamps and data source identifiers to the collected data. The data cleaning layer integrates a data deduplication submodule, a data format verification submodule, and a data integrity check submodule. It uses hash algorithms, regular expressions, and interpolation algorithms to achieve data deduplication, format correction, and missing value completion, respectively. The data storage layer uses a relational database to store structured data and a non-relational database to store unstructured data. It integrates multi-source data through a data warehouse and supports data sharing across business systems. The data quality monitoring layer deploys a real-time monitoring submodule, a quality assessment submodule, and a source analysis submodule. It uses a stream computing framework to achieve real-time data quality assessment, locates the root cause of problems through data lineage technology, and supports threshold alarms. The data application layer provides application services such as precise decision support, customer credit assessment, and market trend forecasting, and outputs business analysis results based on data. The user interaction layer provides web and mobile terminal operation interfaces, supporting the visualization of data quality reports and the submission of user feedback.

2. The coal transaction data quality control system according to claim 1, characterized in that: The multi-source data acquisition module of the data acquisition layer includes an enterprise direct reporting data acquisition submodule, a market data acquisition submodule, and a sensor data acquisition submodule; The enterprise direct reporting data collection submodule connects with the enterprise's internal business system through a Web service interface to collect transaction price, transaction volume, and coal type information in a unified data format. The market data acquisition submodule collects market price indices, supply and demand information, and policy updates through web crawling technology and API interface calls; the sensor data acquisition submodule receives and parses sensor data through an IoT gateway, covering coal quality testing and logistics positioning data.

3. The coal transaction data quality control system according to claim 1, characterized in that: The data cleaning layer's deduplication submodule employs deduplication logic based on hash algorithms and unique identifier fields. It performs hash operations on key fields such as transaction number and sensor device ID to generate unique hash values. Duplicate data is identified by comparing hash values, and the original field values ​​are further compared when hash collisions occur. The data format verification submodule verifies the format of transaction price values ​​and the "YYYY-MM-DD" format of dates using regular expressions and data type checks. Data with incorrect formats is automatically repaired or marked for manual processing. The data integrity check submodule determines integrity constraints based on business rules. Data with missing key fields is discarded directly, and data with missing non-key fields is filled using interpolation algorithms or default values.

4. The coal transaction data quality control system according to claim 1, characterized in that: The real-time monitoring submodule of the data quality monitoring layer deploys monitoring probes at key nodes in the data storage layer and application layer. It captures data operation events through database triggers and log analysis, and processes data samples in real time using the Apache Flink / Storm stream computing framework. It calculates data quality scores and compares them with thresholds to trigger audible and visual alarms and email notifications. The quality assessment submodule establishes an assessment index system that includes accuracy, completeness, consistency, timeliness, and redundancy. It periodically extracts sample data for quantitative assessment and generates visual quality reports. The source tracing analysis submodule records the data flow path through data lineage tracking technology, tracing back from the application layer to the collection layer to locate the root cause of the problem.

5. A method for quality control of coal transaction data, applied to the system described in any one of claims 1-4, characterized in that: The method includes the following steps: S1. Standardized Collection of Multi-Source Data: Through the enterprise direct reporting, market collection, and sensor collection sub-modules of the data collection layer, data from multiple channels is collected according to preset rules, and after adding timestamps and data source identifiers, it is transmitted to the data cleaning layer. S2. Intelligent Data Cleaning and Processing: The data cleaning layer sequentially passes through the deduplication, format verification, and integrity check sub-modules, using hash algorithms, regular expressions, and interpolation algorithms to process the data, generating cleaned data that is then transmitted to the data storage layer, and recording problem data logs. S3. Layered Data Storage and Integration: The data storage layer stores structured and unstructured data through relational and non-relational databases respectively, integrates data using a data warehouse, and supports cross-departmental data sharing; S4. Real-time quality monitoring and source tracing: The data quality monitoring layer captures data operations through monitoring probes, the stream computing framework evaluates the quality in real time, triggers threshold alarms, and locates the root cause of the problem through data lineage technology; S5. Closed-loop feedback for continuous optimization: The user interaction layer collects user data quality feedback, combines it with monitoring data, and uses machine learning to classify and sort the feedback information to generate optimization strategies, adjust cleaning rules and monitoring indicators.

6. The method for quality control of coal transaction data according to claim 5, characterized in that: The specific process of S1 is as follows: S11: The system loads the predefined acquisition task configuration, including acquisition frequency, data fields, and interface protocol; S12: The enterprise direct reporting data collection submodule sends a data request message to the enterprise business system, and receives and parses transaction data transmitted in a unified format; S13: The market data collection submodule accesses the target website periodically through web crawlers to parse web page content, or obtains structured market data through API calls; S14: The sensor data acquisition submodule listens to sensor messages in real time and receives and parses coal quality and positioning data through the IoT gateway; S15: Add timestamps and data source identifiers to all collected data and transmit them to the data cleaning layer.

7. The method for quality control of coal transaction data according to claim 6, characterized in that: The specific process of S2 is as follows: S21: The data cleaning layer receives the collected data and temporarily stores it in the buffer; S22: The data deduplication submodule calculates hash values ​​for key fields, compares hash values ​​to remove duplicate data, and verifies the original field values ​​when hash collisions occur. S23: The data format validation submodule validates the data format using regular expressions, automatically repairs correctable errors, and marks data with uncorrectable format errors. S24: The data integrity check submodule checks whether the data meets the integrity constraints, discards missing data in key fields, and uses Lagrange interpolation to complete missing data in non-key fields. S25: Transmit the cleaned data to the data storage layer and record detailed information on format errors and missing data to the log file.

8. The method for quality control of coal transaction data according to claim 5, characterized in that: The specific process of S4 is as follows: S41: During system initialization, monitoring probes are deployed at key nodes in the data storage layer and application layer to start monitoring, evaluation, and tracing services; S42: The real-time monitoring submodule captures data insertion, update, and deletion events through probes and transmits data samples to the monitoring server. S43: The monitoring server uses a stream computing framework to process samples in real time and calculates data quality scores according to the indicator system. S44: If the score is lower than the preset threshold, trigger an audible and visual alarm and email notification, and record the alarm information; S45: The quality assessment submodule periodically extracts sample data for quantitative assessment and generates a visual report containing indicator scores and trend changes; S46: When a data quality problem is discovered, the source tracing analysis submodule traces the data lineage information from the application layer to the collection layer, locates the problem link and cause, and generates a source tracing report.

9. A method for quality control of coal transaction data according to claim 5, characterized in that: The specific process of S5 is as follows: S51: Users submit data quality issue feedback through the user interaction layer, including a description of the issue, the scope of the affected data, and the severity. S52: The feedback optimization module receives feedback information, associates it with monitoring data, and classifies the feedback information using the K-Means clustering algorithm; S53: Sort problems by severity and scope of impact, and generate optimization strategies; S54: Submit the optimization strategy for review. Once approved, apply it to the system and update the relevant rules and parameters. S55: Regularly review data quality trends, evaluate optimization effects, and iterate and improve management processes and technical solutions.

Images