A method and system for automatic conversion and verification of fuel data
By forming fuel data association units and field-level mapping chains, the problem of inconsistent data standards across security partitions was solved, enabling automated processing and anomaly handling of fuel data, and improving data flow efficiency and accuracy.
Patent Information
- Application Number
- CN202611337894.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-31
- Publication Date
- 2026-11-17
AI Technical Summary
In the process of transferring existing fuel data across safety zones, data standards are not uniform, and it is difficult to uniformly convert multiple types of data. Verification and anomaly handling rely on manual processes, resulting in low processing efficiency, large workload, and a high risk of errors.
By acquiring multi-source fuel data, forming related units based on fuel business identifiers and time information, establishing field-level mapping chains, extracting and transforming data, and performing integrity, logical consistency, and mapping consistency checks, recording transformation tracking information, automatically handling anomalies, and generating transformation reports.
It has enabled automated processing of fuel data, improved the efficiency and accuracy of cross-safety zone transfer, reduced manual processing workload, timely detected and handled data anomalies, and provided a reliable data foundation.
Smart Images

Figure CN123115114A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of fuel data verification, specifically relating to an automatic conversion and verification method and system for fuel data. Background Technology
[0002] With the continuous advancement of digitalization and intelligentization in thermal power plants, fuel management systems are gradually evolving towards centralized data governance, automated data collection, intelligent analysis, and unmanned operation. During operation, fuel systems generate various types of business data, including coal arrival data, contract data, and sampling and analysis data. These data typically originate from different business systems and serve various business scenarios such as coal yard management, metering management, laboratory management, and subsequent analysis and decision-making. To support upper-level applications such as digital coal yards, unmanned operation, and fuel data centers, it is necessary to uniformly process fuel data from different sources and convert it into data that the target system can recognize and use.
[0003] In practical applications, due to network security partitioning and the independent deployment of different business systems, direct data transfer between different security partitions is difficult. For example, fuel-related data such as coal arrival data, contract data, and sampling and analysis data may be located in different security partitions, while the target business system is located in another security partition. This prevents the relevant fuel data from being directly transmitted to the target system, making it difficult for digital coal yards and unattended systems to obtain timely quantitative, qualitative, and price-related data such as coal type, batch, weight, and analysis results. Furthermore, differences in data standards, field definitions, data types, units of measurement, and encoding rules exist between different systems. Even if data transmission is completed, further format adaptation and data conversion are usually required.
[0004] In current fuel data processing, there is often a lack of a unified data mapping and transformation mechanism for different types of data, such as structured tabular data and unstructured log data. This requires staff to extract the necessary information from different data sources and then filter, organize, and enter it according to the target system or a fixed report format. This method is not only cumbersome and involves a large amount of repetitive work, but it is also prone to problems such as numerical entry errors, data omissions, and inconsistencies in logic during manual data entry. When the data volume is large or the data sources are numerous, manual processing methods are insufficient to meet the fuel system's requirements for data timeliness and accuracy.
[0005] Furthermore, existing data processing methods typically require manual verification of data integrity, logical consistency, and compatibility with the target system after data conversion. Corrections are only made when anomalies are found, resulting in a delay in anomaly detection and handling. Simultaneously, there is a lack of effective coordination between data filtering, format conversion, validation, anomaly handling, and result feedback, making it difficult to provide timely feedback and tracking of data processing results. This necessitates a significant amount of repetitive verification and post-processing correction work for staff. Summary of the Invention
[0006] The purpose of this invention is to overcome the problems of inconsistent data standards, difficulty in uniformly converting multiple types of data, and reliance on manual verification, anomaly handling, and result feedback in the process of transferring fuel data across security zones. This results in low data processing efficiency, large workload, and a high risk of errors. The invention provides an automatic fuel data conversion and verification method and system.
[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention proposes an automatic conversion and verification method for fuel data, comprising the following steps: Obtain multi-source fuel data from the source security partition, and associate fuel data from different data sources based on fuel service identifiers and time information in the multi-source fuel data to form fuel data association units; Based on the data standards of the target system in the target safety partition, the core fields to be transformed are determined from the fuel data association unit. Based on the field attributes and data sources of the core fields, a field-level mapping chain is established between the source fields, transformation rules and target fields. Data is extracted and transformed from the core fields according to the field-level mapping chain to obtain the target format data. The source data, transformation rules and transformation results corresponding to each core field are recorded, and transformation tracking information corresponding to the target format data is generated. Based on the data constraints of the target system and the correlation between different fuel data in the fuel data association unit, the target format data is subjected to integrity verification, logical consistency verification and mapping consistency verification to obtain the data verification results. When the data verification result is abnormal, the abnormal field and its corresponding source data and transformation rules are determined according to the transformation tracking information. Data repair and re-transformation are performed on the abnormal field according to the abnormality type. The re-transformed data is then verified again to obtain the verified fuel data. Based on the verified fuel data and the data conversion and verification process, a conversion report is generated, and the verified fuel data is output to the target system corresponding to the target security partition.
[0008] Furthermore, the method for acquiring multi-source fuel data from the source security partition, and then associating fuel data from different data sources based on the fuel service identifier and time information in the multi-source fuel data to form fuel data association units is as follows: Batch identifiers, data source identifiers, and data generation times are extracted from coal arrival data, contract data, and sampling and chemical analysis data, respectively, and the time base for data generation times from different data sources is unified. Fuel data with the same batch identifier and whose data generation time meets the preset time association conditions are identified as the same fuel data association unit. For fuel data lacking a unified batch identifier, a composite association identifier is generated based on coal type information, weight information, and data generation time. The corresponding fuel data association unit is then determined based on the composite association identifier.
[0009] Furthermore, based on the data standards of the target system in the target safety partition, the specific method for determining the core fields to be transformed from the fuel data association unit is as follows: Obtain the target field template corresponding to the target system; The field attributes of each source field in the fuel data association unit are matched with the target field template to determine the source field with the corresponding target field; Based on the required attributes of the target field and the fuel business relationship, the core field is determined from the source fields that have the corresponding target field, and the corresponding target field identifier is configured for the core field.
[0010] Furthermore, based on the field attributes of the core fields and the data source, the specific method for establishing a field-level mapping chain between the source field, transformation rules, and target field is as follows: For structured tabular data, obtain the field name, data type, unit of measurement, and encoding method of the source field; Determine the data type conversion rules, unit of measurement conversion rules, and encoding conversion rules between the source field and the target field respectively; Establish a field-level mapping chain according to the execution order of source field, data type conversion rule, unit of measurement conversion rule, encoding conversion rule and target field, and configure the mapping chain identifier and rule version information for the field-level mapping chain.
[0011] Furthermore, the method for establishing field-level mapping chains is as follows: Determine the corresponding log parsing template based on the source of the log data, and use the log parsing template to identify key field identifiers, field values, and data generation time in the log data; Based on the preset correspondence between the key field identifier and the target field identifier, the identified field value is associated with the corresponding target field; Determine the data type conversion rules, unit of measurement conversion rules, and encoding conversion rules corresponding to the field values, and establish a field-level mapping chain for unstructured data according to the correspondence between log data, identification fields, conversion rules, and target fields.
[0012] Furthermore, based on the field-level mapping chain, data is extracted and transformed from the core fields to obtain the target format data. The source data, transformation rules, and transformation results corresponding to each core field are recorded. The specific method for generating transformation tracking information corresponding to the target format data is as follows: Following the field-level mapping chain, perform data type conversion, unit of measurement conversion, and encoding conversion on the core fields in sequence to obtain the target field value corresponding to the target field. For each target field value, record the corresponding source data identifier, source field identifier, field-level mapping chain identifier, rule version information, intermediate transformation results, and final transformation results; The target field values and their corresponding record information belonging to the same fuel data association unit are associated with the same conversion tracking identifier to form conversion tracking information.
[0013] Furthermore, based on the data constraints of the target system and the correlation between different fuel data in the fuel data association unit, the target format data is subjected to integrity verification, logical consistency verification, and mapping consistency verification. The specific method for obtaining the data verification results is as follows: Based on the required attributes in the target field template, check whether the corresponding target field value exists to obtain the integrity verification result; Based on the pre-established field dependencies between different target fields in the fuel data association unit, logical constraint judgments are performed on the values of the interrelated target fields to obtain the logical consistency verification results. The data type, unit of measurement, and encoding method of the target field value are compared with the data type constraints, unit of measurement constraints, and encoding rules of the corresponding target field to obtain the mapping consistency verification result. The abnormal fields and their corresponding abnormal types are determined based on the integrity verification results, logical consistency verification results, and mapping consistency verification results.
[0014] Furthermore, when anomalies are found in the data verification results, the abnormal fields, their corresponding source data, and transformation rules are determined based on the transformation tracking information. Data repair and re-transformation are performed on the abnormal fields according to the anomaly type. The re-transformed data is then verified again to obtain the verified fuel data. The specific method is as follows: Based on the conversion tracking identifier corresponding to the abnormal field, reverse query the source field, field-level mapping chain, and conversion rule corresponding to the abnormal field; When the exception type is a data missing exception, the source field is re-extracted from the corresponding source data. When the exception type is a data type exception, a unit of measurement exception, or a coding exception, the corresponding data type conversion rule, unit of measurement conversion rule, or coding conversion rule shall be redefined. When the exception type is a logical consistency exception, the associated fields related to the exception field are determined based on the field dependency relationship; Perform partial re-transformation on the abnormal fields and related fields affected by the abnormal fields, and perform incremental verification only on the abnormal fields and related fields after re-transformation to obtain the verified fuel data.
[0015] Furthermore, it also includes: Record the anomaly type, anomaly count, and data repair results for each field-level mapping chain within a preset statistical period; The rule status corresponding to the field-level mapping chain is determined based on the number of exceptions and the result of the re-verification after repair. When the number of exceptions reaches the preset exception condition, the priority of the current conversion rule is reduced, and an alternative conversion rule is determined from the candidate conversion rules that match the corresponding source field and target field. The corresponding fields are transformed and validated using alternative transformation rules, and the rule version information in the field-level mapping chain is updated based on the validation results.
[0016] Secondly, this invention proposes an automatic fuel data conversion and verification system, comprising: The data association module is used to acquire multi-source fuel data in the source security partition, and associate fuel data from different data sources based on the fuel business identifier and time information in the multi-source fuel data to form a fuel data association unit. The mapping construction module is used to determine the core fields to be transformed from the fuel data association unit according to the data standards of the target system in the target safety partition, and to establish a field-level mapping chain between the source fields, transformation rules and target fields according to the field attributes and data sources of the core fields. The data transformation module is used to extract and transform data from core fields according to the field-level mapping chain to obtain target format data, record the source data, transformation rules and transformation results corresponding to each core field, and generate transformation tracking information corresponding to the target format data. The data verification module is used to perform integrity verification, logical consistency verification, and mapping consistency verification on the target format data based on the data constraints of the target system and the correlation between different fuel data in the fuel data association unit, and to obtain the data verification results. The exception handling module is used to determine the abnormal fields and their corresponding source data and conversion rules based on the conversion tracking information when there are anomalies in the data verification results. It then performs data repair and re-conversion on the abnormal fields according to the exception type, and re-verifies the re-converted data to obtain the verified fuel data. The result output module is used to generate a conversion report based on the verified fuel data and the data conversion and verification process, and output the verified fuel data to the target system corresponding to the target security partition.
[0017] Compared with the prior art, the present invention has the following beneficial effects: This invention acquires multi-source fuel data from source security partitions and associates fuel data from different data sources with fuel business identifiers and time information to form fuel data association units. This enables fuel data scattered across different business systems such as coal supply, contracts, and sampling and analysis to be uniformly organized according to actual business relationships. Core fields are determined based on the target system's data standards, and a field-level mapping chain is established between source fields, transformation rules, and target fields. This achieves automatic adaptation and transformation of different data structures, data types, units of measurement, and encoding methods, reducing manual screening, sorting, and format adjustment work. During the data transformation process, this invention records source data, transformation rules, and transformation results, forming transformation tracking information, making the data source and transformation process of the target fields traceable. When anomalies are detected during verification, the source data and transformation rules corresponding to the abnormal fields can be located based on the transformation tracking information, and data repair, re-transformation, and re-verification can be performed for different anomaly types. Through integrity verification, logical consistency verification, and mapping consistency verification, data missing, logical anomalies, and format, unit, or encoding mismatches can be detected and addressed in a timely manner before the data enters the target system. Finally, the system automatically generates a conversion report and outputs verified fuel data, realizing automated closed-loop processing of fuel data association, conversion, verification, repair and feedback. This improves the efficiency, accuracy and traceability of fuel data flow across safety zones, and provides a reliable data foundation for applications such as digital coal yards, unmanned operations and fuel data analysis. Attached Figure Description
[0018] Figure 1 This is a flowchart of Example 1; Figure 2 This is a system diagram of Example 2. Detailed Implementation
[0019] To further understand the content of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.
[0020] Example 1: See Figure 1 An automatic conversion and verification method for fuel data includes the following steps: S1: Obtain multi-source fuel data from the source security partition. Based on the fuel service identifier and time information in the multi-source fuel data, associate the fuel data from different data sources to form a fuel data association unit.
[0021] S2, based on the data standards of the target system in the target safety partition, determines the core fields to be transformed from the fuel data association unit, and establishes a field-level mapping chain between the source fields, transformation rules and target fields based on the field attributes and data sources of the core fields.
[0022] S3 extracts and transforms data from core fields based on the field-level mapping chain to obtain target format data, records the source data, transformation rules and transformation results corresponding to each core field, and generates transformation tracking information corresponding to the target format data.
[0023] S4. Based on the data constraints of the target system and the correlation between different fuel data in the fuel data association unit, perform integrity verification, logical consistency verification and mapping consistency verification on the target format data to obtain the data verification results.
[0024] S5. When there is an anomaly in the data verification result, the abnormal field and its corresponding source data and transformation rules are determined according to the transformation tracking information. Data repair and re-transformation are performed on the abnormal field according to the anomaly type. The re-transformed data is then verified again to obtain the verified fuel data.
[0025] S6 generates a conversion report based on the verified fuel data and the data conversion and verification process, and outputs the verified fuel data to the target system corresponding to the target security partition.
[0026] Example 2: See Figure 2 An automatic fuel data conversion and verification system, comprising: The data association module is used to acquire multi-source fuel data in the source security partition, and associate fuel data from different data sources based on the fuel business identifier and time information in the multi-source fuel data to form a fuel data association unit.
[0027] The mapping construction module is used to determine the core fields to be transformed from the fuel data association unit according to the data standards of the target system in the target safety partition, and to establish a field-level mapping chain between the source fields, transformation rules and target fields based on the field attributes and data sources of the core fields.
[0028] The data transformation module is used to extract and transform data from core fields according to the field-level mapping chain to obtain target format data, record the source data, transformation rules and transformation results corresponding to each core field, and generate transformation tracking information corresponding to the target format data.
[0029] The data verification module is used to perform integrity verification, logical consistency verification, and mapping consistency verification on the target format data based on the data constraints of the target system and the correlation between different fuel data in the fuel data association unit, and to obtain the data verification results.
[0030] The exception handling module is used to determine the abnormal fields, their corresponding source data, and conversion rules based on the conversion tracking information when the data verification result is abnormal. It then performs data repair and re-conversion on the abnormal fields according to the exception type, and re-verifies the re-converted data to obtain the verified fuel data.
[0031] The result output module is used to generate a conversion report based on the verified fuel data and the data conversion and verification process, and output the verified fuel data to the target system corresponding to the target security partition.
[0032] Example 3: This embodiment provides an automatic conversion and verification method for fuel data, addressing issues such as inconsistent data standards between different security zones and fuel business systems, difficulties in uniformly converting multiple data types, and reliance on manual verification and anomaly handling after data conversion. In a specific application scenario, the source security zone can be a security zone storing incoming coal data, contract data, and sampling and analysis data; the target security zone can be a digital coal yard, an unmanned system, or a security zone containing other fuel business systems. The specific method is as follows: The system acquires multi-source fuel data from the source security partition. This multi-source fuel data can include incoming coal data, contract data, and sampling and testing data. Data from different data sources is stored using their respective data structures. The system extracts batch identifiers, data source identifiers, data generation time, coal type, weight, and test results—data relevant to fuel operations—from each data source. Among these, coal type, batch, weight, and test data are the core business content that needs to be focused on during fuel data processing.
[0033] Since different data sources may use different time formats, time precision, or time bases, the first step is to unify the time base of the data generation time in each data source so that data from different systems can be compared on the same time scale.
[0034] For data with a unified batch identifier, data with the same batch identifier are considered as candidate associated data, and it is further determined whether the corresponding data generation time meets the preset time association conditions. The preset time association conditions can be set as preset time windows based on the chronological relationship of different data generation in the fuel business. When candidate associated data is within the corresponding time window, it is grouped into the same fuel data association unit.
[0035] For data from different data sources that lack a unified batch identifier or have a missing batch identifier, the coal type information, weight information, and the unified data generation time are extracted, and a composite association identifier is generated according to a preset combination rule. Data with matching composite association identifiers are grouped into the same fuel data association unit. This allows for the establishment of data-level relationships between different fuel business systems, enabling data from different sources but corresponding to the same fuel business object to form a unified data processing object.
[0036] After forming the fuel data association unit, the target field template corresponding to the target system in the target security partition is obtained. The target field template is used to describe the data structure that the target system can receive, which includes at least the target field name, and configures the data type, unit of measurement, encoding rules and required attributes of each target field according to the data standard of the target system.
[0037] The source fields in the fuel data association unit are matched with the target field templates. During matching, the target fields corresponding to the source fields are determined based on the field name, data type, business meaning, and data source. For source fields that can establish a correspondence, the core fields to be transformed are determined based on the required attributes of the target fields and the business relationships between different fuel data, and a corresponding target field identifier is configured for each core field.
[0038] Therefore, before the data enters the transformation stage, only the data content required by the target system is retained, instead of processing all the data in the source system indiscriminately, which can reduce the amount of data for subsequent data transformation and verification.
[0039] After identifying the core fields, establish a field-level mapping chain based on the data source and field attributes of the core fields.
[0040] For structured tabular data, obtain the field name, data type, unit of measurement, and encoding method corresponding to the core fields, and determine the data type conversion rules, unit of measurement conversion rules, and encoding conversion rules according to the data standards of the corresponding target fields.
[0041] For example, when the data types of the source field and the target field are different, the format is unified through data type conversion rules; when the source field and the target field use different units of measurement, the numerical conversion is completed through unit of measurement conversion rules; when two systems use different encoding methods for the same business status, coal type, or other business content, the source code is converted into a code that the target system can recognize through encoding conversion rules.
[0042] A field-level mapping chain is established according to the data processing order between the source field, data type conversion rules, unit of measurement conversion rules, encoding conversion rules, and target field. Each field-level mapping chain is configured with a unique mapping chain identifier and corresponding rule version information. The rule version information is used to distinguish different conversion rules used at different stages between the same source field and target field.
[0043] For unstructured log data, since there is no fixed table structure directly corresponding to the target fields, the first step is to select the appropriate log parsing template based on the log data source. Log data from different sources can be configured with different log parsing templates to identify key field identifiers, field values, and data generation times in the log text.
[0044] Based on the pre-configured correspondence between the key field identifier and the target field identifier, the field values identified in the log are mapped to the corresponding target fields, and then the corresponding conversion rules are determined according to the data type, unit of measurement and encoding method of the field value itself.
[0045] Therefore, regardless of whether the input data is structured tabular data or unstructured log data, it can ultimately be converted into a unified field-level mapping relationship. By establishing corresponding data mapping mechanisms for different data types, the problem of manually adjusting data formats for different data sources can be reduced. The original solution also explicitly proposed to perform unified format conversion of structured tabular and unstructured log data through mapping relationships.
[0046] After the field-level mapping chain is established, data extraction and transformation are performed on the corresponding core fields in sequence according to the mapping chain.
[0047] For each core field, the corresponding source field value is first read. Then, data type conversion, unit of measurement conversion, and encoding conversion are performed sequentially according to the processing order specified in the field-level mapping chain. The converted data is then written to the corresponding target field to form the target field value. The target field values belonging to the same fuel data association unit are combined to form the corresponding target format data.
[0048] Transformation tracking information is generated synchronously during the data transformation process. Specifically, for each target field value, its source data identifier, source field identifier, field-level mapping chain identifier, rule version information, intermediate transformation results, and final transformation result are recorded. All field transformation records belonging to the same fuel data association unit are configured with the same transformation tracking identifier.
[0049] Therefore, based on the transformation tracking identifier of any target field value, it is possible to query which source data the field value originated from, which field-level mapping chain was used, and which transformation processes it went through, providing a data foundation for the reverse location of subsequent abnormal data.
[0050] After data conversion, the target format data is validated. In this embodiment, data validation includes integrity verification, logical consistency verification, and mapping consistency verification.
[0051] Integrity verification is used to determine whether the data required by the target system is complete. Based on the required attributes configured in the target field template, the fields to be verified are determined, and the existence of the corresponding target field values is checked item by item. When a required field is missing or its corresponding value is empty, the corresponding field is identified as having a data missing anomaly.
[0052] Logical consistency verification is based on the field dependencies between different target fields within the fuel data association unit. These field dependencies are pre-established according to the association rules between fuel business data, describing the data association conditions that different fields must meet. Logical constraints are judged on the interrelated target field values within the same fuel data association unit. When corresponding field values do not conform to the preset field dependencies, the relevant fields are identified as logical consistency anomalies.
[0053] Mapping consistency verification is used to determine whether the transformed data meets the data standards of the target system. It compares the data type, unit of measurement, and encoding method of the target field values with the data type constraints, unit of measurement constraints, and encoding rules specified in the target field template. If any item does not match, the corresponding data type, unit of measurement, or encoding anomaly is determined based on the mismatch.
[0054] Based on the above three types of verification results, data verification results are generated, and abnormal fields and their corresponding exception types are recorded. By performing the above verifications before the data enters the target system, data anomalies can be detected during the transformation process, rather than being manually corrected after the target system uses the data.
[0055] If the data verification result shows no abnormalities, the corresponding fuel data will be used as the verified fuel data.
[0056] When the data validation result is abnormal, the conversion tracking information of the field is read according to the conversion tracking identifier corresponding to the abnormal field. The source field, field-level mapping chain and specific conversion rule that generated the target field value are determined from the conversion tracking information. Then, the corresponding exception handling is performed according to the exception type.
[0057] For data missing anomalies, re-access the source data corresponding to the anomaly field and re-extract the corresponding source field. If the source field value is retrieved again, perform data transformation again according to the corresponding field-level mapping chain.
[0058] For data type anomalies, unit of measurement anomalies, or encoding anomalies, identify the specific conversion step where the anomaly occurred, reselect or adjust the corresponding data type conversion rules, unit of measurement conversion rules, or encoding conversion rules, and then re-execute the data conversion starting from the point of anomaly.
[0059] For logical consistency anomalies, other fields that are related to the abnormal field are queried based on the pre-established field dependency relationships, and the abnormal field and the related fields affected by the abnormal field are jointly identified as fields to be reprocessed.
[0060] During anomaly handling, it's not necessary to reprocess the entire fuel data association unit. Instead, a partial re-transformation is performed on the anomalous field and its associated fields. After the partial re-transformation, the entire verification process is not re-executed; only incremental verification is performed on the anomalous field and its associated fields that have changed. This approach reduces the need for repeated transformations and verifications of data that has already passed validation when a single field becomes anomaly.
[0061] Furthermore, the usage of field-level mapping chains is continuously recorded. Within a preset statistical period, the types and frequency of anomalies generated by each field-level mapping chain, as well as the results of re-verification after data repair, are statistically analyzed.
[0062] The current rule status of a field-level mapping chain is determined based on the number of exceptions corresponding to the same field-level mapping chain and the results of the re-verification after repair. When the number of exceptions generated by a certain transformation rule reaches the preset exception condition, it indicates that the transformation rule can no longer stably adapt to the data standards of the current source data or target system. At this time, the usage priority of the current transformation rule is reduced.
[0063] The system pre-configures one or more candidate transformation rules for the corresponding source and target fields. When the current transformation rule meets the abnormal adjustment conditions, an alternative transformation rule is selected from the candidate rules, and the data transformation and validation of the corresponding field are re-executed using the alternative transformation rule. When the data passes validation after using the alternative transformation rule, the alternative transformation rule is used as the rule adopted in the updated field-level mapping chain, and the corresponding rule version information is updated simultaneously.
[0064] In this way, the abnormal information generated during the verification process can be further fed back to the front-end field mapping and transformation process, so that the field mapping relationship can not only be used to perform data transformation, but also be adjusted according to the actual transformation and verification results.
[0065] After data processing is complete, a conversion report is generated based on the verified fuel data and the entire data conversion and verification process. The conversion report can be organized according to fuel data association units, which records the data source, core field filtering, data conversion status, data verification status, and handling of abnormal data.
[0066] For data that passes verification, output data is generated according to the data interface format required by the target system and output to the target system corresponding to the target security partition; for data that still fails verification after anomaly repair, the corresponding anomaly information and conversion tracking information are retained for subsequent anomaly verification.
[0067] Simultaneously, data filtering results, field transformation results, validation results, and anomaly handling results are recorded in the transformation report, enabling the entire fuel data processing process to be queried and traced based on the transformation report and transformation tracking information. This forms a complete data processing process from data filtering, format conversion, validation, anomaly repair to result feedback. After this process is integrated with the fuel production data base, intelligent data entry robot, and query analysis module, it can further provide a data foundation for digital coal yards, unmanned operations, and related fuel data applications.
[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A method for automatic conversion and verification of fuel data, characterized in that, Includes the following steps: Obtain multi-source fuel data from the source security partition, and associate fuel data from different data sources based on fuel service identifiers and time information in the multi-source fuel data to form fuel data association units; Based on the data standards of the target system in the target safety partition, the core fields to be transformed are determined from the fuel data association unit. Based on the field attributes and data sources of the core fields, a field-level mapping chain is established between the source fields, transformation rules and target fields. Data is extracted and transformed from the core fields according to the field-level mapping chain to obtain the target format data. The source data, transformation rules and transformation results corresponding to each core field are recorded, and transformation tracking information corresponding to the target format data is generated. Based on the data constraints of the target system and the correlation between different fuel data in the fuel data association unit, the target format data is subjected to integrity verification, logical consistency verification and mapping consistency verification to obtain the data verification results. When the data verification result is abnormal, the abnormal field and its corresponding source data and transformation rules are determined according to the transformation tracking information. Data repair and re-transformation are performed on the abnormal field according to the abnormality type. The re-transformed data is then verified again to obtain the verified fuel data. Based on the verified fuel data and the data conversion and verification process, a conversion report is generated, and the verified fuel data is output to the target system corresponding to the target security partition.
2. The method for automatic conversion and verification of fuel data according to claim 1, characterized in that, The specific method for acquiring multi-source fuel data from the source security partition and associating fuel data from different data sources based on fuel service identifiers and time information in the multi-source fuel data to form fuel data association units is as follows: Batch identifiers, data source identifiers, and data generation times are extracted from coal arrival data, contract data, and sampling and chemical analysis data, respectively, and the time base for data generation times from different data sources is unified. Fuel data with the same batch identifier and whose data generation time meets the preset time association conditions are identified as the same fuel data association unit. For fuel data lacking a unified batch identifier, a composite association identifier is generated based on coal type information, weight information, and data generation time. The corresponding fuel data association unit is then determined based on the composite association identifier.
3. The method for automatic conversion and verification of fuel data according to claim 1, characterized in that, Based on the data standards of the target system in the target safety partition, the specific method for determining the core fields to be transformed from the fuel data association unit is as follows: Obtain the target field template corresponding to the target system; The field attributes of each source field in the fuel data association unit are matched with the target field template to determine the source field with the corresponding target field; Based on the required attributes of the target field and the fuel business relationship, the core field is determined from the source fields that have the corresponding target field, and the corresponding target field identifier is configured for the core field.
4. The method for automatic conversion and verification of fuel data according to claim 1, characterized in that, The specific method for establishing a field-level mapping chain between source fields, transformation rules, and target fields based on the field attributes of the core fields and the data source is as follows: For structured tabular data, obtain the field name, data type, unit of measurement, and encoding method of the source field; Determine the data type conversion rules, unit of measurement conversion rules, and encoding conversion rules between the source field and the target field respectively; Establish a field-level mapping chain according to the execution order of source field, data type conversion rule, unit of measurement conversion rule, encoding conversion rule and target field, and configure the mapping chain identifier and rule version information for the field-level mapping chain.
5. The method for automatic conversion and verification of fuel data according to claim 1, characterized in that, The method for establishing a field-level mapping chain is as follows: Determine the corresponding log parsing template based on the source of the log data, and use the log parsing template to identify key field identifiers, field values, and data generation time in the log data; Based on the preset correspondence between the key field identifier and the target field identifier, the identified field value is associated with the corresponding target field; Determine the data type conversion rules, unit of measurement conversion rules, and encoding conversion rules corresponding to the field values, and establish a field-level mapping chain for unstructured data according to the correspondence between log data, identification fields, conversion rules, and target fields.
6. The method for automatic conversion and verification of fuel data according to claim 1, characterized in that, The specific method for extracting and transforming data from core fields based on the field-level mapping chain to obtain target format data, recording the source data, transformation rules, and transformation results corresponding to each core field, and generating transformation tracking information corresponding to the target format data is as follows: Following the field-level mapping chain, perform data type conversion, unit of measurement conversion, and encoding conversion on the core fields in sequence to obtain the target field value corresponding to the target field. For each target field value, record the corresponding source data identifier, source field identifier, field-level mapping chain identifier, rule version information, intermediate transformation results, and final transformation results; The target field values and their corresponding record information belonging to the same fuel data association unit are associated with the same conversion tracking identifier to form conversion tracking information.
7. The method for automatic conversion and verification of fuel data according to claim 1, characterized in that, Based on the data constraints of the target system and the correlation between different fuel data in the fuel data association unit, the target format data is subjected to integrity verification, logical consistency verification, and mapping consistency verification. The specific method for obtaining the data verification results is as follows: Based on the required attributes in the target field template, check whether the corresponding target field value exists to obtain the integrity verification result; Based on the pre-established field dependencies between different target fields in the fuel data association unit, logical constraint judgments are performed on the values of the interrelated target fields to obtain the logical consistency verification results. The data type, unit of measurement, and encoding method of the target field value are compared with the data type constraints, unit of measurement constraints, and encoding rules of the corresponding target field to obtain the mapping consistency verification result. The abnormal fields and their corresponding abnormal types are determined based on the integrity verification results, logical consistency verification results, and mapping consistency verification results.
8. The method for automatic conversion and verification of fuel data according to claim 1, characterized in that, When data verification results are abnormal, the abnormal fields, their corresponding source data, and transformation rules are determined based on the transformation tracking information. Data repair and re-transformation are performed on the abnormal fields according to the abnormality type. The re-transformed data is then verified again to obtain the verified fuel data. The specific method is as follows: Based on the conversion tracking identifier corresponding to the abnormal field, reverse query the source field, field-level mapping chain, and conversion rule corresponding to the abnormal field; When the exception type is a data missing exception, the source field is re-extracted from the corresponding source data. When the exception type is a data type exception, a unit of measurement exception, or a coding exception, the corresponding data type conversion rule, unit of measurement conversion rule, or coding conversion rule shall be redefined. When the exception type is a logical consistency exception, the associated fields related to the exception field are determined based on the field dependency relationship; Perform partial re-transformation on the abnormal fields and related fields affected by the abnormal fields, and perform incremental verification only on the abnormal fields and related fields after re-transformation to obtain the verified fuel data.
9. The method for automatic conversion and verification of fuel data according to claim 1, characterized in that, Also includes: Record the anomaly type, anomaly count, and data repair results for each field-level mapping chain within a preset statistical period; The rule status corresponding to the field-level mapping chain is determined based on the number of exceptions and the result of the re-verification after repair. When the number of exceptions reaches the preset exception condition, the priority of the current conversion rule is reduced, and an alternative conversion rule is determined from the candidate conversion rules that match the corresponding source field and target field. The corresponding fields are transformed and validated using alternative transformation rules, and the rule version information in the field-level mapping chain is updated based on the validation results.
10. An automatic fuel data conversion and verification system, characterized in that, include: The data association module is used to acquire multi-source fuel data in the source security partition, and associate fuel data from different data sources based on the fuel business identifier and time information in the multi-source fuel data to form a fuel data association unit. The mapping construction module is used to determine the core fields to be transformed from the fuel data association unit according to the data standards of the target system in the target safety partition, and to establish a field-level mapping chain between the source fields, transformation rules and target fields according to the field attributes and data sources of the core fields. The data transformation module is used to extract and transform data from core fields according to the field-level mapping chain to obtain target format data, record the source data, transformation rules and transformation results corresponding to each core field, and generate transformation tracking information corresponding to the target format data. The data verification module is used to perform integrity verification, logical consistency verification, and mapping consistency verification on the target format data based on the data constraints of the target system and the correlation between different fuel data in the fuel data association unit, and to obtain the data verification results. The exception handling module is used to determine the abnormal fields and their corresponding source data and conversion rules based on the conversion tracking information when there are anomalies in the data verification results. It then performs data repair and re-conversion on the abnormal fields according to the exception type, and re-verifies the re-converted data to obtain the verified fuel data. The result output module is used to generate a conversion report based on the verified fuel data and the data conversion and verification process, and output the verified fuel data to the target system corresponding to the target security partition.
