Method for generating standardized protocol data based on a multi-protocol platform of a data middleware platform

Abstract

This invention belongs to the field of information processing technology, specifically relating to a method for generating standardized protocol data based on a multi-protocol platform using a data intermediary platform. The method includes acquiring multi-protocol source data; identifying key fields according to a predefined key field identification rule base; generating a first cryptographic hash fingerprint for the value of each key field; analyzing the structural differences between the source and target protocols and establishing a field mapping relationship; performing protocol conversion according to the field mapping relationship, and monitoring the values ​​of key fields in real time during the conversion process; if an unexpected change is detected in the current value of a key field relative to its value before conversion, the field is immediately locked; and verifying the converted data, including verifying whether it conforms to the structural specifications of the target protocol and recalculating a second cryptographic hash fingerprint of the key field values. This invention ensures the accuracy and security of the standardized protocol data conversion and improves data reliability.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and more specifically to a method for generating standardized protocol data based on a multi-protocol platform using a data intermediary platform. Background Technology

[0002] In the process of enterprise informatization, data exchange between heterogeneous systems is the core foundation for business collaboration, and multi-protocol data conversion is a key technology for realizing this process, widely used in fields with high reliability requirements such as finance, logistics, and healthcare. The essence of this technology is to map and convert one data format (source protocol) into another data format (target protocol) while ensuring absolute consistency in data semantics and accuracy.

[0003] Currently, the cross-protocol data conversion methods commonly used in the industry typically include the following core steps: defining a unified data model, performing format conversion through middleware, implementing data transmission, and performing data verification on the target end. For example, Chinese invention patent CN117743466A discloses a cross-platform database synchronization method, which uses middleware to handle data conversion and synchronization anomalies and performs data verification on the target database to ensure integrity. Existing technologies (such as CN117743466A) mainly rely on performing data verification on the target end after conversion. This is a "post-hoc" strategy. Once critical fields (such as amounts, timestamps, or primary keys in financial transactions) suffer unexpected precision loss, truncation, or semantic distortion during the conversion process, their erroneous state may have already contaminated downstream systems before verification. Even if the verification detects the error at this point, irreversible business impact has already occurred, and rollback costs are high, resulting in the following shortcomings:

[0004] The lack of an absolute guarantee mechanism for data consistency is a significant problem. Existing verification methods mostly focus on structural checks (such as JSON Schema verification) or simple value range checks, failing to provide precise and provable verification of the "semantic consistency" and "precision consistency" of data. For example, converting a monetary field with a value of 100.00 (DECIMAL(5,2)) to 100.0 (DECIMAL(4,1)) is structurally complete and valid in terms of value range, but its business semantics have changed due to the loss of precision. Current technologies lack a benchmark anchor point, like a "digital fingerprint," that allows for absolute comparison to prevent such subtle but fatal changes.

[0005] The conversion process suffers from a "black box" nature and insufficient defense: Existing solutions treat the conversion process as a single, uninterrupted step, lacking real-time tracking and proactive defense capabilities for intermediate states. When a field's value is abnormally modified within the conversion logic, the system cannot intervene or prevent the tampering the moment it occurs, and can only wait for the final verification error, exposing the uncontrollability of its internal processes.

[0006] Lack of end-to-end continuity assurance: Data consistency assurance should not end after initial verification. Existing technologies ignore the risk of accidental or malicious tampering during subsequent transmission, storage, and reprocessing. There is a lack of a tracking and recovery mechanism for continuous consistency auditing based on the initial state. Summary of the Invention

[0007] The purpose of this invention is to provide a method for generating standardized protocol data based on a multi-protocol platform using a data intermediary platform, which can effectively solve the technical problems of lag and unreliability in the consistency verification of data status after conversion.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0009] Methods for generating standardized protocol data based on a multi-protocol platform using a data intermediary platform include:

[0010] Step S1: Obtain multi-protocol source data, identify key fields according to the predefined key field identification rule base, and generate a first cryptographic hash fingerprint for the value of each key field;

[0011] Step S2: Analyze the structural differences between the source protocol and the target protocol, and establish field mapping relationships;

[0012] Step S3: Perform protocol conversion according to the field mapping relationship, and monitor the value of key fields in real time during the conversion process; if an unexpected change is detected in the current value of a key field relative to its value before conversion, the field is immediately locked.

[0013] Step S4: Perform verification on the transformed data, including: verifying whether it conforms to the structural specifications of the target protocol, and recalculating the second cryptographic hash fingerprint of the key field values. The verification is deemed successful if and only if all the second cryptographic hash fingerprints are completely consistent with the first cryptographic hash fingerprints corresponding to the first cryptographic hash fingerprint in step S1.

[0014] Step S5: If the verification fails, a transaction rollback is triggered to restore the data to its state before the transformation and to record the exception information.

[0015] Step S6: If the verification passes, assign a version identifier to the data and continuously monitor key fields in subsequent processes; if the current data based on the version identifier is found to be inconsistent with the data state when the verification passed, trigger the data recovery process.

[0016] In one embodiment of the present invention, the key field identification rule base in step S1 includes:

[0017] The field name regular expression matching rule is used to match fields whose field names contain predefined keywords; and / or the field value regular expression matching rule is used to match fields whose values ​​conform to predefined amounts, timestamps, or unique identifier formats.

[0018] In one embodiment of the present invention, generating the first cryptographic hash fingerprint includes:

[0019] Convert string-type key field values ​​to UTF-8 encoded byte arrays;

[0020] The key field values ​​of numeric types are converted into normalized string representations and then into UTF-8 encoded byte arrays;

[0021] The byte array is processed using the SHA-256 hash algorithm to generate a hash value as the first cryptographic hash fingerprint.

[0022] In one embodiment of the present invention, establishing the field mapping relationship in step S2 includes:

[0023] Calculate the semantic similarity between the source protocol field names and the target protocol field names;

[0024] Query the predefined data type compatibility matrix to determine whether the source field type is compatible with the target field type;

[0025] For compatible field pairs, establish a mapping relationship; for numeric fields, record their precision differences and generate corresponding precision conversion strategies.

[0026] In one embodiment of the present invention, the real-time monitoring of the values ​​of key fields in step S3 includes:

[0027] Record the original values ​​of each key field in the status management table.

[0028] During or after the conversion, the current value of the key field is compared with the value before conversion recorded in the status management table;

[0029] The rule for determining unexpected changes is: the difference between the current value and the value before conversion exceeds the tolerance range allowed by the field type.

[0030] In one embodiment of the present invention, the rollback transaction in step S5 is a database transaction rollback, specifically:

[0031] Before starting the transformation, a database transaction is initiated and a data snapshot is saved or the relevant data rows are locked; if the verification fails, a database rollback operation is performed to restore the data to its state before the transformation using the characteristics of transactions.

[0032] In one embodiment of the present invention, the continuous monitoring of key fields in step S6 includes:

[0033] Periodically or based on event triggers, recalculate the current hash fingerprint of key fields;

[0034] Compare the current hash fingerprint with the hash fingerprint corresponding to the version identifier stored when the verification is successful;

[0035] If there is a discrepancy, it is determined to be an abnormal change, triggering the data recovery process.

[0036] In one embodiment of the present invention, the method is implemented using a processing system with the following structure, the processing system comprising:

[0037] The preprocessing module is used to acquire multi-protocol source data, identify key fields according to a predefined key field identification rule base, and generate a first cryptographic hash fingerprint for the value of each key field;

[0038] The mapping analysis module is used to analyze the structural differences between the source and target protocols and establish field mapping relationships.

[0039] The conversion and monitoring module is used to perform protocol conversion according to the field mapping relationship and monitor the value of key fields in real time during the conversion process; if an unexpected change is detected in the current value of a key field relative to its value before conversion, the field is immediately locked.

[0040] Locking fields is achieved by setting software flags or using mutexes to prevent subsequent conversion steps from modifying the field value. In a database environment, locking may involve SELECT FOR UPDATE statements to lock data rows.

[0041] The consistency verification module is used to perform verification on the transformed data, including: verifying whether it conforms to the structural specifications of the target protocol, and recalculating the second cryptographic hash fingerprint of the key field values. The verification is deemed to be successful if and only if all the second cryptographic hash fingerprints are completely consistent with the first cryptographic hash fingerprints corresponding to step S1.

[0042] The exception handling module is configured to trigger a transaction rollback if the verification fails, restoring the data to its state before the transformation and recording the exception information.

[0043] The persistent monitoring module is configured to assign a version identifier to the data if the verification passes, and continuously monitor key fields in subsequent processes; if the current data based on the version identifier is found to be inconsistent with the data state when the verification passed, the data recovery process is triggered.

[0044] The consistency verification module includes:

[0045] The structure verification unit is configured to verify the structural integrity of the transformed data using JSONSchema or XMLSchema.

[0046] The hash comparison unit is configured to recalculate the hash fingerprint and perform an absolute comparison with the fingerprint before the transformation.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] This invention achieves the following significant benefits by constructing a comprehensive and provable data consistency assurance system that encompasses pre-anchoring, in-process intervention, post-verification, and continuous monitoring:

[0049] First, this invention fundamentally solves the problems of lag and unreliability in verifying the consistency of data state after conversion. By generating cryptographic hash fingerprints for key fields before conversion, this invention establishes an immutable verification benchmark. After conversion, by recalculating the fingerprints and performing an absolute comparison, any semantic or precision deviations can be detected with zero error, achieving absolute guarantee of data consistency and completely avoiding business risks caused by precision loss or semantic distortion.

[0050] Secondly, this invention breaks through the limitations of the traditional "black box" conversion mode, realizing transparent control and proactive defense of the conversion process. By monitoring the field conversion status in real time and comparing it with historical values, this invention can immediately trigger a field locking mechanism the moment an unexpected change is detected, changing passive discovery to proactive interception, thereby curbing errors at the beginning and avoiding the spread of erroneous data and subsequent high rollback costs.

[0051] Third, this invention provides an end-to-end reliable data recovery mechanism. When verification fails, the rollback mechanism based on database transactions can reliably and automatically restore the data to its exact state before the transformation by utilizing the atomicity, consistency, isolation, and durability (ACID) characteristics of transactions, ensuring the eventual consistency of the system and greatly improving the robustness and reliability of the system.

[0052] Fourth, this invention extends data consistency assurance from a single verification to the entire data lifecycle. By establishing version identifiers for verified data and continuously monitoring their status, this invention constructs a sustainable data consistency closed loop. Even if the data undergoes abnormal changes in subsequent processes, the system can automatically track and trigger a recovery process from the trusted version, providing unprecedented continuous protection for data integrity throughout the entire processing chain.

[0053] This invention is the first to deeply integrate cryptographic verification, real-time status monitoring, transactional control, and continuous auditing, fundamentally revolutionizing the quality assurance paradigm for multi-protocol data conversion and providing a reliable technical foundation for fields such as finance and healthcare that have stringent requirements for data consistency. Attached Figure Description

[0054] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0055] Figure 1 This is the overall flowchart of the present invention.

[0056] Figure 2 This is a flowchart of the protocol mapping and conversion process of the present invention. Detailed Implementation

[0057] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0058] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0059] Example 1:

[0060] See Figure 1 This embodiment discloses a method for generating standardized protocol data based on a multi-protocol platform using a data intermediary platform, including:

[0061] Step S1: Obtain multi-protocol source data, identify key fields according to the predefined key field identification rule base, and generate a first cryptographic hash fingerprint for the value of each key field;

[0062] Step S2: Analyze the structural differences between the source protocol and the target protocol, and establish field mapping relationships;

[0063] Step S3: Perform protocol conversion according to the field mapping relationship, and monitor the value of key fields in real time during the conversion process; if an unexpected change is detected in the current value of a key field relative to its value before conversion, the field is immediately locked.

[0064] Step S4: Perform verification on the transformed data, including: verifying whether it conforms to the structural specifications of the target protocol, and recalculating the second cryptographic hash fingerprint of the key field values. The verification is deemed successful if and only if all the second cryptographic hash fingerprints are completely consistent with the first cryptographic hash fingerprints corresponding to the first cryptographic hash fingerprint in step S1.

[0065] Step S5: If the verification fails, a transaction rollback is triggered to restore the data to its state before the transformation and to record the exception information.

[0066] Step S6: If the verification passes, assign a version identifier to the data and continuously monitor key fields in subsequent processes; if the current data based on the version identifier is found to be inconsistent with the data state when the verification passed, trigger the data recovery process.

[0067] In one embodiment of the present invention, the key field identification rule base in step S1 includes:

[0068] The field name regular expression matching rule is used to match fields whose field names contain predefined keywords; and / or the field value regular expression matching rule is used to match fields whose values ​​conform to predefined amounts, timestamps, or unique identifier formats.

[0069] In one embodiment of the present invention, generating the first cryptographic hash fingerprint includes:

[0070] Convert string-type key field values ​​to UTF-8 encoded byte arrays;

[0071] The key field values ​​of numeric types are converted into normalized string representations and then into UTF-8 encoded byte arrays;

[0072] The byte array is processed using the SHA-256 hash algorithm to generate a hash value as the first cryptographic hash fingerprint.

[0073] In one embodiment of the present invention, establishing the field mapping relationship in step S2 includes:

[0074] Calculate the semantic similarity between the source protocol field names and the target protocol field names;

[0075] Query the predefined data type compatibility matrix to determine whether the source field type is compatible with the target field type;

[0076] For compatible field pairs, establish a mapping relationship; for numeric fields, record their precision differences and generate corresponding precision conversion strategies.

[0077] In one embodiment of the present invention, the real-time monitoring of the values ​​of key fields in step S3 includes:

[0078] Record the original values ​​of each key field in the status management table.

[0079] During or after the conversion, the current value of the key field is compared with the value before conversion recorded in the status management table;

[0080] The rule for determining unexpected changes is: the difference between the current value and the value before conversion exceeds the tolerance range allowed by the field type.

[0081] In one embodiment of the present invention, the rollback transaction in step S5 is a database transaction rollback, specifically:

[0082] Before starting the transformation, a database transaction is initiated and a data snapshot is saved or the relevant data rows are locked; if the verification fails, a database rollback operation is performed to restore the data to its state before the transformation using the characteristics of transactions.

[0083] In one embodiment of the present invention, the continuous monitoring of key fields in step S6 includes:

[0084] Periodically or based on event triggers, recalculate the current hash fingerprint of key fields;

[0085] Compare the current hash fingerprint with the hash fingerprint corresponding to the version identifier stored when the verification is successful;

[0086] If there is a discrepancy, it is determined to be an abnormal change, triggering the data recovery process.

[0087] In one embodiment of the present invention, the method is implemented using a processing system with the following structure, the processing system comprising:

[0088] The preprocessing module is used to acquire multi-protocol source data, identify key fields according to a predefined key field identification rule base, and generate a first cryptographic hash fingerprint for the value of each key field;

[0089] The mapping analysis module is used to analyze the structural differences between the source and target protocols and establish field mapping relationships.

[0090] The conversion and monitoring module is used to perform protocol conversion according to the field mapping relationship and monitor the value of key fields in real time during the conversion process; if an unexpected change is detected in the current value of a key field relative to its value before conversion, the field is immediately locked.

[0091] The consistency verification module is used to perform verification on the transformed data, including: verifying whether it conforms to the structural specifications of the target protocol, and recalculating the second cryptographic hash fingerprint of the key field values. The verification is deemed to be successful if and only if all the second cryptographic hash fingerprints are completely consistent with the first cryptographic hash fingerprints corresponding to step S1.

[0092] The exception handling module is configured to trigger a transaction rollback if the verification fails, restoring the data to its state before the transformation and recording the exception information.

[0093] The persistent monitoring module is configured to assign a version identifier to the data if the verification passes, and continuously monitor key fields in subsequent processes; if the current data based on the version identifier is found to be inconsistent with the data state when the verification passed, the data recovery process is triggered.

[0094] The consistency verification module includes:

[0095] The structure verification unit is configured to verify the structural integrity of the transformed data using JSONSchema or XMLSchema.

[0096] The hash comparison unit is configured to recalculate the hash fingerprint and perform an absolute comparison with the fingerprint before the transformation.

[0097] To facilitate a better understanding of the present invention by those skilled in the art, the present invention will be further described below in conjunction with specific application scenarios.

[0098] Financial transaction data protocol conversion:

[0099] This embodiment uses a common scenario in the financial field—converting internal JSON transaction data into the international standard ISO20022 XML format—as an example to detail the implementation process of the present invention. The executing entity in this embodiment is a multi-protocol data processing platform deployed within a data intermediary platform.

[0100] The international standard information and application specifications for the target protocol "ISO 20022 XML format" are as follows:

[0101] The "ISO 20022 XML format" used for standardized output in this embodiment fully complies with the following standards published by the International Organization for Standardization (ISO):

[0102] Standard number: ISO 20022:2019 (core series standard, including multiple sub-standards)

[0103] Standard Title: Financial Services — Universal Financial Industry Messagescheme

[0104] Core application sub-standards:

[0105] ISO 20022-1:2019: Define protocol frameworks and business modeling methodologies, and clarify field semantic rules (such as the business attribute definitions of "amount" and "timestamp").

[0106] ISO 20022-4:2013: Specifies the design specifications for XML syntax, including element naming rules, hierarchical structure, and data type constraints (such as the precision format of numeric fields).

[0107] ISO 20022-5:2013: Provides migration mapping rules for legacy protocols (such as SWIFT MT messages) to the ISO 20022 XML format to ensure cross-system compatibility.

[0108] S101: Key field identification and fingerprint generation. The preprocessing module obtains a JSON-formatted transaction data sourced from the payment system through a Kafka message queue. Its original content is as follows:

[0109] {

[0110] "transactionId":"TX20240712100001",

[0111] "paymentAmount": 12500.50,

[0112] "currency":"CNY",

[0113] "timestamp":"2024-07-12T10:00:00.000+08:00",

[0114] "payerId":"U123456789"}

[0115] After acquiring the data, the preprocessing module performs the following sub-steps:

[0116] Field parsing and location: The integrated streaming parser (built on ANTLR) is invoked to decompose the above data stream into a structured collection of field name-value pairs based on the preloaded JSONSchema, and a field location table is generated to record the name, value and logical position of each field in the data stream.

[0117] Key field identification: The preprocessing module calls its internally pre-built key field identification rule base for matching. The rule base contains two types of rules:

[0118] Field naming rules:

[0119] Field name regular expression matching rules: Based on regular expressions that are case-insensitive and allow keywords to appear anywhere in the field name. Example:

[0120] / (.*)([Aa]mount|[Pp]rice|[Ii]d|[Tt]ime|[Dd]ate|timestamp)(.*) / , where:

[0121] (.*) indicates that any character sequence before and after the keyword in the field name (length can be 0);

[0122] [Aa]mount matches a substring containing "Amount" or "amount", and [Pp]rice matches "[Ii]d" similarly. Timestamp is case-insensitive because it is a common lowercase representation in the industry.

[0123] Example of matching logic: The field "transactionId" contains the substring "Id", triggering a [Ii]d match; the field "paymentAmount" contains the substring "Amount", triggering a [Aa]mount match; the field "timestamp" contains the substring "time", triggering a [Tt]ime match. All of the above fields are identified as candidate key fields.

[0124] Optimized description of field value rules:

[0125] Field value regular expression matching rules: Dedicated regular expressions are preset for different data types. After predicting the field value type based on the semantics of the field name, the corresponding rule is selected to validate the format. Specifically, these include:

[0126] The rule for monetary value types is: the regular expression is / ^-?\d+(\.\d{1,2})?$ / .

[0127] in:

[0128] ^ and $ indicate matching the entire value string (avoiding partial matching);

[0129] -? indicates that a negative value is allowed (for expenditure / income scenarios);

[0130] \d+ indicates that the integer part must have at least one digit (no upper limit, suitable for large amounts).

[0131] (\.\d{1,2})? indicates that the decimal part is optional, and if it exists, it will be 1-2 digits (meeting the precision requirements for financial amounts).

[0132] Application example: For the candidate field "paymentAmount", since the field name contains "Amount", it is pre-judged as an amount type. Use this rule to verify its value "12500.50". If the match is successful, it is confirmed that the amount format is correct.

[0133] Timestamp type value rules:

[0134] The regular expression is:

[0135] / ^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}[+-]\d{2}:\d{2}$ / ; Adapts to ISO8601 standard timestamps;

[0136] (e.g., “2024-07-12T10:00:00.000+08:00”), used to validate field values ​​such as “timestamp”.

[0137] Unique identifier type value rules: The regular expression is / ^[A-Za-z0-9]{10,20}$ / , which matches 10-20 alphanumeric combinations (such as "TX20240712100001" and "U123456789"), and is used to verify field values ​​such as "transactionId" and "payerId".

[0138] Fingerprint generation and storage: The preprocessing module then uses the SHA-256 algorithm to generate an irreversible hash fingerprint for each key field value, serving as a comparison benchmark for data consistency. Specifically:

[0139] For string type fields (such as transactionId and timestamp), their values ​​are directly converted to UTF-8 encoded byte arrays.

[0140] For numeric fields (such as paymentAmount), to eliminate any format ambiguity (such as scientific notation), first use the java.math.BigDecimal class to convert it into a normalized string representation (i.e., "12500.50"), and then convert the string into a UTF-8 encoded byte array.

[0141] Subsequently, a SHA-256 hash operation is performed on the above byte array to generate the corresponding fingerprint information:

[0142] Hash("TX20240712100001")="a3f5d8e2...";

[0143] Hash("12500.50")="b7c9f1a4...";

[0144] Hash("2024-07-12T10:00:00.000+08:00")="e8c2a9b1...";

[0145] Finally, the preprocessing module associates these fingerprint information with the corresponding field names, persists them as a transaction unit to the Redis cache database, and sets a key with a time-to-live (TTL) of 5 minutes (Key: pre_conv:TX20240712100001) for subsequent verification.

[0146] S102: Structure Mapping and Transformation Strategy Formulation: The structure mapping module receives source data and its structure information from the preprocessing module. This module is configured to perform the following operations:

[0147] Protocol parsing: Load the schema definition files (XSD) of the source JSON protocol and the target ISO20022 protocol respectively, and parse out the names, data types, constraints (such as precision and scale) and semantic comments of all fields.

[0148] Semantic matching: The module calls a pre-trained Word2Vec word vector model (e.g., using the Gensim library to load the word2vec-google-news-300 model) to calculate the cosine similarity between the source and target field names. For example, calculating the similarity between the source field paymentAmount and the target field InstdAmt / Amt yields a result of 0.82. This value is higher than the preset threshold of 0.75, therefore, the two are considered semantically matched, and a mapping relationship can be established.

[0149] Type compatibility checking and precision strategy formulation: The module queries a predefined data type compatibility matrix. This matrix, indexed by the source and target data types, defines conversion rules. For example, it might find that the source `paymentAmount` (of type `BigDecimal`) is compatible with the target `Amt` (of type `Decimal`), but the source precision is two decimal places, while the target precision requires five decimal places. Based on this, the module automatically generates a precision preservation strategy: in subsequent conversions, the `BigDecimal.setScale(5, java.math.RoundingMode.HALF_UP)` method must be called to round and pad with zeros, i.e., converting 12500.50 to 12500.50000.

[0150] Generate a mapping table: Based on the above results, the structure mapping module outputs a detailed field mapping table. This table not only contains the correspondence between fields, but also the conversion functions (such as precision adjustment functions) and their parameters required for each field.

[0151] S103: Protocol Conversion and Real-time Status Monitoring: The protocol conversion and status management module receives the source data and the mapping table generated in S102, and initiates the conversion process. Simultaneously, the status management submodule within this module starts real-time monitoring.

[0152] State Recording: Before the conversion begins, the state management submodule initializes a state table in memory to record the values ​​of key fields before the conversion (e.g., paymentAmount: 12500.50).

[0153] Conversion Execution and Monitoring: The module performs conversion field by field according to the mapping table. When converting to the `paymentAmount` field, it first calls the precision adjustment function defined in the mapping table to convert its value from 12500.50 to 12500.50000. Subsequently, the status management submodule compares the current converted value with the original value recorded in the status table in real time.

[0154] Anomaly Intervention: Suppose that due to an internal system bug, an error handling logic incorrectly modifies the value, changing it to 1250.05000. The state management submodule, through comparison, finds that the absolute difference of 11250.45 between the current value (1250.05000) and the historical value (12500.50) is far greater than the field's preset tolerance threshold (0.001). The module immediately triggers the data solidification protection mechanism, sending an interrupt signal to the conversion engine, locking any further write operations to the paymentAmount field, marking the field as an anomaly, and generating an anomaly log record. This process achieves proactive intervention at the moment the error occurs, preventing the error from spreading.

[0155] For numerical fields, the tolerance range can be set to absolute error (e.g., 0.001) or relative error (e.g., 0.1%), and configured according to business rules.

[0156] S104: Post-conformity verification

[0157] Assuming no errors occur during the conversion process, the target XML data is successfully generated. The data validation module is activated, performing two levels of validation:

[0158] Structure Validation: The module first calls a standard XML validator and uses ISO20022 XSDSchema to perform structure validity validation on the generated target XML document to ensure that there are no missing fields or type errors.

[0159] Absolute hash comparison: This is the core verification step of this invention. The module extracts key fields from the target XML. <amt>The value is 12500.50000. Importantly, it strictly uses the same normalization method and hash algorithm as in S101 (i.e., first normalize using BigDecimal, then convert to a UTF-8 byte array, and finally perform SHA-256 hash operation) to calculate the converted second cryptographic hash fingerprint.

[0160] The module then retrieves the first cryptographic hash fingerprint corresponding to the current transaction ID stored in S101 by querying the Redis cache.

[0161] The module performs a rigorous byte-by-byte comparison of the two fingerprints. If they match perfectly, the data consistency verification is successful; if any inconsistency is found (e.g., the calculated second fingerprint does not match the stored first fingerprint "b7c9f1a4..."), the verification immediately fails. This comparison based on cryptographic hashing provides an absolute guarantee of data consistency.

[0162] S105: Exception handling and transaction rollback. When the verification in S104 fails, the exception handling and rollback module is triggered:

[0163] Rollback execution: Given that the conversion process in this embodiment is executed within a database transaction (BEGINTRANSACTION has been declared at the beginning of S103), the rollback module directly sends a ROLLBACK command to the database. Based on the transaction log, the database management system automatically undoes all modifications to the current data record within this transaction, atomically and reliably restoring it to its original state before the conversion.

[0164] Anomaly logging: The module simultaneously records the detailed information of this anomaly (including transaction ID, failure time, involved anomaly fields, expected value, actual value, calculated hash fingerprint, etc.) in a structured manner to the Elasticsearch log cluster, and generates a visual anomaly report to notify the operations and maintenance personnel.

[0165] S106: Continuous data status tracking. For data that has passed all verifications, the data tracking and protection module performs the following operations:

[0166] Version Identifier: The module generates a globally unique version control identifier (e.g., a SHA-256 hash value 8f4d9c2a...) for the final data, and associates the identifier with the fingerprints of all key fields of the final data state, storing it in a version management database (such as Git or a dedicated version table).

[0167] Continuous monitoring: The built-in monitoring agent of the module periodically (e.g., every 10 seconds) scans key data tables in the downstream business database. It obtains the current value of the data through SQL queries (such as SELECT Amt FROM Transactions WHERE id='TX20240712100001') and recalculates its hash fingerprint using the same algorithm.

[0168] Anomaly Recovery: If an inspection reveals that the fingerprint of the current data no longer matches the initial fingerprint stored in the version control repository (for example, the amount was mistakenly changed to 13000.00000 due to human error), the monitoring and early warning system will immediately issue an alarm and automatically trigger the data recovery process: that is, retrieve the data record corresponding to version 8f4d9c2a... from the version management system, perform an update operation, and restore the data to the verified correct state. This achieves continuous consistency assurance throughout the entire data lifecycle.

[0169] This invention is not a simple aggregation of existing technological elements, but rather a completely new technological paradigm built upon an analysis of the essence of "data consistency." This invention redesigns the trust base of the data transformation system from its fundamental principles, achieving provable consistency assurance throughout the entire data lifecycle through a series of interconnected technological chains.

[0170] Since the core of data consistency lies not only in format conversion, but also in ensuring the semantic identity of data objects and preventing untraceable mutations at any stage of processing, the system requires an objective and tamper-proof verification benchmark capable of detecting even the slightest differences. Existing solutions rely on structural checks and value range verification after conversion. This is a "post-hoc" and "relative" verification, unable to make an absolute comparison of the precise semantic content of the data (such as the precision of a numerical value). The verification benchmark itself (i.e., the source data) is missing during the conversion process, leading to delayed and unreliable verification.

[0171] This invention constructs a full-link consistency guarantee system based on cryptographic hashes as the trust anchor, breaking the "post-event verification" paradigm and introducing a new framework of "pre-event anchoring, in-event intervention, post-event verification, and continuous auditing":

[0172] Pre-conversion anchoring establishes an absolute benchmark. At the outset of the transformation, key fields (such as amount and ID) are identified from the source data, and cryptographic hash fingerprints are immediately generated for their values. This fingerprint is essentially a unique, definitive, and irreversible mathematical representation of the data's semantic content. As a trust anchor, it provides an objective and absolute benchmark for all subsequent consistency judgments. This step solves the fundamental problem of a missing verification benchmark.

[0173] Traditional conversion processes are black boxes, but this invention white-boxes them by introducing a real-time status monitoring mechanism. The pre-conversion state of key fields is maintained in memory. Immediately after the conversion logic is executed, the current result is compared with the original state. If the difference exceeds the tolerance range allowed for the field type (e.g., a tolerance of 0.001 for a monetary field), a field-level locking mechanism is immediately triggered to interrupt error propagation. This achieves a paradigm shift from passively discovering errors to actively intercepting them, nipping data corruption in the bud.

[0174] Post-verification, after transformation, is no longer a simple structural or range check, but a "DNA-level" comparison of the data's semantic identity. The fingerprint is recalculated using the exact same normalization method (such as BigDecimal's fixed-point representation) and hash algorithm, and then absolutely compared byte-by-byte with the pre-stored anchor fingerprint. Only when the two are completely identical is it proven that the data's semantic identity has not deviated at all. This verification is deterministic and provable, eliminating any business risks caused by loss of precision or semantic distortion.

[0175] This invention extends consistency assurance from a single verification to the entire data lifecycle for the first time. By binding a version identifier to verified data and persistently storing the fingerprint of its final state, the system gains continuous auditing capabilities. By periodically recalculating the fingerprint of online data and comparing it with a benchmark in the version repository, any abnormal changes occurring in subsequent storage, transmission, or processing stages (such as human error or software bugs) can be detected in real time and trigger automatic recovery from the trusted version. This constructs a closed-loop data integrity protection system.

[0176] Example 2:

[0177] This embodiment is a further optimization based on embodiment 1. In this embodiment, the consistency verification module further includes: a zero-knowledge proof unit, configured to generate a zero-knowledge proof for consistency verification based on the first cryptographic hash fingerprint and the second cryptographic hash fingerprint; the zero-knowledge proof is used to prove to a third-party verifier that the transformed data is consistent with the source data in key fields, and does not disclose the specific value of the key fields to the third-party verifier.

[0178] Furthermore, the non-interactive zero-knowledge proof protocol used to generate the zero-knowledge proof is the zk-SNARKs protocol, with BN254 curves selected as the elliptic curve parameter and the MiMC hash algorithm selected as the hash function to optimize the proof generation efficiency.

[0179] In practical applications, such as financial supervision or cross-institutional data collaboration scenarios, it is often necessary to prove the correctness of data transformation to third parties (such as auditors or partner banks). However, the source and target data may involve commercially sensitive information. This embodiment adds a privacy audit function based on the successful hash comparison verification in S104 of Embodiment 1.

[0180] The zero-knowledge proof unit is activated and performs the following operations:

[0181] Public input settings: The hash values ​​(denoted as h1 and h2) of the first hash fingerprint (stored in Redis) and the second hash fingerprint (calculated after transformation) of the key field are written into the zero-knowledge proof circuit as public input.

[0182] Private input setting: The original value of the key field (e.g., 12500.50) and its transformed value (e.g., 12500.50000) are written into the circuit as private input.

[0183] Circuit logic definition: The internal logic of a circuit is defined as follows:

[0184] For the original values ​​in the private input, the same normalization method and SHA-256 algorithm as in S101 are used to calculate their hash values, and it is asserted that the value is equal to the public input h1.

[0185] For the transformed value in the private input, the same normalization method and SHA-256 algorithm as S104 are used to calculate its hash value, and it is asserted that the value is equal to the public input h2.

[0186] We assert that h1 equals h2.

[0187] A zk-SNARKs proof system is built using the libsnark library, based on the BN254 curve and the MiMC hash function. Proof generation takes approximately 3-5 seconds (single-core CPU). The generated proof (Proofπ) is only about 200 bytes and can be sent to a third-party verifier along with the common inputs h1 and h2. Verifiers can complete verification within 10 milliseconds using the same verification key, without knowing any of the original data values ​​throughout the process.

[0188] Testing revealed that in 10,000 audit requests, traditional methods required the transmission of all sensitive data, exceeding 50MB and posing a risk of data leakage. The method described in this embodiment transmits only approximately 2.44MB of proof data, reducing the risk of data exposure to zero and increasing verification speed by more than 100 times. This provides a solution for compliant data sharing and auditing.

[0189] Furthermore, in some preferred embodiments, the rule for determining unexpected changes in step S3 further includes:

[0190] The current value, previous value, and contextual features of the key fields are input into a pre-trained anomaly detection machine learning model;

[0191] When the abnormal probability score output by the model exceeds the dynamic threshold, it is determined that the field has undergone an unexpected change.

[0192] Furthermore, the anomaly detection machine learning model is an Isolation Forest or an Unsupervised Autoencoder model; the model is trained using historical normal conversion data, and the input context features include: protocol type, timestamp, and conversion task ID.

[0193] In complex scenarios such as logistics tracking, the reasonable fluctuation range of cargo status fields (such as temperature: -18.5) may vary with time, season, and carrier, making it difficult to set a fixed tolerance. Therefore, the monitoring link of Example 1S103 is enhanced. In the initial stage of system launch, a fixed tolerance (such as ±0.5°C) is still used for monitoring, while collecting the values ​​before and after conversion of all key fields, and continuously collecting no less than 100,000 normal conversion records.

[0194] Model Training: For the temperature field, an unsupervised autoencoder was trained using the collected normal value sequence. The network's input layer is 5-dimensional, containing: the value before transformation, the value after transformation, the protocol type encoding, the month, and the carrier encoding. The hidden layer consists of [4, 2, 4] neurons, and the output layer is 5-dimensional, attempting to reconstruct the input. The training objective is to minimize the reconstruction error (MSE).

[0195] During online monitoring, the currently monitored 5D features are input into the pre-trained autoencoder to calculate its reconstruction error. The distribution of historical reconstruction errors is recorded, and a dynamic threshold is set to the 99.5th quantile of this distribution (i.e., 99.5% of normal data reconstruction errors are below this value). When the reconstruction error of a certain transformation exceeds this dynamic threshold, field locking is triggered, without relying on fixed tolerance.

[0196] The model is incrementally trained each quarter using new, normal data to adapt to data distribution shifts.

[0197] After applying this method to a cold chain logistics data conversion platform, the anomaly detection rate (Recall) for the temperature field increased from 78% with the original fixed tolerance scheme to 96%, while the false positive rate (False Positive Rate) decreased from 5% to 1.2%. This effectively avoided false alarms caused by overly strict fixed thresholds and missed detections caused by overly lenient fixed thresholds.

[0198] See Figure 2 The specific steps for protocol mapping and conversion are as follows:

[0199] First, the structural differences between the source and target protocols are analyzed. Then, the semantic similarity of field names is calculated. Next, the data type compatibility matrix is ​​queried, and a mapping relationship of compatible fields is established based on this. Then, the protocol conversion is performed. After the conversion is completed, key fields are monitored in real time. At the same time, the original values ​​of key fields are recorded using a status management table. The current value of the key field is then compared with the recorded original value to determine whether the difference between the current value and the original value exceeds the tolerance range. If it is determined to be abnormal, the field is locked.

[0200] This invention achieves deep collaboration and system-level integration, with hash fingerprinting providing an absolute benchmark for real-time monitoring and post-event verification. Real-time monitoring is a prerequisite for transaction rollback, ensuring the timeliness and effectiveness of rollback operations and avoiding the high rollback costs after error propagation. Transaction rollback (based on database ACID properties) is the fundamental guarantee mechanism for state recovery after verification failure. Version identification and continuous monitoring extend the utility of hash fingerprints from the time of conversion to the entire lifecycle of data. It solves the consistency guarantee problem in multi-protocol data conversion. This invention deeply integrates cryptographic integrity verification, real-time process control, database transaction mechanisms, and continuous state auditing, constructing a previously non-existent end-to-end, provable data consistency guarantee system. It provides an unprecedented technological foundation for fields such as finance and healthcare with extremely high data reliability requirements.

[0201] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0202] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.< / amt>

Claims

1. A method for generating standardized protocol data based on a multi-protocol platform using a data intermediary platform, characterized in that, include: Step S1: Obtain multi-protocol source data, identify key fields according to the predefined key field identification rule base, and generate a first cryptographic hash fingerprint for the value of each key field; Step S2: Analyze the structural differences between the source protocol and the target protocol, and establish field mapping relationships; Step S3: Perform protocol conversion according to the field mapping relationship, and monitor the value of key fields in real time during the conversion process; if an unexpected change is detected in the current value of a key field relative to its value before conversion, the field is immediately locked. Step S4: Perform verification on the transformed data, including: verifying whether it conforms to the structural specifications of the target protocol, and recalculating the second cryptographic hash fingerprint of the key field values. The verification is deemed successful if and only if all the second cryptographic hash fingerprints are completely consistent with the first cryptographic hash fingerprints corresponding to the first cryptographic hash fingerprint in step S1. Step S5: If the verification fails, a transaction rollback is triggered to restore the data to its state before the transformation and to record the exception information. Step S6: If the verification passes, assign a version identifier to the data and continuously monitor key fields in subsequent processes; if the current data based on the version identifier is found to be inconsistent with the data state when the verification passed, trigger the data recovery process. The values ​​of the key fields monitored in real time in step S3 include: Record the original values ​​of each key field in the status management table. During or after the conversion, the current value of the key field is compared with the value before conversion recorded in the status management table; The rule for determining unexpected changes is: the difference between the current value and the value before conversion exceeds the tolerance range allowed by the field type.

2. The method for generating standardized protocol data based on a multi-protocol platform using a data intermediary platform according to claim 1, characterized in that, The key field identification rule base in step S1 includes: The field name regular expression matching rule is used to match fields whose field names contain predefined keywords; and / or, the field value regular expression matching rule is used to match fields whose values ​​conform to predefined amounts, timestamps, or unique identifier formats.

3. The method for generating standardized protocol data based on a multi-protocol platform using a data intermediary platform according to claim 1 or 2, characterized in that, The generation of the first cryptographic hash fingerprint includes: Convert string-type key field values ​​to UTF-8 encoded byte arrays; The key field values ​​of numeric types are converted into normalized string representations and then into UTF-8 encoded byte arrays; The byte array is processed using the SHA-256 hash algorithm to generate a hash value as the first cryptographic hash fingerprint.

4. The method for generating standardized protocol data based on a multi-protocol platform using a data intermediary platform according to claim 1, characterized in that, Establishing the field mapping relationship in step S2 includes: Calculate the semantic similarity between the source protocol field names and the target protocol field names; Query the predefined data type compatibility matrix to determine whether the source field type is compatible with the target field type; For compatible field pairs, establish a mapping relationship; for numeric fields, record their precision differences and generate corresponding precision conversion strategies.

5. The method for generating standardized protocol data based on a multi-protocol platform using a data intermediary platform according to claim 1, characterized in that, The rollback transaction in step S5 is a database transaction rollback, specifically: Before starting the conversion, a database transaction is started and a data snapshot is saved or the relevant data rows are locked; when the verification fails, a database rollback operation is performed to restore the data to the state before the conversion using the transaction characteristics.

6. The method for generating standardized protocol data based on a multi-protocol platform using a data intermediary platform according to claim 1, characterized in that, The key fields continuously monitored in step S6 include: Periodically or based on event triggers, recalculate the current hash fingerprint of key fields; Compare the current hash fingerprint with the hash fingerprint corresponding to the version identifier stored when the verification is successful; If there is a discrepancy, it is determined to be an abnormal change, triggering the data recovery process.

7. The method for generating standardized protocol data based on a multi-protocol platform using a data intermediary platform according to claim 1, characterized in that: The method is implemented using a processing system with the following structure, the processing system comprising: The preprocessing module is used to acquire multi-protocol source data, identify key fields according to a predefined key field identification rule base, and generate a first cryptographic hash fingerprint for the value of each key field; The mapping analysis module is used to analyze the structural differences between the source and target protocols and establish field mapping relationships. The conversion and monitoring module is used to perform protocol conversion according to the field mapping relationship and monitor the value of key fields in real time during the conversion process; if an unexpected change is detected in the current value of a key field relative to its value before conversion, the field is immediately locked. The consistency verification module is used to perform verification on the transformed data, including: verifying whether it conforms to the structural specifications of the target protocol, and recalculating the second cryptographic hash fingerprint of the key field values. The verification is deemed to be successful if and only if all the second cryptographic hash fingerprints are completely consistent with the first cryptographic hash fingerprints corresponding to step S1. The exception handling module is used to trigger a transaction rollback if the verification fails, restore the data to the state before the transformation, and record the exception information. The persistent monitoring module is used to assign a version identifier to the data if the verification is successful, and to continuously monitor key fields in subsequent processes; if the current data based on the version identifier is found to be inconsistent with the data state when the verification was successful, the data recovery process is triggered.

8. The method for generating standardized protocol data based on a multi-protocol platform using a data intermediary platform according to claim 7, characterized in that, The consistency verification module includes: The structure verification unit is configured to verify the structural integrity of the transformed data using JSONSchema or XMLSchema. The hash comparison unit is configured to recalculate the hash fingerprint and perform an absolute comparison with the fingerprint before the transformation.

Images