Data cross-system docking method and system based on dynamic metadata analysis

Abstract

The invention relates to the technical field of data processing and heterogeneous system integration, in particular to a data cross-system docking method and system based on dynamic metadata analysis, and the method comprises the steps: obtaining a dynamic data packet of a source system and a target preset metadata structure; calculating a metadata semantic entropy representing uncertainty based on the structural difference characteristics; comparing the entropy value with a preset safety threshold value, and selecting a full-amount dynamic mapping or core field intersection mode to generate a docking instruction; writing the data into a target system, and feeding back and adjusting subsequent semantic entropy calculation parameters based on a writing result; according to the method, the problem of ETL task collapse caused by interface protocol drift between heterogeneous systems is solved, and self-adaptive evolution and self-repair of the system are realized.

Description

Technical Field

[0001] This invention relates to the field of data processing and heterogeneous system integration technology, specifically to a method and system for cross-system data interoperability based on dynamic metadata parsing. Background Technology

[0002] In current heterogeneous system data integration environments, source systems often send dynamic data packets in the form of weakly typed key-value pair sets, and interface protocols frequently experience non-notification-based structural drift, field expansion, or data type changes. To achieve data ingestion, existing solutions generally employ static metadata mapping or fixed Extract, Transform, and Load (ETL) scripts, assuming a strict match between the source data structure and the target storage system's constraints. While this approach offers some processing efficiency when the protocol is stable, it is highly susceptible to triggering integrity constraint violations, deadlocks, or type conversion anomalies in the target database when faced with upstream system issues such as misspelled field names, incompatible data types, or the inclusion of debugging data. This rigid mechanism lacks quantitative assessment and adaptive defense capabilities against data structure uncertainties, leading to frequent ETL task crashes, complete rejection of critical business data from writing, and difficulty in balancing system throughput and security.

[0003] Therefore, how to construct a data integration method that can quantify data structure risks, adaptively switch between full mapping and core field protection strategies according to the degree of risk, and ensure the continuous storage of core data in the database under protocol drift environment has become an urgent technical problem to be solved. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for cross-system data integration based on dynamic metadata parsing, in order to solve the technical problems of ETL task crashes and loss of critical business data caused by interface protocol drift, field expansion, or data type changes in existing heterogeneous system integration. The technical solution of this invention is as follows: A method for cross-system data integration based on dynamic metadata parsing, applied to a data processing system connecting a source system and a target storage system, the method comprising the following steps: The dynamic data packets sent by the source system and the preset metadata structure of the target storage system are obtained, wherein the dynamic data packets are in the form of a weakly typed key-value pair set, and the preset metadata structure defines the standard constraint parameters of the target field. Based on the structural difference characteristics between the key name set in the dynamic data packet and the preset metadata structure, the metadata semantic entropy representing the uncertainty of the data structure is calculated, and the dynamic data packet is prepared for parsing. The semantic entropy of the metadata is compared with a preset security threshold. Based on the comparison result, either the full dynamic mapping mode or the core field intersection mode is selected to generate the corresponding data docking instruction. The data in the dynamic data packet is written into the target storage system through the data docking command, and the subsequent semantic entropy calculation parameters are adjusted based on the writing result feedback.

[0005] Preferably, based on the structural difference characteristics between the key name set in the dynamic data packet and the preset metadata structure, the metadata semantic entropy characterizing the uncertainty of the data structure is calculated, including: Extract the key name and value type of all key-value pairs in the dynamic data packet to construct the current transmission protocol characteristics; Calculate the percentage of unknown key names, the key name character edit distance, and the data type conflict rate between the current transmission protocol features and the preset metadata structure (Schema); Based on preset weighting coefficients, the weighted summation of the percentage of unknown key names, the edit distance of key name characters, and the data type conflict rate is calculated to obtain the semantic entropy of the metadata.

[0006] Preferably, based on the comparison results, either the full dynamic mapping mode or the core field intersection mode is selected, including: In response to the metadata semantic entropy being less than or equal to the preset security threshold, the full dynamic mapping mode is activated. In the full dynamic mapping mode, all key names in the dynamic data packet are directly mapped to column names of the target storage system, and a Structured Query Language (SQL) statement containing all fields is generated as the data docking instruction.

[0007] Preferably, selecting either the full dynamic mapping mode or the core field intersection mode based on the comparison results also includes: In response to the metadata semantic entropy being greater than the preset security threshold, the core field intersection mode is activated; In the core field intersection mode, identify the fields marked as core business in the preset metadata structure, and calculate the intersection of the key name set of the dynamic data packet and the core business fields; Only data belonging to the intersection is extracted to generate a structured query language statement, and non-core fields in the dynamic data packet that are not included in the intersection are discarded to generate the data docking instruction.

[0008] Preferably, the preparation for parsing the dynamic data packet further includes: Monitor whether there are temporary fields or unexpected protocol variation features in the dynamic data packets that conform to the debugging naming rules; Upon detecting the protocol variation feature, a preset numerical penalty term is added to the metadata semantic entropy to increase the value of the metadata semantic entropy; The dynamic data packets are cleaned by removing the temporary fields, and the cleaned data is used as the data to be processed.

[0009] Preferably, writing the data in the dynamic data packet into the target storage system via the data docking instruction includes: Establish a connection channel with the target storage system using a database connection pool; Execute the data docking instructions in batches under the transaction isolation level; If a database deadlock or type conversion exception is triggered during execution, the current transaction is rolled back, the corresponding risk weight coefficient is adjusted to the preset saturation upper limit, and the processing strategy of subsequent data packets is forcibly switched to the core field intersection mode.

[0010] Preferably, the source system includes heterogeneous business systems, and the dynamic data packet is a semi-structured data stream; The preset metadata structure includes the column name definition, data type constraints, and primary key index information of the database table; The metadata semantic entropy is used to quantify the potential pollution risk to the target storage system caused by the structural uncertainty of the dynamic data packet.

[0011] A cross-system data interface system based on dynamic metadata parsing includes the following modules: The data acquisition module is configured to acquire dynamic data packets sent by the source system and a preset metadata structure of the target storage system, wherein the dynamic data packets are in the form of a set of weakly typed key-value pairs; The entropy calculation module is configured to calculate the metadata semantic entropy, which represents the uncertainty of the data structure, based on the structural difference features between the key name set in the dynamic data packet and the preset metadata structure, and to prepare for parsing the dynamic data packet. The strategy decision module is configured to compare the semantic entropy of the metadata with a preset security threshold, and select either the full dynamic mapping mode or the core field intersection mode based on the comparison result to generate the corresponding data docking instruction. The execution feedback module is configured to write the data in the dynamic data packet into the target storage system through the data docking instruction, and adjust the subsequent semantic entropy calculation parameters based on the writing result feedback.

[0012] The beneficial effects of this invention are as follows: 1. This invention quantifies the uncertainty of data structure by calculating the semantic entropy of metadata based on structural difference features, and realizes the selection of the full dynamic mapping mode and the core field intersection mode; when the data structure is stable and the entropy value is low, the full mapping is used to maximize the system throughput; when high-risk protocol drift is detected, the core field protection strategy is automatically switched; this processing strategy effectively solves the problem that the traditional rigid docking mechanism cannot cope with the non-notification changes of heterogeneous systems, and takes into account both efficient transmission and security defense. 2. When a high entropy risk is detected, this invention activates the core field intersection mode. By identifying core business fields and extracting only the intersection data between the core business fields and dynamic data packets, and discarding non-core fields, a loss prevention protection for data docking is constructed. This method avoids the rejection of writing the entire critical business record due to structural errors, spelling drift, or data type conflicts in secondary fields, ensuring the continuous storage of critical business data in abnormal network environments and greatly improving the robustness of the system. 3. This invention calculates the semantic entropy of metadata by the proportion of unknown key names, the key name character edit distance, and the data type conflict rate, and introduces numerical penalty terms and removes temporary fields for unexpected protocol variation features. This deep parsing mechanism can keenly capture fine-grained details such as field spelling errors, data type incompatibility, and unexpected protocol variations, effectively preventing temporary fields or unexpected protocol variation features received by the data processing system from polluting the target storage system and ensuring the accuracy of risk assessment. 4. This invention is configured with a feedback adjustment step based on the write result. When a database deadlock or type conversion anomaly is detected, the current transaction can be automatically rolled back and the subsequent strategy can be forcibly switched. At the same time, the semantic entropy calculation parameters are dynamically adjusted based on the write result feedback. Through this parameter adjustment mechanism, the system can quickly improve its sensitivity to specific risk characteristics after a failure, prevent similar structural errors from causing continuous crashes, and realize the weight adjustment of specific risk characteristics and the forced switching of strategies. Attached Figure Description

[0013] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a structural diagram of the system of the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0015] Example 1: Please see Figure 1A cross-system data connection method based on dynamic metadata parsing is proposed. The method is applied to a data processing system that connects a source system and a target storage system. The method includes the following steps: obtaining dynamic data packets sent by the source system and the preset metadata structure of the target storage system. The dynamic data packets are in the form of a weakly typed key-value pair set, and the preset metadata structure defines the standard constraint parameters of the target field. Based on the structural differences between the key set in the dynamic data packet and the preset metadata structure, the semantic entropy of metadata, which represents the uncertainty of the data structure, is calculated, and the dynamic data packet is prepared for parsing. The semantic entropy of the metadata is compared with the preset security threshold. Based on the comparison result, the full dynamic mapping mode or the core field intersection mode is selected to generate the corresponding data docking instructions. Data from dynamic data packets is written to the target storage system via data docking commands, and subsequent semantic entropy calculation parameters are adjusted based on the write results.

[0016] This embodiment elaborates on the execution logic of the above-mentioned cross-system data connection method, which aims to solve the problem of ETL task crashes caused by the opaque drift of interface protocols between heterogeneous systems; the system executes the data acquisition step and obtains the dynamic data packets sent by the source system through the middleware interface; At the physical level, the dynamic data packet is represented as a set of weakly typed key-value pairs, meaning that the specific field names and data types are not locked during the compilation phase, allowing the upstream system to make non-notification structural changes. At the same time, the system loads a preset metadata structure from the system table of the target storage system. This structure serves as a verification benchmark and strictly defines the standard constraint parameters of the target fields, including column names, type precision, and non-null constraints. The system enters the core computing phase, calculating the semantic entropy of metadata based on the structural differences between the key set in the dynamic data packet and the preset metadata structure. This parameter is a quantitative scalar used to characterize the degree of uncertainty of the current batch data structure relative to the target schema. During this period, the system will perform parsing preparation for the dynamic data packet, including preliminary format cleaning and encoding conversion. The system executes policy decision logic, comparing the calculated metadata semantic entropy with a preset security threshold; the preset security threshold... Instead of fixed empirical values, these values ​​are determined through benchmark testing during the system initialization phase. The system selects a set of sample data packets, such as 1000 packets conforming to a preset metadata structure, as the "golden dataset." This golden dataset consists of benchmark data samples that have passed manual verification or were collected during the system's historical stable operation. The entropy set of these samples is calculated, and the average value is derived. with standard deviation ,set up Thus, based on statistical principles, the boundary between normal fluctuations and structural anomalies is defined; Given that the gold dataset is extremely pure, that is In extreme cases, the calculated threshold may approach zero, causing system overreaction. This embodiment introduces a minimum noise basis constant. The threshold is set to 0.15, a value determined based on 1.5 times the average background noise entropy measured under benchmark conditions. This is intended to filter out floating-point errors in calculations, ensuring the system retains a basic fault tolerance margin. The corrected threshold calculation logic is as follows: in, A preset safety threshold, dimensionless; function Indicates taking and The larger value in the range is used to ensure that the final threshold is not lower than the minimum noise floor; This is the arithmetic mean of the set of entropy values ​​in the gold dataset; The standard deviation of the entropy set is used to characterize the normal fluctuation range; The minimum noise basis constant, with a value of 0.15, is dimensionless. Its physical meaning is an estimate of the background noise entropy of the system under ideal silent conditions, used to prevent noise from spreading when the system is in a state of complete silence. Systemic allergy caused by an excessively low time threshold; It should be understood that a preset safety threshold is... The static baseline is determined during system initialization and does not change with subsequent weighting coefficients. The weighting changes dynamically; the increase in weighting aims to amplify the numerical contribution of specific risk dimensions, making them more likely to break through static limitations. Interception line.

[0017] Specifically, for different combinations of risk characteristics in the weighted summation model, such as high Low Low and Low high High may lead to the same entropy value In the case of, The percentage of unknown key names represents the risk of structural expansion. Edit the distance value for key name characters to represent the risk of name drift; This is a numerical value representing the data type conflict rate, indicating the risk of type incompatibility. These are the weighting coefficients corresponding to the risk characteristics, dimensionless, with an initial preset value of [value missing]. The weighting is determined based on the Analytic Hierarchy Process (AHP), assigning the highest risk priority to data type conflicts. This embodiment adds a risk root cause tracing step to the decision-making logic: while comparing entropy values, the system calculates each risk dimension separately. The contribution value is used to identify the dominant risk characteristic of the current batch, and the dimension with the largest contribution value is marked as the dominant risk characteristic of the current batch. If the entropy value is below the threshold, the system determines that the data structure is stable and selects the full dynamic mapping mode to maximize throughput. If the entropy value is above the threshold, the system determines that there is a high risk of protocol drift, automatically switches to the core field intersection mode, and loads differentiated processing sub-strategies based on the dominant risk characteristic. For example, if the dominant risk characteristic is the data type conflict rate... Then, in the core field intersection pattern, the forced type cleaning logic is activated; if the dominant risk characteristic is the proportion of unknown key names... This activates the rapid filtering logic, thereby enabling targeted defense against different risk scenarios under a unified security mode to ensure the survival of critical data. The system generates corresponding data docking instructions based on the selected mode, which are usually dynamically constructed batch SQL statements, and writes the data to the target storage system through the database connection channel. The system executes a closed-loop feedback mechanism, and adjusts the subsequent semantic entropy calculation parameters based on the write results, such as whether deadlock or type truncation occurs. Specifically, the system maintains a set of dynamic weight parameters for calculating the semantic entropy of metadata, i.e., calculation parameters. When a write failure is detected, the system makes a judgment based on preset exception-feature mapping rules: for example, if the captured exception is a type conversion exception, Structured Query Language (SQL) status code 22003, then the data type conflict rate is locked. To correspond to the risk dimension, the corresponding subscript If the exception is a violation of integrity constraints, then the percentage of unknown key names will be locked. , corresponding subscript Based on this determination, the system sends a feedback signal to the entropy calculation module, increasing the weight coefficient of the corresponding risk dimension. For example, the weight of the data type conflict rate is increased by 0.1; to prevent the infinite accumulation of weights from distorting the calculation model, the system applies a preset parameter saturation limit logic, the calculation formula of which is: in, For the updated number Weighting coefficients for each risk dimension; function Indicates taking and The smaller value in the range is used to ensure that the calculation result does not exceed the preset saturation upper limit; The first step of the current time Initial values ​​for the weighting coefficients of each risk dimension; Increase the step size by the preset weights, and set the value accordingly. This is used to characterize the gain magnitude of risk sensitivity to a single failure; The upper limit of weight saturation is set to . Dimensionless; here Set to much larger The penalty level value is based on the mathematical transformation from probabilistic risk to an absolute penalty factor, allowing... ,symbol This indicates that the sum of the weights is significantly greater than 1.0, ensuring that risk saturation in a single dimension can independently drive the total entropy value to exceed 1.0. ; This parameter setting clarifies the mathematical meaning of the weights in the non-normalized state: that is, transforming them from probability weights to absolute risk penalty factors, allowing... Much larger This ensures that the weight saturation of a single high-risk dimension can independently drive the entropy value. Exceeding the safety threshold; This will Set to much larger The penalty level is based on mathematical robustness considerations: if Only When risk characteristics such as the proportion of unknown key names Minimal, for example ,Right now Only one of the fields When there are unknown fields, the calculated entropy increase is only This value may be much smaller than the preset safety threshold. For example, 0.15, which prevents the system from triggering defenses against fatal errors in a single field, such as an unindexed column that causes a deadlock; By allowing weights to reach The system can calculate This value is greater than the preset safety threshold, which is sufficient to cross the threshold interception line, thereby ensuring that the weight is always within the extended effective control range; It is worth noting that during this adaptive adjustment process, the system allows the sum of the updated weights for each dimension. For values ​​exceeding 1.0, this non-normalization strategy aims to mathematically guarantee the calculated entropy value under the same data characteristics by simply increasing the gain of a specific risk dimension. Monotonically increasing entropy makes it easier to trigger threshold interception; this increases the calculated entropy value in subsequent calculations, making it more likely to exceed the preset security threshold, forcing the system to enter the core field intersection mode; when consecutive successful writes reach a preset batch, for example... When processing batches, the system activates weight decay logic, attempting to gradually reduce the weight coefficients back to their initial values; the specific decay calculation formula is as follows: in, For example, the preset decay step size , The first one set during system initialization The baseline weights for each risk dimension ensure that after the system has passed through the risk period caused by data source fluctuations, it can smoothly return to a normal, low-sensitivity monitoring state, thereby achieving the system's adaptive evolution and self-repair.

[0018] Example 2: Based on the structural differences between the key set in dynamic data packets and the preset metadata structure, the semantic entropy of metadata, which represents the uncertainty of data structure, is calculated. This includes: extracting the key name and value type of all key-value pairs in the dynamic data packets to construct the current transmission protocol features; calculating the proportion of unknown key names, the key name character edit distance, and the data type conflict rate between the current transmission protocol features and the preset metadata structure; and performing a weighted summation of the proportion of unknown key names, the key name character edit distance, and the data type conflict rate based on preset weight coefficients to obtain the semantic entropy of metadata. This embodiment further specifies the metadata semantic entropy calculation step in Embodiment 1. This calculation process is constructed as a multi-dimensional feature weighted model; the system parses the dynamic data packets of the current batch, extracts the key name and value type of all key-value pairs, and executes the deep type probe logic here: For numeric types in JavaScript object notation (JSON), the absolute value of the numeric value modulo 1 is further calculated to see if it is less than a preset floating-point error. If so, it is marked as a preset integer (PROFILED_INT). For string types, the preset date regular expression ^\d{4}-\d{2}-\d{2} (where \d represents numeric characters and {n} represents the number of repetitions, i.e., matching the standard date format of 4 digits - 2 digits - 2 digits) is attempted to match and marked as a preset date (PROFILED_DATE). For null values, they are marked as preset null values ​​(PROFILED_NULL). For strings that do not match a specific rule, they are marked as preset strings (PROFILED_STRING). This constructs the current transport protocol features containing fine-grained type signatures. The system calculates the difference indicators in three dimensions: First, calculate the percentage of unknown key names. This metric is used to measure how many redundant fields undefined in the target database appear in the data packet. Second, calculate the key name character edit distance value. This metric is used to detect whether there are spelling errors or non-standard naming drift in field naming. Third, the data type conflict rate is calculated. This metric is used to detect whether data types with the same name are compatible. Based on preset weighting coefficients, the system performs a weighted summation calculation to obtain the metadata semantic entropy. The specific calculation formula is as follows: in, Metadata semantic entropy, in physical terms, is the degree of disorder of the data structure relative to the target preset structure, and is a scalar quantity. Regarding boundary situations, if Then the system will force To avoid division by zero exceptions, the calculation formula is as follows: in, The number of key names that appear in the dynamic data packet but are not defined in the preset metadata structure; The total number of keys in the dynamic data packet; if Then a mandatory definition is required. ; In the specific calculation, for each key name in the dynamic data packet that does not match exactly... Iterate through all key names in the preset metadata structure. Calculate the edit distance and find the minimum edit distance. and its corresponding target key name To eliminate the influence of string length on the distance value, local normalization is performed, and its calculation formula is as follows: in, This is the normalized single-field edit distance; Source key name Best match target key name Minimum Levenshtein edit distance between them; This is a function for calculating the length of a string. like Then let ; final For all unmatched key names The arithmetic mean; if there are no unmatched key names, then ; The data type conflict rate is a numerical value, and its calculation formula is as follows: in, This refers to the number of fields in the key-name matching field whose actual data type cannot be compatible with the target type through implicit conversion rules; This refers to the total number of fields in a dynamic data packet whose key names can precisely match the preset metadata structure; for To address the boundary cases where the denominator is undefined, this embodiment establishes special handling rules for the covering formula: directly let To characterize extremely high risk, instead of performing division, and skipping subsequent conflict counting steps; Under normal circumstances, This is defined as the number of fields in these matching fields whose actual data type cannot be implicitly converted to the preset target type; The inability to implicitly convert here is determined based on a preloaded type compatibility truth table, which is specifically constructed as a two-dimensional Boolean matrix. row index The fine-grained type signatures identified by the aforementioned depth type probes are enumerated as follows: "PROFILED_STRING,PROFILED_INT,PROFILED_FLOAT,PROFILED_DATE,PROFILED_NULL,PROFILED_BOOL,PROFILED_ARRAY,PROFILED_OBJECT" Column Index The target database's SQL types are enumerated, including VARCHAR, INT, DATE, and BLOB. The matrix element assignment logic follows a strict mode, with the specific rules as follows: like For PROFILED_INT and If the value is a database_integer (DB_INT), it will return True directly because the probe has verified the integer attribute and range of the value. like For PROFILED_STRING and For database variable-length strings (DB_VARCHAR) or database character strings (DB_CHAR), the system automatically checks whether the byte length of the source string is less than or equal to the length constraint defined in the target field. If so, it returns True; otherwise, it returns False. For PROFILED_FLOAT and If the value is DB_INT, return False, indicating a precision loss conflict. For null boundary cases: if If PROFILED_NULL, the system checks the target field. The IsNullable property in the metadata definition returns True if nullability is allowed, otherwise it returns False. Default blocking policy: For all types not explicitly whitelisted by the above rules... The matrix unit returns False by default; this table lookup mechanism based on probe results eliminates the semantic ambiguity risk caused by implicit conversion at the database layer; during the statistical process, the system performs a lookup on each matching field, i.e., the aforementioned... If a field in a collection is retrieved from a table, and the result is False, then the values ​​are incremented. ; This embodiment accurately distinguishes between benign structural expansion and malignant structural damage through multi-dimensional quantitative calculations. In particular, by normalizing the edit distance based on the maximum length, it avoids the bias of long field names naturally having a higher edit distance. At the same time, by assigning a weight coefficient with a high type conflict rate, the system can keenly capture those hidden errors where the key names match but the data types are incompatible, ensuring that the entropy value can truly reflect the potential harm of the data to the target system.

[0019] Example 3: Based on the comparison results, either the full dynamic mapping mode or the core field intersection mode is selected, including: activating the full dynamic mapping mode in response to the metadata semantic entropy being less than or equal to a preset security threshold; in the full dynamic mapping mode, all key names in the dynamic data packet are directly mapped to the column names of the target storage system, and a structured query language statement containing all fields is generated as a data docking instruction.

[0020] This embodiment describes in detail the processing logic under low entropy conditions, namely the execution process of the full dynamic mapping mode; the system compares the calculated metadata semantic entropy with a preset security threshold; in response to the metadata semantic entropy being less than or equal to the preset security threshold, the system determines that the current data source follows the established protocol and has a high signal-to-noise ratio, and then activates the full dynamic mapping mode. In this mode, the system executes a direct mapping strategy, treating all key names in the dynamic data packet as valid column names of the target storage system without additional filtering or pruning. The system iterates through all key-value pairs in the data packet and uses the dynamic SQL function of the Object Relational Mapping (ORM) framework to generate a structured query language statement containing all fields. This statement is sent to the database execution engine as a data docking instruction, typically using batch insertion to optimize network I / O performance. This embodiment maximizes the system's processing efficiency under normal conditions where the data structure is relatively stable. By eliminating complex field validation and cleaning steps and utilizing a full-field mapping strategy, it achieves throughput close to that of native SQL execution, meeting the need for second-level synchronization of massive amounts of data in a trusted network environment.

[0021] Example 4: Based on the comparison results, the system selects either the full dynamic mapping mode or the core field intersection mode. It also includes: activating the core field intersection mode in response to the metadata semantic entropy being greater than a preset security threshold; in the core field intersection mode, identifying the fields marked as core business in the preset metadata structure, calculating the intersection of the key name set of the dynamic data packet and the core business fields; extracting only the data belonging to the intersection to generate a structured query language statement, and discarding non-core fields in the dynamic data packet that are not included in the intersection, in order to generate data docking instructions.

[0022] This embodiment details the defense logic under high entropy conditions, specifically the execution process of the core field intersection mode. In response to the metadata semantic entropy exceeding a preset security threshold, the system determines that the current data source has experienced severe protocol drift or contamination, and then activates the core field intersection mode. At this time, the system reads the dominant risk characteristics identified during the entropy calculation and comparison phase, i.e., the main components causing the increase in entropy, such as high... or high The system reads metadata tags from a preset metadata structure and identifies fields marked as core business data, such as primary key ID and key business timestamps. The system calculates the intersection of the key set of dynamic data packets with the aforementioned core business fields to determine the still usable subset of valid data in the current data packet. Based on this, the system performs cleaning and generation steps, extracting only the data belonging to this intersection to generate structured query language statements. During this process, the system performs adaptive processing on the extracted data based on the dominant risk characteristic: if the dominant risk characteristic is data type conflict rate... The system will enable a strong type compatibility adapter; This adapter incorporates a regular expression-based maximum likelihood type correction algorithm: for fields whose target is numeric but whose value is a dirty string, it uses regular expressions to extract the first matching numeric substring as the valid value; for fields whose target is date but whose format is unknown, it attempts to parse them according to international standards ISO8601, RFC3339, and the common yyyyMMdd format in turn. If all attempts fail, the field value is forcibly set to NULL or the default value defined by the metadata to prevent the database from throwing a type conversion exception. If the dominant risk characteristic is the proportion of unknown key names This means that there are a lot of junk fields in the data packet, so the system activates the whitelist reverse extraction logic in the structure pruning strategy: the system no longer traverses the key name set of the dynamic data packet, but directly traverses the core field list in the preset metadata structure. ,by The time complexity involves searching for the corresponding value in the dynamic data packet; if it exists, it is extracted; otherwise, it is ignored. This reverse extraction method avoids dealing with massive amounts of unknown fields, i.e. The traversal and memory allocation reduce the processing time complexity from Reduce to This significantly optimizes processing performance in scenarios with rampant junk data. At the same time, the system physically discards all non-core fields in dynamic data packets that are not included in the intersection, including unknown or mutated fields that cause entropy to increase. The generated instructions contain only the most basic business data, ensuring that they can pass the database's strong validation constraints. This embodiment constructs a data docking loss mitigation mechanism; when an uncontrollable protocol change occurs in the upstream system, this mode forcibly ensures the continuous storage of core business data by sacrificing the integrity of non-core data; this avoids the entire critical business record being rejected for writing due to a structural error in a minor field, greatly improving the system's survivability in abnormal environments.

[0023] Example 5: The preparation for parsing dynamic data packets also includes: monitoring whether there are temporary fields or unexpected protocol variation features in the dynamic data packets that conform to the debugging naming rules; if protocol variation features are detected, adding a preset numerical penalty to the metadata semantic entropy to increase the value of the metadata semantic entropy; cleaning the dynamic data packets, removing temporary fields, and using the cleaned data as data to be processed.

[0024] This embodiment is a correction and optimization of the entropy calculation process in embodiment 2, which aims to deal with the pollution of development and debugging data. In the parsing preparation stage, the system starts a feature monitoring program to scan whether the key name of the dynamic data packet contains a specific prefix or suffix in order to determine whether there are temporary fields that conform to the debugging naming rules, or unexpected protocol variation features such as all uppercase or all lowercase. In practice, the system has a built-in regular expression rule library. For example, it matches key names that begin with "test," "temp," or "debug," or unofficial version fields with suffixes like "bak" or "v2." Simultaneously, to detect protocol variation characteristics, the system detects sudden changes in key name naming styles, such as a sudden shift from the default underscore naming convention to camelCase, which usually indicates code infiltration from different development teams. In response to detecting these protocol variation characteristics, the system executes a penalty mechanism, adding a preset numerical penalty to the basic metadata semantic entropy. The corrected entropy calculation formula is as follows: in, To correct the semantic entropy of metadata after introducing a penalty term; The semantic entropy of the basic metadata calculated in Example 2; This is the protocol variation penalty factor, with a value of 10.0, dimensionless. This value is set based on historical fault backtracking tests to ensure... To achieve absolute interception; This is a mutation feature indicator function that is used when mutation features in debugging naming rules or protocols are detected. ,otherwise ; The determination of this value is based on backtesting of historical fault data: After analyzing 1,000 past cases of system crashes caused by protocol mutations, it was found that the normal entropy value fluctuation range is usually between [0, 1.5]. In order to ensure that when a clear mutation feature such as the debug_ prefix is ​​detected, the system can switch to the core field intersection mode 100% regardless of how other features behave, an overwhelming value that can cover all normal fluctuations must be introduced. After experimental verification, Setting it to 10.0 can achieve absolute interception of contaminated data while retaining sufficient safety margin, while setting it to a lower value such as 2.0 may produce misjudgments in benign variations with low entropy values; This is an indicator function that takes a value of 1 when a mutation feature is detected, and 0 otherwise. The system performs physical cleaning on dynamic data packets, removing all key-value pairs identified as temporary fields, and passes the cleaned data as data to be processed to the subsequent pattern selection module. This embodiment introduces a penalty mechanism, enabling the system to be acutely aware of informal releases or hot update tests that may be taking place in upstream systems. By rapidly increasing the entropy value, the system can forcibly trigger the intersection mode of core fields, thereby preventing junk fields used for testing from polluting the schema definition of the production environment database, thus acting as a firewall.

[0025] Example 6: The data in the dynamic data packet is written to the target storage system through data docking instructions, including: establishing a connection channel with the target storage system using a database connection pool; executing data docking instructions in batches under the transaction isolation level; if a database lock-up or type conversion exception is triggered during execution, the current transaction is rolled back, the corresponding risk weight coefficient is adjusted to the preset saturation upper limit value, and the processing strategy of subsequent data packets is forcibly switched to the core field intersection mode.

[0026] This embodiment details the transaction control and circuit breaker logic during the data writing phase. The system utilizes a high-performance database connection pool to establish a physical connection channel with the target storage system and configures session parameters. The system initiates a database transaction and sets an appropriate transaction isolation level, executing the generated data docking instructions in batches within this transaction context. The system monitors database feedback signals during execution. If a database deadlock is triggered during execution, specifically, an SQL exception is captured, and its SQLState attribute matches a predefined set of fault codes, including 40001 (representing deadlock), HYT00 (representing lock timeout), or 22003 (representing numerical overflow), or a type conversion exception occurs, i.e., data truncation or mismatch. In response to such severe exceptions, the system immediately performs a rollback operation, undoing all writes in the current batch to protect database consistency. The system triggers a circuit breaker, forcibly locking the data packet processing strategy for the current batch and subsequent data packets within a certain time window to the core field intersection mode, and no longer performing entropy value judgments; at the same time, in order to maintain consistency with the feedback mechanism of Example 1, the system will synchronously adjust the subsequent semantic entropy calculation parameters, specifically executing differentiated parameter locking logic based on fault type: If a type conversion exception is caught, the system will adjust the weighting factor of the data type conflict rate. Update directly to the preset saturation upper limit value ; If a database deadlock exception is captured, the system will assign a weighting factor to the proportion of unknown key names. Updated to The parameter update operation here is a persistent modification of the system's global configuration. Therefore, when the system enters a half-open state after a cooling-off period, such as 300 seconds, the entropy calculation for subsequent data packets will be based on these saturated high-weighted data packets. This mathematically guarantees that when the risk characteristic reappears, the calculated entropy value will be accurate. Able to quickly exceed the preset safety threshold This forces the system to maintain a high sensitivity to similar risks in the early stages of a restart, due to the intersection pattern of core fields. This mechanism ensures that the system will not be attacked again with the same structural characteristics due to excessively rapid weight rollback at the moment of fault recovery; the weight decay logic will only be triggered after multiple consecutive batches, such as the 50 batches described in Example 1, are successfully written, and its calculation formula is as follows: The system gradually exits its defensive state. This differentiated feedback ensures that the entropy value calculated for the first time after the cooldown period accurately reflects the specific risk source that caused the crash. In this embodiment, the time window, i.e., the cooldown period, is preset to 300 seconds. After the cooldown period ends, the system enters a half-open state, allowing entropy calculation and a full attempt to access the next data packet. If the attempt succeeds, the cooldown period is reset. If it fails again, an exponential backoff algorithm is executed to extend the cooldown time. The specific backoff calculation formula is as follows: in, The cooling wait time for the current circuit breaker cycle, in seconds; The base cooldown time is set to 300 seconds, referencing the system mean time between failures (MTBF); constant. The binary exponential backoff base is dimensionless and is set with reference to standard network congestion control algorithms. This represents the cumulative number of consecutive circuit breakers triggered, expressed in times. The maximum cooldown time is set to 3600 seconds. The choice of 2 as the growth factor is based on the principle of the standard binary exponential backoff algorithm. This value has been widely verified in computer network congestion control as the optimal solution for balancing system recovery speed and load pressure. Experimental data shows that under a stress test with a concurrency of 500 transactions per second (TPS), using factor 2 can shorten the time for the database to recover from a locked state by about 40% compared to the linear growth strategy. After the system state stabilizes or after a preset cooling period, the forced locking is released and the adaptive mode is restored.

[0027] Example 7: The source system includes heterogeneous business systems, and the dynamic data packets are in a semi-structured format. The preset metadata structure includes the column name definitions, data type constraints, and primary key index information of the database tables. The metadata semantic entropy is used to quantify the potential pollution risk to the target storage system caused by the structural uncertainty of the dynamic data packets.

[0028] This embodiment defines the specific application scenarios and physical meaning of the parameters of the method; the source system includes heterogeneous business systems, such as medical information systems, logistics management systems or financial transaction systems, etc., and the dynamic data packets are semi-structured data streams, such as JSON, XML or YAML formats; these systems are usually developed by different vendors and have huge differences in interface standards; Dynamic data packets are limited to JSON format data streams. This format naturally supports nesting and weak typing, which is the main cause of structure drift. The preset metadata structure specifically corresponds to the data definition language definition of relational database tables, including column name definitions, data type constraints, and primary key index information used to uniquely identify records. Metadata semantic entropy is physically used to quantify the potential pollution risk to the target storage system caused by the structural uncertainty of dynamic data packets; the higher the entropy value, the greater the probability that the data packet will cause database errors, generate dirty data, or trigger table locks. This embodiment combines abstract algorithmic logic with specific pain points in medical data integration. Addressing the characteristics of high uncertainty in medical data structures, significant differences in field standards, and extremely high consistency requirements, it clarifies the applicable boundaries of the technical solution and demonstrates the specific effectiveness of this method in reducing data integration and maintenance costs and improving data warehouse quality in solving real-world industry challenges.

[0029] Example 8: Please see Figure 2 A cross-system data interface system based on dynamic metadata parsing includes the following modules: a data acquisition module, configured to acquire dynamic data packets sent by the source system and the preset metadata structure of the target storage system, wherein the dynamic data packets are in the form of a set of weakly typed key-value pairs.

[0030] The entropy calculation module is configured to calculate the metadata semantic entropy, which represents the uncertainty of the data structure, based on the structural difference characteristics between the key name set in the dynamic data packet and the preset metadata structure, and to prepare for parsing the dynamic data packet. The strategy decision module is configured to compare the semantic entropy of metadata with a preset security threshold, and select either the full dynamic mapping mode or the core field intersection mode based on the comparison result to generate the corresponding data docking instructions. The execution feedback module is configured to write data from dynamic data packets into the target storage system via data docking commands, and adjust subsequent semantic entropy calculation parameters based on the write result feedback. This embodiment provides a hardware and software architecture system for executing the above method; the data acquisition module, as the system's input gateway, is configured to receive JSON streams via Hypertext Transfer Protocol (HTTP) or a message queue interface and maintain a read connection with the target database metadata table; the entropy calculation module has a built-in string processing and statistical analysis engine, configured to execute the weighted algorithm of Embodiment 2 and output the entropy scalar in real time; The strategy decision module, as the logical control center, stores configurable safety threshold parameters and distributes control signals to the SQL generator based on the comparison results. The execution feedback module includes a Java database connection executor and an exception capture unit, configured to manage database transactions. When a specific SQL exception is captured, it sends a feedback signal to the entropy calculation module to adjust the weights or sends a degradation signal to the strategy decision module. This embodiment achieves decoupling and independent evolution of various functional components through modular architecture design; in particular, the closed-loop control system composed of the execution feedback module and the front-end computing module enables the entire device to have self-learning and self-adaptation capabilities, and can dynamically adjust its robust configuration as the data environment changes, adapting to long-term operation and maintenance needs.

[0031] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all implementation methods here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A method for cross-system data integration based on dynamic metadata parsing, characterized in that, The method is applied to a data processing system that connects a source system and a target storage system, and the method includes the following steps: The dynamic data packets sent by the source system and the preset metadata structure of the target storage system are obtained, wherein the dynamic data packets are in the form of a weakly typed key-value pair set, and the preset metadata structure defines the standard constraint parameters of the target field. Based on the structural difference characteristics between the key name set in the dynamic data packet and the preset metadata structure, the metadata semantic entropy representing the uncertainty of the data structure is calculated, and the dynamic data packet is prepared for parsing. The semantic entropy of the metadata is compared with a preset security threshold. Based on the comparison result, either the full dynamic mapping mode or the core field intersection mode is selected to generate the corresponding data docking instruction. The data in the dynamic data packet is written into the target storage system through the data docking command, and the subsequent semantic entropy calculation parameters are adjusted based on the writing result feedback.

2. The cross-system data interoperability method based on dynamic metadata parsing according to claim 1, characterized in that, The calculation of metadata semantic entropy, which characterizes the uncertainty of the data structure, based on the structural difference features between the key name set in the dynamic data packet and the preset metadata structure, includes: Extract the key name and value type of all key-value pairs in the dynamic data packet to construct the current transmission protocol characteristics; Calculate the percentage of unknown key names, the key name character edit distance, and the data type conflict rate between the current transmission protocol features and the preset metadata structure; Based on preset weighting coefficients, the weighted summation of the percentage of unknown key names, the edit distance of key name characters, and the data type conflict rate is calculated to obtain the semantic entropy of the metadata.

3. The cross-system data interoperability method based on dynamic metadata parsing according to claim 1, characterized in that, The step of selecting the full dynamic mapping mode or the core field intersection mode based on the comparison results includes: In response to the metadata semantic entropy being less than or equal to the preset security threshold, the full dynamic mapping mode is activated. In the full dynamic mapping mode, all key names in the dynamic data packet are directly mapped to column names of the target storage system, and a structured query language statement containing all fields is generated as the data docking instruction.

4. The cross-system data interoperability method based on dynamic metadata parsing according to claim 1, characterized in that, The step of selecting the full dynamic mapping mode or the core field intersection mode based on the comparison results also includes: In response to the metadata semantic entropy being greater than the preset security threshold, the core field intersection mode is activated; In the core field intersection mode, identify the fields marked as core business in the preset metadata structure, and calculate the intersection of the key name set of the dynamic data packet and the core business fields; Only data belonging to the intersection is extracted to generate a structured query language statement, and non-core fields in the dynamic data packet that are not included in the intersection are discarded to generate the data docking instruction.

5. A cross-system data interoperability method based on dynamic metadata parsing according to claim 1, characterized in that, The step of preparing to parse the dynamic data packet also includes: Monitor whether there are temporary fields or unexpected protocol variation features in the dynamic data packets that conform to the debugging naming rules; Upon detecting the protocol variation feature, a preset numerical penalty term is added to the metadata semantic entropy to increase the value of the metadata semantic entropy; The dynamic data packets are cleaned by removing the temporary fields, and the cleaned data is used as the data to be processed.

6. The cross-system data interoperability method based on dynamic metadata parsing according to claim 1, characterized in that, The step of writing data from the dynamic data packet into the target storage system via the data docking command includes: Establish a connection channel with the target storage system using a database connection pool; Execute the data docking instructions in batches under the transaction isolation level; If a database deadlock or type conversion exception is triggered during execution, the current transaction is rolled back, the corresponding risk weight coefficient is adjusted to the preset saturation upper limit, and the processing strategy of subsequent data packets is forcibly switched to the core field intersection mode.

7. The cross-system data interoperability method based on dynamic metadata parsing according to claim 1, characterized in that, The source system includes heterogeneous business systems, and the dynamic data packets are data streams in a semi-structured format; The preset metadata structure includes the column name definition, data type constraints, and primary key index information of the database table; The metadata semantic entropy is used to quantify the potential pollution risk to the target storage system caused by the structural uncertainty of the dynamic data packet.

8. A cross-system data interface system based on dynamic metadata parsing, running in a data processing system, for implementing the cross-system data interface method based on dynamic metadata parsing as described in any one of claims 1-7, characterized in that, Includes the following modules: The data acquisition module is configured to acquire dynamic data packets sent by the source system and a preset metadata structure of the target storage system, wherein the dynamic data packets are in the form of a set of weakly typed key-value pairs; The entropy calculation module is configured to calculate the metadata semantic entropy, which represents the uncertainty of the data structure, based on the structural difference features between the key name set in the dynamic data packet and the preset metadata structure, and to prepare for parsing the dynamic data packet. The strategy decision module is configured to compare the semantic entropy of the metadata with a preset security threshold, and select either the full dynamic mapping mode or the core field intersection mode based on the comparison result to generate the corresponding data docking instruction. The execution feedback module is configured to write the data in the dynamic data packet into the target storage system through the data docking instruction, and adjust the subsequent semantic entropy calculation parameters based on the writing result feedback.

Images