Deterministic Idempotent Reading Method and System Across Heterogeneous Data Sources
By intercepting and parsing read requests at the middleware layer and generating deterministic read tasks, combined with Bloom filters and ladder-like idempotency checks of persistent indexes, the nondeterministic problem of reading across heterogeneous data sources is solved, achieving data consistency and efficient auditing capabilities, and addressing the issues of storage cost and performance degradation in existing technologies.
Patent Information
- Application Number
- CN202611360165.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-03
- Publication Date
- 2026-11-17
AI Technical Summary
In high-concurrency, high-availability distributed systems, read operations across heterogeneous data sources are subject to uncertainty risks, leading to data inconsistency and phantom reads. Furthermore, existing technologies cannot effectively shield the underlying differences between heterogeneous data sources, resulting in storage costs and performance degradation, which cannot meet the long-term, precise auditing needs of the financial and regulatory sectors.
By intercepting and parsing read requests through the middleware layer, deterministic read tasks are generated. Bloom filters and persistent indexes are used for idempotent verification. Combined with hierarchical content semantic fingerprinting technology, deterministic idempotent reads across heterogeneous data sources are achieved, ensuring data consistency and generating unforgeable audit anchors.
It effectively shields the update frequency and version differences of heterogeneous data sources, eliminates the risk of cross-source semantic drift and phantom reads during the retry process, achieves absolute consistency of data entity semantics, reduces the penetration pressure of high-frequency retries on the underlying data source, balances storage costs and system performance, and provides long-term accurate auditing capabilities.
Smart Images

Figure CN123115505A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer distributed systems technology, and in particular to a deterministic idempotent reading method and system for cross-heterogeneous data sources. Background Technology
[0002] With the widespread adoption of microservice architectures in distributed systems, remote calls between services inevitably face uncertainties such as network jitter and momentary node failures. Retry mechanisms have become a standard configuration for ensuring system availability. Current industry research and practice on idempotency primarily focuses on write operations, using database unique keys, distributed locks, or token mechanisms to prevent duplicate insertions or updates. Read operations are generally considered inherently idempotent, meaning multiple reads do not produce side effects. However, in high-concurrency, high-availability distributed environments, the uncertainty of read semantics is increasingly prominent, becoming a key hidden danger affecting the accuracy of business decisions.
[0003] Existing technical solutions mainly fall into two categories: interface-layer deduplication mechanisms, which record request identifiers at the gateway or interface layer and directly return the previous response status or a processing status when a duplicate request is detected. This type of solution only guarantees the idempotency of the request process and does not address the real-time changes in underlying data within the retry span. In extremely high-frequency trading scenarios, if the data source changes within milliseconds, the data obtained from the retry request will be inconsistent with the original request, leading to "phantom reads" in critical business processes such as billing and risk control. Reading solutions based on Multi-Version Concurrency Control (MVCC) rely on the physical implementation of a specific data source (such as a relational database) and achieve consistent reading by maintaining multiple historical versions of data rows. However, in scenarios involving heterogeneous data sources (such as a mix of relational databases and real-time search engines / document databases), the versioning mechanisms, update frequencies, and garbage collection strategies of each data source are independent, making global version alignment difficult. Furthermore, the version cleanup cycle of the data source is not controlled by the middleware layer, failing to meet the needs of finance, regulation, and other fields for long-term, accurate audit backtracking, creating a sharp contradiction between storage costs and audit compliance.
[0004] In summary, existing technologies suffer from the following core defects: First, the risk of "non-determinism" in read semantics, where consistent data snapshots cannot be guaranteed to be seen within the retry lifecycle for the same request identifier; second, the loss of focus in cross-source semantics, as the lack of a unified middleware-level view lock to shield the underlying differences of heterogeneous data sources leads to logical drift in cross-source query results; third, the trade-off between storage space and performance, where storing all historical versions by the data source will result in severe storage skew and write performance degradation; and fourth, the lack of audit verifiability, as existing idempotent components do not save binary fingerprints of return values, making it impossible to provide unforgeable on-site data evidence in the event of disputes, thus lacking technical credibility for post-event tracing. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and to propose a deterministic idempotent reading method and system for cross-heterogeneous data sources.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] On the one hand, a deterministic idempotent reading method across heterogeneous data sources is provided, including the following specific steps:
[0008] S1. The middleware layer intercepts and parses the received read request, and extracts the globally unique request identifier carried in the read request; if the read request does not carry a globally unique request identifier, a default request identifier is generated based on the target data source identifier, query condition parameters, caller source information, business domain identifier and current timestamp thumbnail of the read request.
[0009] Parse the read request to obtain the data source access intent, which includes the target heterogeneous data source set explicitly declared in the read request or implicitly inferred based on the request routing parameters; parse the read request to determine the business consistency level;
[0010] The globally unique request identifier or default request identifier, data source access intent, and business consistency level are deterministically bound to generate a read semantic contract; the read semantic contract is encapsulated into a deterministic read task; the logical timestamp when the read request is intercepted is recorded, and the logical timestamp is encapsulated into the deterministic read task; the middleware layer is configured with a heterogeneous data adaptation driver module, which is used to convert the original results of different data sources into the middleware standard binary snapshot format and restore the middleware standard binary snapshot format into structured data;
[0011] S2. Receive the deterministic read task, and extract the globally unique request identifier or default request identifier and the read semantic contract;
[0012] In the distributed memory storage, a Bloom filter is used for initial screening based on the globally unique request identifier or the default request identifier. If the initial screening is successful, the pre-stored response body is obtained. A contract consistency pre-check is performed on the pre-stored response body to verify whether the data source coverage set and business consistency level associated with the pre-stored response body are consistent with the data source access intent and business consistency level in the reading semantic contract. If they are consistent, the pre-stored response body is returned directly. If they are inconsistent, it is considered a miss.
[0013] If the initial screening fails or the contract consistency pre-check fails, the persistent snapshot index is queried. The persistent snapshot index records the semantic fingerprint associated with a globally unique request identifier or a default request identifier, the data source coverage set, the business consistency level, the snapshot error correction check code, and the semantic fingerprint version chain pointer. If the index is found, the corresponding snapshot file is subjected to an integrity check based on the snapshot error correction check code. If the check passes, the contract consistency pre-check is performed again. If both checks pass, the snapshot file is loaded, restored to structured data by the heterogeneous data adaptation driver module, and then returned. If the integrity check fails or the contract consistency pre-check fails, it is considered a miss.
[0014] If none of the above levels are hit, a penetration command is generated;
[0015] S3. Receive the penetration instruction, and according to the data source access intention, initiate a real-time read request to the downstream heterogeneous data source through the heterogeneous data adaptation driver module;
[0016] Obtain the original results and source response metadata from each heterogeneous data source. The source response metadata includes source identifier, response timing marker, and source transaction status marker. Convert the original results from different data sources into the middleware standard binary snapshot format through the heterogeneous data adaptation driver module.
[0017] Semantic normalization is performed on the data in the middleware standard binary snapshot format. The semantic normalization process includes rearranging the field order according to a preset specification, uniformly filling null values according to a preset rule, and standardizing the data type according to a preset mapping rule. After completing the semantic normalization process, a logical timestamp is extracted from the deterministic reading task, and the logical timestamp is bound to the normalized data.
[0018] Calculating the hierarchical content semantic fingerprint includes calculating the first fingerprint for the structured semantic view corresponding to the bound normalized data, and calculating the second fingerprint for the original binary byte stream of the middleware standard binary snapshot format, and fusing the logical timestamp, the first fingerprint and the second fingerprint to generate the current time dimension semantic fingerprint.
[0019] Generate a cross-source consistency descriptor to record the data source access intent, the actual data source coverage set obtained, the response metadata of each source, the business consistency level, and the hierarchical content semantic fingerprint;
[0020] The following two operations will be executed in parallel:
[0021] (a) The original result is converted into a standard structured data format expected by the caller via the heterogeneous data adaptation driver module and then returned to the caller;
[0022] (b) Initiate an asynchronous task to establish a mapping from a globally unique request identifier or a default request identifier to a content snapshot in the distributed memory storage and persistent snapshot index based on the cross-source consistency descriptor. The mapping includes creating a semantic fingerprint record with the request identifier as the key. The semantic fingerprint record includes the request identifier, the current time dimension semantic fingerprint, the snapshot expiration timestamp, the semantic fingerprint version chain pointer, the data source coverage set, the business consistency level, the record status, and the snapshot error correction check code. The semantic fingerprint version chain pointer is used to point to the storage location of the previous time dimension semantic fingerprint generated by the same request identifier at a historical time point. While establishing the mapping, check whether a physical data snapshot with the same content already exists in the snapshot storage pool with the first fingerprint as the content addressing identifier. If it does not exist, the original result is serialized into a physical data snapshot in binary format and stored in the snapshot storage pool. A dual index is established for the physical data snapshot with the current time dimension semantic fingerprint as the index key and the first fingerprint as the content addressing identifier. If it already exists, the existing physical data snapshot in the snapshot storage pool is reused and an index mapping from the current time dimension semantic fingerprint to the existing physical data snapshot is established.
[0023] The content snapshots are managed for lifecycle according to the data retention policy configured in the business settings.
[0024] The middleware layer periodically performs snapshot consistency maintenance operations: it scans the semantic fingerprint records in the persistent snapshot index, marks semantic fingerprint records whose current time has exceeded the expiration time indicated by the snapshot expiration timestamp as pending archiving, and traces the complete semantic fingerprint version chain of the request identifier according to the semantic fingerprint version chain pointer. It then transfers all expired semantic fingerprint records in the complete semantic fingerprint version chain to the archive index area and deletes the transferred semantic fingerprint records from the persistent snapshot index. For semantic fingerprint records that have been transferred to the archive index area, it further checks whether the physical data snapshots corresponding to each semantic fingerprint in the snapshot storage pool are still referenced by the remaining non-expired semantic fingerprint records in the persistent snapshot index or other archived records in the archive index area. If a physical data snapshot is not referenced by any valid record, the middleware layer deletes the physical data snapshot from the snapshot storage pool.
[0025] On the other hand, a deterministic idempotent read system across heterogeneous data sources is provided to implement deterministic idempotent read methods across heterogeneous data sources, including:
[0026] The request interception and semantic contract anchoring module is used by the middleware layer to intercept read requests, extract or generate request identifiers, parse the data source access intent and business consistency level; generate read semantic contracts, record the logical timestamp when the read request is intercepted, and encapsulate the read semantic contracts and logical timestamps into a deterministic read task.
[0027] The ladder-type idempotency verification module is communicatively connected to the request interception and semantic contract anchoring module. It is used to receive the deterministic reading task, perform ladder-type idempotency verification in distributed memory storage and persistent snapshot index library, and perform initial screening for existence and contract consistency pre-check by Bloom filter. If it fails, it performs snapshot error correction check code verification. If it hits, it loads and returns. If it misses, it generates a penetration instruction.
[0028] The heterogeneous source penetration and snapshot anchoring module is communicatively connected to the ladder-type idempotent verification module. It is used to receive the penetration command, initiate a request to the downstream through the heterogeneous data adaptation driver module configured in the middleware layer, obtain the original result and convert it into the middleware standard binary snapshot format, bind it to the logical timestamp after semantic normalization, calculate the hierarchical content semantic fingerprint to generate a cross-source consistency descriptor, return the result in parallel and asynchronously establish the request identifier to the content snapshot mapping, and perform lifecycle management according to the data retention strategy.
[0029] The beneficial effects of this invention are as follows:
[0030] 1. By reading the semantic contract, the request identifier, data source access intent, and business consistency level are deterministically bound, and contract consistency pre-checks are performed in each level of verification. This effectively shields the differences in update frequency and version mechanism of heterogeneous backend data sources, fundamentally eliminating the risks of cross-source semantic drift and phantom reads during the retry process, and ensuring the absolute consistency of data entity semantics within the same request lifecycle.
[0031] 2. By building a tiered idempotent verification architecture of "in-memory screening - persistent index - source penetration" in the middleware layer, the idempotent boundary is advanced from the traditional protocol layer to the data semantic layer. Without relying on the isolation level or multi-version support of the underlying database, a virtual unified snapshot can be achieved on heterogeneous data sources, significantly reducing the penetration pressure of high-frequency retries on the underlying data sources.
[0032] 3. Employing hierarchical content semantic fingerprinting technology, which integrates semantic view fingerprints and binary integrity fingerprints, it achieves deduplication of physical snapshots across requests while generating unforgeable content-level audit anchors for each read. Combined with business-driven TTL snapshot lifecycle management, it effectively balances storage costs and system performance while ensuring accurate long-term audit backtracking capabilities. Attached Figure Description
[0033] Figure 1 This is a flowchart of the method proposed in Embodiment 1 of the present invention;
[0034] Figure 2 This is a diagram of the middleware layer architecture of Embodiment 1 of the present invention;
[0035] Figure 3This is a framework diagram of the system proposed in Embodiment 2 of the present invention. Detailed Implementation
[0036] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.
[0037] Example 1
[0038] Please see the appendix Figure 1-2 A deterministic idempotent reading method across heterogeneous data sources includes the following specific steps:
[0039] S1. The middleware layer intercepts read requests, extracts or generates request identifiers, and parses the data source access intent and business consistency level; it generates a read semantic contract, records the logical timestamp when the read request is intercepted, and encapsulates the read semantic contract and logical timestamp into a deterministic read task; specifically including:
[0040] S11. Inference of data source access intent and determination of business consistency level, specifically including:
[0041] S111. Upon receiving a read request, the middleware layer first checks if the request header contains an explicitly declared target data source collection field. This field exists in the request header as a key-value pair, with the key name defaulting to `datasource-target` and its value being a comma-separated list of data source identifiers. If this field is detected and its value is not empty, it is directly parsed as the target data source collection. If the field does not exist in the request header or its value is empty, the implicit inference process is initiated. The middleware layer extracts the Uniform Resource Locator (URL) path portion of the request and matches it against a preset routing rule table. The number of entries in this routing rule table ranges from 50 to 500, with a preferred value of 100. Simultaneously, it extracts the service interface identifier invoked by the request, which is either the last segment of the interface path or a custom interface code.
[0042] The query condition parameters are further analyzed to identify the data table names or index names involved. The middleware layer inputs the three dimensions of features—routing path, service interface identifier, and data table or index name—into the data source routing rule engine. This engine uses a priority scoring mechanism, and the middleware layer evaluates candidate data sources. Matching score Calculate using the following formula: In the formula: Candidate data sources Match score; Weights are assigned to the routing paths, with values ranging from 0.1 to 0.8, preferably 0.5; This is a route path matching flag; it is 1 if a route path match is found, and 0 otherwise. Assign a weight to the service interface identifier, with a value ranging from 0.1 to 0.8, and a preferred value of 0.3; This is a matching flag for the service interface identifier; it is 1 if the service interface identifier matches, and 0 otherwise. Assign weights to data table or index names, with values ranging from 0.1 to 0.8, and a preferred value of 0.2. ; This is a flag indicating whether a table or index name matches; it is 1 if the table or index name matches, and 0 otherwise.
[0043] When the matching score of a certain data source reaches or exceeds the total score threshold, that is: When, it is included in the inference results, where The overall score threshold is set between 0.5 and 0.9, with a preferred value of 0.6. Finally, the explicitly extracted or inferred list of data source identifiers, after deduplication, is used as the data source access intent.
[0044] S112. When resolving the business consistency level, the middleware layer first scans the parameter list of the read request to check if there is an explicitly declared business consistency level parameter. This parameter exists as a string in the request body or request header, with the default parameter key name being consistency-level. Its optional value range is limited to three enumerated values: strong consistency level, session consistency level, and eventual consistency level. If the request carries this parameter and its value belongs to the above enumerated value range, it is directly determined as the business consistency level of the current read request. If the request does not carry this parameter or its value exceeds the enumerated value range, the middleware layer initiates the default level inference process.
[0045] The middleware layer extracts the business type identifier associated with the read request. This identifier is provided by the caller in the request header as a business domain identifier. The default length of the business domain identifier is an 8-character alphanumeric combination, with a length ranging from 6 to 16 characters, preferably 8 characters. The middleware layer queries a default consistency level mapping table, which has 30 to 200 entries, preferably 50. In the mapping rules, financial billing business type identifiers are forcibly mapped to a strong consistency level, user session business type identifiers are mapped to a session consistency level, and data display business type identifiers are mapped to an eventual consistency level. For business type identifiers not matched in the mapping table, the eventual consistency level is used as a fallback strategy by default.
[0046] S12. Encapsulation of semantic contract generation and deterministic reading tasks, specifically including:
[0047] S121. After obtaining the globally unique request identifier or default request identifier, the data source access intent, and the business consistency level, the middleware layer initiates the semantic contract generation process. First, the middleware layer creates a new contract structure, which pre-constructs six fields: contract identifier, target data source list, access order dependency, business consistency level, contract generation timestamp, and contract status.
[0048] The contract identifier field is filled with a globally unique request identifier or a default request identifier to ensure a one-to-one correspondence between contracts and requests. The target data source list field is filled with a serialized array of data source identifiers obtained from parsing the data source access intent. The number of array elements ranges from 1 to 20, with a preferred value of 3 to 5. Each identifier's length ranges from 4 to 64 bytes. The access order dependency field describes the read execution order between target data sources and is stored as a directed graph adjacency list. Nodes in the graph are data source identifiers, edges represent dependency directions, and an edge weight of 1 indicates a serial dependency, while a weight greater than 1 indicates a parallel dependency, with the weight representing the concurrency group number. Specifically: Access Order Dependency... Based on edge weight The judgment is as follows: In the formula: This refers to the access order dependency type; This represents the weight of the edges in the directed graph, ranging from 1 to 10. The Business Consistency Level field should be filled with the predefined business consistency level enumeration value. The Contract Generation Timestamp field should be filled with the current system timestamp, with millisecond precision. The Contract Status field is initially set to "active," and the status enumeration values include "active," "expired," and "revoked."
[0049] The above six fields are bound into triples according to a preset contract structure. The contract identifier, the target data source list, and the business consistency level constitute the core triple, while the access order dependency serves as an extended constraint, generating the final read semantic contract. This read semantic contract is persisted to the middleware layer's local memory cache. The cache validity period is preset to 300 seconds, with a value ranging from 60 to 3600 seconds, and a preferred value of 300 seconds. It serves as a verification benchmark in subsequent idempotency checks at various levels to determine whether the cached snapshot data meets the business semantic matching conditions of the current read request.
[0050] S122. After generating the read semantic contract, the middleware layer initiates the deterministic read task encapsulation process. First, the middleware layer creates a task encapsulation structure, which pre-constructs five fields: task identifier, semantic contract, logical timestamp, original query condition parameters, and task creation timestamp. The task identifier field is directly filled with a globally unique request identifier or a default request identifier, strictly consistent with the contract identifier in the read semantic contract. The semantic contract field embeds the generated read semantic contract object by reference to avoid data redundancy. The logical timestamp field is directly filled with the logical timestamp recorded in step S1 when the read request was intercepted. This timestamp is generated by the middleware layer's internal logical clock, decoupled from the physical clock, with millisecond precision. The timestamp length ranges from 13 to 19 digits, with a preferred value of 13 milliseconds. The original query condition parameters field is filled with the serialized query condition portion of the read request. The serialization format is pre-constructed as a JSON string, with a string length ranging from 32 to 4096 bytes, with a preferred value of 256 to 1024 bytes. The task creation timestamp field should be filled with the current system's physical timestamp, with a precision of milliseconds.
[0051] The five fields mentioned above are assembled according to a preset task encapsulation format to generate a deterministic read task. This deterministic read task is submitted to the task scheduling queue of the middleware layer, with a queue depth ranging from 100 to 10000, preferably 1000, waiting for downstream processing modules to consume and execute it.
[0052] S2. Perform a ladder-style idempotent check on the distributed memory storage and persistent snapshot index, followed by an initial existence screening using a Bloom filter and a contract consistency pre-check; if the check fails, perform a snapshot error correction checksum check; if a match is found, load and return; otherwise, generate a penetration instruction; specifically including:
[0053] S21.L1 Distributed Memory Cache Bloom Filtering and Contract Consistency Pre-check specifically includes:
[0054] S211. After receiving a deterministic read task, the downstream processing module of the middleware layer first performs deserialization parsing on the task structure, and extracts the task identifier field value as the request identifier, the semantic contract field value as the read semantic contract, and the original query condition parameter field value as the original query condition parameter.
[0055] Subsequently, the middleware layer creates a validation benchmark object in the local thread context. It writes the target data source list field value from the read semantic contract to the data source access intent attribute of this benchmark object, and the business consistency level field value to the consistency level attribute. The lifecycle of this benchmark object is bound to the processing thread of the current read request; it remains valid as long as the thread is alive, constraining the business semantic matching conditions that cached snapshots must meet in subsequent hit determinations. The memory footprint of the benchmark object is preset to 256 bytes to 4096 bytes, with a preferred value of 1024 bytes.
[0056] S212. The middleware layer uses the request identifier as the query key, calculates the hash value of the query key, with the default hash algorithm being MurmurHash3, and the output length being 32 bits or 128 bits, with 32 bits being the preferred value. The middleware layer maps the calculated hash value to a Bloom filter bit array in distributed in-memory storage. Query Key When mapping to the Bloom filter bit array, the first The bit array index corresponding to each hash function Calculate using the following formula: In the formula: and For query keys The two unsigned integer values obtained by performing two independent hash operations; This is the query key, i.e., a globally unique request identifier or a default request identifier; For the first The bit array index obtained by mapping the hash function. The current false positive rate of the Bloom filter. Estimate using the following formula: In the formula: The length of the Bloom filter bit array, with a value ranging from 10,000 to 10,000,000, and a preferred value of 1000,000. The number of hash functions, ranging from 3 to 10, with 7 being the preferred value; This represents the number of elements currently inserted into the Bloom filter. The false positive rate of the Bloom filter ranges from 0.001 to 0.01, with a preferred value of 0.001.
[0057] If the Bloom filter determines that the request identifier does not exist, it directly marks the L1 layer as a miss and proceeds to query the persistent snapshot index. If the Bloom filter determines that the request identifier may exist, it queries the key-value storage engine in the distributed in-memory storage using the request identifier as the key to obtain the historical data source coverage set, historical business consistency level, and snapshot timestamp of the pre-stored response body and its associated storage. The middleware layer calculates the difference between the current time and the snapshot timestamp. If the difference exceeds the memory storage validity period of the pre-stored response body, it determines that the pre-stored response body has expired and is considered an L1 layer miss; if it does not exceed the difference, it continues to perform the contract consistency pre-check. The memory storage validity period of the pre-stored response body ranges from 60 to 600 seconds, with a preferred value of 300 seconds.
[0058] S213. The middleware layer extracts the target data source list from the validation benchmark object and performs a first comparison with the historical data source coverage set. The first comparison uses set inclusion relationship determination, iterating through each data source identifier in the target data source list and checking if that identifier exists in the historical data source coverage set. If all identifiers in the target data source list are contained in the historical data source coverage set, the first comparison passes. Formally, the first comparison passes if and only if: TargetSet HistorySet is a collection of historical data sources, where TargetSet is the list of target data sources indicated by the data source access intent in the validation baseline object, and HistorySet is the collection of historical data source overlays.
[0059] Simultaneously, the middleware layer performs a second comparison between the historical business consistency level and the business consistency level in the verification benchmark object. This second comparison uses exact string matching; if the character sequences of the two are completely identical, the second comparison passes. If both the first and second comparisons pass, the contract consistency pre-check passes, and the middleware layer directly returns the pre-stored response body as the response result of the current read request. If either the first or second comparison fails, the contract consistency pre-check fails, the initial screening result of the L1 layer is determined to be a miss, and the process proceeds to query the persistent snapshot index.
[0060] S22.L2 persistent index error correction verification and penetration instruction generation specifically includes:
[0061] S221. After determining that the L1 layer has not been hit, the middleware layer initiates a query request to the persistent snapshot index using a globally unique request identifier or a default request identifier as the query key. The persistent snapshot index is implemented using a distributed key-value storage engine. The engine's preset number of shards ranges from 3 to 128, with a preferred value of 16, and the replication factor ranges from 2 to 5, with a preferred value of 3. The query request timeout ranges from 50 to 5000 milliseconds, with a preferred value of 500 milliseconds. The persistent index record is stored with the query key as the primary key. Its data structure is preset to include five fields: semantic fingerprint field, snapshot storage location field, snapshot error correction checksum field, historical data source coverage set field, and historical business consistency level field. The checksum field stores redundancy information that is synchronously written when the physical snapshot file is written to disk. The checksum length ranges from 4 to 64 bytes, with a preferred value of 16 bytes.
[0062] S222. The middleware layer extracts the snapshot error correction checksum and snapshot storage location from the persistent index records. The snapshot storage location is represented in Uniform Resource Locator (URL) format or file system absolute path format, with a path depth ranging from 1 to 10 levels, preferably 3 levels. The middleware layer performs integrity checks on the physical snapshot file corresponding to the snapshot storage location based on the snapshot error correction checksum. The integrity check uses either Cyclic Redundancy Check (CRC) or Reed-Solomon error correction algorithm. The polynomial order of CRC ranges from 8 to 32 bits, preferably 32 bits; the data block size of the Reed-Solomon error correction algorithm ranges from 512 to 4096 bytes, preferably 1024 bytes, and the number of redundant blocks ranges from 2 to 8, preferably 4. If the integrity check fails, it indicates that the physical snapshot file has been corrupted by bit flipping or illegally tampered with, and the middleware layer directly determines that the L2 layer query result is a miss. If the integrity check passes, the middleware layer continues to extract the historical data source coverage set and historical business consistency level from the persistent index records.
[0063] S223. Compare the historical data source coverage set with the data source access intent in the verification benchmark object to determine whether the historical data source coverage set completely covers the target data source list indicated by the data source access intent; formally, this comparison is successful if and only if: TargetSet The `HistorySet` is a set of historical data sources, where `TargetSet` is the list of target data sources indicated by the data source access intent in the validation benchmark object, and `HistorySet` is the set of historical data source coverage. Simultaneously, the historical business consistency level is compared with the business consistency level in the validation benchmark object to determine if they are the same.
[0064] If both comparisons pass, the middleware layer determines that the L2 index has been hit. It then loads the physical snapshot file from the snapshot storage location into a memory buffer. The buffer size ranges from 64 to 8192 kilobytes, with a preferred value of 1024 kilobytes. After loading, the heterogeneous data adaptation driver module restores the physical snapshot file into structured data. The serialization protocol for the restoration process is preset to JSON or Protocol Buffers. The restored data is then returned to the caller. If either comparison fails, the middleware layer determines that the L2 index has not been hit.
[0065] S224. If the initial screening at layer L1 fails or the contract consistency pre-check fails, and the index at layer L2 fails, the middleware layer determines that all idempotency checks at each level have failed and initiates the penetration instruction generation process. The penetration instruction is encapsulated in a structure containing three core fields: a request identifier field, a read semantic contract field, and a raw query condition parameter field. The value of the request identifier field is directly filled with a globally unique request identifier or a default request identifier. The read semantic contract field is embedded using a deep copy method into the read semantic contract object corresponding to the current read request. The raw query condition parameter field is filled with the raw query condition parameters parsed from the deterministic read task. After the penetration instruction is generated, the middleware layer writes it to the task queue of the downstream processing module. The queue length ranges from 100 to 10000, with a preferred value of 1000, awaiting consumption and execution by the heterogeneous source contract fulfillment penetration process in step S3.
[0066] S3. The heterogeneous data adaptation driver module configured in the middleware layer sends a request to the downstream to obtain the original result and convert it into the middleware standard binary snapshot format. After semantic normalization, it is bound to the logical timestamp.
[0067] The process involves calculating hierarchical content semantic fingerprints to generate cross-origin consistency descriptors, returning results in parallel, asynchronously establishing a mapping from request identifiers to content snapshots, and managing the lifecycle according to data retention strategies. Specifically, this includes:
[0068] S31. Real-time penetration of heterogeneous data sources and standardized snapshot generation, specifically including:
[0069] S311. After receiving the penetration command, the downstream processing module of the middleware layer extracts the globally unique request identifier or default request identifier from the request identifier field of the penetration command, extracts the reading semantic contract from the reading semantic contract field, and extracts the original query condition parameters from the original query condition parameter field. The middleware layer parses the data source access intent in the reading semantic contract and extracts a list of target data sources from the data source access intent. This list contains one or more data source identifiers, with the number of data source identifiers ranging from 1 to 20, preferably 3 to 5. At the same time, it extracts the access order dependency relationship between each target data source. This dependency relationship is described in the form of a directed graph, where nodes are data source identifiers and edges represent execution order constraints. The access order dependency relationship includes two types: parallel dependency relationship and serial dependency relationship. The weight of the edge ranges from 1 to 10, with a weight of 1 indicating a serial dependency relationship and a weight greater than 1 indicating a parallel dependency relationship, and the weight value representing the concurrent group number.
[0070] S312. The middleware layer, based on the target data source list and access order dependencies, schedules the corresponding adaptation drivers in the heterogeneous data adaptation driver module, and issues real-time read requests to each target data source according to the access order dependencies. For multiple target data sources indicated by parallel dependencies, the middleware layer issues read requests concurrently. The size of the concurrent thread pool ranges from 2 to 64, with a preferred value of 8. The timeout for a single data source connection ranges from 100 to 10000 milliseconds, with a preferred value of 2000 milliseconds. For multiple target data sources indicated by serial dependencies, the middleware layer issues read requests sequentially according to the order specified by the serial dependencies. The total timeout for the serial link ranges from 500 to 30000 milliseconds, with a preferred value of 10000 milliseconds. The request interval between two adjacent data sources ranges from 0 to 1000 milliseconds, with a preferred value of 10 milliseconds. Each real-time read request carries the original query condition parameters and a globally unique request identifier or a default request identifier.
[0071] S313. The middleware layer receives the raw results returned by each target data source and the corresponding source response metadata. The source response metadata includes a source identifier, which is a string with a length ranging from 4 to 64 bytes. The middleware layer uses a heterogeneous data adaptation driver module to convert the raw results returned by different target data sources into a middleware standard binary snapshot format that conforms to the middleware layer's unified data specification. The conversion process includes protocol parsing, character encoding unification, and field structure flattening. The character encoding is uniformly UTF-8. After conversion, each raw result is associated and bound to its corresponding source identifier, generating a standardized snapshot set with data source lineage markers. The lineage markers are embedded in the snapshot header in key-value pair format, with the key being `source-id` and the value being the corresponding source identifier. The size of a single middleware standard binary snapshot format memory object in the standardized snapshot set ranges from 128 bytes to 50 megabytes, with a preferred value of 64 kilobytes. The upper limit of memory usage for the entire set ranges from 1 megabyte to 100 megabytes, with a preferred value of 10 megabytes. The size range of the physical snapshot file after serialization and disk writing is shown in S343.
[0072] S32. Semantic normalization and time dimension anchoring of multi-source data, specifically including:
[0073] S321. The middleware layer extracts middleware standard binary snapshot format data corresponding to each data source from a standardized snapshot set with data source lineage markers. During extraction, it iterates according to the target data source list order in the data source access intent, aggregating data records from the same logical group into the same memory partition. Group merging employs a hash bucketing strategy, with the data record's rec index belonging to that bucket. Calculate using the following formula: In the formula: This is the identifier of the data source associated with this data record; rec is a single data record in the multi-source dataset to be normalized; This is a bucket index, with values ranging from 0 to... ; This is a hash function that takes a data source identifier as input and outputs an unsigned integer value. This represents the total number of hash buckets, ranging from 16 to 1024, with 256 being the preferred value.
[0074] Data within each bucket is arranged lexicographically by the data source identifier. The maximum number of records in a single bucket ranges from 1,000 to 100,000, with a preferred value of 10,000 records. After merging, a multi-source dataset to be normalized is generated. This dataset is temporarily stored in a memory buffer as a two-dimensional table. The initial capacity of the buffer ranges from 1 megabyte to 100 megabytes, with a preferred value of 10 megabytes. When the size of the multi-source dataset to be normalized reaches 90% of the current buffer capacity, expansion is triggered. The expansion factor ranges from 1.5 to 2.0, with a preferred value of 2.0.
[0075] S322. The middleware layer performs semantic normalization on each data record in the multi-source dataset to be normalized. During field rearrangement, the fields within the record are rearranged according to a preset canonical field order list. The number of entries in the canonical field order list ranges from 10 to 500, with a preferred value of 100. The field matching tolerance is that the field name similarity is not less than 0.85. The similarity algorithm uses Levenstein distance or Jaccard coefficient; if Jaccard coefficient is used, the field name similarity... Calculate using the following formula: In the formula: To standardize the set of characters for field names in the field sequence table; The set of characters representing the actual field names of the data records to be matched; when When they are determined to be the same field.
[0076] When filling null values, a preset null value filling rule applies: string types are filled with an empty string, numeric types with 0, boolean types with false, and time types with January 1, 1970, 00:00:00. During data type conversion, a preset data type mapping rule applies, converting data source-specific types to a unified middleware layer type. The number of mapping rule entries ranges from 20 to 200, with a preferred value of 50.
[0077] S323. After completing semantic normalization, the middleware layer extracts the logical timestamp field value from the task structure of the deterministic read task. This logical timestamp is used as a time dimension anchor and written as a key-value pair into the metadata header of the normalized data. The key is defaulted to `logic-timestamp`, and the value is a 13-digit millisecond-level timestamp string. After binding, a semantically normalized data structure with a time dimension marker is generated. This structure defaults to two fields: a data payload field and a time dimension marker field. The data payload field stores the normalized two-dimensional table data, and the time dimension marker field stores the logical timestamp. The serialized byte stream length ranges from 256 bytes to 50 megabytes, with a preferred value of 1 megabyte. The default serialization format is JSON or Apache Avro.
[0078] S33. Hierarchical semantic fingerprint calculation and cross-source consistency descriptor generation, specifically including:
[0079] S331. The middleware layer extracts normalized data with completed time dimension tag binding from the data payload field of the semantically normalized data structure, and uses it as a structured semantic view. Simultaneously, it extracts the raw binary byte streams in the middleware standard binary snapshot format corresponding to each data source from the standardized snapshot set with data source lineage tags, and uses them as binary views. The structured semantic view and the binary view each allocate independent memory cache areas, with cache sizes ranging from 64 kilobytes to 50 megabytes, preferably 2 megabytes. The two views are stored in different memory formats: the structured semantic view is stored as an object tree with a node depth ranging from 1 to 20 levels, preferably 5 levels; the binary view is stored as a contiguous byte array with an array length ranging from 128 bytes to 10 megabytes, preferably 64 kilobytes. After caching, the middleware layer labels the two views with view type tags: the structured semantic view is labeled "semantic-view," and the binary view is labeled "binary-view," for differentiation in subsequent fingerprint calculation processes.
[0080] S332. The middleware layer performs fingerprint calculation on the structured semantic view. First, the structured semantic view is converted into a standardized structured text string according to preset serialization rules. The serialization rules are preset to JSON compact format or XML standard format, with string encoding uniformly set to UTF-8. The string length ranges from 256 bytes to 10 megabytes, with a preferred value of 512 kilobytes. A hash value is calculated for this structured text string as the first fingerprint. The hash algorithm is preset to SHA-256 or SHA-3, with an output length ranging from 256 to 512 bits, with a preferred value of 256 bits. The first fingerprint is used to characterize the uniqueness of the data's semantic content, serving as the content addressing identifier during subsequent physical snapshot storage deduplication. Simultaneously, the middleware layer directly calculates a hash value for the binary view as the second fingerprint. The hash algorithm used is the same as or different from the first fingerprint, and the output length is consistent with the first fingerprint. The second fingerprint is used to characterize the integrity of the original binary data.
[0081] S333. The middleware layer extracts the logical timestamp from the deterministic read task and converts it into a fixed-length timestamp byte sequence according to a preset encoding format. The preset encoding format is a 64-bit big-endian unsigned long integer, and the byte sequence length is fixed at 8 bytes. The first fingerprint, the second fingerprint, and the timestamp byte sequence are concatenated according to a preset fusion rule, with the preset concatenation order being the first fingerprint first, the second fingerprint in the middle, and the timestamp byte sequence last. The hash value of the concatenated byte stream is calculated using the SHA-256 algorithm as the current time dimension semantic fingerprint. The output length ranges from 256 to 512 bits, with 256 bits being the preferred value. The current time dimension semantic fingerprint simultaneously contains data semantic content information, original binary integrity information, and time dimension anchoring information. The fingerprint string length ranges from 64 to 128 characters, with 64 characters being the preferred value.
[0082] S334. The middleware layer generates a cross-source consistency descriptor, which is stored in structure form and contains six fields by default. The first field is the list of target data sources and access order dependencies, directly copied from the corresponding content in the semantic contract. The second field is the data source coverage set, recording the list of data source identifiers that actually obtained the data. The third field is the source identifier, recording the identity of each data source. The fourth field is the business consistency level, directly copied from the business consistency level value in the semantic contract. The fifth field is the current time dimension semantic fingerprint, filled with the fingerprint value generated in S333. The sixth field is the descriptor generation timestamp, filled with the current system physical timestamp in seconds, with a precision of seconds. The length of the byte stream after serialization of the cross-source consistency descriptor ranges from 512 bytes to 10 kilobytes, with a preferred value of 2 kilobytes. The serialization format is JSON or Protocol Buffers by default.
[0083] S34. Synchronous result return, asynchronous snapshot anchoring, and lifecycle management, specifically including:
[0084] S341. After obtaining the middleware standard binary snapshot format data converted by S313, the middleware layer calls the output formatting component in the heterogeneous data adaptation driver module to convert the snapshot format data into the standard structured data format expected by the caller. The conversion process is based on the response format type declared by the caller in the read request. The response format type includes JSON, XML, or Protocol Buffers; JSON is used by default if not declared. After the conversion is complete, the middleware layer synchronously returns the standard structured data to the caller through the current network connection as the response result of the current read request. After the synchronous return is complete, the middleware layer immediately records the return timestamp. This timestamp is generated by the middleware layer's internal logical clock, with millisecond precision, and the timestamp length ranges from 13 to 19 digits, with 13 digits being preferred. The return timestamp is associated with the request identifier and written to the response log buffer. The buffer refresh interval ranges from 1 second to 60 seconds, with 10 seconds being preferred. The return timestamp is only used for response latency monitoring and audit log recording and does not participate in subsequent snapshot mapping and lifecycle management logic.
[0085] S342. After completing the synchronous return operation, the middleware layer immediately submits the asynchronous task to the background task thread pool, or starts the asynchronous task in parallel with the synchronous return operation. The core number of threads in the asynchronous task thread pool ranges from 2 to 32, with a preferred value of 8. The maximum number of threads ranges from the core number of threads to 128, with a preferred value of 32, and the maximum number of threads is not less than the core number of threads. The thread idle lifespan ranges from 10 seconds to 600 seconds, with a preferred value of 60 seconds. The asynchronous task first reads the cross-source consistency descriptor and extracts the request identifier, the current time dimension semantic fingerprint, the data source coverage set, and the business consistency level. Then, using the request identifier as the key, a semantic fingerprint record is created in the distributed memory storage. This record contains eight fields by default: request identifier field, current time dimension semantic fingerprint field, snapshot expiration timestamp field, semantic fingerprint version chain pointer field, data source coverage set field, business consistency level field, record status field, and snapshot error correction check code field. The initial value of the record status field is active, and the status enumeration values include active, archived, and expired. The value of the semantic fingerprint version chain pointer field is obtained by querying the historical semantic fingerprint records of the same request identifier. If a historical record exists, the new version pointer is pointed to the storage address of the latest historical record, forming a fingerprint evolution chain that extends over time.
[0086] S343. After creating the semantic fingerprint record, the asynchronous task uses the first fingerprint as the content addressing identifier to query whether a physical data snapshot with the same content already exists in the snapshot storage pool. The snapshot storage pool adopts a content-addressable object storage architecture, with the number of storage buckets ranging from 1 to 64, preferably 8. During the query, the hash value of the first fingerprint is calculated and mapped to the corresponding storage bucket. Bucket number mapped to snapshot storage pool Calculate using the following formula: In the formula: The first fingerprint is the hash value of the structured semantic view; The target bucket number in the snapshot storage pool, with a value ranging from 0 to... ; This is a content-addressable hash function. The input is the first fingerprint string, and the output is an unsigned integer value. The total number of buckets in the snapshot storage pool, ranging from 1 to 64, with a preferred value of 8; the timeout for a single query, ranging from 50 milliseconds to 5000 milliseconds, with a preferred value of 200 milliseconds.
[0087] If the query hits, the existing physical data snapshot in the snapshot storage pool is reused, and its snapshot error correction check code is extracted from the persistent index record associated with the existing physical data snapshot. An index mapping is established from the current time dimension semantic fingerprint to the existing physical data snapshot, and the extracted snapshot error correction check code is written to the semantic fingerprint record corresponding to the current request identifier. If the query misses, the original result is serialized into a physical data snapshot in binary format and stored in the snapshot storage pool. The size of the serialized physical snapshot file ranges from 128 bytes to 50 megabytes, with a preferred value of 1 megabyte. A dual index structure is established using the current time dimension semantic fingerprint as the index key of the physical data snapshot and the first fingerprint as the content addressing identifier of the physical data snapshot. After the physical snapshot file is written to disk, the middleware layer calculates the snapshot error correction check code for the physical snapshot file and writes the check code back to the semantic fingerprint record corresponding to the request identifier in the persistent snapshot index library.
[0088] S344. The asynchronous task sets a snapshot expiration timestamp for semantic fingerprint records based on the data retention policy configured in the business settings. The data retention policy is pre-configured by the business side at the interface level, and the configuration dimensions include business type, data sensitivity, and compliance level.
[0089] The absolute expiration timestamp is calculated by adding the snapshot validity period of the business configuration to the descriptor generation timestamp, and then written into the snapshot expiration timestamp field of the semantic fingerprint record. The timestamp precision is in the second range. Calculate using the following formula: In the formula: Generate a timestamp for the descriptor, in seconds; The snapshot validity period configured for the business is in seconds, ranging from 60 to 2,592,000. The preferred value is 604,800 for billing, 600 for display, and 2,592,000 for auditing. This snapshot expiration timestamp serves as a driving signal, triggering periodic snapshot consistency maintenance operations to archive and reclaim expired semantic fingerprint records. After completing the lifecycle management configuration, the asynchronous task batch writes the metadata information of semantic fingerprint records and physical data snapshots to the pre-write log of the persistent snapshot index. The default log flushing strategy is triggered every 1000 records or every 5 seconds, whichever comes first; the preferred value is every 100 records or every 1 second, whichever comes first.
[0090] S345. The middleware layer starts a timed scheduling thread to periodically perform snapshot consistency maintenance operations. The scheduling period ranges from 60 seconds to 3600 seconds, with a preferred value of 300 seconds. During each execution, the semantic fingerprint records in the persistent snapshot index are scanned. For each semantic fingerprint record, the middleware layer extracts its snapshot expiration timestamp field value as the absolute expiration timestamp. and compared with the current system timestamp In comparison, if If the record is expired, it is determined that the record has expired; the semantic fingerprint records that have expired and whose record status is active are selected, and their record status is changed to archived.
[0091] Subsequently, based on the semantic fingerprint version chain pointer, the fingerprint evolution chain of the same request identifier is traced backward to collect the complete semantic fingerprint version chain for that request identifier. All expired semantic fingerprint records in the version chain are transferred in batches to the archive index area. The batch size ranges from 100 to 10,000 records, with a preferred value of 1,000 records. After the transfer is complete, the transferred records are deleted from the persistent snapshot index. For archived semantic fingerprint records, it is further checked whether the physical data snapshots they reference are still referenced by non-expired records in the persistent snapshot index or by other records in the archive index area. Physical data snapshots whose reference count drops to zero are physically deleted from the snapshot storage pool.
[0092] Example 2
[0093] Please see the appendix Figure 3 This provides a deterministic idempotent read system across heterogeneous data sources, used to implement deterministic idempotent read methods across heterogeneous data sources, including:
[0094] The request interception and semantic contract anchoring module is used by the middleware layer to intercept read requests, extract or generate request identifiers, parse the data source access intent and business consistency level; generate read semantic contracts, record the logical timestamp when the read request is intercepted, and encapsulate the read semantic contracts and logical timestamps into a deterministic read task.
[0095] The ladder-type idempotency verification module communicates with the request interception and semantic contract anchoring module. It is used to receive deterministic read tasks and perform ladder-type idempotency verification in distributed memory storage and persistent snapshot index. It performs initial screening for existence and pre-check for contract consistency using a Bloom filter. If it fails, it performs snapshot error correction check code verification. If it hits, it loads and returns; if it misses, it generates a penetration instruction.
[0096] The heterogeneous source penetration and snapshot anchoring module communicates with the ladder-type idempotency verification module to receive penetration instructions. It initiates requests to downstream applications through the heterogeneous data adaptation driver module configured in the middleware layer, obtains the original results, converts them into the middleware standard binary snapshot format, and binds them to logical timestamps after semantic normalization. It calculates hierarchical content semantic fingerprints to generate cross-source consistency descriptors, returns results in parallel, and asynchronously establishes request identifiers to content snapshot mappings, and performs lifecycle management according to data retention strategies.
[0097] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention is limited to these examples; within the framework of the invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of the different aspects of the invention as described above, which are not provided in detail for the sake of brevity.
[0098] This invention is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this specification. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A deterministic idempotent reading method across heterogeneous data sources, characterized in that, include: The middleware layer intercepts read requests, extracts or generates request identifiers, and parses the data source access intent and business consistency level. Generate a read semantic contract, record the logical timestamp when the read request is intercepted, and encapsulate the read semantic contract and logical timestamp into a deterministic read task; A ladder idempotency check is performed on the distributed memory storage and persistent snapshot index, followed by an initial existence screening and contract consistency pre-check using a Bloom filter. If the check fails, a snapshot error correction check code will be performed. If the check code is hit, the data will be loaded and returned. If the check fails, a penetration command will be generated. The heterogeneous data adaptation driver module configured in the middleware layer sends a request to the downstream to obtain the original result and convert it into the middleware standard binary snapshot format. After semantic normalization, it is bound to the logical timestamp. The algorithm calculates hierarchical content semantic fingerprints to generate cross-source consistent descriptors, returns results in parallel, asynchronously establishes request identifiers to content snapshot mappings, and performs lifecycle management according to data retention strategies.
2. The deterministic idempotent reading method across heterogeneous data sources according to claim 1, characterized in that, The middleware layer intercepts read requests, extracts or generates request identifiers, and parses the data source access intent and business consistency level, specifically including: Extract the explicitly declared target data source set from the read request; if not explicitly declared, infer the target data source set according to the routing path in the Uniform Resource Locator, the service interface identifier, and the data table or index name in the query condition parameters, following preset data source routing rules. The explicitly declared or inferred target data source set is then identified as the data source access intent. The target data source set inference employs a priority scoring mechanism, with candidate data source matching scores... for: In the formula: Match weights to the routing paths. For route path matching flags, Match weights to service interface identifiers. Match the flag to the service interface identifier. Match weights to the data table or index name. A flag that matches the name of a data table or index; When the total score threshold is reached or exceeded, it will be included in the inference results; Extract the business consistency level parameter from the read request; if not specified, determine the default business consistency level according to the business type and the preset consistency level mapping table. The business consistency level includes strong consistency level, session consistency level and eventual consistency level.
3. The deterministic idempotent reading method across heterogeneous data sources according to claim 2, characterized in that, The process of generating a read semantic contract involves recording a logical timestamp when the read request is intercepted, and encapsulating the read semantic contract and the logical timestamp into a deterministic read task. Specifically, this includes: The globally unique request identifier or default request identifier, the data source access intent, and the business consistency level are bound together to generate a read semantic contract. This read semantic contract includes a list of target data sources, the access order dependency relationships for each target data source, and the business consistency level; wherein the access order dependency relationships are based on edge weights. The judgment is as follows: In the formula: This refers to the access order dependency type; These are the weights of the edges in the directed graph. The semantic contract, logical timestamp, and original query condition parameters are assembled to generate a deterministic reading task.
4. The deterministic idempotent reading method across heterogeneous data sources according to claim 1, characterized in that, The step-wise idempotency check performed on the distributed memory storage and persistent snapshot index library, after initial existence screening by Bloom filter and contract consistency pre-check, specifically includes: Set the data source access intent and business consistency level in the read semantic contract as the verification benchmark; Using the globally unique request identifier or default request identifier as the query key, a Bloom filter is used for initial screening in the distributed memory storage, where the current false positive rate of the Bloom filter is... for: In the formula: The length of the Bloom filter bit array. The number of hash functions. This represents the number of elements currently inserted into the Bloom filter; if no match is found, the query proceeds to the persistent snapshot index; if a match is found, the pre-stored response body and its associated historical data source coverage set and historical business consistency level are retrieved. The historical data source coverage set is compared with the data source access intent to determine whether it completely covers the target data source list; the historical business consistency level is compared with the business consistency level to determine whether they are the same; if both pass, a pre-stored response body is returned; if either fails, it is determined that a match has not been found and the query is transferred to the persistent snapshot index library.
5. The deterministic idempotent reading method across heterogeneous data sources according to claim 4, characterized in that, If the check fails, a snapshot error correction check code is performed. If the check code is hit, it is loaded and returned; otherwise, a penetration instruction is generated, specifically including: When the L1 layer is not hit, the persistent snapshot index is queried using the globally unique request identifier or the default request identifier as the query key to obtain the persistent index record. The persistent index record includes semantic fingerprint, snapshot storage location, snapshot error correction check code, historical data source coverage set and historical business consistency level. Integrity verification is performed on the corresponding physical snapshot file based on the snapshot error correction check code; if it fails, it is determined that the L2 layer has not been hit; if it passes, the historical data source coverage set and historical business consistency level are extracted. The historical data source coverage set is compared with the data source access intent to determine whether the target data source list is fully covered; the historical business consistency level is compared with the business consistency level to determine whether they are the same; if both pass, the physical snapshot file is loaded and restored to structured data before returning; if any fails, it is determined that the L2 layer is not hit. If neither L1 nor L2 layer is hit, a penetration instruction is generated. The penetration instruction carries the globally unique request identifier or default request identifier, the semantic contract for reading, and the original query condition parameters.
6. The deterministic idempotent reading method across heterogeneous data sources according to claim 3, characterized in that, The heterogeneous data adaptation driver module configured through the middleware layer sends a request to the downstream to obtain the original result and convert it into the middleware standard binary snapshot format, specifically including: Based on the reading semantic contract in the penetration instruction, extract the target data source list and access order dependency relationship, which includes parallel dependency relationship and serial dependency relationship; Based on the target data source list and access order dependency, real-time read requests are sent to downstream heterogeneous data sources through the corresponding adaptation driver; data sources corresponding to parallel dependencies are sent concurrently, and data sources corresponding to serial dependencies are sent sequentially. Receive the raw results and source response metadata returned by each target data source, wherein the source response metadata includes the source identifier; convert each raw result into the middleware standard binary snapshot format and bind it with the corresponding source identifier to generate a standardized snapshot set with data source lineage markers.
7. The deterministic idempotent reading method across heterogeneous data sources according to claim 6, characterized in that, The binding of semantically normalized data with logical timestamps specifically includes: Data in the middleware standard binary snapshot format is extracted from the standardized snapshot set, and grouped and merged according to the data source access intent to generate a multi-source dataset to be normalized; wherein a hash bucketing strategy is used during grouping and merging, and the data record belongs to the bucket index of rec. for: In the formula: This is the identifier of the data source associated with this data record; rec is a single data record in the multi-source dataset to be normalized; This is a bucketed index, with values ranging from 0 to... ; A hash function that takes a data source identifier as input; This represents the total number of hash buckets. Semantic normalization processing is performed on the data records in the multi-source dataset. The semantic normalization processing includes at least one of the following: rearranging fields according to a preset standard field order; uniformly filling null values according to a preset null value filling rule; and converting data types according to a preset data type mapping rule. Extract logical timestamps from the deterministic reading task, bind the logical timestamps to the normalized data, and generate a semantically normalized data structure with time dimension markers.
8. The deterministic idempotent reading method across heterogeneous data sources according to claim 7, characterized in that, The calculation of hierarchical content semantic fingerprints to generate cross-source consistent descriptors specifically includes: Normalized data is extracted from the semantically normalized data structure as a structured semantic view, and the original binary byte stream is extracted from the standardized snapshot set as a binary view; The hash value of the structured semantic view is calculated as the first fingerprint, and the hash value of the binary view is calculated as the second fingerprint; The logical timestamp, the first fingerprint, and the second fingerprint are fused to generate a semantic fingerprint for the current time dimension. Generate a cross-source consistency descriptor, which records the target data source list and access order dependency, data source coverage set and source identifier, business consistency level and current time dimension semantic fingerprint.
9. The deterministic idempotent reading method across heterogeneous data sources according to claim 8, characterized in that, The parallel return results and asynchronous establishment request identifiers are mapped to content snapshots, and lifecycle management is performed according to the data retention strategy, specifically including: The original results are converted into a standard structured data format and then synchronously returned to the caller, with a timestamp recorded. An asynchronous task is initiated to establish a mapping from request identifiers to content snapshots in the distributed memory storage and persistent snapshot index based on cross-source consistency descriptors. A semantic fingerprint record is created using the request identifier as the key. The semantic fingerprint record includes the request identifier, the current time dimension semantic fingerprint, the snapshot expiration timestamp, the semantic fingerprint version chain pointer, the data source coverage set, the business consistency level, the record status, and the snapshot error correction check code. The semantic fingerprint version chain pointer points to the storage location of the previous time dimension semantic fingerprint of the same request identifier. The system queries whether a physical data snapshot with the same content already exists in the snapshot storage pool using the first fingerprint. During the query, the hash value of the first fingerprint is calculated and mapped to the corresponding storage bucket. Bucket number mapped to snapshot storage pool for: In the formula: The target bucket number in the snapshot storage pool, with a value ranging from 0 to... ; Use a content-addressable hash function, with the first fingerprint string as input; The total number of buckets in the snapshot storage pool; if it exists, the existing physical data snapshot is reused, an index mapping from the current time dimension semantic fingerprint to the physical data snapshot is established, and the snapshot error correction check code associated with the existing physical data snapshot is written into the semantic fingerprint record; if it does not exist, the original result is serialized into a physical data snapshot and stored in the snapshot storage pool, and a dual index structure is established with the current time dimension semantic fingerprint as the index key and the first fingerprint as the content addressing identifier. According to the data retention policy configured in the business settings, the snapshot expiration timestamp field value is written to the semantic fingerprint record, where the absolute expiration timestamp is... for: In the formula: Generate timestamps for descriptors. The validity period of the snapshot configured for the business; Periodically scan the semantic fingerprint records in the persistent snapshot index library, transfer expired semantic fingerprint records to the archive index area and delete them; check whether the physical data snapshots in the snapshot storage pool are still being referenced by valid records, and delete unreferenced physical data snapshots.
10. A deterministic idempotent reading system across heterogeneous data sources, characterized in that, A deterministic idempotent reading method for cross-heterogeneous data sources as described in any one of claims 1-9, comprising: The request interception and semantic contract anchoring module is used by the middleware layer to intercept read requests, extract or generate request identifiers, parse the data source access intent and business consistency level; generate read semantic contracts, record the logical timestamp when the read request is intercepted, and encapsulate the read semantic contracts and logical timestamps into a deterministic read task. The ladder-type idempotency verification module is communicatively connected to the request interception and semantic contract anchoring module. It is used to receive the deterministic reading task, perform ladder-type idempotency verification in distributed memory storage and persistent snapshot index library, and perform initial screening for existence and contract consistency pre-check by Bloom filter. If it fails, it performs snapshot error correction check code verification. If it hits, it loads and returns. If it misses, it generates a penetration instruction. The heterogeneous source penetration and snapshot anchoring module is communicatively connected to the ladder-type idempotent verification module. It is used to receive the penetration command, initiate a request to the downstream through the heterogeneous data adaptation driver module configured in the middleware layer, obtain the original result and convert it into the middleware standard binary snapshot format, bind it to the logical timestamp after semantic normalization, calculate the hierarchical content semantic fingerprint to generate a cross-source consistency descriptor, return the result in parallel and asynchronously establish the request identifier to the content snapshot mapping, and perform lifecycle management according to the data retention strategy.
