Semantic identification blockchain retrieval method and system for large-scale distributed data

Abstract

This invention belongs to the field of blockchain technology, specifically relating to a semantic identifier blockchain retrieval method and system for large-scale distributed data. This invention designs a unified semantic identifier model, mapping data records from different sources and in different formats to multi-level scalable semantic paths. It constructs a MIR tree forest, integrating the organization of Trie and the verifiability of Merkle, with each scenario as the root. Only the root hash of each scenario or its aggregate root is maintained in the block header, thereby achieving overall constraint on the state of large-scale off-chain semantic indexes while maintaining lightweight on-chain storage overhead. Simultaneously, this invention designs corresponding index traversal and verification object construction mechanisms around various semantic retrieval modes, enabling clients to verify the correctness of target records with only limited path and node hash information. This provides a new technical path for unified semantic management and efficient verifiable retrieval of multi-source heterogeneous data in the HSB environment.

Description

Technical Field

[0001] This invention belongs to the field of blockchain technology, specifically relating to a semantic identifier blockchain retrieval method and system for large-scale distributed data. Background Technology

[0002] With the deepening application of blockchain technology in fields such as distributed intelligent computing, intelligent manufacturing, and the Internet of Things, blockchain systems are gradually evolving from single business ledgers centered on transaction records to data infrastructure supporting multi-party collaboration. In a typical HybRid-StoRage Blockchain (HSB) architecture, large volumes of business data are hosted off-chain in relational databases, document systems, object storage, or distributed file systems. The blockchain is primarily responsible for storing summary information, access credentials, or index metadata related to the off-chain data, providing a unified, trusted anchor and verifiable execution foundation for multi-party collaboration in heterogeneous environments. As the number of participants and business scenarios continues to increase, the system needs to simultaneously manage heterogeneous data from multiple sources, including experimental platforms, sensing devices, model repositories, and business systems. These data exhibit significant differences in format structure, field semantics, and hierarchical organization. Traditional on-chain record methods based on account-transaction or simple key-value pairs are insufficient to characterize semantic relationships and cannot meet the query requirements for cross-source data aggregation by keywords and semantics.

[0003] Within the existing HSB system, two main technical approaches have emerged for collaborative on-chain and off-chain queries. One approach employs an "off-chain retrieval, on-chain verification" model. This involves jointly maintaining an Authentication Data Structure (ADS) on and off-chain. The off-chain service provider returns the query results and verification objects, while the client verifies the correctness and completeness of the results using the on-chain root digest. However, the existing "off-chain retrieval, on-chain verification" model generally assumes a single cloud server and a known precise access endpoint, making it difficult to adapt to distributed multi-server query environments. The other approach attempts to build an "on-chain index-guided retrieval" mechanism. This extracts keywords or semantic features from off-chain data and constructs a semantic index structure on-chain to support multi-keyword queries, range queries, and cross-chain queries. However, these indexes are mostly customized around specific application scenarios and data types, lacking a unified semantic identifier model that can be reused across scenarios. This makes it difficult to map data from different sources to a consistent semantic hierarchy. Furthermore, when supporting complex semantic conditions such as prefix queries and multi-keyword combinations, existing index structures often have coarse-grained verification object designs, making it difficult to provide fine-grained, formally verifiable guarantees for the correctness of result sets within a specific semantic range while ensuring controllable on-chain overhead. In summary, how to construct a unified semantic identifier space for multi-source heterogeneous data under the HSB architecture and maintain a verifiable semantic index structure on-chain at low cost is a key problem that current technology has not yet effectively solved.

[0004] Therefore, it is very important to design an on-chain semantic indexing scheme that has both unified semantic identification capabilities and supports efficient and verifiable retrieval. Summary of the Invention

[0005] This invention aims to overcome the problem in the prior art of how to construct a unified semantic identifier space for multi-source heterogeneous data under the HSB architecture and maintain a verifiable semantic index structure on the chain at low cost. It provides a semantic identifier blockchain retrieval method and system for large-scale distributed data that can have both unified semantic identifier capabilities and support efficient and verifiable retrieval under a hybrid storage blockchain architecture.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] A semantic-identified blockchain retrieval method for large-scale distributed data includes the following steps:

[0008] S1, Semantic identifier generation:

[0009] The off-chain server performs semantic parsing on the original heterogeneous data, generates a unique semantic identifier path P for each piece of data according to predefined semantic rules, and calculates the data digest meta_hash and off-chain data reference DataRef bound to the path.

[0010] S2, MIR index construction:

[0011] The semantic identifier path P obtained in step S1 is inserted into the MIR index tree of the corresponding scenario. The semantic identifiers of different scenarios are structured and organized through the MIR index forest formed by multiple MIR trees. At the same time, a corresponding scenario root hash (MIR Root hash) is generated for each MIR tree.

[0012] S3, Index Commitment and Maintenance:

[0013] The MIR Root hashes from various scenarios are aggregated into a global index root MIR Forest hash in a fixed order and written into the blockchain block header as an on-chain semantic state commitment. At the same time, the complete MIR index structure is maintained locally on the chain node in a key-value storage manner, and the hash commitment is incrementally updated in a bottom-up manner during subsequent insertions or updates.

[0014] S4 can verify semantic queries:

[0015] The client initiates a query request, and the off-chain node performs semantic matching in the MIR index forest and returns the query results and the corresponding lightweight verification object (VO). The client uses the VO and the MIR Root hash or MIR Forest hash in the block header to reconstruct and verify the results, thereby verifying the correctness and completeness of the query results.

[0016] Preferably, in step S1, the semantic identification path P is as follows: ;

[0017] Here, Scene represents a macro-level application scenario; Type represents a semantic type of data; Subtype* represents an extensible sub-level used for recursive expansion to support multi-level semantic structures; Entity is used to ensure the global uniqueness of the identifier.

[0018] Subtype* employs a variable-length semantic sub-path mechanism, which can be extended according to actual needs;

[0019] Entities are generated using a short hash method based on scenario, semantic path, and record content.

[0020] Preferably, in step S2, the MIR tree contains the following three types of nodes:

[0021] Semantic segment node : Represents an intermediate node with multiple sub-path branches. The node content is a single semantic token used to distinguish different semantic branches at the same level;

[0022] Compressed Nodes Used for path compression, it represents a node formed by merging consecutive unique sub-paths, and its content is a combination of multiple consecutive token sequences;

[0023] leaf node : Represents the endpoint of a complete semantic path, corresponding to a unique entity identifier and associated data, as defined below:

[0024] ;

[0025] For each Scene, the system maintains the root hash of the scene's MIR tree and writes it into the block header as a cryptographic commitment to all current records in that scene.

[0026] Preferably, step S3 includes the following steps:

[0027] S31, Given a new semantic identifier path to be inserted. : ;

[0028] During insertion, the tree structure is traversed level by level. The specific steps are as follows:

[0029] S311, Initial Positioning:

[0030] From the root set of the MIR Forest, search for the corresponding MIR tree by the root node Scene; if it does not exist, create a new tree.

[0031] S312, step-by-step insertion:

[0032] For each in the path ,in First, determine the corresponding node type: if the current node is a semantic segment node, check if it exists in the child node set. If the corresponding branch exists, it will continue down to the corresponding child node; otherwise, a new semantic segment node or leaf node will be created. If the current node is a compressed node, then the compressed node will be set to... The remaining path to be inserted is Calculate the length of the corresponding common prefix. : ;

[0033] The sequence length function is used to return the number of tokens contained in the common prefix token sequence; CommonPrefix(C,R) represents the longest common prefix token sequence of sequence C and sequence R, that is, it compares each token starting from the first token, stops when the first unequal token is found, and returns the sequence composed of all equal tokens before that.

[0034] like This indicates that the compressed node path is completely matched, and the remaining paths will continue to be processed. ;like If the common prefix of the split compressed node is used to form a new semantic segment node, then different suffixes will create new compressed nodes or leaf nodes respectively.

[0035] Preferably, step S3 further includes the following steps:

[0036] S32, the system starts from the leaf node that has changed, recalculates the corresponding node hash, and updates it layer by layer upwards along the parent node relationship until a new scene root hash MIR Root hash is obtained;

[0037] S33, the root hashes of each scenario are then sequentially aggregated into a cross-scenario global hash, MIR Forest hash, and written into the new block header.

[0038] Preferably, in step S4, the query request includes an exact query, a prefix query, or a multi-keyword query; wherein an exact query includes the following process:

[0039] S41, Define an exact query given a complete semantic path : ;

[0040] The corresponding semantic target is a uniquely determined MIR tree Zhongyu corresponding leaf nodes and return the corresponding leaf node associated with it. With metadata summary ;

[0041] Off-chain full nodes in the local index The semantic segments contained within are matched layer by layer starting from the root node. After locating the unique leaf node, the matching continues along the path... The path to the root node collects Merkle authentication information and constructs a membership proof path;

[0042] For each current node at each level along the path, the hash values ​​of all its sibling subtrees, and, if necessary, the aggregate hash of the current node covering the leaf set, are written together into the VO, denoted as . ;

[0043] The client receives the quadruple. Then, first utilize Recalculate the leaf node hash locally Then, Merkle synthesis is performed from bottom to top according to the sibling hashes provided in the VO to obtain the reconstructed root hash. ;

[0044] like and If the value matches the record value in the block header, it verifies that the corresponding record is indeed contained in the semantic index tree promised by the current block and has not been tampered with during transmission and storage.

[0045] Preferably, in step S4, the prefix query includes the following process:

[0046] S42, Prefix query given a certain semantic prefix : ;

[0047] The off-chain node first locates the internal node corresponding to the prefix node in the MIR tree. Then, enumerate all the matched leaf nodes in the corresponding subtree to obtain the result set. ;

[0048] For branches within the prefix, select a set of subtree boundary nodes that will not be expanded further, based on the tree's branching pattern; for subtrees outside the prefix, only return the root hash of the corresponding subtree without expanding the leaves; thus, the VO of a prefix query can be abstracted as... : ;

[0049] in, The set of leading nodes that cover the entire query subtree;

[0050] During client-side verification, on the one hand, all returned paths are checked leaf by leaf to ensure they all have a given prefix. Beginning; then utilizing The content hash of the leaf node, along with the front boundary and sibling hash provided in the VO, are reconstructed by performing Merkle synthesis from bottom to top. and the block header record Compare; if they match, the verification result does indeed correspond to the scenario index tree of the on-chain commitment.

[0051] Preferably, the multi-keyword query includes the following process:

[0052] S43, Set the set of query keywords as ; Filter to meet The set of leaf nodes is ;

[0053] The system first normalizes the input keywords and maps them to a set of matching predicates on the path. Then, it scans the leaf nodes in the MIR tree and selects all leaves that satisfy the predicate constraints to form a hit set.

[0054] In terms of verification, for each hit leaf node The VO should provide the corresponding complete path. , And the authentication path from the leaf to the scene root; for all missing subtrees, a global commitment is made through the aggregate hash of the leading-edge nodes, specifically, verifying the object. Defined as: ;

[0055] in This represents the frontier commitment node given for the subtree that "does not contain all keywords". right Each element in the table gives its destination. The sibling hash sequence (and direction information) is used to reconstruct the root hash on the client side.

[0056] The client first checks each returned path locally to see if it satisfies the keyword set constraint, that is, for each leaf node... verify Subsequently, the member paths and boundary sibling hashes provided in the VO were used to reconstruct the data. and the block header record Compare; if they match, then the returned hit leaf node and its corresponding associated path field and data reference have not been tampered with, and all come from the on-chain promised scenario index tree.

[0057] This invention also provides a semantic-identified blockchain retrieval system for large-scale distributed data, including:

[0058] The semantic identifier generation module enables the off-chain server to perform semantic parsing on the original heterogeneous data, generate a unique semantic identifier path P for each piece of data according to predefined semantic rules, and calculate the data digest meta_hash and off-chain data reference DataRef bound to the path.

[0059] The MIR index building module is used to insert the obtained semantic identifier path P into the corresponding scenario's MIR index tree. The MIR index forest formed by multiple MIR trees is used to organize the semantic identifiers of different scenarios in a structured way, and at the same time, a corresponding scenario root hash (MIR Root hash) is generated for each MIR tree.

[0060] The index commitment and maintenance module is used to aggregate the MIR Root hashes of various scenarios into a global index root MIR Forest hash in a fixed order and write it into the blockchain block header as an on-chain semantic state commitment; at the same time, it maintains the complete MIR index structure locally on the chain node in a key-value storage manner, and incrementally updates the hash commitment in a bottom-up manner during subsequent insertions or updates.

[0061] The verifiable semantic query module enables clients to initiate query requests. Off-chain nodes perform semantic matching in the MIR index forest and return the query results and the corresponding lightweight verification object (VO). Clients use the VO and the MIRRoot hash or MIR Forest hash in the block header to reconstruct and verify the results, thereby verifying the correctness and completeness of the query results.

[0062] Compared with the prior art, the beneficial effects of this invention are: (1) It proposes an encodeable semantic identifier system for heterogeneous data to realize on-chain semantic retrieval: This invention is the first to directly encode semantic information into data identifiers to form semantic identifiers, and embeds scenarios, types, subcategories and keywords hierarchically.<Scene / Type / Subtype* / Entity> Semantic paths enable each entity in the blockchain to not only have a unique identification capability, but also carry a parsable semantic structure; this identification does not need to rely on a specific feature extraction model of the data content, can be reused across multiple heterogeneous data sources such as documents, object storage, logs and structured tables, and can directly support semantic precise positioning, prefix aggregation query and keyword combination retrieval on the chain, breaking through the limitation of traditional blockchains that can only traverse the entire chain or locate single data points by hash; (2) Design a MIR “index forest” that is naturally divided according to semantic scenarios, supporting low-cost on-chain anchoring and scalable maintenance: This invention establishes independent MIR index trees with different semantic scenarios as roots, forming a pluggable and scalable index forest structure. Each tree maintains its semantic level and entity nodes respectively, and only writes the root digest of each tree into the blockchain header, without having to put a large number of index structures on the chain. This design avoids the high maintenance costs caused by solidifying complex semantic structures on the chain, and can naturally expand new semantic branches and subcategories as the business evolves, realizing the reuse and on-demand growth of cross-domain semantic systems, and keeping the blockchain lightweight storage and constant-level update overhead; (3) Proposes a lightweight verification mechanism oriented towards semantic scope to realize verifiable semantic queries: In view of the problem that existing solutions can only verify "whether the data exists" but cannot verify "whether the semantic scope is complete", this invention introduces a verifiable path and frontier node mechanism in the node structure of the MIR forest, so that the client does not need to download the complete index, but can verify whether the query result is real, has not been tampered with, and has not missed entity records in the same semantic scope with only a few sibling hashes and aggregation nodes; This mechanism does not require large-scale zero-knowledge proofs or index on-chain, and can ensure the integrity of prefix queries, multi-keyword queries and semantic aggregation queries. Attached Figure Description

[0063] Figure 1 This is a diagram of an index architecture for the MIR tree in this invention; Figure 2 This is a schematic diagram illustrating a comparison of query performance in a single scenario (computer device) according to an embodiment of the present invention. Figure 3 A schematic diagram illustrating the comparison of average verification time in a single scenario provided by an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the comparison of VO size in a single scenario provided by an embodiment of the present invention; Figure 5This is a schematic diagram illustrating the average query time variation of various query methods under multiple scenarios as the number of scenarios changes, provided in an embodiment of the present invention. Figure 6 A schematic diagram illustrating the comparison of average verification time across multiple scenarios provided in an embodiment of the present invention; Figure 7 This is a schematic diagram illustrating a verification size comparison under multiple scenarios provided in an embodiment of the present invention. Detailed Implementation

[0064] To more clearly illustrate the embodiments of the present invention, specific implementation methods will be described below with reference to the accompanying drawings. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0065] This invention provides a semantic identifier blockchain retrieval method for large-scale distributed data, comprising the following steps:

[0066] 1. Semantic tag generation:

[0067] The off-chain server performs semantic parsing on the original heterogeneous data, generates a unique semantic identifier path P for each piece of data according to predefined semantic rules, and calculates the data digest meta_hash and off-chain data reference DataRef bound to the path.

[0068] 2. MIR index construction:

[0069] The semantic identifier path P obtained in step S1 is inserted into the MIR index tree of the corresponding scenario. The semantic identifiers of different scenarios are structured and organized through the MIR index forest formed by multiple MIR trees. At the same time, a corresponding scenario root hash (MIR Root hash) is generated for each MIR tree.

[0070] 3. Index Commitment and Maintenance:

[0071] The MIR Root hashes from various scenarios are aggregated into a global index root MIR Forest hash in a fixed order and written into the blockchain block header as an on-chain semantic state commitment. At the same time, the complete MIR index structure is maintained locally on the chain node in a key-value storage manner, and the hash commitment is incrementally updated in a bottom-up manner during subsequent insertions or updates.

[0072] 4. Verifiable semantic queries:

[0073] The client initiates a query request, and the off-chain node performs semantic matching in the MIR index forest and returns the query results and the corresponding lightweight verification object (VO). The client uses the VO and the MIR Root hash or MIR Forest hash in the block header to reconstruct and verify the results, thereby verifying the correctness and completeness of the query results.

[0074] This invention employs a lightweight on-chain index and off-chain raw data storage approach. The blockchain is responsible for recording semantic identifier mapping relationships and global index root digests, while off-chain servers are responsible for storing heterogeneous resources such as documents, object files, log data, and model parameters.

[0075] In this system, the four core technological processes are completed collaboratively by different system components, and their corresponding relationships are as follows:

[0076] The semantic tagging method is mainly executed by off-chain servers, which are responsible for semantic parsing and structured encoding of the original heterogeneous data to generate a unified format. The path identifier is used to calculate the corresponding data digest meta_hash and the off-chain data reference DataRef;

[0077] The construction and storage of the MIR index forest are completed collaboratively by the blockchain full node and its local key-value database. The complete MIR tree structure is stored in the local LevelDB-like storage of the chain node, while the root hash MIR Roothash corresponding to each scenario and the aggregated global index root MIR Forest hash are written into the blockchain block header as a trusted commitment to the current semantic index state.

[0078] The index maintenance mechanism is executed by the local index module of the blockchain node. When the semantic identifier path is inserted, split or updated, the MIR tree is incrementally adjusted, and the root digest on the chain is maintained synchronously through the bottom-up hash update mechanism, so that the index structure can continue to evolve without destroying historical blocks.

[0079] The construction and verification of the lightweight verification path (VO) are completed collaboratively by off-chain full nodes and clients. Off-chain nodes generate the minimum necessary Merkle proof object in the MIR index forest according to the query request. The client uses the root hash recorded in the blockchain to reconstruct and verify the VO, thereby ensuring the verifiability of the query results.

[0080] Within the overall framework described above, the specific functions of each system component are as follows:

[0081] 1. Blockchain component

[0082] A block consists of a block header and a block body. The block body records multiple index transactions, each represented as a four-tuple: (scene, path, data_Ref, meta_hash).

[0083] in:

[0084] "Scene" refers to a business or data scenario.

[0085] path is encoded according to semantic rules.<Scene / Type / Subtype* / Entity> path;

[0086] data_Ref points to the location of the data in the chain, which can be a file path, an object storage key, or a data digest;

[0087] meta_hash records a hash digest of off-chain data content or metadata, used for consistency verification.

[0088] The block header records a two-level index summary:

[0089] Maintain a MIR index tree for each scene, and denot its root hash as the MIR Root hash;

[0090] The outer Merkle tree is constructed by taking all MIR Root hashes in a fixed order, and its root hash, MIR Foresthash, is written into the block header for global semantic state proof.

[0091] Only indexed transactions and root summaries are stored on-chain; the complete MIR structure is stored in a LevelDB-like key-value store local to the chain node.

[0092] 2. Off-chain server component

[0093] Off-chain storage is used to store large volumes of data resources in various formats, including material files, document data, model weights, and equipment logs. The system has a built-in semantic engine responsible for extracting key fields from the raw data and automatically generating...<Scene / Type / Subtype* / Entity> Identify and establish the Entity ↔ DataRef mapping relationship. Simultaneously calculate the data meta_hash and include lightweight fields (<scene, path, data_Ref, meta_hash> The index transaction is written to the blockchain.

[0094] 3. Client-side component

[0095] The client is used to initiate semantic queries. The query node performs semantic retrieval through the local MIR structure and returns a VO. The client uses the VO and the on-chain root digest to complete a verifiable query.

[0096] Specifically, the specific process of the semantic identifier blockchain retrieval method for large-scale distributed data provided by this invention is as follows:

[0097] (a) Semantic tagging methods

[0098] To achieve unified expression and structured organization of multi-source heterogeneous data, this invention designs a four-layer semantic tagging structure. Keywords are encoded into semantic tags, and these semantic tags are then combined to form an index tree. Data from different scenarios share a single index tree, thus creating an index forest. The basic form of the tagging is as follows:

[0099] <Scene / Type / Subtype* / Entity> ;

[0100] The meaning of each layer is as follows:

[0101] Scene: Represents a macro-level application scenario (such as supplychain, IoT, intelligent-computing);

[0102] Type: Represents a semantic type of data (such as fRuit, sensoR, model);

[0103] Subtype*: Requires extensible sub-levels (e.g., apple, tempeRatuRe, cnn), which can be recursively expanded to support multi-level semantic structures;

[0104] Entity: Ensures the global uniqueness of the identifier.

[0105] Subtype* employs a variable-length semantic subpath mechanism, which can be extended according to actual needs, such as:

[0106] intelligent-computing / model / cnn / Resnet50 / v1.2 / Entity.

[0107] Entities are generated using a short hash method based on scene, semantic path, and record content. The system first serializes the entire original record into ordered JSON and calculates its SHA-256 hash to obtain the meta_hash. Then, it concatenates the scene, the standardized semantic path excluding the Entity segment, and the meta_hash, uses this as input to calculate the SHA-256 hash again, and extracts the first 10 hexadecimal characters as the entity identifier. ;

[0108] (ii) MIR Forest Index Structure

[0109] On-chain storage, serving as a trusted anchoring layer in a semantic retrieval system, is primarily responsible for the structured organization of semantic identifiers, the verifiable maintenance of retrieval paths, and data integrity verification. This paper establishes an index structure, MIR Forest, within the block body to address the issues of hierarchical management and efficient indexing of semantic identifiers. The MIR Tree in MIR Forest is rooted at the Scene node, expanding layer by layer according to the hierarchical structure of semantic identifiers, and integrating the path organization of Trie trees with the verifiability of Merkle trees. Each tree is independent and loosely coupled, facilitating distributed expansion and parallel processing. Nodes in the MIR are divided by "token," with each token inserted into the MIR as a complete logical key. Furthermore, each node calculates and stores the combined hash values ​​of its child nodes, forming a Merkle hash link from the leaf to the root. Merkle nodes do not affect the indexing function but provide verifiability to the entire tree.

[0110] like Figure 1 As shown, to achieve flexible indexing and efficient verification, the MIR tree contains the following three types of nodes:

[0111] Semantic segment node : Represents an intermediate node with multiple sub-path branches. The node content is a single semantic token used to distinguish different semantic branches at the same level;

[0112] Compressed Nodes Used for path compression, it represents a node formed by merging consecutive unique sub-paths, with the content being a combination of multiple consecutive token sequences; for semantic path indexing, if there are no branches between adjacent levels, they can be merged into a compressed node, with multiple token fragments stored on the edge as labels to achieve path compression.

[0113] leaf node : Represents the endpoint of a complete semantic path, corresponding to a unique entity identifier and associated data, as defined below: ;

[0114] DataRef is an off-chain data reference (such as metadata hash, storage address, etc.) corresponding to the entity.

[0115] For each Scene, the system maintains the root hash of the scene's MIR Tree, the MIR Root hash, and writes it into the block header as a cryptographic commitment for all current records in that scene. To manage multiple scene roots in a unified manner, the MIR Root hash is further aggregated into the MIR Forest hash.

[0116] (III) Index Maintenance Methods

[0117] The MIR forest index of this invention is not directly stored on the blockchain. Instead, only the root digest is recorded on the chain, while the complete structure is stored as key-value pairs in a LevelDB-like storage database local to each node. Each node (semantic segment node, compressed node, or leaf node) is mapped to an independent key-value entry. The key uses the encoded form of its semantic segment or combined semantic segments, and the value contains references to the node's child nodes, the aggregate hash value, and the node type. Based on this storage method, the index can be incrementally modified locally without repeated on-chain writes, thus not affecting the validity of historical blocks and conforming to the design principles of lightweight blockchains.

[0118] Given a new semantic identifier path to be inserted: ;

[0119] During insertion, the tree structure is traversed level by level. The specific algorithm steps are as follows:

[0120] Step 1: Initial Location. From the root set of the MIR Forest, search for the corresponding MIR Tree by the root node Scene. If it does not exist, create a new tree.

[0121] Step 2: Submerged insertion. For each... (in First, determine the node type: if the current node is a semantic segment node, check if it exists in the set of child nodes. If the corresponding branch exists, it will continue down to the corresponding child node; otherwise, a new semantic segment node or leaf node will be created. If the current node is a compressed node, then it is assumed that the compressed node is... The remaining path to be inserted is The length of its common prefix needs to be calculated. :

[0122] like This indicates that the compressed node path is completely matched, and the remaining paths will continue to be processed. ;like If the common prefix of the split compressed node is used to form a new semantic segment node, then different suffixes will create new compressed nodes or leaf nodes respectively.

[0123] Furthermore, MIR incorporates a lazy unrolling mechanism, where the complete path can be stored directly as a single leaf node during the initial insertion. Subsequent insertions only perform the aforementioned splitting operation when the path partially overlaps with existing leaf nodes, thus avoiding unnecessary fine-grained node creation.

[0124] When nodes are added, split, or replaced, this invention does not reconstruct the entire index. Instead, it updates the hash commitments from bottom to top. Starting with the leaf node that has changed, the system recalculates the node's hash and updates it layer by layer upwards along the parent node relationship until a new scenario root hash (MIR Root hash) is obtained. Subsequently, the scenario root hashes are sequentially aggregated into a cross-scenario global hash (MIR Forest hash) and written into the new block header. Because the MIR structure itself is stored in LevelDB, and only the final hash commitment is recorded on-chain, index updates do not cause historical blocks to become invalid, nor do they require rewriting on-chain data. This ensures the immutability of the blockchain structure, the continuous evolution of the index state, and the long-term reliability of semantic retrieval.

[0125] (iv) Verifiable semantic query method

[0126] (1) Precise query:

[0127] Exact query given a complete semantic path :

[0128] Its semantic goal is to uniquely determine the MIR tree. Zhongyu corresponding leaf nodes and return the leaf node associated with it. With metadata summary .

[0129] Off-chain full nodes in the local index The semantic segments contained within are matched layer by layer starting from the root node. After locating the unique leaf node, the matching continues along the path... The path to the root node collects Merkle authentication information and constructs a membership proof path. Specifically, for each current node at each level on the path, the hash values ​​of all its sibling subtrees, and, if necessary, the aggregate hash of the current node covering the leaf set, are written together into the VO, denoted as: ;

[0130] The client receives the quadruple. Then, first utilize Recalculate leaf node hashes locally Then, Merkle synthesis is performed from bottom to top according to the sibling hashes provided in the VO to obtain the reconstructed root hash. .like and If the value matches the record in the block header, it verifies that the record is indeed contained in the semantic index tree promised by the current block and has not been tampered with during transmission and storage; any changes to the path token, or Forgery will cause the synthetic root hash to be inconsistent with the on-chain commitment, which will be identified and rejected by the client.

[0131] (2) Prefix lookup:

[0132] Prefix query given a semantic prefix ;

[0133] The goal is to obtain all semantic paths to The set of leaf nodes with the prefix: ;

[0134] Suitable for aggregated browsing and batch analysis by brand, category, or scenario. Functionally, this type of query can be viewed as a prefix query. A semantic aggregation query of the semantic subtree it covers.

[0135] The off-chain node first locates the internal node corresponding to the prefix node in the MIR tree. Then, enumerate all hit leaf nodes in its subtree to obtain the result set. To reduce the VO size while ensuring verifiability, this invention does not provide a complete membership proof for each leaf individually. Instead, it constructs a set of "frontier nodes" to cover the entire prefix subtree: for branches within the prefix, a set of subtree boundary nodes that are no longer expanded downwards are selected based on the tree's branching pattern; for subtrees outside the prefix, only the root hash of the corresponding subtree is returned without expanding the leaves. Therefore, the VO for prefix queries can be abstracted as: ;

[0136] in, This is the set of leading nodes that cover the entire query subtree.

[0137] During client-side verification, on the one hand, all returned paths are checked leaf by leaf to ensure they all have a given prefix. Beginning; then utilizing The content hash of the middle leaf node, along with the front boundary and sibling hash provided in the VO, are reconstructed by performing a Merkle synthesis from bottom to top. and the block header record Compare. If they match, it can be verified that the returned result does indeed correspond to the scenario index tree of the on-chain commitment, and any comparison with the returned path field... or Any alteration will result in an inconsistency in the root hash, which will then be detected and rejected.

[0138] S42, Prefix query given a certain semantic prefix : ;

[0139] The off-chain node first locates the internal node corresponding to the prefix node in the MIR tree. Then, enumerate all the matched leaf nodes in the corresponding subtree to obtain the result set. ;

[0140] For branches within the prefix, select a set of subtree boundary nodes that will not be expanded further, based on the tree's branching pattern; for subtrees outside the prefix, only return the root hash of the corresponding subtree without expanding the leaves; thus, the VO of a prefix query can be abstracted as... : ;

[0141] in, The set of leading nodes that cover the entire query subtree;

[0142] During client-side verification, on the one hand, all returned paths are checked leaf by leaf to ensure they all have a given prefix. Beginning; then utilizing The content hash of the leaf node, along with the front boundary and sibling hash provided in the VO, are reconstructed by performing Merkle synthesis from bottom to top. and the block header record Compare; if they match, the verification result does indeed correspond to the scenario index tree of the on-chain commitment.

[0143] (3) Multi-keyword search:

[0144] Multi-keyword queries are used to characterize the constraint that "the path must simultaneously contain several semantic segments." Let the set of query keywords be: ;

[0145] For example, in a computer scenario, searching for all devices that simultaneously match "laptop" and "dell" can be considered as filtering to meet the criteria. leaf node set The system first normalizes the input keywords and maps them to a set of matching predicates on the path (which can correspond to semantic segments at a specific level or any level of the path). Then, it scans the leaf nodes in the MIR tree and selects all leaves that satisfy the above predicate constraints to form a hit set.

[0146] In terms of verification, this invention reuses the VO structure idea of ​​prefix query: for each hit leaf node The VO should provide its complete path. , This includes the authentication path from the leaf node to the root of the scene; for all missing subtrees, a global commitment is made through the aggregate hash of the leading node, without expanding the internal leaves, thus avoiding linear expansion of the VO. Formally, it can be written as: ;

[0147] in This represents the frontier commitment node given for the subtree that "does not contain all keywords".

[0148] The client first checks each returned path locally to see if it satisfies the keyword set constraint, that is, for each leaf node... verify (The order of keywords can be ordered or unordered by the query interface); then, the member paths and boundary sibling hashes provided in the VO are used to reconstruct the sequence. and the block header record Compare. If they match, it verifies that the returned hit leaf node and its associated path fields and data references have not been tampered with, and all come from the on-chain committed scenario index tree; any modification to the path fields, or Forgery will disrupt the consistency of the final root hash, thus being detected and rejected by the client.

[0149] In addition, this invention also provides a semantic-identified blockchain retrieval system for large-scale distributed data, including:

[0150] The semantic identifier generation module enables the off-chain server to perform semantic parsing on the original heterogeneous data, generate a unique semantic identifier path P for each piece of data according to predefined semantic rules, and calculate the data digest meta_hash and off-chain data reference DataRef bound to the path.

[0151] The MIR index building module is used to insert the obtained semantic identifier path P into the corresponding scenario's MIR index tree. The MIR index forest formed by multiple MIR trees is used to organize the semantic identifiers of different scenarios in a structured way, and at the same time, a corresponding scenario root hash (MIR Root hash) is generated for each MIR tree.

[0152] The index commitment and maintenance module is used to aggregate the MIR Root hashes of various scenarios into a global index root MIR Forest hash in a fixed order and write it into the blockchain block header as an on-chain semantic state commitment; at the same time, it maintains the complete MIR index structure locally on the chain node in a key-value storage manner, and incrementally updates the hash commitment in a bottom-up manner during subsequent insertions or updates.

[0153] The verifiable semantic query module enables clients to initiate query requests. Off-chain nodes perform semantic matching in the MIR index forest and return the query results and the corresponding lightweight verification object (VO). Clients use the VO and the MIRRoot hash or MIR Forest hash in the block header to reconstruct and verify the results, thereby verifying the correctness and completeness of the query results.

[0154] Based on the technical solution of this invention, the implementation process of this invention in practical applications will be illustrated through the following case scenarios. The specific application implementation scheme is as follows:

[0155] 1. Experimental Platform and Prototype Implementation

[0156] To verify the performance and effectiveness of the semantic indexing and verifiable retrieval mechanism proposed in this invention in practical applications, this embodiment constructs a lightweight blockchain and off-chain index prototype system and conducts a series of experiments in a unified hardware environment. The experimental platform in this embodiment is a workstation equipped with an Intel i7-level multi-core processor, 24GB of RAM, and a 1TB solid-state drive. The prototype system is implemented using Python, and the off-chain index is built using LevelDB. All block writing, index generation, VO construction, and verification processes are completed in the same environment to ensure the fairness and repeatability of the experimental results.

[0157] 2. Datasets and Semantic Encoding

[0158] To evaluate the versatility of this invention across different applications, ten publicly available datasets covering various real-world data sources, including commodities, travel, housing, employment, film and television content, and academic literature, were selected. The dataset sizes ranged from hundreds to hundreds of thousands of records, and formats included Excel and JSON. Each record was encoded as an index transaction, including its scenario identifier, hierarchical semantic path, off-chain data reference, and original record hash, forming the semantic index forest designed in this invention. The specific datasets are shown in Table 1.

[0159] Table 1 Data source tables based on various real-world scenarios

[0160] 3 Experimental Tests

[0161] 3.1 Single-Scenario Extensibility Experiment

[0162] (1) Search performance

[0163] To analyze the retrieval performance of this invention in a large-scale single scenario, this invention selected the largest dataset of computer equipment commodities and tested three typical semantic retrieval types—multi-keyword query, path prefix query, and precise path query—at different scales ranging from 10k to 100k. The baseline for comparison included: (1) basic chained storage without any index structure, which can only locate records by sequentially scanning blocks and does not have semantic query capabilities; (2) the mainstream consortium chain FISCOBCOS, which queries on-chain transactions through the node RPC interface, and its query capabilities and semantic expression capabilities are limited by the chain structure.

[0164] Experimental results show that, Figure 2 As shown, across all test scales, the latency of the precise path query in this invention remained consistently around 1-2 milliseconds, showing almost no change with increasing data size. At the same scale, FISCO BCOS and the plain mode, due to their reliance on sequential traversal, experienced linearly increasing latency with increasing data volume and could not support complex semantic conditions. The latency of path prefix query and multi-keyword combination query remained between approximately 4-50 milliseconds and 40-500 milliseconds, respectively, showing a slight increase with increasing data volume but remaining generally stable.

[0165] In contrast, this invention not only supports richer semantic retrieval capabilities, but also maintains stable and efficient query performance under large-scale data, demonstrating that the designed semantic indexing layer and on-chain commitment structure can effectively separate storage and retrieval tasks, giving full play to the advantages of efficient off-chain indexing and trustworthy on-chain verification.

[0166] (2) Analysis of Validation Object (VO) Size and Validation Overhead

[0167] The experiment also compared the verification objects generated by the three systems. It should be noted that the VO size of the plain mode and FISCO BCOS is mainly related to the number of transactions contained in a single block. In this experiment, the baseline was designed to have a fixed block size of approximately 500 transactions. Therefore, no matter how much the total amount of data on the chain grows, its VO always only contains Merkle paths "within the block it is in," and its size remains basically stable, independent of query semantics.

[0168] In contrast, the VO generated by the MIR in this invention is not bounded by blocks, but rather by a "scenario-level index tree" as its logical structure. Its proof scope covers path segmentation, keyword filtering nodes, prefix matching nodes, and other content within that scenario, used to prove that the query result not only exists but also satisfies the corresponding semantic conditions. Therefore, the size of the MIR VO is primarily related to the "semantic data scale of a single scenario," and is unrelated to on-chain block division. In large-scale tests of a single scenario like PC, since data in the 100,000-level range still has a relatively low index tree height (logarithmic growth), the VO size only increases slightly with the scale.

[0169] Although MIR's VO structure is richer than plain and FISCO, its verification latency remains stable in the millisecond range. Figure 3 As shown, in single-scenario testing, the VO verification for exact queries typically takes no more than 1 millisecond, while the verification time for path prefixes and multi-keyword queries is mostly between 10 and 140 milliseconds. Therefore, the VO overhead of MIR is mainly affected by two parts: firstly, the proof boundary and shared authentication path related to tree height; and secondly, the result list related to the hit count (which grows linearly with the number of hits but has a small unit overhead). Since the hash operation of Python's local verification is much faster than the RPC calls and logical scans on the blockchain node side, the verification overhead remains controllable.

[0170] like Figure 4 As shown, when the record size in a single scenario increases from 10,000 to 100,000, the VO for exact queries only increases from 19 KB to 184 KB. However, with a significantly larger hit size, when the number of keyword query hits increases from 124 to 1,258, the VO increases from 0.28 MB to 2.63 MB; and when the number of prefix query hits increases from 875 to 8,371, the VO size increases from 0.35 MB to 3.32 MB. This indicates that even if MIR supports returning tens of thousands of hit results, its VO is still mainly composed of "logarithmic tree-high authentication paths + a small number of boundary node commitments + a list of DataRefs with linear but minimal constant overhead," growing slowly rather than expanding uncontrollably with the expansion of data size. In a single scenario with 100,000 records, the VO for prefix and multi-keyword high-hit queries remains stable in the range of several MB, thus ensuring semantic verifiability while maintaining deployable communication and storage overhead.

[0171] 3.2 Multi-scenario query scalability testing and result analysis

[0172] (1) Search performance

[0173] In multi-scenario experiments, this invention sequentially wrote ten datasets of different sources and sizes into the same chain, observing the performance changes of each system in PC scenario queries as the number of scenarios on the chain increased from 1 to 10. The retrieval object was always fixed to the path in the PC scenario to reflect whether the system would experience performance degradation due to heterogeneous data and scale expansion on the chain.

[0174] The results are as follows Figure 5 As shown, consistent with the single-scenario results, MIR's three types of semantic retrieval (multi-keyword, prefix, and exact) still maintain stable millisecond or sub-millisecond performance in multiple scenarios; the difference is that as the cumulative number of scenarios on the chain gradually increases, it can be observed that the query time of plain and FISCO BCOS is significantly higher than that of MIR.

[0175] (2) Verifiability

[0176] like Figure 6 and Figure 7 As shown, in terms of verification objects, MIR, plain, and FISCO exhibit completely different scale stability under multiple scenario conditions. The VOs of plain and FISCO are determined by a Merkle tree consisting of approximately 500 transactions within a block, and their size is only related to the number of transactions within the block, not to the total number of scenarios on the chain. Therefore, they maintain a stable structure at the byte to KB level throughout the process. In contrast, the VO of MIR is driven by the scale of data within a scenario. Since the scale of the PC scenario remains constant in multi-scenario experiments, its VO size and verification time also remain stable at the single-scenario level, without changing with the increase in the number of scenarios. This further illustrates that MIR's design inherently possesses semantic partitioning isolation: scenarios do not interfere with each other, thus ensuring that the verifiable structure of the VO expands with the complexity of query semantics, rather than with the overall scale of the chain.

[0177] In summary, multi-scenario experiments demonstrate that the semantically verifiable index proposed in this invention maintains stable query performance across multiple scenarios and multi-business co-chain environments, without degrading as on-chain data volume expands. Furthermore, the VO structure remains independent of other scenarios, ensuring the controllability and predictability of verifiability costs. This fully proves that this invention is suitable for building a blockchain intelligent computing system with unified storage and verifiable queries across multiple domains.

[0178] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A semantic identifier blockchain retrieval method for large-scale distributed data, characterized in that, Includes the following steps: S1, Semantic identifier generation: The off-chain server performs semantic parsing on the original heterogeneous data, generates a unique semantic identifier path P for each piece of data according to predefined semantic rules, and calculates the data digest meta_hash and off-chain data reference DataRef bound to the path. S2, MIR index construction: The semantic identifier path P obtained in step S1 is inserted into the MIR index tree of the corresponding scenario. The semantic identifiers of different scenarios are structured and organized through the MIR index forest formed by multiple MIR trees. At the same time, a corresponding scenario root hash (MIR Root hash) is generated for each MIR tree. S3, Index Commitment and Maintenance: The MIR Root hashes from various scenarios are aggregated into a global index root MIR Forest hash in a fixed order and written into the blockchain block header as an on-chain semantic state commitment. At the same time, the complete MIR index structure is maintained locally on the chain node in a key-value storage manner, and the hash commitment is incrementally updated in a bottom-up manner during subsequent insertions or updates. S4 can verify semantic queries: The client initiates a query request, and the off-chain node performs semantic matching in the MIR index forest and returns the query results and the corresponding lightweight verification object (VO). The client uses the VO and the MIR Root hash or MIR Forest hash in the block header to reconstruct and verify the results, thereby verifying the correctness and completeness of the query results.

2. The semantic identifier blockchain retrieval method for large-scale distributed data according to claim 1, characterized in that, In step S1, the semantic identification path P is as follows: ； Here, Scene represents a macro-level application scenario; Type represents a semantic type of data; Subtype* represents an extensible sub-level used for recursive expansion to support multi-level semantic structures; Entity is used to ensure the global uniqueness of the identifier. Subtype* employs a variable-length semantic sub-path mechanism, which can be extended according to actual needs; Entities are generated using a short hash method based on scenario, semantic path, and record content.

3. The semantic identifier blockchain retrieval method for large-scale distributed data according to claim 2, characterized in that, In step S2, the MIR tree contains the following three types of nodes: Semantic segment node : Represents an intermediate node with multiple sub-path branches. The node content is a single semantic token used to distinguish different semantic branches at the same level; Compressed Nodes Used for path compression, it represents a node formed by merging consecutive unique sub-paths, and its content is a combination of multiple consecutive token sequences; leaf node : Represents the endpoint of a complete semantic path, corresponding to a unique entity identifier and associated data, as defined below: ； For each Scene, the system maintains the root hash of the scene's MIR tree and writes it into the block header as a cryptographic commitment to all current records in that scene.

4. The semantic identifier blockchain retrieval method for large-scale distributed data according to claim 3, characterized in that, Step S3 includes the following steps: S31, Given a new semantic identifier path to be inserted. : ； During insertion, the tree structure is traversed level by level. The specific steps are as follows: S311, Initial Positioning: From the root set of the MIR Forest, search for the corresponding MIR tree by the root node Scene; if it does not exist, create a new tree. S312, step-by-step insertion: For each in the path ,in First, determine the corresponding node type: if the current node is a semantic segment node, check if it exists in the child node set. If the corresponding branch exists, it will continue down to the corresponding child node; otherwise, a new semantic segment node or leaf node will be created. If the current node is a compressed node, then the compressed node will be set to... The remaining path to be inserted is Calculate the length of the corresponding common prefix. : ； The sequence length function is used to return the number of tokens contained in the common prefix token sequence; CommonPrefix(C,R) represents the longest common prefix token sequence between sequence C and sequence R; like This indicates that the compressed node path is completely matched, and the remaining paths will continue to be processed. ;like If the common prefix of the split compressed node is used to form a new semantic segment node, then different suffixes will create new compressed nodes or leaf nodes respectively.

5. The semantic identifier blockchain retrieval method for large-scale distributed data according to claim 4, characterized in that, Step S3 also includes the following steps: S32, the system starts from the leaf node that has changed, recalculates the corresponding node hash, and updates it layer by layer upwards along the parent node relationship until a new scene root hash MIR Root hash is obtained; S33, the root hashes of each scenario are then sequentially aggregated into a cross-scenario global hash, MIR Forest hash, and written into the new block header.

6. The semantic identifier blockchain retrieval method for large-scale distributed data according to claim 5, characterized in that, In step S4, the query request includes an exact query, a prefix query, or a multi-keyword query; The precise query includes the following process: S41, Define an exact query given a complete semantic path : ； The corresponding semantic target is a uniquely determined MIR tree Zhongyu corresponding leaf nodes and return the corresponding leaf node associated with it. With metadata summary ; Off-chain full nodes in the local index The semantic segments contained within are matched layer by layer starting from the root node. After locating the unique leaf node, the matching continues along the path... The path to the root node collects Merkle authentication information and constructs a membership proof path; For each current node at each level along the path, the hash values ​​of all its sibling subtrees, and, if necessary, the aggregate hash of the current node covering the leaf set, are written together into the VO, denoted as . ; The client receives the quadruple. Then, first utilize Recalculate the leaf node hash locally Then, Merkle synthesis is performed from bottom to top according to the sibling hashes provided in the VO to obtain the reconstructed root hash. ; like and If the value matches the record value in the block header, it verifies that the corresponding record is indeed contained in the semantic index tree promised by the current block and has not been tampered with during transmission and storage.

7. The semantic identifier blockchain retrieval method for large-scale distributed data according to claim 6, characterized in that, In step S4, the prefix query includes the following process: S42, Prefix query given a certain semantic prefix : ； The off-chain node first locates the internal node corresponding to the prefix node in the MIR tree. Then, enumerate all the matched leaf nodes in the corresponding subtree to obtain the result set. ; For branches within the prefix, select a set of subtree boundary nodes that will not be expanded further, based on the tree's branching pattern; for subtrees outside the prefix, only return the root hash of the corresponding subtree without expanding the leaves; thus, the VO of a prefix query can be abstracted as... : ； in, The set of leading nodes that cover the entire query subtree; During client-side verification, on the one hand, all returned paths are checked leaf by leaf to ensure they all have a given prefix. Beginning; then utilizing The content hash of the leaf node, along with the front boundary and sibling hash provided in the VO, are reconstructed by performing Merkle synthesis from bottom to top. and the block header record Compare; if they match, the verification result does indeed correspond to the scenario index tree of the on-chain commitment.

8. The semantic identifier blockchain retrieval method for large-scale distributed data according to claim 6, characterized in that, In step S4, the multi-keyword query includes the following process: S43, Set the set of query keywords as ; Filter to meet The set of leaf nodes is ; The system first normalizes the input keywords and maps them to a set of matching predicates on the path. Then, it scans the leaf nodes in the MIR tree and selects all leaves that satisfy the predicate constraints to form a hit set. In terms of verification, for each hit leaf node The VO should provide the corresponding complete path. , And the authentication path from the leaf to the scene root; for all missing subtrees, make a global commitment through the aggregate hash of the leading edge nodes; verify the object. Defined as: ； in This represents the frontier commitment node given for the subtree that "does not contain all keywords"; The client first checks each returned path locally to see if it satisfies the keyword set constraint, that is, for each leaf node... verify Subsequently, the member paths and boundary sibling hashes provided in the VO were used to reconstruct the data. and the block header record Compare; if they match, then the returned hit leaf node and its corresponding associated path field and data reference have not been tampered with, and all come from the on-chain promised scenario index tree.

9. A semantic identifier blockchain retrieval system for large-scale distributed data, used to implement the semantic identifier blockchain retrieval method for large-scale distributed data as described in any one of claims 1-8, characterized in that, The semantic identifier blockchain retrieval system for large-scale distributed data includes: The semantic identifier generation module enables the off-chain server to perform semantic parsing on the original heterogeneous data, generate a unique semantic identifier path P for each piece of data according to predefined semantic rules, and calculate the data digest meta_hash and off-chain data reference DataRef bound to the path. The MIR index building module is used to insert the obtained semantic identifier path P into the corresponding scenario's MIR index tree. The MIR index forest formed by multiple MIR trees is used to organize the semantic identifiers of different scenarios in a structured way, and at the same time, a corresponding scenario root hash (MIR Root hash) is generated for each MIR tree. The index commitment and maintenance module is used to aggregate the MIR Root hashes of various scenarios into a global index root MIR Forest hash in a fixed order and write it into the blockchain block header as an on-chain semantic state commitment; at the same time, it maintains the complete MIR index structure locally on the chain node in a key-value storage manner, and incrementally updates the hash commitment in a bottom-up manner during subsequent insertions or updates. The verifiable semantic query module enables clients to initiate query requests. Off-chain nodes perform semantic matching in the MIR index forest and return the query results and the corresponding lightweight verification object (VO). Clients use the VO and the MIR Roothash or MIR Forest hash in the block header to reconstruct and verify the results, thereby verifying the correctness and completeness of the query results.

Images