A hybrid blockchain-based multi-modal data approximate query method and system

Abstract

This invention discloses a method and system for approximate multimodal data query based on hybrid blockchain, relating to the field of blockchain technology. The invention constructs a verifiable index structure to describe the compressed representation of multimodal data feature vectors and their mapping relationship to off-chain data pointers. It uses only a minimal on-chain space to record index state changes, avoiding the huge storage overhead of directly storing the original high-dimensional feature vectors or complete data on the blockchain. By compressing the feature vectors into PQ codes through product quantization and combining them with an MPT structure to generate verifiable lightweight cryptographic proofs, the authenticity of the multimodal approximate query results is verified. The Merkle proof is independently verified using on-chain root hashes to ensure the results have not been tampered with. The client downloads the original data, performs full-precision feature extraction and distance calculation, compensating for the precision loss caused by PQ compression; thus achieving a complete and reliable closed loop of "efficient approximate search + accurate result output".

Description

Technical Field

[0001] This invention belongs to the field of blockchain technology, and in particular relates to a method and system for approximate querying of multimodal data based on hybrid blockchain. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, multimodal data such as images, videos, text, and audio are widely used in social networks, e-commerce, smart healthcare, and autonomous driving. How to support semantic similarity retrieval (such as "find images similar to this image" or "search for related images using a single sentence") while ensuring data integrity has become a current research hotspot.

[0003] Blockchain technology, due to its decentralized, immutable, and traceable characteristics, is widely used for data storage and integrity protection. However, traditional blockchain systems (such as Ethereum) are primarily designed to process structured transaction records, with limited storage capacity and low throughput, making it difficult to directly handle large-scale, multimodal data. Existing solutions mainly include:

[0004] Pure off-chain storage + on-chain hash anchoring: Raw data is stored in distributed file systems such as IPFS and Filecoin, with only the data hash value written to the blockchain. Feature vector on-chain: Multimodal data is extracted into high-dimensional feature vectors (e.g., 512-dimensional CLIP vectors) using deep learning models and stored directly on the blockchain. Vector compression + off-chain indexing: Feature vectors are compressed using techniques such as Product Quantization (PQ) to improve storage efficiency. Improvements to existing index structures: Some research proposes optimizing blockchain query performance based on structures such as Merkle trees, Cuckoo filters, and B+ trees.

[0005] However, while the pure off-chain storage + on-chain hash anchoring method saves on-chain space, it cannot support content-based similarity retrieval, and the off-chain index is easily tampered with, making it impossible for users to verify the authenticity of the returned results. Methods like on-chain feature vectors support vector similarity calculations, but each vector occupies approximately 2KB of space, which can generate terabytes of data daily in blockchains with thousands of TPS, severely limiting system scalability. In vector compression + off-chain indexing methods, errors introduced during compression will affect retrieval accuracy; furthermore, the lack of cryptographic verification mechanisms in off-chain indexes poses a risk of result forgery. These methods with improved index structures are mainly geared towards keyword matching or numerical range queries and are not suitable for Approximate Nearest Neighbor (ANN) retrieval in high-dimensional vector spaces.

[0006] In summary, existing technologies struggle to balance storage efficiency, query accuracy, and result reliability. Therefore, there is an urgent need for a technical solution that can significantly reduce on-chain overhead while supporting high-precision, verifiable multimodal approximate queries. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a method and system for approximate multimodal data query based on hybrid blockchain. By constructing a three-layer trusted architecture of "on-chain compressed index + off-chain raw data + client-side verification and reordering", it achieves efficient and high-precision multimodal content retrieval while ensuring data integrity and query verifiability. This solves the core problems faced by existing technologies in multimodal data blockchain storage and query, such as high storage overhead, low query accuracy, and unreliable results.

[0008] The technical solution of this invention is as follows:

[0009] On one hand, this invention provides a method for approximate multimodal data query based on hybrid blockchain, comprising the following steps:

[0010] Obtain the raw data of the multimodal dataset, and perform feature extraction and unified vectorization to obtain a high-dimensional feature vector for each raw data.

[0011] Each high-dimensional feature vector is compressed and multiplied and quantized to generate PQ codes;

[0012] Based on the generated PQ code, a verifiable index structure PQ-MPT is constructed;

[0013] Write the root hash of the verifiable index structure PQ-MPT into the blockchain, and form an immutable global state commitment through consensus;

[0014] Obtain user query requests and use the verifiable index structure PQ-MPT to retrieve query results from the multimodal raw data.

[0015] Furthermore, the process of acquiring the multimodal raw data and performing feature extraction and unified vectorization to obtain a high-dimensional feature vector for each raw data point specifically includes:

[0016] A1: Retrieve raw multimodal data, including images, text, audio, and video, from a distributed storage system;

[0017] A2: Preprocess the raw multimodal data to obtain preprocessed multimodal data;

[0018] Specifically:

[0019] For audio, first convert the audio into a waveform and extract the Mel spectrogram;

[0020] For the image and Mel spectrogram, perform the following preprocessing operations: scale the image and Mel spectrogram to a set uniform size, then convert the scaled image and Mel spectrogram to the RGB three-channel color space, and normalize, zero-mean, and variance normalize the pixel values ​​to obtain the preprocessed image and Mel spectrogram.

[0021] For text, perform word segmentation, stop word removal and lowercase conversion, and then truncate or pad to a fixed length;

[0022] For videos, keyframe images are extracted from the video or several frames are obtained by sampling at fixed intervals.

[0023] A3: Perform deep feature extraction and vectorization on the preprocessed multimodal data to obtain high-dimensional feature vectors and store them.

[0024] Furthermore, the step of compressing and multiplying each high-dimensional feature vector to generate PQ codes specifically includes:

[0025] B1: Transform high-dimensional feature vectors The space is divided into Subspaces, and then high-dimensional feature vectors Divide into uniformly along the dimensional axis Non-overlapping sub-vectors The number of subspaces is set, and each subvector corresponds to one subspace;

[0026] B2: Each subspace is trained separately, and each subspace yields a local codebook, which is then combined to form the PQ codebook. ;

[0027] Specifically: Extract high-dimensional feature vectors from all multimodal raw data and divide them uniformly into... After identifying non-overlapping sub-vectors, for each subspace, the k-means clustering algorithm is used to cluster all sub-vectors belonging to that subspace, resulting in clusters of... A central point forms a local codebook. , For the first Local codebooks of each subspace, For the first The first subspace One central point, Number the center point, then The PQ codebook is obtained by integrating the local codebooks. ;

[0028] B3: PQ codebook After training, calculate its SHA-256 hash. It is stored on the blockchain and the PQ codebook is updated regularly to adapt to changes in data distribution;

[0029] B4: Using the PQ codebook For high-dimensional feature vectors Perform product quantization encoding to obtain PQ codes;

[0030] Specifically: for each subvector Calculate its relationship with the local codebook middle Find the optimal cluster center index by using the Euclidean distance between the center points:

[0031] (1);

[0032] in, For the first The optimal cluster center index corresponding to each sub-vector;

[0033] Then the high-dimensional feature vector of The optimal cluster center indices of the subvectors are combined sequentially to generate the final PQ code.

[0034] Furthermore, the construction of the verifiable index structure PQ-MPT based on the generated PQ code specifically includes:

[0035] C1: Construct key-value pairs based on the PQ code corresponding to each piece of raw data;

[0036] PQ code for each piece of raw data Calculate the SHA-256 hash to obtain a 256-bit cryptographic hash value. As the key; the content identifier (CID) of the original data in the distributed storage system is used as the value, thus obtaining key-value pairs. ;

[0037] C2: Construct an MPT tree based on key-value pairs to obtain the index structure PQ-MPT.

[0038] Furthermore, the step of writing the root hash of the verifiable index structure PQ-MPT into the blockchain and forming an immutable global state commitment through consensus specifically includes:

[0039] D1: Write the root hash of the index structure PQ-MPT into the blockchain;

[0040] D2: Deploy a smart contract that stores the root hash of the index structure PQ-MPT and the SHA-256 hash of the PQ codebook. It provides the function of updating the root hash of the on-chain index structure PQ-MPT;

[0041] D3: When the index structure PQ-MPT is updated, the smart contract is invoked to construct a transaction and combine the root hash of the new PQ-MPT with the SHA-256 hash of the PQ codebook. Stored on the blockchain.

[0042] Furthermore, the step of obtaining user query requests and using the verifiable index structure PQ-MPT to retrieve query results from the multimodal raw data specifically includes:

[0043] E1: The user submits a query request through the client;

[0044] E2: Extract features from the query request using the same method as A3 to obtain a high-dimensional query vector;

[0045] E3: Using the PQ codebook For high-dimensional query vectors Compress the code to generate a query PQ code. ;

[0046] E4: Cluster all the PQ codes of the original data into L clusters. Each cluster corresponds to an inverted list that stores the identifiers of all PQ codes belonging to that cluster.

[0047] E5: Calculate the query PQ code Given the distances to the centers of the L clusters, select the τ clusters that are closest to them, and perform a candidate search only in the inverted list corresponding to the selected clusters to obtain the candidate results;

[0048] E6: Generate a proof for each candidate result;

[0049] E7: Verify the proof of the candidate results and retain the candidate results that pass the verification;

[0050] E8: Perform precise reordering on the validated candidate results to obtain the final query results;

[0051] Specifically, for the verified candidate results, the original data is downloaded using their Content Identifier (CID), features are extracted from the original data to obtain a high-dimensional feature vector, the precise distance between the high-dimensional feature vector and the high-dimensional query vector is calculated, the candidate results are reordered, and the final query result is output.

[0052] On the other hand, the present invention also provides a multimodal data approximation query system based on hybrid blockchain, for implementing a multimodal data approximation query method based on hybrid blockchain, including:

[0053] The feature extraction module is used to acquire the raw data of the multimodal data, extract features from it and perform unified vectorization to obtain a high-dimensional feature vector for each raw data.

[0054] The encoding module is used to compress and multiply quantize each high-dimensional feature vector to generate PQ codes;

[0055] The index structure construction module is used to construct a verifiable index structure PQ-MPT based on the generated PQ code;

[0056] The anchoring module is used to write the root hash of the verifiable index structure PQ-MPT into the blockchain, and form an immutable global state commitment through consensus.

[0057] The query module is used to obtain user query requests and retrieve query results from the multimodal raw data using the verifiable index structure PQ-MPT.

[0058] Thirdly, this application proposes an electronic device, including: one or more processors, and a memory for storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform the aforementioned multimodal data approximation query method based on a hybrid blockchain.

[0059] Fourthly, this application proposes a computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the aforementioned multimodal data approximation query method based on a hybrid blockchain.

[0060] Fifthly, this application proposes a computer program product, including a computer program or instructions that, when executed by a processor, implement the aforementioned multimodal data approximation query method based on a hybrid blockchain.

[0061] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0062] The technical solution proposed in this invention constructs a verifiable index structure based on PQ-Merkle Patricia Trie (PQ-MPT) to describe the compressed representation of multimodal data feature vectors and their mapping relationship with off-chain data pointers. It uses only a minimal on-chain space to record index state changes, avoiding the huge storage overhead of directly storing the original high-dimensional feature vectors or complete data on the blockchain. Under this mechanism, a 512-dimensional feature vector is compressed to a 64-bit PQ code through product quantization (PQ), and combined with the MPT structure to generate a verifiable lightweight cryptographic proof. This enables verification of the authenticity of multimodal approximate query results. Users independently verify the Merkle proof using the on-chain root hash, ensuring the results have not been tampered with. The client downloads the original data, performs full-precision feature extraction and distance calculation, compensating for the precision loss caused by PQ compression. This achieves a complete and reliable closed loop of "efficient approximate search + accurate result output". This method significantly reduces the storage burden on the blockchain while ensuring the efficiency of the query process and the credibility of the results. This avoids problems such as data not being able to be uploaded to the chain, unreliable queries, or decreased system performance caused by on-chain capacity limitations, thus ensuring the secure sharing and efficient retrieval of multimodal data in a decentralized environment. Attached Figure Description

[0063] Figure 1 This is an overall flowchart of a multimodal data approximation query method based on hybrid blockchain in an embodiment of the present invention;

[0064] Figure 2 This is a flowchart of the PQ code generation process in an embodiment of the present invention;

[0065] Figure 3 This is a structural diagram of the verifiable index structure PQ-MPT in an embodiment of the present invention;

[0066] Figure 4 This is a schematic diagram of the query process in an embodiment of the present invention;

[0067] Figure 5 This is a flowchart of client verification and reordering in an embodiment of the present invention. Detailed Implementation

[0068] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0069] Example 1:

[0070] A method for approximate multimodal data query based on hybrid blockchain, such as Figure 1 As shown, it includes the following steps:

[0071] Step 1: Obtain the raw data of the multimodal dataset, and perform feature extraction and unified vectorization to obtain a high-dimensional feature vector for each raw data; the modalities include images, text, audio, and video, etc.

[0072] Step 1 is the data preprocessing and semantic encoding stage of this invention. Its core purpose is to convert heterogeneous multimodal raw data into a unified, high-dimensional numerical vector representation, thereby achieving alignment and computability across modal semantic spaces. This step ensures that raw data from different modalities can be similarly measured in the same vector space, laying the foundation for subsequent compression, indexing and retrieval.

[0073] The specific implementation process is as follows:

[0074] Step 1.1: Obtain the raw multimodal data from the distributed storage system, including images, text, audio, and video;

[0075] In this embodiment, user-uploaded multimodal raw data is received via a RESTful API interface, a web portal, or a command-line tool. The data types include, but are not limited to: static images in JPEG, PNG, BMP, and GIF formats; text files in TXT, HTML, PDF, and DOCX formats; audio data in WAV, MP3, and FLAC formats; and video clips in MP4, AVI, and MOV formats. The data is automatically categorized based on file extension, MIME type, or binary signature (magic number), identifying its modality category and recording metadata (such as upload time, user ID, and data tags).

[0076] Step 1.2: Preprocess the raw multimodal data to obtain preprocessed multimodal data;

[0077] For audio, first convert the audio into a waveform and extract the Mel-spectrogram.

[0078] Then, for the image and Mel spectrogram, the following preprocessing operations are performed: the image and Mel spectrogram are scaled to a set uniform size (e.g., 224×224 pixels) using bilinear interpolation or Lanczos resampling algorithm to ensure image quality; then, the scaled image and Mel spectrogram are converted to the RGB three-channel color space, and the pixel values ​​are normalized (i.e., the pixel values ​​are mapped from [0,255] to the range [0.0,1.0]), zero-mean (subtracting the mean vector of the ImageNet dataset [0.485, 0.456, 0.406]), and variance normalized (divided by the standard deviation [0.229, 0.224, 0.225]) to match the input requirements of the pre-trained model, resulting in the preprocessed image and Mel spectrogram;

[0079] For text, perform tokenization (using Byte Pair Encoding or WordPiece), stop word removal, and lowercasing, and then truncate or pad to a fixed length (e.g., 77 tokens) to ensure that the input format conforms to the model specifications. For long texts, sliding window or summary extraction techniques can be used to generate representative segments.

[0080] For video, keyframe images (such as I-frames) are extracted from the video or several frames are obtained by sampling at fixed intervals;

[0081] Step 1.3: Perform deep feature extraction and vectorization on the preprocessed multimodal data to obtain high-dimensional feature vectors and store them;

[0082] Load a pre-trained cross-modal deep neural network model, preferably the CLIP (Contrastive Language-Image Pre-training) model released by OpenAI, which includes independent visual encoders (such as ViT-B / 32 or ResNet-50) and text encoders. The visual encoder takes the pre-processed image and Mel spectrogram as input, and the text encoder takes the pre-processed text as input, and outputs a unified 512-dimensional high-dimensional feature vector. In this unified vector space, semantically similar data (such as "a running dog" and its corresponding image) are geometrically close, thus supporting distance-based similarity retrieval.

[0083] For keyframe images (such as I-frames) extracted from videos or several frames sampled at fixed intervals, average or max pooling is used to obtain a high-dimensional feature vector of 512 dimensions.

[0084] The high-dimensional feature vector generated in this embodiment The high-dimensional feature vector is stored in a distributed storage system in float32 format, with each high-dimensional feature vector occupying 4 bytes × 512 = 2048 bytes (2KB) of space. This high-dimensional feature vector serves as the input for subsequent PQ compression, and its hash value can be used for subsequent data deduplication and integrity verification. All intermediate results (original multimodal data, high-dimensional feature vector, and subsequently obtained PQ codes) are stored in the distributed storage system.

[0085] Step 2: Perform compression and product quantization (PQ) encoding on each high-dimensional feature vector to generate PQ codes;

[0086] Step 2 is the core compression stage of this invention, which aims to solve the storage explosion and performance bottleneck problems caused by directly uploading high-dimensional feature vectors to the blockchain. This step uses Product Quantization (PQ) technology to perform lossy compression on the 512-dimensional floating-point high-dimensional feature vector, generating a compact PQ code of only 8 bytes, thereby reducing the size of on-chain metadata by two orders of magnitude. This step greatly improves the scalability of the blockchain system while ensuring approximate retrieval availability.

[0087] like Figure 2 As shown, the specific implementation process is as follows:

[0088] Step 2.1: Convert the high-dimensional feature vector The space is divided into Subspaces, and then high-dimensional feature vectors Divide into uniformly along the dimensional axis Non-overlapping sub-vectors The number of subspaces is set, and each subvector corresponds to one subspace;

[0089] In this embodiment, the input 512-dimensional high-dimensional feature vector Divide into uniformly along the dimensional axis =8 non-overlapping sub-vectors, each sub-vector has a dimension The value is 64, and the segmentation method is as follows: ,in Indicates the first indivual The sub-vectors of dimension are partitioned based on the assumption that "vector space is decomposable," meaning that each subspace is independent, can be quantized separately, and supports dynamic adjustment. and To adapt to high-dimensional feature vectors of different dimensions in the input;

[0090] Step 2.2: Each subspace is trained separately, and each subspace yields a local codebook, which is then combined to form the PQ codebook. ;

[0091] Specifically: Extract high-dimensional feature vectors from all multimodal raw data and divide them uniformly into... After identifying non-overlapping sub-vectors, for each subspace, the k-means clustering algorithm is used to cluster all sub-vectors belonging to that subspace, resulting in clusters of... A central point forms a local codebook. , For the first Local codebooks of each subspace, For the first The first subspace One central point, Number the center point, then The PQ codebook is obtained by integrating the local codebooks. ;

[0092] In this embodiment of the invention, for each subspace, all corresponding 64-dimensional sub-vectors are extracted from the training set (such as subsets of MS-COCO, Flickr30K, and LAION-5B), and clustered. =256 center points, forming a local codebook By combining the eight local codebooks, the PQ codebook is obtained. And stored on the server, the total size is approximately 8×256×64×4=512KB (float32).

[0093] Step 2.3: PQ Codebook After training, calculate its SHA-256 hash. It is also stored on the blockchain to ensure algorithm consistency and auditability, and supports regular updates of the PQ codebook to adapt to changes in data distribution;

[0094] Step 2.4: Using the PQ codebook The high-dimensional feature vector obtained in step 1 Perform product quantization (PQ) encoding to obtain PQ codes;

[0095] PQ encoding process:

[0096] For each subvector Calculate its relationship with the local codebook middle Find the optimal cluster center index by using the Euclidean distance between the center points:

[0097] (1);

[0098] in, For the first The optimal cluster center index (i.e., a part of the PQ code) corresponding to each sub-vector;

[0099] Then the high-dimensional feature vector of The optimal cluster center indices of the subvectors are combined in order to generate the final PQ code;

[0100] In this embodiment, due to =256, each optimal cluster center index can be represented by an 8-bit (1-byte) unsigned integer; the 8 optimal cluster center indices are combined sequentially to generate the final PQ code: The total length is 8 bytes (64 bits).

[0101] Compression benefit analysis: A single high-dimensional feature vector is compressed from 2048 bytes to 8 bytes, with a compression ratio of 256:1, which significantly reduces the storage pressure on the blockchain;

[0102] Step 3: Based on the generated PQ code, construct a verifiable index structure PQ-MPT;

[0103] Step 3 is the index construction stage of this invention, aiming to create an on-chain index structure that supports cryptographic verification, ensuring that any query result can be independently verified by the client for its authenticity and integrity. This step innovatively combines PQ codes with Merkle Patricia Trie (MPT) to form a "PQ-MPT" structure, the structure of which is as follows: Figure 3 As shown, this solves the trust problem of off-chain indexes being easily tampered with.

[0104] The specific implementation process is as follows:

[0105] Step 3.1: Construct key-value pairs based on the PQ code corresponding to each piece of raw data;

[0106] PQ code for each piece of raw data Calculate the SHA-256 hash to obtain a 256-bit (32-byte) cryptographic hash value. The original data's Content Identifier (CID) in the distributed storage system is used as the key; the value is then used to obtain key-value pairs. ;

[0107] In this embodiment, the content identifier is preferably in IPFS CIDv1 format (e.g., bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylq44jdtj4fcz6dvyq), with a length of 46 bytes.

[0108] Step 3.2: Construct an MPT tree based on key-value pairs to obtain the index structure PQ-MPT;

[0109] Specifically, all key-value pairs are lexicographically sorted, and an MPT (Merkle PatriciaTrie) tree is constructed from bottom to top. Leaf nodes store complete key-value pairs and paths from the root node to the leaf nodes, where the path is the hexadecimal representation of the key. Extension nodes are used to compress common prefix paths. Branch nodes contain 16 pointers to child nodes (corresponding to hexadecimal characters 0-F) and an optional value. Null nodes represent empty paths. The hash of each node (non-null nodes, including leaf nodes, branch nodes, and extension nodes) is the SHA-3 (Keccak-256) digest of its content. The root node outputs the root hash as a unique fingerprint of the entire index state.

[0110] Supports dynamic operations: When adding new multimodal raw data, insert new leaf nodes and update the path; when deleting multimodal raw data, remove the corresponding nodes and recalculate the Keccak-256 hash value and hash digest of each modified node (including leaf nodes, extension nodes, and branch nodes) on the path.

[0111] Supports snapshots and incremental updates, facilitating large-scale data management;

[0112] The index structure is persistently stored in LevelDB or RocksDB, supporting efficient queries;

[0113] Step 4: Write the root hash of the verifiable index structure PQ-MPT into the blockchain, and form an immutable global state commitment through consensus;

[0114] Step 4 is the trust anchoring stage of this invention, which aims to write the root hash of PQ-MPT into the blockchain, forming an immutable, globally consensus-based state commitment. This step is the cornerstone of the entire blockchain system's trustworthiness, ensuring that any tampering with the index can be detected. The specific implementation process is as follows:

[0115] Step 4.1: Write the root hash of the index structure PQ-MPT into the blockchain;

[0116] Step 4.2: Deploy the smart contract, which stores the root hash of the index structure PQ-MPT and the SHA-256 hash of the PQ codebook. The updateIndex function is used to update the root hash of the on-chain index structure PQ-MPT.

[0117] In this embodiment, the smart contract MultimodalIndex is deployed on a blockchain platform such as Tendermint, Ethereum, or Hyperledger Fabric.

[0118] The preferred contract language is Go (Tendermint ABCI).

[0119] State variable definition:

[0120] contract MultimodalIndex {

[0121] bytes32 public rootHash; / / Current PQ-MPT root hash

[0122] bytes32 public codebookHash; / / PQ codebook hash

[0123] uint256 public lastUpdateTime; / / Last update timestamp

[0124] address public owner; / / Contract administrator

[0125] }

[0126] Step 4.3: When the index structure PQ-MPT is updated, the smart contract is invoked to construct a transaction and combine the root hash of the new PQ-MPT with the SHA-256 hash of the PQ codebook. Store on the blockchain;

[0127] In this embodiment, after the index structure PQ-MPT is updated, the smart contract's updateIndex(bytes32 _root, bytes32 _codebookHash) function is called; a transaction is constructed, including function call data, gas limit, nonce, and other fields; the transaction is signed using the administrator's private key; it is broadcast to the blockchain network, verified by consensus nodes, and then packaged into a block; after the transaction is confirmed, rootHash and codebookHash are permanently recorded, forming an immutable historical version;

[0128] Step 5: Obtain the user's query request and use the verifiable index structure PQ-MPT to retrieve the query results from the multimodal raw data;

[0129] Step 5 is the query service and result proof stage of this invention, the core of which lies in achieving a combination of efficient and reliable off-chain approximate search and on-chain verifiability. Through the five-step process of "query encoding → approximate search → proof generation → client verification → precise reordering", it ensures that users can quickly obtain candidate results and independently verify their authenticity.

[0130] The query process is as follows Figure 4As shown below, the specific implementation process is as follows:

[0131] Step 5.1: The user submits a query request through the client (the input form includes images, text phrases, or fragments of academic papers).

[0132] Step 5.2: Extract features from the query request according to the method in Step 1 to obtain a high-dimensional query vector, ensuring that it is in the same semantic space as the multi-modal original data stored off-chain.

[0133] Step 5.3: Use the PQ codebook obtained in Step 2.2 to compress the high-dimensional query vector and generate a query PQ code ;

[0134] In this embodiment, the high-dimensional query vector is divided into 8 64-dimensional sub-vectors. For each sub-vector, calculate its optimal clustering center index with the local codebook to generate a query PQ code ;

[0135] Step 5.4: Cluster the PQ codes of all original data into L clusters, and each cluster corresponds to an inverted list, storing the identifiers of all PQ codes belonging to that cluster.

[0136] In this embodiment, L = 100;

[0137] Step 5.5: Calculate the distance between the query PQ code and the centers of the L clusters, select the τ closest clusters (τ << L), and only perform candidate searches in the inverted lists corresponding to the selected clusters, greatly reducing the computational amount to obtain candidate results.

[0138] In this embodiment, τ = 4;

[0139] Step 5.6: Generate a proof for each candidate result.

[0140] In this step, a cryptographic proof is generated for each candidate result to prove that it is consistent with the root hash of the index structure PQ-MPT stored on the chain. For the PQ code of each candidate result, after the blockchain node calculates the key of the candidate result, it locates the leaf node corresponding to the key in the local index structure PQ-MPT; traverse from the leaf node to the root node upward, collect the hash values of all sibling nodes on the path, form a Merkle proof and return it. This Merkle proof consists of the PQ code, the content identifier CID, and the Merkle path. This proof does not depend on the honesty of the server, and any tampering (PQ code, content identifier CID, Merkle path) will result in verification failure.

[0141] Step 5.7: Verify the proofs of the candidate results and retain the candidate results that pass the verification;

[0142] The client queries the root hash through a blockchain light node or full node to obtain the current global consensus PQ-MPT root hash. Then, for each candidate result, it recalculates and verifies the local root hash based on the Merkle proof (PQ code, content identifier CID, Merkle path). If the two results are different, the candidate result is rejected.

[0143] Step 5.8: Perform precise reordering on the validated candidate results to obtain the final query results;

[0144] PQ compression is a lossy process, which may lead to approximate ranking errors (e.g., the 99th most similar result is ranked 1st). Therefore, this invention performs full-precision re-ranking locally on the client side. For verified candidate results, the original data (image or text) is downloaded using its Content Identifier (CID). Feature extraction is performed on the original data to obtain a 512-dimensional high-dimensional feature vector. The precise distance between the high-dimensional feature vector and the high-dimensional query vector is calculated. The candidate results are then re-ranked, and the final query result is output. The process is as follows: Figure 5 As shown;

[0145] Experiments in this embodiment demonstrate that the CLIP model (ViT-B / 32) extracts 512-dimensional feature vectors from the MS-COCO dataset, which contains 123,287 images and a large amount of multimodal data, including text. The storage overhead of this invention is reduced by 35.76 times compared to directly storing the original vectors, and by half compared to storing only hashes. Furthermore, to explore the impact of multimodal data on blockchain usability, this invention tested the blockchain's throughput; the reduction in transaction size directly improved blockchain performance. In a four-node blockchain, due to the large transaction size (approximately 2KB), the original vector method could only achieve a processing speed of less than 100 transactions per second (TPS). In contrast, this invention has an average transaction size of 48 bytes, and with increased transaction arrival rate, the throughput can be stabilized at a maximum of approximately 3800 transactions per second, which is roughly equivalent to the performance of the hash algorithm. This indicates that the compression technology of this invention effectively improves the overall efficiency of the blockchain network.

[0146] This invention also tested query precision, achieving the highest accuracy (Recall@100 = 0.98, mean precision mAP@100 = 0.85) using the original vector method with full precision search. However, this invention combines on-chain verification with client-side precise reordering, achieving a recall@100 = 0.83 and a mean precision mAP@100 = 0.74, which is very close to the full precision benchmark. It is worth noting that without the reordering step, the mean precision mAP@100 drops to 0.69.

[0147] Example 2:

[0148] A multimodal data approximation query system based on hybrid blockchain, used to implement a multimodal data approximation query method based on hybrid blockchain, includes:

[0149] The feature extraction module is used to acquire the raw data of the multimodal data, extract features from it and perform unified vectorization to obtain a high-dimensional feature vector for each raw data.

[0150] The encoding module is used to perform compression and product quantization (PQ) encoding on each high-dimensional feature vector to generate PQ codes;

[0151] The index structure construction module is used to construct a verifiable index structure PQ-MPT based on the generated PQ code;

[0152] The anchoring module is used to write the root hash of the verifiable index structure PQ-MPT into the blockchain, and form an immutable global state commitment through consensus.

[0153] The query module is used to obtain user query requests and retrieve query results from the multimodal raw data using the verifiable index structure PQ-MPT.

[0154] Example 3:

[0155] This embodiment proposes an electronic device, including: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the aforementioned multimodal data approximation query method based on hybrid blockchain.

[0156] The electronic device may be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements a multimodal data approximation query method based on a hybrid blockchain as described in the embodiments. It is understood that the electronic device may also include input / output (I / O) interfaces and communication components.

[0157] The processor is used to execute all or part of the steps in the multimodal data approximation query method based on hybrid blockchain as described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in an electronic device, as well as application-related data.

[0158] The processor can be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute the multimodal data approximation query method based on hybrid blockchain described in the above embodiments.

[0159] Example 4:

[0160] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0161] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the multimodal data approximation query method based on hybrid blockchain described in various embodiments of this application.

[0162] The aforementioned storage media include: flash memory, hard disks, multimedia cards, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory), random access memory (RAM), static random-access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disks, optical discs, servers, APP (Application) app stores, and other media capable of storing program verification codes. These media store computer programs, which, when executed by a processor, can implement the various steps of the aforementioned multimodal data approximation query method based on hybrid blockchain.

[0163] Example 5:

[0164] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned multimodal data approximation query method based on a hybrid blockchain.

[0165] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0166] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0167] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of this disclosure and its equivalents, then the intent of this disclosure also includes these modifications and variations.

Claims

1. A method for approximate multimodal data query based on hybrid blockchain, characterized in that, Includes the following steps: Obtain the raw data of the multimodal dataset, and perform feature extraction and unified vectorization to obtain a high-dimensional feature vector for each raw data. Each high-dimensional feature vector is compressed and multiplied and quantized to generate PQ codes; Based on the generated PQ code, a verifiable index structure PQ-MPT is constructed; The construction of the verifiable index structure PQ-MPT based on the generated PQ code specifically includes: C1: Construct key-value pairs based on the PQ code corresponding to each piece of raw data; PQ code for each piece of raw data Calculate the SHA-256 hash to obtain a 256-bit cryptographic hash value. As the key; the content identifier (CID) of the original data in the distributed storage system is used as the value, thus obtaining key-value pairs. ; C2: Construct an MPT tree based on key-value pairs to obtain the index structure PQ-MPT; Write the root hash of the verifiable index structure PQ-MPT into the blockchain, and form an immutable global state commitment through consensus; Obtain user query requests and use the verifiable index structure PQ-MPT to retrieve query results from the multimodal raw data; The process of obtaining user query requests and retrieving query results from multimodal raw data using the verifiable index structure PQ-MPT specifically includes: E1: The user submits a query request through the client; E2: Preprocess the query request, extract deep features and vectorize it to obtain a high-dimensional query vector; E3: Using the PQ codebook For high-dimensional query vectors Compress the code to generate a query PQ code. ; E4: Cluster all the PQ codes of the original data into L clusters. Each cluster corresponds to an inverted list that stores the identifiers of all PQ codes belonging to that cluster. E5: Calculate the query PQ code Given the distances to the centers of the L clusters, select the τ clusters that are closest to them, and perform a candidate search only in the inverted list corresponding to the selected clusters to obtain the candidate results; E6: Generate a proof for each candidate result; E7: Verify the proof of the candidate results and retain the candidate results that pass the verification; E8: Perform precise reordering on the validated candidate results to obtain the final query results; Specifically, for the verified candidate results, the original data is downloaded using their Content Identifier (CID), features are extracted from the original data to obtain a high-dimensional feature vector, the precise distance between the high-dimensional feature vector and the high-dimensional query vector is calculated, the candidate results are reordered, and the final query result is output.

2. The method for approximate multimodal data query based on hybrid blockchain according to claim 1, characterized in that, The process of acquiring multimodal raw data, extracting features from it, and uniformly vectorizing it to obtain a high-dimensional feature vector for each raw data point specifically includes: A1: Retrieve raw multimodal data, including images, text, audio, and video, from a distributed storage system; A2: Preprocess the raw multimodal data to obtain preprocessed multimodal data; Specifically: For audio, first convert the audio into a waveform and extract the Mel spectrogram; For the image and Mel spectrogram, perform the following preprocessing operations: scale the image and Mel spectrogram to a set uniform size, then convert the scaled image and Mel spectrogram to the RGB three-channel color space, and normalize, zero-mean, and variance normalize the pixel values ​​to obtain the preprocessed image and Mel spectrogram. For text, perform word segmentation, stop word removal and lowercase conversion, and then truncate or pad to a fixed length; For videos, keyframe images are extracted from the video or several frames are obtained by sampling at fixed intervals. A3: Perform deep feature extraction and vectorization on the preprocessed multimodal data to obtain high-dimensional feature vectors and store them.

3. The method for approximate multimodal data query based on hybrid blockchain according to claim 1, characterized in that, The step of compressing and multiplying quantization encoding each high-dimensional feature vector to generate PQ codes specifically includes: B1: Transform high-dimensional feature vectors The space is divided into Subspaces, and then high-dimensional feature vectors Divide into uniformly along the dimensional axis Non-overlapping sub-vectors The number of subspaces is set, and each subvector corresponds to one subspace; B2: Each subspace is trained separately, and each subspace yields a local codebook, which is then combined to form the PQ codebook. ; Specifically: Extract high-dimensional feature vectors from all multimodal raw data and divide them uniformly into... After identifying non-overlapping sub-vectors, for each subspace, the k-means clustering algorithm is used to cluster all sub-vectors belonging to that subspace, resulting in clusters of... A central point forms a local codebook. , For the first Local codebooks of each subspace, For the first The first subspace One central point, Number the center point, then The PQ codebook is obtained by integrating the local codebooks. ; B3: PQ codebook After training, calculate its SHA-256 hash. It is stored on the blockchain and the PQ codebook is updated regularly to adapt to changes in data distribution; B4: Using the PQ codebook For high-dimensional feature vectors Perform product quantization encoding to obtain PQ codes; Specifically: for each subvector Calculate its relationship with the local codebook middle Find the optimal cluster center index by using the Euclidean distance between the center points: （1）； in, For the first The optimal cluster center index corresponding to each sub-vector; Then the high-dimensional feature vector of The optimal cluster center indices of the subvectors are combined sequentially to generate the final PQ code.

4. The method for approximate multimodal data query based on hybrid blockchain according to claim 1, characterized in that, The step of writing the root hash of the verifiable index structure PQ-MPT into the blockchain and forming an immutable global state commitment through consensus specifically includes: D1: Write the root hash of the index structure PQ-MPT into the blockchain; D2: Deploy a smart contract that stores the root hash of the index structure PQ-MPT and the SHA-256 hash of the PQ codebook. It provides the function of updating the root hash of the on-chain index structure PQ-MPT; D3: When the index structure PQ-MPT is updated, the smart contract is invoked to construct a transaction and combine the root hash of the new PQ-MPT with the SHA-256 hash of the PQ codebook. Stored on the blockchain.

5. A multimodal data approximation query system based on hybrid blockchain, used to implement the multimodal data approximation query method based on hybrid blockchain as described in any one of claims 1-4, characterized in that, include: The feature extraction module is used to acquire the raw data of the multimodal data, extract features from it and perform unified vectorization to obtain a high-dimensional feature vector for each raw data. The encoding module is used to compress and multiply quantize each high-dimensional feature vector to generate PQ codes; The index structure construction module is used to construct a verifiable index structure PQ-MPT based on the generated PQ code; The anchoring module is used to write the root hash of the verifiable index structure PQ-MPT into the blockchain, and form an immutable global state commitment through consensus. The query module is used to obtain user query requests and retrieve query results from the multimodal raw data using the verifiable index structure PQ-MPT.

6. An electronic device, characterized in that, include: One or more processors, and a memory for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform a multimodal data approximation query method based on a hybrid blockchain as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed, cause the processor to perform the multimodal data approximation query method based on a hybrid blockchain as described in any one of claims 1-4.

8. A computer program product, characterized in that, Includes a computer program or instructions that, when executed by a processor, implement the multimodal data approximation query method based on a hybrid blockchain as described in any one of claims 1-4.

Images