Compression index high-efficiency transmission verifiable encryption integration method for mass data

Abstract

The application discloses a kind of mass data-oriented compressed index efficient transmission verifiable encryption integrated method, by segmented Elias-Fano compressed inverted list, RRR bitmap and rANS entropy coding are constructed into compressed index structure that can be directly queried, adaptive block is carried out with α-β link model through network interface, realizes flexible transmission using RaptorQ fountain code and random linear network coding, using encryption coprocessor is encrypted with AEAD-SIV mode and establishes Merkle / KZG verifiable structure, in combination with SE-Bloom supports search.The application can realize index, transmission, encryption end-to-end cooperation, reduce bit cost and tail delay, guarantee integrity, tamper resistance and privacy.

Description

Technical Field

[0001] This invention belongs to the field of computer software and information security technology, and relates to an integrated method for efficient transmission, verification and encryption of compressed indexes for massive data, which can be applied to scenarios such as massive key-document inverted data, time-series logs, object metadata and vector ID lists. Background Technology

[0002] There is a clear contradiction between traditional inverted indexes (PostingList) and sparse bitmaps in terms of storage cost and I / O amplification: without compression, storage overhead is huge; after compression, decompression is usually required before querying, leading to a simultaneous increase in latency and memory usage. In wide area network or cross-data center transmission scenarios, the retransmission overhead and long-tail latency caused by packet loss and jitter are particularly significant; at the same time, cross-domain data transmission requires verifiable integrity, resistance to replay attacks, and privacy protection.

[0003] Most existing technical solutions only perform local optimizations within a single domain, such as index compression, transmission redundancy, or end-to-end encryption, making it difficult to achieve optimal end-to-end coordination among index construction, data stream transmission, and ciphertext verification. This invention addresses this fragmented situation by proposing an integrated solution supported by collaborative hardware modules. Summary of the Invention

[0004] The technical problem to be solved by this invention is: how to enable the compressed index of massive data to be queried directly without complete decompression, to resist packet loss jitter with adaptive redundancy in wide-area transmission, to maintain verifiable integrity and limited searchability in encrypted form, and finally to approach the optimal boundary between bit cost and end-to-end latency under the joint constraints of the three domains.

[0005] To address the aforementioned issues, this invention proposes an integrated method for efficient transmission, verification, and encryption of compressed indexes for massive datasets. Its core principles can be summarized in three points: First, treat "compressed formats that can be directly queried" as first-class citizens, with the index residing in the compressed area, and queries only accessing necessary bits. Second, treat "redundant transmission and coding rate" as runtime decision parameters, allowing the processor to dynamically adjust block size and fountain redundancy based on link α-β costs and real-time packet loss rate. Third, take "verifiability and limited searchability of ciphertext blocks" as security objectives, with the encryption coprocessor supporting integrity verification and equivalent queries without decryption, and without revealing the plaintext structure. Given the accuracy, recall, and security objectives, the entire system minimizes bit cost and end-to-end latency.

[0006] The specific technical solution is as follows: I. Compressed indexes that can be directly queried Let the ascending ID sequence be 0. <x1≤…≤x m≤U, where m is the list length (i.e., the number of elements in the posting), and U is the size of the ID universe (i.e., the largest ID plus 1). The upper bound of the bit cost in Elias-Fano is B. EF (m,U)=m log2(U / m) +2m+o(m). To reduce waste caused by local density fluctuations, the processor segments the sequence into blocks: P={P1,…,P...} K The segments are disjoint and their union is the sum of all IDs, with K segments in total. The goal is to minimize the sum of the costs of each segment. , where span(P k )=max(P k ) min(P k Let )+1 represent the segment span. Dynamic programming can provide an approximate solution in O(mlogm).

[0007] When accessing sequentially or jumping, the high-order bitmap H of EF supports selection operations. The low-order array L is a compressed area resident in memory; combined with the skip list gates (each ω element records a pointer) it supports O(logm) jumps, where ω is the gate interval.

[0008] For a sparse bitmap B∈{0,1} U RRR divides it into blocks and encodes it by the number of 1s within each block. Its bit cost is Where l is the block length, r j Let be the number of 1s in the j-th block. rank1(B,i) and select1(B,k) are completed in O(1) or O(logU).

[0009] Let the symbol distribution be p(s), and the zero-order entropy be... . is the inverted difference Δx i =x i x i-1 During encoding, using rANS can achieve code lengths close to entropy: , ε ans →0 increases with the number of states. Processors can improve throughput by approximately 4-8 times to scalar implementations through SIMD parallel batch processing.

[0010] The combined list access time of PEF+RRR+rANS is T. post (k)≈T gate +O(logm)+O(k). When introducing lossy quantization shift (optional) And |δ i When |≤Δ, And can be eliminated by a second table lookup; Δ is the upper bound of the error, T gateThis refers to the time for gate access.

[0011] II. Resilient and Predictable Transport Layer The link cost is approximately equal to Where α is the delay constant, β is the occupancy factor, and BW is the bandwidth. The processor divides the index and data into blocks of size S, with the block ID denoted as c. The optimal block size satisfies the joint constraints of the link model and retransmission redundancy. , where R(S,p) is the expected retransmission / redundancy overhead at packet loss rate p.

[0012] Using RaptorQ / LT to generate an infinite number of coded slices {y j The decoding success rate under packet loss rate p is} , κ=N recv / N src , where N src N represents the number of source chips. recv The number of received chips. When the number of received chips N recv ≥(1+ε)N src When ε is a small constant, decoding has a high probability of success. For cross-rack / cross-domain paths, the processor can perform random linear network coding (RLNC) at the convergence point: y = Gx, G ∈ F q m×n The full rank probability is approximately Choose q=2 8 In order to enable SIMD optimization.

[0013] End-to-end expected latency is , where q h T represents the compression / coding ratio on the link side. dec For decoding time. Processor co-optimization. , where δ is the upper bound of the failure probability.

[0014] III. Verifiable encryption and limited searchability The cryptographic coprocessor generates a random salt r for each block c. c Related data AD c (Includes version, offset, index summary, etc.), using SIV or XChaCha20-Poly1305: (C c ,tag c )=AEAD Enc (K;nonce c M c AD c ), where K is the key, nonce c For nonce, M c For plaintext, AD cIncludes version, offset, and index digest. SIVs are immune to risks caused by nonce duplication / out-of-order delivery, and ciphertext can be repeatedly transmitted / cached without compromising security.

[0015] ciphertext {C c} Construct a Merkle tree T:h as leaves c =H(C c ),root Receiver verification path π c Does it satisfy Verify(R,C)? c ,π c =true. Multinomial commitments (KZG) can also be used for large volumes of linearized data: construct... Announced commitment C=g f(s) Prove π for position i. i verify This allows us to obtain proof of the constant size and the ability to resist tampering.

[0016] To support finite equivalent queries for "set inclusion / simple range" without revealing plaintext, SE-Bloom is introduced: Let the tags Where PRF is a pseudo-random function. For each inverted block, k hash positions are recorded, and GolombCodedSet / GCS or Bloom is used to store the ciphertext tag positions. The false alarm rates are FPR. BF ≈(1 e -kn / m ) k FPR GCS ≈2 -b Where b is a Golomb parameter. The query only exposes τ and its hash image, not the plaintext term. For forward privacy, each update is performed by a cryptographic coprocessor rotating the salt r with the PRF subkey to block tag linkage.

[0017] IV. Unified Cost-Error-Security Optimization The combined storage and index bit cost is B PEF B RRR The bit costs for segmented Elias-Fano and RRR structures are H0N+ε, respectively. ans For entropy-encoded bits, N is the total number of symbols, and B is the number of bits. aux These are auxiliary bits for gate position, checksum, and tag.

[0018] Given concurrency B, I / O bandwidth BW, and cache hit rate H, the query latency is ,in For the number of bits reached, pruning is performed based on gate position and range. This refers to the random seek time.

[0019] Safety and Error Probability Boundaries: Probability of Successful AEAD Forgery λ is the number of tags; Merkle / KZG false pass probability. Or at most 1 / p; the false alarm threshold of SE-Bloom / GCS is as above, and the false alarm can be reduced to 0 by a second table lookup.

[0020] This invention achieves query performance similar to plaintext without exposing the plaintext index, while simultaneously obtaining verifiable integrity and tamper resistance. Direct querying of the compressed index avoids the memory and latency overhead of complete decompression; elastic transmission adaptively adjusts redundancy based on real-time link status, significantly suppressing tail latency; verifiable encryption supports integrity verification and limited search at the ciphertext level, meeting compliance requirements for cross-domain data transfer. Joint optimization across three domains ensures that storage bit cost, transmission latency, and security strength converge collaboratively under a unified objective function, rather than being isolated local optima. Detailed Implementation

[0021] The specific embodiments of the present invention will be described in detail below.

[0022] 1. Construction of PEF Inverted Index and RRR Bitmap The processor first reads the original document or object ID from memory and sorts it into sequence 0 in ascending order. <x1≤…≤x m ≤U, where m is the list length and U is the size of the ID universe. Then, the difference Δx is calculated. i =x i x i-1 The processor performs dynamic programming, dividing the sequence into disjoint segments P = {P1, ..., P2} in O(mlogm) time. K}, so that the sum of the Elias-Fano bit costs of each segment ,in For each segment, a high-order bitmap H and a low-order array L are generated. H supports the select1 operation, and L resides in the compressed memory area. At the same time, a gate pointer is created for every ω elements and written to the auxiliary structure area of ​​memory to support subsequent O(logm) jumps.

[0023] For sparse attributes or filtered columns, the processor constructs a sparse bitmap B∈{0,1} U Divide the block into blocks of length l using RRR, and then divide the block into blocks of length r based on the number of 1s in each block. j Class encoding is performed, with a bit cost of 1. The rank1 and select1 auxiliary tables of RRR are stored together in memory, so that subsequent queries can be completed in O(1) or O(logU).

[0024] Difference sequence Δx iThe processor writes the rANS entropy code to memory. The expected code length of rANS is... ,in Given zero-order entropy, p(s) is a symbolic distribution, and ε is redundant. ans The state number approaches 0 as it increases. The processor utilizes SIMD instructions to batch process multiple symbols in parallel, increasing encoding throughput to 4-8 times that of scalar implementations. The final memory layout is as follows: The gate position table and the RRR auxiliary table are located in the auxiliary area.

[0025] 2. Query and intersection / union operations When an external request arrives, the processor reads the term set {t} from memory. i The corresponding posting list. For each list, the processor uses gate pointers for jump positioning to find the starting segment in O(logm) time, and then sequentially decodes the rANS differential recovery ID. If multiple lists are involved in intersection, the processor aligns the current segment of each list with gate shifts and sequentially decodes within the intersection candidate range to avoid full decompression.

[0026] If the system enables lossy quantization displacement i =x i +δ i And |δ i If |≤Δ, where Δ is a preset upper bound for error, then candidate results may be falsely rejected due to displacement, and the false rejection probability satisfies the following condition. The processor performs a secondary lookup correction on the candidate ID: it returns to the original uncompressed index or a higher precision level to confirm whether the ID truly meets the query conditions, thereby reducing false rejections to zero.

[0027] During range queries or Top-K queries, the processor first jumps to the vicinity of the target interval using the high-order bitmap H of the PEF and the select1 operation. Then, it uses the RRR bitmap for filtering, decoding only the low-order array L of the filtered segments, significantly reducing the number of bits reached (B). touched .

[0028] 3. Transmission and Decoding Before transmission, the processor measures the current link parameters: delay constant α, occupancy factor β, bandwidth BW, and packet loss rate p. This is based on a link overhead model. The processor solves for the optimal block size. This minimizes the combined cost of end-to-end expected delay and retransmission redundancy, where R(S,p) is the expected retransmission / redundancy cost at packet loss rate p.

[0029] The processor then divides the index and data into blocks. Fragments are created, with block IDs denoted as c. The network interface generates an infinite number of coded fragments {y} using RaptorQ or LT fountain codes.j} and send. The receiving end's network interface continuously receives encoded fragments, and when the number of received fragments reaches a certain threshold... At that time, decoding success rate It is highly likely to be true, where κ=N recv / N src The receiving processor can restore the original data without waiting for the specific lost fragments to be retransmitted.

[0030] In cross-rack or cross-domain paths, the processor at the intermediate aggregation node executes a random linear network code y=Gx, with the coefficient matrix G∈F. q m×n Choose q=2 8 For SIMD optimization. The full-rank probability is approximately This ensures a high probability of decodeability. After decoding, the receiver processor calculates the end-to-end delay and compares it with the target. If necessary, it feeds the measurement results back to the transmitter processor for use in the next round. Adjustment of redundancy rate κ.

[0031] 4. Verifiable encryption and SE-Bloom query The cryptographic coprocessor reads the plaintext block M to be transmitted from memory. c Generate a random salt r for each block c. c Related data AD c AD c It includes context such as version number, offset, and index digest to ensure the binding relationship between blocks. The cryptographic coprocessor uses SIV or XChaCha20-Poly1305 to perform AEAD encryption, and the output... Where K is the master key and nonce c Derived from salt and block ID. SIV mode resists the risk of nonce duplication or out-of-order delivery, allowing ciphertext blocks to be repeatedly transmitted or cached in the network without compromising security.

[0032] After encryption is complete, the encryption coprocessor will process each block of ciphertext C c Construct a Merkle tree using leaves: compute the leaf hash h c =H(C c The root R is obtained by pairing hashes layer by layer. Root R is then published to the receiver via a trusted channel. The receiver receives block C. c Then, request verification path π from the sender or trusted node. c By Verify(R,C) c ,π c This confirms that the block has not been tampered with. For large volumes of linearized data, the cryptographic coprocessor can also construct polynomials. KZG issued a commitment that C=g f(s) Prove π for any position i. isatisfy The verifier only needs to prove the size of the constant to confirm C. i The authenticity of.

[0033] To support finite equivalence queries, the cryptographic coprocessor maintains an SE-Bloom structure. For each search term, tags are calculated. Where PRF is a pseudo-random function. For each inverted block, k hash positions are recorded, and GolombCodedSet or Bloom is used to store the ciphertext tag positions. The querying party only submits... The hash image of the term is not exposed. The cryptographic coprocessor determines "potential inclusion" on SE-Bloom; if a match is found, the corresponding ciphertext block is returned for decryption verification. For forward privacy, each time the index is updated, the cryptographic coprocessor rotates the salt r and the PRF subkey, making the tags of the same term unlinkable at different times, thus blocking temporal correlation analysis.

[0034] 5. Parameter Adaptation and Online Reconfiguration The processor periodically estimates the workload and link status: cache hit rate H, packet loss rate p, bandwidth BW, and query distribution. Based on these measurements, the processor updates the number of segments K and segment boundaries of the segment set P using subgradient or greedy search, ensuring the total bit cost B... total Approaching the optimality of the current working set. Simultaneously, the processor adjusts the fountain redundancy rate κ and the link compression ratio q. h This makes the end-to-end expected delay E[T] e2e Under the decoding success rate constraint P succ ≥1-δ and mean square error constraint MSE(q) h It remains at its minimum when ε ≤ ε.

[0035] Regarding security parameters, the encryption coprocessor adjusts the AEAD tag bit length λ, Bloom / GCS parameter b, and PRF rotation cycle according to the threat model rotation strategy. The processor writes the new parameter configuration to the configuration area of ​​memory and notifies the network interface to switch the encoding parameters. The entire reconfiguration process is completed online without interrupting the query service. The new configuration takes effect when the next batch of data is written, and the data already written remains compatible with the old parameters until it is re-indexed.

Claims

1. A method for efficient transmission, verification, and encryption of compressed indexes for massive amounts of data, characterized in that: include: Construct a compressed index that can be directly queried: Divide the ascending identifier sequence into blocks in memory, build a compressed inverted index for each block, construct a compressed bitmap for sparse attributes, and use entropy coding to encode and decode the difference sequence so that the index can be directly queried without complete decompression; Elastic transmission: The processor fragments the index and data with adaptive block size based on the link overhead model, generates coded fragments through the network interface using forward error correction coding, and transmits them. Network coding is performed at the cross-domain path convergence point. Verifiable encryption: Each block is encrypted in a certified encryption mode by the cryptographic coprocessor and an authentication tag is attached. The ciphertext of the block is organized into a verifiable tree structure or a multinomial commitment structure, generating verifiable roots and proof paths, and establishing a searchable tag structure to support limited equivalent queries without revealing the plaintext. Unified optimization: The processor jointly optimizes the segmentation strategy, coding redundancy rate, link compression ratio and security parameters to minimize storage bit cost and end-to-end latency under given accuracy, recall and security objectives.

2. The method according to claim 1, characterized in that, The construction of a directly queryable compressed index includes: setting the ascending ID sequence to 0. <x1≤…≤x m ≤U, where m is the length of the list, i.e., the number of elements in the posting, and U is the size of the ID universe, i.e., the maximum ID plus 1; divide the sequence into blocks as P={P1,…,P... K The segments are disjoint and their union is the total number of IDs, with K segments in total. The optimal segmentation is approximated in O(mlogm) using dynamic programming, minimizing the sum of the Elias-Fano bit costs of each segment, where a single segment P... k The upper bound of the unit cost is B. EF (|P k |,span(P k )), while span(P k )=max(P k ) min(P k )+1 represents the segment span; the high-order bitmap H of each segment supports selection operations. The low-order array L resides in the compressed region, and combined with the skip list gates where each ω element records a pointer, it supports O(logm) jump access, where ω is the gate interval; for the sparse bitmap B∈{0,1} U Divide into blocks using RRR and classify by the number of 1s within each block; the bit cost is... Where l is the block length, r j Let be the number of 1s in the j-th block; make rank1(B,i) and select1(B,k) complete in O(1) or O(logU); for the inverted index difference Using rANS encoding, the expected code length is achieved. Near-zero entropy Where p(s) is the symbol distribution and the redundancy ε ans →0 increases with the number of states; the processor increases rANS throughput to scalar levels several times through SIMD parallel batch processing.

3. The method according to claim 2, characterized in that, Lossy quantization shifts can be optionally introduced during index construction. And |δ i |≤Δ, where Δ is the upper bound of the error; at this time, the false rejection probability satisfies And false rejections are eliminated by performing a second table lookup.

4. The method according to claim 1, characterized in that, The elastic transmission includes: the processor based on a link overhead model. Determine the adaptive block size, where α is the delay constant, β is the occupancy factor, and BW is the bandwidth. The number of bytes transmitted; an infinite number of coded chips {y} are generated by the network interface using RaptorQ or LT fountain codes. j }, so that the decoding success rate P under the packet loss rate p. succ ≈1 exp( γ(κ κ0), where κ=N recv / N src For the number of received chips N recv With the number of source chips N src The ratio, when N recv ≥(1+ε)N src Decoding is successful when ε is a small constant, and γ and κ0 are success rate parameters; at the convergence node of a cross-rack or cross-domain path, the processor performs random linear network coding y=Gx, where G∈F q m×n For the coefficient matrix, choose q=2 8 For SIMD optimization, the full-rank probability is approximately .

5. The method according to claim 4, characterized in that, The processor jointly optimizes the end-to-end expected latency. , where q h T represents the compression / coding ratio on the link side. dec For decoding time; while satisfying the decoding success rate constraint P succ ≥1 δ and mean square error constraint MSE(q) h Minimize the condition that )≤ε , where δ is the upper bound of the failure probability.

6. The method according to claim 1, characterized in that, The verifiable encryption includes: the encryption coprocessor generating a random salt r for each block c. c Related data AD c AEAD encryption is performed using SIV or XChaCha20-Poly1305, outputting ciphertext C. c With certification tag c That is, (C) c ,tag c )=AEAD Enc (K;nonce c M c AD c ), where K is the key, nonce c For nonce, M c For plaintext, AD c Includes version, offset, and index digest; ciphertext C of each block c Merkle trees are constructed using leaves, and leaf hashes are used to construct them. c =H(C c The root R is obtained through layer-by-layer hash calculation, and the verification path π is used. c Satisfy Verify(R,C) c ,π c =true; or construct a polynomial for a large amount of linearized data. Announced commitment C=g f(s) Prove π for position i i verify This provides proof of the constant size and the ability to resist tampering, where p is the modulus of a finite field.

7. The method according to claim 6, characterized in that, The searchable tag structure is an SE-Bloom structure: Let the tag =w(PRF K (term)), where PRF is a pseudo-random function, and w(·) is a tag mapping function; k hash positions are recorded for the inverted block, and GolombCodedSet or Bloom is used to store the ciphertext tag positions; Bloom false positive rate FPR BF ≈(1 e -kn / m ) k GCS false alarm rate (FPR) GCS ≈2 -b Where b is the Golomb parameter, k is the number of Bloom hashes, m is the size of the bit array, and n is the number of elements; the query only exposes... It also contains the hash image without exposing the plaintext term; each time it is updated, the cryptographic coprocessor rotates the salt r with the PRF subkey to block tag linkability.

8. The method according to claim 1, characterized in that, The unified optimization includes: calculating the total bit cost by the processor. B PEF B RRR The bit costs for segmented Elias-Fano and RRR structures are H0N+ε, respectively. ans For entropy-encoded bits, N is the total number of symbols, and B is the number of bits. aux Auxiliary bits for gate position, checksum, and tag; and calculate query latency. Where B is the concurrency level, H is the cache hit rate, and T is the cache hit rate. seek For random seek time, B touched To query the number of bits reached, T decode This is the decoding time.

Images