Database query verification method and electronic device

Abstract

The application relates to the technical field of database verification, and discloses a database query verification method and electronic equipment. The method comprises the following steps: receiving a structured query request, and generating a query execution plan comprising a plurality of physical operators; executing the query execution plan, so that at least part of the physical operators process input data in a streaming batch processing mode; in the process of physically processing a current data batch, a verification load corresponding to the current data batch is synchronously constructed; after the verification load is constructed, a corresponding proof generation task is submitted to an asynchronous execution process independent of a main calculation path of the physical operators, so that a corresponding verification program is called to generate a batch proof; and after a plurality of data batches generate batch proofs respectively, the plurality of batch proofs are aggregated to obtain an aggregated proof used for verifying a query result. The scheme can adapt to a database streaming batch processing process, and reduce the influence of proof generation on a main execution path.

Description

Technical Field

[0001] This invention relates to the field of verifiable database technology, specifically to a database query verification method and electronic device. Background Technology

[0002] Verifiable database technology is used to verify the correctness and completeness of query results in scenarios where the entities storing and executing data are not entirely trustworthy. This type of technology is typically applied in cloud databases, third-party data processing platforms, and multi-party data collaboration environments. In these scenarios, while users can submit structured query requests to the database system and receive query results, they usually cannot observe whether the server has completed all computational steps according to the predetermined query semantics. Therefore, a mechanism is needed to technically verify the query execution results.

[0003] To explain this issue, we need to first introduce the basic execution method of database queries. After receiving a structured query request, a database system typically doesn't execute the query statement verbatim. Instead, it first generates an execution plan based on the processing objectives expressed in the query. An execution plan can be understood as a set of processing steps and their organizational relationships formed by the database system to complete a query. Execution plans generally include two levels: logical plans and physical plans. The logical plan mainly describes what processing needs to be performed in the query; it can be understood as "what to do." For example, first, filtering records that meet the conditions, then joining records from different data sources according to specified fields, and finally performing statistical analysis on the join results. The physical plan, based on the logical plan, describes "how to do it specifically," further determining the specific execution methods for each process and how data is transferred between processing steps. For example, which traversal method is used when performing filtering, whether hash joins or other join methods are used when performing joins, and what accumulation structure is used when performing aggregations. Therefore, the "physical" in the physical plan does not refer to physical entities, but rather, relative to the query semantics at the logical level, it represents the implementation method facing the actual execution process.

[0004] A physical plan typically consists of multiple physical operators. Physical operators are execution units within a database execution engine responsible for performing specific data processing actions. Essentially, they are still part of a computer program, but in the database domain, they perform a specific type of data processing function. For example, filtering operators are used to select records from input data that meet given conditions; join operators are used to establish matching relationships between two input data streams based on a join key; and aggregation operators are used to merge input data according to a grouping key and calculate sums, counts, etc. Multiple physical operators connected in a upstream-downstream relationship form the actual data processing chain during query execution. The query result is obtained by these physical operators processing the data step by step. For the same query target, the database system may also use different combinations of physical operators or different execution paths depending on the selected physical plan. Therefore, the generation of query results is not only related to the query semantics themselves but also to the processing procedures of each physical operator during actual execution.

[0005] In many database systems, especially in streaming execution engines, physical operators typically advance the processing step-by-step according to data batches. That is, after an upstream operator outputs a batch of data, a downstream operator can immediately begin processing that batch without waiting for all the data to be prepared. This improves the continuity and throughput of data processing. Therefore, database queries in actual operation are usually not a one-time, complete computation, but rather a process in which multiple physical operators continuously execute around different data batches, continuously transmitting intermediate results.

[0006] In existing technologies, a common query verification scheme is to validate query results based on an authentication data structure. For example, the server can build an authentication structure such as a Merkle tree for the original data and include authentication path or digest information related to the result when returning the query result. The client then uses this authentication information to verify whether the result originates from a given dataset and whether the result has been tampered with or omitted. The key to this type of scheme is establishing a correspondence between the query result and the authentication digest, enabling the verifier to determine the credibility of the result based on the result and its accompanying proof.

[0007] However, judging from the implementation of this type of scheme, it mainly verifies the consistency between the result and the underlying authentication structure, focusing on answering the question of "whether the returned result can be supported by the authentication structure." The generation of database query results is usually not just a simple read of a static data set, but involves continuous processing by multiple physical operators. For example, a query might first use a filtering operator to select records, then a join operator to combine data from different sources, and finally an aggregation operator to form statistical results. For this type of query, the final result is not only related to the original data, but also closely related to how each physical operator processes the input data and generates the output data. Because authentication data structure schemes rarely directly describe the data processing procedures within physical operators, it is often difficult to perform fine-grained verification of the entire execution chain when a query involves the continuous execution of multiple physical operators.

[0008] To improve coverage of complex queries, another approach in existing technologies involves incorporating the entire query statement or execution plan as a whole into a unified proof process. In this approach, the system represents the processing logic of multiple physical operators as a unified verification procedure, and then generates a proof of the query results based on this unified verification procedure. Compared to schemes that only verify the result state, this holistic verification method can cover a more complete query semantics in form.

[0009] To improve coverage of complex queries, another approach in existing technologies involves incorporating the entire query statement or execution plan as a whole into a unified proof process. In this approach, the system typically first constructs a unified verification program based on the query statement or execution plan, capable of covering the processing logic of multiple physical operators. Then, the input data, intermediate results, operator parameters, and data transfer relationships between operators involved in the query execution process are organized into proof inputs acceptable to this unified verification program. The unified verification program then centrally verifies the entire query process and generates a proof corresponding to the query result. Compared to schemes that only verify the result state, this holistic verification method no longer focuses solely on the consistency between the final result and a specific digest, but attempts to include the processing logic of multiple physical operators within the verification scope, thus formally covering a more complete query semantics.

[0010] However, database execution engines typically operate with clear operator-level and batch-level boundaries. In other words, query execution doesn't usually proceed by first generating a complete global processing result and then uniformly moving downwards; instead, it's completed step-by-step by multiple physical operators working around batches of data. Each physical operator, when processing its current batch of data, generates corresponding input data, output data, operator parameters, and intermediate processing information. If the verification scheme adopts this holistic approach, this information, generated gradually during operator processing, usually cannot immediately enter the proof generation process as an independent unit after a batch of data is processed. Instead, it needs to wait for the relevant information from subsequent operators to be formed, or for the data relationships of more batches to be completed, before being uniformly organized and arranged according to the requirements of the entire query process. In other words, this type of scheme isn't impossible, but the organization of its verification information usually relies more on the global query context, making it difficult to promptly separate along the natural boundaries of operators and batches.

[0011] This further reveals that, under a holistic verification path, the proof generation process often relies on pre-organized global verification information, which itself is gradually accumulated from the processing results of multiple operators and data batches. Consequently, a strong dependency easily forms between the query computation process and the proof generation process. In other words, proof generation is unlikely to start promptly upon the completion of a particular physical operator's processing of a particular data batch, but is more likely to be initiated later in the query execution. As a result, verification-related data and states tend to accumulate continuously during query execution, and verification overhead is difficult to gradually distribute as data batches progress.

[0012] One of the advantages of streaming execution engines in data processing is their ability to continuously advance query computations in batches, creating a seamless processing flow between upstream and downstream operators. If the verification process cannot proceed synchronously along this batch-based approach and relies primarily on subsequent centralized processing and unified proof, then the verification process struggles to achieve stable coordination with the streaming computation process. This not only hinders the utilization of the streaming execution engine's ability to process data continuously in batches but also tends to concentrate verification overhead in the later stages of query execution, thus increasing its impact on the main execution path. Therefore, how to organize verification information around the batch execution process of physical operators and how to better coordinate the proof generation process with the query computation process to reduce the impact of verification overhead on the main execution path and improve verification efficiency in complex query scenarios remain unresolved technical challenges in verifiable database technology. Summary of the Invention

[0013] The purpose of this invention is to provide a database query verification method and electronic device to alleviate the problem that the database query verification process and the streaming execution process are difficult to coordinate effectively in the prior art, resulting in high verification overhead in complex query scenarios.

[0014] To address the aforementioned technical problems, in one aspect of this application, a database query verification method is provided, comprising: receiving a structured query request and generating a query execution plan including multiple physical operators based on the structured query request; executing the query execution plan such that at least a portion of the multiple physical operators process the input data in a streaming batch processing manner; synchronously constructing a verification payload corresponding to the current data batch during the processing of the current data batch by any of the physical operators in the streaming batch processing manner; after the verification payload is constructed, submitting a proof generation task corresponding to the current data batch to an asynchronous execution process independent of the main computation path of the physical operator, so as to call the verification program corresponding to the physical operator to generate a batch proof; after the multiple data batches generate batch proofs respectively, aggregating the multiple batch proofs to obtain an aggregated proof used to verify the query result corresponding to the structured query request.

[0015] In some implementations, the verification payload includes at least the input data, output data, operator parameters, and verification information characterizing the physical operator processing of the current data batch.

[0016] In some implementations, the verification information includes an execution trace, which is used to characterize the operation process information of the physical operator in processing the current data batch. The operation process information includes at least one of instruction-related information, memory access information, and state change information.

[0017] In some implementations, generating batch proof by calling the verification program corresponding to the physical operator includes: calling the verification program corresponding to the physical operator, performing replay verification or constraint checks on the process of the physical operator processing the current data batch based on the input data, output data, operator parameters and the verification information, and generating the batch proof.

[0018] In some implementations, the verification program runs in a zero-knowledge virtual machine, which generates zero-knowledge proofs about the execution process of the verification program while executing the verification program, and the batch proofs include the zero-knowledge proofs.

[0019] In some implementations, before submitting the proof generation task corresponding to the current data batch to the asynchronous execution process, the method further includes: converting the data format in the database execution engine into a native data structure that the verification program can read, and writing the converted data into the verification payload.

[0020] In some implementations, submitting the proof generation task corresponding to the current data batch to an asynchronous execution process independent of the main computation path of the physical operator includes: encapsulating the verification payload and the identifier or reference of the verification program into an asynchronous task unit, and submitting it to the proof task scheduler in a non-blocking manner, so that the physical operator can continue to process subsequent data batches after submission.

[0021] In some implementations, the aggregation of the multiple batch proofs includes: using a recursive proof method to aggregate the multiple batch proofs layer by layer to generate the aggregated proof.

[0022] In some implementations, the plurality of physical operators include at least one of filtering operators, connection operators, and aggregation operators, and the verification procedure verifies the processing results of the current data batch according to a predetermined processing semantic consistent with the corresponding physical operator.

[0023] In another aspect of this application, an electronic device is also provided, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it causes the electronic device to perform the database query verification method described above.

[0024] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects:

[0025] This invention embeds the construction process of the verification payload into the streaming batch processing of the physical operator, enabling the operator to simultaneously extract verification information related to the current data batch while performing data processing. Since the verification information is not centrally organized after the query is completed, but is gradually formed along the batch in the operator processing path, the centralized processing pressure caused by the post-collection of verification information can be reduced, which is beneficial for the query verification process to better adapt to the batch advancement mechanism of the streaming execution engine.

[0026] In this embodiment of the invention, after the verification payload is constructed, the proof generation task is submitted to an asynchronous execution process independent of the main computation path of the physical operator. This allows the physical operator to continue processing subsequent data batches after the task is submitted. Therefore, the query computation process and the proof generation process can proceed concurrently, and the verification overhead can be distributed along the data processing timeline, rather than being concentrated and blocked at the end of the query execution, thereby helping to improve the overall execution efficiency of database query verification.

[0027] This invention further aggregates multiple batch proofs into aggregated proofs. This preserves the processing granularity of batch verification while generating more compact verification results in the output stage, thus balancing scalability in the streaming process with the consistency of the final verification results.

[0028] In some embodiments, the verification program performs verification according to verification logic consistent with the predetermined semantics of the corresponding physical operator, enabling the processing of different physical operators to be proven separately with their corresponding verification semantics. By breaking down complex queries into batch verification processes oriented towards physical operators, the complexity increase problem caused by holistic unified verification can be mitigated, thus making it more conducive to query verification implementation in complex query scenarios. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the overall architecture of the database query verification system.

[0031] Figure 2 This is a diagram showing the execution plan and physical operator links.

[0032] Figure 3 This is a schematic diagram of the batch execution process of data batches in the physical operator link.

[0033] Figure 4 This is a schematic diagram showing the correspondence between the operator execution process and the verification load construction process.

[0034] Figure 5 A schematic diagram of the data structure used to verify the load.

[0035] Figure 6 This is a schematic diagram of the asynchronous proof scheduling structure.

[0036] Figure 7 This diagram illustrates the interaction between the verification program and the verification payload.

[0037] Figure 8 This is a schematic diagram illustrating the execution and verification process of the filtering operator.

[0038] Figure 9 This is a schematic diagram illustrating the execution and verification process of the connection operator.

[0039] Figure 10This is a schematic diagram illustrating the execution and verification process of the aggregation operator.

[0040] Figure 11 This diagram illustrates the format conversion relationship between the payload and the proof execution environment to verify the payload.

[0041] Figure 12 This is a schematic diagram of the piecewise proof and recursive aggregation process.

[0042] Figure 13 This is a schematic diagram illustrating the basic principle of zero-knowledge proof in database operator verification scenarios. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0044] Before proceeding with each embodiment, let's first combine... Figure 13 This paper provides a unified explanation of the basic principles of zero-knowledge proofs in database operator verification scenarios. Figure 13 The fundamental relation of zero-knowledge proof applicable to this scheme is shown. Figure 13 In this system, the zero-knowledge proof system 1300 may include a prover 1301, a verifier 1302, a verification relation 1303, witness data 1304, common input 1305, batch proof 1306, proof parameters 1307, and verification parameters 1308. The zero-knowledge proof described here does not involve an isolated proof of a specific query result, but rather a verifiable expression of the processing of the current batch of data by the database physical operators. In other words, the content to be proven is usually not merely the existence of a single output result, but rather the existence of a set of witness data 1304 that explains how the output result is formed by the input data, operator parameters, and intermediate processing information, such that this set of witness data 1304, together with the common input 1305, satisfies the constraints defined in the verification relation 1303.

[0045] Figure 13Verification relation 1303 is used to describe the pre-processing semantics of database operators. The pre-processing semantics can be understood as the relationships that a certain type of physical operator should satisfy between input, intermediate processing states, and output, given input data and operator parameters. For filtering operators, verification relation 1303 can be used to characterize the correspondence between input records, filtering conditions, filtering flags, and output records, i.e., which input records satisfy the predicate conditions, which input records should be retained, and whether the output record comes exactly from the input records that satisfy the conditions. For join operators, verification relation 1303 can be used to characterize the correspondence between left and right input records, join keys, matching relationships, and output records, i.e., which left and right record pairs satisfy the join conditions, and whether the output record is obtained by combining record pairs that satisfy the conditions. For aggregation operators, verification relation 1303 can be used to characterize the update relationship between input records, grouping keys, aggregation functions, intermediate states, and final output, i.e., which input records belong to the same group, how the cumulative state corresponding to a certain group changes, and whether the final output is consistent with the cumulative state. It can be seen from this that in this scheme, the verification relation 1303 is not an abstract cryptographic object detached from the database operator, but rather a formal expression carrier for the semantic processing of the database operator.

[0046] Figure 13 Witness data 1304 is used to provide specific values ​​for the variables in verification relation 1303. In this scheme, witness data 1304 is generally not limited to the underlying raw input data, nor is it merely the final output data itself. Instead, it includes the input data, output data, operator parameters, and verification information characterizing the operator processing for the current data batch. This verification information can vary depending on the operator type. For filtering operators, verification information may include filter bitmaps, predicate evaluation results, or execution traces related to the filtering process. For join operators, verification information may include matching hints, index pairs, intermediate index information formed during the construction phase, or execution traces related to the join matching process. For aggregation operators, verification information may include grouping status, intermediate cumulative results, status update process records, or execution traces related to the status update process. This organization method is adopted because the correctness of database operators usually cannot be directly judged based solely on input and output. In many cases, without intermediate processing information, the verifier can only know that a certain output exists, but cannot determine whether that output was obtained by a processing process that conforms to the operator's semantics. Therefore, witness data 1304 is not arbitrary information temporarily added outside of the proof, but a set of necessary values ​​that enable the verification relation 1303 to be instantiated and successfully proven.

[0047] Figure 13The public input 1305 corresponds to the input content visible to and involved in verification on the verification side. There is a division of labor between public input 1305 and witness data 1304. Witness data 1304 mainly carries private or intermediate information that does not need to be explicitly exposed but must participate in the calculation during the proof process; public input 1305 mainly carries information that can be publicly used during verification to determine the boundaries of the proof object and the attribution of the proof result. In this scheme, public input 1305 may include at least one of the following: input data summary, output data summary, operator type identifier, batch identifier, query identifier, and necessary version identifier. For batch-level proofs, public input 1305 can indicate which physical operator, which data batch, and the summary boundaries of the batch's input and output correspond to the current proof. For subsequent aggregated proofs, public input 1305 can also serve as a basis for connecting different batch proofs. For example, the output summary of the upstream operator can serve as a corresponding reference for the input summary of the downstream operator, and multiple batch proofs can also be connected through batch identifiers and summary relationships. Using a summary instead of the raw data as part of the common input 1305 reduces exposure to the underlying data content and makes it easier to control the input size during recursive aggregation.

[0048] Figure 13 In this scheme, the prover 1301 typically corresponds to the database service side that executes database queries and generates proofs, or a proof node that works in collaboration with the database service side. The prover 1301 possesses witness data 1304 and can access proof parameters 1307 used to execute the proof algorithm. Upon receiving the current data batch corresponding to a certain physical operator, the prover 1301 can organize the input data, output data, operator parameters, and verification information corresponding to that data batch into witness data 1304. It then organizes the input / output digest, operator identifier, and batch identifier into common input 1305, and runs the proof algorithm according to the verification relation 1303 to generate batch proof 1306. The batch proof 1306 does not require explicit exposure of the witness data 1304 itself, but it can convince the verification side that a set of witness data 1304 does exist, such that the operator processing of the current data batch satisfies the verification relation 1303. This is of practical significance for this scheme. The database service side does not need to directly expose all the original data and intermediate states in the current data batch to the external verifier, nor does it need to make the verifier re-execute the complete database calculation process. Instead, it uses batch proof 1306 to express that the current batch processing process meets the predetermined semantics.

[0049] Figure 13The verifier 1302 receives the public input 1305, batch proof 1306, and verification parameters 1308, and runs a verification algorithm to determine whether the batch proof 1306 is valid. The verifier 1302 can be an external user of the result or an internal aggregation module responsible for checking the integrity of the proof. If the verification passes, it can be assumed that at least one set of witness data 1304 satisfying the verification relationship 1303 exists, corresponding to the operator processing of the current batch. If the verification fails, it indicates that the current batch has not been successfully proven, or there is an inconsistency between the input-output relationship and the claimed operator processing semantics. Since the verifier 1302 typically does not directly possess the complete witness data 1304, the verification action relies on the cooperation between the public input 1305 and the batch proof 1306. Therefore, the selection of fields in the public input 1305 needs to both identify the object of proof and avoid introducing unnecessary data exposure.

[0050] Figure 13 The proof parameter 1307 and verification parameter 1308 in the code typically originate from the parameter preparation process corresponding to the verification relation 1303. These parameters can take different forms in different zero-knowledge proof systems. Some systems use explicitly distinguishable proof and verification keys, while others use a common set of parameters bound to the program execution environment. In this scheme, the naming convention of the parameters is less important than the fact that the proof side holds the parameters used to generate batch proof 1306, and the verification side holds the parameters used to verify batch proof 1306. This ensures that the verification phase does not require re-executing the entire processing of the database operator. For the verification program running in the zero-knowledge virtual machine, the proof parameter 1307 and verification parameter 1308 can also be represented as environment parameters or their equivalent structures related to the loading of the verification program, the generation of the execution trajectory, and the construction of the proof. With this processing, the processing semantics of the database operator can be incorporated into the provable execution environment through the corresponding verification program, without having to directly move the original database kernel code to the verification side for re-running.

[0051] from Figure 13The relationships shown further illustrate that, in this scheme, the key to zero-knowledge proof is not merely outputting a formal proof result, but rather completing a set of stable mappings. First, the processing semantics of the database physical operators need to be organized as verification relation 1303. Second, the input data, output data, operator parameters, and verification information corresponding to the current data batch need to be organized as witness data 1304. Third, the input-output boundaries and the proof object identifier need to be organized as common input 1305. Fourth, the prover 1301 executes the proof algorithm on witness data 1304 and common input 1305 based on proof parameters 1307, generating batch proof 1306. Fifth, the verifier 1302 executes the verification algorithm on batch proof 1306 based on verification parameters 1308 and common input 1305, thereby determining whether the operator processing of the current data batch is acceptable. Based on this chain, the verification payload in subsequent embodiments is not just an ordinary data encapsulation object, but an organization carrier of witness data 1304; the execution trace, intermediate state, matching prompts and filtering bitmaps are not arbitrarily added information, but intermediate witnesses used to support the establishment of verification relationship 1303.

[0052] Figure 13 The underlying principle illustrated is also directly related to the subsequent recursive aggregation process. A single batch proof 1306 typically only demonstrates that the processing of a specific batch of data by a particular physical operator satisfies the predetermined processing semantics; it cannot independently prove that the entire query chain has been verified. Only when multiple batch proofs 1306 are further organized and aggregated according to input-output summary relationships, batch order relationships, or upstream-downstream operator relationships can a higher-level proof corresponding to the entire query chain be formed. In other words, Figure 13 The common input 1305, besides being used for single-batch verification, also provides the foundation for the connection between subsequent batch proofs. Thus, the role of zero-knowledge proofs in this scheme is no longer limited to local operator-level checks, but can be progressively extended along data batch boundaries and operator boundaries to the entire query process.

[0053] based on Figure 13 The principles illustrated will be further explained in subsequent embodiments: how physical operators synchronously form verification payloads during streaming batch processing; how the verification payloads carry witness data 1304; how input / output summaries enter the proof environment as part of common input 1305; how the verification program translates operator processing semantics into verification relations 1303; and how multiple batch proofs 1306 are recursively aggregated to form aggregated proofs corresponding to the query results. Thus, the correctness of the database query results can be verified without disclosing the underlying original data content or repeatedly executing the complete query.

[0054] In one embodiment, Figure 1The overall architecture of a database query validation system based on a streaming execution engine is shown. This system can be deployed on cloud database nodes, dedicated computing servers, or general-purpose computing devices with database execution capabilities. Figure 1 The database query verification system 100 may include a query receiving unit 101, a plan generation unit 102, an operator execution unit 103, a payload construction unit 104, an asynchronous proof scheduling unit 105, a proof execution unit 106, a proof aggregation unit 107, and a result output unit 108. The query receiving unit 101 receives structured query requests; the plan generation unit 102 converts the structured query requests into a query execution plan including multiple physical operators; the operator execution unit 103 drives the physical operators to process input data batch by batch according to the query execution plan; the payload construction unit 104 synchronously forms verification payloads while the physical operators process batches of data; the asynchronous proof scheduling unit 105 encapsulates the verification payloads into proof tasks and submits them to the asynchronous execution environment; the proof execution unit 106 executes the verification program corresponding to each physical operator and outputs batch proofs; the proof aggregation unit 107 aggregates multiple batch proofs; and the result output unit 108 outputs the aggregated proofs and the verification results associated with the query results.

[0055] The database query validation here does not involve syntax checking of the query statement text itself, nor is it a simple verification of the final result. Instead, it organizes verifiable information during query execution in parallel with the native database execution process. The rationale for this is that query results are typically formed step-by-step by multiple physical operators. If validation is only organized after the query is completed and the results are retrieved, the validation information is easily disconnected from the execution process. Figure 1 The structure shown places query execution and proof generation within the same runtime framework, but they do not share the same main computation path. Operator execution unit 103 is responsible for processing the raw database data, while asynchronous proof scheduling unit 105 and proof execution unit 106 are responsible for proof computation. The two are associated through the intermediate object of verification payload. With this arrangement, the processing rhythm of database operators on data batches can be preserved, and proof tasks can be continuously dispatched in batches, rather than being started all at once after the entire query execution is completed.

[0056] In one embodiment, Figure 2 The schematic structure of the query execution plan and physical operator links is shown. Figure 2 In the process, after the structured query request 201 enters the plan generator 202, a logical plan 203 can be formed first, and then a physical plan 204 can be formed. The physical plan 204 may include a filtering operator 205, a join operator 206, an aggregation operator 207, and a result sorting operator 208. Figure 2The diagram also illustrates a data batch stream 209, which flows sequentially along the filtering operator 205, the concatenation operator 206, and the aggregation operator 207. In some implementations, the number of operators in the physical plan 204 can be greater than [previous number]. Figure 2 The number shown may also include only a subset of operators. Filtering, joining, and aggregation are used as examples because these types of operators are common in query execution, and their respective processing semantics differ significantly, making it easier to illustrate how the validation load changes with the operator type.

[0057] The physical operators in physical plan 204 are not merely abstract names at the logical level, but rather actual program execution units within the database execution engine that participate in data processing. For example, filter operator 205 can perform predicate judgments on each record after reading the current data batch and retain records that meet the conditions. Similarly, join operator 206 can obtain matching data batches from the left and right input streams, establish matching relationships based on the join key, and generate join results. Aggregate operator 207 can maintain cumulative states based on grouping keys and update grouping results on one or more batches of data. Different operators have different internal data paths, temporary states, and output generation methods. Therefore, if you want to verify the query execution process, simply retaining the final result is usually insufficient; you also need to retain information that reflects the semantics of the operator's processing.

[0058] Figure 3 A diagram with Figure 2 The corresponding batch execution process is illustrated. Figure 3 In the process, the input data source 301 is divided into multiple record batches 302, 303, and 304, and each record batch sequentially enters the physical operator link 305. Each operator in link 305, upon receiving a record batch, can execute the calculations for that batch and form the corresponding output batch. For streaming execution engines, this processing method has a crucial characteristic: each batch does not need to wait for all input data to be prepared before reaching the downstream operator. Processing proceeds in a rolling forward manner. This approach is suitable for high-throughput scenarios and long pipeline scenarios because upstream and downstream processes can work concurrently on different batches. If the verification logic cannot adapt to this characteristic and instead heavily relies on a complete global data view, additional waiting will typically be introduced during execution.

[0059] In one embodiment, the formation of the verification payload does not depend on offline backtracking after the query is completed, but rather on synchronous records in the operator execution path. Figure 4 The correspondence between operator execution and load construction is shown. Figure 4In this process, the operator execution path 401 includes an input reading stage 402, a core processing stage 403, and an output generation stage 404. The load construction path 405 is set in parallel with the operator execution path 401. The load construction path 405 may include an input mirroring unit 406, a parameter recording unit 407, an intermediate information acquisition unit 408, and an output encapsulation unit 409. This parallel setting does not require physical separation into two threads; load construction actions can also be inserted during the execution of the same thread. Crucially, input data, operator parameters, intermediate processing status, and output data are not acquired retroactively after the query is completed, but are synchronously organized when the corresponding data batch is processed by the operator.

[0060] Here's a concrete example using a filtering operator. Assume the current data batch contains several user records, each including at least a user identifier field and an age field, with the filtering condition being age greater than 18. When processing this data batch, the filtering operator can iterate through the input batch record by record, reading the age field for each record and performing predicate judgment. If the condition is met, the record is written to the output batch; otherwise, it is skipped. Simultaneously, the input mirroring unit 406 records the set of input records for the current batch, the parameter recording unit 407 records predicate parameters and field position information, the intermediate information acquisition unit 408 records the judgment result or equivalent filtering flag for each record, and the output encapsulation unit 409 records the final retained output record. The resulting verification payload is not just a result summary, but includes the correspondence between the batch's input, processing basis, and output. Subsequent verification programs, upon receiving this verification payload, can replay the judgment process according to the filtering semantics or perform constraint checks on the consistency between the filtering flag and the output result.

[0061] In another implementation, the information collected by the intermediate information acquisition unit 408 does not need to be detailed down to every single record's execution details; instead, it can be represented more compactly. For example, with a filtering operator, the predicate evaluation result for each record can be recorded as a bitmap; with a join operator, matching hints can be recorded as index pairs; and with an aggregation operator, the accumulated state corresponding to each group can be recorded as an intermediate state table. Whether to use finer-grained information typically depends on the implementation of the subsequent verification procedure, available memory, and proof cost. If the collected information is too simplistic, the verification procedure may need to repeat more operations in the proof environment; if the collected information is too detailed, the verification payload will increase. Therefore, in engineering, a trade-off is usually struck between payload size and verification cost. Figure 4 The structure shown allows this trade-off to be made separately for different operator types, without requiring the use of the same granularity.

[0062] In one embodiment, the verification payload can be organized using a uniform container plus operator-specific fields. Figure 5 A schematic diagram of the data structure for verifying the payload is shown. Figure 5 In the validation payload 500, there may be a payload header 501, a pattern definition area 502, an input data area 503, an output data area 504, a parameter area 505, a validation information area 506, and an integrity verification area 507. The payload header 501 may include operator type identifiers, batch identifiers, task identifiers, and version information. The pattern definition area 502 may describe the name, type, order, or column position mapping of each field in the current batch. The input data area 503 and the output data area 504 respectively carry the data content of the current batch before and after entering the physical operator. The parameter area 505 is used to store operator parameters such as predicate conditions, join key definitions, aggregation function types, and grouping key definitions. The validation information area 506 is used to place information characterizing the processing procedure, such as execution traces, filter bitmaps, matching hints, permutation mappings, and grouping intermediate states. The integrity verification area 507 may include summary information calculated for the aforementioned fields to detect whether the payload has been accidentally tampered with during asynchronous transmission or storage.

[0063] This unified container architecture is particularly useful in engineering deployments. Different physical operators in a query plan need to share the same asynchronous proof scheduling facility. If each type of operator has a completely different custom task encapsulation format, the implementation of the asynchronous task queue, serializer, and scheduler becomes significantly more complex. With a unified container, the scheduling layer only needs to identify the operator type identifier in the payload header 501 to distribute tasks to the corresponding verification program; the differences truly related to operator semantics can be stored in the parameter area 505 and the verification information area 506. This doesn't eliminate operator differences, but rather places them at a more manageable level. Figure 5 The schema definition area 502 in the database also has practical significance, because databases often use columnar representation, while verification programs often read data in row-based or linear memory. Without a schema definition, it is difficult to correctly interpret the input byte stream.

[0064] In some implementations, the input data area 503 and the output data area 504 can be laid out in rows. Let the number of records in the current batch be... The serialized length of each record is 100 bytes. ,in Then the total length of the data area for this batch is... It can be represented as:

[0065]

[0066] In the formula, This indicates the total number of bytes of valid data in input data area 503 or output data area 504; This indicates the number of records in the current data batch; Indicates the first The byte length of the record under the current serialization rule. If the record uses variable-length field encoding, then... This total length can change according to the recorded content. It can be written into the payload header (501) for subsequent parsing, and also serve as a task volume indicator during the asynchronous scheduling phase. After reading this length, the scheduler can estimate the memory usage of the current task in the queue and, based on the remaining queue capacity, decide whether to submit it immediately. If a certain data batch corresponds to... If the load exceeds a preset threshold, it is not necessary to send the entire load to the proof environment as is. Instead, it can be further divided on the operator side according to batch boundaries or record ranges. There is a practical consideration for this approach: the proof environment is more suitable for processing inputs with well-defined boundaries and controllable sizes. Excessively large individual loads not only increase queuing time but also easily lead to imbalances in subsequent proof aggregation stages.

[0067] In some implementations, the execution traces in verification information area 506 can be represented as event sequences. These execution traces do not necessarily correspond one-to-one with the raw instructions of the underlying CPU; a more common approach is to record processing events directly related to the operator semantics. For filtering operators, the event sequence may include record read events, predicate evaluation events, and output write events; for hash join operators, the event sequence may include hash table construction events, join key lookup events, and result combination events; for aggregation operators, the event sequence may include grouping location events, cumulative state update events, and grouping result output events. Using this representation, the verification program can directly examine the sequential relationships between events and the correspondence between inputs and outputs based on the operator semantics, without having to reproduce all the underlying implementation details of the database execution engine within the proof environment. This approach reduces the verification program's dependence on the internal structure of the database kernel and facilitates the migration of the verification program between different database engines.

[0068] In one embodiment, the events recorded in the execution trace can be further refined into at least one of instruction-related information, memory access information, and state change information. The instruction-related information here does not require the reconstruction of the original machine instructions at the CPU level, but can be used to characterize the order and type of operations related to the semantics of operator processing. For example, for filtering operators, operation types such as field reading, predicate evaluation, result determination, and output writing can be recorded; for join operators, operation types such as hash construction, key lookup, match confirmation, and result concatenation can be recorded; for aggregation operators, operation types such as grouping and positioning, cumulative calculation, state refresh, and result output can be recorded. By incorporating this type of instruction-related information into the execution trace, the verification program can check whether the operation sequence satisfies the predetermined processing rules around the operator semantics in the proof environment, without relying on the finer-level implementation details within the database execution engine.

[0069] In one embodiment, memory access information can be used to describe the access location, access order, or access range of the current data batch during processing, related to the input buffer, hash table, grouping state table, or output buffer. For hash join operators, the write position of the right input record during the construction phase and the lookup position of the hash bucket during the probing phase can both be characterized by memory access information; for aggregation operators, the location and update of the state table entry corresponding to a certain key can also be reflected by memory access information. The significance of this approach is that the semantics of some operators are not only reflected in the final output result but also in how the intermediate structures are accessed and updated. By incorporating memory access information into the execution trace, the verification program can check whether the access to the intermediate structures corresponds to the input record, join key, or grouping key, thereby reducing the ambiguity caused by inferring from the final result alone.

[0070] In one embodiment, state change information can be used to describe the internal state changes of an operator as it processes the current data batch. This state can be a filtering flag in a filtering operator, a match count or candidate relationship in a join operator, or a cumulative value, grouping state, or intermediate result buffer in an aggregation operator. Taking an aggregation operator as an example, if a key group's cumulative state before processing the current data batch is an input state, and it becomes an output state after processing, then the state change information can link these two states with the current batch record that caused the change. In this way, the verification program not only sees the two endpoints before and after the state update, but can also check whether the state transition conforms to the update rules of the corresponding aggregation function by combining the state change information. Therefore, by recording at least one of instruction-related information, memory access information, and state change information in the execution trace, the verification program can establish a more stable and verifiable representation of the operator processing process without re-relying on the entire underlying implementation of the database execution engine.

[0071] Figure 6 An asynchronous proof scheduling structure is shown. Figure 6 In this process, the main execution thread 601 drives the physical operators to process data batches; the payload buffer 602 temporarily stores newly constructed verification payloads; the task encapsulator 603 generates proof task objects based on the verification payload and operator type; the task queue 604 caches proof tasks to be executed; the proof thread pool 605 executes multiple proof tasks concurrently; and the result collector 606 collects batch proofs and transfers them to the proof aggregator 607. The statement that the main execution thread 601 and the proof thread pool 605 are independent does not mean they share no resources, but rather that the main execution thread 601 does not need to synchronously wait for the proof thread pool 605 to complete the proof of the current batch. Once the current data batch has completed payload construction and been encapsulated by the task encapsulator 603, the main execution thread 601 can continue to drive subsequent data batches forward. Thus, query computation and proof generation can form a pipelined relationship with overlapping time signatures.

[0072] The changes brought about by this asynchronous arrangement are not just concurrent execution in the usual sense, but more importantly, the waiting position has been moved. If proof generation and operator computation share the same main path, then after each batch of data is processed, the main path must stop and wait for the corresponding proof to complete, and the blocking point will frequently fall at the batch boundary. Figure 6 The structure shown primarily waits until the result aggregation stage. In other words, the main execution thread 601 does not need to pause for most batch processing because the proof has not yet returned; the result collector 606 and the proof aggregator 607 only check the completion status when the final aggregated proof needs to be output, or when it is necessary to confirm that all batch proofs have been successfully generated. This structure is more suitable for scenarios with a large number of batches and where the proof cost is significantly higher than the processing cost of the operator itself, because placing the wait after each batch is usually costly in this case.

[0073] To control the resource consumption of the proof thread pool 605, in one embodiment, a queue depth threshold can be set for the task queue 604. And to prove that thread pool 605 sets a concurrent task limit. When the queue is at its current depth satisfy When the number of concurrently running proof tasks is reached, the payload buffer 602 can pause receiving new proof tasks or trigger a backpressure strategy; satisfy In this case, task wrapper 603 can first complete the construction of the task object without actually submitting it to the proof thread pool 605. The backpressure here doesn't necessarily directly block the database execution engine; it can also be controlled hierarchically. For example, first reduce the proof submission speed of low-priority queries, then delay background batch processing tasks; if the queue continues to grow, then apply a gentler rate limit to the main execution thread 601. The reason for this is that proof tasks typically consume a lot of memory and computing resources. If they are continuously dispatched without limit, it can easily conversely crowd out the resources needed for operator execution, affecting the original main query path.

[0074] In a more concrete implementation, the decision to submit a task immediately can be dynamically made based on the estimated proof cost of a single batch. Let the number of input records in the current batch be... The estimated average verification cost per record is Fixed expenses are Then the estimated cost of this batch is... It can be represented as:

[0075]

[0076] In the formula, This indicates the estimated proof cost for the current batch; This represents the fixed overhead required for task initialization, serialization, and starting the verification process. This indicates the number of input records in the current batch; This represents the average validation cost per record. If If the cost exceeds a preset threshold, the batch can be further divided into multiple proof fragments, or the task can be placed in a low-priority queue. Equation (2) here does not require precise prediction of the actual running time and is more suitable as an approximate indicator in scheduling decisions. Under different operator types, Different estimates can be used. For example, the filtering operator. Typically lower than the connection operator This is because join operators often involve matching relationships, temporary states, and more intermediate evidence.

[0077] Figure 7 The interaction between the verification procedure and the verification payload is shown. Figure 7In the process, the verification procedure 701 receives the payload content output by the input parser 702. After processing by the semantic replay unit 703 or the constraint checking unit 704, it generates a batch proof 705 and a common input digest 706. The semantic replay unit 703 and the constraint checking unit 704 are shown side-by-side because both processing methods can be used under different operators and different proof systems. Semantic replay is closer to re-executing the operator semantics in the proof environment, such as evaluating each filter condition or matching each join key. Constraint checking is closer to directly verifying whether the input, output, and intermediate information satisfy a predetermined relationship, such as checking whether the number of records in the filter bitmap matches the number of output records, or checking whether the key values ​​in the matching prompts satisfy the equality condition. In some scenarios, the two can be used in combination: semantic replay is used for key processing steps, and constraint checking is used for auxiliary information. This maintains the clarity of the verification semantics while controlling the execution cost in the proof environment.

[0078] In one embodiment, if the verification program 701 runs in a zero-knowledge virtual machine 703, the input parser 702 can first convert the verification payload into a native input structure that the zero-knowledge virtual machine can accept. Here, the zero-knowledge virtual machine is not a general-purpose operating system virtual machine in the conventional sense, but rather a computational environment capable of outputting zero-knowledge proofs about the execution process while executing the program. This environment focuses more on whether the execution trajectory is provable, rather than simply simulating a certain instruction architecture. To reduce adaptation complexity, the columnar batch structure commonly used in database kernels can be converted to a row-based structure suitable for linear memory access in a zero-knowledge virtual machine on the host side. If necessary, length prefixes can be added to variable-length fields. This approach has clear engineering reasons. Many zero-knowledge execution environments can more easily control the number of memory accesses and constraints when processing linear byte streams. However, if complex columnar structures are directly retained, the verification program often needs to perform more offset calculations and indirect addressing, which can easily increase the proof cost.

[0079] In one embodiment, the significance of running the verification program in the zero-knowledge virtual machine 1104 lies not only in moving existing verification logic to another execution environment, but also in transforming the execution process of the verification logic into a provable execution trajectory. For the filtering, joining, and aggregation operators in this scheme, the verification program itself can maintain an implementation form that is closer to a conventional program structure. The zero-knowledge virtual machine 1104 is responsible for synchronously generating proof materials corresponding to the execution process when executing the verification program. In this way, operator-level verification does not need to reconstruct a completely independent constraint system from the bottom layer for each type of operator. Instead, the verification logic of different operators can be organized into programs first, and then the zero-knowledge virtual machine 1104 provides a unified provable execution framework. For database query verification scenarios that need to cover multiple types of physical operators, this implementation method is more conducive to reusing verification infrastructure.

[0080] In one embodiment, the zero-knowledge proof generated by the zero-knowledge virtual machine 1104 demonstrates that, without disclosing all the witness data, the verification program has indeed completed the corresponding verification execution process for a given public input. The witness data may include input data, output data, operator parameters, and verification information such as execution traces, intermediate states, matching hints, or filtering bitmaps; the public input may include input / output summaries, batch identifiers, and operator identifiers. After the proof outputter 1112 generates the batch proof 1113 based on the results output by the verification logic executor 1110 and the trajectory generator 1111, the external verification side or the upper-layer aggregation side does not need to re-access all the original data or re-execute the original processing flow of the database operators. Instead, the batch proof 1113 can be verified using the verification parameters. This preserves the provability required for query result verification while reducing the direct exposure of underlying data during the verification process.

[0081] In one embodiment, employing a zero-knowledge virtual machine 1104 also facilitates adaptation to the organization of batch verification tasks. The verification payload corresponding to the current data batch, after format conversion, can be directly read as program input to the zero-knowledge virtual machine 1104; verification programs corresponding to different physical operators can be loaded and run separately within the same type of execution environment. Thus, when filtering operators, connection operators, and aggregation operators form batch verification tasks, the asynchronous proof scheduling structure does not need to maintain completely different underlying proof execution frameworks for different operator types, but can schedule different verification programs around the same zero-knowledge virtual machine 1104. For database queries with many operator types and a large number of batches, this combination of programmatic verification and a unified proof environment makes it easier to control system adaptation complexity while ensuring verifiability.

[0082] Figure 8 The execution and verification process of the filtering operator is illustrated. In one embodiment, the filtering operator 801 receives a data batch 802 from upstream and generates an output batch 804 based on the filtering predicate 803. The load construction component 805, which works in parallel with the filtering operator 801, can record the input data while reading the input batch 802, record the filtering criteria when performing predicate judgment, and record the retained results when writing the output batch 804. Figure 8 The document also illustrates a filter verification procedure 806, which receives a filter payload 807 and generates a filter batch proof 808 in a proof environment. The verification process for the filter operator is presented separately here because the filter operator is structurally straightforward, making it easier to clarify the relationship between the input, parameters, intermediate judgments, and output. Once this relationship is clarified, the verification processes for the connection operator and aggregation operator become easier to understand.

[0083] When the filtering operator 801 processes the current batch of data, it can first read the target field position in the pattern definition, and then extract the field values ​​that need to be judged according to the record order. If the filtering predicate 803 is a single-condition expression, such as the age field being greater than 18, then each record only needs to generate a Boolean judgment result; if the filtering predicate 803 is a compound expression, such as the age field being greater than 18 and the status field being active, then each record can generate multiple sub-judgment results internally, and combine them into a total judgment result in the final stage. In a simpler implementation, the load construction component 805 can only retain the final filtering result for each record; in a more granular implementation, the load construction component 805 can also retain sub-judgment results, field value snapshots, or short-circuit evaluation paths. The granularity used is usually related to at which semantic level the verification program 806 intends to check. If the verification program 806 only needs to confirm whether the input and output satisfy the same filtering predicate, then retaining the final filtering result is sufficient to support verification; if it is desired to further verify the evaluation path inside the compound expression in the proof environment, then a finer intermediate state can be retained.

[0084] In one embodiment, the filtering payload 807 may include an input record set, an output record set, a field pattern, filtering predicate parameters, and a filter bitmap. The filter bitmap can be used to indicate whether each record in the current data batch is retained. Suppose the current batch contains... The first input record, the The filtered results of the records are denoted as ,in Indicates the first The records meet the filtering criteria. If the filter criteria are not met, then filter the bitmap. It can be written as If the number of output records is denoted as... If the filtering operator is semantically correct, then the following can be satisfied:

[0085]

[0086] In the formula, This indicates the number of records in the current output batch; This indicates the number of records in the current input batch; Indicates the first The filtering results of the input records. Equation (3) reflects a basic constraint in the filtering semantics, namely, the number of output records should be consistent with the number of elements with a value of 1 in the filtering bitmap. The verification program 806 can use this constraint to perform a quick check first. If the number is inconsistent, there is no need to further compare the record content; if the number is consistent, then continue to compare the retention order and retention content one by one. This layered checking has the advantage that obvious errors can be eliminated at a lower cost first, and then more detailed verification actions can be performed on the remaining cases.

[0087] In some implementations, the filter validator 806 can employ semantic replay. In this case, the validator 806 reads the input records, field patterns, and filter predicate parameters from the filter payload 807, re-evaluates the predicates for each input record in the proof environment, and generates a replay bitmap. Then, the bitmap will be replayed. With filter bitmap The comparison is performed; if they match, the expected output set is extracted from the bitmap and compared with the actual output record set. This implementation is closer to the semantics of database execution itself, and its advantages are intuitive semantics, clear verification logic, and easier location of differences during debugging. If a record is found to be in... The middle is 1 and in If the value is 0, it means that the record was missed in the operator execution path; conversely, if... The middle is 0 and If the value is 1, it indicates that an error has occurred and the passage has been incorrectly approved.

[0088] In other implementations, the filter validator 806 can also use constraint checking. In this case, the validator 806 may not necessarily recalculate all predicates one by one, but directly based on the input records and the filter bitmap. The following relationships are checked with the output record set: all records marked as 1 should appear in the output set; records marked as 0 should not appear in the output set; and the relative order of records in the output set should be consistent with the relative order of these records in the input batch. This implementation is more focused on verifying the input-output mapping relationship than reinterpreting the filter expression itself. When the filter predicate 803 comes from an external, already trusted expression compilation module, and the focus is on verifying the correctness of the operator execution path, constraint checking is often more cost-effective in terms of proof.

[0089] To explain in more detail Figure 8 The working relationship between the various components can be considered in the following example. Assume that input batch 802 contains 4 records with age field values ​​of 16, 20, 31, and 15 respectively, and filter predicate 803 specifies an age greater than 18. Then, when processing this batch, filter operator 801 can obtain a filter bitmap. Output batch 804 contains records 2 and 3. During this process, the load construction component 805 records the input record set, filter predicate parameters, filter bitmap, and output record set, and generates a filtered load 807. After reading this load, the filter verification program 806 can recalculate whether the age field is greater than 18 in the verification environment, obtaining the replay bitmap. .when And the output batch 804 and the press When the extracted expected outputs match, a filter batch proof 808 can be generated. The reason for retaining the filter bitmap instead of just the output record set is that the filter bitmap can directly reflect the retention and rejection relationship of each input record, allowing the verification program to more directly identify which record the problem occurred on.

[0090] Figure 9 The execution and verification process of the connection operator is illustrated. Figure 9 In this process, the connection operator 901 may include a construction phase 902 and a probing phase 903. The construction phase 902 reads records from the right input batch 904 and builds a hash table 905 based on the connection key. The probing phase 903 reads records from the left input batch 906, matches them against the connection key in the hash table 905, and if a match is found, generates a connection result and writes it to the output batch 907. The payload construction component 908 works in parallel with the connection operator 901 to form the connection payload 909. The connection verification program 910 in the proof environment receives the connection payload 909 and generates a connection batch proof 911.

[0091] Verification of join operators is typically more complex than that of filtering operators. This is not because join operators are difficult to describe, but because the generation of the join result depends on the matching relationship between the two inputs. Simply knowing whether the output exists is often insufficient to determine if the matching process is correct; it is also necessary to specify which pair of input records the output corresponds to, or at least why the output record is eligible to appear. Therefore, in one embodiment, the join payload 909, in addition to including the left input batch 906, the right input batch 904, the output batch 907, and the join key parameters, may also include matching hint information. The matching hint information can be represented as a set of index pairs, where each index pair... Indicates the first in the left input batch The record and the first entry in the right input batch The records are concatenated into a single output. This allows the verifier 910 to avoid having to build a hash structure identical to the database kernel from scratch in the proof environment. Instead, it can use matching hints to first locate candidate relationships and then check whether these relationships satisfy the join condition.

[0092] Suppose that there are a total of 906 in the left input batch. There are 904 records in the batch on the right. The set of matching hints is denoted as records. ,in This indicates the number of candidate matching pairs. If the join result is an inner join, the output will show the number of records in batch 907. The number should be consistent with the number of valid matching pairs, that is:

[0093]

[0094] In the above formula, This indicates the number of records in the current connection's output batch; Indicates the first Is the matching suggestion valid? When the left input batch contains the first matching suggestion, the first matching suggestion is valid. The record and the first entry in the right input batch When the connection key of a record meets the predetermined connection conditions, Otherwise, it is 0. For equi-joins, This typically depends on whether the values ​​in the join key field of the two records are equal; for more general join conditions... It can also be determined by combining the evaluation results of additional predicates. Verification program 910 can use equation (4) to first determine whether the number of matching prompts is compatible with the output size, and then further check whether the output content corresponds to these valid matching pairs.

[0095] In one embodiment, the connection verification procedure 910 may employ a cue-assisted constraint checking method. The verification procedure 910 first reads the left input record, right input record, connection key definition, and matching cue set from the connection load 909. Subsequently, for each index pair Verification program 910 checks the left input record. With right input record The connection key is checked to see if it meets the conditions. If it does, the two records are concatenated or projected according to the output mode of the connection operator to generate an expected output record; if it does not meet the conditions, the verification of the corresponding index pair fails. After all index pairs have been processed, the verification program 910 will obtain the expected output set and compare it with the output batch 907. In this method, the matching hints in the connection load 909 help to narrow the search space. The proof environment does not need to enumerate all There are multiple possible combinations, but only candidate matching pairs given by the database execution path need to be checked. This makes it easier to control the proof cost when the data volume is large.

[0096] In another implementation, the join verifier 910 can also employ partial semantic replay. Partial replay here refers to rebuilding a simplified hash index in the proof environment, or logically grouping the right input batch 904 by the join key and then probing the left input batch 906 one by one. The advantage of this approach is that it relies less on matching hints, resulting in more self-consistent verification; the disadvantage is that it places higher demands on the memory and computational overhead of the proof environment. This implementation is easier to implement if the right input batch 904 is small; if the right input batch 904 is large, the more common approach is still for the database execution engine to provide matching hints first, followed by the verifier confirming correctness.

[0097] A specific example can be used to illustrate this. Figure 9 The data relationships are as follows: Assume the left input batch 906 contains 3 records with user identifier fields of 10, 20, and 30 respectively; the right input batch 904 contains 4 records with user identifier fields of 20, 20, 30, and 40 respectively. The join condition is that the user identifiers on both sides are equal. During execution, the join operator 901 can build a hash table 905 using the right input batch 904 in the construction phase 902, and process the left input batch 906 in the probing phase 903 to obtain a matching hint set. Therefore, output batch 907 contains three join results, where the second left record matches the first and second right records respectively, and the third left record matches the third right record. After the load construction component 908 records these matching hints, the join verification program 910 no longer needs to search for whether the first left record and the fourth right record match. Instead, it directly verifies whether each pair of indices in the hints satisfies the join key equality and checks whether the output records are consistent with the hints. If an output record fails to correspond to any valid hint, or if a valid hint lacks a corresponding output, the verification will fail.

[0098] Figure 10 The execution and verification process of the aggregation operator is shown. Figure 10 In the process, the aggregation operator 1001 reads records from the input batch 1002, locates the target group in the grouping state table 1004 according to the grouping key 1003, updates the intermediate state of the corresponding group according to the aggregation function 1005, and finally generates the output batch 1006. The parallel load construction component 1007 can record input data, grouping key definition, aggregation function definition, intermediate state snapshot, and final output, thereby forming the aggregated load 1008. The aggregation verification program 1009 in the proof environment receives the aggregated load 1008 and outputs the aggregated batch proof 1010.

[0099] A key characteristic of aggregation operators is that the processing results of the current batch are not always directly reflected in the output of that batch. Especially in streaming execution engines, some aggregation operators only update the state table 1004 when processing intermediate batches, without immediately outputting the final aggregation result. This means that the validation payload cannot be organized solely around the output batch 1006; it also needs to cover the intermediate batches. For this reason, the aggregation payload 1008 typically includes more intermediate state-related fields than the filtering payload 807 and the join payload 909. Without these fields, the validation program 1009 can only see the input and output, making it difficult to determine whether the state updates conform to grouping and aggregation semantics.

[0100] Let the set of records in input batch 1002 be... The group key extraction function is denoted as Aggregate functions are denoted as For a given set of key values The intermediate aggregation state of this group in the current batch can be denoted as: If the aggregation function is summation, then It can be represented as the sum of the accumulated values ​​in the group; if the aggregation function is a counting function, then... This can be represented as the cumulative number of records in the group. For summation-type aggregations, the state update relationship can be written as follows:

[0101]

[0102] In the formula, This indicates the group key before processing the current batch. The corresponding intermediate input state; This indicates the intermediate output state after processing the current batch; Indicates from record The extracted values ​​to be aggregated; Representing records Belongs to group bond Equation (5) describes the most common type of state update relation. If the aggregation function is counting, then... A uniform value of 1 can be used; if the aggregation function is the maximum or minimum value, the state update can be replaced by the corresponding comparison operation. Verification program 1009 can check whether the state update of each group key satisfies this relationship based on intermediate state snapshots and input record content.

[0103] In one embodiment, the aggregation verification program 1009 may not require the input batch 1002 to be pre-sorted, but instead performs logical grouping based on the grouping key in the proof environment. This logical grouping can be hash grouping or sorted and then segmented. If the number of group keys is small, hash grouping is more direct; if the goal is to reduce random memory access in the proof environment, sorted and then segmented processing makes it easier to control access patterns. The payload construction component 1007 can choose whether to include a group key index table, sorting mapping, or intra-group counting information based on the implementation on the database execution engine side. If this auxiliary information already exists, the verification program 1009 can perform less grouping preparation work; if not included, the verification program 1009 can still recover the grouping relationship from the input records, but the cost will be higher.

[0104] This can be illustrated with a simple example. Figure 10 The process is as follows. Assume that the input batch 1002 contains 5 order records, the grouping key 1003 is the region field, and the aggregation function 1005 is the sum of order amounts. In the current batch, region A corresponds to two records with amounts of 50 and 70 respectively; region B corresponds to three records with amounts of 10, 20 and 40 respectively. If the intermediate state of region A in the status table 1004 is 100 and the intermediate state of region B is 30 before processing the current batch, then after processing, the state of region A becomes 220 and the state of region B becomes 100. The load construction component 1007 writes the state before processing, the state after processing, the input records and the output results into the aggregate load 1008. After reading this information, the aggregation verification program 1009 can regroup according to the region field and verify whether the state updates of region A and region B satisfy equation (5) respectively. If the state update of a certain region is missing a record or is incorrectly mixed with the amount of other regions, then this relationship will not hold.

[0105] Figure 11 The transformation relationship between the verification payload and the proof execution environment is illustrated. In one embodiment, data batches within the database execution engine 1101 are typically stored in a columnar layout 1102. Columnar layout 1102 is suitable for vectorized processing of database operators because contiguous storage of the same field facilitates sequential scanning, batch comparison, and cache reuse. However, the verification program 1103 in the proof execution environment 1104 is often more suitable for reading input in record order, especially when multiple fields need to be combined for semantic checks, where row layout 1105 is more convenient for linear access. Therefore, after the payload is constructed, the columnar layout 1102 can be converted to row layout 1105 by the format conversion unit 1106, and the converted result is written to the verification input area 1107. Figure 11 The diagram also shows a schema mapping table 1108, which is used to illustrate the mapping relationship between the column field order and the row record layout.

[0106] The format conversion here should not be understood as a simple data copy. If not implemented properly, the format conversion itself can introduce high overhead, even offsetting the benefits of asynchronous proof. Therefore, in one embodiment, the format conversion unit 1106 can preferentially adopt zero-copy mapping or segmented copying. For fixed-length fields, such as integer, boolean, or fixed-length character fields, row records can be constructed directly based on field offsets; for variable-length fields, such as string or variable-length binary fields, a length prefix and data segment reference can be written into the row record, and the actual content can be copied when necessary. The reason for adopting this approach is that the storage cost of different field types is not the same. If a deep copy is performed uniformly without distinguishing field types, the additional memory allocation during the payload construction stage will be significant in scenarios with a large number of variable-length fields. If only the length prefix and external reference are retained, the subsequent verification program 1103 will need to solve the cross-region access problem. Therefore, in engineering, a compromise solution is usually chosen based on the field type and deployment environment.

[0107] In one embodiment, the data packet in the verification input area 1107 can consist of three parts: a header, a metadata area, and a data area. The header records the total length of the data packet, the operator type, the batch number, and the version identifier; the metadata area records the number of fields, field types, the null flag layout, and the offset information of each field in the row-based record; the data area stores the specific record content in record order. If the current batch contains... There are 10 records, and the length of each record after row encoding is as follows: Then the starting offset of the record in the data area can be determined using a prefix sum method. Let the first... The starting offset of the record is Then we have:

[0108]

[0109] In the formula, Indicates the first The offset of each record relative to the starting address of the data area; Indicates the first The length of the encoded record. Equation (6) can be used to verify that procedure 1103 locates the first record in the proof environment. For records containing only fixed-length fields, an offset index can also be pre-built on the host side. Similarly, offset calculation is relatively straightforward; for records containing variable-length fields, offset indexes are more valuable because the verifier 1103 can reduce duplicate scans by using pre-stored offsets.

[0110] In some implementations, the verification program 1103 runs in a zero-knowledge virtual machine 1104. This zero-knowledge virtual machine is not intended to simulate the original implementation of the database execution engine, but rather serves as a provable execution container responsible for executing the verification logic and outputting zero-knowledge proofs. Figure 11 In this context, the proof execution environment 1104 may include an input reader 1109, a verification logic executor 1110, a trajectory generator 1111, and a proof outputter 1112. The input reader 1109 is responsible for parsing the data packets in the verification input area 1107; the verification logic executor 1110 is responsible for calling the corresponding verification logic according to the operator type; the trajectory generator 1111 is responsible for generating an execution trajectory that meets the requirements of the proof system during execution; and the proof outputter 1112 is responsible for outputting a batch proof 1113 and a common input digest 1114 based on the execution trajectory and the proof protocol.

[0111] The reason for emphasizing the common input digest 1114 is that batch proofs typically cannot exist in isolation during subsequent aggregation stages. The aggregator needs to know which batch of inputs and outputs each batch proof corresponds to in order to sequentially connect multiple batch proofs. The common input digest 1114 can fulfill this connection function. Taking a certain physical operator as an example, the digest of the input data can represent the input content seen by the operator in the current batch, and the digest of the output data can represent the output content generated in the current batch. If the input digest is denoted as... The output summary is denoted as Then the common input summary 1114 can be written as a tuple It can also be written in a richer structure that includes operator type and batch identifier. Subsequent adjacent operators or adjacent fragments can be compared by comparing summary relationships to confirm whether the data connections are consistent.

[0112] In one embodiment, the digest computation can take the following form. Let the serialized input data byte stream be... The output data byte stream is The hash function is denoted as Then we have:

[0113]

[0114] In the formula, This represents a summary of the input data; This indicates the output data summary; This represents a serialized byte stream of input data; This represents a serialized byte stream of output data; This represents the hash function. A specific hash algorithm is not required; the choice of hash function typically depends on the compatibility of the proof system, implementation complexity, and support from existing cryptographic libraries. The reason for using a hash instead of directly exposing the original data is that the aggregation phase primarily focuses on whether the data relationships between batches can be closed, without needing to reprocess all the original data. This reduces the size of the aggregation input and minimizes additional exposure in scenarios requiring privacy protection.

[0115] Figure 12 The process of piecewise proof and recursive aggregation is illustrated. Figure 12 In the process of executing the query to be verified, multiple batch proofs 1201, 1202, 1203 and 1204 can be generated. These batch proofs first enter the first-level aggregator 1205 and 1206 to obtain intermediate aggregate proofs 1207 and 1208. Subsequently, the intermediate aggregate proofs 1207 and 1208 enter the upper-level aggregator 1209 to generate the final aggregate proof 1210. Figure 12 The diagram also illustrates batch summary chain 1211, used to represent how the input and output summaries corresponding to each batch of proofs are passed and checked together during the aggregation process. Such tree-like aggregation is not required to be a fixed binary structure; a multi-branch structure can also be used depending on the task size. Figure 12 The binary aggregation shown is because the description is more intuitive and the recursive aggregation relationship is clearer.

[0116] In one embodiment, fragmented proofs are typically generated when a single batch payload is too large or the cost of proving a single batch is too high. Fragmentation here doesn't necessarily change the batch boundaries of the database's native operators; it can also involve finer-grained segmentation of a single batch payload at the proof layer. For example, a single data batch processed by a certain join operator may already be formed, but due to the large size of the right input data, the join verification cost for that batch is too high. Therefore, the batch payload can be divided into multiple proof fragments based on record range, hash bucketing, or matching hint intervals. After fragmentation, each fragment can generate an independent batch proof, which is then processed by… Figure 12 The aggregator summarizes the data layer by layer. The purpose of this approach is not to change the original execution result of the database, but to make the proof generation process closer to the scale that the proof environment can handle.

[0117] In one embodiment, aggregators 1205, 1206, and 1209 can run dedicated aggregation verification programs. These programs no longer re-verify the input-output semantics of the underlying operators; instead, they verify the validity of lower-level proofs and whether the common input digests between lower-level proofs satisfy a predetermined connection relationship. If the output digest in a lower-level proof is inconsistent with the expected input digest upstream, the aggregation verification program will not continue generating upper-level proofs. This allows for hierarchical processing of local operator verification and global link verification. Local operator verification is handled by the verification program corresponding to each operator; global link verification is handled by the recursive aggregation program. A direct result of this hierarchical approach is that the underlying operator verification logic does not need to concern itself with the global structure of complex queries, and the upper-level aggregation program does not need to reinterpret the internal semantics of each type of operator.

[0118] A more intuitive example can be used to illustrate this. Figure 12 The working principle is as follows: Suppose a query execution plan contains three main operators: filtering, joining, and aggregation. Each operator generates several batches of proofs. The first batch output digest of the filtering operator corresponds to the first batch input digest of the joining operator, and the aggregated output digest of the joining operator corresponds to the input digest of the aggregation operator. If only batch proofs within each operator are generated, without cross-operator aggregation, it can only be concluded that each batch has passed its own verification, but it cannot be concluded whether these batches constitute the same complete query chain. Figure 12 The batch digest chain 1211 in the code is designed to address this issue. During the recursive aggregation process, not only are the validity of each lower-level proof verified, but the digest relationships they carry are also checked to see if they can be closed, thus connecting the scattered local proofs into the final aggregate proof 1210 corresponding to the entire query process.

[0119] In one embodiment, if the final aggregated proof 1210 is used to return to an external verifier, the result output unit can simultaneously output a query result summary, the final aggregated proof 1210, and verification-related metadata. This metadata may include query identifier, plan version identifier, timestamp, batch range, operator range, or proof protocol version information. The external verifier does not necessarily need to replay the entire database computation process; it can execute the verification procedure based solely on the final aggregated proof 1210 and the corresponding metadata. Thus, the amount of interaction between the database execution node and the external verifier is typically lower than the amount of interaction if all batch proofs were directly transmitted. If the external scenario prioritizes link integrity, only the final aggregated proof 1210 and the necessary summary can be output; if the external scenario also requires locating the failure point, some intermediate aggregated proofs can be included for layered troubleshooting.

[0120] In addition to the methods and processes described above, implementation on the electronic device side can also be carried out relatively directly. Figure 1Each unit in the illustrated system can be implemented by one or more processors in conjunction with memory and program instructions. Processors may include general-purpose central processing units, dedicated accelerators, or secure execution units; memory may include volatile and non-volatile memory. When the program instructions are executed by the processor, the electronic device can perform actions such as receiving structured query requests, generating query execution plans, batch processing of physical operators, constructing verification payloads, submitting asynchronous proof tasks, generating batch proofs, and recursively aggregating output. If the electronic device simultaneously hosts both a database execution environment and a proof execution environment, a shared bus or shared memory region can be used to transmit the verification payload; if the two are deployed separately, they can interact through inter-process communication or remote calls. The deployment method typically depends on computing resources, isolation requirements, and the security boundary design of the proof environment.

[0121] In one different embodiment, the database execution environment and the proof execution environment may not be deployed on the same physical node. The database execution node is responsible for operator processing and verification payload construction, while the remote proof node is responsible for receiving the verification payload and generating batch proofs. In this case, Figure 6 The payload buffer 602 and task queue 604 can be expanded into a network transmission buffer, while the result collector 606 is responsible for receiving the proof results returned from the remote end. This deployment method is suitable for scenarios where the database nodes themselves have high loads, and dedicated proof resources are provided centrally. However, remote deployment introduces additional network latency and payload transmission overhead, making the control of the verification payload size, digest compression, and sharding strategies more important. If the network becomes a limiting factor, the system can also send the verification payloads of key operators only to the remote end, while using a lighter verification strategy for the remaining operators locally.

[0122] In one embodiment, by Figures 1 to 12 The entire processing flow allows verification information to be synchronously organized along the batch execution boundaries of physical operators, and proof generation tasks to be asynchronously dispatched along the same batch boundaries. This way, query computation does not have to wait for the completion of each batch's proof before continuing, and the proof overhead can be progressively distributed as batches advance. The processing semantics of different operators such as filtering, joining, and aggregation are described separately through their respective verification payloads and verification procedures, thus the verification process no longer stops at the final result level but covers the operator-level processing. For complex queries, recursive aggregation further connects the dispersed batch proofs, preventing the final output from becoming fragmented. This overall solution, while maintaining the batch-driven execution mode of the streaming engine, provides proof results suitable for query chain verification.

[0123] In actual deployment, Figure 6The asynchronous proof scheduling structure shown does not always operate with a single queue and a single thread pool. Scheduling strategies often change depending on query load, operator type, and machine resource conditions. A common scenario is that interactive queries aim to return results and corresponding proofs as quickly as possible, while background batch processing tasks prioritize throughput. If both types of tasks are placed in the same scheduling queue, the former is easily slowed down by the latter's large batch of proof tasks. Therefore, in one embodiment, task queue 604 can be further divided into a high-priority queue, a normal-priority queue, and a delayed execution queue. The high-priority queue can be used to carry proof tasks corresponding to interactive queries, the normal-priority queue for regular online queries, and the delayed execution queue for log verification, historical data replay verification, or large batches of offline job proof tasks. When scheduling tasks, proof thread pool 605 can first pull tasks from the high-priority queue, and then retrieve tasks from the normal-priority queue or the delayed execution queue based on remaining resources. This does not change the main structure of batch-level asynchronous submission; it simply adds a finer layer of resource allocation logic before tasks enter the proof environment.

[0124] In one embodiment, the scheduler can also adjust task priorities based on operator type. This is because the verification costs for different operators typically vary significantly. Filtering operators have relatively straightforward verification logic, and their proof tasks often have short lifecycles; connection operators' verification tasks may involve more matching hints and more complex intermediate relationships; and aggregation operators' verification tasks may be accompanied by more state updates. If scheduling is mixed without differentiation, heavier tasks can easily occupy execution slots in the proof thread pool for extended periods, making it difficult for lighter tasks to complete in a timely manner. Therefore, scheduling weights can be preset for different operators. For example, filtering tasks can use a higher fast dequeue weight to shorten the waiting time for short tasks; connection tasks can use controlled concurrency to avoid multiple heavy tasks simultaneously consuming too much memory; and aggregation tasks can have their priorities further subdivided based on the current state table size. This arrangement is closer to scheduling practices in actual engineering. It does not pursue mathematical optimality but prioritizes avoiding significant imbalances.

[0125] In some implementations, the scheduler performs a feasibility check before a task enters the proof thread pool 605. If the verification payload of a task is large, the estimated proof cost is high, and the current memory level is high, the task can first enter a delayed execution state or be split into multiple subtasks. This splitting is not a re-division of the original execution batches of database operators, but rather a finer-grained division of the proof payload. For example, with the join operator, if the matching hint set of a batch is too large, it can be split into multiple sub-hint intervals according to the left input record range or the right input bucket number; with the aggregation operator, if the number of group keys involved in a batch is too large, it can be partitioned by group key hash or split by sorted segments. The split subtasks are independent in the proof layer. After all are completed, they are recursively aggregated to restore the proof relationship corresponding to the original batch. The value of this approach is not in reducing the total workload, but in transforming large tasks from difficult-to-schedule single tasks into multiple medium-sized tasks that are easier to execute interleaved.

[0126] Anomaly handling also plays a role in this scheme. Database operators may not always successfully generate verification payloads when processing batches of data. For example, the input data pattern may differ from the pattern recorded in the planned generation phase, fields may be missing during payload construction, or the proof task may fail to execute in the zero-knowledge virtual machine. In one embodiment, anomalies can be categorized into recoverable and unrecoverable anomalies. Recoverable anomalies typically refer to situations that do not change the query result itself but affect the continued progress of the proof process, such as a missing optional verification information field or a batch summary recalculation failure that still allows for rereading of the original batch data. In such cases, the system can reconstruct the current batch payload or revert to a more conservative verification granularity. Unrecoverable anomalies include situations where the input-output relationship cannot be reconstructed, key verification parameters are missing, or the proof environment returns a deterministic failure. When an unrecoverable anomaly is encountered, the result output unit 108 can stop outputting the final aggregated proof and instead output the query result and verification failure flag. While this does not provide a complete proof, it avoids outputting results with erroneous proof states.

[0127] In one embodiment, the result output unit 108 can maintain a batch status table. Each entry in the batch status table can include a batch identifier, operator identifier, payload status, proof status, and aggregation status. The payload status indicates whether the verification payload of the current batch has been generated, is complete, or has been split; the proof status indicates whether the batch proof has been submitted, is in progress, succeeded, or failed; and the aggregation status indicates whether the proof corresponding to this batch has entered the upper-level recursive aggregation. Using a status table has a direct advantage: the system can track the current stage of each batch in complex queries or long-running scenarios. If the final aggregation stage fails, the status table can quickly pinpoint which operator, which batch, and which level of aggregation has the problem. This information may not necessarily be exposed to external verification parties, but it is valuable for the system's own error recovery and debugging.

[0128] In one embodiment, multiple physical operators in the query execution plan do not necessarily require verification of equal strength. For some low-cost and semantically simple operators, full payload and full proof can be enabled; for some marginal operators with a small impact, a simplified verification strategy can be adopted. Simplification here does not mean eliminating verification, but rather reducing the types of verification information, or only verifying input / output summaries and some key constraints. The reason for this hierarchical approach is that not all operators in a real system are worth investing the same amount of proof cost. Some projection operators simply rearrange field order or delete non-critical fields; retaining a complete execution trace for them may not yield high returns. Conversely, operators such as filtering, joining, and aggregation, which directly affect the result content and the number of records, are more suitable for retaining more complete verification information. In a more specific deployment, the plan generation unit 102 can assign different verification level tags to different operators based on operator type, query level, and available resources, and the payload construction unit 104 and asynchronous proof scheduling unit 105 can then select the processing path based on these tags.

[0129] Besides deployment within database nodes, a collaborative approach between edge and central nodes can also be considered. Assume data undergoes initial filtering and partial aggregation at the edge nodes, then the intermediate results are uploaded to the central node for further joins and global aggregation. In this scenario, edge nodes can generate batch proofs for local filtering and aggregation operators, while the central node generates new batch proofs for subsequent operators. The recursive aggregation phase can either involve partial aggregation at the edge nodes followed by uploading the partial aggregation proofs to the central node, or centralized aggregation at the central node. The choice typically depends on network conditions and the computing power of the edge nodes. If the edge nodes possess sufficient proof capabilities, partial aggregation can reduce the amount of data uploaded; if the edge nodes are only suitable for lightweight preprocessing, the aggregation task can be left to the central node. This change does not alter the main chain of the method; it still involves batch-based payload construction, asynchronous proof generation, and layer-by-layer aggregation, only the physical entities involved in the execution differ.

[0130] In one embodiment, the reuse of existing operator monitoring information within the database engine can also be considered. Many database execution engines themselves record operator-level statistics, such as the number of input records, the number of output records, filter hit rate, join hits, or aggregation group counts. These statistics are originally used for performance analysis and query tuning, but in this solution, they can also be written into the validation payload as supplementary fields for validation information. This has two advantages. First, the statistics themselves can reflect some macroscopic characteristics of operator behavior, and the validation program can use these statistics to perform low-cost checks, such as whether the number of output records matches the number of records retained in the bitmap, and whether the join output size matches the number of matching hints. Second, these statistics are often already calculated internally by the database engine, and reusing them will not significantly increase additional overhead. Of course, statistics can only serve as auxiliary information and cannot replace input / output and key intermediate information. This is because statistics usually only reflect quantitative relationships and are insufficient to prove the correctness of specific recorded content.

[0131] The preceding implementations largely revolve around the normal execution path. However, considering retries, another issue arises: is it possible for the same batch to be proven repeatedly? This problem is not uncommon in asynchronous architectures. The main execution thread 601 may have already submitted a batch of proof tasks, but due to network timeouts, thread pool exceptions, or the result collector 606 failing to register them in time, the scheduler mistakenly believes the task has not yet been executed and submits the same batch again. To avoid duplicate aggregation, in one embodiment, a stable task identifier can be attached to each batch of proofs. The stable task identifier can be generated jointly by the query identifier, operator identifier, batch identifier, and payload digest. When the aggregator receives proof results, it first checks if a stable task identifier already exists; if it exists and the digest is consistent, the later result is marked as a duplicate and not included in the aggregation; if it exists but the digest is inconsistent, it indicates that an inconsistent payload has occurred in the same batch, which usually requires triggering exception handling. In this way, the system can maintain the idempotency of the aggregated input in the event of retries.

[0132] In one embodiment, constraints can also be placed on the relative relationship between the submission time of proof tasks and the execution time of queries. If the main execution path of a query has ended, but a large number of proof tasks are still stuck in the queue, the system can enter "convergence mode." Convergence mode refers to pausing the entry of new tasks into the queue and prioritizing the allocation of existing resources to incomplete proof and aggregation tasks, so that the final aggregated proof is generated as quickly as possible. The difference between convergence mode and normal pipeline mode lies in the shift in the focus of resource allocation. Normal mode prioritizes the continuous advancement of the main execution path, while convergence mode prioritizes the rapid closure of the proof chain. The switch between these two modes is not because the former mode has failed, but because the focus differs at different stages of query processing. If the same resource strategy is always applied, the query tail may experience situations where execution has ended but proofs are delayed for an extended period, affecting the overall response experience.

[0133] From the perspective of implementation structure, Figures 1 to 12 The proposed solution does not require a complete rewrite of the database execution engine. A more common approach is to insert payload construction logic into the processing loops of existing physical operators, add an asynchronous proof thread pool outside the existing execution threads, and add a proof aggregation step to the result output path. The advantage of this modification is that the intrusion into the native database execution path is relatively controllable. For existing database kernels, directly rewriting the entire execution framework is usually costly and not conducive to gradual deployment. With an insertive approach, key operators such as filtering, joining, and aggregation can be started first, and more operator types can be covered as needed. This gradual deployment method is more adaptable to the engineering boundaries of existing database systems.

[0134] As can be seen from the processing described in the aforementioned implementation, verification information is not collected centrally after the query ends, but is organized synchronously along with the batch execution of physical operators; proof generation is not completed serially along the main operator path, but is concurrently advanced as asynchronous tasks. In this way, the waiting position in the main query path is shifted backward, and verification tasks are gradually distributed along the batch boundaries. Combined with dedicated verification procedures for operators such as filtering, joining, and aggregation, as well as sharded proof and recursive aggregation structures, the correctness of local operator processing and the continuity of the global query chain can ultimately be included in the proof scope. For complex query scenarios, this structure is more suitable for the working mode of streaming execution engines and makes it easier to control the proof cost within a schedulable range.

[0135] In one embodiment, the electronic device can serve as Figure 1 The illustrated database query verification system 100 is the carrier entity. This electronic device may include a processor, memory, a communication interface, and a bus structure. The processor executes program instructions in the memory, which stores the database execution program, verification payload construction program, asynchronous task scheduler, verification program, and proof aggregation program. The communication interface receives structured query requests and outputs query results and aggregated proofs. The bus structure connects the processor, memory, and communication interface. In a common deployment, the processor may include a set of computational cores for database operator execution and a set of computational cores for proof task execution. The two sets of cores can be physically separated or share the same processor cluster but use different scheduling domains. The former is more conducive to avoiding the proof task and the main query path from competing for the same resources, while the latter is more likely to improve overall hardware utilization under low load.

[0136] When the program instructions in memory are executed by the processor, the electronic device can perform the following actions: receive a structured query request and generate a query execution plan including multiple physical operators based on the query semantics; drive at least a portion of the multiple physical operators to process the input data in a streaming batch processing manner; synchronously construct a verification payload corresponding to the current data batch while any physical operator is processing the current data batch; after the verification payload is constructed, submit the proof generation task corresponding to the current data batch to an asynchronous execution process independent of the main computation path of the physical operators to call the corresponding verification program to generate batch proofs; after multiple data batches have generated batch proofs respectively, aggregate these batch proofs to obtain an aggregated proof used to verify the query results. The implementation of the electronic device here is consistent with the aforementioned method implementation, only the perspective changes from the step flow to the device execution capability. The device form indicates that this solution is not limited to abstract software logic, but can be implemented in a specific processor, memory, and operating environment.

[0137] In one embodiment, the electronic device may also include a task state management area. This area can be located in memory or a dedicated cache. Its function is to record the current state of verification payloads, proof tasks, and aggregation tasks. Each record in the task state management area may include a task identifier, query identifier, operator identifier, batch identifier, payload summary, current execution stage, last update time, and exception flag. When scheduling proof tasks, the processor can first query this management area to determine whether the current batch has been submitted, is being executed, or has entered a retry state due to failure. Without this management area, managing task retries, duplicate submissions, and failure rollbacks in the asynchronous execution chain becomes difficult. Especially in cases with large query volumes and a large number of batches, if the task state only exists within a thread, it is prone to losing consistency during system fluctuations.

[0138] In one embodiment, the communication interface of an electronic device can handle two types of data exchange. One type is the exchange of query requests and results at the database level, and the other is the exchange of verification payloads, batch proofs, and aggregate proofs at the proof level. If the database execution environment and the proof execution environment are deployed on the same device, the communication interface mainly faces external queryers and external verifiers; if they are deployed separately, the communication interface also needs to forward verification payloads and proof results within the internal network. To prevent communication from becoming a bottleneck, unnecessary raw debugging information is usually not sent to the remote end during the payload construction phase. Instead, only the necessary fields that can support verification are sent first. This trade-off is important in practice. The more debugging information, the easier it is to troubleshoot, but the higher the transmission cost during remote proof; if only the minimum necessary fields are retained, the proof cost is more controllable, but the problem location will be slightly slower when errors occur. Therefore, some implementations set up a "run mode" and a "diagnostic mode," with the former prioritizing bandwidth control and the latter prioritizing the retention of more context information.

[0139] In one embodiment, the program product may also be provided as a computer-readable storage medium. This storage medium may store program instructions for performing the aforementioned methods. When these program instructions are read and executed by a processor, the device can perform actions such as receiving structured query requests, generating query execution plans, performing physical operator streaming batch processing, constructing verification payloads, submitting asynchronous proof tasks, generating batch proofs, and recursively aggregating output. The storage medium can be a disk, solid-state drive, read-only memory, erasable and rewritable memory, or other media capable of storing program code. The program product form is mentioned here because in some deployments, this solution may be delivered as a database plugin, standalone middleware, or loadable module, rather than necessarily as a native component of the database kernel provided with the device.

[0140] In one embodiment, if deployed as a database plugin, the plugin can insert into the query plan generation and operator execution chain without changing the query syntax interface. After receiving operator lifecycle events exposed by the database kernel, the plugin can register payload construction callbacks, task dispatch callbacks, and result collection callbacks. This deployment method is suitable for scenarios where existing database systems do not require large-scale kernel refactoring. The limitations of the plugin approach are also relatively clear. If the database kernel does not provide sufficiently rich operator event interfaces, the intermediate information that the plugin can obtain will be limited, and the granularity of the verification payload may have to be reduced. In contrast, if the solution is deployed as an internal module of the execution engine, it is easier to access the internal state of operators and intermediate buffers, but the engineering modifications will be more significant. Both deployment methods can achieve batch-level verification, but the richness of the verification information obtained and the modification costs differ.

[0141] In one embodiment, different sets of verification programs can be pre-configured within the device to accommodate the different characteristics of filtering, joining, and aggregation operators. These verification program sets can be indexed by operator type or by a combination of operator type and data pattern. The former is simpler to manage and suitable for scenarios with relatively fixed operator logic; the latter is more convenient for specific optimizations for different field types, different null value rules, or different aggregation functions. For example, a verification program for integer equality filtering can be simpler than a verification program for string pattern matching filtering; a verification program for single-key equality joining will differ from a verification program for multi-key combination joining in input parsing and matching prompt validation; and summation aggregation and deduplication counting aggregation differ even more in their state update relationships. After these differences are pre-organized through the verification program set, the task encapsulator 603 can directly reference the matching verification program when generating the proof task, without having to generate all the verification logic temporarily at runtime.

[0142] In one embodiment, if the electronic device supports hardware acceleration, some proof generation actions can be delegated to a dedicated acceleration unit. This dedicated acceleration unit can be an accelerator for large integer operations, hash operations, or polynomial operations, or a general-purpose parallel unit suitable for executing the underlying arithmetic processes of zero-knowledge proofs. Using an acceleration unit does not change the main chain of methods; the verification payload is still constructed first, and then the proof task is submitted. Only some tasks in the proof thread pool 605 are forwarded to dedicated hardware for execution. The practical significance of this is that some underlying operations in zero-knowledge proofs are highly repetitive, and hardware acceleration can often reduce the time spent on a single batch of proofs. However, whether to introduce an acceleration unit requires consideration of deployment complexity, development costs, and operational difficulty. For scenarios with small proof sizes or low query loads, relying entirely on general-purpose processors can also meet the requirements.

[0143] In one embodiment, a more complete running example of the entire process flow can also be provided. Assume a structured query request is received, which requests filtering records from the user table that are older than 18, joining these records with the order table by user identifier, and then counting the number of orders by region field. The plan generation unit 102 generates a physical plan 204 based on the query request, containing a filtering operator 205, a join operator 206, and an aggregation operator 207. The operator execution unit 103 begins processing user and order table data in batches. While processing the current batch of user table data, the filtering operator 205 synchronously constructs a filtering payload, which includes input records, output records, filtering parameters, and a filter bitmap. After the payload construction is complete, the asynchronous proof scheduling unit 105 encapsulates the filtering payload into a proof task and submits it. The main execution path does not wait for the proof task to complete but continues to advance to the next batch of user data and subsequent join processes. After receiving the filtered result batch, the join operator 206 constructs a join payload, which includes left and right input records, join key definitions, output records, and matching hints. Aggregation operator 207 then constructs an aggregation payload based on the join results. The aggregation payload includes input records, grouping key definitions, aggregation functions, intermediate states, and output records. Proof execution unit 106 calls the corresponding verification program to generate filtered batch proofs, join batch proofs, and aggregation batch proofs. Proof aggregation unit 107 performs recursive aggregation based on these batch proofs and their summary relationships, ultimately generating an aggregation proof corresponding to the query result. Result output unit 108 outputs the query result and the aggregation proof. In this way, the entire query synchronously generates verification information during the native streaming execution process and generates verifiable results in the independent proof path.

[0144] One easily overlooked point in this example is that the proofs for filtering, joining, and aggregation are not required to be completed strictly in the order of the main query path. The order of the main query path ensures the correct progress of data computation, while the asynchronous proof path focuses more on whether dependencies are satisfied. Filter batch proofs can occur before, after, or concurrently with a join batch proof, as long as all dependent batch proofs are completed before the final aggregation. This loose coupling is precisely one of the advantages of asynchronous architecture. If the proof execution order were also completely bound to the order of the main query path, the flexibility of asynchronous scheduling would be reduced, and the system would more closely resemble a serial proof model.

[0145] As can be seen from the aforementioned implementation methods, this scheme does not perform an external verification of the final result after the query ends. Instead, it extracts verification information synchronously during the batch execution of physical operators and concurrently advances the proof task along the batch boundaries. Since the verification payload corresponds to the operator semantics, filtering, joining, and aggregation operations can each be checked in the proof environment. Because the proof task is independent of the main operator computation path, the middle of the query execution usually does not need to wait for the proof to complete. Since batch proofs can be recursively aggregated, the final output can maintain a consistent proof form suitable for external verification. In this way, the continuous batch processing characteristic of the streaming execution engine can be preserved, and the verification overhead is more dispersed along the batch processing process rather than concentrated after the query ends. For complex query scenarios involving multiple physical operators and many intermediate results, this organization method more easily balances execution continuity and verification scalability.

[0146] It should be understood that the method steps in the foregoing embodiments can be implemented by program instructions controlling related hardware, or by dedicated circuits, programmable logic devices, or a combination thereof. Correspondingly, the systems, devices, modules, units, or components in the foregoing embodiments can be implemented in software, hardware, or a combination of both. The division of modules, units, or components is merely a logical division for the purpose of illustrating the technical solution; in actual implementation, they can be combined, split, or integrated as needed.

[0147] In one embodiment, the electronic device may include a processor, a memory, and a communication interface, wherein the memory is used to store program instructions, and the processor is used to call and execute the program instructions to implement all or part of the steps in the foregoing method embodiments. The electronic device may be a server, a terminal device, an edge computing node, a cloud computing device, or other device with data processing capabilities.

[0148] In one embodiment, this application may also be implemented in the form of a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to implement all or part of the steps in the foregoing method embodiments. The computer-readable storage medium may be a read-only memory, random access memory, flash memory, hard disk, solid-state drive, optical disk, or other non-transitory storage medium.

[0149] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0150] Furthermore, the terms "including," "comprising," and "having" used in the specification are all non-exclusive inclusions; the terms "first," "second," etc., are only used to distinguish technical features and do not indicate limitations on order, quantity, or importance. The execution order of each step in the method embodiments is also not absolutely limited. Without departing from the technical concept of this application, the steps can be adjusted in order, executed in parallel, combined, or split for execution.

Claims

1. A database query verification method, comprising: Receive a structured query request and generate a query execution plan including multiple physical operators based on the structured query request; The query execution plan is executed such that at least a portion of the plurality of physical operators process the input data in a streaming batch processing manner; During the processing of the current data batch by any of the physical operators that are processed in the streaming batch processing mode, a verification payload corresponding to the current data batch is constructed synchronously. After the verification payload is constructed, the proof generation task corresponding to the current data batch is submitted to an asynchronous execution process independent of the main computation path of the physical operator, so as to call the verification program corresponding to the physical operator to generate batch proof; After generating batch proofs for multiple data batches, the multiple batch proofs are aggregated to obtain an aggregated proof used to verify the query results corresponding to the structured query request.

2. The database query verification method as described in claim 1, wherein, The verification payload includes at least the input data, output data, operator parameters, and verification information used to characterize the physical operator processing of the current data batch.

3. The database query verification method as described in claim 2, wherein, The verification information includes execution traces, which are used to characterize the operation process information of the physical operator in processing the current data batch. The operation process information includes at least one of instruction-related information, memory access information, and state change information.

4. The database query verification method as described in claim 2 or 3, wherein, The verification program corresponding to the physical operator is invoked to generate batch proofs, including: The verification program corresponding to the physical operator is invoked to perform replay verification or constraint checks on the process of the physical operator processing the current data batch based on the input data, output data, operator parameters, and verification information, and to generate the batch proof.

5. The database query verification method as described in claim 4, wherein, The verification program runs in a zero-knowledge virtual machine, which generates zero-knowledge proofs about the execution process of the verification program while executing the verification program. The batch proofs include the zero-knowledge proofs.

6. The database query verification method as described in any one of claims 1 to 5, wherein, Before submitting the proof generation task corresponding to the current data batch to the asynchronous execution process, the process also includes: The data format in the database execution engine is converted into a native data structure that the verification program can read, and the converted data is written into the verification payload.

7. The database query verification method according to any one of claims 1 to 6, wherein, Submitting the proof generation task corresponding to the current data batch to an asynchronous execution process independent of the main computation path of the physical operator includes: The verification payload and the identifier or reference of the verification program are encapsulated into an asynchronous task unit and submitted to the proof task scheduler in a non-blocking manner, so that the physical operator can continue to process subsequent data batches after submission.

8. The database query verification method as described in any one of claims 1 to 7, wherein, The aggregation process for the multiple batches of proof includes: The multiple batch proofs are aggregated layer by layer using a recursive proof method to generate the aggregated proof.

9. The database query verification method according to any one of claims 1 to 8, wherein, The plurality of physical operators include at least one of filtering operators, connection operators, and aggregation operators, and the verification program verifies the processing results of the current data batch according to a predetermined processing semantic consistent with the corresponding physical operator.

10. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when executed by the processor, the computer program causes the electronic device to perform the database query verification method according to any one of claims 1 to 9.

Images