A system, method and device for computing SQL tasks based on Apache Flink

Abstract

Embodiments of the present application disclose a system, method and device for computing SQL tasks based on Apache Flink. In the embodiments of the present application, the system comprises a first intelligent agent, at least one second intelligent agent and a server; wherein the first intelligent agent and the at least one second intelligent agent are configured to obtain a SQL task and a corresponding target data table, and configure MPC metadata; the first intelligent agent is further configured to parse the target SQL task, and generate a first plaintext calculation result and a first ciphertext calculation result; the at least one second intelligent agent is further configured to parse the target SQL task, and generate at least one second plaintext calculation result and at least one second ciphertext calculation result; and the first intelligent agent is further configured to integrate the first plaintext calculation result, the first ciphertext calculation result, the at least one second plaintext calculation result and the at least one second ciphertext calculation result, and generate a target result. Through the above method, the efficiency and accuracy of SQL task parsing can be improved in a multi-party collaborative analysis scenario.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a system, method, and apparatus for computing SQL tasks based on Apache Flink. Background Technology

[0002] With the rapid development of the digital economy, digital advertising has become a core battleground for brand marketing. The entire digital advertising delivery chain involves users, media, advertisers, supply-side platforms (SSPs), ad exchanges (ADXs), demand-side platforms (DSPs), data management platforms (DMPs), and customer data platforms. Platform (CDP) and other parties, also known as business parties, frequently transfer data across platforms, domains, and organizations. While improving marketing efficiency, this also brings serious data privacy and security challenges. Because it is difficult to effectively integrate data from various parties without leaking original information, a serious "data silo" problem has been caused, which has restricted the implementation of key capabilities such as advertising effectiveness measurement, audience expansion, and cross-channel attribution.

[0003] In existing technologies, various Secure Multi-Party Computation (MPC) protocols have been proposed, such as those based on Garbled Circuits, Secret Sharing, Homomorphic Encryption, and Private Set Intersection (PSI). These protocols can collaboratively complete joint computations without exposing the private inputs of each party. However, these MPC protocols are mainly designed for specific computing tasks, have weak versatility, high development thresholds, and high performance overhead. Furthermore, they lack deep integration with mainstream big data processing frameworks, making it difficult to meet the industrial requirements of high throughput, low latency, and complex structured query language (SQL) analysis in advertising scenarios.

[0004] In conclusion, improving the efficiency and accuracy of SQL task parsing in multi-party collaborative analysis scenarios is a problem that needs to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a system, method and apparatus for computing SQL tasks based on Apache Flink, which can improve the efficiency and accuracy of SQL task parsing in multi-party collaborative analysis scenarios.

[0006] In a first aspect, embodiments of the present invention provide a system for computing SQL tasks based on Apache Flink. The system includes: a first agent, at least one second agent, and a server; wherein the first agent and at least one second agent are configured to acquire a Structured Query Language (SQL) task and a corresponding target data table, and configure Multi-Party Computation (MPC) metadata for at least two agents based on the target data table; the first agent is further configured to send the target SQL task and a security policy configuration file to the server, wherein the security policy configuration file includes the MPC metadata; the server is configured to send the target SQL task and the security policy configuration file to the at least one second agent; the first agent is further configured to parse the target SQL task to generate a first plaintext computation result and a first ciphertext computation result; the at least one second agent is further configured to parse the target SQL task to generate at least one second plaintext computation result and at least one second ciphertext computation result, and send the at least one second plaintext computation result and at least one second ciphertext computation result to the first agent; the first agent is further configured to integrate the first plaintext computation result, the first ciphertext computation result, the at least one second plaintext computation result, and the at least one second ciphertext computation result to generate a target result.

[0007] Secondly, embodiments of the present invention provide a method for computing SQL tasks based on Apache Flink. The method includes: obtaining a Structured Query Language (SQL) task and a corresponding target data table, wherein the target data table includes multiple data required to execute the SQL task; configuring Multi-Party Secure Computation (MPC) metadata according to the target data table, wherein the MPC metadata is the data corresponding to the SQL task; sending the target SQL task and a security policy configuration file, wherein the security policy configuration file includes the MPC metadata; parsing the target SQL task according to the security policy configuration file to generate a first plaintext computation result and a first ciphertext computation result; receiving at least one second plaintext computation result and at least one second ciphertext computation result; and integrating the first plaintext computation result, the first ciphertext computation result, the at least one second plaintext computation result, and the at least one second ciphertext computation result to generate a target result.

[0008] Optionally, the method further includes storing the target result.

[0009] Optionally, the security policy configuration file may also include: encryption parameters, Flink cluster configuration parameters, and data partitioning strategies.

[0010] Optionally, the step of parsing the target SQL task according to the security policy configuration file to generate a first plaintext calculation result and a first ciphertext calculation result specifically includes: parsing the target SQL task according to the security policy configuration file to generate an original execution plan; splitting the original execution plan to generate a plaintext plan and a ciphertext plan; generating the first plaintext calculation result according to the plaintext plan, and generating the first ciphertext calculation result according to the ciphertext plan.

[0011] Optionally, generating the first ciphertext calculation result based on the ciphertext plan specifically includes: rewriting the ciphertext plan according to the security policy configuration file to generate an updated ciphertext plan; and generating the first ciphertext calculation result based on the updated ciphertext plan.

[0012] Thirdly, embodiments of the present invention provide a method for computing SQL tasks based on Apache Flink. The method includes: obtaining a Structured Query Language (SQL) task and a corresponding target data table, wherein the target data table includes multiple data required to execute the SQL task; configuring Multi-Party Secure Computation (MPC) metadata according to the target data table, wherein the MPC metadata is the data corresponding to the SQL task; receiving a security policy configuration file for the target SQL task and a first agent, wherein the security policy configuration file includes the MPC metadata; parsing the target SQL task according to the security policy configuration file to generate a second plaintext computation result and a second ciphertext computation result; and sending the second plaintext computation result and the second ciphertext computation result.

[0013] Optionally, the step of parsing the target SQL task according to the security policy configuration file to generate a second plaintext calculation result and a second ciphertext calculation result specifically includes: parsing the target SQL task according to the security policy configuration file to generate an original execution plan; splitting the original execution plan to generate a plaintext plan and a ciphertext plan; generating the second plaintext calculation result according to the plaintext plan, and generating the second ciphertext calculation result according to the ciphertext plan.

[0014] Optionally, generating the second ciphertext calculation result based on the ciphertext plan specifically includes: rewriting the ciphertext plan according to the security policy configuration file to generate an updated ciphertext plan; and generating the second ciphertext calculation result based on the updated ciphertext plan.

[0015] Fourthly, embodiments of the present invention provide a method for computing SQL tasks based on Apache Flink, the method comprising: receiving a target SQL task and a security policy configuration file, wherein the security policy configuration file includes the multi-party secure computing metadata; and sending the target SQL task and the security policy configuration file.

[0016] Fifthly, embodiments of the present invention provide an apparatus for computing SQL tasks based on Apache Flink. The apparatus includes: a first acquisition unit, configured to acquire a Structured Query Language (SQL) task and a corresponding target data table, wherein the target data table includes multiple data required to execute the SQL task; a first calculation unit, configured to configure Multi-Party Secure Computation (MPC) metadata according to the target data table, wherein the MPC metadata is the data corresponding to the SQL task; a first sending unit, configured to send the target SQL task and a security policy configuration file, wherein the security policy configuration file includes the MPC metadata; a first generation unit, configured to parse the target SQL task according to the security policy configuration file to generate a first plaintext calculation result and a first ciphertext calculation result; a first receiving unit, configured to receive at least one second plaintext calculation result and at least one second ciphertext calculation result; and a processing unit, configured to integrate the first plaintext calculation result, the first ciphertext calculation result, the at least one second plaintext calculation result, and the at least one second ciphertext calculation result to generate a target result.

[0017] Optionally, the device further includes a storage unit for storing the target result.

[0018] Optionally, the security policy configuration file may also include: encryption parameters, Flink cluster configuration parameters, and data partitioning strategies.

[0019] Optionally, the first generation unit is specifically used to: parse the target SQL task according to the security policy configuration file to generate an original execution plan; split the original execution plan to generate a plaintext plan and a ciphertext plan; generate the first plaintext calculation result according to the plaintext plan, and generate the first ciphertext calculation result according to the ciphertext plan.

[0020] Optionally, the first generation unit is further configured to: rewrite the ciphertext plan according to the security policy configuration file to generate an updated ciphertext plan; and generate the first ciphertext calculation result according to the updated ciphertext plan.

[0021] In a sixth aspect, embodiments of the present invention provide an apparatus for computing SQL tasks based on Apache Flink. The apparatus includes: a second acquisition unit, configured to acquire a Structured Query Language (SQL) task and a corresponding target data table, wherein the target data table includes multiple data required to execute the SQL task; a second computing unit, configured to configure Multi-Party Secure Computation (MPC) metadata according to the target data table, wherein the MPC metadata is the data corresponding to the SQL task; a second receiving unit, configured to receive the target SQL task and a security policy configuration file of a first agent, wherein the security policy configuration file includes the MPC metadata; a second generation unit, configured to parse the target SQL task according to the security policy configuration file to generate a second plaintext computation result and a second ciphertext computation result; and a second sending unit, configured to send the second plaintext computation result and the second ciphertext computation result.

[0022] Optionally, the second generation unit is specifically used to: parse the target SQL task according to the security policy configuration file to generate an original execution plan; split the original execution plan to generate a plaintext plan and a ciphertext plan; generate the second plaintext calculation result according to the plaintext plan, and generate the second ciphertext calculation result according to the ciphertext plan.

[0023] Optionally, the second generation unit is further configured to: rewrite the ciphertext plan according to the security policy configuration file to generate an updated ciphertext plan; and generate the second ciphertext calculation result according to the updated ciphertext plan.

[0024] In a seventh aspect, embodiments of the present invention provide an apparatus for computing SQL tasks based on Apache Flink, the apparatus comprising: a third receiving unit for receiving a target SQL task and a security policy configuration file, wherein the security policy configuration file includes the multi-party secure computation metadata; and a third sending unit for sending the target SQL task and the security policy configuration file.

[0025] Eighthly, embodiments of the present invention provide an electronic device including a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect or any one of the possible methods of the first aspect.

[0026] In a ninth aspect, embodiments of the present invention provide a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method as described in the first aspect or any one of the possible methods of the first aspect.

[0027] In this embodiment of the invention, a system for computing SQL tasks based on Apache Flink is designed, including a first agent, at least one second agent, and a server. The first agent and at least one second agent are used to obtain a Structured Query Language (SQL) task and a corresponding target data table, and configure Multi-Party Computation (MPC) metadata for at least two agents based on the target data table. The first agent is also used to send the target SQL task and a security policy configuration file to the server, wherein the security policy configuration file includes the MPC metadata. The server is used to send the target SQL task and the security policy configuration file to the at least one second agent. The first agent is also used to parse the target SQL task, generating a first plaintext computation result and a first ciphertext computation result. The at least one second agent is also used to parse the target SQL task, generating at least one second plaintext computation result and at least one second ciphertext computation result, and sending the at least one second plaintext computation result and at least one second ciphertext computation result to the first agent. The first agent is also used to integrate the first plaintext computation result, the first ciphertext computation result, the at least one second plaintext computation result, and the at least one second ciphertext computation result to generate a target result. Through the above system, with SQL as the unified entry point, different intelligent agents can support different MPC protocols. The system includes multiple intelligent agents and can support dynamic switching of multiple MPC protocols and distributed computing, thereby improving the accuracy of SQL task parsing in multi-party collaborative analysis scenarios. Attached Figure Description

[0028] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which: Figure 1 This is a schematic diagram of a system for computing SQL tasks based on Apache Flink, as described in an embodiment of the present invention. Figure 2 This is a flowchart of a method for computing SQL tasks based on Apache Flink in an embodiment of the present invention; Figure 3 This is a schematic diagram of a splitting and rewriting method in an embodiment of the present invention; Figure 4 This is a schematic diagram of the interaction between the encrypted calculation engine and the present invention. Figure 5 This is a flowchart of another method for computing SQL tasks based on Apache Flink in an embodiment of the present invention; Figure 6This is a flowchart of another method for computing SQL tasks based on Apache Flink in an embodiment of the present invention; Figure 7 This is a flowchart of another method for computing SQL tasks based on Apache Flink in an embodiment of the present invention; Figure 8 This is a schematic diagram of an apparatus for computing SQL tasks based on Apache Flink in an embodiment of the present invention; Figure 9 This is a schematic diagram of another apparatus for computing SQL tasks based on Apache Flink in an embodiment of the present invention; Figure 10 This is a schematic diagram of another apparatus for computing SQL tasks based on Apache Flink in an embodiment of the present invention; Figure 11 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0029] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0030] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0031] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0032] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0033] In existing technologies, global regulations such as the General Data Protection Regulation (GDPR) and the Personal Information Protection Law (PIPL) impose strict requirements on user privacy protection, prohibiting the plaintext sharing and cross-domain tracking of original user identifiers. These original user identifiers include Identifier for Advertisers (IDFA), International Mobile Equipment Identity (IMEI), and Device Identifier (Device ID). Advertisers urgently need to integrate their consumer behavior data across all touchpoints, including e-commerce, physical stores, applications (APPs), and social media, to build unified user profiles and conversion attribution models for precise targeting and business growth. However, due to fragmented data ownership, security and compliance constraints, and technical barriers, it is difficult to effectively integrate data from various parties without disclosing original information, resulting in a serious "data silo" problem that hinders the implementation of key capabilities such as advertising effectiveness measurement, audience expansion, and cross-channel attribution. To address the aforementioned issues, existing technologies have proposed various Secure Multi-Party Computation (MPC) protocols, such as those based on Garbled Circuits, Secret Sharing, Homomorphic Encryption, and Private Set Intersection (PSI). These protocols collaboratively complete joint computations without exposing the private inputs of each party. However, these protocols are primarily designed for specific computational tasks, exhibiting weak versatility, high development barriers, and significant performance overhead. Furthermore, they lack deep integration with mainstream big data processing frameworks, making it difficult to meet the industrial demands of high throughput, low latency, and complex SQL analysis in advertising scenarios. Consequently, the industry has encapsulated some MPC capabilities as dedicated services, such as private intersection and joint modeling. However, these services generally suffer from closed protocols, poor scalability, and disconnect from existing data infrastructure. This is particularly problematic in advertising joint operation scenarios that require support for dynamic SQL queries, multi-stage Extract-Transform-Load (ETL), and integrated stream-batch analysis, where they cannot efficiently and accurately parse tasks. Therefore, improving the efficiency and accuracy of SQL task parsing in multi-party collaborative analysis scenarios is a problem that needs to be solved.

[0034] In this embodiment of the invention, to solve the above problems, a system for computing SQL tasks based on Apache Flink is proposed, specifically as follows: Figure 1 As shown, the system includes: a first intelligent agent 101, at least one second intelligent agent 102, and a server 103; The first agent and at least one second agent are configured to acquire a Structured Query Language (SQL) task and a corresponding target data table, and configure Multi-Party Computation (MPC) metadata for at least two agents based on the target data table. The first agent is also configured to send the target SQL task and a security policy configuration file to a server, wherein the security policy configuration file includes the MPC metadata. The server sends the target SQL task and the security policy configuration file to the at least one second agent. The first agent is further configured to parse the target SQL task, generating a first plaintext computation result and a first ciphertext computation result. The at least one second agent is further configured to parse the target SQL task, generating at least one second plaintext computation result and at least one second ciphertext computation result, and sending the at least one second plaintext computation result and at least one second ciphertext computation result to the first agent. The first agent is also configured to integrate the first plaintext computation result, the first ciphertext computation result, the at least one second plaintext computation result, and the at least one second ciphertext computation result to generate a target result.

[0035] In this embodiment of the invention, the first agent 101 and at least one second agent 102 are agents (Agents) of the Apache Flink distributed computing cluster, which are respectively deployed with privacy computing capabilities. The Apache Flink is a distributed stream processing framework that supports high throughput, low latency real-time data stream computing and provides a processing model that combines exact one-time semantics and batch-stream unified processing.

[0036] The first intelligent agent, the second intelligent agent, and the server will be described in detail below.

[0037] In this embodiment of the invention, a method for computing SQL tasks based on Apache Flink is proposed for the first intelligent agent, specifically as follows: Figure 2 As shown, it includes the following: Step S201: Obtain the Structured Query Language (SQL) task and the corresponding target data table.

[0038] Specifically, the target data table includes multiple data sets required to execute the SQL task, each with different availability and visibility.

[0039] For example, the specific definitions of data availability and visibility are shown in Table 1 below: Table 1

[0040] Among them, invisible data cannot be listed either locally or securely transmitted after encryption.

[0041] In one possible implementation, the data can be e_id, aid, e_dims, e_t, and ds. This is only an example and the specific implementation should be determined based on the actual situation.

[0042] Step S202: Configure multi-party secure computation (MPC) metadata according to the target data table.

[0043] Specifically, the Secure Multi-Party Computation (MPC) metadata is the data corresponding to the SQL task, and the MPC is based on cryptographic technology to realize multi-party data fusion and joint computation.

[0044] In one possible implementation, the MPC metadata is shown in Table 2 below: Table 2

[0045] Step S203: Send the target SQL task and security policy configuration file.

[0046] Specifically, the security policy configuration file includes the multi-party secure computation metadata, and also includes encryption parameters, Flink cluster configuration parameters, and data partitioning strategies.

[0047] In one possible implementation, the first agent submits the target SQL task and security policy configuration file to the server (Service) through a secure communication protocol. The target SQL task can also be called the target SQL job or target SQL task, which is the SQL task that needs to be parsed. This is only an example for illustration.

[0048] Step S204: Parse the target SQL task according to the security policy configuration file to generate a first plaintext calculation result and a first ciphertext calculation result.

[0049] In one possible implementation, before the first agent parses the target SQL task according to the security policy configuration file, a pre-validation is performed based on the pre-configured MPC metadata to ensure the syntactic correctness of the input target SQL task and the completeness of the column attributes of the MPC metadata (which can also be called the source table). At the same time, the first agent has authorized the metadata viewing permissions to other participants, i.e., at least one second entity. Otherwise, the target SQL task parsed jointly by multiple parties is an illegal task and cannot be submitted to multiple agents for execution.

[0050] In this embodiment of the invention, the step of parsing the target SQL task according to the security policy configuration file to generate a first plaintext calculation result and a first ciphertext calculation result specifically includes: parsing the target SQL task according to the security policy configuration file to generate an original execution plan; splitting the original execution plan to generate a plaintext plan and a ciphertext plan; generating the first plaintext calculation result according to the plaintext plan, and generating the first ciphertext calculation result according to the ciphertext plan; wherein, the step of generating the first ciphertext calculation result according to the ciphertext plan specifically includes: rewriting the ciphertext plan according to the security policy configuration file to generate an updated ciphertext plan; and generating the first ciphertext calculation result according to the updated ciphertext plan.

[0051] In one possible implementation, the architecture of the first intelligent agent integrates an MPC engine to achieve data privacy protection. The MPC engine includes a multi-party execution plan generation engine, a plaintext computation engine, and a ciphertext computation engine. The multi-party execution plan generation engine, based on the Apache Flink computing framework, is used to generate the original execution plan and traverse the original execution plan from bottom to top, performing a legality verification process and a splitting and rewriting process to generate the plaintext plan and the updated ciphertext plan. The plaintext computation engine is used to execute the plaintext plan, and the ciphertext computation engine is used to execute the updated ciphertext plan.

[0052] In this embodiment of the invention, the legality verification process combines the MPC metadata configuration of the data corresponding to the input target SQL task to complete the derivation and verification of data availability and visibility. The verification rules for different operators are shown in Table 3 below: Table 3

[0053] The operators and verification rules in Table 3 above are designed in detail according to the MPC protocol, and are only illustrated here as examples.

[0054] In one possible implementation, the splitting and rewriting process involves splitting and rewriting the original execution plan. This process traverses the original execution plan from bottom to top, starting with the input data and coloring it according to the data holder. Simultaneously, operators are rewritten, and finally, based on the coloring results, the original execution plan is split into several subgraphs. Each participant corresponds to one or more subgraphs, and each participant is an agent. The subgraphs include plaintext subgraphs and ciphertext subgraphs. The plaintext subgraphs call the plaintext computation engine to complete calculations, and the ciphertext subgraphs call the ciphertext computation engine, communicating with the ciphertext computation engines of other participants via network to execute the MPC protocol and complete calculations.

[0055] For example, the target SQL task expression template is as follows: INSERT INTO result SELECT a.id, FROM a JOIN b ON a.id = b.id; The target SQL task expression consists of INSERT statements and multiple query statements. The INSERT statements correspond to nodes that persist the analysis results to a storage service such as OSS or ODPS. The query statements correspond to operations such as JOIN and UNION, and the source table is obtained using TableScan to determine the splitting and rewriting logic. A corresponding MPC execution plan is generated, with the specific splitting and rewriting as follows: Figure 3 As shown: The original execution plan consists of a SINK node, a RelNode node, and two TableScan nodes from top to bottom. Traversing the original execution plan from bottom to top, the two TableScan nodes are colored to generate a SINK node, a RelNode node, a colored first TableScan node, and a colored second TableScan node. The two TableScan nodes are colored differently. Then, the plan is split to generate two subgraphs. Figure 1 Kazuko Figure 2 The sub Figure 1 This includes a SINK node, a RelNode node, and a first TableScan node; the sub-nodes... Figure 2 Including the first RelNode node and the second TableScan node, the two subgraphs can communicate with each other. After the split and rewrite is completed, each participant performs plaintext or ciphertext calculations based on the sub-execution plan held by itself.

[0056] In one possible implementation, the encrypted computation engine is based on multiple MPC cryptographic protocols to implement multiple sub-plans generated by the original execution plan. Different sub-plans correspond to different subgraphs, and each subgraph is computed by a corresponding operator. That is, the original execution plan generates MPC operator distributed computation logic. Specifically, the first and second intelligent agents are based on the Apache Flink framework. The Flink Runtime is the core underlying execution engine of Apache Flink, which is the foundation supporting the operation of the entire upper-layer functions of Flink. It implements and extends the Flink Process Operator. The Flink Process Operator is the lowest-level and most flexible operator base class in Apache Flink. The worker nodes of the Process Operator are implemented using a hybrid Java and Python programming approach. The Java processing function of the worker is responsible for starting the Python process and using Google Remote Procedure Call (GRP). Call (gRPC) communicates with the Python process, which is responsible for the specific implementation of the encrypted operators. The Java side is unaware of the existence of other business Flink tasks; all communication between different business parties is implemented on the Python side, which ensures the security of data communication through asymmetric encryption. The encrypted operator implementation on the Python side includes an initialization phase, a handshake phase, an encryption phase, and a termination phase. Taking two participating parties, namely the first agent Alice and the second agent Bob, the specific process is as follows: Figure 4As shown, the initialization phase is divided into Master nodes and Slave nodes based on roles. For example, the first agent Alice is the Master node, and the second agent Bob is the Slave node. The Master node registers services, and the Slave node discovers services, such as ZooKeeper. Simultaneously, a peer-to-peer transmission channel between the Master and Slave nodes is initialized using cryptography (Crypten), where Crypton is an open-source privacy-preserving machine learning framework based on PyTorch. This framework is designed to enable researchers to perform distributed model training and inference without disclosing the original data. It achieves encrypted data computation through secure multi-party computation (MPC), allowing multiple participants to collaboratively train the model without sharing plaintext data. The Master and Slave nodes generate a public-private key pair, a secret shared pseudo-random seed, based on an asymmetric key algorithm (e.g., RSA), and then exchange keys through a registered service. Simultaneously, a global random seed is generated based on a symmetric encryption algorithm for message encryption and decryption. Furthermore, other parameters of the customized MPC protocol are initialized. During the handshake phase, the Master and Slave nodes generate a signature algorithm based on the exchanged secret shared pseudo-random seed for authentication, then complete key negotiation for the message transmission encryption protocol, fulfilling the pre-processing protocol for MPC computation. In the encryption phase, the MPC protocol is executed, producing ciphertext results, which are then aggregated to one party; for example, the Slave node sends the ciphertext results to the Master node. In the termination phase, both parties' ciphertext computation engines complete their execution. This process supports distributed computation, running simultaneously on multiple workers, improving computational efficiency.

[0057] Step S205: Receive at least one second plaintext calculation result and at least one second ciphertext calculation result.

[0058] Specifically, the first agent receives at least one second plaintext calculation result and at least one second ciphertext calculation result sent by at least one second agent.

[0059] Step S206: Integrate the first plaintext calculation result, the first ciphertext calculation result, the at least one second plaintext calculation result, and the at least one second ciphertext calculation result to generate the target result.

[0060] In one possible implementation, after step S206, other steps are included, including: storing the target result, specifically, outputting the target result to a designated data storage component.

[0061] In this embodiment of the invention, a method for computing SQL tasks based on Apache Flink is proposed for the second intelligent agent, specifically as follows: Figure 5 As shown, it includes the following: Step S501: Obtain the Structured Query Language (SQL) task and the corresponding target data table.

[0062] Specifically, the target data table includes multiple data sets required to execute the SQL task, each with different availability and visibility.

[0063] Step S502: Configure multi-party secure computation (MPC) metadata according to the target data table.

[0064] Specifically, the Multi-Party Secure Computation (MPC) metadata is the data corresponding to the SQL task, and the MPC is based on cryptographic technology to realize multi-party data fusion and joint computation.

[0065] The specific processing procedures of steps S501 and S502 are the same as those of steps S201 and S202, and will not be repeated here.

[0066] Step S503: Receive the security policy configuration file of the target SQL task and the first agent.

[0067] Specifically, the security policy configuration file includes the multi-party secure computation metadata.

[0068] In one possible implementation, the second agent receives the target SQL task sent by the server and the security policy configuration file of the first agent.

[0069] Step S504: Parse the target SQL task according to the security policy configuration file to generate a second plaintext calculation result and a second ciphertext calculation result.

[0070] Specifically, the target SQL task is parsed according to the security policy configuration file to generate an original execution plan; the original execution plan is split into a plaintext plan and an ciphertext plan; a second plaintext calculation result is generated according to the plaintext plan, and a second ciphertext calculation result is generated according to the ciphertext plan. Specifically, generating the second ciphertext calculation result according to the ciphertext plan includes: rewriting the ciphertext plan according to the security policy configuration file to generate an updated ciphertext plan; and generating the second ciphertext calculation result according to the updated ciphertext plan.

[0071] Step S505: Send the second plaintext calculation result and the second ciphertext calculation result.

[0072] Specifically, the second intelligent agent transmits the second plaintext calculation result and the second ciphertext calculation result to the first intelligent agent via ciphertext transmission.

[0073] In this embodiment of the invention, a method for computing SQL tasks based on Apache Flink is proposed for the server, specifically as follows: Figure 6 As shown, it includes the following: Step S601: Receive the target SQL task and security policy configuration file.

[0074] Specifically, the security policy configuration file includes the multi-party secure computation metadata.

[0075] In this embodiment of the invention, the server is a centralized task orchestration service and a computing coordination hub. The server receives the target SQL task and security policy configuration file sent by the first intelligent agent.

[0076] Step S602: Send the target SQL task and the security policy configuration file.

[0077] Specifically, the server sends the target SQL task and the security policy configuration file to at least one second agent. For example, if the service has only two participants, it sends the task to one second agent; if the service has only three participants, it sends the task to two second agents. This is just an example.

[0078] The following example, using two participating parties, will illustrate the specific processing procedure in detail. Figure 7As shown, the system includes the following: The system for computing SQL tasks based on Apache Flink includes Agent A, a server, and Agent B; Agent A and Agent B each configure MPC metadata according to the data table required by the task, and then send the target SQL task and MPC metadata to the server. The server then sends the target SQL task and MPC metadata to Agent B; Agent A performs calculations using its own MPC engine to generate a first plaintext result and a first ciphertext result. Specifically, the MPC engine includes a multi-party execution plan generation engine, a plaintext calculation engine, and a ciphertext calculation engine; similarly, Agent B performs calculations using its own MPC engine to generate a second plaintext result and a second ciphertext result. Specifically, the MPC engine includes a multi-party execution plan generation engine, a plaintext calculation engine, and a ciphertext calculation engine; the second plaintext result and the second ciphertext result are sent to Agent A, and Agent B... A integrates the first plaintext calculation result, the first ciphertext calculation result, the at least one second plaintext calculation result, and the at least one second ciphertext calculation result to generate a target result, which can also be called an aggregated plaintext-ciphertext result, and stores the target result.

[0079] Through the above embodiments, a secure computing abstraction for cross-domain advertising data collaboration has been constructed. This allows advertisers, media outlets, DMPs, and other parties to directly use familiar SQL syntax to complete complex joint analysis tasks such as cross-domain audience insights, conversion attribution, frequency control, and performance measurement without exchanging raw data or exposing user privacy. This not only significantly lowers the barrier to entry for MPC technology but also achieves seamless integration of secure computing capabilities with the existing big data ecosystem. It provides a practical, scalable, and high-performance engineering solution for solving the core problem of "data available but not visible" in the advertising industry, helping brands achieve compliant growth across all business domains.

[0080] In this embodiment of the invention, Flink SQL is used as a unified entry point to shield the complexity of the underlying MPC, support distributed computing, support dynamic switching of multiple MPC protocols, and adapt to different security and performance trade-off scenarios. It adopts a multi-language collaborative architecture of Java and Python to balance high-performance core computing and flexible algorithm extension.

[0081] Through the above embodiments, built on the Apache Flink engine, it fully inherits its distributed, high-throughput, low-latency stream and batch computing capabilities, enabling real-time and offline joint analysis under privacy protection without reconstructing existing data links. Using Flink SQL as a unified programming interface, users can transparently execute encrypted computations without modifying existing SQL statements or learning cryptography. At the underlying level, operator-level modifications intelligently schedule plaintext operators, MPC encrypted operators, or a mixture of both, achieving enhanced security while maintaining code integrity. Through MPC metadata configuration, it supports making specified sensitive columns (e.g., user ID, phone number) invisible, while other non-sensitive fields are still efficiently computed in plaintext, balancing data availability and privacy compliance. It supports pluggable integration of multiple MPC protocols (e.g., PSI, Secret Sharing, homomorphic encryption, etc.), and has been verified in real-world scenarios to stably support multi-party secure analysis tasks with tens of millions of data points per hour. It fully supports Flink SQL with single, two, and three or more participating parties. The task covers complex multi-role collaborative analysis scenarios such as advertising attribution, cross-domain audience insight, and joint risk control. It has strong business generalization capabilities and supports cross-cloud platform deployment.

[0082] In this embodiment of the invention, an apparatus for computing SQL tasks based on Apache Flink is provided, namely a first intelligent agent, such as... Figure 8 As shown, it specifically includes: a first acquisition unit 801, a first calculation unit 802, a first transmission unit 803, a first generation unit 804, a first receiving unit 805, and a processing unit 806; The system comprises the following components: a first acquisition unit 801, configured to acquire a Structured Query Language (SQL) task and a corresponding target data table, wherein the target data table includes multiple data items required to execute the SQL task; a first calculation unit 802, configured to configure Multi-Party Computation (MPC) metadata based on the target data table, wherein the MPC metadata is the data corresponding to the SQL task; a first sending unit 803, configured to send the target SQL task and a security policy configuration file, wherein the security policy configuration file includes the MPC metadata; a first generation unit 804, configured to parse the target SQL task according to the security policy configuration file, generating a first plaintext calculation result and a first ciphertext calculation result; a first receiving unit 805, configured to receive at least one second plaintext calculation result and at least one second ciphertext calculation result; and a processing unit 806, configured to integrate the first plaintext calculation result, the first ciphertext calculation result, the at least one second plaintext calculation result, and the at least one second ciphertext calculation result to generate a target result.

[0083] Furthermore, the device also includes a storage unit for storing the target result.

[0084] Furthermore, the security policy configuration file also includes: encryption parameters, Flink cluster configuration parameters, and data partitioning strategies.

[0085] Further, the first generation unit is specifically used to: parse the target SQL task according to the security policy configuration file to generate an original execution plan; split the original execution plan to generate a plaintext plan and a ciphertext plan; generate the first plaintext calculation result according to the plaintext plan, and generate the first ciphertext calculation result according to the ciphertext plan.

[0086] Furthermore, the first generation unit is specifically used to: rewrite the ciphertext plan according to the security policy configuration file to generate an updated ciphertext plan; and generate the first ciphertext calculation result according to the updated ciphertext plan.

[0087] In this embodiment of the invention, a device for computing SQL tasks based on Apache Flink is provided, namely a second intelligent agent, such as... Figure 9 As shown, it specifically includes: The second acquisition unit 901, the second calculation unit 902, the second receiving unit 903, the second generation unit 904, and the second sending unit 905; The second acquisition unit 901 is used to acquire a Structured Query Language (SQL) task and a corresponding target data table, wherein the target data table includes multiple data required to execute the SQL task; the second calculation unit 902 is used to configure Multi-Party Secure Computation (MPC) metadata according to the target data table, wherein the MPC metadata is the data corresponding to the SQL task; the second receiving unit 903 is used to receive the target SQL task and a security policy configuration file of the first agent, wherein the security policy configuration file includes the MPC metadata; the second generation unit 904 is used to parse the target SQL task according to the security policy configuration file to generate a second plaintext calculation result and a second ciphertext calculation result; and the second sending unit 905 is used to send the second plaintext calculation result and the second ciphertext calculation result.

[0088] Further, the second generation unit is specifically used to: parse the target SQL task according to the security policy configuration file to generate an original execution plan; split the original execution plan to generate a plaintext plan and a ciphertext plan; generate the second plaintext calculation result according to the plaintext plan, and generate the second ciphertext calculation result according to the ciphertext plan.

[0089] Furthermore, the second generation unit is specifically used to: rewrite the ciphertext plan according to the security policy configuration file to generate an updated ciphertext plan; and generate the second ciphertext calculation result according to the updated ciphertext plan.

[0090] In this embodiment of the invention, an apparatus, namely a server, is provided for computing SQL tasks based on Apache Flink. Figure 10 As shown, it specifically includes: a third receiving unit 1001 and a third sending unit 1002; wherein, the third receiving unit 1001 is used to receive the target SQL task and the security policy configuration file, wherein the security policy configuration file includes the multi-party secure computation metadata; the third sending unit 1002 is used to send the target SQL task and the security policy configuration file.

[0091] Figure 11 This is a schematic diagram of the structure of the electronic device described in an embodiment of the present invention. Figure 11 As shown, it includes a general computer hardware architecture, which includes at least a processor 1101 and a memory 1102. The processor 1101 and the memory 1102 are connected via a bus 1103. The memory 1102 is adapted to store instructions or programs executable by the processor 1101. The processor 1101 can be a standalone microprocessor or a collection of one or more microprocessors. Thus, the processor 1101 executes the instructions stored in the memory 1102 to perform the method flow of the embodiments of the present invention as described above, thereby realizing data processing and control of other devices. The bus 1103 connects the above-mentioned components together, and also connects the above-mentioned components to the display controller 1104, the display device, and the input / output (I / O) device 1105. The input / output (I / O) device 1105 can be a mouse, keyboard, modem, network interface, touch input device, motion-sensing input device, printer, and other devices known in the art. Typically, the input / output device 1105 is connected to the system via an input / output (I / O) controller 1106.

[0092] The instructions stored in memory 1102 are executed by at least one processor 1101 to: obtain a Structured Query Language (SQL) task and a corresponding target data table; configure Multi-Party Computation (MPC) metadata according to the target data table; send the target SQL task and a security policy configuration file; parse the target SQL task according to the security policy configuration file to generate a first plaintext computation result and a first ciphertext computation result; receive at least one second plaintext computation result and at least one second ciphertext computation result; and integrate the first plaintext computation result, the first ciphertext computation result, the at least one second plaintext computation result, and the at least one second ciphertext computation result to generate a target result.

[0093] Specifically, the electronic device includes: one or more processors 1101 and a memory 1102. Figure 11 Take processor 1101 as an example. Processor 1101 and memory 1102 can be connected via a bus or other means. Figure 11 Taking a bus connection as an example, memory 1102, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Processor 1101 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory 1102, thereby realizing the system described above for calculating SQL tasks based on Apache Flink.

[0094] Memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store an option list, etc. Furthermore, memory 1102 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 1102 may optionally include memory remotely located relative to processor 1101, and these remote memories may be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0095] One or more modules are stored in memory 1102 and, when executed by one or more processors 1101, execute the system for computing SQL tasks based on Apache Flink in any of the above method embodiments.

[0096] As those skilled in the art will recognize, various aspects of the embodiments of the present invention can be implemented as a system, method, or computer program product. Therefore, various aspects of the embodiments of the present invention can take the form of a completely hardware implementation, a completely software implementation (including firmware, resident software, microcode, etc.), or an implementation combining software and hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." Furthermore, various aspects of the embodiments of the present invention can take the form of a computer program product implemented in one or more computer-readable media having computer-readable program code implemented thereon.

[0097] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, (but not limited to) an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media will include: an electrical connection having one or more wires, a portable computer floppy disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the context of embodiments of the present invention, a computer-readable storage medium can be any tangible medium capable of containing or storing a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0098] Computer-readable signal media may include propagated digital signals having computer-readable program code implemented therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and can communicate, propagate, or transmit a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0099] Program code implemented on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, or any suitable combination thereof.

[0100] Computer program code for performing operations relating to various aspects of embodiments of the present invention can be written in any combination of one or more programming languages, including: object-oriented programming languages ​​such as Java, Smalltalk, C++, etc.; and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code can be executed as a standalone software package entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet provided by an Internet service provider).

[0101] The flowchart illustrations and / or block diagrams of the methods, apparatus (systems), and computer program products according to embodiments of the present invention describe various aspects of the embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions (executed via the processor of the computer or other programmable data processing apparatus) create means for implementing the functions / actions specified in the flowchart and / or block diagram blocks or blocks.

[0102] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus or other means to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of writing that includes instructions that implement the functions / actions specified in flowchart and / or block diagram blocks or blocks.

[0103] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operable steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide for implementing the functions / actions specified in flowchart and / or block diagram blocks or blocks.

[0104] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0105] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding access points are provided for users to choose to authorize or refuse processing. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

Claims

1. A system for computing SQL tasks based on Apache Flink, characterized in that, The system includes: A first intelligent agent, at least one second intelligent agent, and a server; The first agent and at least one second agent are used to obtain a Structured Query Language (SQL) task and a corresponding target data table, and to configure Multi-Party Secure Computation (MPC) metadata for at least two agents based on the target data table. The first intelligent agent is also used to send a target SQL task and a security policy configuration file to the server, wherein the security policy configuration file includes the multi-party secure computation metadata; The server is used to send the target SQL task and security policy configuration file to the at least one second intelligent agent; The first intelligent agent is also used to parse the target SQL task and generate a first plaintext calculation result and a first ciphertext calculation result; The at least one second agent is further configured to parse the target SQL task, generate at least one second plaintext calculation result and at least one second ciphertext calculation result, and send the at least one second plaintext calculation result and at least one second ciphertext calculation result to the first agent; The first intelligent agent is also used to integrate the first plaintext calculation result, the first ciphertext calculation result, the at least one second plaintext calculation result, and the at least one second ciphertext calculation result to generate the target result.

2. A method for computing SQL tasks based on Apache Flink, characterized in that, The method includes: Obtain a Structured Query Language (SQL) task and its corresponding target data table, wherein the target data table includes multiple data items required to execute the SQL task; Configure multi-party secure computation (MPC) metadata according to the target data table, wherein the multi-party secure computation (MPC) metadata is the data corresponding to the SQL task; Send the target SQL task and security policy configuration file, wherein the security policy configuration file includes the multi-party secure computation metadata; The target SQL task is parsed according to the security policy configuration file to generate a first plaintext calculation result and a first ciphertext calculation result; Receive at least one second plaintext calculation result and at least one second ciphertext calculation result; The first plaintext calculation result, the first ciphertext calculation result, the at least one second plaintext calculation result, and the at least one second ciphertext calculation result are integrated to generate the target result.

3. The method according to claim 2, characterized in that, The method further includes: Store the target result.

4. The method according to claim 2, characterized in that, The security policy configuration file also includes: encryption parameters, Flink cluster configuration parameters, and data partitioning strategy.

5. The method according to claim 2, characterized in that, The step of parsing the target SQL task according to the security policy configuration file to generate a first plaintext calculation result and a first ciphertext calculation result specifically includes: The target SQL task is parsed according to the security policy configuration file to generate the original execution plan; The original execution plan is split into plaintext and ciphertext plans; The first plaintext calculation result is generated according to the plaintext plan, and the first ciphertext calculation result is generated according to the ciphertext plan.

6. The method according to claim 5, characterized in that, The step of generating the first ciphertext calculation result according to the ciphertext plan specifically includes: The ciphertext plan is rewritten according to the security policy configuration file to generate an updated ciphertext plan; The first ciphertext calculation result is generated based on the updated ciphertext plan.

7. A method for computing SQL tasks based on Apache Flink, characterized in that, The method includes: Obtain a Structured Query Language (SQL) task and its corresponding target data table, wherein the target data table includes multiple data items required to execute the SQL task; Configure multi-party secure computation (MPC) metadata according to the target data table, wherein the multi-party secure computation (MPC) metadata is the data corresponding to the SQL task; Receive the security policy configuration file of the target SQL task and the first intelligent agent, wherein the security policy configuration file includes the multi-party secure computation metadata; The target SQL task is parsed according to the security policy configuration file to generate a second plaintext calculation result and a second ciphertext calculation result; Send the second plaintext calculation result and the second ciphertext calculation result.

8. The method according to claim 7, characterized in that, The step of parsing the target SQL task according to the security policy configuration file to generate a second plaintext calculation result and a second ciphertext calculation result specifically includes: The target SQL task is parsed according to the security policy configuration file to generate the original execution plan; The original execution plan is split into plaintext and ciphertext plans; The second plaintext calculation result is generated according to the plaintext plan, and the second ciphertext calculation result is generated according to the ciphertext plan.

9. The method according to claim 8, characterized in that, The step of generating the second ciphertext calculation result according to the ciphertext plan specifically includes: The ciphertext plan is rewritten according to the security policy configuration file to generate an updated ciphertext plan; The second ciphertext calculation result is generated based on the updated ciphertext plan.

10. An apparatus for computing SQL tasks based on Apache Flink, characterized in that, The device includes: The first acquisition unit is used to acquire a Structured Query Language (SQL) task and a corresponding target data table, wherein the target data table includes multiple data required to execute the SQL task; The first computing unit is configured to configure multi-party secure computation (MPC) metadata according to the target data table, wherein the multi-party secure computation (MPC) metadata is the data corresponding to the SQL task; The first sending unit is used to send the target SQL task and the security policy configuration file, wherein the security policy configuration file includes the multi-party secure computation metadata; The first generation unit is used to parse the target SQL task according to the security policy configuration file and generate a first plaintext calculation result and a first ciphertext calculation result. The first receiving unit is configured to receive at least one second plaintext calculation result and at least one second ciphertext calculation result; The processing unit is used to integrate the first plaintext calculation result, the first ciphertext calculation result, the at least one second plaintext calculation result, and the at least one second ciphertext calculation result to generate the target result.

11. An apparatus for computing SQL tasks based on Apache Flink, characterized in that, The device includes: The second acquisition unit is used to acquire a Structured Query Language (SQL) task and a corresponding target data table, wherein the target data table includes multiple data required to execute the SQL task; The second computing unit is configured to configure multi-party secure computation (MPC) metadata according to the target data table, wherein the multi-party secure computation (MPC) metadata is the data corresponding to the SQL task; The second receiving unit is used to receive the security policy configuration file of the target SQL task and the first intelligent agent, wherein the security policy configuration file includes the multi-party secure computation metadata; The second generation unit is used to parse the target SQL task according to the security policy configuration file and generate a second plaintext calculation result and a second ciphertext calculation result. The second sending unit is used to send the second plaintext calculation result and the second ciphertext calculation result.

12. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 2-9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 2-9.

Images