Data processing method and device, equipment and storage medium

Abstract

The application discloses a data processing method, device and equipment and a storage medium, and the method comprises the following steps: obtaining a database script statement, the database script statement being associated with first sample data and second sample data, the first sample data being sample data in a local database of a first client terminal; determining a second client terminal according to the database script statement, the second sample data being sample data in a local database of the second client terminal; and performing data alignment of the first sample data and the second sample data with the second client terminal according to the database script statement and an encryption algorithm, to obtain intersection data of the first sample data and the second sample data, the intersection data being used for federated learning. The application can align sample data on different client terminals in federated learning, reduce the complexity of sample data alignment in federated learning, and improve the efficiency of sample data alignment in federated learning.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] Federated machine learning, also known as federated learning, enables collaborative data use and modeling among various parties without requiring the data to leave the local machine, making it a common method in privacy-preserving computing.

[0003] In federated learning, multiple participants in the machine learning model training process possess different but aligned data. To ensure that the modeling effect of federated learning is not significantly different from that of modeling by combining the data from each participant, data alignment between different participants is necessary before model training.

[0004] In federated learning, the data of different participants are stored locally, and the data alignment between different participants is a cross-platform or cross-network data alignment, the complexity of which needs to be reduced. Summary of the Invention

[0005] The main objective of this invention is to provide a data processing method, apparatus, device, and storage medium, addressing the technical problem of high complexity in aligning data between different client terminals in federated learning.

[0006] To achieve the above objectives, the present invention provides a data processing method applied to a first client terminal, the method comprising:

[0007] Obtain a database script statement, which is associated with a first sample data and a second sample data, wherein the first sample data is sample data in the local database of the first client terminal;

[0008] According to the database script statement, the second client terminal is determined, and the second sample data is the sample data in the local database of the second client terminal;

[0009] Based on the database script statements and encryption algorithm, the first sample data and the second sample data are aligned with the second client terminal to obtain the intersection data of the first sample data and the second sample data. The intersection data is used for federated learning.

[0010] Optionally, the database script statement includes the identification information of the first sample data. Before aligning the first sample data and the second sample data with the second client terminal, the method further includes:

[0011] Based on the identification information of the first sample data, the first sample data is obtained from the local database of the first client terminal.

[0012] Optionally, the database script statement further includes identification information of the second sample data, and determining the second client terminal based on the database script statement includes:

[0013] The second client terminal is determined based on the identification information of the second sample data and the preset sample data distribution information. The sample data distribution information is used to indicate the correspondence between the identification information of the sample data and the client terminal to which it belongs.

[0014] Optionally, the database script statement further includes sample alignment reference information, which includes one or more of the following: sample ID, sample features. The step of aligning the first sample data and the second sample data with the second client terminal according to the database script statement and the encryption algorithm includes:

[0015] According to the encryption algorithm, the data is aligned with at least one first element value and at least one second element value with the second client terminal to obtain the intersection data;

[0016] Wherein, the first element value is the element value in the first sample data that corresponds to the sample alignment reference information, and the second element value is the element value in the second sample data that corresponds to the sample alignment reference information.

[0017] Optionally, the step of aligning the data with the second client terminal using at least one first element value and at least one second element value according to the encryption algorithm to obtain the intersection data includes:

[0018] Encrypt each of the first element values ​​to obtain the first encrypted data;

[0019] The first encrypted data is sent to the second client terminal, and the second encrypted data returned by the second client is received. The second encrypted data is associated with each of the encrypted second element values.

[0020] Based on the first encrypted data and the second encrypted data, the encrypted values ​​of each first element and each encrypted value of each second element are aligned to obtain the intersection data.

[0021] Optionally, before determining the second client terminal based on the database script statement, the method further includes:

[0022] The database script statements are compiled to obtain compiled syntax units;

[0023] Based on the compiled syntax unit, the identification information of the first sample data, the identification information of the second sample data, and the sample alignment reference information are obtained.

[0024] Optionally, the database script statement is a Structured Query Language (SQL) statement, which includes the filename of the first sample data, the filename of the second sample data, and the sample alignment reference information.

[0025] Optionally, the method further includes:

[0026] The intersection data is sent to the second client terminal.

[0027] Optionally, before performing data alignment of the first sample data and the second sample data with the second client terminal, the method further includes:

[0028] The database script statement is sent to the second client terminal.

[0029] The present invention also provides a data processing apparatus, the apparatus comprising:

[0030] The acquisition module is used to acquire database script statements, which are associated with first sample data and second sample data. The first sample data is sample data in the local database of the first client terminal.

[0031] The determination module is used to determine the second client terminal based on the database script statement, wherein the second sample data is sample data in the local database of the second client terminal;

[0032] The intersection module is used to perform data alignment between the first sample data and the second sample data with the second client terminal according to the database script statement and encryption algorithm, so as to obtain the intersection data of the first sample data and the second sample data, and the intersection data is used for federated learning.

[0033] The present invention also provides a data processing apparatus, the data processing apparatus comprising: a memory, a processor, and a data processing program stored in the memory and executable on the processor, wherein the data processing program, when executed by the processor, implements the steps of the data processing method as described in any of the preceding claims.

[0034] The present invention also provides a computer-readable storage medium storing a data processing program, which, when executed by a processor, implements the steps of the data processing method as described in any of the preceding claims.

[0035] In this invention, after acquiring the database script statements associated with the first and second sample data, the first client terminal determines the second client terminal where the second sample data resides based on the database script statements. Then, according to the database script statements and an encryption algorithm, it aligns the first and second sample data with the data from the second client terminal to obtain the intersection of the first and second sample data. Therefore, while ensuring data security in federated learning, this invention utilizes database script statements to achieve data alignment between different client terminals, effectively reducing the complexity and improving the efficiency of data alignment between different client terminals. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present invention;

[0037] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of the present invention;

[0038] Figure 3 A flowchart illustrating another data processing method provided in an embodiment of the present invention;

[0039] Figure 4 Example diagram of data alignment between mechanism A and mechanism B provided in an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of the structure of a data processing device provided by the present invention;

[0041] Figure 6 This is a schematic diagram of the structure of a data processing device provided by the present invention.

[0042] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0043] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0044] Figure 1 This is an example diagram of an application scenario provided by an embodiment of the present invention.

[0045] like Figure 1As shown, the participants in federated learning include a server and K client terminals. During the federated learning process, the server distributes a global model to each client terminal. Each client terminal uses its local data to train the global model distributed by the server, obtains the trained model parameters, and uploads them to the server. The server aggregates the model parameters uploaded by each client terminal to obtain an updated global model. This process is repeated until the aggregated global model converges.

[0046] Each client terminal may include terminal equipment and / or a server.

[0047] Federated learning includes two approaches: horizontal federated learning and vertical federated learning.

[0048] In horizontal federated learning, the local data of different client terminals participating in the federated learning process contain user data from different users, but there is significant overlap in user characteristics. For example, client terminal A has user data for users a, b, c, and d, and the user characteristics in this data include age, occupation, and income. Client terminal B has user data for users e, f, and g, and the user characteristics in this data include age, income, and spending history. It is evident that client terminal A and client terminal B possess user data from different users, but both user data in client terminal A and client terminal B include the user characteristics of age and income.

[0049] In vertical federated learning, the local data of different client terminals participating in the federated learning process contain overlapping user data, but the user characteristics within this data differ. For example, client terminal A possesses user data for users a, b, c, and d, with user characteristics including age, occupation, and income. Client terminal C possesses user data for users b, c, d, and f, with user characteristics including consumption and travel records. It is evident that both client terminals A and C possess user data for users b, c, and d, but the user characteristics in client terminal A's data differ from those in client terminal C's data.

[0050] Before performing federated learning modeling in horizontal or vertical federated learning, it is necessary to align the data between different client terminals, also known as "data collision", to obtain the intersection data between different client terminals. Based on the intersection data, horizontal or vertical federated modeling is then performed.

[0051] For example, based on Figure 1In the application scenario shown, before the server distributes the global model to each client terminal, or before each client terminal trains the global model distributed by the server using local data, different client terminals need to perform data alignment, or in other words, they need to perform a brute-force attack to obtain their respective intersection data. Based on the local data and the intersection data, each client terminal trains the global model distributed by the server.

[0052] For example, client terminal A and client terminal B can be aligned based on user characteristics. The intersection of their data then consists of age and income. Client terminal A can train a global model based on the age and income of its users, and client terminal B can also train a global model based on the same data. This results in a final global model aggregated by the server that approximates the effect of modeling using user data from both client terminal A and client terminal B together.

[0053] In federated learning, user data across client terminals is typically stored in databases. Each client terminal has its own mature database scripting language for data analysis and processing, and the data remains within its own terminal. While using database scripting languages ​​to find the intersection of data is relatively simple and efficient, aligning user data from different client terminals requires database operations across multiple parties and involves data encryption. Traditional methods of data statistics and analysis using database scripting languages ​​are usually only suitable for finding the intersection of local user data on one client terminal and are difficult to directly apply to data alignment in federated learning.

[0054] In view of this, embodiments of the present invention provide a data processing method. Through database script statements related to first and second sample data, a first client terminal is instructed to align the first and second sample data. After obtaining the database script statements, the first client terminal determines the second client terminal based on the database script statements. Then, based on the database script statements and an encryption algorithm, the first client terminal aligns the first and second sample data with the second client terminal to obtain the final intersection data. Here, the first and second client terminals are different client terminals in federated learning. Therefore, embodiments of the present invention utilize the characteristic that client terminals in federated learning use databases to store user data. By employing database script statements and encryption algorithms, the alignment of sample data between different client terminals in federated learning is achieved while ensuring the data security of user data on the client terminals. This reduces the complexity of sample data alignment between different client terminals in federated learning and improves the efficiency of sample data alignment between different client terminals in federated learning.

[0055] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention. It is applied to a first client terminal, such as... Figure 2 As shown, the method may include:

[0056] Step 201: Obtain the database script statement. The database script statement is associated with the first sample data and the second sample data. The first sample data is the sample data in the local database of the first client terminal.

[0057] Database script statements are programming statements within database scripting languages. Database scripting languages ​​are non-procedural programming languages ​​used to perform operations on databases (such as creating data tables, querying databases, and updating databases).

[0058] The database script statements may include identification information of the first sample data (e.g., the database name of the database containing the first sample data, the table name of the table containing the first sample data) and identification information of the second sample data (e.g., the database name of the database containing the second sample data, the table name of the table containing the second sample data), or may include identification information of the first client terminal (e.g., the device identifier and network address of the first client terminal) and identification information of the second client terminal (e.g., the device identifier and network address of the second client terminal), used to instruct the first client terminal and the second client terminal to perform data alignment of the first sample data and the second sample data.

[0059] The first sample data refers to sample data in the local database of the first client terminal, and the second sample data refers to sample data in the local database of the second client terminal. The first sample data includes sample data for one or more samples, wherein the sample data includes the sample ID and sample features of each sample, and each sample ID is unique. Similarly, the second sample data includes sample data for one or more samples, wherein the sample data includes the sample ID and sample features of each sample, and each sample ID is unique.

[0060] In horizontal federated learning, the sample IDs in the first and second sample data are different, but the sample features overlap. For example, the first sample data includes feature a1 of sample a, and the second sample data includes sample feature a1 of sample e. In vertical federated learning, the sample IDs in the first and second sample data overlap. For example, the first sample data includes sample feature a1 of sample d, and the second sample data includes sample feature b1 of sample d.

[0061] Optionally, the sample can be users, and the sample data can be user data, including user ID and user characteristics. The user ID is unique. The user ID may include one or more of the following: user number, ID card number, bank card number, terminal device number, etc., and the user characteristics may include one or more of the following: user's name, age, occupation, income, consumption history, etc.

[0062] Optionally, retrieve pre-set database script statements.

[0063] Optionally, the database script statement input by the user can be obtained. For example, if the user inputs a database script statement on the terminal device of the first client terminal, the terminal device of the first client terminal will send the user-input database script statement to the server of the first client terminal so that the server of the first client terminal can align the first sample data and the second sample data with the server of the second client terminal.

[0064] Optionally, it can receive database script statements sent by a second client terminal.

[0065] For example, before aligning the first and second sample data, the first client terminal can send a data alignment request to the second client terminal. In response to the received data alignment request, the second client terminal sends pre-configured database script statements to the first client terminal. Alternatively, when the current time is detected to be a preset time, the second client terminal can proactively send database script statements to the first client terminal.

[0066] Step 202: Determine the second client terminal based on the database script statements.

[0067] Specifically, after obtaining the database script statement, if the database script statement includes the identification information of the second sample data, the second client terminal where the second sample data is located can be determined based on the identification information of the second sample data; or, if the database script statement includes the identification information of the second client terminal, the second client terminal where the second sample data is located can be determined based on the identification information of the second client terminal; or, if the database script statement is sent from the second client terminal to the first client terminal, the second client terminal can be determined based on the sending device of the database script statement.

[0068] Optionally, when determining the second client terminal where the second sample data is located based on the identification information of the second sample data, the second client terminal is determined based on the identification information of the second sample data and preset sample data distribution information. The sample data distribution information indicates the correspondence between the identification information of the sample data and its associated client terminal. Using the identification information of the second sample data as an index, the client terminal corresponding to the identification information of the second sample data is searched within the sample data distribution information, and that client terminal is identified as the second client terminal. Therefore, by recording the client terminals distributed across different sample data through the sample data distribution information, another client terminal can be accurately and quickly identified during data alignment.

[0069] Step 203: Based on the database script statements and encryption algorithm, perform data alignment between the first sample data and the second sample data with the second client terminal to obtain the intersection data of the first sample data and the second sample data. The intersection data is used for federated learning.

[0070] To ensure data security, an asymmetric encryption algorithm is used. Examples of asymmetric encryption algorithms include RSA, Digital Signature Algorithm (DSA), and Elliptic Curve Cryptography (ECC).

[0071] Specifically, after obtaining the database script statement, the first client terminal retrieves the first sample data according to the database script statement. The first client terminal can encrypt the first sample data using an encryption algorithm. The second client terminal can determine the second sample data using the same database script statement, encrypt the second sample data using the same encryption algorithm, and send the encrypted second sample data to the first client terminal. The first client terminal performs data alignment based on the first and second sample data that have undergone the same encryption process. The intersection data of the encrypted first sample data and the encrypted second sample data is then used to obtain the intersection data of the first and second sample data based on the correspondence between the first sample data and the encrypted first sample data. For example, it obtains the user ID that both the first and second sample data possess.

[0072] When the first client terminal encrypts the first sample data using the encryption algorithm, it can encrypt some or all of the data in the first sample data. For example, it can encrypt the sample ID of each sample in the first sample data, or encrypt the sample features of each sample in the first sample data, or encrypt both the sample ID and sample features of each sample in the first sample data. Similarly, when the second client terminal encrypts the second sample data using the encryption algorithm, it can encrypt some or all of the data in the second sample data.

[0073] If the sample IDs of each sample in the first sample data are encrypted, and the sample IDs of each sample in the second sample data are also encrypted, then the first client terminal and the second client terminal can obtain the intersection data of the sample IDs of the first sample data and the sample IDs of the second sample data. Similarly, if the sample features of each sample in the first sample data are encrypted, and the sample features of each sample in the second sample data are also encrypted, then the first client terminal and the second client terminal can obtain the intersection data of the sample features of the first sample data and the sample features of the second sample data.

[0074] Optionally, the first client terminal and the second client terminal can respectively perform data alignment on the encrypted first sample data and the encrypted second sample data, and each obtains the intersection data; or, the data alignment on the first client terminal is performed on the encrypted first sample data and the encrypted second sample data is performed on the second client terminal, and the first client terminal sends the obtained intersection data to the second client terminal; or, the data alignment on the second client terminal is performed on the encrypted first sample data and the encrypted second sample data is performed on the second client terminal, and the second client terminal sends the intersection data to the first client terminal.

[0075] Optionally, when retrieving the first sample data in the first client terminal according to the database script statement, all local sample data in the local database on the first client terminal can be retrieved, or the local sample data for data alignment pre-set by the user in the local database on the first client terminal can be retrieved. Alternatively, when retrieving the first sample data in the first client terminal according to the database script statement, the sample data corresponding to the identifier information of the first sample data can be searched in the local database on the first client terminal based on the identifier information of the first sample data, thus obtaining the first sample data. Therefore, the first sample data for data alignment can be specified through the database script statement.

[0076] The data processing method provided in this embodiment leverages the characteristic of client terminals in federated learning using databases to store user data. It employs database script statements and encryption algorithms to achieve sample data alignment between the first and second client terminals. Therefore, while ensuring the data security of user data on the client terminals, it achieves sample data alignment between different client terminals in federated learning, reducing the complexity of sample data alignment between different client terminals and improving the efficiency of sample data alignment between different client terminals in federated learning.

[0077] In some embodiments, the database script statement further includes sample alignment reference information, which includes one or more of the following: sample ID, sample feature, etc. The sample alignment reference information is used to determine the range for data alignment between the first sample data and the second sample data. If the sample alignment reference information is a sample ID, the database script statement indicates that the sample IDs of each sample in the first sample data are aligned with the sample IDs of each sample in the second sample data. If the sample alignment reference information is a sample feature, the database script statement indicates that the sample features of each sample in the first sample data are aligned with the sample features of each sample in the second sample data. Therefore, based on the database script statement, the alignment of sample IDs and / or sample features between the first and second sample data is achieved.

[0078] When the database script statement includes sample alignment reference information, and the sample alignment reference information includes sample ID and / or sample features, according to the encryption algorithm, the first client terminal and the second client terminal perform data alignment of at least one first element value and at least one second element value to obtain intersection data. Here, the first element value is the element value in the first sample data corresponding to the sample alignment reference information, and the second element value is the element value in the second sample data corresponding to the sample alignment reference information.

[0079] Specifically, the first client terminal retrieves sample alignment reference information from a database script statement and obtains the first element value corresponding to the sample alignment reference information from the first sample data. The first client terminal encrypts each first element value using an encryption algorithm. The second client terminal retrieves sample alignment reference information from a database script statement and obtains the second element value corresponding to the sample alignment reference information from the second sample data, and encrypts each second element value using an encryption algorithm. Based on the intersection of the first and second element values ​​after the same encryption process, the first client terminal can determine the intersection data between the first and second sample data.

[0080] In the case where the database script statement includes the identification information of the first sample data, the identification information of the second sample data, and the sample alignment reference information, and the sample alignment reference information includes the sample ID and / or sample features, Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention. It is applied to a first client terminal, such as... Figure 3 As shown, the method may include:

[0081] Step 301: The first client terminal obtains the database script statements.

[0082] Step 302: The second client terminal obtains the database script statements.

[0083] Optionally, the first client terminal obtains a database script statement pre-set on the first client terminal, and the second client terminal obtains a database script statement pre-set on the second client terminal, wherein the database script statement of the first client terminal is consistent with the database script statement of the second client terminal.

[0084] Optionally, the first client terminal obtains the database script statement input by the user, and the second client terminal obtains the database script statement input by the user. For example, the user of the first client terminal and the user of the second client terminal agree to input the same database statement into the first client terminal and the second client terminal, respectively.

[0085] Optionally, the first client terminal obtains the database script statement input by the user and sends the database script statement to the second client terminal; or, the second client terminal obtains the database script statement input by the user and sends the database script statement to the second client terminal. Therefore, it is essential to ensure that the database script statement obtained by the first client terminal is consistent with the database script statement obtained by the second client terminal.

[0086] The contents of the database script statements, the first sample data, and the second sample data can be referred to in the aforementioned embodiments, and will not be repeated here.

[0087] It should be noted that steps 301 and 302 can be executed simultaneously or asynchronously. There is no restriction on the order in which steps 301 and 302 are executed.

[0088] Step 303: The first client terminal obtains at least one first element value from the first sample data according to the database script statement, and encrypts each first element value to obtain the first encrypted data.

[0089] Specifically, the first client terminal can obtain the data identifier and sample alignment reference information of the first sample data in the database script statement. Based on the data identifier of the first sample data, the first client terminal obtains the first sample data from the local database. After obtaining the first sample data, it retrieves at least one element value corresponding to the sample alignment reference information from the first sample data to obtain at least one first element value.

[0090] If the sample alignment reference information includes a sample ID, then the element value corresponding to the sample ID is obtained from the first sample data. For example, if the first sample data includes sample data of sample a, sample b, sample c, and sample d, and the sample IDs of sample a, sample b, sample c, and sample d are 1, 2, 3, and 4 respectively, then the element values ​​corresponding to the sample IDs in the first sample data include 1, 2, 3, and 4.

[0091] If the sample alignment reference information includes sample features, then the element value corresponding to that sample feature is obtained from the first sample data. For example, if the sample alignment reference information includes user age, then the user age of each sample needs to be obtained from the first sample data.

[0092] Specifically, since asymmetric encryption algorithms use public and private keys, to ensure data security, the first client terminal and the second client terminal can each hold one of the public and private keys. For example, the first client terminal holds the public key, and the second client terminal holds the private key; or, the first client terminal holds the private key, and the second client terminal holds the public key. Taking the first client terminal holding the public key as an example, the first client terminal can encrypt each first element value according to the public key and the encryption algorithm to obtain the first encrypted data.

[0093] Step 304: The first client terminal determines the second client terminal based on the database script statement.

[0094] Specifically, the first client terminal can obtain the identification information of the second sample data in the database script statement, and determine the second client terminal based on the identification information of the second sample data and the preset sample data distribution information. For details, please refer to the relevant content in the foregoing embodiments, which will not be repeated here.

[0095] Optionally, after obtaining the database script statement, the first client terminal inputs the database script statement into a compiler to compile it, obtaining a compiled syntax unit. Based on the compiled syntax unit, it obtains the identification information of the first sample data, the identification information of the second sample data, and the sample alignment reference information. The syntax unit is machine language that can be understood by a computer, and the identification information of the first sample data, the identification information of the second sample data, and the sample alignment reference information can be directly read from the compiled syntax unit.

[0096] Step 305: The first client terminal sends the first encrypted data to the second client terminal.

[0097] Specifically, after obtaining the first encrypted data, the first client terminal sends the first encrypted data to the second client terminal, and the first encrypted data is used in the data encryption process of the second client terminal.

[0098] Step 306: The second client terminal obtains at least one second element value from the second sample data according to the database script statement, and obtains the second encrypted data according to the first encrypted data and each second element value.

[0099] The second encrypted data is associated with each encrypted second element value.

[0100] Specifically, the process by which the second client terminal obtains at least one second element value from the second sample data according to the database script statement can be referred to the process by which the first client terminal obtains at least one first element value from the first sample data according to the database script statement, and will not be repeated here.

[0101] Specifically, after receiving the first encrypted data, the second client terminal can process the first encrypted data using a private key and an encryption algorithm to obtain the first encrypted data encrypted with the private key. The second client terminal then processes each second element value using the private key and the encryption algorithm to obtain each second element value encrypted with the private key. The second encrypted data is then obtained from the first encrypted data and the second element values ​​encrypted with the private key. Therefore, by encrypting the first sample data at the first client terminal and encrypting the second sample data at the second client terminal, the data security of both the first and second sample data is ensured.

[0102] Optionally, after obtaining the database script statement, the second client terminal inputs the database script statement into the compiler to compile the database script statement and obtain the compiled syntax unit. Based on the compiled syntax unit, it obtains the identification information of the first sample data, the identification information of the second sample data, and the sample alignment reference information.

[0103] Step 307: The second client terminal sends the second encrypted data to the first client terminal.

[0104] Step 308: The first client terminal aligns the encrypted first element values ​​and encrypted second element values ​​based on the first encrypted data and the second encrypted data to obtain the intersection data.

[0105] Specifically, after receiving the second encrypted data, the first client terminal, since the second encrypted data includes first encrypted data encrypted with a private key and a second element value encrypted with a private key, and since the first encrypted data is a first element value encrypted with a public key, restores the public key-encrypted portion of the first encrypted data based on the first encrypted data and the first encrypted data encrypted with a private key, obtaining only the first element value encrypted with the private key. After obtaining the first element value encrypted with a private key and the second element value encrypted with a private key, the first client terminal can perform an intersection operation on them to obtain the intersection data of the first element value encrypted with a private key and the second element value encrypted with a private key. Since the first client terminal possesses the first element value, based on the intersection data of the first element value encrypted with a private key and the second element value encrypted with a private key, it can obtain the intersection data of the first element value and the second element value, which is also the intersection data of the first sample data and the second sample data obtained based on the database script statement. Therefore, throughout the entire process of processing the first and second sample data at the first and second client terminals, the data security of the first and second sample data is ensured, as is the data security of the intersection data between the first and second sample data, ultimately completing the alignment between the first and second sample data.

[0106] The data processing method provided in this embodiment, through database script statements and encryption algorithms including the identification information of the first sample data, the identification information of the second sample data, and sample alignment reference information, achieves data alignment of the first sample data and the second sample data between the first client terminal and the second client terminal while ensuring the data security of the first sample data and the second sample data. Moreover, by combining database script language with data encryption based on database script statements and data encryption, and taking advantage of the simplicity and efficiency of the database script language in solving intersection problems, the efficiency of data alignment between the first client terminal and the second client terminal in federated learning is improved.

[0107] In some embodiments, when encrypting each first element value, to prevent a second client terminal from decrypting the first element value using its private key, the first client terminal can first generate a corresponding random number for each first element value; different element values ​​correspond to different random numbers. The first client terminal encrypts the random data using its public key to obtain the obfuscation factor for each first element value. A hash value for each first element is obtained using a hash algorithm. Based on the obfuscation factor and hash value of each first element, the public key encryption value corresponding to each first element is obtained. The public key encryption values ​​corresponding to each first element constitute the first encrypted data.

[0108] Correspondingly, after receiving the first encrypted data, the second client terminal can process the public key encrypted values ​​in the first encrypted data using its private key to obtain the first encrypted data encrypted with the private key. Since the hash value of the first element in each public key encrypted value in the first encrypted data is not encrypted with the public key, only the obfuscation factor is encrypted with the public key, the processing of each public key encrypted value by the second client terminal using its private key is equivalent to decrypting the obfuscation factor in each public key encrypted value and encrypting the public key hash value of each first element in each encrypted value.

[0109] Accordingly, when the second client terminal encrypts each second element value using its private key, it can first obtain the hash value of each second element value using the same hash algorithm as the first client terminal. Then, it encrypts the hash value corresponding to each second element value using its private key to obtain the encrypted private key value for each second element value. Therefore, the second encrypted data includes the encrypted private key values ​​of each second element value and the first encrypted data encrypted with the private key. The encrypted private key values ​​in the first encrypted data are the first element values ​​encrypted with the obfuscation factor and the private key.

[0110] Accordingly, after receiving the second encrypted data, the first client terminal, having stored obfuscation factors for each first element value, can process each private key encrypted value in the first encrypted data to remove the obfuscation factors, obtaining only the first element value encrypted with the private key. Therefore, the first client terminal obtains both the first and second element values ​​encrypted with the private key, and can further obtain the intersection of these two values.

[0111] As an example, the first client terminal first generates a corresponding random number for each first element value, encrypts the random number using a public key, and obtains the obfuscation factor for each first element value. Then, it obtains the hash value of each first element value, obfuscates the hash value using the obfuscation factor, and obtains the obfuscated result for each first element value. Finally, it performs a modulo operation on the obfuscated results of each first element value, and the resulting obfuscated results are used to compose the first encrypted data. For example, the first encrypted data Y... A It can be represented as:

[0112] Y A ={ri e %n*H(ui)%n}, where the public key is represented as (n,e), r i u is a random number corresponding to the i-th first element value. i ∈X A Let X represent the value of the i-th first element. A H(u) represents the set of all first-element values.i ) represents u i The hash value.

[0113] As an example, in the process of encrypting each second element value using a preset private key, the second client terminal first obtains the hash value of each second element value, then hashes these multiple hash values ​​again to obtain the final hash value. Finally, it encrypts the final hash value using the private key to obtain the encrypted private key value corresponding to each second element value. For example, the set Z of encrypted private key values ​​corresponding to each second element value. B It can be represented as:

[0114] Z B ={H(H(u)} j )) d %n}, where u j ∈X B Let X represent the value of the i-th second element. B Let d represent the set of all values ​​of the second element, and let (n, d) be the private key.

[0115] As an example, the second client uses its private key to encrypt the first encrypted data, resulting in the first encrypted data Z encrypted with the private key. A It can be represented as:

[0116]

[0117] As an example, after receiving the second sub-encrypted data, the first client terminal processes the first encrypted data encrypted with the public key to obtain the set of first element values ​​encrypted with the private key:

[0118] D A ={H(ri*(H(ui))} d / ri%n)}={H(H(u i )) d Therefore, the first client terminal can D. A and Z B The intersection is calculated to obtain the intersection data. In this formula, the first client terminal corresponds to Z. A A simulation operation (i.e., the division modulo operation in the formula) was performed to remove the random number r. i The effect of the obfuscation factor is used to obtain the first sample data encrypted with the private key.

[0119] In some embodiments, the database script statement is a Structured Query Language (SQL) statement, which includes the filenames of the first sample data, the filenames of the second sample data, and sample alignment reference information. For example, the SQL statement can be represented as:

[0120] "select featureA,featureB from A join B on ID_A=ID_B".

[0121] Where A represents the first sample data and B represents the second sample data. ID_A and ID_B are sample alignment reference data, where ID_A represents the sample ID in the first sample data and ID_B represents the sample ID in the second sample data. featureA represents the sample feature in the first sample data and featureB represents the sample feature in the second sample data. Since the sample features in the second sample data are unknown to the first client terminal, featureB can be understood as all sample features in the second sample data.

[0122] Specifically, since SQL statements are a database scripting language, after obtaining the SQL statement, the first client terminal can input it into a compiler to convert it into machine language, resulting in a syntax unit that the machine can understand. From this syntax unit, the identification information of the first sample data, the identification information of the second sample data, and the sample alignment reference information are obtained.

[0123] As an example, Figure 4 This is an example diagram illustrating data alignment between Institution A and Institution B. Institution A can be understood as the first client terminal mentioned above, and Institution B as the second client terminal. Data table A represents the first sample data, and data table B represents the second sample data. The target SQL represents the database script statement. The Federated AI Technology Enabler (FATE) system is a computing framework for federated learning. The FATE system includes a compiler, a distributed computing system, and a storage system, enabling compilation, computation, and storage operations during the federated learning process.

[0124] like Figure 4 As shown, Organization A and Organization B compile the target SQL to obtain syntax units. Based on the syntax units and sample data distribution information, they obtain the distribution information of data tables A and B, i.e., the client terminals where data table A and data table B reside. They can also obtain sample alignment reference information from the syntax units. Then, based on the sample reference alignment information, they perform the data encryption and alignment operations described in any of the above method embodiments on data tables A and B in the distributed computing system.

[0125] The FATE system can reside on the servers of Organization A and Organization B. Users of Organization A and Organization B can input the target SQL statement on their respective terminal devices. The terminal devices then send the target SQL statement to their respective servers.

[0126] In some embodiments, the second client terminal stores a public key and a private key, and can send the public key to the first client terminal. Alternatively, the first client terminal can store a public key and a private key, and can send the public key to the second client terminal, thereby enabling the first client terminal and the second client terminal to perform the above-mentioned data encryption operations using a unified public key and private key.

[0127] Figure 5 This is a schematic diagram of a data processing device provided in an embodiment of the present invention. Figure 5 As shown, the data processing device may include:

[0128] The acquisition module 501 is used to acquire database script statements. The database script statements are associated with first sample data and second sample data. The first sample data is sample data in the local database of the first client terminal.

[0129] The determination module 502 is used to determine the second client terminal based on the database script statement, and the second sample data is the sample data in the local database of the second client terminal;

[0130] The intersection module 503 is used to align the first sample data and the second sample data with the second client terminal according to the database script statement and encryption algorithm to obtain the intersection data of the first sample data and the second sample data. The intersection data is used for federated learning.

[0131] The data processing device provided in this embodiment can be used to execute the technical solutions provided in any of the foregoing method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0132] In one possible implementation, the database script statement includes the identification information of the first sample data, and the acquisition module 501 is further configured to: acquire the first sample data from the local database of the first client terminal based on the identification information of the first sample data.

[0133] In one possible implementation, the database script statement also includes identification information of the second sample data. The determining module 502 is specifically used to: determine the second client terminal based on the identification information of the second sample data and the preset sample data distribution information. The sample data distribution information is used to indicate the correspondence between the identification information of the sample data and the client terminal to which it belongs.

[0134] In one possible implementation, the database script statement further includes sample alignment reference information, which includes one or more of the following: sample ID, sample features. The intersection module 503 is specifically used to: perform data alignment with the second client terminal based on the encryption algorithm, with at least one first element value and at least one second element value, to obtain intersection data; wherein, the first element value is the element value in the first sample data that corresponds to the sample alignment reference information, and the second element value is the element value in the second sample data that corresponds to the sample alignment reference information.

[0135] In one possible implementation, the intersection module 503 is specifically used to: encrypt each first element value to obtain first encrypted data; send the first encrypted data to a second client terminal and receive second encrypted data returned by the second client, wherein the second encrypted data is associated with each encrypted second element value; and perform data alignment on each encrypted first element value and each encrypted second element value according to the first encrypted data and the second encrypted data to obtain intersection data.

[0136] In one possible implementation, the data processing device further includes a compilation unit for compiling database script statements to obtain compiled syntax units. The acquisition module 501 is further configured to: obtain identification information of the first sample data, identification information of the second sample data, and sample alignment reference information based on the compiled syntax units.

[0137] In one possible implementation, the database script statement is a Structured Query Language (SQL) statement, which includes the filename of the first sample data, the filename of the second sample data, and sample alignment reference information.

[0138] In one possible implementation, the data processing apparatus further includes a transceiver unit 504 for sending the intersection data to a second client terminal.

[0139] In one possible implementation, the transceiver module 504 is also used to: send database script statements to a second client terminal.

[0140] The data processing apparatus provided in any of the foregoing embodiments is used to execute the technical solutions of any of the foregoing method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0141] Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention. Figure 6As shown, the device may include: a memory 601, a processor 602, and a data processing program stored in the memory 601 and executable on the processor 602. When the data processing program is executed by the processor 602, it implements the steps of the data processing method as described in any of the foregoing embodiments.

[0142] Optionally, the memory 601 can be either standalone or integrated with the processor 602.

[0143] The implementation principle and technical effects of the device provided in this embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0144] This invention also provides a computer-readable storage medium storing a data processing program, which, when executed by a processor, implements the steps of the data processing method as described in any of the foregoing embodiments.

[0145] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0146] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention.

[0147] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0148] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0149] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0150] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0151] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0152] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0153] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0154] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A data processing method, characterized in that, Applied to the first client terminal, including: A database script statement is obtained, which is associated with first sample data and second sample data. The first sample data is sample data in the local database of the first client terminal. The database script statement includes the identification information of the first sample data and the identification information of the second sample data. The sample data in the first sample data and the second sample data includes the sample ID and sample features of the sample. Based on the identification information of the first sample data, the first sample data is obtained from the local database of the first client terminal; In the preset sample data distribution information, the identification information of the second sample data is used as an index to find the customer terminal corresponding to the identification information of the second sample data, and the customer terminal is identified as the second customer terminal. The second sample data is the sample data in the local database of the second customer terminal. The sample data distribution information is used to indicate the correspondence between the identification information of the sample data and the customer terminal to which it belongs. Based on the database script statements and encryption algorithm, the first sample data and the second sample data are aligned with the second client terminal to obtain the intersection data of the first sample data and the second sample data. The intersection data is used for federated learning.

2. The method according to claim 1, characterized in that, The database script statement also includes sample alignment reference information, which includes one or more of the following: sample ID, sample features. The step of aligning the first sample data and the second sample data with the second client terminal according to the database script statement and the encryption algorithm includes: According to the encryption algorithm, the data is aligned with at least one first element value and at least one second element value with the second client terminal to obtain the intersection data; Wherein, the first element value is the element value in the first sample data that corresponds to the sample alignment reference information, and the second element value is the element value in the second sample data that corresponds to the sample alignment reference information.

3. The method according to claim 2, characterized in that, The step of aligning the data with the second client terminal using at least one first element value and at least one second element value according to the encryption algorithm to obtain the intersection data includes: Encrypt each of the first element values ​​to obtain the first encrypted data; The first encrypted data is sent to the second client terminal, and the second encrypted data returned by the second client is received. The second encrypted data is associated with each of the encrypted second element values. Based on the first encrypted data and the second encrypted data, the encrypted values ​​of each first element and each encrypted value of each second element are aligned to obtain the intersection data.

4. The method according to claim 2, characterized in that, Before determining the second client terminal based on the database script statement, the method further includes: The database script statements are compiled to obtain compiled syntax units; Based on the compiled syntax unit, the identification information of the first sample data, the identification information of the second sample data, and the sample alignment reference information are obtained.

5. The method according to claim 2, characterized in that, The database script statement is a Structured Query Language (SQL) statement, which includes the filename of the first sample data, the filename of the second sample data, and the sample alignment reference information.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The intersection data is sent to the second client terminal.

7. The method according to any one of claims 1-5, characterized in that, Before performing data alignment of the first sample data and the second sample data with the second client terminal, the method further includes: The database script statement is sent to the second client terminal.

8. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire a database script statement, which is associated with first sample data and second sample data. The first sample data is sample data in the local database of the first client terminal. The database script statement includes the identification information of the first sample data and the identification information of the second sample data. The sample data in the first sample data and the second sample data includes the sample ID and sample features of the sample. The module acquires the first sample data from the local database of the first client terminal based on the identification information of the first sample data. The determination module is used to find the client terminal corresponding to the identification information of the second sample data in the preset sample data distribution information, using the identification information of the second sample data as an index, and determine the client terminal as the second client terminal. The second sample data is sample data in the local database of the second client terminal. The sample data distribution information is used to indicate the correspondence between the identification information of the sample data and the client terminal to which it belongs. The intersection module is used to perform data alignment between the first sample data and the second sample data with the second client terminal according to the database script statement and encryption algorithm, so as to obtain the intersection data of the first sample data and the second sample data, and the intersection data is used for federated learning.

9. A data processing device, characterized in that, The data processing device includes: a memory, a processor, and a data processing program stored in the memory and executable on the processor, wherein the data processing program, when executed by the processor, implements the steps of the data processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a data processing program, which, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1-7.

Images