Federal data processing method and system based on multistage adaptive acceleration strategy

Abstract

The invention discloses a federal data processing method and system based on a multistage adaptive acceleration strategy. The method comprises the steps that an initiator obtains federal joint data set scale information; based on the scale information and a preset first threshold value and a preset second threshold value, adaptively selecting a CPU distributed parallel computing mode, a GPU single-card parallel computing mode or a GPU multi-card parallel computing mode; according to the selected mode, collaborative partners execute federal feature-based collinearity calculation under privacy protection to obtain a feature correlation coefficient matrix of the joint data set; and finally, calculating a variance expansion factor VIF of each feature and screening the features. According to the method, optimal computing resources are dynamically matched through a multi-stage self-adaptive strategy, the performance bottleneck problem faced by large-scale feature collinearity computing in federated learning is solved, and on the premise that data privacy is guaranteed, the order of magnitude of computing efficiency is improved.

Description

Technical Field

[0001] This invention relates to the fields of privacy computing and high-performance computing, and more specifically, to a federated data processing method and system based on a multi-level adaptive acceleration strategy. Background Technology

[0002] With increasingly stringent data security and privacy regulations, federated learning, a distributed machine learning paradigm that allows "the model to move while the data remains stationary," has been widely applied in fields such as finance and healthcare. In vertical federated learning scenarios, collinearity partners possess the same sample population but different feature dimensions. Feature engineering is required before joint modeling, and feature multicollinearity diagnosis is a crucial step. Variance inflation factor (VIF) is a classic metric for measuring feature multicollinearity, but its calculation depends on the correlation coefficient matrix of all features.

[0003] In a federated environment, calculating the feature correlation coefficient matrix between the initiator and collaborators faces two core challenges: first, privacy and security, ensuring that the raw data of either party is not leaked during the calculation process; second, computational performance, as feature collinearity calculation is inherently a computationally intensive task involving large-scale matrix multiplication, iterative inversion, and other operations. Existing technical solutions mainly suffer from the following shortcomings: 1. Fixed computing models lead to performance bottlenecks: Existing solutions typically employ either a single CPU cluster for distributed computing or a single GPU for accelerated computing. For small to medium-sized datasets (less than tens of millions), GPU acceleration may actually be inefficient due to data transfer and initialization overhead; while for large-scale datasets (hundreds of millions or more), pure CPU computing is extremely time-consuming and cannot meet the real-time requirements of business operations.

[0004] 2. Lack of dynamic resource scheduling strategies: Current federated computing frameworks fail to effectively combine the elastic scalability of distributed systems with the high parallel computing capabilities of GPU hardware. They either rely entirely on frameworks such as Spark for CPU-based distributed computing or simply offload all computing to a single GPU card, lacking intelligent strategies for dynamically partitioning tasks and allocating computing resources (multiple CPU cores, multiple GPU cards) based on data scale and feature dimensions.

[0005] 3. Severe performance overhead under secure computation protocols: To protect privacy, federated feature collinearity computation requires the use of secure multi-party computation protocols such as secret sharing. These protocols introduce a large number of communication rounds and additional local computations (such as encoding and masking operations), further degrading the efficiency of already time-consuming matrix multiplication operations, making them almost infeasible for large-scale data.

[0006] Therefore, there is an urgent need for a federated data processing method that can intelligently adapt to different data scales and efficiently utilize heterogeneous computing resources (CPU clusters, single GPUs, and multiple GPUs) under the privacy constraints of federated learning, in order to support the large-scale, high-dimensional joint modeling needs of actual business applications. Summary of the Invention

[0007] This invention provides a federated data processing method and system based on a multi-level adaptive acceleration strategy to solve the problems of performance bottlenecks caused by fixed computing modes, lack of dynamic resource scheduling strategies, and severe performance loss under secure computing protocols in the prior art.

[0008] To achieve the above objectives, on the one hand, this invention provides a federated data processing method based on a multi-level adaptive acceleration strategy. The method includes: S1, the initiator obtains the local dataset size information of the collaborator and calculates the joint dataset size information; based on the joint dataset size information, a preset first threshold, and a preset second threshold, it automatically selects a calculation mode matching the joint dataset size information; S2, based on the selected calculation mode, it collaborates with the collaborator to perform federated feature collinearity calculation under privacy protection conditions to obtain a feature correlation coefficient matrix of the joint dataset; wherein, the federated feature collinearity calculation involves a secure multiplication protocol based on secret sharing; S3, based on the feature correlation coefficient matrix, it calculates the variance inflation factor (VIF) of each feature and performs feature filtering based on the VIF to obtain a feature set for joint modeling; S4, it performs multi-round model training on the feature set to obtain a joint risk control model; S5, it inputs the data to be detected into the joint risk control model for prediction to obtain a risk control assessment result.

[0009] Optionally, the joint dataset size information is the product of the sample size and the number of features of the joint dataset; the automatic selection of a computing mode matching the joint dataset size information based on the joint dataset size information, a preset first threshold, and a preset second threshold includes: when the joint dataset size information is less than the preset first threshold, selecting the CPU+RDD distributed computing mode; when the joint dataset size information is greater than or equal to the preset first threshold and less than the preset second threshold, selecting the distributed and GPU single-card accelerated computing mode; when the joint dataset size information is greater than or equal to the preset second threshold, selecting the distributed and GPU multi-card parallel accelerated computing mode.

[0010] Optionally, when the CPU+RDD distributed computing mode is selected, the execution of federated feature collinearity computation includes: partitioning the initiator's local dataset X into multiple first data blocks by row, and instructing the collaborator to partition its local dataset Y into multiple second data blocks in the same way; performing data standardization for each pair of corresponding first data blocks Xi and second data blocks Yi; generating shared random number matrices Rx and Ry using secret sharing between the initiator and the collaborator; encoding each pair of standardized first data blocks Xi and second data blocks Yi; masking each pair of encoded first data blocks Xi and second data blocks Yi according to the shared random number matrices Rx and Ry to obtain each pair of corresponding first mask matrices and second mask matrices; performing a secure multiplication protocol based on secret sharing on each pair of corresponding first mask matrices and second mask matrices according to the shared random number matrices Rx and Ry to obtain the local correlation coefficient matrix of each pair of corresponding first data blocks Xi and second data blocks Yi; aggregating the local correlation coefficient matrices of all data block pairs to obtain the feature correlation coefficient matrix of the complete federated dataset.

[0011] Optionally, when the distributed and GPU single-card accelerated computing mode is selected, the execution of federated feature collinearity calculation includes: sharding, standardizing, encoding, and masking the local dataset X of the initiator and the local dataset Y of the collaborator using a CPU-distributed approach to obtain each pair of corresponding first and second mask matrices; executing a secure multiplication protocol based on secret sharing on each pair of corresponding first and second mask matrices, wherein the multiplication operation in the secure multiplication protocol is accelerated using a GPU single-card acceleration approach; converting the matrix data participating in the multiplication operation into Cupy arrays; performing matrix dot product operations on the Cupy arrays on a single GPU; converting the GPU calculation results into NumPy arrays for subsequent aggregation to obtain the feature correlation coefficient matrix of the complete joint dataset.

[0012] Optionally, when selecting the distributed and GPU multi-card parallel acceleration computing mode, the execution of federated feature collinearity calculation includes: sharding, standardizing, encoding, and masking the local dataset X of the initiator and the local dataset Y of the collaborator using a CPU-distributed approach to obtain each pair of corresponding first and second mask matrices; executing a secure multiplication protocol based on secret sharing on each pair of corresponding first and second mask matrices, wherein the multiplication operation in the secure multiplication protocol is accelerated by GPU multi-card acceleration: dividing the matrix data participating in the multiplication operation into multiple column blocks by column and allocating a GPU device to each column block; executing the secure matrix multiplication subtask required for the calculation of its corresponding column block in parallel on each allocated GPU device; and aggregating the calculation results of all GPU devices to obtain the feature correlation coefficient matrix of the complete joint dataset.

[0013] On the other hand, this invention provides a federated data processing system based on a multi-level adaptive acceleration strategy. The system includes: a mode selection unit, used by the initiator to obtain local dataset size information from collaborators and calculate joint dataset size information, automatically selecting a calculation mode matching the joint dataset size information based on the joint dataset size information, a preset first threshold, and a preset second threshold; a federated computation unit, used to, according to the selected computation mode, collaborate with the collaborators to perform federated feature collinearity calculation under privacy protection conditions to obtain a feature correlation coefficient matrix of the joint dataset; wherein the federated feature collinearity calculation involves a secure multiplication protocol based on secret sharing; a feature selection unit, used to calculate the variance inflation factor (VIF) of each feature based on the feature correlation coefficient matrix, and perform feature selection based on the VIF to obtain a feature set for joint modeling; a model training unit, used to perform multiple rounds of model training on the feature set to obtain a joint risk control model; and a prediction unit, used to input the data to be detected into the joint risk control model for prediction to obtain a risk control assessment result.

[0014] Optionally, the joint dataset size information is the product of the sample size and the number of features of the joint dataset; the automatic selection of a computing mode matching the joint dataset size information based on the joint dataset size information, a preset first threshold, and a preset second threshold includes: when the joint dataset size information is less than the preset first threshold, selecting the CPU+RDD distributed computing mode; when the joint dataset size information is greater than or equal to the preset first threshold and less than the preset second threshold, selecting the distributed and GPU single-card accelerated computing mode; when the joint dataset size information is greater than or equal to the preset second threshold, selecting the distributed and GPU multi-card parallel accelerated computing mode.

[0015] Optionally, when the CPU+RDD distributed computing mode is selected, the execution of federated feature collinearity computation includes: partitioning the initiator's local dataset X into multiple first data blocks by row, and instructing the collaborator to partition its local dataset Y into multiple second data blocks in the same way; performing data standardization for each pair of corresponding first data blocks Xi and second data blocks Yi; generating shared random number matrices Rx and Ry using secret sharing between the initiator and the collaborator; encoding each pair of standardized first data blocks Xi and second data blocks Yi; masking each pair of encoded first data blocks Xi and second data blocks Yi according to the shared random number matrices Rx and Ry to obtain each pair of corresponding first mask matrices and second mask matrices; performing a secure multiplication protocol based on secret sharing on each pair of corresponding first mask matrices and second mask matrices according to the shared random number matrices Rx and Ry to obtain the local correlation coefficient matrix of each pair of corresponding first data blocks Xi and second data blocks Yi; aggregating the local correlation coefficient matrices of all data block pairs to obtain the feature correlation coefficient matrix of the complete federated dataset.

[0016] Optionally, when the distributed and GPU single-card accelerated computing mode is selected, the execution of federated feature collinearity calculation includes: sharding, standardizing, encoding, and masking the local dataset X of the initiator and the local dataset Y of the collaborator using a CPU-distributed approach to obtain each pair of corresponding first and second mask matrices; executing a secure multiplication protocol based on secret sharing on each pair of corresponding first and second mask matrices, wherein the multiplication operation in the secure multiplication protocol is accelerated using a GPU single-card acceleration approach; converting the matrix data participating in the multiplication operation into Cupy arrays; performing matrix dot product operations on the Cupy arrays on a single GPU; converting the GPU calculation results into NumPy arrays for subsequent aggregation to obtain the feature correlation coefficient matrix of the complete joint dataset.

[0017] Optionally, when selecting the distributed and GPU multi-card parallel acceleration computing mode, the execution of federated feature collinearity calculation includes: sharding, standardizing, encoding, and masking the local dataset X of the initiator and the local dataset Y of the collaborator using a CPU-distributed approach to obtain each pair of corresponding first and second mask matrices; executing a secure multiplication protocol based on secret sharing on each pair of corresponding first and second mask matrices, wherein the multiplication operation in the secure multiplication protocol is accelerated by GPU multi-card acceleration: dividing the matrix data participating in the multiplication operation into multiple column blocks by column and allocating a GPU device to each column block; executing the secure matrix multiplication subtask required for the calculation of its corresponding column block in parallel on each allocated GPU device; and aggregating the calculation results of all GPU devices to obtain the feature correlation coefficient matrix of the complete joint dataset.

[0018] The beneficial effects of this invention are: This invention provides a federated data processing method and system based on a multi-level adaptive acceleration strategy. By introducing dataset size awareness and a three-level adaptive switching strategy (CPU distributed / CPU distributed and GPU single-card / CPU distributed and GPU multi-card), this invention can dynamically select the most economical and efficient computing mode according to the computational load of the actual task. This avoids the problems of wasted GPU resources for small tasks and insufficient CPU computing power for large tasks, enabling near-optimal processing speeds for data scales ranging from tens of thousands to hundreds of millions. The federated feature collinearity computation of this invention is entirely based on privacy protection, without disclosing the original data information of the initiator and collaborators, meeting the basic privacy requirements of federated learning. Attached Figure Description

[0019] Figure 1 This is a flowchart of a federated data processing method based on a multi-level adaptive acceleration strategy provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a federated data processing system based on a multi-level adaptive acceleration strategy provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0021] With increasingly stringent data security and privacy regulations, federated learning, a distributed machine learning paradigm that allows "the model to move while the data remains stationary," has been widely applied in fields such as finance and healthcare. However, in scenarios involving joint risk control modeling between banks and e-commerce platforms, multicollinearity exists between the bank's features and the e-commerce platform's features, leading to instability in the parameters of the logistic regression model. The parameters (i.e., weights) of the logistic regression model oscillate between bank features and e-commerce platform features. The parameters change significantly with each training iteration, leading to incorrect distribution of feature contributions. In one training iteration, the weights of bank features are very high, while the weights of e-commerce platform features are close to zero. However, in a new batch of samples, the weights of e-commerce platform features surge, rendering the business interpretation completely unreliable. Therefore, federated feature collinearity calculation is needed to obtain the feature correlation coefficient matrix of the joint dataset. Based on this matrix, the variance inflation factor (VIF) of each feature is calculated. By filtering out features with a VIF greater than a certain threshold, the remaining features are used for risk control modeling, thus preventing the aforementioned problems.

[0022] When performing federated feature collinearity computation, existing solutions typically employ either a single CPU cluster for distributed computing or a single GPU for accelerated computing. For small to medium-sized datasets (less than tens of millions), GPU acceleration may actually be inefficient due to data transfer and initialization overhead; while for massive datasets (hundreds of millions), pure CPU computation is extremely time-consuming and cannot meet the real-time requirements of business applications. Current federated computing frameworks fail to effectively combine the elastic scalability of distributed systems with the high parallel computing capabilities of GPU hardware. They either rely entirely on frameworks like Spark for CPU-based distributed computing or simply offload all computation to a single GPU, lacking intelligent strategies for dynamically partitioning tasks and allocating computing resources (multiple CPU cores, multiple GPU cards) based on data size and feature dimensions.

[0023] Therefore, there is an urgent need for a federated data processing method that can intelligently adapt to different data scales and efficiently utilize heterogeneous computing resources (CPU clusters, single GPUs, and multiple GPUs) under the privacy constraints of federated learning.

[0024] Figure 1 This is a flowchart of a federated data processing method based on a multi-level adaptive acceleration strategy provided by an embodiment of the present invention; as shown below. Figure 1 As shown, the method includes: S1. The initiator obtains the local dataset size information of the partner and calculates the joint dataset size information. Based on the joint dataset size information, the preset first threshold and the preset second threshold, the initiator automatically selects the calculation mode that matches the joint dataset size information. Specifically, the initiator obtains the dataset size information (sample size × number of features) from both the local dataset and the collaborator's dataset. The initiator and collaborator have the same sample population but different feature dimensions. A joint dataset size is calculated based on the initiator's local dataset size information and the obtained collaborator's dataset size information. This joint dataset size is the product of the sample size and the number of features in the joint dataset, i.e., sample size × number of joint features. This joint dataset size information is then compared with a preset first threshold (T_cpu) and a preset second threshold (T_gpu_single), dynamically selecting a calculation mode that matches the joint dataset size information.

[0025] In an optional implementation, the automatic selection of a calculation mode matching the joint dataset size information based on the joint dataset size information (sample size × number of joint features), a preset first threshold, and a preset second threshold includes: When the size of the joint dataset is less than a preset first threshold, the CPU+RDD distributed computing mode is selected. When the size of the joint dataset is greater than or equal to a preset first threshold and less than a preset second threshold, select the distributed and GPU single-card accelerated computing modes. When the size of the joint dataset is greater than or equal to a preset second threshold, select the distributed and GPU multi-card parallel acceleration computing mode.

[0026] In a specific embodiment, the initiating bank A and the cooperating e-commerce platform B fuse data based on IDs. Bank A has 100 features, including P0 (average credit card usage rate in the past 6 months), P1 (total loan liabilities in the past 12 months), P2 (monthly repayment amount), ... P99, each of which is a vector with 1 million dimensions, i.e., 1 million samples. The dataset of Bank A constitutes a matrix X = (1,000,000, 100). E-commerce platform B has the same batch of consumption data from 1 million users, with 100 features, including Q0 (average consumption amount in the past 6 months), Q1 (number of large-amount purchases in the past 12 months), Q2 (installment purchase amount), ... Q99, etc. The dataset of E-commerce platform B constitutes a matrix Y = (1,000,000, 100). E-commerce platform B sends its matrix size (sample size, feature dimensions) to Bank A.

[0027] Bank A automatically executes a selection strategy based on the sample size, feature dimensions, preset first threshold, and preset second threshold of the joint dataset: matrix_size (T_cpu) is The value of T_gpu_single is 3*10^8.

[0028] In this case, the sample size multiplied by the number of joint features is 1,000,000 * 200 = 2 * 10^8. if matrix_size < T_cpu: use CPU + RDD elif matrix_size < T_gpu_single: use GPU single-card else: Use GPU multi-cards In other words, if the size of the joint dataset is less than T_cpu, it represents a small-scale dataset, which can be computed using a CPU+RDD distributed computing mode. If the size of the joint dataset is greater than or equal to T_cpu and less than T_gpu_single, it represents a medium-scale dataset, which can be computed using a distributed computing mode and a single GPU acceleration mode. If the size of the joint dataset is greater than or equal to T_gpu_single, it represents a large-scale dataset, which can be computed using a distributed computing mode and a multi-GPU parallel acceleration mode. In this embodiment, T_cpu T_gpu_single is a medium-sized dataset, so the GPU single-card acceleration computing mode is used.

[0029] S2. Based on the selected computation mode, in collaboration with the partners, perform federated feature collinearity computation under privacy protection conditions to obtain the feature correlation coefficient matrix of the joint dataset; wherein, the federated feature collinearity computation involves a secure multiplication protocol based on secret sharing; S3. Calculate the variance inflation factor (VIF) of each feature based on the feature correlation coefficient matrix, and perform feature filtering based on the VIF to obtain a feature set for joint modeling. S4. Train the feature set through multiple rounds of model training to obtain a joint risk control model; In an optional implementation, when the CPU+RDD distributed computing mode is selected, the execution of federated feature collinearity computation includes: The initiator's local dataset X is sharded into multiple first data blocks by row, and the collaborator is instructed to shard its local dataset Y into multiple second data blocks in the same manner; For each pair of corresponding first data block Xi and second data block Yi, data standardization is performed; The initiator and collaborators use secret sharing to generate a shared random number matrix Rx, Ry; Encode each pair of corresponding first data blocks Xi and second data blocks Yi that are standardized; Each pair of corresponding first data blocks Xi and second data blocks Yi after encoding is masked according to the shared random number matrices Rx and Ry to obtain the first mask matrix and the second mask matrix for each pair. For each pair of corresponding first mask matrices and second mask matrices, a secure multiplication protocol based on secret sharing is performed according to the shared random number matrices Rx and Ry to obtain the local correlation coefficient matrix of each pair of corresponding first data block Xi and second data block Yi; By aggregating the local correlation coefficient matrices of all data block pairs, the feature correlation coefficient matrix of the complete joint dataset is obtained.

[0030] In one specific embodiment, bank A partitions its local dataset X into ten first data blocks Xi (X0, X1, X2, X3, X4, X5, X6, X7, X8, X9) by row, with each first data block containing 100,000 samples. The sample size of each first data block can be the same or different, and this application does not impose any restrictions on this. E-commerce platform B partitions its local dataset Y into ten second data blocks Yi (Y0, Y1, Y2, Y3, Y4, Y5, Y6, Y7, Y8, Y9) in the same manner, with each second data block containing 100,000 samples. There are a total of 10 pairs of corresponding first data blocks Xi and second data blocks Yi. For each pair of corresponding first data block Xi and second data block Yi, perform data standardization; The following explanation uses the first data block X0 and the second data block Y0 of the first pair as an example: The first data block X0 and the second data block Y0 are Z-score standardized so that the mean of each feature is 0 and the standard deviation is 1.

[0031] Bank A and e-commerce platform B pre-agreed upon a pseudo-random number generator (PRG) and a shared seed. Each party independently generates a shared random matrix locally: Bank A generates Rx and Ry, and e-commerce platform B also generates the same Rx and Ry. Furthermore, the size of the Rx matrix is ​​the same as X0, and the size of the Ry matrix is ​​the same as Y0.

[0032] The standardized first data block X0 and second data block Y0 are floating-point number matrices. To prevent floating-point precision issues and adapt to secure computation, each element in the standardized first data block X0 and second data block Y0 is encoded as a fixed-point number. For further use in the secure multiplication protocol, both parties split Rx and Ry into odd-row matrices Rx_odd and Ry_odd, and even-row matrices Rx_even and Ry_even.

[0033] Bank A and e-commerce platform B calculate the local correlation coefficient matrix corr0, partitioned into z1, z2, and corr = X0*Y0 = z1+z2, respectively, based on distributed RDDs. The specific calculation process is as follows: (a) Party A calculates the first mask matrix arr_a1=X0+Rx, arr_e1=Rx_odd+Rx_even, arr_f1=Ry_even-Ry_odd, and sends the first mask matrix arr_a1 to Party B. (b) Party B calculates the second mask matrix arr_b1=Ry-Y0, arr_e1=Rx_odd+Rx_even, arr_f1=Ry_even-Ry_odd, and sends the second mask matrix arr_b1 to Party A; (c) A calculates arr_ae = arr_a1 + Rx, arr_ee = Rx_odd + arr_e1; (d) A calculates the local correlation coefficient matrix corr0 piecewise z1 = arr_ae arr_ee* arr_f1; (e) B calculates arr_bb = Y0 - arr_b1, arr_ff = Ry_even + arr_f1; (f) B calculates the local correlation coefficient matrix arr_bb arr_ff; (g) Party B sends z2 to Party A, and Party A calculates the local correlation coefficient matrix corr0 = z1 + z2 for the first data block X0 and the second data block Y0. In the above calculation process, Party A is unaware of the data in Party B's second data block Y0, and Party B is unaware of the data in Party A's first data block X0, thus ensuring that the data is available but not visible.

[0034] Using the same method described above, A calculates the local correlation coefficient matrices corr0, corr1, corr2, corr3, corr4, corr5, corr6, corr7, corr8, and corr9 for each pair of corresponding first data blocks Xi and second data blocks Yi. A aggregates the local correlation coefficient matrices corresponding to all data block pairs to obtain the feature correlation coefficient matrix corr of the complete joint dataset.

[0035] A calculates the variance inflation factor (VIF) of each feature using corr. The variance inflation factor (VIF) of the first feature is calculated as |det(corr_i)| / |det(corr)|, where corr_i is the submatrix after removing the i-th row and i-th column; det(corr) is the determinant of the feature correlation coefficient matrix of the joint dataset, and det(corr_i) is the determinant of the submatrix after removing the i-th row and i-th column.

[0036] The variance inflation factor (VIF) of each feature in the joint dataset of parties A and B was obtained. Features with VIF greater than the preset third threshold were filtered out. The remaining features can be used for joint risk control modeling. That is, the remaining features are used for multiple rounds of model training to obtain the joint risk control model. Since highly collinear features are removed, the model parameters no longer oscillate violently during training. The weight allocation of bank features and e-commerce features will be more reasonable, and the interpretability and generalization ability of the model will be significantly enhanced.

[0037] This step allows CPU computation to meet performance requirements when data is small, avoiding the overhead of GPU data copying.

[0038] In an optional implementation, when the distributed and GPU-accelerated computing modes are selected, the execution of federated feature collinearity computation includes: The local dataset X of the initiator and the local dataset Y of the collaborator are sharded, standardized, encoded, and masked in a CPU-distributed manner to obtain a first mask matrix and a second mask matrix for each pair. For each pair of corresponding first and second mask matrices, a secure multiplication protocol based on secret sharing is executed, wherein the multiplication operation in the secure multiplication protocol is accelerated by a single GPU card: Convert the matrix data involved in the multiplication operation into a Cupy array; Perform matrix dot product operations on Cupy arrays on a single GPU; The GPU computation results are converted into NumPy arrays for subsequent aggregation to obtain the feature correlation coefficient matrix of the complete joint dataset.

[0039] In a specific embodiment, if the joint dataset size information is medium-sized as in the case, then distributed + GPU single card acceleration is used, mainly using GPU acceleration for the most time-consuming calculations (d) and (f) mentioned above; the rest are the same as the calculation methods in the small-scale computing mode.

[0040] The following explanation only covers (d) and (f) using GPU acceleration: (d)z1= arr_ae arr_ee arr_ae, arr_b1, arr_ee, and arr_f1 are all NumPy arrays containing fixed-point numbers. The fixed-point numbers are then converted to integers. 6))) convert the NumPy arrays into integer arrays, then convert these integer NumPy arrays into Cupy arrays. Set the computing device to CUDA to accelerate the computation (i.e., matrix multiplication is performed on a single GPU). After steps d) and (f) are completed, convert the Cupy arrays back into NumPy arrays and continue the subsequent computation until the feature correlation coefficient matrix corr of the complete joint dataset is obtained. Based on the feature correlation coefficient matrix, calculate the variance inflation factor (VIF) of each feature, and perform feature selection based on the VIF to obtain a feature set for joint modeling. Train the model using the feature set to obtain the joint risk control model.

[0041] In an optional implementation, when the distributed and GPU multi-card parallel accelerated computing mode is selected, the execution of federated feature collinearity calculation includes: The local dataset X of the initiator and the local dataset Y of the collaborator are sharded, standardized, encoded, and masked in a CPU-distributed manner to obtain a first mask matrix and a second mask matrix for each pair. For each pair of corresponding first and second mask matrices, a secure multiplication protocol based on secret sharing is executed, wherein the multiplication operation in the secure multiplication protocol is accelerated by GPU multi-card acceleration: The matrix data involved in the multiplication operation is divided into multiple column blocks according to columns, and a GPU device is allocated to each column block; On each allocated GPU device, the secure matrix multiplication subtasks required for the computation of its corresponding column blocks are executed in parallel. The computational results from all GPU devices are aggregated and merged to obtain the feature correlation coefficient matrix of the complete joint dataset.

[0042] In a specific embodiment, if the joint dataset size information is a large-scale dataset, then distributed + GPU multi-card acceleration is used, mainly using GPU multi-card acceleration for the most time-consuming computations of (d) and (f) mentioned above; the rest are the same as the computation method in the small-scale computing mode.

[0043] The following explanation only covers (d) and (f) using GPU acceleration: (d)z1= arr_ae arr_ee arr_ae is divided into multiple column blocks by column, and arr_b1, arr_ee, and arr_f1 are divided into multiple column blocks by column in the same way. For example, if they are all divided into 4 blocks, then a GPU device is allocated to each column block. On the first GPU device, the first column blocks of arr_ae, arr_b1, arr_ee, and arr_f1 undergo matrix multiplication in step (d). On the second GPU device, the second column blocks of arr_ae, arr_b1, arr_ee, and arr_f1 undergo matrix multiplication in step (d). On the third GPU device, the third column blocks of arr_ae, arr_b1, arr_ee, and arr_f1 undergo matrix multiplication in step (d). On the fourth GPU device, the fourth column blocks of arr_ae, arr_b1, arr_ee, and arr_f1 undergo matrix multiplication in step (d). It is important to note that the four GPU devices execute the computation tasks in parallel, and the computation process performed by each GPU device is the same as that performed by a single GPU device in the medium-sized dataset described above. After the four GPU devices obtain the calculation results, the calculation results of all GPU devices are aggregated. After steps d) and f are completed, the subsequent calculations are continued until the feature correlation coefficient matrix corr of the complete joint dataset is obtained. Based on the feature correlation coefficient matrix, the variance inflation factor (VIF) of each feature is calculated, and feature selection is performed based on the variance inflation factor (VIF) to obtain a feature set for joint modeling. The feature set is then used for model training to obtain the joint risk control model.

[0044] This approach divides the massive correlation calculation task into columns and schedules it to be computed simultaneously on four GPU cards on the server. Each GPU card processes the correlation calculation for a portion of the feature columns, and the results are finally merged through high-speed interconnect. The computation speed in this mode can be hundreds of times faster than the pure CPU mode, making it possible to process such ultra-large-scale data.

[0045] S5. Input the data to be detected into the joint risk control model for prediction to obtain the risk control assessment result.

[0046] For example, inputting 100 feature data of the latest users of Bank A and e-commerce B into the joint risk control model for prediction yields risk control assessment results.

[0047] In an optional implementation, the method of the present invention can automatically adjust the number of training rounds, the number of matrix row shards, the number of matrix column shards, and whether to enable single-card or multi-card parallel acceleration computation based on the joint dataset size information. The performance can be automatically and dynamically optimized under different hardware environments without the need for manual parameter tuning. If a GPU card is detected to be faulty or overloaded, the multi-card scheduling unit can automatically reassign the task to other available GPUs, or in extreme cases, downgrade to single-card or pure CPU mode to ensure the final completion of the computing task.

[0048] In summary, this invention creatively solves the performance bottleneck problem in large-scale federated feature engineering by deeply integrating a multi-level adaptive acceleration strategy with a privacy-preserving computation protocol in federated learning. The method of this invention allocates appropriate computational resources based on data volume partitioning, thereby greatly unleashing the potential of joint modeling within the "red line" of data privacy.

[0049] Figure 2 This invention provides a federated data processing system based on a multi-level adaptive acceleration strategy, such as... Figure 2 As shown, the system includes: The mode selection unit 201 is used for the initiator to obtain the local dataset size information of the partner and calculate the joint dataset size information, and automatically select the calculation mode that matches the joint dataset size information based on the joint dataset size information, the preset first threshold and the preset second threshold. The federated computing unit 202 is configured to, in collaboration with the partners, perform federated feature collinearity computation under privacy protection conditions, according to the selected computing mode, to obtain the feature correlation coefficient matrix of the joint dataset; wherein, the federated feature collinearity computation involves a secure multiplication protocol based on secret sharing; The feature filtering unit 203 is used to calculate the variance inflation factor (VIF) of each feature based on the feature correlation coefficient matrix, and to perform feature filtering based on the variance inflation factor (VIF) to obtain a feature set for joint modeling. Model training unit 204 is used to perform multiple rounds of model training on the feature set to obtain a joint risk control model; The prediction unit 205 is used to input the data to be detected into the joint risk control model for prediction, and obtain the risk control assessment result.

[0050] In an optional implementation, the joint dataset size information is the product of the sample size and the number of features of the joint dataset; The automatic selection of a calculation mode that matches the joint dataset size information based on the joint dataset size information, a preset first threshold, and a preset second threshold includes: When the size of the joint dataset is less than a preset first threshold, the CPU+RDD distributed computing mode is selected. When the size of the joint dataset is greater than or equal to a preset first threshold and less than a preset second threshold, select the distributed and GPU single-card accelerated computing modes. When the size of the joint dataset is greater than or equal to a preset second threshold, select the distributed and GPU multi-card parallel acceleration computing mode.

[0051] In an optional implementation, when the CPU+RDD distributed computing mode is selected, the execution of federated feature collinearity computation includes: The initiator's local dataset X is sharded into multiple first data blocks by row, and the collaborator is instructed to shard its local dataset Y into multiple second data blocks in the same manner; For each pair of corresponding first data block Xi and second data block Yi, data standardization is performed; The initiator and collaborators use secret sharing to generate a shared random number matrix Rx, Ry; Encode each pair of corresponding first data blocks Xi and second data blocks Yi that are standardized; Each pair of corresponding first data blocks Xi and second data blocks Yi after encoding is masked according to the shared random number matrices Rx and Ry to obtain the first mask matrix and the second mask matrix for each pair. For each pair of corresponding first mask matrices and second mask matrices, a secure multiplication protocol based on secret sharing is performed according to the shared random number matrices Rx and Ry to obtain the local correlation coefficient matrix of each pair of corresponding first data block Xi and second data block Yi; By aggregating the local correlation coefficient matrices of all data block pairs, the feature correlation coefficient matrix of the complete joint dataset is obtained.

[0052] In an optional implementation, when the distributed and GPU-accelerated computing modes are selected, the execution of federated feature collinearity computation includes: The local dataset X of the initiator and the local dataset Y of the collaborator are sharded, standardized, encoded, and masked in a CPU-distributed manner to obtain a first mask matrix and a second mask matrix for each pair. For each pair of corresponding first and second mask matrices, a secure multiplication protocol based on secret sharing is executed, wherein the multiplication operation in the secure multiplication protocol is accelerated by a single GPU card: Convert the matrix data involved in the multiplication operation into a Cupy array; Perform matrix dot product operations on Cupy arrays on a single GPU; The GPU computation results are converted into NumPy arrays for subsequent aggregation to obtain the feature correlation coefficient matrix of the complete joint dataset.

[0053] In an optional implementation, when the distributed and GPU multi-card parallel accelerated computing mode is selected, the execution of federated feature collinearity calculation includes: The local dataset X of the initiator and the local dataset Y of the collaborator are sharded, standardized, encoded, and masked in a CPU-distributed manner to obtain a first mask matrix and a second mask matrix for each pair. For each pair of corresponding first and second mask matrices, a secure multiplication protocol based on secret sharing is executed, wherein the multiplication operation in the secure multiplication protocol is accelerated by GPU multi-card acceleration: The matrix data involved in the multiplication operation is divided into multiple column blocks according to columns, and a GPU device is allocated to each column block; On each allocated GPU device, the secure matrix multiplication subtasks required for the computation of its corresponding column blocks are executed in parallel. The computational results from all GPU devices are aggregated and merged to obtain the feature correlation coefficient matrix of the complete joint dataset.

[0054] The system of the present invention corresponds to the method described above, and the specific implementation of the system will not be repeated here.

[0055] The beneficial effects of this invention are: This invention provides a federated data processing method and system based on a multi-level adaptive acceleration strategy. By introducing dataset size awareness and a three-level adaptive switching strategy (CPU distributed / CPU distributed and GPU single-card / CPU distributed and GPU multi-card), this invention can dynamically select the most economical and efficient computing mode according to the computational load of the actual task. This avoids the problems of wasted GPU resources for small tasks and insufficient CPU computing power for large tasks, enabling near-optimal processing speeds for data scales ranging from tens of thousands to hundreds of millions. The federated feature collinearity computation of this invention is entirely based on privacy protection, without disclosing the original data information of the initiator and collaborators, meeting the basic privacy requirements of federated learning. For medium and large-scale data, this invention innovatively accelerates the most time-consuming core matrix multiplication operation in the federated secure computing protocol through GPU hardware. Especially in ultra-large-scale scenarios, a strategy of data slicing and multi-GPU thread pool parallelism is proposed, achieving fine-grained parallelism of computational tasks, achieving a performance improvement of tens of times compared to pure CPU distributed computing.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A federated data processing method based on a multi-level adaptive acceleration strategy, characterized in that, include: S1. The initiator obtains the local dataset size information of the partner and calculates the joint dataset size information. Based on the joint dataset size information, the preset first threshold and the preset second threshold, the initiator automatically selects the calculation mode that matches the joint dataset size information. S2. Based on the selected computation mode, in collaboration with the partners, perform federated feature collinearity computation under privacy protection conditions to obtain the feature correlation coefficient matrix of the joint dataset; wherein, the federated feature collinearity computation involves a secure multiplication protocol based on secret sharing; S3. Calculate the variance inflation factor (VIF) of each feature based on the feature correlation coefficient matrix, and perform feature filtering based on the VIF to obtain a feature set for joint modeling. S4. Train the feature set through multiple rounds of model training to obtain a joint risk control model; S5. Input the data to be detected into the joint risk control model for prediction to obtain the risk control assessment result.

2. The method according to claim 1, characterized in that: The size information of the joint dataset is the product of the sample size and the number of features of the joint dataset; The automatic selection of a calculation mode that matches the joint dataset size information based on the joint dataset size information, a preset first threshold, and a preset second threshold includes: When the size of the joint dataset is less than a preset first threshold, the CPU+RDD distributed computing mode is selected. When the size of the joint dataset is greater than or equal to a preset first threshold and less than a preset second threshold, select the distributed and GPU single-card accelerated computing modes. When the size of the joint dataset is greater than or equal to a preset second threshold, select the distributed and GPU multi-card parallel acceleration computing mode.

3. The method according to claim 2, characterized in that, When the CPU+RDD distributed computing mode is selected, the execution of federated feature collinearity computation includes: The initiator's local dataset X is sharded into multiple first data blocks by row, and the collaborator is instructed to shard its local dataset Y into multiple second data blocks in the same manner; For each pair of corresponding first data block Xi and second data block Yi, data standardization is performed; The initiator and collaborators use secret sharing to generate a shared random number matrix Rx, Ry; Encode each pair of corresponding first data blocks Xi and second data blocks Yi that are standardized; Each pair of corresponding first data blocks Xi and second data blocks Yi after encoding is masked according to the shared random number matrices Rx and Ry to obtain the first mask matrix and the second mask matrix for each pair. For each pair of corresponding first mask matrices and second mask matrices, a secure multiplication protocol based on secret sharing is performed according to the shared random number matrices Rx and Ry to obtain the local correlation coefficient matrix of each pair of corresponding first data block Xi and second data block Yi; By aggregating the local correlation coefficient matrices of all data block pairs, the feature correlation coefficient matrix of the complete joint dataset is obtained.

4. The method according to claim 2, characterized in that, When the distributed and GPU single-card accelerated computing modes are selected, the execution of federated feature collinearity calculation includes: The local dataset X of the initiator and the local dataset Y of the collaborator are sharded, standardized, encoded, and masked in a CPU-distributed manner to obtain a first mask matrix and a second mask matrix for each pair. For each pair of corresponding first and second mask matrices, a secure multiplication protocol based on secret sharing is executed, wherein the multiplication operation in the secure multiplication protocol is accelerated by a single GPU card: Convert the matrix data involved in the multiplication operation into a Cupy array; Perform matrix dot product operations on Cupy arrays on a single GPU; The GPU computation results are converted into NumPy arrays for subsequent aggregation to obtain the feature correlation coefficient matrix of the complete joint dataset.

5. The method according to claim 2, characterized in that, When the distributed and GPU multi-card parallel acceleration computing mode is selected, the execution of federated feature collinearity calculation includes: The local dataset X of the initiator and the local dataset Y of the collaborator are sharded, standardized, encoded, and masked in a CPU-distributed manner to obtain a first mask matrix and a second mask matrix for each pair. For each pair of corresponding first and second mask matrices, a secure multiplication protocol based on secret sharing is executed, wherein the multiplication operation in the secure multiplication protocol is accelerated by GPU multi-card acceleration: The matrix data involved in the multiplication operation is divided into multiple column blocks according to columns, and a GPU device is allocated to each column block; On each allocated GPU device, the secure matrix multiplication subtasks required for the computation of its corresponding column blocks are executed in parallel. The computational results from all GPU devices are aggregated and merged to obtain the feature correlation coefficient matrix of the complete joint dataset.

6. A federated data processing system based on a multi-level adaptive acceleration strategy, characterized in that, include: The mode selection unit is used by the initiator to obtain the local dataset size information of the partner and calculate the joint dataset size information. Based on the joint dataset size information, the preset first threshold and the preset second threshold, it automatically selects the calculation mode that matches the joint dataset size information. The federated computing unit is used to collaborate with the partners to perform federated feature collinearity computation under privacy protection conditions, according to the selected computing mode, to obtain the feature correlation coefficient matrix of the joint dataset; wherein, the federated feature collinearity computation involves a secure multiplication protocol based on secret sharing; The feature filtering unit is used to calculate the variance inflation factor (VIF) of each feature based on the feature correlation coefficient matrix, and to perform feature filtering based on the variance inflation factor (VIF) to obtain a feature set for joint modeling. The model training unit is used to train the feature set through multiple rounds of model training to obtain a joint risk control model. The prediction unit is used to input the data to be detected into the joint risk control model for prediction, and obtain the risk control assessment result.

7. The system according to claim 6, characterized in that: The size information of the joint dataset is the product of the sample size and the number of features of the joint dataset; The automatic selection of a calculation mode that matches the joint dataset size information based on the joint dataset size information, a preset first threshold, and a preset second threshold includes: When the size of the joint dataset is less than a preset first threshold, the CPU+RDD distributed computing mode is selected. When the size of the joint dataset is greater than or equal to a preset first threshold and less than a preset second threshold, select the distributed and GPU single-card accelerated computing modes. When the size of the joint dataset is greater than or equal to a preset second threshold, select the distributed and GPU multi-card parallel acceleration computing mode.

8. The system according to claim 7, characterized in that, When the CPU+RDD distributed computing mode is selected, the execution of federated feature collinearity computation includes: The initiator's local dataset X is sharded into multiple first data blocks by row, and the collaborator is instructed to shard its local dataset Y into multiple second data blocks in the same manner; For each pair of corresponding first data block Xi and second data block Yi, data standardization is performed; The initiator and collaborators use secret sharing to generate a shared random number matrix Rx, Ry; Encode each pair of corresponding first data blocks Xi and second data blocks Yi that are standardized; Each pair of corresponding first data blocks Xi and second data blocks Yi after encoding is masked according to the shared random number matrices Rx and Ry to obtain the first mask matrix and the second mask matrix for each pair. For each pair of corresponding first mask matrices and second mask matrices, a secure multiplication protocol based on secret sharing is performed according to the shared random number matrices Rx and Ry to obtain the local correlation coefficient matrix of each pair of corresponding first data block Xi and second data block Yi; By aggregating the local correlation coefficient matrices of all data block pairs, the feature correlation coefficient matrix of the complete joint dataset is obtained.

9. The system according to claim 7, characterized in that, When the distributed and GPU single-card accelerated computing modes are selected, the execution of federated feature collinearity calculation includes: The local dataset X of the initiator and the local dataset Y of the collaborator are sharded, standardized, encoded, and masked in a CPU-distributed manner to obtain a first mask matrix and a second mask matrix for each pair. For each pair of corresponding first and second mask matrices, a secure multiplication protocol based on secret sharing is executed, wherein the multiplication operation in the secure multiplication protocol is accelerated by a single GPU card: Convert the matrix data involved in the multiplication operation into a Cupy array; Perform matrix dot product operations on Cupy arrays on a single GPU; The GPU computation results are converted into NumPy arrays for subsequent aggregation to obtain the feature correlation coefficient matrix of the complete joint dataset.

10. The system according to claim 7, characterized in that, When the distributed and GPU multi-card parallel acceleration computing mode is selected, the execution of federated feature collinearity calculation includes: The local dataset X of the initiator and the local dataset Y of the collaborator are sharded, standardized, encoded, and masked in a CPU-distributed manner to obtain a first mask matrix and a second mask matrix for each pair. For each pair of corresponding first and second mask matrices, a secure multiplication protocol based on secret sharing is executed, wherein the multiplication operation in the secure multiplication protocol is accelerated by GPU multi-card acceleration: The matrix data involved in the multiplication operation is divided into multiple column blocks according to columns, and a GPU device is allocated to each column block; On each allocated GPU device, the secure matrix multiplication subtasks required for the computation of its corresponding column blocks are executed in parallel. The computational results from all GPU devices are aggregated and merged to obtain the feature correlation coefficient matrix of the complete joint dataset.

Images