A multi-algorithm fusion big data analysis method and system based on integration collaboration

Abstract

The application discloses a kind of based on integration cooperation's multi-algorithm fusion big data analysis method, specifically includes: multi-algorithm fusion engine layer receives the feature stream, strategy stream and resource quota from integration cooperative control layer;Call multiple basic models in algorithm warehouse, the basic model includes machine learning model, deep learning model, graph model or federal learning model;Sparse gating network is generated through sparse gating weight;The activated basic model output is weighted and fused based on the sparse gating weight, and multi-model fusion output is obtained;In the training process, differential privacy protection is executed through security aggregation gateway;Performance drift index is continuously monitored in collaborative feedback channel, if it is detected that performance decreases more than preset threshold and continues to exceed tolerance round, then automatically roll back to best model state or trigger incremental training;Training result and performance index are pushed to A / B index board in real time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, and mainly relates to a multi-algorithm fusion big data analysis method and system based on integrated collaboration. BACKGROUND

[0002] In the era of big data, the scale of data grows exponentially, and single algorithm cannot meet the analysis needs of high-dimensional heterogeneous and real-time evolution. Financial risk control, smart medical care, industrial internet and other fields have explored the path of multi-algorithm fusion: Ant's "Ant Shield" platform integrates tree model + deep learning to realize millisecond-level anti-fraud; Tencent "Lingkan" uses graph neural network and knowledge graph to identify black production links; Siemens MindSphere uses CNN + LSTM fusion to predict equipment failure. These cross-domain practice cases prove that the application of multi-algorithm fusion technology in various industries has gradually landed, and has shown great potential in actual operation. However, the current integrated collaborative multi-algorithm technology has the limitation of single algorithm, which brings great challenges in big data analysis. SUMMARY

[0003] In the face of the four bottlenecks of the existing architecture, that is, the unified access gateway only supports single modal model, the data governance and storage layer lacks dynamic fusion strategy, the integrated collaborative control layer cannot be flexibly scheduled, and the multi-algorithm fusion engine lacks privacy protection and the intelligent application collaborative layer has high expansion cost, the present application introduces Sidecar + OPA + xDS to realize zero-trust policy hot plug in the unified access gateway, establishes field-level bloodline governance driven by Schema Registry in the data governance and storage layer, deploys resource-privacy-sparse-depth four-dimensional linkage scheduling mechanism in the integrated collaborative control layer, adopts sparse gating meta-graph network in the multi-algorithm fusion engine to realize dynamic fusion, and outputs real-time early warning, decision support, personalized recommendation and joint risk control services to the intelligent application collaborative layer in a modular plug-and-play manner, so as to open up the end-to-end closed loop from access, governance, collaboration and fusion to application, realize minute-level expansion and zero-downtime update. According to the first aspect of the present application, a multi-algorithm fusion big data analysis method based on integrated collaboration is provided, and the specific steps are as follows:

[0004] S1, the multi-algorithm fusion engine layer receives the feature flow, the policy flow and the resource quota from the integrated collaborative control layer;

[0005] S2, calling a plurality of basic models in the algorithm warehouse, the basic models include machine learning models, deep learning models, graph models or federated learning models;

[0006] S3. Generate sparse gating weights through a sparse gating network, wherein: the sparse gating network calculates the activation scores of each base model based on the input features; performs Top-K filtering on the activation scores, retaining only the K base models with the highest scores; performs Softmax normalization on the scores of the retained models to obtain the sparse gating weights;

[0007] S4. Based on the sparse gating weights, the outputs of the activated basic models are weighted and fused to obtain the multi-model fusion output;

[0008] S5. During training, differential privacy protection is performed through a secure aggregation gateway, including: pruning the gradients of each model; adding calibrated Gaussian noise to the pruned gradients; aggregating the noisy gradients and calculating the average gradient for model parameter updates.

[0009] S6. The collaborative feedback channel continuously monitors performance drift metrics. If a performance drop exceeds a preset threshold and continues to exceed the tolerance number of rounds, it will automatically roll back to the best model state or trigger incremental training.

[0010] S7. Training results and performance metrics are pushed to the A / B dashboard in real time.

[0011] Preferably, the sparse gating weights are calculated using the following formula:

[0012] ;

[0013] ;

[0014] Where x represents the input feature vector. and These represent the weights and biases of the hidden layer in the gated network, respectively. Indicates the weights of the output layer of the gated network. Let represent the bias of the output layer of the gated network, TopK(z,K) represent the K largest values ​​in the return vector z, g represent the sparse gate weight vector with only K non-zero values, h represent the hidden layer output of the gated network, and z represent the original output layer values ​​of the gated network. This represents the k-th element in vector z. This represents the k-th gate value after sparsification, where k represents the position index.

[0015] Preferably, the calculation formula for the multi-model fusion output is as follows:

[0016] ;

[0017] in, Represents the k-th base model. denotes the sparse gating weight corresponding to the k-th model, k denotes the position index, denotes the multi-model fusion output, x denotes the input feature vector.

[0018] The calculation formula of the differential privacy protection performed by the priority, secure aggregation gateway is as follows:

[0019] ;

[0020] wherein B denotes a batch of gradient sample sets, denotes the gradient of the i-th sample, denotes a gradient clipping function, denotes a Gaussian noise with a mean of 0 and a standard deviation of C, denotes the average gradient after noise addition and aggregation.

[0021] Preferably, the multi-algorithm fusion engine layer further comprises a unified access gateway layer before the multi-algorithm fusion engine layer, which is configured to, before writing any data, acquire or register the latest data format from a metadata registration center in a data governance and storage layer through a remote API call.

[0022] Wherein, the streaming acquisition module and the batch acquisition module encapsulate events according to the data format; and a Sidecar deployed at the outlet performs data desensitization, zero-trust encryption and exception interception processing, and then accesses the metadata registration center again to confirm version consistency, so as to output clean and traceable standardized event streams to the data governance and storage layer.

[0023] Preferably, the unified access gateway layer is configured with an event back-checking mechanism, which is configured to verify the metadata version consistency after data encapsulation and encryption processing, to ensure that the data format is consistent with the version registered in the Schema Registry.

[0024] Preferably, the multi-algorithm fusion engine layer further comprises a data governance and storage layer before the multi-algorithm fusion engine layer, which is configured to govern, store and manage features of the standardized event streams from the unified access gateway layer, wherein each component follows a dependency order of first governance, then service, and finally consumption, to form a one-way and acyclic data dependency chain.

[0025] Preferably, the data governance and storage layer comprises:

[0026] A data lake module configured to store standardized events in business time partitions and perform incremental compression;

[0027] A metadata center configured to generate field-level provenance information, life cycle labels and model cards at each write;

[0028] A vector index service and feature warehouse configured to keep snapshots consistent with the metadata center.

[0029] Preferably, an integrated collaborative control layer is included before the multi-algorithm fusion engine layer, which is used to realize a unified scheduling and feedback loop for policies, computing power and privacy between the data governance and storage layer and the multi-algorithm fusion engine layer.

[0030] Preferably, the integrated collaborative control layer includes a data collaboration center, an algorithm collaboration center, a computing power collaboration center, and a security collaboration center, wherein,

[0031] The data collaboration center is used to read metadata and data snapshots from the Schema Registry and Iceberg in the data governance and storage layer in real time, and calculate data quality scores based on field-level lineage provided by the metadata center;

[0032] The algorithm collaboration center calls the automated machine learning module to perform hyperparameter search based on the data quality score, and generates candidate algorithms.

[0033] The computing power collaboration center allocates resource quotas and GPU multi-instance slices based on a comprehensive four-dimensional factor: task urgency, model complexity, privacy budget, and available computing power.

[0034] The security collaboration center broadcasts policy updates to each node of the system through the control channel and pushes the feature stream and policy stream to the multi-algorithm fusion engine layer through the data channel.

[0035] Preferably, the multi-algorithm fusion engine layer is followed by an intelligent application collaboration layer, which is used to call the real-time services of the multi-algorithm fusion engine through REST or gRPC interfaces to achieve real-time early warning, decision support, personalized recommendation and joint risk control.

[0036] Preferably, the intelligent application collaboration layer is configured with a business feedback mechanism, which is used to send the real effect data generated by the business end embedding points back to the unified access gateway layer, and form a minute-level closed-loop optimization through each layer, and support one-click generation of compliance reports that conform to the PDF / A standard, so as to realize end-to-end continuous iteration.

[0037] According to a second aspect of the present invention, a computer program product is provided, on which one or more computer programs are stored, which, when executed by a computer processor, implement the method described above.

[0038] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects:

[0039] The method is applied in practice. Practical application shows that the multi-algorithm fusion big data analysis system based on integrated collaboration realized by the method breaks the limitation of traditional single algorithm and fixed fusion strategy, realizes cross-domain data security sharing and algorithm dynamic collaboration, significantly improves analysis accuracy, shortens system expansion period from weeks to minutes, synchronously reduces operation and compliance costs, and has wide industrialization promotion value. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings are included to provide a further understanding of embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the application. Other embodiments and many of the intended advantages of the present application will be readily appreciated as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings. The elements of the drawings are not necessarily to scale relative to each other. Like reference numerals designate corresponding similar parts.

[0041] Figure 1 A flowchart of a multi-algorithm fusion big data analysis method based on integrated collaboration is shown according to an embodiment of the present application.

[0042] Figure 2 is a structural schematic diagram of a computer system of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0043] The present application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for ease of description.

[0044] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0045] Figure 1 A flowchart of a multi-algorithm fusion big data analysis method based on integrated collaboration is shown according to an embodiment of the present application, as shown in Figure 1

[0046] Unified access gateway layer:

[0047] ​The gateway layer acquires or registers the latest data format from the metadata registry (Schema Registry) located in the data governance and storage layer through a remote API call before writing any data; the stream collection (Kafka / Flink) and batch collection (Spark) encapsulate events according to this format; the Sidecar performs de-identification, zero-trust encryption, and exception interception at the export, and after processing, it again checks the metadata registry (Schema Registry) to confirm version consistency, providing "clean and traceable" data for the subsequent governance and storage layer. The standard event stream is then output to the data governance and storage layer.

[0048] Data governance and storage layer:

[0049] Within the data governance and storage layer, components follow the dependency order of "governance first, service second, and consumption last", forming a one-way, acyclic dependency chain. The data lake (Iceberg) receives standardized events from the unified access gateway layer, partitions and writes to disk according to business time, and triggers incremental compression; the metadata center synchronously generates field-level provenance, lifecycle labels, and model cards (Model Card) during each write, and maintains snapshot consistency with the vector index service (Milvus) and feature storage (Feast), thereby providing "highly trusted and highly available" data views and feature assets for the integrated collaborative control layer.

[0050] The metadata center triggers the model card (Model Card) automatic update mechanism when data is written, recording dataset version, field provenance, feature changes, and usage range information to support subsequent model tracing and compliance auditing.

[0051] The data lake module dynamically partitions according to business time when receiving new event writes, and triggers an incremental compression strategy based on file size and time window.

[0052] Integrated collaborative control layer:

[0053] The data coordination center reads the metadata and data snapshots of Schema Registry and Iceberg in the data governance and storage layer in real time, and calculates the data quality score (DQ Score) in real time according to the field-level blood relationship provided by the metadata center; the algorithm coordination center calls the AutoML hyperparameter search to generate candidate algorithms according to the score; the computing power coordination center allocates resource quotas and GPU MIG slices under the comprehensive four-dimensional factor; the security coordination center broadcasts policy updates to all nodes through the control channel (Raft), and pushes the feature and policy stream to the downstream multi-algorithm fusion engine through the data channel (zero-copy message queue), and the feedback result (drift index, rollback instruction) of the engine is written back to the data coordination center through the data channel, realizing the unified scheduling of strategy, computing power and privacy, and forming a control-execution-feedback closed loop.

[0054] The multi-algorithm fusion engine layer:

[0055] S1, the multi-algorithm fusion engine layer receives the feature stream, the policy stream and the resource quota from the integrated collaborative control layer;

[0056] S2, call multiple basic models in the algorithm warehouse, the basic model includes a machine learning model, a deep learning model, a graph model or a federated learning model;

[0057] S3, generate sparse gating weights through a sparse gating network, wherein: the sparse gating network calculates the activation score of each basic model according to the input feature; Top-K screening is performed on the activation score, and only the K basic models with the highest score are reserved; the Softmax normalization is performed on the score of the reserved model to obtain the sparse gating weight; the sparse gating network includes a two-layer full connection structure, the first layer adopts GeLU activation function, and the second layer outputs the model activation score.

[0058] The calculation formula of the sparse gating weight is as follows:

[0059] ;

[0060] ;

[0061] Wherein, x represents an input feature vector, and respectively represent the weight and bias of the hidden layer of the gating network, represent the weight of the output layer of the gating network, represent the bias of the output layer of the gating network, TopK(z,K) represents the maximum K values in the vector z, g represents the sparse gating weight vector, only K non-zero values, h represents the hidden layer output of the gating network, z represents the original output layer value of the gating network, represents the kth element in the vector z, denotes the k-th gating value after the sparsification process, and k denotes a position index.

[0062] In the calculation of the sparse gating weight, a Top-K sparse gating mechanism is embedded in multi-model fusion, and a Softmax normalization + negative infinity shielding structure is adopted to form a sparse and derivable gating process.

[0063] In model integration, a trainable sparse selection layer is introduced to realize "learnable model activation", which is different from traditional static model weighting.

[0064] S4, weighting and fusing the activated base model output based on the sparse gating weight to obtain a multi-model fusion output, and a calculation formula of the multi-model fusion output is as follows:

[0065]

[0066] wherein, denotes the k-th base model, denotes the sparse gating weight corresponding to the k-th model, k denotes a position index, denotes the multi-model fusion output, and x denotes an input feature vector. The "sparse gating output" is directly input as a dynamic weight of multi-model fusion to form an input-driven dynamic model integration structure; the structure essentially constitutes a "model-level attention layer", which is different from the feature-level Attention mechanism and belongs to a high-level structural innovation.

[0067] S5, in the training process, performing differential privacy protection through a secure aggregation gateway, including: clipping each model gradient; adding a calibrated Gaussian noise to the clipped gradient; aggregating the noisy gradient and calculating the average gradient for model parameter update;

[0068] A calculation formula of the secure aggregation gateway performing differential privacy protection is as follows:

[0069]

[0070] wherein, B denotes a gradient sample set of a batch, denotes the gradient of the i-th sample, denotes a gradient clipping function, denotes a Gaussian noise with a mean of 0 and a standard deviation of C, I denotes a unit matrix, denotes the average gradient after noise addition and aggregation.

[0071] ​​In the multi-algorithm fusion scene, the differential privacy gradient aggregation is combined with the sparse gating training mechanism to realize the "sparse model selection + privacy safe update" double constraint training; the coupling control of noise variance σ and gradient clipping threshold C provides a new path for high-precision privacy training in the federal scene; the differential privacy process is creatively integrated into the multi-model fusion engine instead of being executed independently.

[0072] In differential privacy training, the gradient loses part of the directional information after being added with noise, resulting in performance degradation when the model is fused.

[0073] The privacy gradient spectrum reconstruction mechanism (PGSR) recovers part of the structural gradient information through sparse reconstruction in the frequency domain without breaking the privacy budget chain, so that differential privacy training and multi-model fusion achieve a higher precision balance; the specific steps are as follows:

[0074] For each model's gradient tensor, use the fast Fourier transform to map it from the time / space domain to the frequency domain; in the frequency domain, different frequency bands of data represent different levels of information: the low-frequency part represents the main structural information of the gradient, and the high-frequency part usually contains noise or detailed information.

[0075] After adding noise, the frequency domain representation of each gradient tensor is decomposed into several frequency bands; on each frequency band, record its energy distribution and calculate the gradient energy spectrum features. These features can be obtained by calculating the magnitude spectrum or phase spectrum in the frequency spectrum. For each model, record its energy distribution in different frequency bands.

[0076] Based on the gradient energy spectrum features of the low-frequency part, a sparse reconstruction network is used to recover the structural information of the gradient, which introduces a sparse constraint for regularization, and the specific formula is as follows: wherein, represents the gradient after sparse reconstruction, G represents the target gradient to be optimized, represents the gradient of the low-frequency part L extracted from the frequency spectrum, F(G) represents the fast Fourier transform operation on the gradient G, which maps the gradient G from the time or space domain to the frequency domain, represents the low-frequency part of the transformed gradient F(G), represents norm, in the context, the norm measures the difference between the low-frequency part gradient and the sparse gradient after transformation, represents the regularization parameter, which controls the strength of sparsification, denotes norm, for sparsification, arg denotes obtaining an optimal G that minimizes the value of the objective function. By sparsely reconstructing the spectral components of the low-frequency part, direct access to the original high-frequency information or sensitive data is avoided.

[0077] The privacy consumption of the PGSR process is monitored in real time by the privacy budget chain verification module, and the differential privacy algorithm is used to calculate the privacy expenditure of each model; when gradient reconstruction is performed, each reconstruction of the low-frequency part consumes a part of the privacy budget, and it is necessary to ensure that the total privacy consumption does not exceed the total budget, if the budget is exceeded, the system will automatically downgrade to the traditional differential privacy mode, and directly use the noisy gradient update, thereby ensuring the privacy security.

[0078] The reconstructed gradient is input to the fusion layer gate network, and the fusion layer gate network dynamically adjusts the fusion weight according to the gradient structure and spectral characteristics of the model; the gate network adopts an adaptive gating mechanism, and at each update, the contribution of each model is weighted according to the stability of the gradient and the noise level. Specifically, the network selectively focuses on the key information of the gradient through the self-attention mechanism, and in the federated learning or high-noise environment, the Top-K model weighted accuracy and the stability of the training are significantly improved by weighting and fusing the output of each model.

[0079] During the training process, if it is detected that the gradient of a certain model is affected by excessive noise (such as privacy noise from local data), the proportion of the low-frequency part and the high-frequency part in the reconstruction process is automatically adjusted, so that the accuracy of the reconstruction remains at a balance point between efficiency and privacy security.

[0080] S6, the performance drift index is continuously monitored by the collaborative feedback channel, and if the performance decrease is detected to exceed a preset threshold and lasts for more than a tolerance round, the system automatically rolls back to the best model state or triggers incremental training; the collaborative feedback channel records the performance of each round of model and compares the best performance, and if the performance decreases for consecutive rounds and exceeds a preset difference, the model parameter rollback is performed.

[0081] S7, the training results and performance indicators are pushed to the A / B indicator board in real time.

[0082] According to the present application, a model fusion training pseudo code is as follows:

[0083] This code shows the model fusion based on sparse gating, differential privacy training and collaborative feedback process.

[0084] (1) Sparse gating mechanism: through Top-K sparsification, the computational complexity is greatly reduced, making it possible to fuse a large number of models while maintaining the model expression ability.

[0085] (2) Differential Privacy Protection: Add strictly calibrated noise at the gradient level to prevent training data leakage and ensure overall privacy consumption controllable through Privacy Budget Chain.

[0086] (3) Collaborative Feedback Loop: Real-time monitoring of model performance, automatic rollback to the best state or trigger incremental training once data drift or performance decline is detected, ensuring system robustness.

[0087] (4) Resource Coordination: Apply isolated GPU computing power fragments through "GPU MIG" before training to achieve efficient resource utilization.

[0088] """

[0089] import torch

[0090] import torch.nn as nn

[0091] import torch.nn.functional as F

[0092] import numpy as np

[0093] from typing import List

[0094] ---1. Sparse Gating Network Definition---

[0095] class SparseGatingNetwork(nn.Module):

[0096] def __init__(self, input_dim, num_models, top_k, hidden_dim=64):

[0097] super(SparseGatingNetwork, self).__init__()

[0098] self.top_k = top_k

[0099] self.fc1 = nn.Linear(input_dim, hidden_dim)

[0100] self.fc2 = nn.Linear(hidden_dim, num_models)

[0101] def forward(self, x):

[0102] h = F.gelu(self.fc1(x)) # Use the GeLU activation function

[0103] logits = self.fc2(h)

[0104] # Keep TopK, set the rest to negative infinity

[0105] topk_vals, topk_indices = torch.topk(logits, self.top_k, dim=-1)

[0106] sparse_logits = torch.full_like(logits, -float('inf'))

[0107] sparse_logits.scatter_(-1, topk_indices, topk_vals)

[0108] # Generate sparse weights

[0109] gate_weights = F.softmax(sparse_logits, dim=-1)

[0110] return gate_weights

[0111] --- 2. Safe Aggregate Functions ---

[0112] def secure_aggregate(clipped_grads: List[torch.Tensor], clip_norm:float, noise_multiplier: float):

[0113] """

[0114] Achieve differential privacy and security aggregation;

[0115] """

[0116] # Gradient Summation

[0117] summed_grads = torch.stack(clipped_grads).sum(dim=0)

[0118] # Add Gaussian noise

[0119] noise_stddev = clip_norm * noise_multiplier

[0120] noise = torch.normal(0, noise_stddev, size=summed_grads.shape,device=summed_grads.device)

[0121] noised_sum = summed_grads + noise

[0122] # Calculate the average gradient

[0123] aggregated_grads = noised_sum / len(clipped_grads)

[0124] return aggregated_grads

[0125] ---3. Collaborative Feedback Monitor---

[0126] class FeedbackMonitor:

[0127] def __init__(self, threshold=0.05, patience=3):

[0128] self.threshold = threshold # Performance degradation threshold

[0129] self.patience = patience # Tolerance rounds

[0130] self.best_performance = None

[0131] self.counter = 0

[0132] self.best_model_state = None

[0133] def check_rollback(self, current_perf, model):

[0134] if self.best_performance is None or current_perf > self.best_performance:

[0135] self.best_performance = current_perf

[0136] self.best_model_state = copy.deepcopy(model.state_dict())

[0137] self.counter = 0

[0138] else:

[0139] self.counter += 1

[0140] # If performance continues to decline beyond the threshold and the tolerance round, a rollback will be triggered.

[0141] if self.counter >= self.patience and (self.best_performance - current_perf) > self.threshold:

[0142] print("Performance drift detected! Triggering rollback to optimal model state.")

[0143] model.load_state_dict(self.best_model_state)

[0144] self.counter = 0

[0145] return True # Rollback complete

[0146] return False # No rollback

[0147] ---4. Main Training Loop (Summary)---

[0148] def main_training_loop():

[0149] # Initialization: Load models, gating networks, etc. from the algorithm repository

[0150] base_models = load_models() # Load models from the algorithm repository (ML / DL / graph / federation)

[0151] gate_net = SparseGatingNetwork(input_dim=100, num_models=len(base_models), top_k=2)

[0152] optimizer = torch.optim.Adam([*gate_net.parameters(), *base_models.parameters()])

[0153] feedback_monitor = FeedbackMonitor()

[0154] # Training parameters

[0155] clip_norm = 1.0

[0156] noise_multiplier = 0.1 # Differential privacy noise multiplier

[0157] for epoch in range(100):

[0158] for batch_x, batch_y in data_loader:

[0159] # Forward propagation: Sparse gated fusion:

[0160] gate_weights = gate_net(batch_x)

[0161] output = torch.zeros_like(batch_y)

[0162] for i, model in enumerate(base_models):

[0163] # Only compute activated models, save computation resources

[0164] if gate_weights[0, i] > 0:

[0165] output += gate_weights[:, i].view(-1,1) * model(batch_x)

[0166] loss = F.mse_loss(output, batch_y)

[0167] # Backward propagation

[0168] optimizer.zero_grad()

[0169] loss.backward()

[0170] # (Optional) Secure aggregation with differential privacy: clip and noisily aggregate gradients

[0171] # clipped_grads = [clip_grad(model, clip_norm) for model in base_models]

[0172] # aggregated_grads = secure_aggregate(clipped_grads, clip_norm, noise_multiplier)

[0173] # Update models using aggregated_grads...

[0174] # Regular update

[0175] optimizer.step()

[0176] # Evaluate performance on validation set

[0177] val_perf = evaluate_model()

[0178] # Collaborative feedback: check performance drift and decide whether to roll back

[0179] feedback_monitor.check_rollback(val_perf, base_models)

[0180] # Push metrics to A / B dashboard in real-time

[0181] push_to_dashboard(val_perf, loss.item())。

[0182] Intelligent application collaboration layer:

[0183] The intelligent application collaboration domain integrates the real-time interface of the fusion engine through REST / gRPC calls to complete real-time early warning, decision support, personalized recommendations, and joint risk control. The fusion engine applies for GPU multi-instance slicing (MIG) to obtain isolated computing resources from the computing power scheduling module before training.

[0184] Business-side tracking points return real effects to the unified access gateway layer, re-entering the closed loop of each layer to achieve minute-level autonomous optimization and upgrade, and supporting one-key export of PDF / A compliance reports to complete continuous iteration from the "collection-treatment-fusion-use" end to end.

[0185] The following reference is made to Figure 2It shows a schematic diagram of the structure of a computer system 200 suitable for implementing electronic devices according to embodiments of the present application. Figure 2 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0186] like Figure 2 As shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 202 or programs loaded from storage section 208 into random access memory (RAM) 203. The RAM 203 also stores various programs and data required for the operation of the system 200. The CPU 201, ROM 202, and RAM 203 are interconnected via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0187] The following components are connected to I / O interface 205: an input section 206 including a keyboard, mouse, etc.; an output section 207 including a liquid crystal display (LCD) and speakers, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card and a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to I / O interface 205 as needed. A removable medium 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 210 as needed so that computer programs read from it can be installed into storage section 208 as needed.

[0188] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 209, and / or installed from the detachable medium 211. When the computer program is executed by the central processing unit (CPU) 201, the above-described functions defined in the methods of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be either a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may, for example, be but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. In the present application, the computer-readable signal medium can include a data signal that propagates in a baseband or as part of a carrier wave, in which the computer-readable program code is carried. Such a propagated data signal can take on many forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable storage medium that can be used to carry or store program code for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted as program code signals using any suitable medium, including but not limited to wireless, wire line, optical fiber, RF, or any suitable combination of the above.

[0189] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0190] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0191] The modules involved in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware.

[0192] As another aspect, the present application also provides a computer readable storage medium, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device. The computer readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: receive the feature flow, the policy flow and the resource quota from the integrated collaborative control layer by the multi-algorithm fusion engine layer; call a plurality of basic models in the algorithm warehouse, the basic models including a machine learning model, a deep learning model, a graph model or a federated learning model; generate sparse gating weights through a sparse gating network; weight fuse the activated basic model output based on the sparse gating weights to obtain a multi-model fusion output; in the training process, perform differential privacy protection through a secure aggregation gateway; continuously monitor the performance drift index through the collaborative feedback channel, and if it is detected that the performance decline exceeds a preset threshold and lasts for more than a tolerance round, automatically roll back to the best model state or trigger incremental training; and push the training result and the performance index to an A / B index board in real time.

[0193] The above description is merely the preferred embodiments of the present application and the explanation of the technical principles applied. It should be understood by those skilled in the art that the inventive scope involved in the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or their equivalent features without departing from the inventive concept. For example, the technical solutions formed by replacing the above features with the technical features disclosed in the present application (but not limited to) having similar functions.

Claims

1. A multi-algorithm fusion big data analysis method based on integrated collaboration, characterized in that, include: S0, Unified Access Gateway Layer, is used to obtain or register the latest data format by remotely calling the metadata registry center in the data governance and storage layer before writing any data; The streaming acquisition module and the batch acquisition module encapsulate events according to the data format; and the Sidecar deployed at the exit performs data desensitization, zero-trust encryption and anomaly interception processing. After processing, it accesses the metadata registry center again to confirm version consistency, thereby outputting a clean and traceable standardized event stream to the data governance and storage layer. S1. The multi-algorithm fusion engine layer receives feature streams, policy streams, and resource quotas from the integrated collaborative control layer. S2. Call multiple basic models in the algorithm repository, including machine learning models, deep learning models, graph models, or federated learning models; S3. Generate sparse gating weights through a sparse gating network, wherein: the sparse gating network calculates the activation scores of each base model based on the input features; performs Top-K filtering on the activation scores, retaining only the K base models with the highest scores; performs Softmax normalization on the scores of the retained models to obtain the sparse gating weights; S4. Based on the sparse gating weights, the outputs of the activated basic models are weighted and fused to obtain the multi-model fusion output; S5. During training, differential privacy protection is performed through a secure aggregation gateway, including: pruning the gradients of each model; adding calibrated Gaussian noise to the pruned gradients; aggregating the noisy gradients and calculating the average gradient for model parameter updates; the calculation formula for differential privacy protection performed by the secure aggregation gateway is as follows: Where B represents a set of gradient samples in a batch, This represents the gradient of the i-th sample. Represents the gradient clipping function. This indicates that the mean is 0 and the standard deviation is 0. C's Gaussian noise, Let I represent the average gradient after adding noise and aggregating, and let I represent the identity matrix. S6. The collaborative feedback channel continuously monitors performance drift metrics. If a performance drop exceeds a preset threshold and continues to exceed the tolerance number of rounds, it will automatically roll back to the best model state or trigger incremental training. S7. Training results and performance metrics are pushed to the metrics dashboard in real time; Following the S8 multi-algorithm fusion engine layer, there is also an intelligent application collaboration layer, which is used to call the real-time services of the multi-algorithm fusion engine through REST or gRPC interfaces to achieve real-time early warning, decision support, personalized recommendation and joint risk control.

2. The multi-algorithm fusion big data analysis method according to claim 1, characterized in that, The formula for calculating the sparse gating weights is as follows: Where x represents the input feature vector, and These represent the weights and biases of the hidden layer in the gated network, respectively. Indicates the weights of the output layer of the gated network. Let represent the bias of the output layer of the gated network, TopK(z,K) represent the K largest values ​​in the return vector z, g represent the sparse gate weight vector with only K non-zero values, h represent the hidden layer output of the gated network, and z represent the original output layer values ​​of the gated network. This represents the k-th element in vector z. This represents the k-th gate value after sparsification, where k represents the position index.

3. The multi-algorithm fusion big data analysis method according to claim 1, characterized in that, The calculation formula for the multi-model fusion output is as follows: in, Represents the k-th base model. This represents the sparse gating weights corresponding to the k-th model, where K represents the total number of base models with the highest scores, and k represents the position index. This represents the output of multi-model fusion, and x represents the input feature vector.

4. The multi-algorithm fusion big data analysis method according to claim 1, characterized in that, The unified access gateway layer is configured with an event feedback mechanism to verify the consistency of metadata versions after data encapsulation and encryption, ensuring that the data format matches the version registered in the Schema Registry.

5. The multi-algorithm fusion big data analysis method according to claim 1, characterized in that, The multi-algorithm fusion engine layer is preceded by a data governance and storage layer, which is used to govern, store and manage the standard event stream from the unified access gateway layer. The components follow the dependency order of governance first, service then consumption, forming a unidirectional and acyclic data dependency chain.

6. The multi-algorithm fusion big data analysis method according to claim 5, characterized in that, The data governance and storage layer includes: The data lake module is used to partition and store standardized events according to business time and perform incremental compression; Metadata center, used to generate field-level lineage information, lifecycle tags and model cards on each write; Vector indexing service and feature repository are used to keep snapshots consistent with the metadata center.

7. The multi-algorithm fusion big data analysis method according to claim 5, characterized in that, The multi-algorithm fusion engine layer is preceded by an integrated collaborative control layer, which is used to achieve unified scheduling and feedback loop of policies, computing power and privacy between the data governance and storage layer and the multi-algorithm fusion engine layer.

8. The multi-algorithm fusion big data analysis method according to claim 7, characterized in that, The integrated collaborative control layer includes a data collaboration center, an algorithm collaboration center, a computing power collaboration center, and a security collaboration center, wherein... The data collaboration center is used to read metadata and data snapshots from the Schema Registry and Iceberg in the data governance and storage layer in real time, and calculate data quality scores based on field-level lineage provided by the metadata center; The algorithm collaboration center calls the automated machine learning module to perform hyperparameter search based on the data quality score, and generates candidate algorithms. The computing power collaboration center allocates resource quotas and GPU multi-instance slices based on a comprehensive four-dimensional factor: task urgency, model complexity, privacy budget, and available computing power. The security collaboration center broadcasts policy updates to each node of the system through the control channel and pushes the feature stream and policy stream to the multi-algorithm fusion engine layer through the data channel.

9. The multi-algorithm fusion big data analysis method according to claim 1, characterized in that, The intelligent application collaboration layer is configured with a business feedback mechanism, which is used to send the real effect data generated by the business end tracking points back to the unified access gateway layer. Through each layer, a minute-level closed-loop optimization is formed, and it supports one-click generation of compliance reports that conform to the PDF / A standard, realizing end-to-end continuous iteration.

10. A computer program product, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-9.

11. A computing system, characterized in that, It includes a processor and a memory, the processor being configured to perform the method as described in any one of claims 1-9.

Images