Deep learning-based multi-source information data fusion processing method and system

Abstract

The application discloses a multi-source information data fusion processing method and system based on deep learning, and relates to the technical field of intelligent information processing, which comprises the following steps: each participating node extracts features from local multi-source data and fuses the features into unified feature representation, aggregates the unified feature representation into a local prototype vector according to a label category and calculates a quality score, trains a local deep learning model to obtain a total amount of local samples; the quality score is used to determine the prototype disturbance noise intensity, which is inversely proportional to the quality score, and the noise is superimposed and uploaded with the model gradient; the server respectively performs weighted average on the same prototype vector and gradient according to the quality score and the total amount of local samples as weights and then issues the weighted average; each participating node constructs a prototype consistency loss and a classification loss based on classification prediction, and the training target is a weighted combination of the two, and the prototype consistency loss weight is inversely proportional to the quality score. The application uniformly regulates the noise intensity, the aggregation weight and the training focus according to the quality score, and realizes the collaborative optimization of privacy and performance.

Description

Technical Field

[0001] This invention relates to the field of intelligent information processing technology, specifically to a method and system for multi-source information data fusion processing based on deep learning. Background Technology

[0002] With the rapid development of the Internet of Things (IoT) and edge computing, scenarios involving the distributed holding of multi-source heterogeneous data are prevalent in fields such as medical diagnosis, industrial quality inspection, and intelligent transportation. Each participating node locally possesses multiple modalities of data, including images, text, and sensor time series data. This necessitates the fusion of information from multiple sources to complete joint modeling while protecting data privacy. Federated learning, as a mainstream framework for distributed privacy computing, allows participating nodes to train their models locally and then upload the model parameters to an aggregation server for global aggregation, preventing the leakage of raw data. It has become a research hotspot in the field of multi-source data fusion. Under the federated learning framework, how to effectively fuse the multi-modal data held locally by each participating node while protecting the data privacy of each participating node during model parameter uploading and aggregation is a core problem that urgently needs to be solved. CN115859367B discloses a privacy-preserving method and system for multimodal federated learning. After extracting features from multimodal data locally at each participating node, it generates differential privacy perturbation data through a generative adversarial network (GAN). Localized differential privacy noise is then applied to the local model parameters before being uploaded to an aggregation server for global aggregation. However, CN115859367B has the following shortcomings: First, applying the same intensity of differential privacy noise to all participating nodes fails to consider the differences in local data quality among nodes. Model parameters of higher-quality nodes are subjected to unnecessary noise perturbation, wasting privacy budget and reducing global model accuracy. Second, it does not employ a prototype mechanism for class-level knowledge aggregation, failing to mitigate data heterogeneity issues through prototype alignment. Furthermore, existing prototype-based federated learning schemes transmit prototype vectors in plaintext, lacking a privacy protection mechanism. Third, noise intensity, aggregation weights, and local training emphasis are set independently, lacking a coordinated control mechanism with data quality as a unified link, making it difficult to balance privacy protection strength and model performance.

[0003] Therefore, this invention proposes a method and system for multi-source information data fusion processing based on deep learning. Summary of the Invention

[0004] To address the aforementioned technical issues, this paper provides a method and system for multi-source information data fusion processing based on deep learning. This technical solution solves the problems of wasted privacy budget due to uniform noise, lack of privacy protection in prototype plaintext transmission, and lack of coordination among noise aggregation training in multi-source data federated learning.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method and system for multi-source information data fusion processing based on deep learning, including: Step S1: Each participating node extracts features from local multi-source data and fuses them into a unified feature representation. The data is then aggregated into a local prototype vector according to the label category, and a quality score is calculated. The unified feature representation is used to train a local deep learning model to obtain classification prediction and model gradient, and the total number of local samples is counted. Step S2: Each participating node determines the prototype perturbation noise intensity based on the quality score. The noise intensity is inversely proportional to the quality score. The corresponding intensity noise is superimposed on the local prototype vector and uploaded together with the local model gradient to the aggregation server. Step S3: The aggregation server uses the quality score as the weight to calculate the weighted average of the prototype vectors of the same category to obtain the global prototype vector set, and uses the total number of local samples as the weight to calculate the weighted average of the model gradient to obtain the global model gradient, and distributes it to each participating node. Step S4: Each participating node constructs a prototype consistency loss based on the distance between the global and local prototype vectors, and constructs a classification loss based on classification prediction. The training objective is a weighted combination of the classification loss and the prototype consistency loss. The weight of the prototype consistency loss is inversely proportional to the quality score.

[0006] Preferably, in step S1, the local multi-source data includes at least two modalities from images, text, and sensor time-series data. The extraction and fusion of features into a unified feature representation involves extracting modal features for the image modality using a convolutional network, for the text modality using a text encoder, and for the sensor time-series modality using a time-series coding network. After mapping each modal feature to the same dimension, they are concatenated into a joint feature matrix. The attention score is calculated by the dot product of the learnable query vector and the joint feature matrix and normalized to the attention weight using softmax. The unified feature representation is obtained by weighted summation of each modal feature using the attention weight.

[0007] Preferably, in step S1, aggregating local prototype vectors by label category and calculating quality scores includes: The local prototype vector of the corresponding label category is obtained by taking the mean of the uniform feature representations of the same label category. The intra-class tightness is the reciprocal of the average distance between each uniform feature representation and the local prototype vector within the same label category. The inter-class separation is the minimum distance between the local prototype vector of the corresponding label category and the local prototype vectors of other label categories. The quality score is calculated by multiplying the sample size of the corresponding label category, the intra-class tightness, and the inter-class separation, and then normalizing the result.

[0008] Preferably, in step S1, the local deep learning model sequentially includes: The feature mapping layer and the classification output layer are used to obtain classification predictions. The classification predictions are obtained by nonlinear transformation of the feature mapping layer using a unified feature representation. The classification loss between the classification prediction and the label category of each sample is calculated using cross-entropy loss. The classification loss is used to backpropagate the local deep learning model to obtain the model gradient. The total number of local samples is the sum of the number of samples of each label category in the local model.

[0009] Preferably, in step S2, the prototype disturbance noise intensity is determined based on the quality score, and the noise intensity is inversely proportional to the quality score as follows: The quality scores of each participating node are normalized to obtain a normalized quality score. The local privacy budget of each participating node is determined by the product of the preset global privacy budget and the normalized quality score. The sensitivity is determined by the component range of the dimension to which the local prototype vector belongs. The standard deviation of the noise is determined by the ratio of the sensitivity to the local privacy budget as the noise intensity. The perturbed prototype vector is obtained by superimposing Gaussian noise with the corresponding standard deviation on the local prototype vector. The perturbed prototype vector and the model gradient are uploaded to the aggregation server.

[0010] Preferably, in step S3, the global prototype vector set is obtained by weighting the prototype vectors of the same category using quality scores as weights: The aggregation server takes prototype vectors from participating nodes that belong to the same tag category, uses the normalized quality score of the tag category of each participating node as the aggregation weight, and performs a weighted average of each prototype vector to obtain a global prototype vector set for the corresponding tag category. The global prototype vector sets of each tag category are then concatenated according to the tag category to obtain a global prototype vector set.

[0011] Preferably, in step S3, the global model gradient is obtained by weighting the model gradient using the total local sample size as the weight: The aggregation server uses the normalized value of the total local sample volume of each participating node as the gradient aggregation weight, and calculates the weighted average of the model gradients uploaded by each participating node to obtain the global model gradient. The global prototype vector set and the global model gradient are then distributed to each participating node.

[0012] Preferably, in step S4, the prototype consistency loss constructed from the distance between the global and local prototype vectors is the square of the Euclidean distance between the global prototype vector set and the local prototype vectors, and the classification loss constructed from the classification prediction is the cross-entropy between the classification prediction of the local deep learning model and the corresponding label category.

[0013] Preferably, in step S4, the weight of the prototype consistency loss is inversely proportional to the quality score as follows: The normalized quality score is used as the weight of the classification loss, and the difference between 1 and the normalized quality score is used as the weight of the prototype consistency loss. The training objective is the product of the classification loss weight and the classification loss plus the product of the prototype consistency loss weight and the prototype consistency loss.

[0014] Preferably, a deep learning-based multi-source information data fusion processing system includes: The local processing module, located at each participating node, is used to extract features from local multi-source data and fuse them into a unified feature representation, aggregate them into local prototype vectors according to label categories and calculate quality scores, train local deep learning models with the unified feature representation to obtain classification predictions and model gradients, and count the total number of local samples. The noise perturbation module is located at each participating node and is used to determine the prototype perturbation noise intensity based on the quality score. The noise intensity is inversely proportional to the quality score. The corresponding intensity noise is superimposed on the local prototype vector and uploaded together with the local model gradient to the aggregation server. The global aggregation module, located on the aggregation server, is used to calculate the global prototype vector set by weighting the prototype vectors of the same category with the quality score as the weight, and to calculate the global model gradient by weighting the model gradient with the total number of local samples as the weight, and then distribute it to each participating node. The local training module, located on each participating node, is used to construct the prototype consistency loss based on the distance between the global and local prototype vectors, and to construct the classification loss based on classification prediction. The training objective is a weighted combination of the classification loss and the prototype consistency loss, and the weight of the prototype consistency loss is inversely proportional to the quality score.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes an adaptive allocation of prototype perturbation noise intensity based on the quality scores of each participating node. Higher quality scores result in lower noise intensity, achieving differentiated privacy protection where high-quality prototype vectors are subjected to weaker perturbations and low-quality prototype vectors to stronger perturbations. This significantly improves global model accuracy under the same privacy budget. The invention also proposes introducing a prototype mechanism to aggregate local prototype vectors at each participating node by label category, achieving class-level knowledge aggregation and prototype alignment to overcome data heterogeneity issues. Furthermore, it combines differential privacy noise with the prototype mechanism, superimposing noise on local prototype vectors before uploading them to the aggregation server. This protects the data privacy of each participating node while preserving the ability of prototype alignment to mitigate data heterogeneity. Finally, it proposes using quality scores as a unified link to collaboratively regulate noise intensity, global aggregation weights, and prototype consistency loss weights. This achieves coordinated optimization of privacy protection intensity, global knowledge aggregation emphasis, and local training emphasis, overcoming the difficulty of balancing privacy protection and model performance. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a system framework diagram of the present invention. Detailed Implementation

[0017] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0018] Reference Figure 1 As shown, a multi-source information data fusion processing method based on deep learning includes: Step S1: Each participating node extracts features from local multi-source data and fuses them into a unified feature representation. The data is then aggregated into a local prototype vector according to the label category, and a quality score is calculated. The unified feature representation is used to train a local deep learning model to obtain classification prediction and model gradient, and the total number of local samples is counted. Step S2: Each participating node determines the prototype perturbation noise intensity based on the quality score. The noise intensity is inversely proportional to the quality score. The corresponding intensity noise is superimposed on the local prototype vector and uploaded together with the local model gradient to the aggregation server. Step S3: The aggregation server uses the quality score as the weight to calculate the weighted average of the prototype vectors of the same category to obtain the global prototype vector set, and uses the total number of local samples as the weight to calculate the weighted average of the model gradient to obtain the global model gradient, and distributes it to each participating node. Step S4: Each participating node constructs a prototype consistency loss based on the distance between the global and local prototype vectors, and constructs a classification loss based on classification prediction. The training objective is a weighted combination of the classification loss and the prototype consistency loss. The weight of the prototype consistency loss is inversely proportional to the quality score.

[0019] In the specific implementation process, each participating node first extracts and fuses features from the various modalities of data it holds locally. Taking a medical diagnosis scenario as an example, each participating node locally holds image modalities (such as CT images), text modalities (such as electronic medical records), and sensor time-series modalities (such as electrocardiogram signals). These multimodal data need to be fused into a unified feature representation before subsequent training and prototype construction can proceed. The calculation of the quality score comprehensively considers the number of samples in each category, intra-class density, and inter-class separation, which can quantify the reliability of the category information contained in each local prototype vector. The purpose of counting the total number of local samples is to provide a weight basis for the weighted aggregation of model gradients in S3. The larger the sample size of the participating node, the greater its gradient contribution to the global model gradient. Determining the noise intensity based on the quality score in S2 is a key step in achieving a balance between privacy protection and model accuracy. The higher the quality score, the more reliable the prototype vector is, and applying weaker noise can meet the privacy requirements, thereby avoiding excessive perturbation of high-quality prototypes; the lower the quality score, the stronger the noise needs to be applied to protect prototypes that may contain abnormal information. In S3, the aggregation server performs weighted aggregation of prototype vectors and model gradients. Prototype vectors are weighted by their quality scores to ensure that high-quality prototypes occupy a larger proportion of the global prototype. Model gradients are weighted by their local sample volume to ensure that nodes with sufficient data contribute more to the update of the global model. In S4, the training objective is a weighted combination of classification loss and prototype consistency loss. The weight of prototype consistency loss is inversely proportional to the quality score; that is, nodes with higher quality scores trust their own prototypes more and follow the global prototype less. Nodes with lower quality scores need to align more closely with the global prototype to obtain global knowledge.

[0020] By adopting the above technical solution, noise intensity, aggregation weights and training emphasis can be adjusted simultaneously through quality scoring, thereby achieving synergistic optimization of privacy protection and model performance.

[0021] In step S1, the local multi-source data includes at least two modalities from images, text, and sensor time-series data. The extraction and fusion of features into a unified feature representation involves extracting modal features for the image modality using a convolutional network, for the text modality using a text encoder, and for the sensor time-series modality using a time-series coding network. After mapping each modal feature to the same dimension, they are concatenated into a joint feature matrix. The attention score is calculated by the dot product of the learnable query vector and the joint feature matrix and normalized to the attention weight using softmax. The unified feature representation is obtained by weighting and summing the attention weights of each modal feature.

[0022] In the specific implementation, the three modalities employ corresponding feature extraction networks: the image modality uses a convolutional neural network (such as ResNet) to extract spatial feature vectors, the text modality uses a text encoder (such as BERT or Transformer encoder) to extract semantic feature vectors, and the sensor temporal modality uses a temporal coding network (such as LSTM or TCN) to extract temporal feature vectors. The original dimensions of the feature vectors for each modality may differ, requiring a linear mapping layer to map them to the same hidden dimension d for subsequent concatenation. After mapping, the feature vectors of the K modalities are concatenated row-wise to form a joint feature matrix. In the attention fusion stage, a learnable query vector q (with dimension d) is introduced. The dot product of q and each row of the joint feature matrix U is calculated to obtain K attention scores. After softmax normalization, K attention weights are obtained. The attention weights are then used to weight and sum the feature vectors of each modality to obtain the final unified feature representation with dimension d. This attention fusion mechanism can automatically learn the importance of each modality according to task requirements, avoiding information redundancy problems caused by simple concatenation or average fusion. In another implementation, attention fusion can also employ a multi-head attention mechanism, which involves setting up multiple sets of query vectors, calculating attention weights for each, and then concatenating the results to capture modal interaction relationships in different subspaces.

[0023] In step S1, aggregating local prototype vectors by label category and calculating quality scores includes: The local prototype vector of the corresponding label category is obtained by taking the mean of the uniform feature representations of the same label category. The intra-class tightness is the reciprocal of the average distance between each uniform feature representation and the local prototype vector within the same label category. The inter-class separation is the minimum distance between the local prototype vector of the corresponding label category and the local prototype vectors of other label categories. The quality score is calculated by multiplying the sample size of the corresponding label category, the intra-class tightness, and the inter-class separation, and then normalizing the result.

[0024] In the specific implementation process, assuming that a participating node has C local label categories, for the c-th category, the average of the unified feature representations of all Nc samples in that category is taken to obtain the local prototype vector Pc of the c-th category. The intra-class tightness is calculated as follows: calculate the Euclidean distance between all Nc unified feature representations in the c-th category and the local prototype vector Pc, take the average distance, and then take the reciprocal. That is, intra-class tightness is positively correlated with the degree of intra-class aggregation; the tighter the intra-class tightness, the larger this value. The inter-class separation is calculated as follows: calculate the Euclidean distance between the local prototype vector Pc of the c-th category and the local prototype vectors of the other C-1 categories, and take the minimum value as the inter-class separation. That is, the larger the distance between the c-th category and its nearest neighbor category, the greater the separation. The quality score is calculated as follows: multiply the sample size of the corresponding label category, the intra-class tightness, and the inter-class separation to obtain the quality score of that label category. The quality scores for all C categories are normalized by dividing each quality score by the sum of the quality scores for all categories, so that the sum of the normalized quality scores is 1. In another implementation, the normalization can also be achieved using max-min normalization, which involves subtracting the minimum value from each quality score and then dividing by the difference between the maximum and minimum values.

[0025] In step S1, the local deep learning model includes, in sequence: The feature mapping layer and the classification output layer are used to obtain classification predictions. The classification predictions are obtained by nonlinear transformation of the feature mapping layer using a unified feature representation. The classification loss between the classification prediction and the label category of each sample is calculated using cross-entropy loss. The classification loss is used to backpropagate the local deep learning model to obtain the model gradient. The total number of local samples is the sum of the number of samples of each label category in the local model.

[0026] In practical implementation, the feature mapping layer consists of two fully connected layers and a ReLU activation function, nonlinearly mapping a unified feature representation of dimension d to a new feature space. The classification output layer is a fully connected layer with a softmax activation function, mapping the output of the feature mapping layer to a C-dimensional probability vector as the classification prediction. During training, the classification loss between the classification prediction and the true label category is calculated using cross-entropy loss, and the gradient of the classification loss with respect to the parameters of each layer of the local deep learning model is calculated using the backpropagation algorithm, i.e., the model gradient. The total local sample size is the sum of the sample sizes for each label category, i.e., the sum of the sample sizes for all label categories in the local model. In another implementation, the feature mapping layer can also use a residual connection structure to alleviate gradient vanishing, and a Dropout layer can be added before the classification output layer to prevent overfitting.

[0027] In step S2, the prototype disturbance noise intensity is determined based on the quality score, and the noise intensity is inversely proportional to the quality score: The quality scores of each participating node are normalized to obtain a normalized quality score. The local privacy budget of each participating node is determined by the product of the preset global privacy budget and the normalized quality score. The sensitivity is determined by the component range of the dimension to which the local prototype vector belongs. The standard deviation of the noise is determined by the ratio of the sensitivity to the local privacy budget as the noise intensity. The perturbed prototype vector is obtained by superimposing Gaussian noise with the corresponding standard deviation on the local prototype vector. The perturbed prototype vector and the model gradient are uploaded to the aggregation server.

[0028] In the specific implementation process, a global privacy budget is preset as a system hyperparameter to control the overall privacy protection strength; the smaller the global privacy budget, the stronger the privacy protection. The local privacy budget of each participating node is determined by the product of the global privacy budget and the corresponding normalized quality score; nodes with higher quality scores are allocated larger local privacy budgets. Sensitivity is determined by calculating the value range of each dimension component of the local prototype vector, and using the maximum difference between the maximum and minimum values ​​of each dimension component as the sensitivity. In another implementation, sensitivity can also be determined by L2 norm clipping; that is, after clipping the local prototype vector to a preset clipping threshold using the L2 norm, the sensitivity is twice the clipping threshold. The noise standard deviation is determined by the ratio of sensitivity to the local privacy budget; the standard deviation is the noise intensity. A Gaussian distribution with a mean of zero and a standard deviation corresponding to the noise intensity is sampled and superimposed on the local prototype vector to obtain a perturbed prototype vector. This noise superposition mechanism satisfies the mathematical guarantee of differential privacy, where the privacy relaxation parameter is a preset value.

[0029] In step S3, the global prototype vector set is obtained by weighting the prototype vectors of the same category with quality scores as weights: The aggregation server takes prototype vectors from participating nodes that belong to the same tag category, uses the normalized quality score of the tag category of each participating node as the aggregation weight, and performs a weighted average of each prototype vector to obtain a global prototype vector set for the corresponding tag category. The global prototype vector sets of each tag category are then concatenated according to the tag category to obtain a global prototype vector set.

[0030] In the specific implementation process, after the aggregation server receives the perturbation prototype vectors and model gradients uploaded by K participating nodes, for a certain label category, it selects the set of nodes holding samples of the corresponding category from the K participating nodes. Using the normalized quality score of each participating node for the corresponding category as the aggregation weight, it calculates a weighted average of the perturbation prototype vectors uploaded by each participating node under that category to obtain the global prototype vector set for the corresponding label category. After calculating the global prototype vector set for all label categories, they are concatenated in label category order to form the global prototype vector set.

[0031] In step S3, the global model gradient is obtained by weighting the model gradient using the total local sample size as the weight: The aggregation server uses the normalized value of the total local sample volume of each participating node as the gradient aggregation weight, and calculates the weighted average of the model gradients uploaded by each participating node to obtain the global model gradient. The global prototype vector set and the global model gradient are then distributed to each participating node.

[0032] In the specific implementation process, the aggregation server normalizes the total local sample volume of each participating node, that is, the gradient aggregation weight of each node is the proportion of its local sample volume to the sum of the local sample volumes of all nodes. The global model gradient is obtained by weighted averaging of the uploaded model gradients of each participating node using the gradient aggregation weights of each participating node. After aggregation is completed, the aggregation server distributes the global prototype vector set and the global model gradient to each participating node.

[0033] In step S4, the prototype consistency loss is constructed by the distance between the global and local prototype vectors, which is the square of the Euclidean distance between the global prototype vector set and the local prototype vector. The classification loss is constructed by the classification prediction, which is the cross-entropy between the classification prediction of the local deep learning model and the corresponding label category.

[0034] In practical implementation, the prototype consistency loss for each participating node is the squared Euclidean distance between the global prototype vector set and the local prototype vector. The global prototype vector set is distributed by the aggregation server, and the local prototype vector is the corresponding category prototype vector local to that node. The classification loss is the cross-entropy between the classification prediction of the local deep learning model for the unified feature representation and the true label category; the average classification loss is taken for all local samples. In another implementation, the prototype consistency loss can also use cosine similarity loss instead of the squared Euclidean distance, i.e., subtracting the difference in cosine similarity between the global prototype vector set and the local prototype vector as the loss value. This is suitable for scenarios where the feature vector direction is more important than the absolute distance.

[0035] In step S4, the weight of the prototype consistency loss is inversely proportional to the quality score: The normalized quality score is used as the weight of the classification loss, and the difference between 1 and the normalized quality score is used as the weight of the prototype consistency loss. The training objective is the product of the classification loss weight and the classification loss plus the product of the prototype consistency loss weight and the prototype consistency loss.

[0036] In practice, the training objective for each participating node on a given label category is a weighted combination of classification loss and prototype consistency loss: the normalized quality score is used as the weight for classification loss, and the difference between 1 and the normalized quality score is used as the weight for prototype consistency loss. The training objective is the product of the classification loss weight and the classification loss itself, plus the product of the prototype consistency loss weight and the prototype consistency loss. When the normalized quality score is high, the classification loss has a larger weight, and the node focuses more on optimizing local classification performance; when the normalized quality score is low, the prototype consistency loss has a larger weight, and the node focuses more on aligning with the global prototype to acquire global knowledge. The intuition behind this weight allocation strategy is that categories corresponding to high-quality prototypes already possess reliable local knowledge and do not need to excessively follow the global prototype; categories corresponding to low-quality prototypes need to rely more on the global prototype to compensate for the deficiencies in local data. The overall training objective for the participating node is obtained by summing the training objectives for all local categories. In another implementation, the weight mapping function can also adopt an exponential decay form, that is, the weight of the prototype consistency loss decays exponentially with the increase of the normalized quality score, and the decay coefficient is a preset value, so that the weight decreases more drastically with the decrease of the quality score.

[0037] A deep learning-based multi-source information data fusion processing system includes: The local processing module, located at each participating node, is used to extract features from local multi-source data and fuse them into a unified feature representation, aggregate them into local prototype vectors according to label categories and calculate quality scores, train local deep learning models with the unified feature representation to obtain classification predictions and model gradients, and count the total number of local samples. The noise perturbation module is located at each participating node and is used to determine the prototype perturbation noise intensity based on the quality score. The noise intensity is inversely proportional to the quality score. The corresponding intensity noise is superimposed on the local prototype vector and uploaded together with the local model gradient to the aggregation server. The global aggregation module, located on the aggregation server, is used to calculate the global prototype vector set by weighting the prototype vectors of the same category with the quality score as the weight, and to calculate the global model gradient by weighting the model gradient with the total number of local samples as the weight, and then distribute it to each participating node. The local training module, located on each participating node, is used to construct the prototype consistency loss based on the distance between the global and local prototype vectors, and to construct the classification loss based on classification prediction. The training objective is a weighted combination of the classification loss and the prototype consistency loss, and the weight of the prototype consistency loss is inversely proportional to the quality score.

[0038] In the specific implementation process, the hardware implementation of each module is as follows: the local processing module and the noise perturbation module are deployed on the GPUs or TPUs of each participating node, responsible for local feature extraction, prototype construction, noise superposition, and parameter uploading; the global aggregation module is deployed on the GPU cluster of the aggregation server, responsible for receiving the perturbation prototype vectors and model gradients uploaded by each participating node and performing weighted aggregation; the local training module is also deployed on the GPUs or TPUs of each participating node, responsible for receiving the global prototype vector set and global model gradients issued by the aggregation server, updating the local model parameters, and calculating the next round of training objectives. The perturbation prototype vectors and model gradients are transmitted between each participating node and the aggregation server via a TLS encrypted communication link to ensure security during transmission.

[0039] For example, each participating node can be a mobile electronic device such as a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, wearable device, super mobile personal computer, netbook, or personal digital assistant, or a non-mobile electronic device such as a server, network attached storage, personal computer, television, ATM, or self-service machine. This application embodiment does not impose specific limitations. The electronic device used for multi-source information data fusion processing in this application embodiment can be an electronic device with an operating system, such as Android, iOS, or other possible operating systems. This application embodiment does not impose specific limitations.

[0040] A readable non-transitory storage medium includes a computer-readable non-transitory storage medium, such as a computer read-only memory, random access memory, magnetic disk, or optical disk, etc., on which a program or instruction is stored, and when the program or instruction is executed by a processor, it implements the steps of the deep learning-based multi-source information data fusion processing method provided in any embodiment of this application.

[0041] In summary, this application achieves synergistic optimization of privacy protection and model performance by calculating the quality score of each local prototype vector and simultaneously adjusting the noise intensity, aggregation weights, and training focus based on the quality score, thus significantly improving the global model accuracy under the same privacy budget.

[0042] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A multi-source information data fusion processing method based on deep learning, characterized in that, Includes the following steps: Step S1: Each participating node extracts features from local multi-source data and fuses them into a unified feature representation. The data is then aggregated into a local prototype vector according to the label category, and a quality score is calculated. The unified feature representation is used to train a local deep learning model to obtain classification prediction and model gradient, and the total number of local samples is counted. Step S2: Each participating node determines the prototype perturbation noise intensity based on the quality score. The noise intensity is inversely proportional to the quality score. The corresponding intensity noise is superimposed on the local prototype vector and uploaded together with the local model gradient to the aggregation server. Step S3: The aggregation server uses the quality score as the weight to calculate the weighted average of the prototype vectors of the same category to obtain the global prototype vector set, and uses the total number of local samples as the weight to calculate the weighted average of the model gradient to obtain the global model gradient, and distributes it to each participating node. Step S4: Each participating node constructs a prototype consistency loss based on the distance between the global and local prototype vectors, and constructs a classification loss based on classification prediction. The training objective is a weighted combination of the classification loss and the prototype consistency loss. The weight of the prototype consistency loss is inversely proportional to the quality score.

2. The multi-source information data fusion processing method based on deep learning according to claim 1, characterized in that, In step S1, the local multi-source data includes at least two modalities from images, text, and sensor time-series data. The extraction and fusion of features into a unified feature representation involves extracting modal features for the image modality using a convolutional network, for the text modality using a text encoder, and for the sensor time-series modality using a time-series coding network. After mapping each modal feature to the same dimension, they are concatenated into a joint feature matrix. The attention score is calculated by the dot product of the learnable query vector and the joint feature matrix and normalized to the attention weight using softmax. The unified feature representation is obtained by weighting and summing the attention weights of each modal feature.

3. The multi-source information data fusion processing method based on deep learning according to claim 1, characterized in that, In step S1, aggregating local prototype vectors by label category and calculating quality scores includes: The local prototype vector of the corresponding label category is obtained by taking the mean of the uniform feature representations of the same label category. The intra-class tightness is the reciprocal of the average distance between each uniform feature representation and the local prototype vector within the same label category. The inter-class separation is the minimum distance between the local prototype vector of the corresponding label category and the local prototype vectors of other label categories. The quality score is calculated by multiplying the sample size of the corresponding label category, the intra-class tightness, and the inter-class separation, and then normalizing the result.

4. The multi-source information data fusion processing method based on deep learning according to claim 1, characterized in that, In step S1, the local deep learning model includes, in sequence: The feature mapping layer and the classification output layer are used to obtain classification predictions. The classification predictions are obtained by nonlinear transformation of the feature mapping layer using a unified feature representation. The classification loss between the classification prediction and the label category of each sample is calculated using cross-entropy loss. The classification loss is used to backpropagate the local deep learning model to obtain the model gradient. The total number of local samples is the sum of the number of samples of each label category in the local model.

5. The multi-source information data fusion processing method based on deep learning according to claim 1, characterized in that, In step S2, the prototype disturbance noise intensity is determined based on the quality score, and the noise intensity is inversely proportional to the quality score: The quality scores of each participating node are normalized to obtain a normalized quality score. The local privacy budget of each participating node is determined by the product of the preset global privacy budget and the normalized quality score. The sensitivity is determined by the component range of the dimension to which the local prototype vector belongs. The standard deviation of the noise is determined by the ratio of the sensitivity to the local privacy budget as the noise intensity. The perturbed prototype vector is obtained by superimposing Gaussian noise with the corresponding standard deviation on the local prototype vector. The perturbed prototype vector and the model gradient are uploaded to the aggregation server.

6. The multi-source information data fusion processing method based on deep learning according to claim 1, characterized in that, In step S3, the global prototype vector set is obtained by weighting the prototype vectors of the same category with quality scores as weights: The aggregation server takes prototype vectors from participating nodes that belong to the same tag category, uses the normalized quality score of the tag category of each participating node as the aggregation weight, and performs a weighted average of each prototype vector to obtain a global prototype vector set for the corresponding tag category. The global prototype vector sets of each tag category are then concatenated according to the tag category to obtain a global prototype vector set.

7. The multi-source information data fusion processing method based on deep learning according to claim 1, characterized in that, In step S3, the global model gradient is obtained by weighting the model gradient using the total local sample size as the weight: The aggregation server uses the normalized value of the total local sample volume of each participating node as the gradient aggregation weight, and calculates the weighted average of the model gradients uploaded by each participating node to obtain the global model gradient. The global prototype vector set and the global model gradient are then distributed to each participating node.

8. The multi-source information data fusion processing method based on deep learning according to claim 1, characterized in that, In step S4, the prototype consistency loss is constructed by the distance between the global and local prototype vectors, which is the square of the Euclidean distance between the global prototype vector set and the local prototype vector. The classification loss is constructed by the classification prediction, which is the cross-entropy between the classification prediction of the local deep learning model and the corresponding label category.

9. The multi-source information data fusion processing method based on deep learning according to claim 1, characterized in that, In step S4, the weight of the prototype consistency loss is inversely proportional to the quality score: The normalized quality score is used as the weight of the classification loss, and the difference between 1 and the normalized quality score is used as the weight of the prototype consistency loss. The training objective is the product of the classification loss weight and the classification loss plus the product of the prototype consistency loss weight and the prototype consistency loss.

10. A deep learning-based multi-source information data fusion processing system, applied to the deep learning-based multi-source information data fusion processing method as described in any one of claims 1-9, characterized in that, include: The local processing module, located at each participating node, is used to extract features from local multi-source data and fuse them into a unified feature representation, aggregate them into local prototype vectors according to label categories and calculate quality scores, train local deep learning models with the unified feature representation to obtain classification predictions and model gradients, and count the total number of local samples. The noise perturbation module is located at each participating node and is used to determine the prototype perturbation noise intensity based on the quality score. The noise intensity is inversely proportional to the quality score. The corresponding intensity noise is superimposed on the local prototype vector and uploaded together with the local model gradient to the aggregation server. The global aggregation module, located on the aggregation server, is used to calculate the global prototype vector set by weighting the prototype vectors of the same category with the quality score as the weight, and to calculate the global model gradient by weighting the model gradient with the total number of local samples as the weight, and then distribute it to each participating node. The local training module, located on each participating node, is used to construct the prototype consistency loss based on the distance between the global and local prototype vectors, and to construct the classification loss based on classification prediction. The training objective is a weighted combination of the classification loss and the prototype consistency loss, and the weight of the prototype consistency loss is inversely proportional to the quality score.

Images