Distributed multi-source image fusion method and system based on privacy protection

Abstract

The present invention belongs to the field of image processing technology, and provides a distributed multi-source image fusion method and system based on privacy protection, including obtaining a global model; training the global model, obtaining local model parameters and sending them to a server for aggregation, obtaining a new global model, and sending the new global model to each client; classifying the clients into clusters according to task categories, training the new global model with specific tasks, obtaining specific task model parameters, aggregating the specific task model parameters in the cluster, and distributing the aggregated parameters to each client in the cluster, updating the global model to obtain a fusion network model; training a detail network model, and adding the outputs of the detail network model and the fusion network model to obtain a fusion result. The present invention can effectively improve the quality and efficiency of multimodal image fusion while ensuring data privacy and security, while reducing the operating cost and energy consumption of the system.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to a distributed multi-source image fusion method and system based on privacy protection. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] In the context of contemporary globalization and informatization, data security and privacy protection have become challenges we all face. Federated learning (FL), as a cutting-edge distributed learning method, has been widely recognized by the scientific community for its unique advantages in data privacy protection. On the one hand, FL allows shared models to be learned from distributed data sources without centralizing or directly exchanging data, thereby effectively preventing data leakage. On the other hand, FL allows devices to train local models in a distributed manner and adjust training methods as needed, which is consistent with the real-world scenario of distributed deployment of multi-source sensors.

[0004] As more and more industries and fields generate large amounts of multi-source image data, how to safely and efficiently fuse this data has become an important issue in scientific research and technological development. From the perspective of basic research, the study of a distributed general fusion framework for multi-source images is not only an important supplement and extension to existing data processing and machine learning theories, but also an in-depth exploration of data privacy protection technology. This research will help promote scientific development in the fields of federated learning, image processing, data encryption, and provide new theoretical foundations and technical methods for solving data security and privacy protection. At the same time, it will also promote the cross-integration of fields such as computer vision and artificial intelligence with the field of data security, and promote the international scientific development trend towards a safer and smarter direction.

[0005] Generally speaking, devices with multi-source sensors adopt a distributed working mode, collect real-world scene data in a decentralized manner, and then collectively transmit multimodal images to the server for fusion model training. The well-trained model is sent back to the device by the server for unified deployment. However, due to different hardware specifications, target scenarios, and fusion tasks across devices, providing customized models increases the costs associated with training, deploying, and maintaining fusion models. A feasible solution is to train a common fusion model for all devices. After receiving the data collected by the device, the server uses a unified training method to train different fusion tasks. The trained common model is then deployed on all devices. However, existing fusion methods focus on centralized collection and training of data, which compromises the privacy of device data.

[0006] As data security has become a prominent research field, several methods have been proposed at home and abroad to solve the data security problem in image fusion. For example, a privacy-preserving medical image fusion framework based on blockchain, BMIF, designs a consensus mechanism to save the computing power of the blockchain, and uses efficient homomorphic encryption to encrypt the training process of the fusion model. There is also an encrypted domain medical image fusion scheme based on affine Hill cipher, which does not increase the storage burden of the image and ensures that the provider's computational workload is very small. A medical image fusion method based on non-subsampled shearlet transform in a secure environment is also proposed. This method uses a random generator to encrypt / decrypt the fused image in a secure environment and tests the results of the fused image. However, the existing fusion methods for protecting sensor data privacy are mainly concentrated in the field of medical image fusion, and lack general privacy protection methods for other modes.

[0007] General image fusion methods can reduce the overhead of training, deploying, and maintaining models on distributed devices, while leveraging cross-task versatility to enhance fusion performance. In the field of general models, there are mainly: FusionDN, which uses adaptive weighting to train the network to complete various fusion tasks. Based on FusionDN, U2Fusion integrates the EWC mechanism to prevent catastrophic forgetting problems across multiple tasks. FusionDN and U2Fusion can achieve comparable performance on most fusion tasks with a single model, and many methods train multiple models by adjusting hyperparameters to adapt to different fusion tasks. SwinFusion uses a scintillation transformer to establish long-range dependencies of images to capture the global information of the image, and trains multiple fusion models suitable for various fusion tasks through a carefully designed texture detail loss. However, although these general methods adopt a unified training rule and adjust hyperparameters to adapt the fusion model to various tasks, they are insufficient in selecting the optimal training method for a specific fusion task. Summary of the invention

[0008] In view of the above defects, the present disclosure proposes a distributed multi-source image fusion method and system based on privacy protection. Based on the federated learning framework, it not only protects the privacy of data, but also makes full use of the local data of each sensor, improving the quality and efficiency of image fusion. It also supports real-time operation to ensure that the optimal fusion effect can be achieved under various operating conditions. By intelligently allocating computing tasks to each node in the network, the burden on the central server is reduced, and the overall scalability and robustness of the system are improved.

[0009] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0010] The first aspect of the present disclosure provides a distributed multi-source image fusion method based on privacy protection, comprising the following steps:

[0011] Get the initialized global model;

[0012] Use local data to train the global model, obtain local model parameters, send the local model parameters to the server, aggregate the local model parameters to obtain a new global model, and send the new global model to each client;

[0013] The clients are classified into clusters according to the task categories, and the new global model is trained with the specific tasks to obtain the specific task model parameters. The specific task model parameters are aggregated in the cluster, and the aggregated parameters are distributed to each client in the cluster. The global model is updated to obtain the fusion network model.

[0014] The detailed network model is trained using local data, and the outputs of the detailed network model and the fusion network model are added together to obtain the fusion result.

[0015] As a further implementation method, the natural image training set MSCOCO is used to train the encoder and dual decoder of the global model to initialize the global model.

[0016] As a further implementation, the global model includes an encoder and a decoder, wherein the decoder is a dual-branch decoder.

[0017] As a further implementation method, the encoder of the global model is trained using local data, and the dual decoder is used to constrain the feature distribution to obtain the encoder parameters of the local model.

[0018] As a further implementation, the detail network model and the fusion network model have the same structure, which consists of an encoder-decoder network consisting of a dense network and four convolutional layers.

[0019] As a further implementation, the detailed network model is stored locally on the client and does not participate in the model upload, download or aggregation process.

[0020] As a further implementation, the local data is multi-source images, including: medical images, infrared and visible light images, multi-exposure images and multi-focus images.

[0021] A second aspect of the present disclosure provides a distributed multi-source image fusion system based on privacy protection, comprising:

[0022] The initialization module is configured to: obtain an initialized global model;

[0023] The task sharing module is configured to: train the global model using local data to obtain local model parameters, send the local model parameters to the server, aggregate the local model parameters to obtain a new global model, and send the new global model to each client;

[0024] The task-specific module is configured to: classify the clients into clusters according to the task categories, train the new global model using the specific tasks to obtain the specific task model parameters, aggregate the specific task model parameters in the cluster, distribute the aggregated parameters to each client in the cluster, and update the global model to obtain the fusion network model;

[0025] The output module is configured to: train the detail network model using local data, add the outputs of the detail network model and the fusion network model to obtain a fusion result.

[0026] The third aspect of the present disclosure provides a medium on which a program is stored. When the program is executed by a processor, the steps in the distributed multi-source image fusion method based on privacy protection described in the first aspect of the present disclosure are implemented.

[0027] The fourth aspect of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the distributed multi-source image fusion method based on privacy protection described in the first aspect of the present disclosure are implemented.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention proposes a distributed multi-source image fusion method and system based on privacy protection, which is implemented based on a federated learning framework and can achieve effective fusion of multiple source sensor data without directly exchanging source data. It not only protects the privacy of the data, but also makes full use of the local data of each sensor, improving the quality and efficiency of image fusion. It solves the problems of data privacy protection and heterogeneous data processing that are not fully considered in traditional multi-source image fusion technology.

[0030] This paper proposes a distributed multi-source image fusion method and system based on privacy protection, which adopts an efficient data encoding and decoding network model, reduces the size of the model, and ensures that even resource-constrained devices can quickly process and participate in the learning process. The lightweight and efficient network model design reduces communication overhead.

[0031] This paper proposes a distributed multi-source image fusion method and system based on privacy protection, which is applicable to sensors of various distributed training fusion models, including medical sensors (MED), infrared and visible light sensors (IV), multi-exposure sensors (ME) and multi-focus sensors (MF). The universal algorithm design enables the data collected by each sensor to play a richer role. Combining the advantages of each task and integrating through algorithm optimization, the conflicts and negative effects that may be caused by different optimization strategies in traditional fusion technology are avoided. This paper is aimed at the distribution problems of realistic sensors and data privacy issues, and has achieved better fusion effects in multiple public multi-source image data sets.

[0032] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0034] Figure 1 is a flow chart of a distributed multi-source image fusion method based on privacy protection disclosed in the present invention;

[0035] Figure 2 is an iterative flow chart of the disclosed algorithm;

[0036] Figure 3 It is the overall structure diagram of the disclosed algorithm;

[0037] Figure 4 is a network model structure diagram of the present disclosure;

[0038] Figure 5 is an example of multi-focus fusion using the fusion method disclosed in the present invention, wherein (a) is a near-focus image, (b) is a far-focus image, and (c) is a fusion image;

[0039] Figure 6 is an example of multi-exposure fusion using the fusion method disclosed herein, wherein (a) is an exposure image, (b) is a low-light image, and (c) is a fusion image;

[0040] Figure 7 is an example of medical image fusion using the fusion method disclosed herein, wherein (a) is an MRI image, (b) is a PET image, and (c) is a fusion image;

[0041] Figure 8This is an example of using the fusion method of the present invention to fuse infrared and visible light images, where (a) is an infrared image, (b) is a visible light image, and (c) is a fusion image. DETAILED DESCRIPTION

[0042] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0043] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0044] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

[0045] Embodiment 1

[0046] like Figure 1-4 As shown, the first embodiment of the present disclosure provides a distributed multi-source image fusion method based on privacy protection, comprising the following steps:

[0047] S1 obtains the initialized global model;

[0048] S2 uses local data to train the global model, obtains local model parameters, sends the local model parameters to the server, aggregates the local model parameters to obtain a new global model, and sends the new global model to each client;

[0049] S3 classifies the clients into clusters according to the task categories, trains the new global model with specific tasks, obtains the specific task model parameters, aggregates the specific task model parameters in the cluster, distributes the aggregated parameters to each client in the cluster, and updates the global model to obtain the fusion network model;

[0050] S4 uses local data to train the detail network model, adds the outputs of the detail network model and the fusion network model to obtain the fusion result.

[0051] In order to achieve privacy-preserving distributed multi-source image fusion, the server is used to initialize and aggregate local models to obtain a global model, which is then distributed to the corresponding sensors. The sensors use local datasets to train the global model in a task-agnostic and task-specific manner, and optimize the local models based on the global model. In addition, based on the trained fusion model, the locally stored detail models are further trained. A privacy-preserving distributed image fusion method is designed for realistic data distribution scenarios. The method is based on the FedAvg algorithm in the Federated Learning framework and consists of the following four parts: data preparation, network design, model training, and loss function.

[0052] Among them, model training includes four stages: server initialization stage, task sharing stage, task specific stage and local specific stage. In the server initialization stage, we use natural images to pre-train the encoder and dual decoder on the server side, which enhances the ability of the encoder to extract deep features from the input image. Subsequently, the parameters of the pre-trained model are evenly distributed to all sensors. In the task sharing stage, the parameters of the dual decoder are determined to control the distribution of the encoder, and the encoder is trained to extract domain important information from the local data of the sensor. In order to improve the ability of the encoder to extract features, the present disclosure aggregates the local encoder parameters obtained from all sensors and then distributes them back to the sensor. In the task-specific stage, the encoder is fixed and the decoder parameters on the sensor side are optimized according to their respective training methods. We aggregate the decoder parameters from the same task to avoid conflicts caused by different optimization methods for different tasks in the task sharing stage.

[0053] The specific implementation steps of this disclosure are as follows:

[0054] First, data preparation. We validated our method in medical image fusion, infrared and visible light image fusion, multi-exposure image fusion, and multi-focus image fusion. The training and test sets of all fusion tasks are from publicly available datasets: MRI and PET images from the Harvard Medical School website are used for medical image fusion tasks; TNO is used for infrared and visible light image fusion tasks; the dataset provided by Cai et al. (2018) (Cai, J., Gu, S., & Zhang, L. (2018). Learning a deep single image contrast enhancer from multi-exposure images. IEEE Transactions on Image Processing, 27(4), 2049–2062.) is used for multi-exposure image fusion tasks; the dataset provided by Nejati et al. (2015) (Multi-focus image fusion using dictionary-based sparse representation. Information Fusion, 25, 72–84.) is used for multi-focus image fusion tasks. In these four image fusion tasks, in order to obtain more training data, we adopted the expansion strategy of cropping and decomposition. Specifically, we cropped infrared and visible light images into 35,810 image blocks of size 128×128 as training sets. For the multi-exposure image fusion task, we selected 300 images of true labels, overexposed images, and underexposed images respectively, and cropped 900 images into 43,893 image block pairs of size 512×512 for training. For the multi-focus image fusion task, we cropped 32,000 image block pairs of size 128×128 for near-focus and far-focus images for training. For medical image fusion, we used MRI and PET medical images, selected 252 pairs of images for operation, and cropped them into 40,824 image blocks of size 128×128.

[0055] At the same time, since we are based on a federated learning model, the initial server-side model training uses 20,000 images selected from the MSCOCO data for training, and the size is uniformly adjusted to 64×64 during training.

[0056] The second is network design. The present invention consists of three network models: the codec of the fusion network ( , ), details network Codec, dual branch decoder The server initial stage and task sharing stage use the encoder and dual-branch decoder network, and the task-specific stage and local-specific stage use the encoder and decoder network.

[0057] To reduce the loss of details due to over-aggregation of parameters and ensure better feature extraction from various images, the encoder of the fusion network uses dense connections to extract features from the input image. To ensure the timeliness of sensor communication, a lightweight network with four convolutional layers is used as the fusion network decoder to reconstruct the image. The fusion network uses group normalization as the normalization layer to reduce the difference in feature distribution between local sensor networks, improve aggregation efficiency, and generate high-quality fused images. The local detail network is used to enhance the fused image with texture and detail information.

[0058] The local detail network consists of an encoder-decoder network consisting of a dense network and four convolutional layers. The local detail network adopts batch normalization as the normalization layer to better adapt to the local training strategy and data distribution of the network. A dual-branch dual decoder consisting of two decoders is used to standardize the feature distribution of the encoder for different task data, and the encoder is trained to retain important features of the input image. The dual decoder adopts batch normalization as its normalization layer and does not participate in the local training and aggregation of the model. In the network structure of the present disclosure, the convolution kernel is uniformly specified as 3×3, and its padding is 1. For the activation function, leakyReLU is the choice of all layers except the last layer, which uses Tanh to produce an output tensor between [-1,1].

[0059] The network model is trained in four stages: server initialization stage, task sharing stage, task specific stage and local specific stage.

[0060] The server initialization phase initializes the global model through the server, which includes two phases: initial training and distributed global model. Initial training is performed on the server side, and then the initialized model parameters are sent to all sensors. In order to accelerate the convergence of the model and save the computing resources of the sensor, we use the natural image training set MSCOCO to train the encoder and dual decoder of the fusion model in the server initialization phase. and Image of and The selected image is used as the input image ( , ) Input fusion encoder .based on Extracted features, dual-branch decoder Reconstruct these features into an output image and . and They are all trainable during the server initialization phase, aiming to train and reconstruct the input image to improve the fusion encoder Feature extraction capability and dual-branch decoder We use the following autoencoding loss function Update during initialization and Parameters:

[0061]

[0062]

[0063]

[0064] The loss function we designed consists of two parts, one is the gradient loss function of the input image , and the other part is the strength loss function ,in and represents the height and width of the image. Therefore, is the total number of pixels in the image. Represents the image gradient, calculated using the Sobel operator, where is the L1 norm, is the square of the F-norm. The penalty parameter is used to adjust the proportion of the gradient loss function and is set to 10.

[0065] like Figure 3 As shown, the task sharing phase includes two phases: training and aggregation. The training phase is performed locally on the sensor in a distributed manner, while the aggregation phase is performed on the server side.

[0066] Training: To compensate for domain differences and extract important features from different task images, we train the fusion encoder in a distributed client-side manner on the sensor. , and use a dual decoder To constrain the feature distribution. We extract multi-source images from the local client dataset and Yes, and use them as Then The features extracted by Reconstruct the output image and . The parameters of are trainable, and The parameters of are fixed. On the one hand, we train Adapt to the local data domain; on the other hand, we Constrain the distribution of feature maps. We use the following To update Parameters.

[0067]

[0068] in, =10 is the penalty parameter used to constrain The proportion of loss.

[0069] No. Round The update rules of the local parameters of each sensor are as follows:

[0070]

[0071] Indicates During the round of testing Sensor parameter, Indicates During the round of testing Sensor parameter, is the learning rate. Subtraction is used because we want to adjust the parameters in the opposite direction of the gradient of the loss function (the fastest rising direction), so that the value of the loss function decreases.

[0072] Aggregation: In the model aggregation phase, the server collects locally trained , and aggregate its model parameters according to the FedAvg algorithm. The present invention uses data from all participating sensors to jointly train By aggregating local parameters, we can obtain wheel The global parameters of , the aggregation equation is as follows:

[0073]

[0074] in, for wheel The global parameters of is the total number of local sensors participating in the distributed training in the present invention, For the During the round of testing Sensor Parameters. After aggregation, the parameters are The global model is distributed to the individual sensors involved in the training.

[0075] The task-specific phase also includes two phases: task-specific training and task-specific aggregation. The training phase is performed in a distributed manner at the sensor end.

[0076] Training: Different from the unified training method adopted by general fusion models, federated learning allows customers to train their local models separately. Therefore, we design unique fusion losses for various fusion tasks, enabling each task to achieve state-of-the-art performance and classify sensors into clusters according to their task categories. Sensors are first trained with parameters The aggregate encoder extracts the features of local data and then uses the parameters of the previous round distribution as The decoder reconstructs the image We use the sensor-specific task loss function Fusion to update local parameters. No. Sensors:

[0077]

[0078] in, Indicates the cluster in the tth round Middle Local model of a sensor Parameters. For the Clustering during rounds Middle Local model of a sensor Parameters. is the learning rate of the optimizer Admm, set the value to 0.01.

[0079] Aggregation: For different fusion tasks with different optimization objectives, it is crucial to solve the parameter aggregation conflict problem caused by different parameter optimization directions between tasks. We aggregate decoders in clusters to coordinate the optimization work across tasks. Sensors belonging to cluster k aggregate their models according to the following formula Parameters:

[0080]

[0081] in Indicates Cluster in round Global Model Parameters, Indicates the currently aggregated cluster. is the fusion task set, Represents a cluster Next, set the parameter Assign to cluster A single sensor in.

[0082] In the local specific phase, the local specific phase involves training the detail network stored locally on the sensor. The data training is performed locally and does not involve communication with the server. In order to recover the detail information lost due to parameter aggregation, we build a detail network to supplement the results of the fusion network with texture and detail information. The detail network is stored locally on the sensor and does not participate in the model upload, download or aggregation process. We use the local data from the jth sensor as the input of the fusion network and the detail network respectively, and add their outputs to obtain the local fusion result We use cluster Corresponding detailed loss function Update detailed network The parameters of sensor j belong to,

[0083]

[0084] in, represents the detailed network parameters of the jth sensor in cluster k in round t, represents the detailed network parameters stored locally on the jth sensor in the previous round, Set the value to 0.01.

[0085] Since we use a unified model framework, in the specific task stage, infrared and visible image fusion and medical image fusion are trained in an unsupervised manner, and the loss function The definition is as follows:

[0086]

[0087] in, Represents visible light or MRI images, represents infrared or PET images, represents the fused output, (·) indicates the operation of taking a larger pixel value. Multi-focus image fusion and multi-exposure image fusion, The definition is as follows:

[0088]

[0089] in, represents the true label, (·) indicates structural similarity calculation, Represents the penalty parameter, and its value is set to 10.

[0090] In the specific task stage, for infrared image and visible image fusion and medical image fusion, we force the detail network to extract texture information from visible images or MRI images and extract saliency information from infrared images or PET images. The definition of the detail loss is as follows:

[0091]

[0092]

[0093]

[0094] represents the strength constraint loss, represents the gradient constrained loss, where and Represents the height and width of the image, so, is the total number of pixels in the image, is the L1 norm. The penalty parameter is used to adjust the proportion of the gradient loss function and is set to 100. For the L1 norm constraint ratio, set the value to 10.

[0095] For multi-focus image fusion and multi-exposure image fusion, we force the detail network to learn the texture details and pixel intensity of the true label information. The relevant detail loss is defined as follows:

[0096]

[0097] in, is the square of the F norm, The penalty parameter is used to adjust the proportion of the gradient loss function and is set to 10.

[0098] Our method can be used in conjunction with other federated algorithms, including other advanced models such as FedProx, FedOpt, or Transformer in FL, Graph Neural Networks (GNNs) in FL models, or some more advanced high-order methods can be used to further improve the method described in the present invention.

[0099] like Figure 5-8As shown in the figure, four specific examples of image fusion using the distributed multi-source image fusion method based on privacy protection of the present invention are given. In smart city applications, the present invention can help city managers achieve more efficient city management and service optimization by integrating data from multiple systems such as transportation, environmental monitoring and public safety. In intelligent transportation systems, the present method can be used to integrate data from multiple sensors such as vehicle-mounted cameras, radars, optical sensors, etc. This fusion helps to improve the environmental perception ability of vehicles and promote the development of safer autonomous driving technology. In the field of telemedicine and health monitoring, the present invention can be used to integrate image data from different medical devices (such as MRI, CT scans and X-rays) to help doctors make more accurate diagnoses without leaking patients' privacy information. Systems used for security and monitoring can use the present invention to fuse video streams from multiple surveillance cameras to achieve more comprehensive monitoring coverage while protecting the data privacy of the monitored area. In summary, the present invention will be applied at multiple levels such as smart cities, intelligent transportation, medical auxiliary diagnosis, and security monitoring.

[0100] Embodiment 2

[0101] Embodiment 2 of the present disclosure provides a distributed multi-source image fusion system based on privacy protection, including:

[0102] The initialization module is configured to: obtain an initialized global model;

[0103] The task sharing module is configured to: train the global model using local data to obtain local model parameters, send the local model parameters to the server, aggregate the local model parameters to obtain a new global model, and send the new global model to each client;

[0104] The task-specific module is configured to: classify the clients into clusters according to the task categories, train the new global model using the specific tasks to obtain the specific task model parameters, aggregate the specific task model parameters in the cluster, distribute the aggregated parameters to each client in the cluster, and update the global model to obtain the fusion network model;

[0105] The output module is configured to: train the detail network model using local data, add the outputs of the detail network model and the fusion network model to obtain a fusion result.

[0106] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0107] Embodiment 3

[0108] Embodiment 3 of the present disclosure provides a medium on which a program is stored. When the program is executed by a processor, the steps in the distributed multi-source image fusion method based on privacy protection as described in Embodiment 1 of the present disclosure are implemented.

[0109] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0110] Embodiment 4

[0111] Embodiment 4 of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the distributed multi-source image fusion method based on privacy protection as described in Embodiment 1 of the present disclosure are implemented.

[0112] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0113] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A distributed multi-source image fusion method based on privacy protection, characterized in that: The following steps are involved: Get the initialized global model; Use local data to train the global model, obtain local model parameters, send the local model parameters to the server, aggregate the local model parameters to obtain a new global model, and send the new global model to each client; The clients are classified into clusters according to the task categories, and the new global model is trained with the specific tasks to obtain the specific task model parameters. The specific task model parameters are aggregated in the cluster, and the aggregated parameters are distributed to each client in the cluster. The global model is updated to obtain the fusion network model. Use local data to train the detail network model, add the outputs of the detail network model and the fusion network model to obtain the fusion result; The global model includes an encoder and a decoder, wherein the decoder is a dual-branch decoder; The encoder of the global model is trained using local data, and the dual decoder is used to constrain the feature distribution to obtain the encoder parameters of the local model. The detailed network model is stored locally on the client and does not participate in the model upload, download, or aggregation process.

2. The distributed multi-source image fusion method based on privacy protection as described in claim 1, characterized in that: The natural image training set MSCOCO is used to train the encoder and dual decoder of the global model and initialize the global model.

3. The distributed multi-source image fusion method based on privacy protection as claimed in claim 1, characterized in that: The structures of the detail network model and the fusion network model are the same, both consisting of a dense network and an encoder-decoder network consisting of four convolutional layers.

4. The distributed multi-source image fusion method based on privacy protection as claimed in claim 1, characterized in that: The local data are multi-source images, including: medical images, infrared and visible light images, multi-exposure images and multi-focus images.

5. A distributed multi-source image fusion system based on privacy protection, characterized in that: include: The initialization module is configured to: obtain an initialized global model; The task sharing module is configured to: train the global model using local data to obtain local model parameters, send the local model parameters to the server, aggregate the local model parameters to obtain a new global model, and send the new global model to each client; The task-specific module is configured to: classify the clients into clusters according to the task categories, train the new global model using the specific tasks to obtain the specific task model parameters, aggregate the specific task model parameters in the cluster, distribute the aggregated parameters to each client in the cluster, and update the global model to obtain the fusion network model; The output module is configured to: train the detail network model using local data, add the outputs of the detail network model and the fusion network model to obtain a fusion result; The global model includes an encoder and a decoder, wherein the decoder is a dual-branch decoder; The encoder of the global model is trained using local data, and the dual decoder is used to constrain the feature distribution to obtain the encoder parameters of the local model. The detailed network model is stored locally on the client and does not participate in the model upload, download, or aggregation process.

6. A medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the distributed multi-source image fusion method based on privacy protection as described in any one of claims 1 to 4 are implemented.

7. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the distributed multi-source image fusion method based on privacy protection as described in any one of claims 1 to 4 are implemented.

Images