Colon polyp semantic segmentation deep learning method based on federal learning privacy protection

Through federated learning and privacy protection technology, the problem of data sharing difficulties in medical image processing is solved, cross-system deep model training and privacy protection are achieved, and the performance and privacy of the semantic segmentation model of colon polyps are improved.

CN118984228APending Publication Date: 2024-11-19CHINA ELECTRONICS STANDARDIZATION INST +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410414129.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-04-08
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In medical image processing, due to the confidentiality limitation of patient data, it is difficult for hospitals to share data, resulting in insufficient training data in deep learning models in colon polyps semantic segmentation, affecting the accuracy of the model.

Method used

Adopting a privacy protection method based on federated learning, through random encryption algorithm pool and differential privacy mechanism, encrypted data transmission and parameter aggregation between the client and the server side is realized to ensure data privacy, while crossing without sharing data System deep model training.

Benefits of technology

The overall performance of the colon polyp semantic segmentation model is improved, the privacy of client data is ensured, and the accuracy is similar to the original model, while avoiding the risk of data sharing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118984228A_ABST
    Figure CN118984228A_ABST
Patent Text Reader

Abstract

The invention discloses a colon polyp semantic segmentation deep learning method based on federated learning privacy protection, and particularly relates to the field of federated learning privacy protection and medical semantic segmentation, and the method comprises the following steps: S1, carrying out the data preprocessing of a Kvasair-SEG data set and a CVC-ClinicDB data set, and carrying out the one-hot coding of a real image in the data set; s2, constructing a federal learning framework, initializing a random encryption algorithm pool, randomly selecting the number of clients, and respectively deploying client and server side models; s3, loading the original picture into Dataloader, setting hyper-parameters, and sending the hyper-parameters into a client model for learning; the method provides a new thought for the development of deep learning and privacy protection in the field of medical image processing, and through verification, the actual effect of a federal learning privacy protection model can reach the accuracy of an original model, and better privacy protection can be achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of privacy protection of federated learning and medical semantic segmentation, and more specifically, the present invention relates to a deep learning method for semantic segmentation of colon polyps based on privacy protection of federated learning. Background Art

[0002] With the rapid development of artificial intelligence technology, all walks of life are introducing artificial intelligence technology, such as autonomous driving, image generation, AR / VR, medical image processing and other fields. Among them, the development momentum of medical image processing is very rapid. Most of the current symptom analysis is assisted by deep learning technology to roughly determine the cause of the disease and the location of the disease, and finally the doctor makes a diagnosis. However, since the patient's medical data is confidential information, various hospitals cannot share information, which will lead to a lack of samples during deep learning training. In deep learning, the lack of training data will directly lead to a decrease in model accuracy. Accuracy is particularly important in the field of medical image processing and is related to the patient's life safety. Therefore, the present invention proposes a deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection, in order to solve the above problems. Summary of the invention

[0003] To achieve the above object, the present invention provides the following technical solutions:

[0004] The deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection includes the following steps:

[0005] S1. Preprocess the Kvasir-SEG dataset and the CVC-ClinicDB dataset, and then perform one-hot encoding on the real images in the dataset.

[0006] S2. Build a federated learning framework, initialize a random encryption algorithm pool, randomly select the number of clients, and deploy client and server models respectively;

[0007] S3, load the original image into Dataloader, set hyperparameters, and send it to the client model for learning;

[0008] S4, perform differential privacy and homomorphic encryption on the client parameters, encrypt the client information using Hash encoding, and send the client parameters and client information to the server;

[0009] S5. The server receives the parameters and aggregates them. After the aggregation is completed, the parameters are encrypted again and sent to the client.

[0010] S6: The client receives the parameter information sent down and stops iterating if the target iteration effect is achieved, otherwise it continues iterating;

[0011] S7. The client performs inference prediction on colon polyp segmentation and visualizes the results.

[0012] In a preferred embodiment, in S2, the processing process uses random preprocessing including affine transformation of data and color contrast enhancement.

[0013] In a preferred embodiment, the specific steps in S1 are: using open source colon polyp datasets, namely Kvasir-SEG and CVC-ClinicDB public datasets, dividing the data in the datasets into two categories: original images and annotated images, the total amount of Kvasir-SEG dataset is 1000 images, the total amount of CVC-ClinicDB dataset is 612 images, and the division ratio of the data volume training set, validation set and test set is 8:1:1;

[0014] The original image of the dataset is reshaped into a shape of (352, 352, 3), and the shape of the annotated image becomes (352, 352, 2). The third dimension is the category information, which contains the background and target areas respectively. Data augmentation operations are performed on the original image and the annotated image.

[0015] The specific steps in S2 are as follows:

[0016] S21. To build a federated learning framework, you need to deploy server-side and client-side models separately. The client and server use the same model. The server randomly selects any number of users from the clients that request to join to join the learning system.

[0017] S22, initialize the random encryption algorithm pool, and select the encryption algorithm and differential privacy mechanism to be used from the algorithm pool.

[0018] In a preferred embodiment, the training steps in S3 are as follows:

[0019] S31, use RGB image patches of size 352×352×3 cropped from the dataset for training;

[0020] S32. Select the AdamW optimizer, set the batch size to 4, set the initial learning rate to 1e-4, set the learning rate self-adjustment strategy, and use the Dice supervision method. If Dice does not improve within ten rounds, actively reduce the learning rate by half. Set the early stopping strategy. When the learning rate is 1e-6 and Dice does not improve within ten rounds, stop training.

[0021] S33. The loss function uses a joint loss function for training. This function includes two loss functions, WIoU and WBCE. The real image and the predicted image are used as inputs to the loss function. Before input, the real image is one-hot encoded, and the categories are merged into the second dimension. A category-based mask is made for the entire binary image. The formula for Loss is:

[0022]

[0023] in and Represent weighted IoU loss and weighted BCE loss respectively, y, p represent ground truth and predicted image respectively, i represents the number of layers of the feature pyramid, and the loss function is calculated for the output of each layer and summarized as the total loss during training.

[0024] In a preferred embodiment, the encryption process in S4 is:

[0025] S41. Perform differential privacy on the parameters trained by the client, that is, add noise. Differential privacy is defined as follows: for a random algorithm M, P M is the set of all values ​​that can be output by algorithm M, if for any pair of adjacent data sets X and X′, P M For any subset S of , the algorithm satisfies:

[0026] P r [M(x)∈S]≤e ε P r [M(x′)∈S]

[0027] Then the algorithm M is said to satisfy ε-differential privacy, where the parameter ε is the privacy protection budget. The smaller ε is, the higher the privacy protection degree is; the larger ε is, the higher the data availability is; when ε = 0, the output probability of M for the two data sets is exactly the same;

[0028] The two different differential privacy mechanisms are similar and are defined as follows:

[0029] Given a function f:N |x| →R k , then the Laplace mechanism is defined as:

[0030]

[0031] where Y i is an independent and identically distributed random variable that conforms to the Laplace distribution or Gaussian distribution, x is the queried data, f(x) is the query function, the output is a k-dimensional vector representing the result, and ε is the privacy budget;

[0032] S42. The data with differential privacy added is homomorphically encrypted. The encryption algorithm pool includes RSA, Paillier and other algorithms. The key is generated and issued by the server.

[0033] In a preferred embodiment, the deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection also includes evaluating the colon polyp segmentation inference prediction results and the actual true value of the client to obtain a deviation evaluation value, and then comparing the deviation evaluation value with a preset comparison threshold. If the deviation evaluation value is greater than or equal to the preset comparison threshold, a first-class feedback signal is issued, otherwise a second-class feedback signal is issued.

[0034] Technical effects and advantages of the present invention:

[0035] The present invention introduces a federated learning framework in the field of semantic segmentation of colon polyps, improves the overall performance of the model, and provides a case for cross-system training of the semantic segmentation model of colon polyps. The random encryption pool technology is adopted to randomly select the homomorphic encryption algorithm and the differential privacy mechanism at the beginning of training. The client source encryption technology is adopted to perform hash encoding on the client source information, which facilitates the rapid search of the client while also encrypting the client source.

[0036] This invention introduces privacy protection measures in the field of colon polyp segmentation. It uses a federated learning framework to achieve cross-system deep model training without sharing data. Secondly, it uses a random encryption algorithm pool to add a combination of differential privacy and homomorphic encryption technology, which greatly improves the privacy of data. It constructs a client source encryption method to ensure that the client source cannot be traced and protect the privacy of the client.

[0037] The traditional federated learning algorithm uses a single encryption structure, which still has a certain risk of being cracked. The new structure proposed in the present invention can realize a combination of multiple encryption methods, which is more difficult to be cracked. The method proposed in the present invention provides a new idea for the development of deep learning and privacy protection in the field of medical image processing, and it has been verified that the actual effect of the model using federated learning privacy protection can achieve the accuracy of the original model, and can achieve better privacy protection. While ensuring that each medical system operates independently without sharing data information, it also achieves joint modeling and improves model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings;

[0039] Figure 1 This is a method block diagram of the deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection in the present invention.

[0040] Figure 2 This is the overall structure diagram of the neural network model in Example 3 of the present invention.

[0041] Figure 3 This is a flowchart of the simplified training of the client in the present invention.

[0042] Figure 4 Visualization results of experimental data in this invention Figure 1 .

[0043] Figure 5 Visualization results of experimental data in this invention Figure 2 .

[0044] Figure 6 This is Table 1 in the present invention.

[0045] Figure 7 This is Table 2 in the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Reference Figure 1 - Figure 7 The following embodiments are obtained:

[0048] Example 1

[0049] The deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection includes the following steps:

[0050] S1. Preprocess the Kvasir-SEG dataset and the CVC-ClinicDB dataset, and then perform one-hot encoding on the real images in the dataset.

[0051] S2. Build a federated learning framework, initialize a random encryption algorithm pool, randomly select the number of clients, and deploy client and server models respectively;

[0052] S3, load the original image into Dataloader, set hyperparameters, and send it to the client model for learning;

[0053] S4, perform differential privacy and homomorphic encryption on the client parameters, encrypt the client information using Hash encoding, and send the client parameters and client information to the server;

[0054] S5. The server receives the parameters and aggregates them. After the aggregation is completed, the parameters are encrypted again and sent to the client.

[0055] S6: The client receives the parameter information sent down and stops iterating if the target iteration effect is achieved, otherwise it continues iterating;

[0056] S7. The client performs inference prediction on colon polyp segmentation and visualizes the results.

[0057] In S2, the processing process uses random preprocessing including affine transformation of data and color contrast enhancement to increase data diversity and alleviate the overfitting problem caused by limited data volume.

[0058] The specific steps in S1 are: using the open source colon polyp datasets, namely Kvasir-SEG and CVC-ClinicDB public datasets, the data in the datasets are divided into two categories: original images and annotated images. The total number of Kvasir-SEG datasets is 1,000 images, and the total number of CVC-ClinicDB datasets is 612 images. This is because the data volume training set, validation set, and test set are divided in a ratio of 8:1:1;

[0059] The original image of the dataset is reshaped into a shape of (352, 352, 3), and the shape of the annotated image becomes (352, 352, 2). The third dimension is the category information, which contains the background and target areas respectively. Data augmentation operations are performed on the original image and the annotated image.

[0060] The specific steps in S2 are as follows:

[0061] S21. To build a federated learning framework, you need to deploy server-side and client-side models separately. The client and server use the same model. The server randomly selects any number of users from the clients that request to join to join the learning system.

[0062] S22, initialize the random encryption algorithm pool, and select the encryption algorithm and differential privacy mechanism to be used from the algorithm pool.

[0063] The training steps in S3 are as follows:

[0064] S31, use RGB image patches of size 352×352×3 cropped from the dataset for training;

[0065] S32. Select the AdamW optimizer, set the batch size to 4, set the initial learning rate to 1e-4, set the learning rate self-adjustment strategy, and use the Dice supervision method. If Dice does not improve within ten rounds, actively reduce the learning rate by half. Set the early stopping strategy. When the learning rate is 1e-6 and Dice does not improve within ten rounds, stop training.

[0066] S33. The loss function uses a joint loss function for training. This function includes two loss functions, WIoU and WBCE. The real image and the predicted image are used as inputs to the loss function. Before input, the real image is one-hot encoded, and the categories are merged into the second dimension. A category-based mask is made for the entire binary image. The formula for Loss is:

[0067]

[0068] in and Represent weighted IoU loss and weighted BCE loss respectively, y, p represent ground truth and predicted image respectively, i represents the number of layers of the feature pyramid, and the loss function is calculated for the output of each layer and summarized as the total loss during training.

[0069] The encryption process in S4 is:

[0070] S41. Perform differential privacy on the parameters trained by the client, that is, add noise. Differential privacy is defined as follows: for a random algorithm M, P M is the set of all values ​​that can be output by algorithm M, if for any pair of adjacent data sets X and X′, P M For any subset S of , the algorithm satisfies:

[0071] P r [M(x)∈S]≤e ε P r [M(x′)∈S]

[0072] Then the algorithm M is said to satisfy ε-differential privacy, where the parameter ε is the privacy protection budget. The smaller ε is, the higher the privacy protection degree is; the larger ε is, the higher the data availability is; when ε = 0, the output probability of M for the two data sets is exactly the same;

[0073] The two different differential privacy mechanisms are similar and are defined as follows:

[0074] Given a function f:N |x| →R k , then the Laplace mechanism is defined as:

[0075]

[0076] where Y i is an independent and identically distributed random variable that conforms to the Laplace distribution or Gaussian distribution, x is the queried data, f(x) is the query function, the output is a k-dimensional vector representing the result, and ε is the privacy budget;

[0077] S42. The data with differential privacy added is homomorphically encrypted. The encryption algorithm pool includes RSA, Paillier and other algorithms. The key is generated and issued by the server.

[0078] Example 2

[0079] The deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection includes the following steps:

[0080] S1. Preprocess the Kvasir-SEG dataset and the CVC-ClinicDB dataset, and then perform one-hot encoding on the real images in the dataset.

[0081] S2. Build a federated learning framework, initialize a random encryption algorithm pool, randomly select the number of clients, and deploy client and server models respectively;

[0082] S3, load the original image into Dataloader, set hyperparameters, and send it to the client model for learning;

[0083] S4, perform differential privacy and homomorphic encryption on the client parameters, encrypt the client information using Hash encoding, and send the client parameters and client information to the server;

[0084] S5. The server receives the parameters and aggregates them. After the aggregation is completed, the parameters are encrypted again and sent to the client.

[0085] S6: The client receives the parameter information sent down and stops iterating if the target iteration effect is achieved, otherwise it continues iterating;

[0086] S7. The client performs inference prediction on colon polyp segmentation and visualizes the results.

[0087] Among them, in S2, the processing process adopts random preprocessing including affine transformation of data and color contrast enhancement.

[0088] In S5, the server receives the parameters from the client, first verifies the client source, decrypts the client code, and then uses the public key to decrypt the parameter information. It then uses the federated equal rights algorithm to aggregate the parameters, and finally encrypts the aggregated parameters again and sends them to the client together with the key.

[0089] In S6, the client receives the model parameters sent by the server and decrypts them first. If the early stopping condition is met, the iteration stops, otherwise the iteration continues.

[0090] In S7, the trained client can be directly used for inference prediction of colon polyp segmentation and visualization of prediction results.

[0091] The deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection also includes evaluating the inference prediction result and the actual true value of colon polyp segmentation on the client to obtain a deviation evaluation value, and then comparing the deviation evaluation value with a preset comparison threshold. If the deviation evaluation value is greater than or equal to the preset comparison threshold, a first-class feedback signal is issued, otherwise a second-class feedback signal is issued; specifically: mIoU and Dice are both indicators for evaluating the performance of semantic segmentation models. mIoU is obtained by calculating the intersection over union ratio of each category and then calculating the average value of the IoU of all categories. It is a global evaluation indicator for measuring the positioning accuracy of the model for objects of each category. The Dice coefficient is a set similarity measurement function, which is usually used to calculate the similarity of two samples, and its value range is [0,1]. In this context, X-GT represents the real segmentation image, and Y-Pred represents the predicted segmentation image. The Dice coefficient can be understood as the dot product between the predicted segmentation map and the real segmentation map, and then the results of the dot product are added, and finally divided by the sum of the elements of the predicted segmentation map and the real segmentation map. When Dice is equal to 1, it means that the segmentation result is completely consistent with the standard segmentation result; when Dice is equal to 0, it means that the segmentation result is completely inconsistent with the standard segmentation result; the evaluation operation of the colon polyp segmentation inference prediction result and the actual true value on the client refers to obtaining the indicators mIoU and Dice respectively. The deviation evaluation value calculation formula is: PG=A*f1+B*f2, where A represents the indicator mIoU, B represents the indicator Dice, f1+f2=1, and f1 and f2 are both positive numbers, PG is the deviation evaluation value, f1 and f2 respectively represent the degree of influence of the indicators mIoU and Dice on the deviation evaluation value PG, and then the deviation evaluation value is compared with the preset comparison threshold. If the deviation evaluation value is greater than or equal to the preset comparison threshold, a first-class feedback signal is issued, indicating that the deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection has achieved the expected effect; otherwise, a second-class feedback signal is issued, indicating that the deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection has not achieved the expected effect.

[0092] Example 3

[0093] A colon polyp semantic segmentation deep learning method based on federated learning privacy protection can construct a colon polyp semantic segmentation neural network model. The specific steps include:

[0094] I. Data Preprocessing

[0095] Ⅱ. Build a federated learning framework, initialize the random encryption algorithm pool, randomly select the number of clients, and deploy the client and server models respectively.

[0096] III. Load training data, set hyperparameters, and send to client model learning.

[0097] IV. Perform differential privacy and homomorphic encryption on the client parameters, encrypt the client information using Hash encoding, and send the client parameters and client information to the server.

[0098] Ⅴ. The server receives the parameters and aggregates them. After aggregation, the parameters are encrypted again and sent to the client.

[0099] VI. The client receives the parameter information sent down and stops iterating if the target iteration effect is achieved, otherwise it continues iterating.

[0100] VII. The client performs inference prediction for colon polyp segmentation and visualizes the results.

[0101] The overall structure of a neural network model for semantic segmentation of colon polyps is as follows: Figure 2 shown.

[0102] Example 4

[0103] Figure 4 Visualizing results for experimental data Figure 1 That is, a visualization diagram of the test results of the Kvasir-SEG dataset, where DPNet is the model proposed in the present invention, DPNet is a model that uses federated learning, and the experimental data comes from the Kvasir-SEG dataset.

[0104] Figure 5 Visualizing results for experimental data Figure 2 That is, a visualization diagram of the test results of the CVC-ClinicDB dataset, where DPNet is the model proposed in the present invention, DPNet is a model that uses federated learning, and the experimental data comes from the CVC-ClinicDB dataset.

[0105] More specifically: the SPResNet structure is used, which is divided into four layers from bottom to top. The feature map between each layer is reduced by 1 / 2. ResNet50 is used as the backbone network of SPResNet for feature extraction. We introduce the strip pooling module at the end of each residual block to obtain more image information. The use of SPResNet as the feature extraction path is based on the fact that the pyramid structure can obtain different feature information by using a bottom-up structure. The feature image in the low-dimensional space contains a lot of spatial information and object location information. This information helps to determine the size and shape of the segmented object and its specific location inside the image. In the commonly used pyramid structure, the low-dimensional feature map and the upsampled high-dimensional feature map are added at the pixel level to ensure that the detailed features can be effectively restored after layer-by-layer extraction. After multiple feature extractions, the information in the high dimension lacks the perception of object details, but is endowed with sensitivity to object categories. The information contained in the pixel is several times that of the low-dimensional space. The specific category of the object can be judged based on the information contained in the pixel, and irrelevant information can be excluded. In the image segmentation process, the proportion of the segmented area in the image can be effectively judged.

[0106] Traditional global average pooling and maximum and minimum pooling operations can only rely on local windows to obtain image information within the window. This operation can extract information within the window but does not take into account the relationship with neighboring pixels. The operating logic of stripe pooling is similar to the self-attention mechanism and non-local network. The difference is that a strip-shaped pooling area is used instead of the original N×N area, that is, the pooling kernel is replaced with N×1 or 1×N, which can effectively allow the backbone network to effectively capture long-distance dependencies. And because the pooling kernel is narrow and long, it is conducive to capturing contextual relationships and preventing interference from irrelevant areas. This module consists of two pooling parts, namely vertical stripe pooling and horizontal stripe pooling. The two are added together to obtain long-distance image information. The stripe pooling module formula is as follows:

[0107]

[0108]

[0109] The formula above represents the pooling operation in the horizontal direction, and the average pooling operation is performed on the corresponding rows. The h in the equation represents the horizontal direction, i represents the position of the corresponding row, W represents the total number of rows, ∑ 0≤j≤W x i,j Represents the accumulation of the current row of elements. The formula below represents the vertical pooling operation to perform an average pooling operation on the column. The v in the equation represents the vertical direction, j represents the position of the corresponding column, H represents the total number of columns, ∑ 0≤i≤H x i,j Represents the accumulation of the current column number of elements, and effectively avoids the interference of irrelevant information by performing pooling operations in different directions.

[0110] The PVT structure also uses the feature pyramid structure as the main feature extraction structure, and uses PVTv2 as the backbone network, using the pre-trained weights on ImageNet. Compared with other Transformer structures that require the original image to be extracted by CNN to form a feature image, PVT can directly use the original image as input, and convert the image into the image block required by the Transformer encoder by performing image block embedding (Patch Embed) and position encoding (Pos it ion Encode) on the original image. The overall structure is divided into four parts, each of which has two units, namely image block embedding and position encoding. Image block embedding divides an entire original image or extracted feature image into blocks of the same size. Position encoding adds a position code to each image block to indicate the position of the image block in the entire image, which facilitates the network to learn the connection between image blocks. Position encoding uses a learnable position code. By initializing the learnable parameters, the position of the image block is learned during the network training process. The encoder part of the Transformer serializes the input image block through the standard Transformer encoder structure and performs linear mapping. After stage I, the original image is scaled to 1 / 8 of the original size, and then reduced by 1 / 2 in each subsequent stage, and finally reduced to 1 / 32 of the original image in the last layer.

[0111] Since the PVT network downsamples the feature image after each part and scales the image to 1 / 2 of the original, the image block embedding information and position encoding information learned in the previous part are invalid for the new image. Therefore, new image block embedding and position information are required for the downsampled feature map at each stage to ensure the new feature image is divided into blocks and position encoded. The PVT Transformer uses a multi-head self-attention mechanism module to perform attention calculations on each input image block, calculate the correlation between different image blocks, and then obtain the global feature connection of the entire image.

[0112] DPNet as a whole uses SPResNet and PVT to form two upper and lower feature extraction channels. The SPResNet channel is based on CNN, which enables the network to extract local features. This part can well preserve important information such as the details of the segmented area. PVT can extract features from a global perspective and realize the correlation of global features, and can better learn the overall contour of the object and the position information of the segmented area in the image. In view of the different attention paid by the Transformer structure to objects of different sizes in images of different sizes, we use multi-scale image training, so that we can achieve good learning effects on targets of different sizes in an image. Both networks have a four-layer structure. The output images of each layer are of the same size but different dimensions. The output feature maps of the two networks are reduced to the same dimension and then concatenated. The whole is sent to the subsequent up-sampled image restoration. After the link block (CB), the input feature image is reduced in dimension to facilitate pixel-wise addition with the feature map of the lower layer. The CB module embeds group normalization, SiLU activation function and residual connections, which can reduce the information loss problem caused by the dimensionality reduction upsampling of the information merged by the two backbone networks in the concatenate stage.

[0113] In our experiments, we found that high-dimensional semantic information cannot directly get the desired results for segmentation tasks through prediction output, because the semantic information of high-dimensional feature images is extremely rich, and the expansion of the receptive field also compresses the image into a very small range, and the position of the pixel block in the original image can only be judged based on the relative position. However, the information brought by the high-dimensional feature image after continuous fusion of low-dimensional feature images is crucial for the judgment of the segmented area. Therefore, we designed the Aggregate Attention Module (AAM), in which we fuse the high-dimensional semantic information of the highest layers of the two networks, fuse the features extracted by each network, and perform the attention mechanism to obtain the attention points of two different structures. AAM can be written as the following formula:

[0114]

[0115]

[0116] Here y represents the top-level input of SPResNet. Represents the top-level input of PVT, Concate represents the fusion of two feature images according to the dimension, Q, K, and V represent the matrices corresponding to the Queery, Key, and Value of the image after the fusion feature, respectively. Represents the reduction factor.

[0117] The present invention obtains the following form:

[0118] Table 1 Performance comparison of different models on Kvasir-SEG and CVC-ClinicDB datasets;

[0119] Table 1 uses the Dice score and mIIoU indicators for comparison, using the same epochs.

[0120] Table 2 Evaluation data of the DPNet model under different clients and different learning rounds when using the federated learning framework, where Global represents the evaluation data of the server-side model after aggregating parameters.

[0121] The following conclusions were drawn: Privacy protection measures were introduced in the field of colon polyp segmentation. The federated learning framework was used to achieve cross-system deep model training without sharing data. Secondly, the random encryption algorithm pool was used to increase the combination of differential privacy and homomorphic encryption technology, which greatly improved the privacy of the data. A client-source encryption method was constructed to ensure that the client source could not be traced and protect the privacy of the client. The traditional federated learning algorithm uses a single encryption structure and still has a certain risk of being cracked. Our new structure can realize a combination of multiple encryption methods and is more difficult to crack. The method we proposed provides new ideas for the development of deep learning and privacy protection in the field of medical image processing. And after our verification, the actual effect of our model using federated learning privacy protection can achieve the accuracy of the original model and can achieve better privacy protection.

[0122] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0123] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0124] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0125] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0126] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection, characterized in that: The following steps are involved: S1. Preprocess the Kvasir-SEG dataset and the CVC-ClinicDB dataset, and then perform one-hot encoding on the real images in the dataset. S2. Build a federated learning framework, initialize a random encryption algorithm pool, randomly select the number of clients, and deploy client and server models respectively; S3, load the original image into Dataloader, set hyperparameters, and send it to the client model for learning; S4, perform differential privacy and homomorphic encryption on the client parameters, encrypt the client information using Hash encoding, and send the client parameters and client information to the server; S5. The server receives the parameters and aggregates them. After the aggregation is completed, the parameters are encrypted again and sent to the client. S6: The client receives the parameter information sent down and stops iterating if the target iteration effect is achieved, otherwise it continues iterating; S7. The client performs inference prediction on colon polyp segmentation and visualizes the results.

2. The deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection according to claim 1, characterized in that: In S2, the processing adopts random preprocessing including affine transformation of data and color contrast enhancement.

3. The deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection according to claim 2, characterized in that: The specific steps in S1 are: using the open source colon polyp datasets, namely Kvasir-SEG and CVC-ClinicDB public datasets, the data in the datasets are divided into two categories: original images and annotated images. The total number of Kvasir-SEG datasets is 1,000 images, and the total number of CVC-ClinicDB datasets is 612 images. The data volume training set, validation set, and test set are divided in a ratio of 8:1:1; The original image of the dataset is reshaped into a shape of (352, 352, 3), and the shape of the annotated image becomes (352, 352, 2). The third dimension is the category information, which contains the background and target areas respectively. Data augmentation operations are performed on the original image and the annotated image.

4. The deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection according to claim 3, characterized in that: The specific steps in S2 are as follows: S21. To build a federated learning framework, you need to deploy server-side and client-side models separately. The client and server use the same model. The server randomly selects any number of users from the clients that request to join to join the learning system. S22, initialize the random encryption algorithm pool, and select the encryption algorithm and differential privacy mechanism to be used from the algorithm pool.

5. The deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection according to claim 4, characterized in that: The training steps in S3 are as follows: S31, use RGB image patches of size 352×352×3 cropped from the dataset for training; S32. Select the AdamW optimizer, set the batch size to 4, set the initial learning rate to 1e-4, set the learning rate self-adjustment strategy, and use the Dice supervision method. If Dice does not improve within ten rounds, actively reduce the learning rate by half. Set the early stopping strategy. When the learning rate is 1e-6 and Dice does not improve within ten rounds, stop training. S33. The loss function uses a joint loss function for training. This function includes two loss functions, WIoU and WBCE. The real image and the predicted image are used as inputs to the loss function. Before input, the real image is one-hot encoded, and the categories are merged into the second dimension. A category-based mask is made for the entire binary image. The formula for Loss is: in and Represent weighted IoU loss and weighted BCE loss respectively, y, p represent ground truth and predicted image respectively, i represents the number of layers of the feature pyramid, and the loss function is calculated for the output of each layer and summarized as the total loss during training.

6. The deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection according to claim 5, characterized in that: The encryption process in S4 is: S41. Perform differential privacy on the parameters trained by the client, that is, add noise. Differential privacy is defined as follows: for a random algorithm M, P M is the set of all values ​​that can be output by algorithm M, if for any pair of adjacent data sets X and X′, P M For any subset S of , the algorithm satisfies: P r [M(x)∈S]≤e ε P r [M(x′)∈S] Then the algorithm M is said to satisfy ε - Differential privacy, where the parameter ε is the privacy protection budget; The two different differential privacy mechanisms are similar and are defined as follows: Given a function f:N |x| →R k , then the Laplace mechanism is defined as: where Y i is an independent and identically distributed random variable that conforms to the Laplace distribution or Gaussian distribution, x is the queried data, f(x) is the query function, the output is a k-dimensional vector representing the result, and ε is the privacy budget; S42: The data with differential privacy added is homomorphically encrypted, and the server generates and sends the key.

7. The deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection according to claim 6, characterized in that: The deep learning method for semantic segmentation of colon polyps based on federated learning privacy protection also includes evaluating the colon polyp segmentation inference prediction results and actual true values ​​of the client to obtain a deviation evaluation value, and then comparing the deviation evaluation value with a preset comparison threshold. If the deviation evaluation value is greater than or equal to the preset comparison threshold, a first-class feedback signal is issued, otherwise a second-class feedback signal is issued.