Multi-device gastritis image federated learning method based on topn queue and weight fft transform

Abstract

The application discloses a multi-device gastritis image federal learning method based on a TopN queue and a weight FFT transformation, a TopN queue and a multi-device aggregation model are designed so that the model can fully utilize cross-device information and historical information of the previous round or even several previous rounds, thereby improving the accuracy of the aggregated model; secondly, the frequency domain information of the model parameters is fused to align the parameters of different clients, thereby effectively reducing the model drift phenomenon caused by the data heterogeneity of different clients; finally, in order to reduce the risk of local data overfitting of scattered clients, the application can effectively reduce the overfitting phenomenon of local clients by means of a local loss regular term based on KL divergence, so that the model is optimized to a global model with better effect.

Description

Technical Field

[0001] This invention relates to the field of image recognition, specifically to a multi-device federated learning method for gastritis images based on TopN queues and weighted FFT transform. Background Technology

[0002] Biometric identification has a wide range of applications in daily life. Stomach cancer is one of the three leading causes of cancer death worldwide, and the diagnosis of gastritis is crucial for the early detection of stomach cancer. Deep learning combined with the medical field can help advance smart healthcare; therefore, to realize the deep application of deep learning in gastritis images, a large number of relevant images should first be collected. However, the collection of gastritis images currently faces the following three difficulties:

[0003] 1) When annotating medical images related to gastritis, it is necessary to combine expert opinions and biopsy results for judgment. Therefore, the annotation of gastritis datasets is difficult, and it is difficult for a single hospital to collect a large number of related images.

[0004] 2) Acquiring a large amount of labeled data by combining data from multiple dispersed institutions is a traditional data collection method. However, in the field of gastritis medicine, image data contains relevant patients' private information, making it difficult to share directly. On the other hand, gastritis data distributed in different hospitals have heterogeneous characteristics (i.e., non-IID distribution). Previous studies have found that data heterogeneity can cause the aggregation model to drift away from the theoretical optimum.

[0005] 3) Gastritis detection involves various imaging devices. Currently, there are two mainstream gastroscopy technologies: white light endoscopy (WLI) and linked color imaging (LCI) endoscopy. How to efficiently combine data from different endoscopes to effectively compensate for the scarcity of gastroscopy image data is a challenge, but relevant research is lacking. Therefore, this invention, based on images collected by multiple WLI and LCI endoscopes, further designs a federated learning algorithm for intelligent gastritis detection, promoting the further development and implementation of scenarios where multiple hospitals collaborate on developing intelligent gastritis detection models.

[0006] Based on existing federated learning algorithms for medical images, firstly, previous studies, addressing the problem of imbalanced data distribution (i.e., Non-IID), only aggregated training information from the current round and information from the same device, without utilizing historically high-performing models or models from various different devices. Secondly, previous studies only considered the information distribution of the original model parameters in the original domain, without exploring the frequency domain information distribution of the model parameters. Finally, previous models did not incorporate regularization information during client-side training, preventing the client-side model from overfitting to local data. Summary of the Invention

[0007] To address the aforementioned shortcomings in existing technologies, this invention provides a multi-device federated learning method for gastritis images based on TopN queues and weighted FFT transform.

[0008] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0009] A multi-device federated learning method for gastritis images based on TopN queues and weighted FFT transform includes the following steps:

[0010] S1. Construct an aggregation model based on TopN queues and multi-device information and train the model to obtain a server model consisting of N best historical training models and N client models in the current round.

[0011] S2. Use weighted FFT transformation to align the parameters of the client model to obtain the feature-aligned client model;

[0012] S3. Optimize the client-side model by incorporating the KL divergence between the global model feature vector and the local model feature vector into the loss function calculation of the client-side model to reduce overfitting.

[0013] Furthermore, step S1 specifically includes the following steps:

[0014] S11. Record the N historically trained models to form a historical model set, denoted as:

[0015] Top N ={M top1 M top2 ,…,M topN}

[0016] Among them, M topN The top N models are the best trained models in history, and topN is the corresponding model number;

[0017] S12. After each client model update, compare it with the last one in the historical model set to determine whether the performance of the updated client model on the IID data is better than the last one in the historical model set. If so, use the updated client model to update the last one in the historical model set. If not, discard the updated client model and wait for the next update.

[0018] S13. Merge the current N client models with the updated historical model set to obtain the server model.

[0019] Furthermore, the server model is represented as follows:

[0020] M server =∑Top N +(1-)∑(×CurN ),λ∈(0,1)

[0021] Among them, M server For the server-side model, Top N A collection of historical models, Cur N The current round has N client models, where λ is the coefficient term.

[0022] Furthermore, step S2 specifically includes the following steps:

[0023] S21. Convert the parameter matrix of a CNN layer in the client model into a two-dimensional parameter matrix;

[0024] S22. Use Fourier transform to transform the two-dimensional parameter matrix obtained in S21 to the frequency domain space;

[0025] S23. Perform low-pass filtering on the two-dimensional parameter matrix transformed to the frequency domain to align the heterogeneous characteristic parameters of the client model.

[0026] S24. Use control parameters to control the low-pass filter range and align the amplitude frequency component and phase frequency component of the i-th client model respectively.

[0027] S25. Perform inverse Fourier transform on the amplitude and phase frequency components obtained in S24 to obtain the feature-aligned client model.

[0028] Furthermore, the specific method for transforming the two-dimensional parameter matrix obtained in S21 to the frequency domain using Fourier transform in S22 is as follows:

[0029]

[0030] in, R is the real number domain space, I and O are the number of input and output channels, h1 and h2 are the kernel space sizes, x and y are the parameter domain space variables, and m and n are the frequency domain space variables.

[0031] Furthermore, the specific method of low-frequency filtering in S23 is as follows:

[0032]

[0033] Where I and O are the number of input and output channels, h1 and h2 are the size of the convolution kernel space, r1 is the low-pass radius, and m and n are the frequency domain space variables.

[0034] Furthermore, the specific method for aligning the amplitude and phase frequency components of the i-th client model in S24 is as follows:

[0035]

[0036]

[0037] Among them, F A (w k )F P (w k ) represent the amplitude and phase frequency components, respectively; M is the aforementioned low-pass filter matrix; and W is the corresponding weighting coefficient.

[0038] Furthermore, the specific method of inverse Fourier transform in S25 is as follows:

[0039] w k =F -1 (|F A (w k ),F P (w k )|)

[0040] Furthermore, the loss function in S3 is expressed as:

[0041] l = CrossEmtroyLoss(y pred ,y label )+μKl(v glob ,v local )

[0042] CrossEntroyLoss is the traditional method for calculating cross-entropy, where y pred ,y label Let v be the predicted label and the true label, respectively. μ is the coefficient of the regularization term, where μ ∈ (0,1). Kl is the relative entropy calculation, i.e., the KL divergence. glob v local These are the feature vectors obtained from the global model and the feature vectors obtained from the local client model, respectively.

[0043] The present invention has the following beneficial effects:

[0044] 1) Effectively utilize information from multi-device endoscopes and improve the aggregation efficiency of federated learning by using TopN queues and multi-device models.

[0045] 2) It can make full use of the frequency domain information of the model parameters and perform a weighted frequency domain feature alignment method, which reduces the drift phenomenon of the local model and makes the aggregated model more robust on the Non-IID data distribution.

[0046] 3) Global model correction information was added to the optimization target loss term on the local client, which reduced the overfitting phenomenon of the model on local Non-IID data.

[0047] 4) By combining federated learning algorithms and a sufficient standard labeled dataset of gastritis atrophy and intestinal metaplasia, a real-time and accurate intelligent computer image recognition system for atrophy and intestinal metaplasia is realized. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the multi-device gastritis federated learning method based on TopN queues and weighted FFT transformation.

[0049] Figure 2 This is a schematic diagram of the multi-device aggregation model structure in an embodiment of the present invention.

[0050] Figure 3 This is a schematic diagram of the local loss correction method according to an embodiment of the present invention.

[0051] Figure 4 This is a schematic diagram illustrating the distribution of Non-IID data used in embodiments of the present invention. Detailed Implementation

[0052] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0053] A multi-device federated learning method for gastritis images based on TopN queues and weighted FFT transform, such as Figure 1 As shown, it includes the following steps:

[0054] S1. Construct an aggregation model based on TopN queues and multi-device information and train the model to obtain a server model consisting of N best historical training models and N client models in the current round.

[0055] WLI and LCI optical devices share imaging similarities, and previous studies have shown that dual transfer between white light and LCI optical devices can effectively improve model performance. Therefore, in federated learning, models using both white light and LCI optical devices can compensate for data scarcity and thus improve model performance.

[0056]

[0057] Where ω represents the weight factor of the model.

[0058] like Figure 2As shown, in our experimental scenario, each hospital simultaneously employs both traditional white light equipment and advanced LCI (Liquid Crystal Interference) equipment. The model's global feature layer is aggregated as a whole. Experimental results verify that this aggregation method is superior to aggregation using client-side models within the devices. Furthermore, previous federated learning training processes exhibited a regression phenomenon, primarily due to the forgetting of global information caused by Non-IID client data. Therefore, we corrected this regression by introducing historical training information.

[0059] Specifically, it includes the following steps:

[0060] S11. Record the N historically trained models to form a historical model set, denoted as:

[0061] Top N ={M top1 M top2 ,…,M topN}

[0062] Among them, M topN These are the N best-performing models in the historical training, with topN being the corresponding model numbers.

[0063] S12. Compare the updated client-side model with the last model in the historical model set after each update. Determine if the updated client-side model performs better on the IID data than the last model in the historical model set. If so, update the last model in the historical model set with the updated client-side model; otherwise, discard the updated client-side model and wait for the next update. After each client-side model update, compare it with Top... N The model is compared to the last-ranked record in the set. If the model outperforms the last-ranked record on the IID data, the model for that record is replaced to maintain the Top ranking. N The advanced nature of the set.

[0064] S13. Merge the current N client models with the updated historical model set to obtain the server model.

[0065] To maintain the timeliness of aggregated information, the model parameter information of the N clients in the current round is represented as follows: The final server-side model was developed by Top. N Sets and Cur N The collection and integration yields:

[0066]

[0067] Where W is the normalized accuracy of the model on the IID data.

[0068] S2. Use weighted FFT transformation to align the parameters of the client model to obtain the feature-aligned client model;

[0069] In this embodiment, a weighted FFT transform is used to align the parameters of the client-side models. The data on each client exhibits a non-IID distribution. The ordinary FedAvg direct weighted aggregation method ignores the negative impact of client-side models with local feature distributions on the global model aggregated on the server. Therefore, to mitigate the drift phenomenon of the global model, we use a Fourier transform based on neural network parameters to perform global feature alignment on the non-IID clients, specifically including the following steps:

[0070] S21. Convert the parameter matrix of a CNN layer in the client model into a two-dimensional parameter matrix;

[0071] First, let's consider how the parameters of a convolutional CNN layer can transform the matrix into... A two-dimensional parameter matrix.

[0072] S22. Use Fourier transform to transform the two-dimensional parameter matrix obtained in S21 to the frequency domain space;

[0073] The parameter matrix is ​​transformed to the frequency domain using Fourier transform:

[0074]

[0075] S23. Perform low-pass filtering on the two-dimensional parameter matrix transformed to the frequency domain to align the heterogeneous characteristic parameters of the client model.

[0076] In this embodiment, the original network parameters are subjected to low-frequency filtering to align with the Non-IID client model parameters. The distribution of the Non-IID data used is as follows: Figure 3 As shown. The low-pass filter M is characterized as follows:

[0077]

[0078] S24. Use control parameters to control the low-pass filter range and align the amplitude frequency component and phase frequency component of the i-th client model respectively.

[0079] In the low-pass filter function, 'r' controls the range of the low-pass filter. The amplitude and phase components of the i-th local client are aligned using parameters, as follows:

[0080]

[0081]

[0082] The calculation method for W is the same as described above.

[0083] S25. Perform inverse Fourier transform on the amplitude and phase frequency components obtained in S24 to obtain the feature-aligned client model.

[0084] Finally, the feature-aligned client model is obtained through inverse Fourier transform:

[0085] w k = -1 (| A (w k ),F P (w k )|)

[0086] S3. Optimize the client-side model by incorporating the KL divergence between the global model feature vector and the local model feature vector into the loss function calculation of the client-side model to reduce overfitting.

[0087] To address the local client overfitting problem, this invention proposes a local regularization term. By leveraging the advantages of the global model, a global model correction factor is introduced into the local loss calculation. This factor aims to narrow the gap between the global and local models. Using KL divergence, which effectively measures the similarity between two probability distributions, the KL divergence between the global and local model feature vectors is incorporated into the local model's loss calculation as one of the optimization objectives. This effectively reduces local client overfitting, allowing the model to optimize towards the better-performing global model. Figure 4 As shown. The loss is calculated on the local client as follows:

[0088] l = CrossEntroyLoss(y pred y label )+μl con

[0089] in:

[0090] l con =Kl(v glob ,v local ).

[0091] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0094] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0095] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A multi-device federated learning method for gastritis images based on TopN queues and weighted FFT transform, characterized in that, Includes the following steps: S1. Construct an aggregation model based on TopN queues and multi-device information, and train the model to obtain a server-side model composed of N historically best-trained models and N client models in the current round. This includes the following steps: S11. Record the N historically trained models to form a historical model set, denoted as: in, For the N best models trained in history, t opN Assign a model number; S12. After each client model update, compare it with the last one in the historical model set to determine whether the performance of the updated client model on the IID data is better than the last one in the historical model set. If so, use the updated client model to update the last one in the historical model set. If not, discard the updated client model and wait for the next update. S13. Merge the current N client models with the updated historical model set to obtain the server model; S2. Perform parameter alignment on the client model using weighted FFT transformation to obtain the feature-aligned client model. This includes the following steps: S21. Convert the parameter matrix of a CNN layer in the client model into a two-dimensional parameter matrix; S22. Use Fourier transform to transform the two-dimensional parameter matrix obtained in S21 to the frequency domain space; S23. Perform low-pass filtering on the two-dimensional parameter matrix transformed to the frequency domain to align the heterogeneous feature parameters of the client model. S24. Control the low-pass filter range using control parameters, and align them respectively with the... i Amplitude and phase components of a client-side model; S25. Perform inverse Fourier transform on the amplitude frequency component and phase frequency component obtained in S24 to obtain the feature-aligned client model. S3. Optimize the client-side model by incorporating the KL divergence between the global model feature vector and the local model feature vector into the loss function calculation of the client-side model to reduce overfitting.

2. The multi-device federated learning method for gastritis images based on TopN queues and weighted FFT transform according to claim 1, characterized in that, The server model is represented as follows: in, For the server-side model, A collection of historical models. The current round has N client models. This is the coefficient term.

3. The multi-device federated learning method for gastritis images based on TopN queues and weighted FFT transform according to claim 1, characterized in that, The specific method for transforming the two-dimensional parameter matrix obtained in S21 to the frequency domain using Fourier transform in S22 is as follows: in, , R For the real number field, I These represent the number of input and output channels, These represent the kernel space size, x, y These are the parameter domain space variables, m、n These are frequency domain spatial variables.

4. The multi-device federated learning method for gastritis images based on TopN queues and weighted FFT transform according to claim 1, characterized in that, The specific method of low-frequency filtering in S23 is as follows: in, I These represent the number of input and output channels, These represent the kernel space size, Low-pass radius, m、n These are frequency domain spatial variables.

5. The multi-device federated learning method for gastritis images based on TopN queues and weighted FFT transform according to claim 1, characterized in that, The specific method for aligning the amplitude and phase frequency components of the i-th client model in S24 is as follows: in, These are the amplitude and phase frequency components, respectively. The aforementioned low-pass filter matrix, These are the corresponding weighting coefficients.

6. The multi-device federated learning method for gastritis images based on TopN queues and weighted FFT transform according to claim 1, characterized in that, The specific method of inverse Fourier transform in S25 is as follows: 。 7. The multi-device federated learning method for gastritis images based on TopN queues and weighted FFT transform according to claim 1, characterized in that, In S3, the loss function is expressed as: in, The traditional cross-entropy calculation method, These are the predicted labels and the true labels, respectively. The coefficient of the regularization term and , The relative entropy is calculated as the KL divergence. These are the feature vectors obtained from the global model and the feature vectors obtained from the local client model, respectively.

Images