Multi-modal federated learning method and device based on image segmentation and storage medium

Abstract

The invention discloses a multi-mode federated learning method and device based on image segmentation and a storage medium, and relates to the technical field of machine learning, and the method comprises the steps: converting an input image of a client from a spatial domain to a frequency domain through discrete cosine transform, thereby obtaining a local intermediate frequency spectrum of the client, and obtaining a local intermediate frequency spectrum of the client; according to the technical scheme, the local intermediate frequency spectrum of each client is aggregated at the server side, and by adopting the scheme, only the intermediate frequency coefficient containing the basic characteristics is transmitted, so that the transmitted data volume can be effectively reduced. Besides, a transmission unit number matrix between a local intermediate frequency spectrum and a global intermediate frequency consensus is calculated through EMD improved by a Sinkhorng algorithm, an optimal transmission scheme can be obtained, the global intermediate frequency consensus is iteratively updated by using the optimal transmission scheme, and a client is further guided to reconstruct structural features and detail information by means of the global intermediate frequency consensus, so that the optimal transmission scheme is obtained. Therefore, light-weight and accurate medical image segmentation of the image segmentation model in multiple modes is enhanced, and the balance of model efficiency and precision is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, and particularly relates to a multi-modal federated learning method based on image segmentation, a device and a storage medium. BACKGROUND

[0002] With the rapid development of multimedia technology in the medical field, federated medical image segmentation technology can show great potential in medical multimedia applications under the premise of not affecting privacy by utilizing rich image data. Existing research has gradually extended federated learning (FL) from single modality to general multi-modal segmentation. Multimodal federated learning (MFL) is committed to integrating distributed multi-modal data from different clients to improve feature representation capability under the premise of protecting user privacy, and has received widespread attention in the field of medical image analysis in recent years.

[0003] However, in order to effectively integrate multi-modal information, MFL still faces many challenges, such as inconsistent cross-modal generalization ability and high communication cost. In related technologies, feature alignment, frequency domain modeling and domain adaptation strategies are used to improve generalization performance, and global-local aggregation strategies and model compression techniques are used to alleviate communication bottlenecks in order to reduce communication overhead.

[0004] However, related technologies regard communication efficiency and model generalization ability as independent optimization targets, ignoring the trade-off relationship between the two, which makes it difficult to balance the model between generalization performance and client resource constraints, resulting in a decline in model performance. However, model performance directly determines the stability and reliability of medical image segmentation in clinical decision-making. Suboptimal or incorrect output may lead to irreversible medical events or adverse events, thereby endangering patient safety and overall clinical outcomes. SUMMARY

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an overall description of the application, nor is it intended to determine key / important components or delineate the scope of protection of these embodiments, but as a prelude to the detailed description below.

[0006] The embodiments of the present application provide a multi-modal federated learning method based on image segmentation, a device and a storage medium, which can effectively improve the generalization ability of multi-modal medical image segmentation tasks while reducing communication overhead.

[0007] In a first aspect, the embodiments of the present application provide a multi-modal federated learning method based on image segmentation, comprising:

[0008] performing discrete cosine transform on an input image of a client to obtain a frequency domain matrix of the input image;

[0009] obtaining a local intermediate frequency spectrum of the client according to the frequency domain matrix of the input image;

[0010] aggregating the local intermediate frequency spectrum of each of the clients at a server side to obtain a global intermediate frequency consensus;

[0011] calculating a transmission unit quantity matrix between the local intermediate frequency spectrum and the global intermediate frequency consensus by using a Sinkhorn algorithm, and iteratively updating the global intermediate frequency consensus by using the transmission unit quantity matrix until a maximum global intermediate frequency consensus is obtained;

[0012] obtaining a target frequency domain matrix according to the maximum global intermediate frequency consensus and the frequency domain matrix of the input image;

[0013] training an image segmentation model of the client by using the target frequency domain matrix to obtain a trained image segmentation model.

[0014] Optionally, the obtaining of the local intermediate frequency spectrum of the client according to the frequency domain matrix of the input image comprises:

[0015] determining an intermediate frequency region of the frequency domain matrix according to a preset low frequency region radius and a preset high frequency region radius;

[0016] obtaining an intermediate frequency component of the input image from the intermediate frequency region;

[0017] aggregating the intermediate frequency components of all the input images of the client to obtain the local intermediate frequency spectrum of the client.

[0018] Optionally, the calculating of the transmission unit quantity matrix between the local intermediate frequency spectrum and the global intermediate frequency consensus by using the Sinkhorn algorithm comprises:

[0019] constructing a local discrete frequency distribution based on the local intermediate frequency spectrum, and constructing a global discrete frequency distribution based on the global intermediate frequency consensus;

[0020] calculating the transmission unit quantity matrix by using the Sinkhorn algorithm based on an EMD distance between the local discrete frequency distribution and the global discrete frequency distribution.

[0021] Optionally, the iteratively updating of the global intermediate frequency consensus by using the transmission unit quantity matrix until the maximum global intermediate frequency consensus is obtained comprises:

[0022] aligning the local intermediate frequency spectrum and the global intermediate frequency consensus by using the transmission unit quantity matrix;

[0023] The aligned local intermediate frequency spectrum is aggregated to obtain an updated global intermediate frequency consensus;

[0024] When the difference between the global intermediate frequency consensus updated by two adjacent iterations is less than a preset threshold, the iteration is stopped, and the maximum global intermediate frequency consensus is output.

[0025] Optionally, the local discrete frequency distribution is constructed based on the local intermediate frequency spectrum, including:

[0026] A set of frequency points is extracted from the local intermediate frequency spectrum;

[0027] The spatial position and energy weight corresponding to each frequency point in the set of frequency points are obtained;

[0028] The local discrete frequency distribution is obtained according to the spatial position and energy weight of each frequency point.

[0029] Optionally, the target frequency domain matrix is obtained according to the maximum global intermediate frequency consensus and the frequency domain matrix of the input image, including:

[0030] The maximum global intermediate frequency consensus is linearly transformed to obtain a query vector;

[0031] The low-frequency component and the high-frequency component are extracted from the frequency domain matrix of the input image by using the query vector;

[0032] The low-frequency component, the high-frequency component, and the maximum global intermediate frequency consensus are aggregated to obtain the target frequency domain matrix.

[0033] Optionally, the image segmentation model of the client is trained by using the target frequency domain matrix, including:

[0034] The target frequency domain matrix is input into the image segmentation model to obtain a predicted segmentation result;

[0035] The overlap between the predicted segmentation result and the real segmentation area is optimized by a target loss function.

[0036] Optionally, the image segmentation model of the client is trained by using the target frequency domain matrix, further including:

[0037] The frequency enhancement feature is obtained according to the low-frequency component and the high-frequency component of the input image;

[0038] The difference between the frequency enhancement feature and the maximum global intermediate frequency consensus is optimized by a target loss function.

[0039] In a second aspect, an electronic device is provided, which includes a processor and a memory storing program instructions, wherein the processor is configured to execute the method according to the first aspect or any possible implementation manner when running the program instructions.

[0040] In a third aspect, a computer readable storage medium is provided, which stores a computer program, wherein the computer program is executed by a processor to implement the method according to the first aspect or any possible implementation manner.

[0041] The method for multi-modal federated learning based on image segmentation provided in the embodiments of the present application converts the input image of the client from the spatial domain to the frequency domain by using the discrete cosine transform, thereby obtaining the local intermediate frequency spectrum of the client, aggregates the local intermediate frequency spectrum of each client at the server end, and adopts this scheme, which can effectively reduce the amount of data transmitted by transmitting only the intermediate frequency coefficients containing basic features, thereby reducing the communication overhead and improving the communication efficiency. In addition, the Sinkhorn algorithm improved EMD is applied to calculate the transmission unit quantity matrix between the local intermediate frequency spectrum and the global intermediate frequency consensus, which can obtain the optimal transmission scheme, and the global intermediate frequency consensus is iteratively updated by using the optimal transmission scheme, and the global intermediate frequency consensus is further used to guide the client to reconstruct the structural features and the detailed information, thereby enhancing the light and accurate medical image segmentation of the image segmentation model under multi-modal, and ensuring the balance between the model efficiency and the accuracy. At the same time, the Sinkhorn algorithm is introduced as an efficient approximation of EMD, which provides a simple and efficient iterative process based on matrix, and greatly reduces the computational overhead.

[0042] The foregoing general description and the following description are only exemplary and explanatory, and are not intended to limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0043] One or more embodiments are exemplarily illustrated by the corresponding drawings, which do not constitute a limitation on the embodiments, and elements with the same reference numerals are considered to be similar elements, the drawings do not constitute a proportional limitation, and wherein:

[0044] Figure 1 a schematic diagram of the computational efficiency between different medical image segmentation methods under MFL setting;

[0045] Figure 2 a schematic diagram for quantitatively evaluating the similarity of CT and MRI in the spatial domain and the frequency domain by SSIM and NMSE;

[0046] Figure 3 a general architecture diagram of FedMFD provided in the embodiments of the present application;

[0047] Figure 4 A flowchart of a multi-modal federated learning method based on image segmentation provided by an embodiment of the present application is shown in FIG. 1.

[0048] Figure 5 A dataset details diagram of an image segmentation task provided by an embodiment of the present application is shown in FIG. 2.

[0049] FIG. 6(a) is a comparison diagram of Dice scores of different frequency band combinations in a heart segmentation task provided by an embodiment of the present application.

[0050] FIG. 6(b) is a comparison diagram of Dice scores of different frequency band combinations in a liver segmentation task provided by an embodiment of the present application.

[0051] Figure 7 A comparison table of generalization performance of heart and liver segmentation provided by an embodiment of the present application is shown in FIG. 7.

[0052] Figure 8 A visual comparison diagram of segmentation results of a liver segmentation task provided by an embodiment of the present application is shown in FIG. 8.

[0053] Figure 9 A comparison diagram of computational efficiency of heart and liver segmentation provided by an embodiment of the present application is shown in FIG. 9.

[0054] Figure 10 A performance diagram of different methods in a heart and liver segmentation task provided by an embodiment of the present application is shown in FIG. 10.

[0055] Figure 11 An ablation study diagram of a FedMFD scheme provided by an embodiment of the present application is shown in FIG. 11.

[0056] Figure 12 A schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 12. DETAILED DESCRIPTION

[0057] In order to make the objects, technical solutions and advantages of the present application clearer, the following will further describe the embodiments of the present application in conjunction with the drawings.

[0058] It should be clear that the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0059] The following description refers to the accompanying drawings. Unless otherwise indicated, same or similar elements in different drawings are denoted by same or similar reference numerals. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0060] In the description of the present application, it should be understood that the terms "first", "second", "third" and the like are only used to distinguish similar objects, and do not necessarily describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, "multiple" refers to two or more, unless otherwise specified. The association between the associated objects is described, which means that there can be three relationships, for example, A and / or B, which can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. The character " / " generally represents that the associated objects before and after are in an "or" relationship.

[0061] In the medical image segmentation task, FL as a distributed learning framework provides an effective solution for privacy protection and cross-institutional collaboration modeling without sharing the original data. Although the classic federated optimization algorithms FedAvg and FedProx have been widely applied in this field, there are still challenges, such as model performance decline and training instability when facing multi-modal and non-IID data. In recent years, personalized federated, dynamic optimization of local and global models, local calibration and other methods have tried to improve the generalization of FL. Although progress has been made in segmentation accuracy, most methods treat communication efficiency and generalization separately, ignoring their trade-offs. In multi-modal medical image segmentation, high communication overhead and limited generalization remain major challenges.

[0062] Various strategies such as feature alignment, frequency domain modeling, and domain adaptation have been proposed in the related art to improve performance under non-independent and identical distribution (non-IID) conditions. For example, FedDG utilizes scenario learning in frequency space to enhance generalization between clients, while FL-MRCM adjusts latent feature distribution to mitigate performance decline in multi-modal learning. To reduce communication overhead, traditional solutions mainly focus on global-local aggregation strategies and model compression techniques, aiming to alleviate communication bottlenecks by reducing synchronization frequency or payload size. Although they have achieved success, these methods often involve trade-offs, compromising segmentation accuracy or generalization performance, as shown in Figure 1 , Figure 1 a schematic diagram of the computational efficiency between different medical image segmentation methods under MFL settings, Figure 1 The bubble size in the present technology reflects the related computational cost. The existing technology treats communication efficiency and generalization as independent targets, lacking a unified optimization perspective. For example, FEAL reaches convergence after 95 rounds, but its training process is time-consuming, and each iteration is relatively inefficient.

[0063] In general, existing research on multi-modal federated learning treats communication efficiency and model generalization ability as independent optimization objectives, ignoring the trade-off between the two, resulting in a difficult balance between model generalization performance and client resource constraints. However, model performance directly determines the stability and reliability of medical image segmentation in clinical decision-making, and suboptimal or erroneous outputs can lead to irreversible medical events or adverse events, thereby endangering patient safety and overall clinical outcomes.

[0064] Even within the same organ, spatial domain (e.g., CT and MRI) similarity indicates low correlation between different domains. In contrast, the mid-frequency band shows significantly stronger correlation across modalities in the frequency domain, as shown in Figure 2 Figure 2 To quantitatively evaluate the similarity of CT and MRI in the spatial and frequency domains by SSIM and NMSE, by using the structural similarity index measure (SSIM) and the normalized mean square error (NMSE) to quantitatively evaluate the similarity of CT and MRI in the spatial and frequency domains, it can be seen that SSIM increases with the increase of structural similarity, while NMSE decreases with the decrease of intensity difference.

[0065] Based on this, the embodiments of the present application provide a multi-modal federated learning method based on image segmentation, equipment and storage medium, so as to reduce the communication overhead of federated learning under the image segmentation task while improving the multi-modal generalization ability of the model.

[0066] Figure 3 is the overall architecture diagram of FedMFD provided by the embodiments of the present application, in combination with Figure 3 As shown in the figure, the embodiments propose a mid-frequency consensus driven multi-modal federated segmentation framework (FedMFD), aiming to effectively improve the generalization ability of multi-modal medical image segmentation tasks while reducing communication overhead.

[0067] As shown in Figure 3 , FedMFD adopts a three-stage framework. In stage 1, the mid-frequency sampling stage, each client transforms each input image from the time domain to the frequency domain by discrete cosine transform (DCT), then applies a pre-defined binary mask M to extract the mid-frequency band, from which the corresponding energy spectrum is calculated and uploaded to the server. In this stage, by transmitting only the mid-frequency coefficients with basic anatomical features, the communication overhead can be significantly reduced. In stage 2, the maximum mid-frequency consensus module is designated as FedEMD, the server uses the Sinkhorn-Knopp algorithm optimized EMD to calculate the number of repeated transmission units between the local mid-frequency spectrum of the client and the global mid-frequency consensus, and iteratively reconstructs the global mid-frequency consensus until the difference between the global mid-frequency consensus of the two iterations reaches the convergence threshold, i.e.​ The maximum global mid-frequency consensus is obtained. In this stage, the FedEMD module on the server side uses EMD to calculate the optimal transmission plan and iteratively reconstructs the global Mid-Freq consensus with the maximum information. This process enhances segmentation generalization under modal differences and background interference. Finally, in order to further solve the modal difference caused by only mid-frequency sharing, in the third stage, a frequency-aware cross-attention mechanism (FCA) is used to guide the extraction of low-frequency and high-frequency information, which is then fused with the maximum global mid-frequency consensus, thereby realizing adaptive selection of information components in the entire frequency spectrum. In this stage, the FCA mechanism uses the global mid-frequency consensus to guide the extraction of local low-frequency and high-frequency features, and integrates the three, which can effectively suppress model drift.

[0068] The framework successfully realizes an effective trade-off between communication efficiency and model generalization, reducing the amount of data transmitted between clients and servers, while improving the generalization ability of MFL.

[0069] Figure 4 is a flowchart of a multi-modal federated learning method based on image segmentation provided by an embodiment of the present application. As shown in Figure 4 The multi-modal federated learning method based on image segmentation provided by the embodiment of the present application comprises the following steps:

[0070] S401, performing discrete cosine transform on the input image of the client to obtain a frequency domain matrix of the input image;

[0071] S402, obtaining the local mid-frequency spectrum of the client according to the frequency domain matrix of the input image;

[0072] S403, aggregating the local mid-frequency spectrum of each client on the server side to obtain a global mid-frequency consensus;

[0073] S404, calculating the transmission unit quantity matrix between the local mid-frequency spectrum and the global mid-frequency consensus by the Sinkhorn algorithm, and iteratively updating the global mid-frequency consensus by using the transmission unit quantity matrix until the maximum global mid-frequency consensus is obtained;

[0074] S405, obtaining a target frequency domain matrix according to the maximum global mid-frequency consensus and the frequency domain matrix of the input image;

[0075] S406, training the image segmentation model of the client by using the target frequency domain matrix to obtain a trained image segmentation model.

[0076] The method provided by the embodiment of the application is based on image segmentation and multi-modal federated learning. The input image of the client is converted from a spatial domain to a frequency domain by using discrete cosine transformation, thereby obtaining a local intermediate frequency spectrum of the client. The local intermediate frequency spectrum of each client is aggregated at a server end. By transmitting only intermediate frequency coefficients containing basic features, the amount of data transmitted can be effectively reduced, thereby reducing communication overhead and improving communication efficiency. In addition, the Sinkhorn algorithm is applied to calculate a transmission unit quantity matrix between the local intermediate frequency spectrum and a global intermediate frequency consensus, so as to obtain an optimal transmission scheme. The global intermediate frequency consensus is iteratively updated by using the optimal transmission scheme. The global intermediate frequency consensus is further used to guide the client to reconstruct structural features and detailed information, thereby enhancing the segmentation generalization capability of the image segmentation model under modal difference and background interference and significantly improving segmentation precision. Meanwhile, the Sinkhorn algorithm is introduced as an efficient approximation of EMD. The algorithm provides a simple and efficient iterative process based on a matrix, thereby greatly reducing calculation overhead.

[0077] Optionally, in step S401, the input image of the client is subjected to discrete cosine transformation to obtain a frequency domain matrix of the input image. In this embodiment, direct transmission of the image by the client in federated learning can cause privacy leakage and large consumption of bandwidth and storage resources. To solve this problem, in this embodiment, the input image of the client is subjected to discrete cosine transformation to obtain a frequency domain matrix of the input image. As shown in the following formula, the spatial data is converted into frequency components by applying discrete cosine transformation (DCT) at the client, thereby realizing effective compression and feature extraction for image and signal processing. Specifically, the nth client is denoted as cn n , and given an image sample X ∈ R from a sample data set H×W×C (C = 1), the formula for obtaining the frequency domain representation of the image sample by applying DCT is as follows:

[0078]

[0079] where X(x, y) is the input image, G(u, y) represents an intermediate result obtained by applying 1D discrete cosine transformation (DCT) in the row direction of the input image X(x, y); F(u, v) represents the final two-dimensional DCT coefficient obtained by again performing 1D discrete cosine transformation in the column direction on the intermediate result G(u, y); H is the height (i.e., the number of rows) of the input image; W is the width (i.e., the number of columns) of the input image; and a(u) and a(v) are normalization factors for ensuring the orthogonality of the discrete cosine transformation.

[0080] Optionally, in the step S402, the local mid-frequency spectrum of the client is obtained according to the frequency domain matrix of the input image, including: determining a mid-frequency region of the frequency domain matrix according to a preset low-frequency region radius and a high-frequency region radius; obtaining the mid-frequency component of the input image from the mid-frequency region; and then aggregating the mid-frequency components of all input images of the client to obtain the local mid-frequency spectrum of the client.

[0081] Since the same organ in different medical imaging modalities (e.g., CT and MRI) mainly occurs in the high-frequency and low-frequency bands, wherein the low-frequency part mainly represents the structural outline of the image, and the high-frequency part contains more detailed edge-related information. However, the mid-frequency band reflects the difference in style and structure of the image with higher correlation. Therefore, sharing only the mid-frequency information not only can reduce the privacy risk, but also can reduce the transmission burden of the client data.

[0082] In order to ensure the structural consistency of the frequency domain information transmitted by each client and the aggregation alignment, the embodiment further introduces a unified mid-frequency band extraction mechanism. Specifically, for the frequency domain matrix of each image X in the sample data set , the frequency coordinates (u, v) are defined based on the origin (0, 0) in the upper left corner of the frequency domain matrix. The frequency radius at the coordinates (u, v) is given by . The radius of the high-frequency region and the radius of the low-frequency region are obtained according to prior knowledge, so as to determine the mid-frequency region M, which is determined by the following formula:

[0083]

[0084] wherein r min is the low-frequency region radius; and r max is the high-frequency region radius.

[0085] Further, based on the mid-frequency region M, the mid-frequency component of the frequency domain matrix is defined as follows:

[0086]

[0087] Next, in order to generate the client-level frequency representation, the energy information of all input images at the same frequency coordinates is aggregated within the client to obtain the mid-frequency aggregated spectrum representation of the client, which is shown in the following formula:

[0088]

[0089] wherein is the local mid-frequency spectrum; is the sample data set; is the mid-frequency component of the frequency domain matrix of the image X.

[0090] After obtaining the local intermediate frequency spectrum of the client, the local intermediate frequency spectrum of each client is uploaded to the server end, and the local intermediate frequency spectrum of each client is aggregated at the server end to obtain a global intermediate frequency consensus;

[0091] Optionally, in the step S404, the Sinkhorn algorithm is used to calculate the transmission unit quantity matrix between the local intermediate frequency spectrum and the global intermediate frequency consensus, including: constructing a local discrete frequency distribution based on the local intermediate frequency spectrum, and constructing a global discrete frequency distribution based on the global intermediate frequency consensus; then, based on the EMD distance between the local discrete frequency distribution and the global discrete frequency distribution, the Sinkhorn algorithm is used to calculate the transmission unit quantity matrix.

[0092] Specifically, the construction of the local discrete frequency distribution based on the local intermediate frequency spectrum includes: extracting a frequency point set from the local intermediate frequency spectrum; obtaining the spatial position and energy weight corresponding to each frequency point in the frequency point set; and obtaining the local discrete frequency distribution according to the spatial position and energy weight of each frequency point.

[0093] In order to use the frequency characteristics of the client level for subsequent EMD distance calculation, the embodiment of the application extracts a frequency point set from the local intermediate frequency spectrum , and constructs a frequency characteristic set in the form of a discrete distribution. For a frequency point i in the frequency point set, its coordinates are represented as (u i ,v i ), its corresponding spatial position is represented as p i , and it is assigned an energy weight s i . The specific formula of the energy weight s i is as follows:

[0094]

[0095] where N n is the total number of frequency points in the local intermediate frequency spectrum .

[0096] Next, the local discrete frequency distribution is constructed according to the spatial position and energy weight of each frequency point in the local intermediate frequency spectrum , and the formula is as follows:

[0097]

[0098] where P n is the local discrete frequency distribution, p i represents the spatial position of the frequency point in the local intermediate frequency spectrum , and s i represents the energy weight of the frequency point in the local intermediate frequency spectrum .

[0099] Similarly, the global discrete frequency distribution of the global mid-frequency consensus is constructed, and the formula is as follows:

[0100] P global ={(p j ,s j )}

[0101] Wherein, P global is the global discrete frequency distribution, p j represents the spatial position of the frequency point in the global mid-frequency consensus, and s j represents the energy weight of the frequency point in the global mid-frequency consensus.

[0102] Then, the cosine distance cost matrix C ij is constructed using the spatial positions of the frequency points in the local discrete frequency distribution and the global discrete frequency distribution:

[0103]

[0104] Further, the number of units transported at the minimum cost is calculated using the EMD formula, and the EMD formula is as follows:

[0105]

[0106] subject to γ ij ≥0,i=1,…,N n ,j=1,…,N m

[0107]

[0108] Wherein, represents the transportation unit matrix; N m is the total number of frequency points in the global mid-frequency consensus F global ; the objective function is the earth moving distance (EMD) between P n and P global ; γ ij is the transportation unit matrix, which represents the number of units transported from the i-th frequency point of the local discrete frequency distribution to the j-th frequency point of the global discrete frequency distribution.

[0109] However, the traditional EMD calculation method has high calculation cost. In order to achieve the lightweight optimization goal, the Sinkhorn-Knopp algorithm is introduced as an efficient approximation of EMD. Compared with the traditional EMD with a cubic time complexity O(n 3 ), the Sinkhorn algorithm rewrites the original linear programming problem into a simple and efficient matrix-based iterative process, significantly reducing the calculation overhead to O(T·n 2). In which, Sinkhorn algorithm introduces an entropy regularization term H (γ), the definition formula is:

[0110]

[0111] In which, γ ij represents the transmission unit quantity matrix; N n is the total number of local intermediate frequency spectrum intermediate frequency points; N m is the global intermediate frequency consensus F global intermediate frequency points.

[0112] And, the Sinkhorn distance between P n and P global is expressed by the following formula:

[0113] D Sinkhorn = EMD (P n , P global ) - εH (γ)

[0114] In which, D Sinkhorn is the Sinkhorn distance between P n and P global ; ε is the regularization coefficient, used to balance the transmission cost and the entropy of the solution, so as to control the smoothness of the final transmission scheme, and ε> 0.

[0115] Optionally, in the step S404, the global intermediate frequency consensus is iteratively updated by using the transmission unit quantity matrix until the maximum global intermediate frequency consensus is obtained, comprising: aligning the local intermediate frequency spectrum with the global intermediate frequency consensus by using the transmission unit quantity matrix; aggregating the aligned local intermediate frequency spectrum to obtain the updated global intermediate frequency consensus; when the difference between the global intermediate frequency consensus of adjacent two times of iterative updating is lower than a preset threshold, stopping the iterative updating, and outputting the maximum global intermediate frequency consensus.

[0116] In this embodiment, after completing the matching process based on the optimal transmission, the global intermediate frequency consensus is iteratively updated by aggregating the information of all clients after alignment. Specifically, the transmission unit quantity matrix is represented as In which, K represents the total number of clients, and t represents the current iteration step number. Each client aligns its local intermediate frequency spectrum with the global intermediate frequency consensus by using the transmission unit quantity matrix. The global intermediate frequency consensus corresponding to the iteration t of the client c n is denoted as Its definition is as follows:

[0117]

[0118] In which, represents the client c nThe intermediate frequency spectrum value at the i-th frequency point, represents the number of transmission units matrix at the iteration step t-1.

[0119] Subsequently, the aligned results of all clients are aggregated to form an updated global intermediate frequency consensus. Specifically, according to the different iteration steps t, the initial local intermediate frequency spectrum or the aligned local intermediate frequency spectrum is used to calculate the average value to obtain the global intermediate frequency consensus. The global intermediate frequency consensus is updated in the following formula:

[0120]

[0121] In this embodiment, in the initial stage of iteration (t = 0), the local intermediate frequency spectrum of each client is aggregated to initialize the global intermediate frequency consensus, that is, the average value of the initial local intermediate frequency spectrum of all clients is taken as the initial global intermediate frequency consensus. Then, the structure characteristics of each client are aligned with the global intermediate frequency consensus, and the global representation is gradually optimized until convergence. Specifically, in each iteration t, the current global intermediate frequency consensus is taken as the reference target, and the number of transmission units matrix γ between each local intermediate frequency spectrum and the global intermediate frequency consensus ij of the last round is calculated. In each client, the local intermediate frequency spectrum is aligned with the global intermediate frequency consensus by using the number of transmission units matrix γ ij , and the local intermediate frequency spectrum of all clients after alignment is averaged at the server side to obtain the global intermediate frequency consensus of the next round.

[0122] The iterative updating process will continue until the global intermediate frequency consensus between two consecutive iterations changes by less than a preset threshold δ, and the final result is output. The final matrix F ′ global is the final determined global intermediate frequency consensus, that is, the maximum global intermediate frequency consensus. The constructed matrix effectively captures the most representative shared frequency characteristics between clients under the premise of ensuring the minimum cost of structure alignment. Subsequently, the matrix will be transmitted to each client to guide the extraction of local features.

[0123] Optionally, in the step S405, the target frequency domain matrix is obtained according to the maximum global intermediate frequency consensus and the frequency domain matrix of the input image, including: performing linear transformation on the maximum global intermediate frequency consensus to obtain a query vector; extracting low frequency components and high frequency components from the frequency domain matrix of the input image by using the query vector; and then, aggregating the low frequency components, the high frequency components, and the maximum global intermediate frequency consensus to obtain the target frequency domain matrix.

[0124] To further improve the efficiency of client utilization of local frequency domain information, this embodiment proposes a frequency domain feature extraction method based on a cross-attention mechanism. Specifically, this implementation employs two independent cross-attention modules, where F... ′ global As a query vector Q, it is used to guide the search for low-frequency components. (Low-Freq) and high-frequency components (High-Freq) feature extraction. The calculation method for this cross-attention is defined as follows:

[0125]

[0126] Among them, Q m =Proj Q (F ′ global ), where is the query vector, determined by the maximum global mid-frequency consensus F. ′ global Obtained through linear transformation, it is used to guide feature extraction in the attention mechanism; These are the local low-frequency components from the client, obtained through two linear mappings, respectively. The extracted key and value vectors are used to calculate their relevance to the query vector; similarly, These are the local high-frequency components from the client, obtained through two linear mappings, respectively. The extracted key vector and value vector.

[0127] Furthermore, in order to construct a complete frequency consensus that integrates client-specific characteristics and global structure, this embodiment will use low-frequency feature F ′ l High-frequency characteristics F ′ h With the largest global intermediate frequency consensus F ′ global Aggregation is performed to obtain the client's final frequency domain matrix. That is, the target frequency domain matrix, which is defined as follows:

[0128]

[0129] Concat(·) represents a splicing operation aligned by frequency position, used to restore a relatively complete frequency domain arrangement.

[0130] The final target frequency domain matrix contains both the client's unique personalized feature response and shared multimodal structural information, thus effectively guiding the client's local image segmentation task.

[0131] Optionally, in the step S406, the image segmentation model of the client is trained by using the target frequency domain matrix, including: inputting the target frequency domain matrix into the image segmentation model to obtain a predicted segmentation result; and optimizing an overlapping degree between the predicted segmentation result and a real segmentation region by using a target loss function. The target loss function is composed of two loss terms, which are and

[0132] In order to effectively utilize the fused frequency domain representation in the medical image segmentation task, a task-driven supervised loss function is designed in the embodiment. In the embodiment, after the target frequency domain matrix is input into the image segmentation model, a predicted segmentation result is obtained. In view of the fact that the foreground target in the medical image usually has sparsity and fuzzy boundary, DiceLoss is adopted as a main loss function in the embodiment to directly optimize the overlapping degree between the predicted region and the real region. The loss function is defined as follows:

[0133]

[0134] wherein, Y represents the real label, Dice(·) is a Dice coefficient, and is used to measure the overlapping degree between the predicted region and the real region.

[0135] Optionally, in the step S406, the image segmentation model of the client is trained by using the target frequency domain matrix, and further including: acquiring a frequency enhanced feature according to the low-frequency component and the high-frequency component of the input image; and optimizing a difference between the frequency enhanced feature and the maximum global mid-frequency consensus by using the target loss function.

[0136] In order to further enhance the structural consistency between the features extracted from the low-frequency and high-frequency components and the mid-frequency representation, a structure alignment regularization term is introduced in the embodiment, and the structure alignment regularization term is defined as follows:

[0137]

[0138] wherein, F ′ = F ′ l + F ′ h represents the frequency enhanced feature extracted under the guidance of the mid-frequency spectrum, and Dist(·) represents an L2 distance function, which is used to measure the difference between two representations.

[0139] The final joint optimization objective function is:

[0140] ​

[0141] The loss function not only guarantees segmentation accuracy, but also enhances the model's ability to capture modal invariant features through frequency domain structure guidance, thereby improving the model's generalization performance in non-independent and identically distributed (non-IID) scenarios.

[0142] To embody the beneficial effects of the above embodiments, the embodiments of the present application provide a method for experimentally verifying the above multi-modal federated learning method based on image segmentation.

[0143] The embodiments verify the proposed framework on four publicly available datasets, covering cross-modality segmentation tasks of the same organ, each involving both CT and MRI modalities. The datasets used include MICCAI FLARE22 Challenge, CHAOS, CHD68, and MSDHeart, covering segmentation tasks of multiple organs such as lungs, hearts, and livers, with specific details as shown in Table 1. Figure 5 Figure 5 The data set details of the image segmentation task provided by the embodiments of the present application are shown in the diagram. To comprehensively evaluate segmentation performance, four widely used indicators are adopted: Dice coefficient (Dice), Hausdorff distance (HD), and mean intersection over union (mIoU). Among them, the Dice coefficient is a set similarity measure indicator, used to quantify the degree of overlap between the predicted mask and the true label, and the closer the value is to 1, the higher the segmentation accuracy and spatial alignment. In addition to segmentation accuracy, communication efficiency is quantified by the number of convergence rounds, total training time, and precision change trajectory, to comprehensively reflect the computational and communication overhead in the federated learning scenario.

[0144] In the federated learning (FL) process, all clients use uniform hyperparameter settings, and the input data is uniformly adjusted to axial slices with a size of 384x384. In the FedMFD model training, reasonable hyperparameters are configured, including a learning rate (lr) of 1e-4, an optimizer of ADAM, a batch size of 16, and a training round of 150 rounds. In addition, considering the model complexity and data characteristics, parameters such as hidden layer dimension, Dropout rate, and weight decay are appropriately adjusted to ensure the stability of the training process and the generalization ability of the model. To reduce the randomness of experimental results, each experiment is independently run 10 times, and the final evaluation indicators are reported in the form of "mean ± standard deviation" to reflect the overall performance and cross-sample variance of the model. All experiments are implemented based on Python 3.8 and PyTorch 1.7.1, and trained on an NVIDIA RTX 3090 GPU (24GB of video memory) to ensure computational efficiency and the reliability of the results.

[0145] ​As shown in FIG. 6, FIG. 6 is a comparison diagram of Dice scores of different frequency band combinations in a heart and liver segmentation task provided by an embodiment of the application, wherein the X axis and the Y axis represent the frequency bands of CT and MRI images respectively. The quantitative analysis reveals the differentiated contribution of different frequency bands in medical image segmentation, providing a more objective theoretical basis for the sharing of mid-frequency information. Taking liver segmentation as an example, low-frequency information (+Low, Dice: 53.28%) can capture the overall structure, but the boundary accuracy is insufficient; high-frequency information (+High, Dice: 71.04%) contains detailed features, but is easily disturbed by noise, resulting in unstable segmentation results; and mid-frequency information (+Mid, Dice: 94.76%) achieves a good balance between information acquisition and noise suppression, and can effectively encode key boundary transition features. The mid-frequency band naturally has noise suppression capability while encoding boundary transition information. Although full-band activation can obtain the optimal segmentation effect, it depends on the transmission of complete data, which brings bandwidth pressure and privacy risk. The experimental results further confirm that the mid-frequency band isolation strategy achieves an optimal trade-off between segmentation performance, communication efficiency and privacy protection: while maintaining near-optimal accuracy, it significantly reduces data transmission volume, and effectively enhances the privacy protection capability of patient medical data through the natural filtering mechanism of frequency domain information.

[0146] The present embodiment evaluates six state-of-the-art federated medical image segmentation methods on multi-modal liver and heart segmentation tasks. The baselines include FedAvg, which uses a simple average, and FedBN, which uses local batch normalization to improve generalization. Among the advanced methods, FedALA captures the required information in the global model for the client models in personalized FL. FEAL introduces Dirichlet priors to model uncertainty and calibrate data evaluation, thereby achieving more effective information sample selection under domain transfer in the federated learning scenario. HarmoFL averages the amplitudes to align the device distribution, FedMix mixes local and global models to enhance robustness. FAMNet utilizes frequency domain similarity to reduce domain variance.

[0147] As Figure 7 With Figure 8 as Figure 7The generalization performance comparison table of heart and liver segmentation provided by the embodiments of the present application needs to be explained that all comparison methods are evaluated under consistent client dataset settings to ensure fair comparison. Bold indicates the best value, and underlined indicates the second best value. In this setting, corresponding to each client node. The upward arrow (↑) indicates that a higher value corresponds to better performance, while the downward arrow (↓) indicates the opposite. As can be seen from the figure, FedMFD achieves the best mIoU and Dice score in both liver and heart segmentation tasks. In the liver segmentation task, FedMFD achieves an mIoU of 93.99% and a Dice of 95.14%, which is 0.71% and 9.66% higher than FedALA, respectively. In the heart segmentation task, its mIoU and Dice are 94.14% and 95.27%, respectively, which not only outperforms FedALA and FedBN, but also shows smaller performance fluctuations under the condition of multi-source heterogeneous data, reflecting good stability. Figure 8 The visual comparison diagram of segmentation results of the liver segmentation task provided by the embodiments of the present application. The first 1-2 rows and the third 3-4 rows in the figure correspond to the visual results on MRI and CT, respectively. It can be seen that FedMFD has clearer segmentation quality than the other seven competitors. Especially noteworthy is that FedMFD performs well in terms of anatomical structure fidelity of the heart, effectively depicting fine boundaries and internal structures, significantly improving the segmentation effect of complex heart contours and reducing the common artifact phenomenon in other methods. It also performs well in boundary positioning and structure restoration, with Hausdorff distance reduced to 13.72 and 10.15 in liver and heart segmentation, respectively, which is significantly better than FedAvg and HarmoFL (HD> 30). In addition, FedMix and FedAvg perform well on CT images, but have obvious deviations on MRI images, indicating that they only have advantages in a single modality. In contrast, FedMFD has good adaptability in both CT and MRI dual modalities, ensuring consistent high-precision performance across modalities.

[0148] As Figure 9 indicated, Figure 10 Figure 9 The computational efficiency comparison diagram of heart and liver segmentation provided by the embodiments of the present application. The bold indicates the best value. FedMFD significantly improves the communication efficiency in the heterogeneous multi-modal federated segmentation task, achieving convergence at the 82nd and 101st rounds in the heart and liver segmentation tasks, respectively, with a total training time of only 3410.8 seconds and 3879.6 seconds, which is the lowest among all methods. Figure 10 ​Performance of different methods in the heart and liver segmentation tasks provided by the embodiments of the present application; the four subgraphs in the figure show the performance of different methods in the heart and liver segmentation tasks, as can be seen from the figure, the FedMFD method performs best in all four subgraphs, always leading other methods and achieving the highest accuracy and Dice score. FedMFD performs rapidly in terms of accuracy improvement, surpassing other methods in about 90 minutes of training, with an accuracy close to 90%. Compared with baseline methods such as FedAvg, FedBN and FedMix, FedMFD reduces the total training time by up to 39.16% and the convergence round by 16.81%. This is due to its stronger cross-modal generalization ability, which enables a faster and more effective optimization process. Compared with the recently proposed FAMNet and FEAL methods, FedMFD still maintains an advantage in training efficiency and single-round computation time, requiring only about 42 seconds per round, showing stronger deployment adaptability, especially for resource-constrained medical device environments. FedMFD exhibits robust performance in both large organ (liver) and small organ (heart) tasks, while other methods generally exhibit significant fluctuations when facing modal heterogeneity or task complexity. This further verifies that the frequency-aware design of FedMFD effectively alleviates the communication bottleneck and enhances the training stability in different anatomical structure tasks.

[0149] The embodiments of the present application perform ablation experiments on multi-modal segmentation tasks to evaluate the contributions of two key modules in the proposed method, FedEMD and FACA (frequency-aware cross-attention mechanism). Specifically, “w / o FedEMD” means removing the FedEMD strategy on the server side, and only aggregating shared intermediate frequency information through simple averaging; “w / o FCA” means disabling the frequency-aware cross-attention mechanism. While the base variant removes both modules, it serves as the baseline configuration for the model. As shown in Figure 11 Figure 11 The ablation study diagram of the FedMFD scheme provided by the embodiments of the present application, where “w / o FedEMD” and “w / o FCA” represent two FedMFD variants with the frequency consistency module and the frequency-aware cross-attention mechanism removed, respectively. “base” represents FedMFD without the two components. As can be seen from the figure, removing the FedEMD module significantly reduces communication efficiency. In addition, removing either module will result in a significant decline in generalization performance; when both are removed, the segmentation performance in the unseen field drops to the lowest level. The experimental results meet expectations, highlighting the complementary role of FedEMD and FCA in overall performance improvement.

[0150] ​In the above embodiments, in terms of communication efficiency, the client images are converted from spatial domain to frequency domain coefficients by discrete cosine transform (DCT), and only the medium frequency components are selected for uploading, thereby reducing transmission redundancy. In terms of generalization ability, the above embodiments introduce the Earth Mover's Distance (EMD) to quantify the maximum similarity of medium frequency features between clients, and generate a global frequency consensus based on the optimal transmission scheme. With the guidance of the global frequency consensus, the clients reconstruct the structural features and detailed information, thereby enhancing the segmentation generalization performance of the model under the interference of modal difference and background noise. A large number of experiments on multiple computed tomography (CT) and magnetic resonance imaging (MRI) modal dataset prove that FedMFD is superior to existing mainstream methods in terms of segmentation accuracy and communication efficiency, and exhibits significant performance advantages.

[0151] In conjunction with Figure 12 As shown in the above embodiments, the electronic device 1200 includes a processor 1201 and a memory 1202. Optionally, the device can also include a communication interface 1203 and a bus 1204. The processor 1201, the communication interface 1203, and the memory 1202 can communicate with each other through the bus 1204. The communication interface 1203 can be used for information transmission. The processor 1201 can invoke the logical instructions in the memory 1202 to execute the multi-modal federated learning method in the above embodiments.

[0152] In addition, the logical instructions in the memory 1202 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium.

[0153] The memory 1202 as a computer readable storage medium can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the method in the embodiments of the present application. The processor 1201 executes the function application and data processing by running the program instructions / modules stored in the memory 1202, that is, implements the multi-modal federated learning method in the above embodiments.

[0154] The memory 1202 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 1202 can include a high-speed random access memory, and can also include a non-volatile memory.

[0155] The embodiment of the application provides a storage medium, which stores computer executable instructions configured to perform the multi-modal federated learning method in the above embodiment.

[0156] The storage medium described above can be a transitory computer readable storage medium or a non-transitory computer readable storage medium.

[0157] The technical solution of the embodiment of the application can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes one or more instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiment of the application. The storage medium described above can be a non-transitory storage medium, including a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes, or a transitory storage medium.

[0158] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0159] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0160] In the embodiments disclosed herein, the disclosed methods, products (including but not limited to apparatuses, devices, etc.), can be implemented in other manners. For example, the described apparatus embodiments can be implemented only in a form of a logical function, and can be implemented by using a manner such as software (for example, a computer program) or a combination of hardware and software. In a component, a plurality of components, or a combination thereof, can be implemented. In some embodiments, a plurality of components or a combination thereof can be integrated in a machine to realize an apparatus according to the embodiments. In some embodiments, a plurality of components or a combination thereof can be integrated in one physical device or in a virtual machine. In some embodiments, a plurality of components or a combination thereof can be implemented in an addressable storage medium by using instructions.

[0161] The flowcharts and block diagrams in the drawings show the possible architectural, functional, and operational scenarios of systems, methods, and computer program products according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a portion of code that contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the drawings, the operations or steps corresponding to different blocks can also occur in an order different from that disclosed in the descriptions, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. Each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by using a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by using a combination of dedicated hardware and computer instructions.

Claims

1. A multi-modal federated learning method based on image segmentation, characterized in that, The method comprises: performing discrete cosine transform on an input image of a client to obtain a frequency domain matrix of the input image; obtaining a local intermediate frequency spectrum of the client according to the frequency domain matrix of the input image; aggregating the local intermediate frequency spectrum of each client on a server side to obtain a global intermediate frequency consensus; calculating a transmission unit quantity matrix between the local intermediate frequency spectrum and the global intermediate frequency consensus by using a Sinkhorn algorithm, and iteratively updating the global intermediate frequency consensus by using the transmission unit quantity matrix until a maximum global intermediate frequency consensus is obtained; obtaining a target frequency domain matrix according to the maximum global intermediate frequency consensus and the frequency domain matrix of the input image; training an image segmentation model of the client by using the target frequency domain matrix to obtain a trained image segmentation model.

2. The method of claim 1, wherein, The method comprises: determining an intermediate frequency region of the frequency domain matrix according to a preset low frequency region radius and a high frequency region radius; obtaining an intermediate frequency component of the input image from the intermediate frequency region; aggregating the intermediate frequency components of all input images of the client to obtain the local intermediate frequency spectrum of the client.

3. The method of claim 1, wherein, The method comprises: constructing a local discrete frequency distribution based on the local intermediate frequency spectrum, and constructing a global discrete frequency distribution based on the global intermediate frequency consensus; calculating the transmission unit quantity matrix by using the Sinkhorn algorithm based on an EMD distance between the local discrete frequency distribution and the global discrete frequency distribution.

4. The method of claim 1, wherein, The method comprises: aligning the local intermediate frequency spectrum and the global intermediate frequency consensus by using the transmission unit quantity matrix; aggregating the aligned local intermediate frequency spectrum to obtain an updated global intermediate frequency consensus; stopping the iterative updating when a difference between the global intermediate frequency consensus obtained by two adjacent iterations is lower than a preset threshold, and outputting the maximum global intermediate frequency consensus.

5. The method of claim 3, wherein, The method comprises: extracting a frequency point set from the local intermediate frequency spectrum; obtaining a spatial position and an energy weight corresponding to each frequency point in the frequency point set; obtaining the local discrete frequency distribution according to the spatial position and the energy weight of each frequency point.

6. The method of claim 1, wherein, The method comprises: performing linear transformation on the maximum global intermediate frequency consensus to obtain a query vector; extracting a low frequency component and a high frequency component from the frequency domain matrix of the input image by using the query vector; aggregating the low frequency component, the high frequency component and the maximum global intermediate frequency consensus to obtain the target frequency domain matrix.

7. The method of claim 6, wherein, The method comprises: inputting the target frequency domain matrix into the image segmentation model to obtain a predicted segmentation result; An overlap between the predicted segmentation result and a real segmentation region is optimized by a target loss function.

8. The method of claim 7, wherein, The training of the image segmentation model of the client by the target frequency domain matrix further includes: Frequency enhancement features are obtained according to the low-frequency component and the high-frequency component of the input image. A difference between the frequency enhancement features and the maximum global mid-frequency consensus is optimized by a target loss function.

9. An electronic device comprising a processor and a memory having stored thereon program instructions, wherein, The processor is configured to execute the multi-modal federated learning method according to any one of claims 1-8 when the program instructions are executed.

10. A storage medium storing program instructions, characterized in that, The program instructions are executed to perform the multi-modal federated learning method according to any one of claims 1-8.

Images