Heart ultrasound image processing method based on federated learning
By constructing a global federated model using federated learning technology, the problems of insufficient data privacy and accuracy in early CHD detection were solved. This enabled efficient and secure image processing through cross-hospital collaborative learning, improving the accuracy and efficiency of CHD detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- WUHAN UNIV
- Filing Date
- 2026-01-27
- Publication Date
- 2026-05-05
AI Technical Summary
Existing technologies struggle to accurately detect early congenital heart disease (CHD) while protecting patient privacy, especially given the limited scale and availability of data during the first trimester. This leads to insufficient samples and inaccurate training of deep learning models.
A federated learning-based method for processing cardiac ultrasound images is adopted. By constructing a global federated model, the client trains sub-models on the local sample set and uploads local prototype parameters to the server for aggregation, generating global prototype parameters, updating sub-model parameters, and finally constructing a global federated model for image processing.
It effectively solves the problem of data heterogeneity across hospitals, improves the accuracy and efficiency of early CHD detection, protects patient privacy, enhances the efficiency of collaborative learning and diagnostic accuracy among multiple hospitals, and supports early prenatal screening.
Smart Images
Figure CN121983252A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of medical image processing and federated learning technology, and in particular to a method for processing cardiac ultrasound images based on federated learning. Background Technology
[0002] Congenital heart disease (CHD) is the most common birth defect, and early CHD detection is crucial for effective intervention programs. Currently, prenatal detection of CHD mainly relies on ultrasound examinations performed during the second trimester (18-22 weeks of gestation). This relatively late detection window limits intervention options and carries the possibility of missed diagnoses. Identifying severe CHD only in late pregnancy complicates the decision to terminate the pregnancy and hinders effective intervention. Therefore, advancing the detection window to the first trimester (11-14 weeks of gestation) represents a critical but unmet technical need.
[0003] However, first-pregnancy data is limited in both scale and availability, and CHD testing in the first pregnancy relies on the experience of technicians, thus limiting the effectiveness of CHD testing. While aggregating medical samples from multiple institutions could improve the accuracy of first-pregnancy CHD testing, this approach poses serious risks to patient privacy and raises significant ethical concerns.
[0004] Some related technologies can process ultrasound images from the first trimester by training deep learning models. However, training deep learning models for CHD detection faces many challenges. Given the limited scale and availability of data from the first trimester, individual institutions often lack a large number of high-quality training samples. Collecting training samples and training the model both require a significant amount of time, making it impossible to guarantee the accuracy of deep learning models used for CHD detection.
[0005] Therefore, there is currently a lack of ultrasound image processing methods that can achieve early detection of CHD while protecting patient privacy. Summary of the Invention
[0006] This application provides a cardiac ultrasound image processing method based on federated learning to overcome the shortcomings of related technologies. The technical solution is as follows: Firstly, this application provides a federated learning-based method for processing cardiac ultrasound images, applied to a client-side application, the method comprising: Retrieve the global federated model issued by the server; The global federated model is used as a local sub-model, and the local sub-model is trained on the local sample set to output the prediction results; During the training process, local prototype parameters are constructed and uploaded to the server. Obtain the global prototype parameters generated by the server based on the aggregation of local prototype parameters of all clients, and calculate the loss function based on the prediction results and the global prototype parameters; The sub-model parameters of the local sub-model are updated based on the loss function, and the updated sub-model parameters are uploaded to the server. The server performs global model aggregation on the sub-model parameters of each client to construct a new global federated model; The server determines whether the new global federated model has converged, and the converged global federated model is used to process the target cardiac ultrasound image to obtain the image processing result.
[0007] In one alternative embodiment of the first aspect, the process of constructing the local sample set includes: Acquire multiple local sample cardiac ultrasound images; Obtain the image type of each sample cardiac ultrasound image and extract various medical concept parameters corresponding to each sample cardiac ultrasound image; The image type label for each sample of cardiac ultrasound image is obtained based on the image type, and the medical concept parameter label for each sample of cardiac ultrasound image is obtained based on the various medical concept parameters. The local sample set is constructed based on the multiple sample cardiac ultrasound images, the corresponding image type labels, and the medical concept parameter labels.
[0008] In one alternative of the first aspect, training a local sub-model on a local sample set and outputting prediction results includes: Each sample echocardiogram image in the local sample set is processed by a local sub-model to obtain prediction results for the sample echocardiogram image, including image type prediction results and medical concept parameter prediction results. During the processing, the local sub-model is used to extract features from each sample of cardiac ultrasound image to obtain an image feature vector. Each sample of echocardiogram image is input into a function used to extract medical concept parameter feature vectors to obtain the medical concept parameter feature vectors.
[0009] In one alternative of the first aspect, the local prototype parameters include medical concept prototype parameters and image feature prototype parameters; The process of constructing local prototype parameters during training includes: Based on the feature vector of the medical concept parameters, medical concept prototype parameters are constructed, including positive prototype parameters and negative prototype parameters corresponding to each type of medical concept prototype parameters. Based on the image type prediction results, the image feature vectors of each image type are clustered. Based on the average value of all image feature vectors under each image type, the prototype parameters of the image features of the current client corresponding to each image type are obtained.
[0010] In one alternative to the first aspect, the step of obtaining the global prototype parameters generated by the server based on the aggregation of local prototype parameters from all clients includes: Obtain the local prototype parameters for each client on the server side; By aggregating the prototype parameters corresponding to each type of medical concept parameter learned by each client through the server, the global prototype parameters of the corresponding type of medical concept parameter are obtained. By aggregating the negative prototype parameters corresponding to the prototype parameters of each type of medical concept learned by each client through the server, the global negative prototype parameters of the corresponding type of medical concept parameters are obtained. By aggregating the image feature prototype parameters learned by each client for each image type on the server, the global image feature prototype parameters for each image type are obtained. The global prototype parameters are distributed to each client via the server. The global prototype parameters include global negative prototype parameters and global positive prototype parameters for each type of medical concept parameter, as well as global image feature prototype parameters corresponding to each image type.
[0011] In one alternative of the first aspect, the loss function includes a default loss term, a concept regularization loss term, a federalized clinical concept alignment loss term, and a federalized disease prototype contrast loss term. The total loss function is obtained by weighted summation of each loss term, using the formula: ; in, Denotes the loss function, This refers to the default loss term. This represents the regularization loss term of the concept. This indicates the alignment loss term of the federal clinical concept. This represents the loss term compared to the federal disease prototype.
[0012] In one alternative to the first aspect, the calculation of the default loss term applies the formula: ; ; in, This refers to the default loss term; This represents a function for calculating cross-entropy loss; The image type prediction result for the i-th sample cardiac ultrasound image of the k-th client; The image type label for the i-th sample cardiac ultrasound image of the k-th client; This represents the total number of sample cardiac ultrasound images for the k-th client; This is the image feature vector corresponding to the i-th sample cardiac ultrasound image of the k-th client; The structure mask is preset during the construction of the local sample set; This represents the function used to compute the attention map.
[0013] Secondly, this application also provides a cardiac ultrasound image processing device based on federated learning, including a client and a server: The client is used to obtain the global federation model issued by the server; The client is also used to train the local sub-model on the local sample set and output the prediction results by using the global federated model as a local sub-model. The client is also used to construct local prototype parameters during training and upload the local prototype parameters to the server. The client is also used to obtain global prototype parameters generated by the server based on the aggregation of local prototype parameters of all clients, and to calculate a loss function based on the prediction results and the global prototype parameters; The client is also used to update the sub-model parameters of the local sub-model based on the loss function, and upload the updated sub-model parameters to the server; The server is also used to perform global model aggregation on the sub-model parameters of each client to construct a new global federated model; The server is also used to determine whether the new global federated model has converged. The client uses the converged global federated model to process the target cardiac ultrasound image and obtain the image processing result.
[0014] Thirdly, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method provided by the first aspect of this application or any implementation thereof.
[0015] Fourthly, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided by the first aspect of this application or any implementation thereof.
[0016] The beneficial effects of the technical solution provided in this application include at least the following: The cardiac ultrasound image processing method provided in this application effectively solves the problem of cross-hospital data heterogeneity through federated learning technology, and improves the early ultrasound image processing capability of congenital heart disease (CHD) while protecting patient privacy. The multi-party collaborative learning process not only avoids the sharing and transmission of raw medical data, fundamentally eliminating the risk of patient privacy leakage, but also optimizes the collaborative learning efficiency between multiple hospitals through a two-level knowledge dissemination mechanism.
[0017] Specifically, this application enhances the consistency of clinical concept understanding across institutions and the intra-class consistency and inter-class discriminative power of image feature space by constructing global prototype parameters, thereby significantly improving the fine-grained discrimination capability for different CHD subtypes. This is of great significance for advancing CHD screening in the first trimester, enabling earlier identification of severe CHD and supporting effective intervention plans. Furthermore, this method improves the efficiency and security of multi-hospital collaboration, promotes the synergistic optimization of medical privacy protection and diagnostic accuracy, and provides an efficient and safe new solution for early prenatal screening. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a schematic flowchart of a federated learning-based cardiac ultrasound image processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a cardiac ultrasound image processing device based on federated learning provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0021] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not limited to the steps or modules listed, but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to such process, method, product, or apparatus.
[0022] It should be noted that the terms "first" and "second" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in an order other than those described or illustrated herein.
[0023] It should be noted that Federated Learning (FL) enables multiple hospitals to share a global federated model without sharing the original patient data. This can protect data privacy and ensure a sufficient number of training samples, thereby improving the accuracy of the trained model and providing a promising approach to the effectiveness of early pregnancy CHD detection.
[0024] The present application will now be described in detail with reference to specific embodiments.
[0025] Next, combine Figure 1 This application introduces a federated learning-based cardiac ultrasound image processing method provided by embodiments of the present application, which is applied to the client side of each institution. For details, please refer to... Figure 1 , Figure 1 The illustration shows a flowchart of a federated learning-based cardiac ultrasound image processing method according to an embodiment of this application, with each client acting as the execution entity. The method includes the following steps: S101, obtain the global federation model issued by the server; S102, the global federated model is used as a local sub-model, the local sub-model is trained on the local sample set, and the image processing result is output. S103, During the training process, local prototype parameters are constructed and uploaded to the server; S104, obtain the global prototype parameters generated by the server based on the local prototype parameters of each client, and calculate the loss function based on the image processing result and the global prototype parameters; S105, Update the sub-model parameters of the local sub-model based on the loss function, and upload the updated sub-model parameters to the server; S106, The server performs global model aggregation on the sub-model parameters of each client to construct a new global federated model; S107, The server determines whether the new global federation model has converged; S108 uses the converged global federated model to process the target cardiac ultrasound image, obtaining the image processing result.
[0026] In some embodiments, the global federated model in S101 can be pre-built on the server side, for example, using the ViT deep learning model as the base model as the global federated model for the processing task of fetal cardiac ultrasound images.
[0027] In some embodiments, in S102, the server can distribute the global federated model to the client of each institution (e.g., a hospital). Each client can receive the global federated model, use the global federated model as a local sub-model, and train the local sub-model based on the local sample set.
[0028] Specifically, each client can build a local sample set based on historical echocardiogram image data. The local sample set construction process includes: Acquire multiple local sample cardiac ultrasound images; Obtain the image type of each sample cardiac ultrasound image and extract various medical concept parameters corresponding to each sample cardiac ultrasound image; The image type label for each sample of cardiac ultrasound image is obtained based on the image type, and the medical concept parameter label for each sample of cardiac ultrasound image is obtained based on the various medical concept parameters. The local sample set is constructed based on the multiple sample cardiac ultrasound images, the corresponding image type labels, and the medical concept parameter labels.
[0029] The image type label can be understood as the classification result determined after analyzing the cardiac ultrasound image, including normal type and various congenital heart disease subtypes. Congenital heart disease subtypes include, but are not limited to, tricuspid valve hypoplasia, hypoplastic ventricle, etc. This application embodiment does not limit this. The image type label can indicate whether the corresponding image has normal features or abnormal features such as tricuspid valve hypoplasia, hypoplastic ventricle, etc. This application embodiment does not limit this.
[0030] It should be noted that medical concept parameters can be understood as the extraction results of relevant medical parameters based on cardiac ultrasound images. The types of medical parameters include, but are not limited to, the number of heart chambers, the number of blood flow bands, and blood flow symmetry. The parameter values corresponding to the number of heart chambers, the number of blood flow bands, and blood flow symmetry can be easily extracted by analyzing cardiac ultrasound images, and then used as the medical concept parameters.
[0031] Specifically, generating medical concept parameter labels based on the medical concept parameters corresponding to each image can be understood as follows: comparing the value of each type of medical concept parameter corresponding to each image with the standard value of the corresponding type of medical concept parameter, performing binarization processing on the medical concept parameter according to the comparison result, and using the binarization result (0 or 1) as the medical concept parameter label; wherein, the standard value of each type of medical concept parameter can be determined based on medical common sense in the relevant field.
[0032] For example, under normal circumstances, a standard cardiac ultrasound image should show more than two ventricles and a double-beam, symmetrical blood flow pattern. If these standard conditions are not met, it indicates that there are abnormalities in the relevant parameters. Medical concept parameters that do not meet the standards are binarized to the value 1, and medical concept parameters that meet the standards are binarized to the value 0. Based on the binarization results of each medical concept parameter, a medical concept parameter label is constructed.
[0033] For example, each type of medical concept parameter can be binarized separately. Taking the number of heart chambers as an example, if the number of heart chambers is less than the standard value of the number of heart chambers, then the medical concept parameter of the number of heart chambers is binarized to 0; otherwise, it is binarized to 1.
[0034] Specifically, the constructed local sample set can be represented as: ; Where k represents the kth client (hospital). This represents the total number of cardiac ultrasound images for the k-th client. This represents the echocardiogram image of the i-th sample from the k-th client. For the image type label of the i-th sample cardiac ultrasound image of the k-th client, This represents the medical concept parameter label corresponding to the i-th sample cardiac ultrasound image of the k-th client. Specifically, The medical concept parameter labels include the binarized results of each type of medical concept parameter (including the number of heart chambers, the number of blood flow bands, and blood flow symmetry).
[0035] For example, if there are G-class medical concept parameters, then It can be represented as: ; in, This represents the binarization result of the g-th medical concept parameter corresponding to the i-th sample cardiac ultrasound image.
[0036] It should be noted that the data from different hospitals exhibits heterogeneity, specifically demonstrating a non-independent and identically distributed (NonIID) property. Specifically, due to differences in imaging equipment, terminology, and staff experience among hospitals, medical CHD data distributed across different hospitals naturally exhibit a heterogeneous distribution. This cross-silo distribution shift significantly hinders the convergence speed of the global federated model and further reduces the effectiveness of CHD detection.
[0037] This application embodiment can provide a powerful federal medical signal by binarizing each type of medical concept parameter during the sample set construction process, so as to unify the measurement standards of medical concept parameters in different hospitals, perceive the representative pattern of potential congenital heart disease across hospitals, and thus calibrate client drift.
[0038] Specifically, in S102, the local sub-model can process each sample cardiac ultrasound image in the local sample set separately, based on the prediction results of the sample cardiac ultrasound images. The prediction results include a local sub-model based on sample cardiac ultrasound images. Image type prediction results obtained from processing and medical concept parameter prediction results .
[0039] Image type prediction results This can be understood as the subtype of congenital heart disease in patients as reflected in the sample echocardiogram images, and the prediction results of medical conceptual parameters. It can be understood as the binarized values of various medical concept parameters (such as the number of heart chambers, the number of blood flow bands, and blood flow symmetry) predicted by the local sub-model based on sample cardiac ultrasound images.
[0040] In some embodiments, the local submodel is based on a local sample set. During training, local prototype parameters can be constructed. S103 includes: S1031, using a local sub-model to analyze the cardiac ultrasound images of each sample. Feature extraction is performed separately to obtain image feature vectors. and medical concept parameter feature vector .
[0041] Specifically, medical concept parameter feature vectors can be extracted using a concept embedding architecture, applying the formula: ; in, This represents a function for extracting feature vectors of medical concept parameters. Represents the range of the real number field, The dimension of the concept embedding vector. This represents the medical concept parameter feature vector obtained by client k based on the i-th sample cardiac ultrasound image.
[0042] Specifically, the local prototype parameters include medical concept prototype parameters and image feature prototype parameters. Medical concept prototype parameters can be constructed based on the feature vector of the medical concept parameters, specifically including: S1032, Obtain the initial prototype parameters corresponding to each type of medical concept parameter. and initial negative prototype parameters ; S1033, calculate the initial prototype parameters corresponding to the medical parameter feature vectors for each type of medical concept parameter. The Euclidean distance and the initial negative prototype parameters corresponding to each type of medical concept parameter. Euclidean distance: , ; S1034 can convert the feature vector of each medical parameter into a single feature vector. Euclidean distance to the initial prototype parameters and the Euclidean distance to the initial negative prototype parameter. The difference is input into the sigmoid function to obtain the probability that the corresponding medical concept parameter meets the standard: ; S1035, based on the feature vector of each medical parameter The corresponding probability values update the initial prototype parameters above. and initial negative prototype parameters ; S1036, obtained from the client The prototype parameters corresponding to each type of medical concept parameter learned and negative prototype parameters Output the prototype parameters of the medical concept.
[0043] Understandably, the positive prototype parameter can be regarded as the average representation of the standard conditions for each type of medical concept parameter corresponding to each sample cardiac ultrasound image in the current hospital's sample set. Similarly, the negative prototype parameter can be regarded as the average representation of the standard conditions for each type of medical concept parameter corresponding to each sample cardiac ultrasound image in the current hospital's sample set.
[0044] Simultaneously, the following steps can be performed based on each image feature vector. Constructing image feature prototype parameters : S1037, Prediction results based on image type Cluster the image feature vectors for each image type, and obtain the prototype parameters of the image features for hospital k corresponding to image type m based on the average of all image feature vectors for each image type. Apply the formula: ; in, This indicates that the prediction within hospital k is the first... The total number of image feature vectors of image type; express The i-th image feature vector Image feature vectors of a sample cardiac ultrasound image.
[0045] Further, step S1038 is executed to upload the extracted concept prototype parameters and image prototype parameters to the server.
[0046] Next, each hospital's client k uploads its local prototype parameters to the server via the network. The server then aggregates the local prototype parameters from each client to obtain the global prototype parameters. The global prototype parameters include global medical concept prototype parameters (including global positive prototype parameters and global negative prototype parameters) and global image feature prototype parameters. The specific steps are as follows: S201, aggregating each client The prototype parameters corresponding to each type of medical concept parameter learned This yields the global positive prototype parameters of the corresponding type of medical concept parameters. Apply the formula: ; Aggregate each client The negative prototype parameters corresponding to the prototype parameters of each type of medical concept learned This yields the global negative prototype parameters of the corresponding type of medical concept parameters. Apply the formula: ; S202, aggregate the image feature prototype parameters learned by each client for each image type to obtain the global image feature prototype parameters for each image type. Apply the formula: ; Where K represents the total number of clients (the total number of hospitals participating in federated learning to train the global federated model).
[0047] S203, the server sends the global prototype parameters, including the global negative prototype parameters and global positive prototype parameters for each type of medical concept parameter, as well as the global image feature prototype parameters corresponding to each image type, to each client.
[0048] In some embodiments, in S104, each client can receive global prototype parameters sent by the server and calculate the loss function by combining the global prototype parameters with the prediction results based on the sample cardiac ultrasound images in S102.
[0049] Specifically, the loss function includes several loss terms: default loss term, concept regularization loss term, federal clinical concept alignment loss term, and federal disease prototype contrast loss term.
[0050] Specifically, the calculation process for the default loss term uses the following formula: ; ; in, This is a pre-set structural mask used during the construction of the local sample set for S102. Specifically, it is a mask for key fetal heart structures, which can be obtained through manual annotation based on experience.
[0051] This represents the image type prediction result based on the local sub-model. relative to image type label Cross-entropy loss, This represents the function used to calculate the cross-entropy loss. This represents the function used to compute the attention map.
[0052] Specifically, the calculation process for the concept regularization loss term uses the following formula:
[0053] in, This represents the prediction results of medical concept parameters calculated using the local sub-model. Binary cross-entropy loss with medical concept parameter labels; It is a positive number whose value is less than the threshold, used to prevent the denominator from being 0.
[0054] Specifically, the calculation process for the Federated Clinical Conceptual Alignment (FCCA) uses the following formula: ; Where G represents the set of types of medical concept parameters.
[0055] Specifically, the calculation process for the federal disease prototype comparison loss term includes: For each image feature vector Normalization is performed to obtain the normalized image feature vector. : ; Normalize each image feature prototype parameter to obtain the normalized image feature prototype vector. : ; in, This represents the prototype parameter of the m-th image feature, and each prototype parameter corresponds to an image type.
[0056] The vector similarity between the normalized image feature vector and the image feature prototype vector for each image type can be calculated: ; in, This represents the temperature coefficient, and the subscript m indicates the image type.
[0057] Furthermore, the contrast loss is calculated based on similarity:
[0058] in, This indicates that an exponentiation operation is being performed.
[0059] Indicates: Based on current sample cardiac ultrasound images Image feature vectors and current image type labels The similarity of the prototype parameters of the image features of all other types of images that are not the same is calculated separately, and the similarity exponent values are summed.
[0060] This indicates the similarity calculated between the image feature vector of the current sample cardiac ultrasound image and the image feature prototype parameters of the image type corresponding to the current image type label.
[0061] Furthermore, the Federated Disease Prototype Contrast (FDPC) term is calculated based on the contrast loss of each sample's echocardiogram image, using the following formula: ; Then, a weighted sum is performed on each loss term to obtain the total loss function, using the formula: ; Here, γ is a balancing hyperparameter used to control the relative contribution of image feature prototype contrastive learning.
[0062] Further, step S105 is executed, backpropagation is performed based on the value of the loss function, the sub-model parameters of the local sub-model are updated, the updated sub-model parameters are obtained, and the sub-model parameters are uploaded to the server.
[0063] Further, in step S106, the server receives the sub-model parameters uploaded by each client k, and uses a federated averaging algorithm to aggregate the sub-model parameters uploaded by each client k in the global model, thereby constructing a new global federated model.
[0064] Understandably, after completing steps S101-S106 above, one round of training is finished. Step S107 can then be executed on the server side to determine whether the new global federated model has converged. This determination can be based on the validation set and includes the following steps: Input the validation set into the new global federated model, and calculate the F1 score, sensitivity, and specificity based on the processing results output by the new global federated model, applying the following formulas respectively: ; ; ; In this context, TP represents true positive, TN represents true negative, FP represents false positive, and FN represents false negative. Precision represents accuracy, and Recall represents recall (sensitivity). A true positive means the validation sample's true label is positive, and the model's detection result also indicates a positive result; a true negative means the validation sample's true label is negative, and the model's detection result also indicates a negative result; a false positive means the validation sample's true label is negative, but the model's detection result incorrectly indicates a positive result; a false negative means the validation sample's true label is positive, but the model's detection result incorrectly indicates a negative result.
[0065] Therefore, it can be determined whether the model has converged. If the model has not converged, proceed to step S101 to distribute the new global federated model to each client and start the next round of training until step S107 determines that the model has converged.
[0066] If the model is determined to be converged, then step S108 is executed. Each hospital can use the converged trained model to process the target cardiac ultrasound image, so that the trained global federated model can obtain the image processing result based on the input target cardiac ultrasound image, and give the image type classification result and medical concept parameter analysis result.
[0067] It should be noted that the (n+1)th iteration can update the local medical concept prototype parameters based on the global medical concept prototype parameters obtained in the nth iteration, using the following formula: ; ; in, This is the momentum factor.
[0068] Based on the above formulas, the positive and negative prototype parameters updated based on the global medical concept prototype parameters are obtained respectively, and used in the calculation of the (n+1)th iteration, specifically: In step S1032 of the (n+1)th iteration, the positive prototype parameter updated based on the global medical concept prototype parameter can be used as the initial positive prototype parameter. Similarly, the negative prototype parameter obtained from the global update can be used as the initial negative prototype parameter.
[0069] It should be noted that during the first iteration, initial positive prototype parameters and initial negative prototype parameters can be pre-generated based on the current sample set.
[0070] In one specific embodiment, the federated model can be trained based on hospitals from four different regions in China, and experiments can be conducted on real first-trimester CHD datasets collected from each hospital: Specifically, these include Hospital A (North China), Hospital B (Central China), Hospital C (XH, South China), and Hospital D (South China). The data is based on a total of 5108 fetal echocardiogram images, covering categories such as normal, atrial and ventricular septal defect (AVSD), single ventricle (FSV), and hypoplastic ventricle (HV).
[0071] To simulate the federated learning process in a real-world scenario, this application's embodiment uses a four-hospital federated training setup for evaluation: Client distribution: Hospital A (1278 samples) - Hospital B (1452 samples) - Hospital C (1189 samples) - Hospital D (1189 samples) By inputting data from different hospitals, the model's adaptability to heterogeneous medical data was tested. To further evaluate the model's generalization ability, zero-shot testing was also conducted on a cross-hospital test set. This approach comprehensively verifies the method's robustness and knowledge dissemination efficiency in real-world medical environments.
[0072] To verify the effectiveness of the embodiments of this application, the embodiments of this application are compared with existing federated learning methods, which mainly include: FedAvg, MOON, FedOPT, FedProc, FedDyn, FedProx, and FedPL.
[0073] The federated model was tested based on datasets from four hospitals. The F1-score, sensitivity, and specificity results are shown in Table 1.
[0074]
[0075] Among them, Overall represents the global average result, Normal represents the test result under the normal classification, and AVSD, FSV and HV represent the test results of three subtypes of congenital heart disease: Atrioventricular Septal Defect, Functional Single Ventricle and Hypoplastic Ventricles, respectively.
[0076] As can be seen from the table, the embodiments of this application demonstrate superior performance in federal CHD diagnostic tasks. In terms of overall F1 score, the embodiments of this application achieve a 3.5% improvement over the existing best method, FedProx (0.766 vs 0.740). In the difficult category HV, the embodiments of this application show a significant improvement over existing methods, increasing from the best 0.542 to 0.635, an improvement of 17.1%. Training curves show that it maintains stable performance growth in cross-hospital testing, indicating that the model of the embodiments of this application has stronger generalization and consistency.
[0077] The above data demonstrates that the embodiments of this application, through a global knowledge propagation strategy based on federated concept alignment, can adaptively integrate clinical concept understanding across hospitals (aligning various medical concept parameters), avoiding the limitations of concept drift. The embodiments of this application, through comparative learning of global image feature prototype parameters, force alignment of image feature spaces across hospitals, effectively improving the model's image classification ability (i.e., determining the disease type reflected in the ultrasound image through the model's output classification results), while simultaneously enhancing intra-class consistency and inter-class discriminability of the feature space.
[0078] The following are apparatus embodiments of this application, which can be used to execute the method embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of this application.
[0079] Please see below. Figure 2 The following is a schematic diagram of the structure of a cardiac ultrasound image processing device based on federated learning, which is an exemplary embodiment of this application. The device includes a client and a server. The client is used to obtain the global federation model issued by the server; The client is also used to train the local sub-model on the local sample set and output the prediction results by using the global federated model as a local sub-model. The client is also used to construct local prototype parameters during training and upload the local prototype parameters to the server. The client is also used to obtain global prototype parameters generated by the server based on the aggregation of local prototype parameters of all clients, and to calculate a loss function based on the prediction results and the global prototype parameters; The client is also used to update the sub-model parameters of the local sub-model based on the loss function, and upload the updated sub-model parameters to the server; The server is also used to perform global model aggregation on the sub-model parameters of each client to construct a new global federated model; The server is also used to determine whether the new global federated model has converged. The client uses the converged global federated model to process the target cardiac ultrasound image and obtain the image processing result.
[0080] It should be noted that the apparatus provided in the above embodiments, when executing a federated learning-based cardiac ultrasound image processing method, is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their implementation process is detailed in the method embodiments, which will not be repeated here.
[0081] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.
[0082] Please see Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of this application.
[0083] like Figure 3 As shown, the electronic device includes a processor and a memory.
[0084] In this embodiment, the processor is the control center of the computer system, and can be a processor of a physical machine or a processor of a virtual machine. The processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor can be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array).
[0085] A processor can also include a main processor and a coprocessor. The main processor is used to process data in the wake-up state and is also called the CPU (Central Processing Unit). The coprocessor is a low-power processor used to process data in the standby state.
[0086] The memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments of this application, the non-transitory computer-readable storage media in the memory are used to store at least one instruction, which is executed by a processor to implement the methods in the embodiments of this application.
[0087] In some embodiments, the electronic device further includes a peripheral device interface and at least one peripheral device. The processor, memory, and peripheral device interface are connected via a bus or signal line. Each peripheral device is connected to the peripheral device interface via a bus, signal line, or circuit board. Specifically, the peripheral device includes: a display screen, a camera, and audio circuitry. The peripheral device interface can be used to connect at least one I / O (Input / Output) related peripheral device to the processor and memory.
[0088] In some embodiments of this application, the processor, memory, and peripheral device interfaces are integrated on the same chip or circuit board; in other embodiments of this application, any one or two of the processor, memory, and peripheral device interfaces can be implemented on separate chips or circuit boards. This application does not specifically limit the implementation in this regard.
[0089] The electronic device structural block diagrams shown in the embodiments of this application do not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or use different component arrangements.
[0090] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the methods in any of the foregoing embodiments. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.
[0091] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A method for processing cardiac ultrasound images based on federated learning, characterized in that, Applied to a client, the method includes: Retrieve the global federated model issued by the server; The global federated model is used as a local sub-model, and the local sub-model is trained on the local sample set to output the prediction results; During the training process, local prototype parameters are constructed and uploaded to the server. Obtain the global prototype parameters generated by the server based on the aggregation of local prototype parameters of all clients, and calculate the loss function based on the prediction results and the global prototype parameters; The sub-model parameters of the local sub-model are updated based on the loss function, and the updated sub-model parameters are uploaded to the server. The server performs global model aggregation on the sub-model parameters of each client to construct a new global federated model; The server determines whether the new global federated model has converged, and the converged global federated model is used to process the target cardiac ultrasound image to obtain the image processing result.
2. The method for processing cardiac ultrasound images based on federated learning according to claim 1, characterized in that, The process of constructing the local sample set includes: Acquire multiple local sample cardiac ultrasound images; Obtain the image type of each sample cardiac ultrasound image and extract various medical concept parameters corresponding to each sample cardiac ultrasound image; The image type label for each sample of cardiac ultrasound image is obtained based on the image type, and the medical concept parameter label for each sample of cardiac ultrasound image is obtained based on the various medical concept parameters. The local sample set is constructed based on the multiple sample cardiac ultrasound images, the corresponding image type labels, and the medical concept parameter labels.
3. The method for processing cardiac ultrasound images based on federated learning according to claim 2, characterized in that, The process of training a local sub-model on a local sample set and outputting prediction results includes: Each sample echocardiogram image in the local sample set is processed by a local sub-model to obtain prediction results for the sample echocardiogram image, including image type prediction results and medical concept parameter prediction results. During the processing, the local sub-model is used to extract features from each sample of cardiac ultrasound image to obtain an image feature vector. Each sample of echocardiogram image is input into a function used to extract medical concept parameter feature vectors to obtain the medical concept parameter feature vectors.
4. The method for processing cardiac ultrasound images based on federated learning according to claim 3, characterized in that, The local prototype parameters include medical concept prototype parameters and image feature prototype parameters; The process of constructing local prototype parameters during training includes: Based on the feature vector of the medical concept parameters, medical concept prototype parameters are constructed, including positive prototype parameters and negative prototype parameters corresponding to each type of medical concept prototype parameters. Based on the image type prediction results, the image feature vectors of each image type are clustered. Based on the average value of all image feature vectors under each image type, the prototype parameters of the image features of the current client corresponding to each image type are obtained.
5. The method for processing cardiac ultrasound images based on federated learning according to claim 4, characterized in that, The steps for obtaining the global prototype parameters generated by the server based on the aggregation of local prototype parameters from all clients are as follows: Obtain the local prototype parameters for each client on the server side; By aggregating the prototype parameters corresponding to each type of medical concept parameter learned by each client through the server, the global prototype parameters of the corresponding type of medical concept parameter are obtained. By aggregating the negative prototype parameters corresponding to the prototype parameters of each type of medical concept learned by each client through the server, the global negative prototype parameters of the corresponding type of medical concept parameters are obtained. By aggregating the image feature prototype parameters learned by each client for each image type on the server, the global image feature prototype parameters for each image type are obtained. The global prototype parameters are distributed to each client via the server. The global prototype parameters include global negative prototype parameters and global positive prototype parameters for each type of medical concept parameter, as well as global image feature prototype parameters corresponding to each image type.
6. The method for processing cardiac ultrasound images based on federated learning according to claim 5, characterized in that, The loss function includes a default loss term, a concept regularization loss term, a federalized clinical concept alignment loss term, and a federalized disease prototype comparison loss term. The total loss function is obtained by weighted summation of each loss term, using the formula: ； in, Denotes the loss function, This refers to the default loss term. This represents the regularization loss term of the concept. This indicates the alignment loss term of the federal clinical concept. This represents the loss term compared to the federal disease prototype.
7. The method for processing cardiac ultrasound images based on federated learning according to claim 6, characterized in that, The calculation process for the default loss term uses the following formula: ； ； in, This refers to the default loss term; This represents a function for calculating cross-entropy loss; The image type prediction result for the i-th sample cardiac ultrasound image of the k-th client; The image type label for the i-th sample cardiac ultrasound image of the k-th client; This represents the total number of sample cardiac ultrasound images for the k-th client; This is the image feature vector corresponding to the i-th sample cardiac ultrasound image of the k-th client; The structure mask is preset during the construction of the local sample set; This represents the function used to compute the attention map.
8. A cardiac ultrasound image processing device based on federated learning, characterized in that, Includes client and server: The client is used to obtain the global federation model issued by the server; The client is also used to train the local sub-model on the local sample set and output the prediction results by using the global federated model as a local sub-model. The client is also used to construct local prototype parameters during training and upload the local prototype parameters to the server. The client is also used to obtain global prototype parameters generated by the server based on the aggregation of local prototype parameters of all clients, and to calculate a loss function based on the prediction results and the global prototype parameters; The client is also used to update the sub-model parameters of the local sub-model based on the loss function, and upload the updated sub-model parameters to the server; The server is also used to perform global model aggregation on the sub-model parameters of each client to construct a new global federated model; The server is also used to determine whether the new global federated model has converged. The client uses the converged global federated model to process the target cardiac ultrasound image and obtain the image processing result.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.