Training method and device of federal learning medical model based on cross-modality

Abstract

The invention provides a training method and device for a federated learning medical model based on cross-modality, and relates to the technical field of computers, and the method comprises the steps: inputting single-modality medical data into a feature extraction layer of an initial global model, so as to generate a single-modality feature vector; traversing in a public database based on the single-mode feature vector to determine an opposite-end retrieval feature vector corresponding to the single-mode feature vector; generating a conversion feature vector according to the single-mode feature vector and the disease prompt feature vector; fusing the opposite-end retrieval feature vector and the conversion feature vector to generate a target opposite-end feature vector; inputting the single-mode feature vector and the target opposite-end feature vector into an encoder of an initial global model, and determining a disease prediction type through the processing of a classifier; training an encoder and a classifier of the initial global model according to the difference between the disease prediction type and the disease marking type to generate a target global model; and sending the target global model to a server.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a training method and device for a federated learning medical model based on cross-modal. BACKGROUND

[0002] In real medical scenarios, due to sensor failures, inconsistent data collection, privacy regulations, and the like, each medical institution, which can also be referred to as a client, often only holds partial modal data, such as only image data or only text data, and it is difficult to form complete multi-modal data. When training a medical model using only the partial modal data, the model training is insufficient, and the performance is significantly reduced.

[0003] In related technologies, missing data is usually replaced by retrieved data, but this method easily loses key semantic information of the original available modal, or missing data can also be supplemented by a simple filling strategy, but this method introduces semantic noise, causing the model optimization direction to deviate from the global optimum, further increasing the model bias. Therefore, under the condition of modal loss, how to improve the accuracy of medical model training is crucial. SUMMARY

[0004] The present application provides a training method and device for a federated learning medical model based on cross-modal.

[0005] According to a first aspect of the present application, a training method for a federated learning medical model based on cross-modal is provided, which comprises:

[0006] inputting single-modal medical data into a feature extraction layer of an initial global model to generate a single-modal feature vector, wherein the single-modal medical data is medical image data or medical text data, and the initial global model is initialized by a server and then distributed to a client;

[0007] based on the single-modal feature vector, performing traversal in a public database to determine a peer retrieval feature vector corresponding to the single-modal feature vector, wherein the public database includes vector pairs of image feature vectors and text feature vectors;

[0008] generating a conversion feature vector according to the single-modal feature vector and a disease prompt feature vector;

[0009] fusing the peer retrieval feature vector and the conversion feature vector to generate a target peer feature vector;

[0010] inputting the single-modal feature vector and the target peer feature vector into an encoder of the initial global model to determine a disease prediction type through processing of a classifier;

[0011] Based on the difference between the disease prediction type and the disease labeling type, the encoder and classifier of the initial global model are trained to generate the target global model;

[0012] The target global model is sent to the server, so that the server can aggregate and generate a global model based on the target global model uploaded by each client.

[0013] Optionally, the step of traversing a public database based on the single-modal feature vector to determine the peer retrieval feature vector corresponding to the single-modal feature vector includes:

[0014] The single-modal feature vector is matched with the same-end feature vectors in the public database to determine the similarity.

[0015] The counterpart feature vector in the vector pair to which the highest similarity-corresponding same-end feature vector belongs is determined as the counterpart retrieval feature vector corresponding to the single-modal feature vector.

[0016] Optionally, determining the counterpart feature vector in the vector pair to which the highest similarity-corresponding same-end feature vector belongs as the counterpart retrieval feature vector corresponding to the single-modal feature vector includes:

[0017] When there are multiple highest similarity values, the counterpart feature vectors in the vector pairs to which the same-end feature vectors corresponding to each highest similarity value belong are fused to generate the counterpart retrieval feature vector corresponding to the single-modal feature vector.

[0018] Optionally, before traversing the public database based on the single-modal feature vector to determine the peer retrieval feature vector corresponding to the single-modal feature vector, the method further includes:

[0019] The raw medical data is processed to generate triplet data, wherein the triplet data includes medical images, text reports, and disease tags;

[0020] Feature extraction is performed on the triplet data to generate vector pairs corresponding to the triplets, wherein the image feature vectors and text feature vectors in the vector pairs have semantic consistency.

[0021] Optionally, generating a transformation feature vector based on the single-modal feature vector and the disease indication feature vector includes:

[0022] The single-modal feature vector is matched with each of the disease indication feature vectors to determine the degree of matching between the single-modal feature vector and each of the disease indication feature vectors;

[0023] Based on the matching degree, the single-modal feature vector is updated to generate the target single-modal feature vector;

[0024] The target single-modal feature vector is processed to generate a transformed feature vector.

[0025] Optionally, updating the unimodal feature vector based on the matching degree to generate a target unimodal feature vector includes:

[0026] When the single-modality feature vector includes multiple sub-feature vectors, the sub-feature vectors are matched with each of the disease indication feature vectors to determine the matching degree.

[0027] Based on the matching degree, an attention weight matrix is ​​generated;

[0028] Based on the attention weight matrix, the disease perception feature vectors associated with each disease prompt feature vector are fused to generate a target sub-feature vector;

[0029] The single-modal feature vector is updated based on the target sub-feature vector to generate the target single-modal feature vector.

[0030] Optionally, processing the target single-modal feature vector to generate a transformed feature vector includes:

[0031] The target single-modal feature vector is input into the cross-modal converter, and after processing by the cross-modal converter, a transformed feature vector is generated.

[0032] According to a second aspect of this application, a training apparatus for a cross-modal federated learning healthcare model is provided, comprising:

[0033] The first generation module is used to input single-modal medical data into the feature extraction layer of the initial global model to generate single-modal feature vectors. The single-modal medical data is medical image data or medical text data. The initial global model is initialized by the server and then sent to the client.

[0034] The first determining module is used to traverse a public database based on the single-modal feature vector to determine the peer retrieval feature vector corresponding to the single-modal feature vector, wherein the public database includes vector pairs composed of image feature vectors and text feature vectors;

[0035] The second generation module is used to generate a transformation feature vector based on the single-modal feature vector and the disease indication feature vector;

[0036] The third generation module is used to fuse the peer-to-peer retrieval feature vector with the transformed feature vector to generate the target peer-to-peer feature vector;

[0037] The second determining module is used to input the single-modal feature vector and the target counterpart feature vector into the encoder of the initial global model, so as to determine the disease prediction type after processing by the classifier;

[0038] The fourth generation module is used to train the encoder and classifier of the initial global model based on the difference between the disease prediction type and the disease labeling type, so as to generate the target global model.

[0039] The sending module is used to send the target global model to the server, so that the server can aggregate and generate a global model based on the target global model uploaded by each client.

[0040] Optionally, the first determining module includes:

[0041] The first determining unit is used to match the single-modal feature vector with the same-end feature vectors in the public database to determine the similarity.

[0042] The first determining unit is used to determine the counterpart feature vector in the vector pair to which the highest similarity-corresponding same-end feature vector belongs as the counterpart retrieval feature vector corresponding to the single-modal feature vector.

[0043] Optionally, the first determining unit is specifically used for:

[0044] When there are multiple highest similarity values, the counterpart feature vectors in the vector pairs to which the same-end feature vectors corresponding to each highest similarity value belong are fused to generate the counterpart retrieval feature vector corresponding to the single-modal feature vector.

[0045] Optionally, the first determining module is further configured to:

[0046] The raw medical data is processed to generate triplet data, wherein the triplet data includes medical images, text reports, and disease tags;

[0047] Feature extraction is performed on the triplet data to generate vector pairs corresponding to the triplets, wherein the image feature vectors and text feature vectors in the vector pairs have semantic consistency.

[0048] Optionally, the second generation module includes:

[0049] The second determining unit is used to match the single-modal feature vector with each of the disease indication feature vectors to determine the degree of matching between the single-modal feature vector and each of the disease indication feature vectors;

[0050] An update unit is used to update the single-modal feature vector according to the matching degree to generate a target single-modal feature vector;

[0051] The generation unit is used to process the target single-modal feature vector to generate a transformed feature vector.

[0052] Optionally, the update unit is specifically used for:

[0053] When the single-modality feature vector includes multiple sub-feature vectors, the sub-feature vectors are matched with each of the disease indication feature vectors to determine the matching degree.

[0054] Based on the matching degree, an attention weight matrix is ​​generated;

[0055] Based on the attention weight matrix, the disease perception feature vectors associated with each disease prompt feature vector are fused to generate a target sub-feature vector;

[0056] The single-modal feature vector is updated based on the target sub-feature vector to generate the target single-modal feature vector.

[0057] Optionally, the generation unit is specifically used for:

[0058] The target single-modal feature vector is input into the cross-modal converter, and after processing by the cross-modal converter, a transformed feature vector is generated.

[0059] According to a third aspect of this application, an electronic device is provided, comprising: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements any of the above-described training methods for a cross-modal federated learning medical model.

[0060] According to a fourth aspect of this application, a computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed by a processor, implement any of the above-described training methods for a cross-modal federated learning medical model.

[0061] In summary, the training method and apparatus for a cross-modal federated learning medical model provided in this application have at least the following beneficial effects: First, single-modal medical data can be input into the feature extraction layer of an initial global model to generate single-modal feature vectors. The single-modal medical data can be medical image data or medical text data. The initial global model is initialized by the server and then distributed to the client. Subsequently, based on the single-modal feature vectors, a public database can be traversed to determine the corresponding peer-to-peer feature vectors. The public database includes vector pairs consisting of image feature vectors and text feature vectors. Then... Based on the unimodal feature vector and disease suggestion feature vector, a transformed feature vector is generated. Then, the peer retrieval feature vector is fused with the transformed feature vector to generate the target peer feature vector. Next, the unimodal feature vector and the target peer feature vector are input into the encoder of the initial global model. After processing by the classifier, the disease prediction type is determined. Then, based on the difference between the disease prediction type and the disease labeling type, the encoder and classifier of the initial global model are trained to generate the target global model. The target global model is sent to the server, where the server aggregates the target global models uploaded by each client to generate the global model. Therefore, for unimodal medical data lacking another modality, the disease suggestion feature vector can be used to generate a transformed feature vector that retains the key semantic knowledge of the existing modality. This transformed feature vector is then fused with the peer retrieval feature vector determined through a public database to generate the target peer feature vector. This reduces information loss and noise problems caused by filling in missing modalities, thereby improving the performance of federated learning medical models in the case of missing modalities. Attached Figure Description

[0062] To more clearly illustrate the specific embodiments of this application or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 A flowchart illustrating a training method for a cross-modal federated learning healthcare model provided for embodiments of this application;

[0064] Figure 2 A schematic diagram illustrating the training process of a cross-modal federated learning healthcare model provided for an embodiment of this application;

[0065] Figure 3 A structural diagram of a training device for a cross-modal federated learning healthcare model provided for embodiments of this application;

[0066] Figure 4 This is a structural diagram of an electronic device provided as an embodiment of the present application. Detailed Implementation

[0067] To make the above and other features and advantages of this application clearer, the application is further described below with reference to the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explanation to those skilled in the art, and are exemplary only, not restrictive.

[0068] In the following description, numerous specific details are set forth to provide a thorough understanding of this application. However, it will be apparent to those skilled in the art that the specific details are not required to practice this application. In other instances, well-known steps or operations have not been described in detail to avoid obscuring this application.

[0069] The training method for a cross-modal federated learning healthcare model provided in this application embodiment can be executed by the training device for a cross-modal federated learning healthcare model provided in this application embodiment, which can be configured in an electronic device.

[0070] refer to Figure 1 This application provides a training method for a cross-modal federated learning healthcare model, the method comprising:

[0071] Step 101: Input the single-modal medical data into the feature extraction layer of the initial global model to generate a single-modal feature vector. The single-modal medical data is medical image data or medical text data. The initial global model is initialized by the server and then sent to the client.

[0072] It is understandable that, due to sensor malfunctions, inconsistent data collection, and privacy regulations, each client often only holds partial modal medical data, such as only medical image data or only medical text data. Therefore, the single-modal medical data in the embodiments of this application may be medical image data but lacking corresponding medical text data, or medical text data but lacking corresponding medical image data; this application does not limit this.

[0073] In addition, due to the differences in data distribution among clients in a federated learning environment, if the feature extraction layer is continuously updated, different clients may adjust their feature spaces toward their local optimal direction, which may lead to feature space misalignment during aggregation and thus affect model convergence. Therefore, the feature extraction layer in the initial global model provided in this application is in a frozen state.

[0074] Step 102: Based on the single-modal feature vector, traverse the public database to determine the corresponding peer retrieval feature vector. The public database includes vector pairs consisting of image feature vectors and text feature vectors.

[0075] The public database stores a large number of vector pairs, each containing different modal representations of the same medical entity. That is, the image feature vector and the text feature vector in each vector pair have semantic consistency. For example, the image feature vector and the text feature vector in a certain vector pair may both represent "inflammation", or the image feature vector and the text feature vector in a certain vector pair may both represent "hand fracture", etc. This application does not limit this.

[0076] Furthermore, the peer-to-peer retrieval feature vector can be understood as a feature vector of a different modality from the unimodal feature vector. For example, if the unimodal medical data is medical image data, then the unimodal feature vector is the medical image feature vector, and the peer-to-peer retrieval feature vector is the medical text feature vector; or, if the unimodal medical data is medical text data, then the unimodal feature vector is the medical text feature vector, and the peer-to-peer retrieval feature vector is the medical image feature vector. This application does not limit this.

[0077] Optionally, the raw medical data can be processed first to generate triple data, which includes medical images, text reports, and disease tags. Then, features can be extracted from the triple data to generate vector pairs corresponding to the triples, where the image feature vectors and text feature vectors in the vector pairs have semantic consistency.

[0078] Raw medical data can be obtained through any viable means or method, such as obtaining it from public datasets or through questionnaires. This raw medical data can then be cleaned and preprocessed to generate triplet data including medical images, text reports, and disease labels. This triplet data can then be stored in a public database, which can be represented as D. pub =(x i ,x t ,y), where x i For medical images, x t For text or diagnostic reports, y represents the disease label. When extracting features from triplet data, a frozen feature extraction layer can be used to process the data to generate vector pairs consisting of image feature vectors and text feature vectors. The image feature vectors and text feature vectors in each vector pair have semantic consistency and can represent the same medical fact or the same medical case, etc.

[0079] Optionally, the unimodal feature vector can be matched with the same-end feature vectors in the public database to determine the similarity. Then, the opposite feature vector in the vector pair to which the same-end feature vector with the highest similarity belongs can be determined as the opposite retrieval feature vector corresponding to the unimodal feature vector.

[0080] In this context, the same-end feature vector can be understood as a feature vector belonging to the same modality as the single-modal feature vector, while the opposite-end feature vector can be understood as a feature vector belonging to a different modality than the single-modal feature vector. For example, if the single-modal feature vector is a medical image feature vector, then the same-end feature vector is the image feature vector in the public database, and the opposite-end feature vector is the text feature vector in the public database; or, if the single-modal feature vector is a medical text feature vector, then the same-end feature vector is the text feature vector in the public database, and the opposite-end feature vector is the image feature vector in the public database, etc. This application does not impose any limitations on this.

[0081] Alternatively, the similarity between a unimodal feature vector and its counterparts can be determined using distance formulas, such as Euclidean distance or Manhattan distance. This application does not limit this approach.

[0082] Therefore, in this embodiment, the single-modal feature vector can be matched with the same-end feature vectors in the public database to determine the similarity between the single-modal feature vector and each same-end feature vector. Generally, a higher similarity indicates that the single-modal feature vector and the same-end feature vector are more consistent and have smaller differences, while a lower similarity indicates that the single-modal feature vector and the same-end feature vector are more inconsistent and have larger differences. Thus, the same-end feature vector corresponding to the highest similarity can be determined first, and the opposite feature vector forming a vector pair with this same-end feature vector is the opposite retrieval feature vector corresponding to this single-modal feature vector.

[0083] Optionally, when there are multiple highest similarity values, the counterpart feature vectors in the vector pairs to which the same-end feature vectors corresponding to each highest similarity value belong can be fused to generate counterpart retrieval feature vectors corresponding to the single-modal feature vectors.

[0084] For example, when the similarity between a single-modal feature vector and three similar-end feature vectors is the highest, the counterpart feature vectors in the vector pairs to which these three similar-end feature vectors belong can be fused. For example, the three corresponding counterpart feature vectors can be directly fused, or the three counterpart feature vectors can be weighted and fused according to their weights. This application does not limit this.

[0085] Therefore, in this embodiment, the counterpart feature vector can be determined from the vector pair to which the counterpart feature vector belongs based on the similarity between the unimodal feature vector and the same-end feature vector in the public database. Since the same-end feature vector is the sample in the public database that is closest to the unimodal feature vector, its corresponding counterpart feature vector also has reliability and authenticity.

[0086] Step 103: Generate a transformation feature vector based on the single-modal feature vector and the disease indication feature vector.

[0087] The disease suggestion feature vector can be a feature vector corresponding to a disease type or disease name, possessing semantic knowledge related to the disease type or name and containing contextual knowledge of a specific disease. There can be multiple disease suggestion feature vectors, each corresponding to a different disease category. Therefore, in this embodiment, a single-modal feature vector can be used as a query to actively search for its own disease-related context within the aforementioned disease suggestion feature vectors. For example, if the single-modal feature vector contains semantic knowledge related to "pulmonary edema," the query can activate the disease suggestion feature vector related to "pulmonary edema" through attention weights, thereby strengthening the prediction signal for that disease. The strengthened and activated single-modal feature vector is then determined as the transformation feature vector.

[0088] Optionally, the unimodal feature vector can be matched with each disease suggestion feature vector to determine the matching degree between the unimodal feature vector and each disease suggestion feature vector. Based on the matching degree, the unimodal feature vector is updated to generate a target unimodal feature vector. Then, the target unimodal feature vector is processed to generate a transformed feature vector.

[0089] The system includes multiple disease-indicating feature vectors. A single-modal feature vector can be matched with each disease-indicating feature vector individually to determine the matching degree between the single-modal feature vector and each disease-indicating feature vector. Then, disease-indicating feature vectors with high matching degrees can be fused with single-modal feature vectors to update the single-modal feature vector, thereby generating the target single-modal feature vector. Alternatively, a threshold can be pre-set, and one or more disease-indicating feature vectors with matching degrees greater than the threshold can be fused with single-modal feature vectors to update the single-modal feature vector, thereby generating the target single-modal feature vector. This application does not limit the scope of this method.

[0090] Optionally, when the single-modal feature vector includes multiple sub-feature vectors, the sub-feature vectors can be matched with each disease prompt feature vector separately to determine the matching degree. Then, an attention weight matrix can be generated based on the matching degree, and the disease perception feature vectors associated with each disease prompt feature vector can be fused based on the attention weight matrix to generate the target sub-feature vector. Then, the single-modal feature vector is updated based on the target sub-feature vector to generate the target single-modal feature vector.

[0091] In this context, a sub-feature vector can be understood as a local feature vector representing a different region or location within a single-modality feature vector. All of these sub-feature vectors constitute the single-modality feature vector. Each sub-feature vector is then matched against each disease-suggestion feature vector to determine the degree of matching between them. This degree of matching can then be used as a weight value, or it can be further processed, such as through normalization or standardization, to generate an attention weight matrix. This attention weight matrix characterizes the degree of matching between each sub-feature vector and each disease-suggestion feature vector.

[0092] Furthermore, each disease-hinting feature vector has an associated disease-perceived feature vector, which contains relevant semantic information about the disease. Therefore, in this embodiment, after determining the attention weight matrix, the disease-perceived feature vectors associated with each disease-hinting feature vector can be weighted and fused according to the weight values ​​in the attention weight matrix to generate corresponding target sub-feature vectors. These target sub-feature vectors can then be fused with the unimodal feature vectors. For example, the target sub-feature vector can replace the corresponding sub-feature vector to update the unimodal feature vector; alternatively, the target sub-feature vector can be fused with its corresponding sub-feature vector separately to update the unimodal feature vector, and the updated result is determined as the target unimodal feature vector.

[0093] Therefore, in this embodiment, by matching multiple sub-feature vectors corresponding to the single-modal feature vector one by one with the disease indication feature vector, and generating an attention weight matrix based on the matching degree, the disease perception feature vector is then fused based on the attention weight matrix. This allows local regions highly correlated with a specific disease, such as lesion regions, to extract rich semantic information from their matched disease indication and disease perception feature vectors, achieving precise enhancement. Regions unrelated to the disease, such as normal tissue background, are not incorrectly enhanced, effectively avoiding noise introduction and improving the accuracy and reliability of the target sub-feature vector generation. Subsequently, updating the single-modal feature vector based on the target sub-feature vector ensures that the generated target single-modal feature vector retains key information from the original data while possessing strong disease semantic perception capabilities through semantic enhancement.

[0094] Optionally, after determining the target single-mode feature vector, the target single-mode feature vector can be input into the cross-modal converter to generate a transformed feature vector through the cross-modal converter.

[0095] The cross-modal converter can be used to convert the current single-modal feature vector into the transformed feature vector corresponding to the missing modality. The cross-modal converter can be a one-dimensional convolutional layer, a fully connected network, or a recurrent neural network, etc. This application does not limit it in this regard.

[0096] Optionally, after inputting the target single-modal feature vector into the cross-modal converter, it can be further transformed into a fixed-length vector through adaptive pooling. Thus, through one-dimensional convolution and adaptive pooling, the transformed feature vector r of the missing modality can be generated while fully preserving the key information of the original features. t :

[0097]

[0098] Where Pool1D(·) represents the adaptive pooling operation, Activation(·) represents the activation function of the Gaussian error linear unit, and Conv1D i→t (·) indicates one-dimensional convolution processing. This represents the target single-modal feature vector.

[0099] Step 104: The peer-to-peer retrieved feature vector and the transformed feature vector are fused to generate the target peer-to-peer feature vector.

[0100] Once the peer retrieval feature vector and the transformed feature vector are determined, they can be fused. There are several ways to fuse them, such as directly concatenating them or fusion by weighting them according to their weights to generate the target peer feature vector.

[0101] Optionally, a dual-path fusion module (DPFM) can be used to fuse the peer-to-peer retrieval feature vector and the transformed feature vector. Since the peer-to-peer retrieval feature vector retrieved from the public database captures features specific to a particular modality, the transformed feature vector generated by the cross-modal converter retains the original semantic features present in the available modalities. To reconcile these complementary advantages, the transformed feature vector r can be... t Retrieved counterpart retrieval feature vector The target end feature vector can be generated by linear combination as follows:

[0102]

[0103] Here, λ is a hyperparameter, also known as a weight, which reflects the relative contribution of each part. By adjusting λ, the confidence of the model in the converter and the retrieved data can be adjusted.

[0104] Therefore, in this embodiment, the dual-path fusion module can be used to fuse the transformed feature vector with the peer retrieval feature vector to generate the target peer feature vector corresponding to the missing modality. This combines the original information of the currently available modality data with the relevant knowledge of similar data in the public database, thereby significantly reducing noise accumulation in the missing modality filling process and improving the accuracy and reliability of the target peer feature vector without adding additional labeled data.

[0105] Step 105: Input the single-modal feature vector and the target counterpart feature vector into the encoder of the initial global model so that the disease prediction type can be determined after processing by the classifier.

[0106] The encoder of the initial global model is a trainable encoder, which can be initialized by the server based on a pre-trained ViLT model; the classifier is also a trainable module, which can be initialized by the server through a normal distribution or other methods and then sent to the client.

[0107] Step 106: Based on the difference between the disease prediction type and the disease labeling type, train the encoder and classifier of the initial global model to generate the target global model.

[0108] The difference between the disease prediction type and the disease label type output by the initial global model can be determined using a loss function. For example, the cross-entropy loss function can be used to calculate the loss value between the two; the smaller the loss value, the better the performance of the initial global model. After determining the loss value, the gradient of each trainable parameter in the initial global model, such as the gradients of the weights and biases of the encoder and classifier, can be calculated using the backpropagation algorithm, and then each parameter is updated separately. This process is repeated multiple times on the local client, that is, through multiple iterations, until the initial global model converges or reaches the preset number of training epochs. Finally, a target global model with improved performance on the local client is generated, providing a foundation for subsequent federated learning.

[0109] Step 107: Send the target global model to the server so that the server can generate a global model based on the target global model sent by each client.

[0110] Understandably, local clients can send the trained target global model to the server. After receiving the target global models sent by each client, the server can aggregate the target global models and generate a new global model through a federated averaging algorithm.

[0111] Optionally, to ensure the accuracy of aggregation, the weight of the target global model in the aggregation can be determined based on the number of samples in each client's local training dataset. For example, the larger the amount of training data a client has, the greater its weight in the model aggregation process. Thus, by using a weighted average aggregation method, the global model can fully absorb the data distribution characteristics of each client, making it more accurate and reliable. Afterward, the server can distribute the newly generated global model to each client to start the next round of training, and then repeat the above process until the final generated global model converges.

[0112] The training method for cross-modal federated learning healthcare models provided in this application can be applied to any training scenario for federated learning healthcare models. The following section combines... Figure 2 The training process of the cross-modal federated learning healthcare model provided in this application is briefly described.

[0113] It is understood that the training method for the cross-modal federated learning healthcare model provided in this application can also be called the retrieval-augmented cross-modal converter tuning (RACCT) framework. The training phase of the RACCT framework can be divided into three steps: initialization and distribution of the initial global model, local training, and aggregation and reallocation.

[0114] likeFigure 2 As shown, the initial global model is first initialized and distributed, following standard federated learning specifications. The initial global model parameters are initialized on the server side, and then the server distributes them to all clients for local training. Figure 2 Step 1: Distribute the global model. To ensure the accuracy and reliability of training, the frozen feature extraction layer ε can be obtained by initializing the pre-trained ViLT. emb and trainable ViLT Transformer encoder ε enc And initialize the trainable classifier using a normal distribution. Then the initial global model is distributed to the client.

[0115] Subsequently, for each local client k, if the medical data is multimodal, i.e., the medical data is an image-text pair, then the image-text pair can be fed into the frozen feature extraction layer ε. emb In the process of generating medical text feature vectors r t and medical image feature vector r i Then, the medical text feature vector r t and medical image feature vector r i ViLTTransformer encoder ε enc and classifier The processing outputs the final disease prediction type.

[0116] Additionally, for unimodal data, the problem of missing modalities can be addressed using a cross-modal converter with disease-aware prompt (CCDP).

[0117] If the medical data is monomodal, such as consisting only of medical image data or only of medical text data, this monomodal medical data can be denoted as D. u =(x i The input (y) is fed into a frozen feature extraction layer to process the medical data and output a single-modal feature vector. Where n is the length of the single-modal feature vector sequence. Additionally, all disease categories, such as "atelectasis," "cardiac hypertrophy," "pulmonary consolidation," and "pulmonary edema," along with their corresponding task-specific contextual knowledge and descriptive information, can be input into the frozen feature extraction layer to generate a disease-aware prompt, including a disease prompt feature vector and a disease-aware feature vector, which can be denoted as... Where C represents the number of disease categories, and d represents the hidden dimension of the disease perception cue representation.

[0118] Subsequently, in the Cross Attention module, the unimodal feature vector can be used as the query Q, and the disease suggestion feature vector and the disease perception feature vector can be used as the key K and value V, respectively, to suggest the representation of the existing modality. While retaining the key information in the original data, it focuses on key fine-grained features in the representation space and outputs the target unimodal feature vector.

[0119]

[0120] in, Let r represent the target unimodal feature vector, Attn() represent the attention function, and r i f represents a single-modal eigenvector. Q f represents the projection function of the query. K The projection function for the key, f V P represents the projection function of the value. I This represents the disease suggestion feature vector, where the key is the disease suggestion feature vector and the value is the disease perception feature vector. d represents the hidden layer dimension.

[0121] Then, a cross-modal converter is used to convert it into the transformation feature vector corresponding to the missing modality in order to fill in the missing modality.

[0122] Alternatively, based on a single-modal feature vector, the most similar end-to-end feature vector can be selected from a public database to determine the corresponding end-to-end retrieval feature vector, i.e. Figure 2 The ViLT Transformer encoder uses a distance-based retrieval system. The transformed feature vector and the peer retrieval feature vector are then fused in a dual-path fusion module (DPFM) to generate the final target peer feature vector corresponding to the missing modality. The single-modality feature vector and the target peer feature vector are then input into the ViLT Transformer encoder ε. enc and classifier The system can output the final disease prediction type.

[0123] The client can then train the encoder and classifier of the initial global model based on the difference between the disease prediction type and the disease labeling type to generate the target global model. For single-modal clients that only process single-modal data, it can be done as follows: Figure 2 Step 2, shown in the left-hand section, involves local training of a single-modal client. For multimodal clients processing multimodal data, the following can be used: Figure 2Step 2, shown on the right side, involves local training of the multimodal client. Afterward, the client can upload the generated target global model to the server. Figure 2 Upload local models in the process.

[0124] Optionally, during the training of the ViLT Transformer encoder, low-rank adaptation (LoRA) fine-tuning can be used, or other methods can be used, which are not limited in this application.

[0125] The server can aggregate the target global model uploaded by each client according to the federated averaging algorithm, that is... Figure 2 The local models in the dataset are aggregated to generate a global model, which is then distributed to each client to begin the next round of training. This process can be repeated until the generated global model converges.

[0126] Figure 2 The unimodal medical data on the left consists of medical image data and missing medical text data. Figure 2 The right side contains multimodal medical data, which includes both medical image data and corresponding medical text data.

[0127] The following section compares the performance of the RACCT method provided in this application with four other methods—FedFeatGen, CAR-MFL, RAGPT, and MoMKE—in medical tasks.

[0128] First, a pre-trained ViLT model was used as the initial global model. Three publicly available datasets—MIMIC-CXR, NIH Open-I, and CheXpert—were used as three clients in the joint learning environment. AUROC was used as the evaluation metric. The best scores on the datasets are shown in bold. In Table 1, "M" represents a multimodal client with both image and text data, and "U" represents a single-modal client with only image data. For the global test set, text modality data was randomly dropped with a probability of η%.

[0129] Table 1. Performance Comparison Results of RACCT with Four Baseline Models

[0130]

[0131] As shown in Table 1, RACCT outperforms all competing baselines on the test dataset in disease prediction tasks across different datasets, validating the effectiveness of the RACCT method.

[0132] In this embodiment, unimodal medical data can first be input into the feature extraction layer of an initial global model to generate unimodal feature vectors. The unimodal medical data can be medical image data or medical text data. The initial global model is initialized by the server and then sent to the client. Subsequently, based on the unimodal feature vectors, a public database can be traversed to determine the corresponding peer retrieval feature vectors. The public database includes vector pairs consisting of image feature vectors and text feature vectors. Then, based on the unimodal feature vectors and disease prompt feature vectors, a transformation feature vector is generated. The peer retrieval feature vector and the transformation feature vector are then fused to generate the target peer feature vector. The unimodal feature vector and the target peer feature vector are then input into the encoder of the initial global model. After processing by the classifier, the disease prediction type is determined. Then, based on the difference between the disease prediction type and the disease labeling type, the encoder and classifier of the initial global model are trained to generate a target global model. The target global model is then sent to the server so that the server can aggregate and generate a global model based on the target global models uploaded by each client. Therefore, for single-modal medical data lacking another modality, disease-indicating feature vectors can be used to generate transformation feature vectors that retain key semantic knowledge of the existing modality. These transformation feature vectors can then be fused with peer-to-peer retrieval feature vectors determined through public database searches to generate target peer-to-peer feature vectors. This reduces information loss and noise issues caused by filling in missing modalities, thereby improving the performance of federated learning medical models in the absence of modalities.

[0133] According to this application, a training device 300 for a cross-modal federated learning healthcare model is provided, such as... Figure 3 As shown, the device includes a first generation module 310, a first determination module 320, a second generation module 330, a third generation module 340, a second determination module 350, a fourth generation module 360, and a sending module 370.

[0134] The first generation module 310 is used to input single-modal medical data into the feature extraction layer of the initial global model to generate a single-modal feature vector. The single-modal medical data is medical image data or medical text data. The initial global model is initialized by the server and then sent to the client.

[0135] The first determining module 320 is used to traverse a public database based on the single-modal feature vector to determine the peer retrieval feature vector corresponding to the single-modal feature vector, wherein the public database includes vector pairs composed of image feature vectors and text feature vectors.

[0136] The second generation module is used to generate a transformation feature vector 330 based on the single-modal feature vector and the disease indication feature vector.

[0137] The third generation module is used to fuse the peer-to-peer retrieval feature vector with the transformed feature vector to generate the target peer-to-peer feature vector 340.

[0138] The second determining module 350 is used to input the single-modal feature vector and the target counterpart feature vector into the encoder of the initial global model, so as to determine the disease prediction type after processing by the classifier.

[0139] The fourth generation module 360 ​​is used to train the encoder and classifier of the initial global model based on the difference between the disease prediction type and the disease labeling type, so as to generate the target global model.

[0140] The sending module 370 is used to send the target global model to the server, so that the server can aggregate and generate a global model based on the target global model uploaded by each client.

[0141] Optionally, the first determining module 320 includes:

[0142] The first determining unit is used to match the single-modal feature vector with the same-end feature vectors in the public database to determine the similarity.

[0143] The first determining unit is used to determine the counterpart feature vector in the vector pair to which the highest similarity-corresponding same-end feature vector belongs as the counterpart retrieval feature vector corresponding to the single-modal feature vector.

[0144] Optionally, the first determining unit is specifically used for:

[0145] When there are multiple highest similarity values, the counterpart feature vectors in the vector pairs to which the same-end feature vectors corresponding to each highest similarity value belong are fused to generate the counterpart retrieval feature vector corresponding to the single-modal feature vector.

[0146] Optionally, the first determining module 320 is further configured to:

[0147] The raw medical data is processed to generate triplet data, wherein the triplet data includes medical images, text reports, and disease tags;

[0148] Feature extraction is performed on the triplet data to generate vector pairs corresponding to the triplets, wherein the image feature vectors and text feature vectors in the vector pairs have semantic consistency.

[0149] Optionally, the second generation module 330 includes:

[0150] The second determining unit is used to match the single-modal feature vector with each of the disease indication feature vectors to determine the degree of matching between the single-modal feature vector and each of the disease indication feature vectors;

[0151] An update unit is used to update the single-modal feature vector according to the matching degree to generate a target single-modal feature vector;

[0152] The generation unit is used to process the target single-modal feature vector to generate a transformed feature vector.

[0153] Optionally, the update unit is specifically used for:

[0154] When the single-modality feature vector includes multiple sub-feature vectors, the sub-feature vectors are matched with each of the disease indication feature vectors to determine the matching degree.

[0155] Based on the matching degree, an attention weight matrix is ​​generated;

[0156] Based on the attention weight matrix, the disease perception feature vectors associated with each disease prompt feature vector are fused to generate a target sub-feature vector;

[0157] The single-modal feature vector is updated based on the target sub-feature vector to generate the target single-modal feature vector.

[0158] Optionally, the generation unit is specifically used for:

[0159] The target single-modal feature vector is input into the cross-modal converter, and after processing by the cross-modal converter, a transformed feature vector is generated.

[0160] The training device for a cross-modal federated learning medical model provided in this application can first input unimodal medical data into the feature extraction layer of an initial global model to generate unimodal feature vectors. The unimodal medical data can be medical image data or medical text data. The initial global model is initialized by the server and then distributed to the client. Subsequently, based on the unimodal feature vectors, a public database is traversed to determine the corresponding peer retrieval feature vectors. The public database includes vector pairs consisting of image feature vectors and text feature vectors. Then, based on the unimodal feature vectors and disease indication feature vectors, a transformed feature vector is generated. The peer retrieval feature vector and the transformed feature vector are then fused to generate a target peer feature vector. The unimodal feature vector and the target peer feature vector are then input into the encoder of the initial global model. After processing by a classifier, the disease prediction type is determined. Then, based on the difference between the disease prediction type and the disease labeling type, the encoder and classifier of the initial global model are trained to generate a target global model. The target global model is then sent to the server, allowing the server to aggregate and generate a global model based on the target global models uploaded by each client. Therefore, for single-modal medical data lacking another modality, disease-indicating feature vectors can be used to generate transformation feature vectors that retain key semantic knowledge of the existing modality. These transformation feature vectors can then be fused with peer-to-peer retrieval feature vectors determined through public database searches to generate target peer-to-peer feature vectors. This reduces information loss and noise issues caused by filling in missing modalities, thereby improving the performance of federated learning medical models in the absence of modalities.

[0161] It should be understood that the specific features, operations, and details described herein with respect to the methods of this application can also be similarly applied to the apparatus and system of this application, or vice versa. Furthermore, each step of the methods of this application described above can be performed by a corresponding component or unit of the apparatus or system of this application.

[0162] It should be understood that the various modules / units of the device of this application can be implemented wholly or partially through software, hardware, firmware, or a combination thereof. Each module / unit can be embedded in the processor of the electronic device in hardware or firmware form or independent of the processor, or it can be stored in the memory of the electronic device in software form for the processor to call to execute the operation of each module / unit. Each module / unit can be implemented as an independent component or module, or two or more modules / units can be implemented as a single component or module.

[0163] like Figure 4As shown, this application provides an electronic device 400, which includes a processor 401 and a memory 402 storing computer program instructions. The processor 401 executes the computer program instructions to implement the steps of the aforementioned training method for a cross-modal federated learning medical model. This electronic device 400 can be broadly categorized as a server, terminal, or any other electronic device with the necessary computing and / or processing capabilities.

[0164] In one embodiment, the electronic device 400 may include a processor, memory, network interface, communication interface, etc., connected via a system bus. The processor of the electronic device 400 can be used to provide necessary computing, processing, and / or control capabilities. The memory of the electronic device 400 may include non-volatile storage media and internal memory. The non-volatile storage media may store an operating system, computer programs, etc. The internal memory can provide an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface and communication interface of the electronic device 400 can be used to connect and communicate with external devices via a network. When the computer program is executed by the processor, it performs the steps of the method of this application.

[0165] This application provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the above-described training method for a cross-modal federated learning medical model.

[0166] Those skilled in the art will understand that the method steps of this application can be performed by a computer program instructing related hardware, such as electronic device 400 or a processor. The computer program can be stored in a non-transitory computer-readable storage medium, and its execution causes the steps of this application to be performed. Depending on the context, any reference herein to memory, storage, or other media may include non-volatile or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0167] The technical features described above can be combined arbitrarily. Although not all possible combinations of these technical features are described, any combination of these technical features should be considered to be covered by this specification, provided that such combination does not contain contradictions.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A training method for a cross-modal federated learning healthcare model, characterized in that, include: Single-modal medical data is input into the feature extraction layer of the initial global model to generate a single-modal feature vector. The single-modal medical data is medical image data or medical text data. The initial global model is initialized by the server and then sent to the client. Based on the single-modal feature vector, a traversal is performed in a public database to determine the peer retrieval feature vector corresponding to the single-modal feature vector. The public database includes vector pairs consisting of image feature vectors and text feature vectors. Based on the single-modal feature vector and the disease indication feature vector, a transformation feature vector is generated; The peer-to-peer retrieval feature vector and the transformed feature vector are fused to generate the target peer-to-peer feature vector; The single-modal feature vector and the target counterpart feature vector are input into the encoder of the initial global model, and the disease prediction type is determined after processing by the classifier. Based on the difference between the disease prediction type and the disease labeling type, the encoder and classifier of the initial global model are trained to generate the target global model; The target global model is sent to the server, so that the server can aggregate and generate a global model based on the target global model uploaded by each client.

2. The method as described in claim 1, characterized in that, The step of traversing a public database based on the single-modal feature vector to determine the peer retrieval feature vector corresponding to the single-modal feature vector includes: The single-modal feature vector is matched with the same-end feature vectors in the public database to determine the similarity. The counterpart feature vector in the vector pair to which the highest similarity-corresponding same-end feature vector belongs is determined as the counterpart retrieval feature vector corresponding to the single-modal feature vector.

3. The method as described in claim 2, characterized in that, The step of determining the counterpart feature vector in the vector pair to which the highest similarity-corresponding same-end feature vector belongs as the counterpart retrieval feature vector corresponding to the single-modal feature vector includes: When there are multiple highest similarity values, the counterpart feature vectors in the vector pairs to which the same-end feature vectors corresponding to each highest similarity value belong are fused to generate the counterpart retrieval feature vector corresponding to the single-modal feature vector.

4. The method as described in claim 2, characterized in that, Before traversing the public database based on the single-modal feature vector to determine the peer retrieval feature vector corresponding to the single-modal feature vector, the method further includes: The raw medical data is processed to generate triplet data, wherein the triplet data includes medical images, text reports, and disease tags; Feature extraction is performed on the triplet data to generate vector pairs corresponding to the triplets, wherein the image feature vectors and text feature vectors in the vector pairs have semantic consistency.

5. The method as described in claim 1, characterized in that, The step of generating a transformed feature vector based on the single-modal feature vector and the disease indication feature vector includes: The single-modal feature vector is matched with each of the disease indication feature vectors to determine the degree of matching between the single-modal feature vector and each of the disease indication feature vectors; Based on the matching degree, the single-modal feature vector is updated to generate the target single-modal feature vector; The target single-modal feature vector is processed to generate a transformed feature vector.

6. The method as described in claim 5, characterized in that, The step of updating the single-modal feature vector based on the matching degree to generate a target single-modal feature vector includes: When the single-modality feature vector includes multiple sub-feature vectors, the sub-feature vectors are matched with each of the disease indication feature vectors to determine the matching degree. Based on the matching degree, an attention weight matrix is ​​generated; Based on the attention weight matrix, the disease perception feature vectors associated with each disease prompt feature vector are fused to generate a target sub-feature vector; The single-modal feature vector is updated based on the target sub-feature vector to generate the target single-modal feature vector.

7. The method as described in claim 5, characterized in that, The process of processing the target single-modal feature vector to generate a transformed feature vector includes: The target single-modal feature vector is input into the cross-modal converter, and after processing by the cross-modal converter, a transformed feature vector is generated.

8. A training device for a cross-modal federated learning healthcare model, characterized in that, include: The first generation module is used to input single-modal medical data into the feature extraction layer of the initial global model to generate single-modal feature vectors. The single-modal medical data is medical image data or medical text data. The initial global model is initialized by the server and then sent to the client. The first determining module is used to traverse a public database based on the single-modal feature vector to determine the peer retrieval feature vector corresponding to the single-modal feature vector, wherein the public database includes vector pairs composed of image feature vectors and text feature vectors; The second generation module is used to generate a transformation feature vector based on the single-modal feature vector and the disease indication feature vector; The third generation module is used to fuse the peer-to-peer retrieval feature vector with the transformed feature vector to generate the target peer-to-peer feature vector; The second determining module is used to input the single-modal feature vector and the target counterpart feature vector into the encoder of the initial global model, so as to determine the disease prediction type after processing by the classifier; The fourth generation module is used to train the encoder and classifier of the initial global model based on the difference between the disease prediction type and the disease labeling type, so as to generate the target global model. The sending module is used to send the target global model to the server, so that the server can aggregate and generate a global model based on the target global model uploaded by each client.

9. The apparatus as claimed in claim 8, characterized in that, The first determining module includes: The first determining unit is used to match the single-modal feature vector with the same-end feature vectors in the public database to determine the similarity. The first determining unit is used to determine the counterpart feature vector in the vector pair to which the highest similarity-corresponding same-end feature vector belongs as the counterpart retrieval feature vector corresponding to the single-modal feature vector.

10. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the training method for a cross-modal federated learning medical model as described in any one of claims 1-7.

Images